Thursday, 2022-02-24

*** rlandy|ruck|bbl is now known as rlandy|out02:06
*** pojadhav|out is now known as pojadhav|ruck02:52
*** frenzy_friday is now known as frenzyfriday|rover04:26
frenzyfriday|roverhey pojadhav|ruck, good morning. I will be away for a couple of hours from 11 IST04:27
*** soniya29 is now known as soniya29|afk04:44
*** ysandeep|out is now known as ysandeep04:53
pojadhav|ruckfrenzyfriday|rover, good morning.. ack05:26
*** ysandeep is now known as ysandeep|away05:55
*** soniya29|afk is now known as soniya2906:40
*** ysandeep|away is now known as ysandeep07:04
*** amoralej|off is now known as amoralej07:09
* frenzyfriday|rover is back07:42
ysandeepfrenzyfriday|rover: hey o/ Is security done with the reproducer env, or are they still debugging there?08:01
* pojadhav|ruck lunch08:17
frenzyfriday|roverysandeep, Hey, I think they are still debugging08:25
frenzyfriday|roverThat is the last update I see on the cix08:26
ysandeepfrenzyfriday|rover, ack08:28
*** jpena|off is now known as jpena08:34
*** ysandeep is now known as ysandeep|lunch09:07
amoralejfrenzyfriday|rover, we've moved opstools repo to use centos mirror, let us know if you observe anything abnormal09:53
amoralejrlandy|out, ^09:53
frenzyfriday|roveramoralej, thanks, I'll keep an eye10:02
*** ysandeep|lunch is now known as ysandeep10:19
*** rlandy|out is now known as rlandy|ruck11:18
rlandy|ruckamoralej: thanks - will watch11:18
rlandy|ruckfrenzyfriday|rover: pojadhav|ruck: hey - how are things?11:23
pojadhav|ruckrlandy|ruck, going good..11:23
rlandy|ruckakahat: pls bring your criteria patch up at today's scrum meeting11:23
rlandy|ruckso we can talk about it and decide how best to use it11:24
rlandy|ruckpojadhav|ruck: frenzyfriday|rover: let's sync about test skips still under investigation11:24
pojadhav|ruckrlandy|ruck, sure11:24
rlandy|ruckhttps://meet.google.com/ckn-byik-stb?pli=1&authuser=011:24
rlandy|ruckpojadhav|ruck: ^^11:24
rlandy|ruckjust waiting for frenzyfriday|rover to respond11:25
frenzyfriday|roverrlandy|ruck, yep, joining11:25
*** dviroel_ is now known as dviroel11:28
frenzyfriday|roverrlandy|ruck, pojadhav|ruck https://review.rdoproject.org/r/c/testproject/+/3972911:28
pojadhav|ruckrlandy|ruck, https://review.rdoproject.org/r/c/testproject/+/3972911:28
frenzyfriday|roverrlandy|ruck, pojadhav|ruck https://review.rdoproject.org/r/c/testproject/+/3973711:29
ysandeepbhagyashris_: hey o/ left a comment here: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/312030/3#message-fd64997c9dc894feea3b7b704073d07e267988f5 , What do you think about this?11:32
akahatrlandy|ruck, okay11:34
ysandeepfolks o/ please add in your review list: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/311972 and https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/83067311:44
frenzyfriday|roverrlandy|ruck, https://review.rdoproject.org/r/c/testproject/+/3964011:44
rlandy|ruckdviroel: hi - are we clear on https://trello.com/c/F00VnyBm/2336-cixlp1956475tripleociproa-centos9-nova-migration-breaks-because-sftp-server-is-not-part-of-the-migration-wrapper now?11:48
rlandy|ruckpatch is merged11:48
dviroelrlandy|ruck: oh, so I will trigger testproject with skiplist revert. If master works, we need to proceed with u-c updates on requirements and rdoinfo.11:50
amoralejmay i get reviews on https://review.opendev.org/c/openstack/tripleo-quickstart/+/830556 ?11:52
rlandy|ruckvoted11:55
*** bhagyashris_ is now known as bhagyashris11:57
bhagyashrisysandeep, looking ..11:57
* pojadhav|ruck need a tea break12:06
mariosfrenzyfriday|rover: pojadhav|ruck: o/ aware of that? looks like new (CI)  gate blocker https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-buildimage-overcloud-full-centos-7-train 12:13
marioshmm did we just have a train promotion? 12:18
*** amoralej is now known as amoralej|lunch12:19
frenzyfriday|rovermarios, checking12:20
mariosfrenzyfriday|rover: ack i think i found the problem 12:21
mariosfrenzyfriday|rover: but would be good to have a bug for the tracking12:21
mariosfrenzyfriday|rover: tripleo-ci gate blocker12:22
mariosfrenzyfriday|rover: also that https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-buildimage-ironic-python-agent-centos-7-train 12:22
*** sshnaidm is now known as sshnaidm|off12:23
mariosfrenzyfriday|rover: https://review.opendev.org/c/openstack/tripleo-common/+/828206/3#message-7276b58e6ab4ad8fe52190f300883666f95bb2f6 12:26
mariosfrenzyfriday|rover: if you are busy right now i can file the bug let me know?12:26
frenzyfriday|rovermarios, yeah would be great if you could file the bug :D , sorry12:28
marios frenzyfriday|rover: k np12:30
rlandy|ruckmarios: hey - thank s- we just just promote train12:33
*** ysandeep is now known as ysandeep|afk12:33
rlandy|ruckif you have a fix, great12:34
mariosrlandy|ruck: no fix revert 12:34
mariosrlandy|ruck: maybe sbaker can come up with a fix12:34
mariosrlandy|ruck: https://review.opendev.org/c/openstack/tripleo-common/+/830715 because of https://review.opendev.org/c/openstack/tripleo-common/+/828206/3#message-7276b58e6ab4ad8fe52190f300883666f95bb2f6 12:34
rlandy|ruckysandeep|afk: when you are back, pls ping re: ssl on cockpit12:34
mariosrlandy|ruck: (bug incoming sec)12:34
rlandy|ruckmarios++ thank you12:34
mariosrlandy|ruck: blocks our gate https://zuul.openstack.org/status#830132 12:34
mariosfrenzyfriday|rover: pojadhav|ruck: there https://bugs.launchpad.net/tripleo/+bug/1962168 12:36
pojadhav|ruckmarios, thank you!12:37
frenzyfriday|rovermarios, thank you!12:37
rcastillorlandy|ruck: managed to reproduce the libvirt minimal issue on ci.centos12:38
mariosnp frenzyfriday|rover it blocks my patch ;) https://zuul.openstack.org/status#83013212:38
rlandy|ruckrcastillo++12:38
rcastillothe undercloud image just plain won't even bother trying to lease an ip and I have no idea why12:38
rlandy|ruckrcastillo: can you open a bug with details12:38
rlandy|ruckrcastillo: we can try get the DF to help12:39
rlandy|ruckrcastillo: add as much detail as you can to the bug and then we can ping out and ask cloudnull and friends for help there12:40
rlandy|ruckTengu also has experience there12:40
* Tengu has no experience at all12:40
Tengu:)12:41
* Tengu knows nothing12:41
rlandy|ruckno - you're our last hope!!!12:41
Tenguwhat's the issue actually?12:41
Tengurcastillo: ah, OC provisioning issue? on cs9? with bios or uefi?12:42
rlandy|ruckTengu: this is the undercloud12:43
rlandy|ruckvirt master c912:43
Tenguoh. UC not getting network?12:43
dviroelarxcruz: hi, around?12:45
rlandy|ruckTengu: ack12:46
Tengurlandy|ruck: didn't get that one unfortunately.12:46
rlandy|rucksee logs in https://jenkins-cloudsig-ci.apps.ocp.ci.centos.org/job/tripleo-quickstart-promote-master-centos9-current-tripleo-delorean-minimal/27/console12:46
rlandy|ruckok - so you're not hitting that12:47
Tengunot on my lab, nope..12:48
*** ysandeep|afk is now known as ysandeep12:54
*** carloss is now known as carloss|afk12:56
bhagyashrisfolks 13:00
bhagyashrisscrum time13:00
bhagyashrischandankumar, rcastillo dviroel 13:01
*** amoralej|lunch is now known as amoralej13:19
chandankumarhttps://review.rdoproject.org/r/q/topic:cs9-component-jobs-audit13:24
chandankumarhttps://review.rdoproject.org/r/q/topic:cs9-component-jobs-audit13:25
chandankumarhttps://review.rdoproject.org/r/q/topic:oooq-container-updates-wallaby-components13:25
chandankumarhttps://review.opendev.org/q/topic:c9_undercloud_upgrade13:26
marioshttps://review.opendev.org/q/topic:c9_wallaby_gates13:28
marioshttps://bugs.launchpad.net/tripleo/+bug/196216813:29
marioshttps://etherpad.opendev.org/p/tripleoci-wallaby-centos-8-913:29
rcastillohttps://bugs.launchpad.net/tripleo/+bug/196217813:39
ysandeep* 311972: Add standalone multinode ipa on RHOSP17 RHEL9 | https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/13:47
ysandeep* 830673: Add openstack_git_root for Ansible repo path | https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/830673 13:47
ysandeep* 312889: Add dib_release in release file. | https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/31288913:47
ysandeep* 313687: Add 17 on rhel9 details in ruck_rover config file | https://code.engineering.redhat.com/gerrit/c/openstack/rrcockpit/+/31368713:47
bhagyashrishttps://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3879413:50
ysandeeprlandy|ruck, frenzyfriday|rover pojadhav|ruck: fyi.. steve patches merged today but looks like they broke c8 - someone created a revert - https://review.opendev.org/c/openstack/diskimage-builder/+/830717 13:52
pojadhav|ruckysandeep, i am not sure but this bug looks like related one : https://bugs.launchpad.net/tripleo/+bug/196216813:53
marioshttps://review.opendev.org/c/openstack/tripleo-common/+/828206 vs https://review.opendev.org/c/openstack/diskimage-builder/+/828617 13:56
chandankumarysandeep: https://review.rdoproject.org/r/c/config/+/3970513:59
*** dviroel is now known as dviroel|brb14:01
*** carloss|afk is now known as carloss14:22
*** dviroel|brb is now known as dviroel14:23
*** pojadhav|ruck is now known as pojadhav|brb14:28
*** pojadhav|brb is now known as pojadhav|afk14:32
* pojadhav|afk back in hour14:32
rlandy|ruckfrenzyfriday|rover: can you add ysandeep and my keys to the cockpit test server?14:32
rlandy|ruckwe need to try out downstream change14:32
frenzyfriday|roverhey soniya29 : https://trello.com/c/EsgceuAN/2342-cixlp1960310tripleociproa-compute-tests-eg-testrebuildserverwithvolumeattached-failing-with-volume-id-failed-to-reach-available - there is a requirements patch which is blocking patches fixing the tempest tests. When you get some timew could you pls look into it?14:33
frenzyfriday|roverrlandy|ruck, ack. Or lemme create a password and add to the infra doc. That would be earier14:34
Tengurcastillo: is it the latest cs9 image? maybe it has an issue? reading https://bugs.launchpad.net/tripleo/+bug/1962178 sounds weird.14:43
frenzyfriday|roverhttps://review.rdoproject.org/r/c/testproject/+/39642 is still running. I will check back in some time to get the compute logs14:48
rlandy|ruckfrenzyfriday|rover: sure - feel free to pass on 14:54
rlandy|ruckwhen you are ready14:54
chandankumarrlandy|ruck: marios I think I have got the answer for second question https://review.opendev.org/c/openstack/tripleo-ci/+/830818/4#message-03b5f9462f43fdcf122ee4ff2e60cfd9a64ae4ee14:55
rlandy|ruckmarios: frenzyfriday|rover: pojadhav|afk: https://zuul.openstack.org/status#830825 - looks like the revert will work14:55
chandankumarwe need to do little duplicay14:55
chandankumarif a project is again defined in required-projects under child job with different branch14:55
mariosack cool rlandy|ruck 14:56
chandankumarthen child job project name different branch wins14:56
marioschandankumar: i think best to post it and show us instead of trynig to describe it will be hard to visualise it ;)14:56
marioschandankumar: and a lot of effort from you to try and describe (!)14:56
chandankumarmarios: rlandy|ruck if free https://meet.google.com/bsi-ynsu-qat?authuser=0&hl=en14:57
chandankumarThen I can show14:57
rlandy|ruckchandankumar: need to join the df call now14:57
chandankumarsorry14:58
chandankumarI did something wrong14:58
chandankumarneed to do some differnet expermient14:58
marioschandankumar: k 14:58
frenzyfriday|roverrlandy|ruck, ack. Found some compute logs for the other tempest card: https://bugs.launchpad.net/tripleo/+bug/1959328/comments/614:59
* dviroel manila community meetings15:07
rlandy|ruckfrenzyfriday|rover: thank you - will check15:12
soniya29frenzyfriday|rover, can you pass me the cix link again?15:39
soniya29frenzyfriday|rover, i lost that comment15:39
rlandy|ruckysandeep: can you access the cockpit test server 15:42
rlandy|ruckI still can't15:42
ysandeeprlandy|ruck, frenzyfriday|rover that server have password based ssh disabled.15:43
ysandeepfrenzyfriday|rover, please add our keys15:43
ysandeepfrenzyfriday|rover, https://github.com/sandeepyadav93.keys15:43
ysandeepfrenzyfriday|rover, please add all the keys from https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/tenant_vars/common.yml#L31-L63 to that server.15:45
ysandeepor may be pojadhav|afk if you have access to cockpit development server15:46
rcastilloTengu: sorry for the delay, added the details you asked for in a comment on the bug15:48
Tengurcastillo: lemme check15:49
Tengurcastillo: errrr - wait, I thought it was the undercloud failing, not the OC... ?!15:50
Tengurcastillo: or is the oc-full used as undercloud as well?15:50
rcastilloThis job runs the convert-image role15:50
rcastilloTIL that exists15:50
Tenguand here I'm lost - convert-image?15:51
Tenguas said: I'm pretty sure it's in the image itself, probably some issues with the interface actual name vs expected name from nmcli15:52
rcastillohttps://opendev.org/openstack/tripleo-quickstart/src/branch/master/roles/convert-image15:52
rcastillospecifically it runs virt-customize with this https://opendev.org/openstack/tripleo-quickstart/src/branch/master/roles/convert-image/templates/convert_image.sh.j215:52
rcastillosomething's probably going wrong there15:52
Tenguah, yes, more than probably.15:53
Tenguit's creating the eth0 ifcfg files, while the kernel exposes an ens315:53
Tenguor something15:53
Tenguguess that's the issue.15:53
rcastillothose lines are only for c8 and under15:53
Tenguoh. didn't see the "if".15:54
Tenguhm. so basically.... I should be able to just check the image content.15:55
Tengubrb15:56
rlandy|ruckfrenzyfriday|rover: thanks - looking at those logs15:56
rlandy|ruckrhos-17 should promote :)15:57
chandankumarrlandy|ruck: let me know your thoughts on this https://review.opendev.org/c/openstack/tripleo-ci/+/830836/3#message-b6a2b79e39fb7df4b7a1919264d7989463e7363c once zuul shares the result15:58
rlandy|rucksure15:58
chandankumarsee ya!15:59
*** dviroel is now known as dviroel|lunvh16:03
*** dviroel|lunvh is now known as dviroel|lunch16:03
Tengurcastillo: hum, just like that, I don't see why it would fail. Getting someone from HardProv might be worth the ping.16:05
ykarelTengu, rcastillo we have seen that in past with c8, and it required cleanup of ifcfg-files16:11
ykarelmay be u already ruled out that part?16:11
rlandy|ruckmarios: hey still around?16:12
marioso/ rlandy|ruck 16:12
rlandy|ruckTengu: ykarel: rcastillo: we had that in baremetal16:13
rlandy|ruckhad to remove that interface16:13
ykarelnot in baremetal, but on undercloud16:13
rlandy|ruckwe may still have those roles somewhere16:13
rlandy|ruckmarios: hey - question about OCP cluster16:13
rlandy|ruckactually ... https://meet.google.com/miz-item-ejj?pli=1&authuser=0 - quicker16:14
ykarelhttps://bugs.launchpad.net/tripleo/+bug/1932019 was the previous bug16:15
ykarelso likely just  need to update the condition?16:15
ykarelto support centos 916:15
rcastillorlandy|ruck: ykarel: this? https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/playbooks/baremetal-prep-virthost.yml#L116 16:15
ysandeeprlandy|ruck, lets work on cockpit on Monday, its late here and frenzyfriday|rover seems away.16:16
rlandy|ruckysandeep: sure16:16
ykarelrcastillo, what ifcfg files you see at /etc/sysconfig/network-scripts/ifcfg* ?16:16
ykarelon undercloud16:16
*** ysandeep is now known as ysandeep|out16:17
rlandy|ruckmarios: lost you - joined again16:19
ykarelrcastillo, and by fixing condition i meant https://opendev.org/openstack/tripleo-quickstart/src/branch/master/roles/convert-image/templates/convert_image.sh.j2#L3216:23
rcastilloykarel: just ifcfg-lo16:23
ykarelrcastillo, ok then fix ^16:23
ykarelrcastillo, and also just check rpm -qa|grep network-scripts16:23
rcastilloahh, seems simple enough16:24
*** pojadhav|afk is now known as pojadhav|out16:25
pojadhav|outfrenzyfriday|rover, rlandy|ruck, leaving for the day... left status on tripleo-ci chat..16:26
pojadhav|outbhagyashris_, ^^16:26
frenzyfriday|roverpojadhav|out, good night :)16:29
* frenzyfriday|rover reading up16:30
*** ykarel is now known as ykarel|away16:31
frenzyfriday|roverysandeep|out, rlandy|ruck can you try ssh centos@10.0.111.235 now? Added the keys16:34
frenzyfriday|roverrlandy|ruck, https://review.rdoproject.org/r/c/testproject/+/39642 finished running. I have the node on hold. Will check tomorrow morning and get the compute logs16:36
rlandy|ruckfrenzyfriday|rover: thanks - works for me now16:40
rlandy|ruckysandeep|out, is out16:40
rcastilloykarel++ that did the trick16:40
rlandy|ruckwe will try this tomorrow16:40
frenzyfriday|roverack16:41
frenzyfriday|roverrlandy|ruck, I am leaving for the day16:41
rcastillorlandy|ruck: https://review.opendev.org/c/openstack/tripleo-quickstart/+/83087016:48
*** marios is now known as marios|out16:51
rlandy|ruckrcastillo: hmm ... why did we exclude 9 there?16:57
*** dviroel|lunch is now known as dviroel16:58
rcastillorlandy|ruck: not clear to me from the old bug and commit message17:03
rcastillothis only hits libvirt jobs afaik17:03
rlandy|ruckrcastillo: ok - well if that is unblock you, great17:05
rlandy|ruckcan you try the deploy again on your env?17:05
rcastillorlandy|ruck: sure, I already tried it17:09
rcastillogot past that point, running into some other issue now17:10
rlandy|rucklunch brb17:38
*** amoralej is now known as amoralej|off17:42
*** jpena is now known as jpena|off17:53
*** rlandy|ruck is now known as rlandy|ruck|mtg17:59
dviroelade_lee: o/18:35
dviroelade_lee: https://review.opendev.org/c/openstack/tripleo-ci/+/824479/10..11//COMMIT_MSG18:35
dviroelade_lee: "Depends=On"18:35
ade_leedviroel, :)18:35
dviroellol18:36
ade_leegood catch :)18:36
dviroelade_lee: saw when adding a new depends-on below18:37
ade_leedviroel, that should take care of scenario4 18:37
dviroelade_lee: yeah, in the end I added latest-devel container for Ceph too, but we can try again later without pointing to latest-devel18:40
dviroelwas before finding depends=on issue18:41
rlandy|ruck|mtgdropping review time - nobody there19:06
rlandy|ruck|mtgwill carry on reviews myself19:06
dviroelrlandy|ruck|mtg: oh, forgot this one19:09
dviroelmy bad19:09
rlandy|ruck|mtgno worries19:10
*** rlandy|ruck|mtg is now known as rlandy|ruck19:19
rlandy|ruckdviroel: as a manila core, what do you think of: https://review.rdoproject.org/r/q/topic:cs9-component-jobs-audit?19:27
* dviroel looking19:31
dviroelrlandy|ruck: looking into periodic-tripleo-ci-centos-9-standalone-manila-master logs - there is no manila running, so I don't understand the purposes of this job19:39
dviroelscenario004 has manila, with ceph backend and runs 1 tempest job 19:39
* dviroel 1 manila tempest job + other tempest jobs19:42
* dviroel was adding a scenario test here https://review.opendev.org/c/openstack/tripleo-ci/+/828922 - will continue that soon 19:44
rlandy|ruckok - so pls comment19:46
rlandy|ruckand we can remove19:46
dviroelack19:50
dviroeldone19:55
dviroelade_lee: there is a small difference in changelogs of certmonger (c8 x c9)20:24
dviroelade_lee: https://centos.pkgs.org/9-stream/centos-appstream-x86_64/certmonger-0.79.14-5.el9.x86_64.rpm.html  and https://centos.pkgs.org/8-stream/centos-appstream-x86_64/certmonger-0.79.13-5.el8.x86_64.rpm.html20:24
ade_leedviroel, looking 20:24
dviroelnot sure if is mandatory 20:25
ade_leedviroel, yeah -- I'm going to try to reproduce the error here this afternoon .  The change I was looking for was  Fix local CA to work under FIPS (#1954618)20:27
ade_leewhich is present in both changelogs20:27
dviroelade_lee: ok20:28
dviroelrlandy|ruck: wrt https://trello.com/c/F00VnyBm/2336-cixlp1956475tripleociproa-centos9-nova-migration-breaks-because-sftp-server-is-not-part-of-the-migration-wrapper - testing now with skiplist revert - i was chasing common component promotion - now it is in tripleo-ci-testing20:28
ade_leedviroel, did we figure out why scenario 7 is not using ecdsa?20:28
dviroelade_lee: i have a guess20:28
dviroelade_lee: https://review.opendev.org/c/openstack/tripleo-quickstart/+/830884/1/config/general_config/featureset030.yml20:29
dviroelade_lee: not all featuresets have this configuration, and it might be messing up with tempest profile overrides20:30
ade_leedviroel, ok -- featureset30 == scenrio 7?20:30
dviroelade_lee: scenario007-multinode-oooq-container uses fs03020:32
ade_leedviroel, cool - lets see if it works :)20:32
ade_leedviroel, now we just need to figure out how to make ipa not be fips-enabled20:34
dviroelade_lee: didn't take a look at yet20:34
ade_leeack20:35
dviroelade_lee: did you have that issue in C8?20:35
ade_leedviroel, looking at the c8 job --- I feel like we added something to make the ipa node not be fips enabled20:36
dviroelade_lee: 2022-02-07 23:14:12.950760 | primary | FIPS mode is enabled.20:39
dviroel2022-02-07 23:14:12.613516 | secondary | FIPS mode is enabled.20:39
dviroelc8 ^20:39
ade_leedviroel, ah, well maybe not then :/20:40
ade_leedviroel, it'll be easiest if we could just have the ipa node not have fips enabled20:41
dviroelade_lee: will take a look in details20:42
ade_leedviroel, cool thanks20:42
rlandy|ruckdviroel: ok - thanks20:48
*** dviroel is now known as dviroel|afk20:54
*** dviroel|afk is now known as dviroel23:37
*** rlandy|ruck is now known as rlandy|out23:38

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!