Tuesday, 2022-03-08

« Monday, 2022-03-07 Index Wednesday, 2022-03-09 »
*** rlandy|ruck is now known as rlandy|out01:13
*** frenzy_friday is now known as frenzyfriday|rover03:00
*** ysandeep|out is now known as ysandeep03:45
*** soniya29 is now known as soniya29|ruck05:21
*** ysandeep is now known as ysandeep|lunch07:11
*** frenzy_friday is now known as frenzyfriday|rover07:44
*** ysandeep|lunch is now known as ysandeep07:50
*** jpena|off is now known as jpena08:33
frenzyfriday|roverhey folks, need some review on https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/40189 - remove fs39 from criteria08:37
frenzyfriday|roverbot:add review list https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4018908:37
* frenzyfriday|rover thinks bot is dead08:38
soniya29|ruckfrenzyfriday|rover, reviewed :)08:50
chandankumarfrenzyfriday|rover: commented09:09
*** frenzy_friday is now known as frenzyfriday|rover10:01
frenzyfriday|roverchandankumar, I updated https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4018910:01
*** ysandeep is now known as ysandeep|afk10:04
dpawlikfrenzyfriday|rover: hey, are you ok with https://review.rdoproject.org/r/c/config/+/39047 ?10:21
chandankumarfrenzyfriday|rover: Hello, How is the container build job doing for all releases?10:25
chandankumarI wanted to merge few patches related to container build reparenting10:26
chandankumarit might break few stuff10:26
*** frenzy_friday is now known as frenzyfriday|rover10:51
frenzyfriday|roverchandankumar, it passed for master and wallaby, checking for the rest10:51
frenzyfriday|roverdpawlik, thanks for updating https://review.rdoproject.org/r/c/config/+/39047/10:52
soniya29|ruckykarel, cs9 fs20 and cs9 full tempest scenario is failing on master and the failure are observed specifically on neutron tempest tests10:56
soniya29|ruckykarel, https://bugs.launchpad.net/tripleo/+bug/196390710:57
ykarelsoniya29|ruck, when they started to fail?10:58
soniya29|ruckykarel, after cloud update mostly10:59
soniya29|ruckykarel, last green run was seen on 3rd march10:59
soniya29|ruckafter that it is either skipping or failing11:00
ykarelsoniya29|ruck, okk will check11:05
soniya29|ruckykarel, ack11:05
ykarelsoniya29|ruck, seems it's blocking promotions too?11:05
ykareladd the tags so it lands to CIX11:05
soniya29|ruckykarel, yup11:05
dpawliknp frenzyfriday|rover. Will be nice to merge it and to switch tripleomastercentos9 also to this repository11:14
dpawlikfrenzyfriday|rover: I'm still working on opensearch on opendev, but hope soon it will be done and ready to publish so I can do some progress with quay  :)11:15
*** rlandy|out is now known as rlandy|ruck11:15
rlandy|rucksoniya29|ruck: frenzyfriday|rover: let's sync meet.google.com/vzm-nrah-qqf11:17
rlandy|ruckmarios: chandankumar: dviroel|out: ysandeep|afk: pls join if you are around11:18
mariosrlandy|ruck: ack o/ gimme 1 min11:18
*** dviroel|out is now known as dviroel11:22
*** ysandeep|afk is now known as ysandeep11:30
*** pojadhav is now known as pojadhav|brb11:32
chandankumarhttps://bugs.launchpad.net/tripleo/+bug/196367511:33
*** pojadhav|brb is now known as pojadhav11:41
* bhagyashris tea brb11:42
frenzyfriday|roverfolks, we are using topic:chasing_promotions_080322 for promotions testprojs11:45
ysandeeprlandy|ruck, soniya29|ruck We promoted 16.2 11:46
ysandeepchecking 1711:46
rlandy|ruckysandeep: nice good start11:48
ysandeeprlandy|ruck, I might kill some component lines in order to run 17 ovb jobs - Otherwise I need to wait for couple of hours - will be eod for me by then.11:48
rlandy|ruckysandeep: whatever you need to do11:48
*** ykarel is now known as ykarel|mtg12:02
rlandy|ruckcurrent-tripleo/2022-03-08 12:07  - victoria12:08
rlandy|ruckwe're getting there12:08
rlandy|ruckfrenzyfriday|rover: looks like we're missing fs001 ussuri12:10
rlandy|ruckis that in rerun somewhere?12:10
rlandy|ruckotherwise I'll testproject it12:10
frenzyfriday|roverrlandy|ruck, checking12:12
frenzyfriday|roverrlandy|ruck, https://review.rdoproject.org/r/c/testproject/+/40196 running it now12:13
rlandy|ruckfrenzyfriday|rover: thanks12:13
frenzyfriday|roverhistory is not green, checking12:13
frenzyfriday|roverinconsistent failures, lets see how the testproj goes12:19
ysandeeprlandy|ruck, let reduce 16.2 run to once a day  https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/31652012:21
rlandy|ruckysandeep: voted12:24
rlandy|ruckmerge when you are ready12:24
ysandeepthanks!12:24
*** ysandeep is now known as ysandeep|mtg12:27
rlandy|ruckysandeep|mtg: chandankumar: marios: dasm|off: rcastillo: pojadhav: akahat: arxcruz|off: soniya29|ruck: frenzyfriday|rover: dviroel: arxcruz|off: bhagyashris: invited you all to a OSP-18 planning meeting today at 3:30 UTC - this is optional ...12:33
rlandy|ruckwe will have the PMs around12:33
ysandeep|mtgrlandy|ruck, ack thanks!12:34
rlandy|ruckto explain how we do our planning 12:34
rlandy|ruckas a lagging team12:34
dviroelack12:34
akahatack12:34
bhagyashrisack12:34
rlandy|ruckTC, UA etc, are required :)12:34
ysandeep|mtgI will be there12:34
bhagyashris0/12:34
chandankumarack12:34
rlandy|ruckthere is a link to the JIRA dashboard12:35
rlandy|ruckfrom other teams planning12:35
rlandy|ruckin the meeting invite12:35
chandankumarfrenzyfriday|rover: on fs020 master and full tempest scenario failures, I am waiting for fs020 job to complete12:52
chandankumarthen I will update the bug based on my reports12:52
frenzyfriday|roverchandankumar, thanks12:53
chandankumarfrenzyfriday|rover: soniya29|ruck https://paste.openstack.org/raw/bxb3HmQlks2yHSxUN0c6/12:53
pojadhavrlandy|ruck, ack12:58
soniya29|ruckchandankumar, ack12:59
rlandy|ruckdviroel, https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/83244613:00
bhagyashrisrlandy|ruck, soniya29|ruck https://hackmd.io/P5sTPMSBREe5kBS_4jDhIg#2022-03-08-tcuapm-planning-for-sprint-5913:00
bhagyashrisUA/TC/PM sync13:01
bhagyashrischandankumar, ^13:01
bhagyashrisrlandy|ruck, https://meet.google.com/jpp-qwbd-seb?authuser=013:01
rlandy|ruckdviroel: https://logserver.rdoproject.org/55/36255/15/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-wallaby/3ef08a0/job-output.txt13:02
rlandy|rucksearch that for "cat out resolv.conf to see ending  DNS config"13:02
rlandy|ruckyou'll see it's changed13:02
rlandy|ruckie: it has the right value13:02
dviroelrlandy|ruck: ack - i see thanks - i have a node hold to debug13:12
*** ykarel|mtg is now known as ykarel13:29
bhagyashrisarxcruz, sshnaidm, rlandy, rlandy|ruck, marios, ysandeep, bhagyashris, svyas, soniya29, pojadhav, akahat, chandankumar, frenzy_friday, anbanerj,  dviroel, rcastillo, dasm, jm1, rlandy|ruck soniya29|ruck frenzyfriday|rover 13:30
bhagyashrishttps://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg?both#2022-03-08-Community-Call13:31
*** ysandeep|mtg is now known as ysandeep13:42
Tengurcastillo: do you want me to bootstrap an env for the uefi failure? or are you all good to get it?13:51
rcastilloTengu: that could be helpful. I'm deploying my own atm but I won't know if it'll get to the node provision issue yet13:54
Tengurcastillo: 'k. lemme start the NUC. guess you'd like an access to the env - care to provide me your ssh key?13:55
Tengu(public key, of course)13:55
ysandeepjm1++ 14:00
pojadhavjm1++14:00
rlandy|ruckjm1++ nicely done14:00
chandankumarthanks jm1, very useful session :-)14:00
mariosnice one jm1 :)14:01
bhagyashrisjm1, ++14:01
rcastillojm++ thanks14:02
rcastilloTengu:  https://github.com/eltrufas.keys14:02
Tengurcastillo: good - lemme add it to my env. I'll provide you the ssh command shortly.14:03
*** pojadhav is now known as pojadhav|afk14:05
Tenguhumpf. authorized_keys is terrible in ansible.14:06
rlandy|ruckchandankumar: soniya29|ruck: any news on tempest failures for master?14:13
chandankumarrlandy|ruck: bug updated with my finding https://bugs.launchpad.net/tripleo/+bug/196390714:16
chandankumarlinked with this AMQP server on standalone.ctlplane.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer14:17
chandankumarmay be we can start with pidone then to nova or neutron ?14:17
rlandy|ruckpidone maybe14:17
rlandy|ruckchandankumar: start with lmiccini14:19
rlandy|ruckchandankumar: #rhos-dev14:20
chandankumarrlandy|ruck: done14:22
rlandy|ruckchandankumar: ack - ok - ipa job about o report there as well14:23
chandankumarrlandy|ruck: https://review.rdoproject.org/zuul/status#4019514:24
*** sshnaidm is now known as sshnaidm|afk14:25
chandankumarrlandy|ruck: please merge this one https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/31593214:27
chandankumarit is tested14:27
marioschandankumar: rlandy|ruck: fyi at least some of the wallaby issues are duplicate for that bug I commented there https://bugs.launchpad.net/tripleo/+bug/1963907/comments/3 (i marked my bug duplicate just now)14:30
*** pojadhav|afk is now known as pojadhav14:32
rlandy|ruckmarios: chandankumar: so we're following the same set of tempest failures in master c9 and wallaby c9?14:33
rlandy|ruckfrenzyfriday|rover: what's going on with c8 wallaby?14:33
chandankumarrlandy|ruck: have not checked wallaby line https://bugs.launchpad.net/tripleo/+bug/196390714:34
rlandy|ruckchandankumar: from marios comment above14:34
*** dasm|off is now known as dasm14:35
dasmo/14:35
ykarelsoniya29|ruck, purshed a patch for https://bugs.launchpad.net/tripleo/+bug/196390714:35
ykareltesting it in testproject, hope it fixes it14:35
rlandy|ruckykarel++14:36
chandankumarykarel: \o, you mean to say it is not linked with load issue14:37
chandankumar?14:37
rlandy|ruckmarios: ^^14:37
rlandy|ruckmay work for wallaby as well?14:37
frenzyfriday|roverrlandy|ruck, tempest failed in periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby14:37
frenzyfriday|roverchecking details14:37
mariosrlandy|ruck: ack will cycle back... there are multiple issues for wallaby 14:39
mariosrlandy|ruck: (so far         * https://bugs.launchpad.net/tripleo/+bug/196413314:40
marioshttps://bugs.launchpad.net/tripleo/+bug/1964129 14:40
mariosrlandy|ruck: will update you later anyway 14:40
ykarelchandankumar, it shouldn't seeing the type of failures, /me reads your comments now14:41
chandankumarykarel: check #rhos-dev14:41
ykarellooking14:42
rlandy|ruckmarios: should I ping fmount with https://bugs.launchpad.net/tripleo/+bug/1964133?14:42
rlandy|ruckchandankumar: soniya29|ruck: are https://logserver.rdoproject.org/95/40195/1/check/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-master/12aaeb6/logs/undercloud/var/log/tempest/stestr_results.html.gz different failures or the same from  https://bugs.launchpad.net/tripleo/+bug/1963907?14:45
rlandy|ruckDetails: Fault: {'code': 400, 'created': '2022-03-08T14:03:44Z', 'message': "Host 'standalone-0.ooo.test' is not mapped to any cell"}. Server boot request ID: req-b3f265d0-171b-4ea4-be1b-03edf82a1600.14:45
rlandy|ruckdifferent?14:45
chandankumarrlandy|ruck: from here https://logserver.rdoproject.org/95/40195/1/check/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-master/12aaeb6/logs/undercloud/var/log/extra/errors.txt.gz I can say similar failure14:46
mariosrlandy|ruck: yes sounds good thanks 16:42 < rlandy|ruck> marios: should I ping fmount with 14:46
chandankumarAMQP server on standalone-0.ctlplane.ooo.test:5672 is unreachable14:46
ykarelchandankumar, the load should not be causing all those failures, also tempest scenarios are passing, neutron-tempest one failing let's see how test run goes14:47
rlandy|ruckchandankumar: ok - at least we are following one set ... 14:47
chandankumarykarel: ok, thanks for looking into this :-)14:47
dasmOo. My first 3 months just ended!14:51
ysandeepdasm: congrats!14:51
dasmwith all new things coming my way i thought i'm here at least 6 months already xD14:52
rlandy|ruckfrenzyfriday|rover: let's chat re: wallaby14:54
rlandy|ruckhttps://meet.google.com/xry-budm-gkc?pli=1&authuser=014:54
dasmysandeep: thanks! :)14:55
chandankumarysandeep: dviroel quick merge https://review.opendev.org/c/openstack/tripleo-ci/+/83251614:55
chandankumardasm: congratulations :-)14:55
dasmchandankumar: \o/14:56
chandankumardviroel: when you get some time , please have a look at https://review.opendev.org/c/openstack/tripleo-ci/+/832254 thanks :-)14:57
ysandeepchandankumar, I have a doubt regarding parenting - if you have 5 mins for gmeet14:57
chandankumarysandeep: sure14:58
ysandeepchandankumar, meet.google.com/dkt-bqhx-zqg14:58
chandankumarysandeep: https://code.engineering.redhat.com/gerrit/c/testproject/+/31633515:02
ysandeepchandankumar, https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/31647415:02
*** sshnaidm|afk is now known as sshnaidm15:02
chandankumarrlandy|ruck: dasm https://review.rdoproject.org/r/c/testproject/+/4019515:10
chandankumarfs039 fs064 passed15:10
dasmchandankumar: nice. openssl downgrade worked?15:10
chandankumardasm: yes yes15:10
dasmsweet!15:11
dasmso we're in a better shape now?15:11
*** ysandeep is now known as ysandeep|dinner15:14
chandankumardasm: not yet, we are hitting this bug https://bugs.launchpad.net/tripleo/+bug/1963907  currently in few of the jobs blocking promotion15:14
rlandy|ruckchandankumar: yay15:14
rlandy|ruckone step15:15
rlandy|ruckfrenzyfriday|rover: train promoted15:20
rlandy|ruckcurrent-tripleo/2022-03-08 13:01 15:20
frenzyfriday|roverrlandy|ruck, cool!15:20
rlandy|rucksoniya29|ruck: ^^15:20
rlandy|rucksource_registry.namespace": "tripleotraincentos8"15:21
rlandy|ruckyep15:21
rlandy|rucktrain, osp-16.2, victoria15:21
rlandy|ruckgetting there15:21
frenzyfriday|roverrlandy|ruck, do you have some time now before the planning to discuss tempest fail on wallaby15:22
rlandy|ruckok15:22
frenzyfriday|roverhttps://meet.google.com/xry-budm-gkc?pli=1&authuser=015:22
rlandy|rucksoniya29|ruck: is your testproject stuck on internal? 316224,115:23
rlandy|ruck9 hours15:23
soniya29|ruckrlandy|ruck, ack15:27
soniya29|ruckrlandy|ruck, yeah, seems stucked15:28
rlandy|rucksoniya29|ruck: abandon your testproject - not needed15:28
soniya29|ruckrlandy|ruck, okay15:28
rlandy|ruck16.2 promoted15:28
chandankumarrlandy|ruck: meeting time15:31
rlandy|rucksoniya29|ruck: osp-18 planning meeting15:32
rlandy|ruckdviroel: marios: rcastillo: dasm: ^^ if interested15:32
dasmright.15:32
dviroelack - joining15:32
soniya29|ruckrlandy|ruck, i am in15:33
mariosrlandy|ruck: yes thanks joining 15:42
mariosrlandy|ruck: fyi 5 incoming cix for wallaby (well 4 one is the duplicate tempest one) https://bugs.launchpad.net/tripleo/+bug/1964133 https://bugs.launchpad.net/tripleo/+bug/1964138 https://bugs.launchpad.net/tripleo/+bug/1964129 https://bugs.launchpad.net/tripleo/+bug/1964142 & marked duplicate https://bugs.launchpad.net/tripleo/+bug/1964131 15:44
mariosrlandy|ruck: please sanity check flags etc but they should join the cix board 15:44
mariosrlandy|ruck: all confirmed with multiple runs, latest yesterday and/or today 15:44
mariosfrenzyfriday|rover: soniya29|ruck: ^^ 15:44
soniya29|ruckmarios, ack15:46
frenzyfriday|rovermarios, thanks for filing the bugs! https://bugs.launchpad.net/tripleo/+bug/1964138 - I think there is the same bug already, checking if it is for a different job15:47
frenzyfriday|rovermarios, https://bugs.launchpad.net/tripleo/+bug/1964142 as well I think15:47
soniya29|ruckykarel, thanks15:54
mariosfrenzyfriday|rover: ack feel free to mark duplicate and comment etc thank you15:54
frenzyfriday|roverrlandy|ruck, https://bugs.launchpad.net/tripleo/+bug/1964150 - bug for fs001 tempest neutron read timeout16:02
frenzyfriday|roverchecking 3516:03
rlandy|ruckmarios; frenzyfriday|rover: chandankumar: thank you - reading back16:03
*** ysandeep|dinner is now known as ysandeep16:07
* soniya29|ruck leaving for today16:07
rlandy|ruckmarios: will check - thanks - just talking with fmount16:08
rlandy|ruckfrenzyfriday|rover: thanks - you can put in the testprojects before you are EoD16:11
rlandy|ruckwill watch16:11
frenzyfriday|roverrlandy|ruck, ack16:11
frenzyfriday|roverrlandy|ruck, soniya29|ruck I will be on pto tomorrow16:12
rlandy|ruckreally hoping to promote ussuri16:12
rlandy|ruckthen we are down c8, c9 wallaby and master16:12
rlandy|ruckchandankumar: are there any patches you need merged today?16:12
chandankumarrlandy|ruck: yes16:13
chandankumarrlandy|ruck: https://review.opendev.org/c/openstack/tripleo-ci/+/832254 and https://review.opendev.org/c/openstack/tripleo-ci/+/83251616:13
chandankumarrebasing one of them16:13
ysandeeprlandy|ruck, fyi.. 17 jobs are still rerunning: https://code.engineering.redhat.com/gerrit/c/testproject/+/309664 16:15
* ysandeep out for the day, folks fyi.. I will be on PTO tomorrow16:15
chandankumarrlandy|ruck: https://review.opendev.org/c/openstack/tripleo-ci/+/83115116:16
chandankumarysandeep: enjoy your pto, thanks for all the help on downstream merge reparenting :-)16:16
ysandeepchandankumar: thanks! :D16:17
chandankumarrlandy|ruck: https://review.rdoproject.org/r/c/rdo-jobs/+/40112 and https://review.rdoproject.org/r/c/rdo-jobs/+/4015216:17
chandankumarand rest I will merge tomorrow and test16:17
chandankumarand revert back for rdo16:17
chandankumarthat's it16:17
* rlandy|ruck looks16:19
rlandy|ruckchandankumar; to check ...16:19
rlandy|ruckhttps://review.opendev.org/c/openstack/tripleo-ci/+/831151 is mergeable?16:20
rlandy|ruckand16:20
rlandy|ruck https://review.rdoproject.org/r/c/rdo-jobs/+/40112 and https://review.rdoproject.org/r/c/rdo-jobs/+/40152 - can be merged independently?16:20
rlandy|ruckchandankumar: ^^?16:21
chandankumarrlandy|ruck:  https://review.opendev.org/c/openstack/tripleo-ci/+/831151 https://review.opendev.org/c/openstack/tripleo-ci/+/832254/10 and https://review.opendev.org/c/openstack/tripleo-ci/+/83251616:21
chandankumarrlandy|ruck: yes both rdo reviews are independent16:21
chandankumardoes not break anything16:21
chandankumarsee ya!16:22
rlandy|ruck"tripleoussuri" is promoting!!16:23
rlandy|ruckfrenzyfriday|rover: ^^16:23
frenzyfriday|rover\0/16:23
rlandy|ruckgetting there16:24
*** ysandeep is now known as ysandeep|out16:30
*** ysandeep|out is now known as ysandeep|PTO16:30
frenzyfriday|roverrlandy|ruck, I have added the testprojs in tripleo-ci chat16:31
rlandy|ruckfrenzyfriday|rover: awesome thanks16:39
*** dviroel is now known as dviroel|lunch16:57
frenzyfriday|roverrlandy|ruck, https://bugs.launchpad.net/tripleo/+bug/1964199 - for fs35 timeout/tempest fail16:57
frenzyfriday|roverDid not find any usefull logs though16:57
rlandy|ruckmeeting - will look later16:58
rlandy|ruckpls pass to soniya in the chat16:58
frenzyfriday|roverack16:59
* frenzyfriday|rover leaving for the day17:00
*** frenzyfriday|rover is now known as frenzyfriday|pto17:00
*** jpena is now known as jpena|off17:32
rlandy|ruckchandankumar: fyi ... https://logserver.rdoproject.org/12/40212/1/check/periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-master/3ed2023/logs/undercloud/var/log/tempest/stestr_results.html.gz17:43
rlandy|ruckno luck on ykarel's change17:43
*** dviroel|lunch is now known as dviroel17:52
dviroelrlandy|ruck: hey - do we want to maintain different LP bug for different branches?18:06
rlandy|ruckdviroel: are they different errors?18:06
rlandy|ruckwhich bug?18:07
dviroelrlandy|ruck: no - 1) https://bugs.launchpad.net/tripleo/+bug/1964138 and 2) https://bugs.launchpad.net/tripleo/+bug/196367518:07
dviroel1) is the one marios created18:07
rlandy|ruckdviroel: duping marios bug to 196367518:09
rlandy|ruckdviroel: are there other dups?18:11
* rlandy|ruck needs to looks through the list18:11
dviroelhttps://bugs.launchpad.net/tripleo/+bug/196414218:11
dviroel^ this one dasm also temporary fixed with openssl downgrade, right?18:12
dviroelsame issue then18:12
rlandy|ruckdasm: ^^ pls update the two CIX cards18:12
rlandy|ruckwith your workaround patches18:13
dviroelso there are 3 more issue in wallaby-c9: ceph, container image prepare and DNS one (debugging)18:14
rlandy|ruckceph I know about - pinged fmount18:15
rlandy|ruckimage prep I saw18:16
rlandy|ruckdviroel: are you debugging the DNS one?18:16
rlandy|rucknot sure what (debugging) means18:16
rlandy|ruckdviroel: think this is also a dup https://bugs.launchpad.net/tripleo/+bug/196414218:18
rlandy|ruckto dasm's fix18:19
rlandy|ruckworkaround18:19
rlandy|ruckdasm: also the bug needs an update with your workaround18:20
rlandy|ruckhttps://bugs.launchpad.net/tripleo/+bug/196354518:20
rlandy|ruckpls keep details together18:20
dviroelrcastillo: yes, I am debugging 18:22
dviroelrlandy|ruck: ^18:22
rlandy|ruckdviroel: dup'ed two18:23
dviroelrlandy|ruck: ack - agree18:23
rlandy|ruckso there are only two new ones for wallaby c918:23
rlandy|ruckceph one18:23
rlandy|ruckand image prepare18:23
rlandy|ruckthree are dup'ed18:23
dviroelyes18:23
rlandy|ruckdviroel: which one are you working on?18:23
rlandy|ruckdns one should be workedarounf by dasm's change18:24
dviroelrlandy|ruck: are you sure that also fixes DNS issue? openssl?18:25
dviroelis there other change?18:25
rlandy|ruckdviroel: let's clarify ...18:25
rlandy|ruckhttps://meet.google.com/hje-bygj-xyr?pli=1&authuser=018:26
rlandy|ruckhttps://review.rdoproject.org/r/c/testproject/+/4019518:26
dviroelrlandy|ruck: https://opendev.org/openstack/ansible-role-tripleo-modify-image/commit/e4c84adc36e4aadee6188778d616a39e7d9bb8f518:47
dviroelis this just for debug?18:47
rlandy|rucklooking18:47
dviroelrlandy|ruck: related to https://bugs.launchpad.net/tripleo/+bug/196412918:48
rlandy|ruckI guess to verify we actually update the containers that are relevant18:48
rlandy|ruckhmmm ... what did I break  10 months ago????18:49
dviroelthe failure is i/o timetout18:49
rlandy|ruck buildah run 18:49
dviroeltimeout*18:49
rlandy|ruckdid we get an updated buildah?18:49
rlandy|ruck"buildah run 192.168.24.1-working-container yum list installed18:50
rlandy|ruckpretty basic18:50
dviroelyep18:50
rlandy|ruckdid not get container create message from subprocess18:50
* rlandy|ruck checking buildah versions18:51
rlandy|ruckmaster works though18:51
sshnaidmrcastillo, commented on https://review.opendev.org/c/openstack/ansible-collections-openstack/+/82853919:21
rlandy|ruckchandankumar: not sure about merge order of all of https://review.rdoproject.org/r/c/rdo-jobs/+/4011319:22
rlandy|ruckcan you explain at tomorrow's review time?19:23
rcastillosshnaidm: that's a good question, I'll bring up a devstack to make sure19:23
dasmrlandy|ruck: thanks. i added extra context to openssl bugs on cix20:03
dviroelrlandy|ruck: so, wrt periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-wallaby. This file (https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-wallaby/482c76e/logs/undercloud/etc/NetworkManager/conf.d/nm-no-resolv-handling.conf.txt.gz) was responsible for disabling NM at the very beginning of 20:10
dviroelthe job20:10
dviroelrlandy|ruck: i still don't know who was creating that file and where20:10
dviroelrlandy|ruck: but it doesn't exist anymore.20:11
rlandy|ruckha20:14
rlandy|ruckand it does not exist in the failing jobs?20:14
dviroelrlandy|ruck: yes20:14
rlandy|ruckcan we put it back :)20:15
dviroelrlandy|ruck: i was trying to find out where it was added - since it is not the default behavior - disable dns 20:16
dviroelit is Glean20:21
dviroelhttps://opendev.org/opendev/glean/src/branch/master/glean/init/nm-no-resolv-handling.conf20:29
dviroeltrying that https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/832670 - testing here: https://review.rdoproject.org/r/c/testproject/+/3864621:07
dviroelade_lee_: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario001-standalone-master/82d3ffa/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz21:09
dviroelade_lee_: this is not fips21:09
ade_lee_dviroel, oh interesting -- looks like you got some more output21:17
dviroelrlandy|ruck: tempest test failing with "'message': "Host 'standalone.localdomain' is not mapped to any cell" - seems a real issue - master and wallaby21:38
dviroelnot sure if affects other branches21:38
rlandy|ruckdviroel: ack - I see that a lot21:38
rlandy|ruckcan you comment on bug pls21:38
rlandy|ruckand leave notes on tripleo-ci chat21:38
rlandy|ruckdviroel: I don;t see the image update error any more21:39
dviroelrlandy|ruck: is there a bug for it?21:39
rlandy|ruckor the ceph deploy error21:39
rlandy|ruckthere is a bug for neutron failures - let me check if it's the same21:39
rlandy|ruckdviroel: https://bugs.launchpad.net/bugs/196390721:40
rlandy|ruckdviroel: I am on a node right now21:42
rlandy|ruckwatching tempest run21:42
rlandy|ruck2022-03-08 21:41:57.920 163137 INFO paramiko.transport [-] Authentication (password) failed.21:42
rlandy|ruck2022-03-08 21:41:57.921 163137 WARNING tempest.lib.common.ssh [-] Failed to establish authenticated ssh connection to cirros@192.168.24.169 (Authentication failed.). Number attempts: 14. Retry after 15 seconds.: paramiko.ssh_exception.AuthenticationException: Authentication failed.21:42
rlandy|ruck2022-03-08 21:41:42.460 163137 INFO paramiko.transport [-] Authentication (publickey) failed.21:42
rlandy|ruck2022-03-08 21:41:42.731 163137 INFO paramiko.transport [-] Authentication (password) failed.21:42
dviroelis the SSH timeout issue that you are hitting?21:43
rlandy|ruckdviroel; ^^ looks like the error we solved with the keys21:43
rlandy|ruck38.102.83.7321:43
rlandy|ruckkeeps spinning on that21:44
rlandy|ruckhttps://logserver.rdoproject.org/15/151734d416f03f3b5c9c93b1cb999f490c49589c/openstack-periodic-integration-stable1-cs9/periodic-tripleo-ci-centos-9-scenario007-standalone-wallaby/6f7fd60/logs/undercloud/var/log/tempest/stestr_results.html.gz21:44
rlandy|ruck^^ has Details: {'code': 400, 'created': '2022-03-08T21:33:48Z', 'message': "Host 'standalone.localdomain' is not mapped to any cell"}21:44
dviroelrlandy|ruck: yes, Host is not mapped to any cell - seems a different issue from LPs21:45
dviroelfails before - to create a server21:45
rlandy|ruckhttps://bugs.launchpad.net/kolla-ansible/+bug/183500221:46
dviroelSSH timeouts looks like will need keys fix on tempest plugins21:46
rlandy|ruckdviroel: all jobs running test_network_basic_ops are hitting  "Host 'standalone.localdomain' is not mapped to any cell"}21:48
rlandy|ruck^6 probably worth a directed bug21:48
rlandy|ruckto see if we can fix that21:48
rlandy|ruckhttps://logserver.rdoproject.org/53/21853/62/check/periodic-tripleo-ci-centos-9-scenario004-standalone-master/f4a6d91/logs/undercloud/var/log/tempest/stestr_results.html.gz21:48
rlandy|ruckhttps://logserver.rdoproject.org/53/21853/62/check/periodic-tripleo-ci-centos-9-scenario004-standalone-master/f4a6d91/logs/undercloud/var/log/tempest/stestr_results.html.gz21:49
rlandy|rucksame on all21:49
rlandy|ruckdviroel: ^^ can you create a new bug for that21:49
dviroelcompute also hitting https://logserver.rdoproject.org/53/21853/62/check/periodic-tripleo-ci-centos-9-standalone-wallaby/ed691f7/logs/undercloud/var/log/tempest/stestr_results.html.gz21:49
rlandy|ruckI can try see if we can close some others21:49
dviroelrlandy|ruck: yes I can21:50
rlandy|ruckthank you21:50
rlandy|ruckhttps://logserver.rdoproject.org/15/151734d416f03f3b5c9c93b1cb999f490c49589c/openstack-periodic-integration-stable1-cs9/periodic-tripleo-ci-centos-9-scenario003-standalone-wallaby/70aca9e/logs/undercloud/var/log/tempest/stestr_results.html.gz - ssh timeout21:51
rlandy|rucksomewhere we need to fix those keys21:51
dviroelI think that will need to be fixed in tempest and their plugins - we already fixed that in manila-tempest-plugin for instance - neutron fix is in progress 21:54
dviroelthere are some specific tests that create their own flavor, server and do ssh into vms to execute tests - the fix is on ssh command21:55
rcastillorlandy|ruck: so... the centosci libvirt job uses overcloud-full because whole disk images don't work with libvirt deploys yet22:00
rlandy|ruckrcastillo: so we know that for a fact?22:00
rlandy|ruckor was that just for c8?22:00
rcastillorlandy|ruck: and uefi boot is broken for overcloud-full22:00
rlandy|ruckcorrect22:00
rlandy|ruck" whole disk images don't work with libvirt deploys yet"22:00
rlandy|ruck^^ so then there is no way to make this work22:01
rcastillorlandy|ruck: I'm trying to run a libvirt deploy right now to confirm22:01
rlandy|ruckwho confirmed ^^?22:01
rlandy|rucksteve?22:01
rcastillosteve22:01
rlandy|ruckok - then there is no way out 22:01
rlandy|ruckis there an ETA?22:01
rlandy|ruckcan we bios boot?22:01
rcastillohe suggested doing bios boot22:01
dviroelrlandy|ruck: https://bugs.launchpad.net/tripleo/+bug/196426922:03
* dviroel need to go afk22:03
*** dviroel is now known as dviroel|afk22:03
rlandy|ruckdviroel|afk++ thanks22:05
rcastillorlandy|ruck: I'll try using the whole disk images on c9 and if that fails I'll try setting boot mode to bios as as second choice22:05
rlandy|ruckrcastillo: ok - if not - we give this up until we have support23:16
rcastillorlandy|ruck: ack23:22
*** rlandy|ruck is now known as rlandy|out23:36
« Monday, 2022-03-07 Index Wednesday, 2022-03-09 »

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!