Monday, 2022-06-27

*** ysandeep|out is now known as ysandeep02:04
ysandeeprlandy|out, >> https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-8-bm_envD-3ctlr_1comp-featureset035-rhos-16.2&skip=0 - think we broke bm with your review -> My patches were not merged, they got blocked in gate - recheck dance ongoing 02:06
*** ysandeep is now known as ysandeep|afk05:15
akahat|ruckarxcruz, frenzyfriday o/05:46
akahat|ruckarxcruz, frenzyfriday promoter is failing with wrong credentials for qay.rdoproject.org05:47
*** ysandeep|afk is now known as ysandeep05:56
jm1good morning ci team ☺️06:28
*** jm1|ruck is now known as jm1|rover06:29
jm1akahat|ruck: hello :) what is going on?06:29
akahat|ruckjm1,  o/06:29
akahat|ruckjm1, chasing with promoter.06:30
jm1akahat|ruck: ack, rlandy left us notes and todos in rr hackmd ^^06:30
jm1akahat|ruck: i will work on those notes. maybe you want to add something for me?06:31
akahat|ruckjm1, yes. noted. 06:31
akahat|ruckjm1, gate is looks good for now.06:31
jm1akahat|ruck: ok, cool :)06:33
jm1akahat|ruck: created a new day in rr notes, copied over last friday and removed obsolete entries06:35
akahat|ruckjm1, ok06:37
*** amoralej|off is now known as amoralej07:05
*** frenzyfriday|PTO is now known as frenzy_friday07:13
frenzy_fridayakahat|ruck, checking07:13
akahat|ruckfrenzy_friday, i've updated passwords. 07:14
akahat|ruckit's working now.07:14
frenzy_fridayoh, cool!07:14
frenzy_fridayis it tagging current-tripleo on quay.rdoproj?07:14
akahat|ruckfrenzy_friday, yeah.. and pushing it to quay.io.07:32
* akahat|ruck lunch 07:32
frenzy_fridaycool07:39
*** jpena|off is now known as jpena07:43
jm1ysandeep: recheck'ed your 16.2 bm job here because it failed on ssl error which is different from errors before https://code.engineering.redhat.com/gerrit/c/testproject/+/21164308:15
ysandeepjm1, i see same issue locally http://pastebin.test.redhat.com/106101709:03
ysandeepI would ping #rhos-delivery too see if there is an ongoing issue.09:04
jm1ysandeep: see #openstack-pcci for updates ;)09:06
ysandeepack09:07
*** vkmc_ is now known as vkmc09:10
amoralejchandankumar, how are the molecule jobs on cs9? still wip?09:12
*** rlandy|out is now known as rlandy09:29
chandankumaramoralej: almost on it09:29
rlandyakahat|ruck: jm1:  hi - how are things?09:30
jm1rlandy, akahat|ruck: sync?09:30
rlandyack ... https://meet.google.com/syt-ovim-szg?pli=1&authuser=0 ... akahat|ruck; jm1 09:31
ysandeeprlandy, starting your day early today? good morning09:31
rlandyysandeep: ack - mom decided to come visit and help out while I am down and out with my knee injury09:31
rlandystarting early - taking some time off this afternoon09:31
rlandyysandeep: going to look at bm failure09:31
ysandeeprlandy, I tried to look but its failing at earlier step now - cert issue ongoing.09:32
mariosrlandy: hows the knee09:37
ysandeeprlandy, >> starting early - taking some time off this afternoon - ack, hope you recover fully soon, good to have mom's back.09:42
mariosi see thanks ysandeep so i guess not great09:48
*** ysandeep is now known as ysandeep|afk09:50
rlandyakahat|ruck: https://review.rdoproject.org/r/c/testproject/+/3625410:22
*** ysandeep|afk is now known as ysandeep10:38
rlandymarios: getting better - it's all green and purple now - so hopefully we're moving through the healing phases10:49
marios:/10:50
rlandyysandeep: asked jm1 to touch base with you - ack there is the cert issue - but there is still a legit virt issue there - maybe a reset needed on the box 10:50
ysandeeprlandy: just to confirm what do you mean by reset - reboot?10:51
ysandeepI am waiting for cert issue to resolve to troubleshoot further10:53
mariosysandeep: rlandy: maybe this is what you referring to https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43714 ? 10:54
ysandeephey o/ marios, its a different issue: https://sf.hosted.upshift.rdu2.redhat.com/logs/43/211643/146/check/periodic-tripleo-ci-rhel-8-bm_envD-3ctlr_1comp-featureset035-rhos-16.2/1cffbc0/logs/rdoci-hp-01.v100.rdoci.lab.eng.rdu2.redhat.com/home/zuul/repo_setup.log 10:57
mariosysandeep: ah ok 11:10
bhagyashrisfrenzy_friday, hey jfyi i have added story for 17.1 work https://issues.redhat.com/browse/TRIPLEOCI-1091 plz add sub story or reviews etc 11:11
reviewbotDo you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks.11:11
frenzy_fridaybhagyashris, yep, thanks. I added a hachmd to the description. Adding the sub tasks soon11:12
bhagyashrissure 11:12
rlandyysandeep: yep - or removal of that dir11:13
chandankumarrlandy: arxcruz hello11:15
chandankumarrlandy: arxcruz I think we can drop following files https://paste.centos.org/view/raw/9d36ed48 from log collection11:15
chandankumarit might help post_failures11:15
chandankumarthese seems to be unwanted files 11:15
rlandychandankumar: arxcruz: let's discuss at scrum11:16
chandankumarrlandy: sure11:16
rlandyfeel free to put up a patch in the mean time11:16
chandankumarsure11:16
ysandeepmarios, chandankumar review time11:17
mariosysandeep: sorry joining11:24
rlandyTengu: added a note on DF meeting call - CI section:  Need reviews and merge on Chandan’s patch: https://review.opendev.org/c/openstack/tripleo-ansible/+/839688 11:25
reviewbotI have added your review to the Review list11:26
rlandypls ask the team to review/merge that when ready11:26
Tengurlandy: linked to his mail earlier today pointing to the 2 LPs?11:26
Tengu(gosh, I *hate* virt-sysprep and the way it doesn't want to run script as it should....)11:27
chandankumarTengu: I have added two more lps11:28
Tenguchandankumar: ;_;11:29
Tenguchandankumar: care to list them? I'll add them to our checkpoint to raise awareness and call for help.11:32
Tenguchandankumar: on my side, I'm working on a local reproducer for that podman/container_manage thingy.11:32
chandankumarsending via email11:32
Tenguthanks!11:32
chandankumar- tripleo-ansible-centos-stream-molecule-tripleo_update_trusted_cas11:32
chandankumar  - https://bugs.launchpad.net/tripleo/+bug/197980711:32
chandankumar- tripleo-ansible-centos-stream-molecule-tripleo-modules11:32
chandankumar  - https://bugs.launchpad.net/tripleo/+bug/197980711:32
chandankumar- tripleo-ansible-centos-stream-molecule-tripleo_container_manage11:33
chandankumar  - https://bugs.launchpad.net/tripleo/+bug/197981011:33
chandankumar- tripleo-ansible-centos-stream-molecule-tripleo_redhat_enforce11:33
chandankumar  - https://bugs.launchpad.net/tripleo/+bug/197997111:33
chandankumar- tripleo-ansible-centos-stream-molecule-tripleo_collectd11:33
chandankumar  - https://bugs.launchpad.net/tripleo/+bug/197997211:33
Tenguhmmm there's a dup'11:33
Tengusame ID for the update_trusted_cas + tropleo-modules11:34
chandankumarTengu: the failures are same11:35
Tenguoh, ok11:35
chandankumarIt might be all related to podman11:35
Tengu'k11:36
Tenguso of course I took the easy one ;)11:36
rlandysoniya29: https://opendev.org/openstack/openstack-tempest-skiplist/src/branch/master/roles/validate-tempest/vars/tempest_skip.yml11:38
pojadhavrlandy, https://opendev.org/openstack/openstack-tempest-skiplist/src/branch/master/roles/validate-tempest/vars/tempest_skip.yml#L106111:39
pojadhavsoniya29, ^11:39
soniya29pojadhav, tempest.scenario.test_network_basic_ops.TestNetworkBasicOps11:40
soniya29pojadhav, marios, rlandy, https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/84735111:44
rlandyakahat|ruck: looks like network wallaby c8 is past node provision11:45
rlandyreal issue on failing master tests ... pls see https://logserver.rdoproject.org/54/36254/125/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master/1b5536d/logs/undercloud/home/zuul/undercloud_install.log.txt.gz11:45
chandankumarrlandy: arxcruz https://review.opendev.org/c/openstack/tripleo-ci/+/84777111:45
chandankumarmore files excludes11:45
rlandychandankumar: thanks11:46
arxcruzchandankumar wow, we were collecting /opt/git ?11:47
arxcruz:O11:47
chandankumararxcruz: yes11:48
soniya29rlandy, marios, we are not skipping the test - tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_router_rescheduling11:48
rlandyakahat|ruck: jm1: multiple failures on gate now11:48
rlandypls look11:48
rlandy<13>Jun 27 11:27:59 puppet-user: Error: Evaluation Error: Operator '[]' is not applicable to an Undef Value. (file: /etc/puppet/modules/openstacklib/manifests/defaults.pp, line: 9, column: 7) on node undercloud.localdomain11:49
rlandy+ rc=111:49
rlandyfailing master11:49
mariossoniya29: thank you11:51
mariossoniya29: i'll revisit on next reviews 11:52
reviewbotDo you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks.11:52
*** dviroel|out is now known as dviroel12:01
*** amoralej is now known as amoralej|lunch12:06
bhagyashrisrlandy, chandankumar updated  https://review.opendev.org/c/openstack/tripleo-ci/+/84297612:15
chandankumarbhagyashris: thanks! will take a look12:15
rlandybhagyashris: going to miss some of our scrum to join the DF call12:15
bhagyashrisrlandy, ack12:16
rlandyand talk about wallaby blockers12:16
rlandybhagyashris; will join in 512:31
bhagyashrisok12:31
rlandyjust getting cix created12:31
bhagyashrishttps://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/40210112:39
bhagyashrishttps://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/40288912:39
rlandychandankumar: ysandeep: trying to help bhagyashris get some patches through here  - pls can you vote/merge if happy ttps://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/40210112:41
chandankumarrlandy: checking downstream ones12:41
chandankumarbhagyashris: can you pass the rdo review also?12:41
*** arxcruz is now known as arxcruz|brb12:42
arxcruz|brbbrb 12:42
ysandeeprlandy, looking12:43
rlandyysandeep: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/402889 as well pls12:43
rlandyjust rebased12:43
chandankumarbhagyashris: left a comment https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/40210112:52
chandankumarjm1: please have a look https://github.com/containers/ansible-podman-collections/pull/442 when free, thanks!12:54
ysandeepbhagyashris, rlandy don't we want to create option job for ipa and multinode-container in upstream?12:56
ysandeepI don't find why we are making this exception in commit msg.12:57
*** amoralej|lunch is now known as amoralej13:04
bhagyashrisysandeep, i not sure but looks like we added option job for standalone scenarios only if team wants to add for ovb amd multinode and ipa one then i will add for those as well13:05
ysandeepbhagyashris, lets discuss this in scrum before you make any changes13:06
jm1akahat|ruck: is https://bugs.launchpad.net/tripleo/+bug/1979986 the same that rlandy submitted in https://bugs.launchpad.net/tripleo/+bug/1979985 ?13:06
jm1akahat|ruck: ah, already marked as duplicate13:07
bhagyashrisysandeep, sure13:08
rlandyakahat|ruck: jm1: ok - we have a CIX card for https://trello.com/c/roBlJfnB/2594-cixlp1979985tripleociproa-master-jobs-are-failing-undercloud-install-error-facter-error-while-resolving-custom-fact-osx-undefine13:17
jm1ysandeep: an idea for working around the ssl cert issue https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/417005 what do you think? worth it or keep it abandoned? < rlandy13:18
rlandyjm1: already abandoned13:20
rlandy??13:20
jm1rlandy: because it touches the release files and that somehow does not look like a quick fix for our 16.2 fixes.. but wanted to have ysandeep's opinion on that anyway 13:21
jm1rlandy: if it is worthwhile to follow, i can restore13:21
rlandyone sec- just testprojecting factor fix patch13:22
ysandeepjm1, you can go ahead with that but we just need it on 16.2/17 file but okay this way as well.13:22
* ysandeep notes there are some obsolete files which we should remove 13:24
jm1ysandeep: no idea which places are required so i just replaced all broken occurences of that url ^^13:24
ysandeepjm1, that's fine - I should have done the cleanup already 13:25
* ysandeep updated https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/417005 for testing13:34
jm1ysandeep: ok, thanks!13:34
dasmo/13:47
*** arxcruz|brb is now known as arxcruz13:48
rlandyjm1: akahat|ruck: ugh  - gate just reset13:53
rlandyarxcruz: chandankumar: going to miss some of scrum call to talk to DF13:54
rlandyafterwards will need to talk with you about log upload13:54
rlandypls follow #opendev13:54
chandankumarrlandy: ok13:57
rlandymutinode impacted13:58
dasmwill join standup after rbac13:59
bhagyashrisakahat|ruck,  chandankumar soniya29 14:01
bhagyashrisscrum time14:01
dviroeloh14:02
ysandeeprlandy, jm1 this worked https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/417005 , +2ed 14:03
ysandeepwe can revert once cert issue clears14:04
jm1ysandeep: great ^^14:04
ysandeepjm1, the other issue is weird because I can run those same commands manually as zuul user on the bm machine.14:08
bhagyashrishttps://review.opendev.org/c/openstack/tripleo-ci/+/84297614:08
rlandyysandeep: the cix called they said this was cleared14:12
ysandeepnice.. we can revert in that case.. /me proposing a revert14:14
jm1rlandy, ysandeep: i still have this issue, easy to test with curl -L -O https://download.devel.redhat.com/rel-eng/RCMTOOLS/rcm-tools-rhel-8-baseos.repo14:14
ysandeepjm1, its waiting for me now: http://pastebin.test.redhat.com/106112814:15
ysandeepworking*14:15
jm1ysandeep: on baremetal or from your home?14:15
ysandeepmy laptop14:15
jm1ysandeep: try from bm14:16
akahat|ruckjm1, https://review.opendev.org/c/openstack/puppet-tripleo/+/84776614:16
akahat|ruckhttps://logserver.rdoproject.org/21/846121/1/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/07b360d/logs/undercloud/home/zuul/undercloud_install.log.txt.gz14:16
akahat|ruckhttps://logserver.rdoproject.org/66/847766/3/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035/38a2802/logs/undercloud/home/zuul/undercloud_install.log.txt.gz14:16
akahat|ruckjm1, seen this before?14:16
akahat|ruckUNAUTHORIZED for quay.rdoproject.org 14:16
ysandeepjm1, worked on bm as well: http://pastebin.test.redhat.com/106112914:17
akahat|ruckand it's target registry is also not correct.14:17
akahat|ruckrlandy, ^^14:17
jm1ysandeep, rlandy: ah, now it works for me too. great :)14:18
jm1akahat|ruck: nope, not seen that before14:18
ysandeeprlandy, https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/416621 , we can merge the revert.14:19
soniya29marios, https://review.rdoproject.org/r/c/testproject/+/4010214:24
jm1ysandeep: how to "close" https://bugzilla.redhat.com/show_bug.cgi?id=2101405 ?14:24
ysandeepjm1, issue is not replicating in local tests, we can take a look together after scrum call.14:26
dasmakahat|ruck: o/ do you need a help with tempest skiplist?14:28
akahat|ruckdasm, o/ 14:29
akahat|ruckdasm, i'm trying to get some basic model working.. but it taking some time.14:30
akahat|ruckyou are welcome to contribue.t14:31
akahat|ruckcontribute*14:31
dasmakahat|ruck: do you have some patches? or design doc? 14:31
akahat|ruckdasm, yeah.. let's talk after the meeting.14:32
dasmk14:32
dasmfrenzy_friday: https://review.opendev.org/c/opendev/elastic-recheck/+/84740514:43
frenzy_fridaycool! I'll take a look, thanks14:43
ysandeepTengu, hey o/ when you have a chance today/tomorrow can we talk about your nftables work: https://issues.redhat.com/browse/TRIPLEOCI-1072 and how tripleo-ci team and help here, I see you have abandoned some of the patches.14:51
Tenguysandeep: np - basically, no more undercloud.conf param, but it's replaced by https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/84700214:54
ysandeepTengu: thanks i will go through ^^ and will ping you if something is unclear. 14:55
dasmTengu: just asked question on the review: > Is move to nftables one-direction? Are we gonna need to support nftables and iptables?14:56
Tengusure14:56
Tengudasm: imho we should drop iptables once the move to nftables is done. especially since iptables isn't iptables anymore, but a wrapper called "iptables-nft"14:56
dasmso, there is no need to test both config, iiuc?14:57
bhagyashrishttps://review.opendev.org/c/openstack/tripleo-ci/+/847771/2/toci-quickstart/config/collect-logs.yml14:57
Tengudasm: yep14:58
dasmack14:58
Tengudasm: basically, as just pushed on the change, we're removing a compatibility layer14:58
Tenguwell. not removing. bypassing is the correct word.14:58
dasmthat makes sense14:58
ykarelakahat|ruck, jm1|rover is the kernel panic issue with cirros vm a known one?14:58
ykarelKernel panic - not syncing: IO-APIC + timer doesn't work!  Boot with apic=debug and send a report.  Then try booting with the 'noapic' option.14:58
Tengudasm: `iptables' commands will stay on the system, and would allow others such as Neutron to still use it.14:58
dasmTIL about iptables-nft 14:59
Tengu:)14:59
akahat|ruckykarel, no. we didn't saw any related issue recently. Can you share logs?15:00
ykarelakahat|ruck, i see it's happening from long15:00
ykarellike some failures in https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-wallaby15:00
Tengudasm: basically, you can install iptables-legacy and switch back to actual iptables by updating the alternatives - but you'd lose all the existing rules, of course. Sooo.. iptables-save, update-alternative, and iptables-restore.15:00
ykarelmay be also see in others jobs too, i just checked ^ and -master one15:01
dasmyeah, i'm just checking: https://www.redhat.com/en/blog/using-iptables-nft-hybrid-linux-firewall15:01
Tengu^^15:02
Tenguit was a fun journey in the Firewall World15:02
akahat|ruckykarel, this issue is not consistent. Sometimes tempest tests failed with different issues.. but once we rechecked it we didn't see that issue get reproduced.15:03
ykarelmay be good to check from logstash on what all's impacted15:03
ykarelyes not consistent, but may block promotions and would need rerun etc15:04
dasmakahat|ruck: wanna quick sync?15:04
*** pojadhav is now known as pojadhav|afk15:04
ysandeepjm1, you want to meet now?15:04
akahat|ruckdasm, yeah.. give me a min.15:04
dasmk15:05
jm1ysandeep: sure15:05
ysandeepjm1, https://meet.google.com/kzv-boir-shk15:05
akahat|ruckykarel, i need to take a look in to the jobs.. 15:06
chandankumarmarios: ysandeep updated https://review.opendev.org/c/openstack/tripleo-ci/+/847771 feel free to take a look, thanks15:06
chandankumarrlandy: https://review.opendev.org/c/openstack/tripleo-ci/+/847771 please get this merged Eod15:06
ykarelakahat|ruck, ack yes please check, if logstash is working that would be easier to check with15:07
chandankumarsee ya!15:07
akahat|ruckykarel, yeah.. thanks!!15:07
akahat|ruckdasm, https://meet.google.com/zdy-vsqa-wxi15:08
rlandychandankumar: I can w+ that15:10
rlandyanyone else with issue on what we are excluding???15:10
ykarelakahat|ruck, also https://bugs.launchpad.net/nova/+bug/1939108, so have a workaround available for that issue15:15
ykarelother workaround was to use cirros uec image in past15:15
marioswill do chandankumar 15:25
* dviroel lunch15:27
*** dviroel is now known as dviroel|lunch15:27
dasmakahat|ruck: https://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg?both#2022-06-28-Community-Call15:31
*** marios is now known as marios|out15:39
dasmbrb15:57
ysandeeprlandy: you are genius, reboot fixed 16.2 bm job failure16:02
ysandeepjm1++ 16:03
rlandyysandeep: happy to hear16:03
ysandeeprlandy, jm1 bm job in rerun https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status#211643 16:05
* ysandeep out16:09
*** ysandeep is now known as ysandeep|PTO16:09
jm1ysandeep: awesome, thank you and have a nice pto :)16:18
jm1rlandy: updated rr notes, bz bugs and cix cards with current status on our two periodic-tripleo-ci-rhel-8-bm_envD-3ctlr_1comp-featureset035-rhos-16.2 bugs16:21
jm1akahat|ruck: ^16:21
jm1rlandy, akahat|ruck: eod now16:22
rlandyjm1: thank you16:22
*** dviroel|lunch is now known as dviroel16:37
*** amoralej is now known as amoralej|off16:51
rlandylunch brb17:17
*** Guest3279 is now known as rcastillo17:20
*** pojadhav|afk is now known as pojadhav17:36
rlandydasm: are you out on thursday?18:36
rlandyyour calendar is marked busy18:36
*** rlandy is now known as rlandy|afk18:40
dasmrlandy|afk: i'm taking training, but i'm not fully busy18:54
*** jpena is now known as jpena|off19:06
*** rlandy|afk is now known as rlandy19:23
* dviroel out 21:40
*** dviroel is now known as dviroel|out21:40
* dasm too is leaving22:16
dasmtake care!22:16
dasmo/22:16
*** dasm is now known as dasm|off22:16
* rcastillo out as well o/22:30

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!