Tuesday, 2022-06-28

*** rlandy is now known as rlandy|out02:43
*** jm1|ruck is now known as jm1|rover06:18
jm1Guten Morgen :)06:19
jm1akahat|ruck: Hello, mr ruck :) How's CI?06:19
*** amoralej|off is now known as amoralej06:46
akahat|ruckjm1, o/07:04
akahat|ruckjm1, for now looks good.. paining patches got merged.. trying to get job green run.07:04
chandankumarakahat|ruck: jm1 if you see any post failure in tripleo jobs today, please let me know07:20
akahat|ruckchandankumar, okay.07:21
dpawlikarxcruz, rlandy|out, marios: Hello folks. I would like to ask if we can move forward with decommissioning old registry and move to quay.rdoproject.org07:31
chandankumardpawlik: I think we need to wait till we decomission victoria branch07:35
chandankumarwhich is in progress07:35
* akahat|ruck lunch07:39
*** jpena|off is now known as jpena07:42
dpawlikack07:45
mariosdpawlik: o/ thanks chandankumar victoria jobs should be gone in the next couple weeks07:59
*** undefined_ is now known as Guest350107:59
mariosdpawlik: chandankumar: added note in todays community call https://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg?view#2022-06-28-Community-Call 08:01
dpawlikthanks08:06
chandankumarmarios: thanks!08:08
* pojadhav stepping out for few hrs08:58
Tenguchandankumar: heya! sorry for re-asking - do you have any plan in getting the ubi9 molecule work backported to wallaby? imho, it should be, because it already helped tracking issues that may affect osp-17 downstream - and we'd need that work downstream anyway, since we're running osp-17 on el9 only09:09
chandankumarlet me check the list of patches to backport09:11
chandankumarTengu: I think we can do that https://review.opendev.org/q/topic:unified_mol_config09:12
chandankumarwe need to backport 8 patches approx and then we are done09:12
Tenguchandankumar: would be really, really good.09:12
Tenguchandankumar: and the ones I'm working on for the tripleo-ansible corrections - but yeah.09:13
chandankumaryup yup!09:14
Tenguspeeaking of that...09:16
Tenguthe podman-collection will need a real update in the end.09:16
Tengunot sure that will be enough, but I hope so.09:16
Tenguat least, it shouldn't hurt.09:16
*** rlandy|out is now known as rlandy09:40
jm1akahat|ruck: any reruns in progress which are not documented in rr notes?09:40
rlandydpawlik: hi - what chandankumar and marios said ... is that ok for you?09:41
rlandyjm1: akahat|ruck: hi - let's sync09:43
rlandyjm1: akahat|ruck: https://meet.google.com/isg-voky-mbw?pli=1&authuser=009:44
dpawlikrlandy: it is :)09:44
chandankumarakahat|ruck: <13>Jun 28 10:08:10 puppet-user: Error: Evaluation Error: Operator '[]' is not applicable to an Undef Value. (file: /etc/puppet/modules/mysql/manifests/server/config.pp, line: 37, column: 8) on node undercloud.localdomain10:18
chandankumarhttps://logserver.rdoproject.org/88/839688/91/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001/cbefef1/logs/undercloud/home/zuul/undercloud_install.log.txt.gz10:18
chandankumaris this a known issue?10:18
akahat|ruckchandankumar, https://bugs.launchpad.net/tripleo/+bug/197998510:18
chandankumarakahat|ruck: thanks!10:21
rlandychandankumar: you need container rebuild10:22
rlandyor master promo to fix that10:22
rlandycheck jobs with provider jobs should be ok10:23
rlandyjm1 is trying to get a master promo to fix the rest of jobs10:23
chandankumarrlandy: jm1 thanks! 10:23
jm1akahat|ruck, rlandy: updated rr notes with testprojects and sync'ed "known bugs" with cix cards10:23
rlandyjm1++ thank you10:23
akahat|ruckjm1, great!! thanks man!10:24
jm1akahat|ruck, rlandy: also added ykarel's kernel panic issue in periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-wallaby to rr notes in case its coming up again10:27
jm1its under "intermittent failures"10:27
ykarelnot just wallaby i have seen it in master too, multiple times10:28
jm1ykarel: ack, thanks, updated rr hackmd :)10:29
rlandyah - ok - checking how frequent10:29
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/59/417259/1/check/periodic-tripleo-ci-rhel-9-standalone-full-tempest-api-rhos-17/29023a2/job-output.txt11:06
rlandyrepo setup issue11:06
rlandyjm1: akahat|ruck: ^^ failed the 17 test projects 11:09
rlandymay be a hitch11:09
rlandypls watch and report to delivery if needed11:09
* akahat|ruck brb11:20
jm1rlandy: probably a hitch, i can download the required libxml2 package just fine11:35
jm1rlandy: akahat|ruck recheck'ed the job, lets see what happens next11:36
*** dviroel|out is now known as dviroel11:37
rlandyjm1: thanks - discussion on delivery11:37
rlandyakahat|ruck: jm1: train should promote now11:39
rlandyrechecked master kvm11:39
akahat|ruckrlandy, jm1 and c9 wallaby too.11:39
akahat|ruckTrain is promoting.11:39
rlandyakahat|ruck: for c9 wallaby looks like we are missing fs39 and fs03511:40
akahat|ruckrlandy, those are running currently.. once those passed. It will promote.11:40
rlandyack11:40
rlandyhopefully11:40
rlandyakahat|ruck: even if they fail - we should probably skip and promote 11:41
rlandylet's see what the damage is11:41
akahat|ruckrlandy, yes. that is last option.11:41
rlandybecause we need the promotion to clear the network line11:41
rlandyakahat|ruck: wrt components on c9 wallaby ...11:41
rlandyclients, network, galnce11:42
rlandyglance11:42
rlandyakahat|ruck: ^^ tracking anything else?11:42
rlandyclients and glance11:43
chandankumarrlandy: Tengu can you approve this series https://review.opendev.org/q/topic:tripleo_project_level_queue when free, thanks!11:43
rlandy^^ could clear those11:43
rlandychandankumar: ack - will w+ those11:43
akahat|ruckrlandy, yes .. i'm tracking11:43
chandankumarrlandy: thanks!11:43
Tenguthanks for checking rlandy  - I'm neck deep in that new podman-4 thing with the healthchecks...11:43
Tengudid I already say I hate podman's healthchecks?11:44
rlandydone11:45
* bhagyashris tea brb11:46
rlandyjm1: ok - we should be able to promote 17 on rhel-8 now11:46
* jm1 lunch11:52
*** amoralej is now known as amoralej|lunch12:28
akahat|ruckTrain promoted12:29
akahat|ruckrlandy, https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/f40a926/logs/undercloud/var/log/tempest/stestr_results.html.gz12:30
akahat|ruckfs 38 is failed for tempst12:31
akahat|ruckshould i rerun it or skip and promote?12:31
rlandyakahat|ruck: rerun12:31
rlandybut I may skip it12:31
akahat|ruckrlandy, okay12:31
rlandyif fs035 passes12:31
rlandyakahat|ruck: at this point, you know :)12:31
rlandyakahat|ruck: fs035 passed12:32
rlandyakahat|ruck: let's give fs039 one more shot12:32
rlandyif it does not pass this time, we skip and promote12:32
rlandyso like one more hour12:33
akahat|ruckrlandy, I'll re-triggering it.. will get something at eod.12:33
rlandyakahat|ruck: pls put in the patch to skip fs039 in the mean time12:33
rlandyso it starts running12:33
dviroelmarios: chandankumar rlandy - it seems that we might have a good topic on podified core meeting - happening now12:33
rlandywe may or may not merge it12:33
akahat|ruckrlandy, okay12:33
mariosdviroel: thanks middle of something for mixed rhel will join the bigger sync in a bit 12:34
rlandydviroel: thanks 12:35
rlandydviroel: are you going to talk about what you and sandeep have worked on?12:38
dviroelrlandy: not really, actually, we made more research than pocs, they seems to be focusing on a specific deployment12:50
rlandydviroel: well - at least what derek was telling us12:50
rlandyspeak up :)12:50
*** Guest3501 is now known as rcastillo12:59
rlandyakahat|ruck: at your eod - let's also consider the master jobs - if we shouls skip and promote there13:03
rlandywe need the new containers13:03
rlandyso probably13:03
akahat|ruckrlandy, ack13:04
*** dasm|off is now known as dasm13:04
dasmo/13:04
*** amoralej|lunch is now known as amoralej13:09
jm1dasm: good morning :)13:13
dasmjm1: good morning :)13:14
rlandydasm: so wrt thursday13:14
rlandyif I move the podified ci meeting to 2:30 utc13:14
rlandythursday13:14
rlandycan you make that?13:15
dasmrlandy: yes, it's 7:30am for me.13:15
* dasm hopes we're talking about 2:30pm utc ;)13:15
rlandynvm  emilien can't make it13:16
rlandydasm: ysandeep|PTO: chandankumar: dviroel: marios: booked next weeks operator sync with emilien on thursday13:19
rlandyonly slot I could get everyone13:19
chandankumarrlandy: ack, thanks!13:20
dviroelrlandy: ok13:20
rlandyakahat|ruck: also considering skipping failing master13:20
dasmrlandy: oh, i though we were talking about this Thursday :) Thank you13:21
rlandylet's consider after community call13:21
rlandywe were13:21
dasmok13:21
bhagyashrisTripleo CI community call in 6 mins13:24
bhagyashrisarxcruz, rlandy, marios, ysandeep, bhagyashris, svyas, soniya29, pojadhav, akahat, chandankumar, frenzy_friday, anbanerj, dviroel, rcastillo, dasm, jm1, marios, akahat|ruck 13:24
bhagyashrishttps://meet.google.com/igc-nxwj-gws13:25
bhagyashrishttps://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg?both#2022-06-28-Community-Call @ line 2913:25
* dviroel will join soon, brb13:27
mariosbhagyashris: thanks joining in a sec 13:31
*** dviroel is now known as dviroel|biab13:52
rlandymarios, chandankumar: ysandeep|PTO: dviroel|biab: pls send me your nicks on slack14:04
mariosrlandy: sec i have to login been a while14:09
ysandeep|PTOrlandy, My full name - Sandeep Yadav14:10
mariosrlandy: yah same for me full name (actually using red hat sso to sign in)14:13
mariosrlandy: see pvt about that ^ 14:14
dasmakahat|ruck++ thanks for presenting14:15
rlandyakahat|ruck: pls put in two patches14:18
rlandyon eto skip fs039 for wallaby14:18
rlandyand one to skip fs020 and internal for master14:18
akahat|ruckrlandy, okay.. on it.14:18
rlandywe'll see in 30 mins which ine we want to merge14:18
*** dviroel|biab is now known as dviroel14:21
dasmarxcruz: o/ qq. for a skiplist. if tempest has 100 tests, and 20 tests are on skiplist. Are we running 100 tests and ignoring results of 20 tests from skiplist? Or are we running 80 tests?14:22
dasmcc akahat|ruck  ^14:22
arxcruzdasm we are running 80 tests 14:24
rlandyakahat|ruck: at this point - I'm pretty into merging both and promoting14:24
dasmack14:24
rlandywe have less to lose by cleaning the component lines14:24
akahat|ruckrlandy, fs 39 is running... we can wait for few mins.. to check it's results14:25
dasmarxcruz: so we still need to run periodic jobs to verify skiplist tests. so next step would be to revive periodic job.14:25
akahat|ruckhttps://review.rdoproject.org/zuul/stream/af57357adf66424587cc64f0dd230cf9?logfile=console.log14:25
rlandyakahat|ruck; and wallaby clients should also promote: https://review.rdoproject.org/zuul/stream/5bf9b233f99c47a384c65f65b8c8249b?logfile=console.log14:30
rlandyso only network left after we do the main promotion14:30
rlandy16.2 and 17 promoted14:31
rlandyakahat|ruck: jm1: ^^14:31
rlandywe;re getting there14:31
akahat|ruckcool.14:31
rlandystill ned 17 on rhel-9 - waiting on updatde FDP there14:31
arxcruzdasm so we need to work on this: https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/rrcockpit/files/mariadb/run.sh#L80-L126=14:35
arxcruzto update the jobs that we want to read the skip list14:36
arxcruzdasm here's the go code that handle that https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/rrcockpit/files/mariadb/skiplist.go14:36
arxcruzdasm but we also have a python version https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/rrcockpit/files/mariadb/skiplist.py14:36
arxcruzi use the go code because it's much more faster than the python version, it's like go code run in 14 secondes while python run in almost one minute 14:37
dasmhow read_skipped is triggered?14:37
arxcruzdasm but since it was me who was working on that, i choose the go one14:37
jm1rlandy: good good :)14:37
arxcruzdasm https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/rrcockpit/files/mariadb/run.sh#L172=14:38
dasmarxcruz: how `run.sh` is triggered?14:38
dasmis it some cron job?14:38
akahat|ruckrlandy, fs020 where it is failing.. i didn't see anything related to fs20 on downstream.14:38
arxcruzdasm https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/rrcockpit/files/mariadb/Dockerfile#L15=14:39
akahat|ruckcould you please point me there?14:39
dasmarxcruz: how docker is deployed? xD14:39
arxcruzdasm https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/rrcockpit/files/docker-compose.yml14:40
dasmarxcruz: how docker-compose is starting? ;)14:40
rlandyakahat|ruck: master fs20 is still running14:40
rlandyrunning tempest now14:40
arxcruzdasm the role https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/infra-setup/roles/rrcockpit14:40
arxcruzthis role runs on the cockpit 14:40
dasmarxcruz: what i mean -- how many manual steps are required to get it running on prod14:40
arxcruzdasm oh, not too much, we have ansible-pull in the cockpit vm 14:41
dasmarxcruz: is it automatically done? or does someone need to take care of that?14:41
arxcruzwhen some patch is merged, ansible-pull will get and update it 14:42
dasmack14:42
dasmthat's what i needed to know :)14:42
dasmthank you14:42
arxcruzdasm the skiplist code i show you feed this http://dashboard-ci.tripleo.org/d/3pUqDadGk/tempest-skipped-tests?orgId=114:42
arxcruzdasm but don't trust this data right now, since it's not updated with the newest jobs 14:43
dasmarxcruz: so, going back to updating the script. can we change "skiplist" to always run *all*? without need to feed it like with shell script?14:43
dasm(i can investigate it in a bit, but if you know the answer, that would be good)14:43
dasmok, i see now what you pointed to. skiplist tool does all the thing.14:45
* dviroel needs to be afk for a bit14:52
*** dviroel is now known as dviroel|afk|lunch14:52
rlandyakahat|ruck: ok - we need that patch to skip fs039 pls14:53
rlandyfor wallaby14:53
rlandyjust faied14:53
rlandyfailed14:53
akahat|ruckrlandy, okay. on it.14:53
rlandymay even be lgit bit we still need to promote now14:53
rlandyand maybe it will  get fixed with more promoted components14:53
rlandyakahat|ruck: I am preparing the master patch - pls just submit the wallaby one15:01
rlandyso the promotion can start while you are still here15:01
rlandyand we can check the promoter is ok 15:01
mariosi see arxcruz in the photo :D15:02
marioso/15:02
arxcruzwhich photo ?15:02
rlandyhttps://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43773 Temp skip fs020 and kvm-internal to promote master 15:02
mariosarxcruz: all hands presentation 15:04
mariosarxcruz: eoghan talking about berlin15:04
mariosshowed group photo arxcruz 15:04
jm1rlandy: c9 ovb fs039 in failed in akahat|ruck's rerun https://review.rdoproject.org/r/c/testproject/+/41469/8 because "puppet-user:   net-snmp-agent-libs-1:5.9.1-7.el9.x86_64: Cannot download, all mirrors were already tried without success", the next run failed on tempest.. looks like a lot of intermittent failures..15:07
rlandyjm1: ack - I asked akahat|ruck to put in a patch to skip that15:07
rlandyso we can promote15:07
rlandywaiting for that patch15:07
rlandyjm1: similar patch to promote master: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43773 Temp skip fs020 and kvm-internal to promote master [NEW] 15:08
rlandyakahat|ruck: need help with the criteria patch?15:08
rlandycan make that change now if you don't have it15:08
jm1rlandy: btw akahat|ruck's last rerun of fs39 had a POST_FAILURE 🙄15:09
akahat|ruckrlandy, https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/84798215:09
akahat|ruckrlandy, no. I'll promote it from promoter server.15:10
rlandyakahat|ruck: ok - pls go ahead15:10
rlandypls just revert afterwards15:10
jm1rlandy, akahat|ruck: eod now. please keep rr hackmd updated, then i will have a look in my morning :)15:11
rlandyjm1: have a good night15:12
rlandyakahat|ruck: sorry - wrt skip I was referring to criteria15:13
rlandyakahat|ruck: so you'd need to skip in criteria15:13
rlandyor skip on promoter server15:13
rlandyagreed?15:13
akahat|ruck:)15:14
akahat|ruckrlandy, yes. running in mins15:14
rlandysure15:14
akahat|ruckrlandy, http://promoter.rdoproject.org/promoter_logs/centos9_wallaby.log15:16
akahat|ruckpromoting15:16
akahat|ruckI'll restore promoter once c9-wallaby promoted.15:16
rlandyakahat|ruck++ thank you15:16
rlandygreat15:16
* akahat|ruck dinner back in few hours15:18
rlandyakahat|ruck: Temp skip kvm-internal to promote master  https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4377315:20
rlandyfor after wallaby promotes15:20
dasmarxcruz: i'm trying to build go package locally and i'm having issues. are you able to verify if that works for you?15:21
arxcruzdasm go run doesn't work? let me check15:21
dasmarxcruz: i can't pull dependencies15:21
dasmif i'm reading correctly *.go and *.py are different15:21
arxcruzdasm yes, they do the same, but different languages15:22
arxcruzdasm feel free to use the python one, it's slower than the go one, but well...15:22
arxcruzdasm it was more me learning go 15:22
dasmarxcruz: py has hardcoded job name: https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/rrcockpit/files/mariadb/skiplist.py#L3615:23
arxcruzdasm probably, it's been a while last time i touch this code to be honest15:24
dasmarxcruz: i would prefer to continue working on *.go but if i won't it run in reasonable time, i might be forced to restore *.py version. any objections?15:24
arxcruzdasm no, the go version because of the new changes in the go need some update to load the modules15:25
dasmi think so. hence i'm not sure how it's working :)15:25
dasmif it hasn't been updated for last few months 15:25
arxcruzdasm because we are building inside the container, we are using an old version of go 15:25
dasmah15:25
dasmi see it: https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/rrcockpit/files/mariadb/Dockerfile#L115:26
dasmso, sooner than later we might encounter issues with the docker image15:27
dasmpy3.6 is deprecated already15:27
dasmnot sure when it's gonna be pulled down15:27
arxcruzdasm go mod init skiplist15:27
arxcruzdasm go mod tidy15:27
arxcruzdasm go build15:27
arxcruzdasm profit!15:27
dasmnice arxcruz++15:28
rlandyakahat|ruck: merging https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43773 to promo master15:43
rlandycurrent-tripleo/2022-06-28 15:57 15:57
rlandywooho wallaby c9 pormoted15:58
rlandylunch - bbiab15:58
*** rlandy is now known as rlandy|afk15:59
rlandy|afkakahat|ruck: rekicked network jobs15:59
*** marios is now known as marios|out16:00
*** dviroel|afk|lunch is now known as dviroel16:23
akahat|ruckrlandy|afk, it's passed here, https://code.engineering.redhat.com/gerrit/c/testproject/+/417253 no need to skip.16:28
akahat|ruckpromoting master16:28
*** amoralej is now known as amoralej|off16:36
akahat|ruckrlandy|afk, https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43777 restore sc10-internal 16:36
*** rlandy|afk is now known as rlandy17:11
rlandyakahat|ruck: 6th time lucky :)17:12
* rlandy reverts17:12
rlandymaster promo - current-tripleo/2022-06-28 17:11 17:12
akahat|ruckrlandy, https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43777 instead of revert merge this.17:13
rlandydone17:14
rlandythanks akahat|ruck17:14
rlandyakahat|ruck: and we had a merge party in gate17:14
rlandyyay today17:14
akahat|ruckhehehe.17:15
rlandyakahat|ruck: I have the wallaby c8 and c9 network component jobs in rerun17:15
rlandywe need to kick master network and tripleo17:15
rlandydviroel: rcastillo: dasm: have a clashing meeting with review time today - pls carry on w/o me17:19
rcastillorlandy: ack17:19
*** jpena is now known as jpena|off17:28
dasmack17:31
*** rlandy is now known as rlandy|afk18:00
akahat|ruckrlandy|afk, just saw your msg.. have you kicked master network and tripleo?>18:00
akahat|rucktripleo component jobs rerun: https://review.rdoproject.org/r/c/testproject/+/4127818:07
*** akahat|ruck is now known as akahat|out18:11
dasmdviroel: o/ any idea why is it failing linters? https://review.rdoproject.org/r/c/rdo-jobs/+/43778 18:14
dasmlocally i have no issues with that. 18:15
dasm(venv) [dasm@fedora rdo-jobs]$ echo $?18:15
dasm018:15
dviroeldasm: o/ do a git diff in your local env, sometimes, linter already fixes some issues after running18:17
dviroeldasm:  Fix End of Files.........................................................Failed18:18
dviroeldasm: you have 2 blank lines an the end of the file18:19
dasmheh.18:20
dasmdviroel++18:20
dasmlocally diff shows nothing. 18:20
dviroeloh, because the linter fix these issues automatically18:23
dviroeldo you run 'tox -e linters'?18:24
dasmyes, i did18:25
dviroelok, not sure why didn't work for you18:28
dasmno worries. thanks for catching that one18:33
rlandy|afkakahat|out: just network18:45
rlandy|afktripleo will kick shortly18:45
rlandy|afk17 on 9 promoted18:59
* dasm is stepping away. doc's appointment.19:34
*** dasm is now known as dasm|afk19:34
*** dviroel is now known as dviroel|out21:19
*** rlandy|afk is now known as rlandy|out22:09

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!