Wednesday, 2022-03-23

*** rlandy|bbl is now known as rlandy|out02:05
*** pojadhav|out is now known as pojadhav|rover04:25
*** bhagyashris_ is now known as bhagyashris04:55
*** ysandeep|out is now known as ysandeep04:56
mariosarxcruz|ruck: pojadhav|rover: o/ morning any urgent things need reviews for blockers/bugs? 07:03
pojadhav|rovermarios, \0 good morning, not atm on downstream side.. :)07:05
marioso/ k pojadhav|rover 07:07
*** pojadhav|rover is now known as pojadhav|lunch07:13
*** ysandeep is now known as ysandeep|afk07:19
mariosarxcruz: o/ morning let us know if you need some reviews for urgent bugs/blockers etc or anything else07:26
*** amoralej|off is now known as amoralej07:32
*** pojadhav|lunch is now known as pojadhav|rover07:39
*** ysandeep|afk is now known as ysandeep08:04
*** arxcruz is now known as arxcruz|ruck08:06
arxcruz|ruckmarios ack 08:06
arxcruz|ruckmarios chandankumar https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/40826 I'm removing fs035 from wallaby centos 9 and 8 so we can promote 08:11
arxcruz|ruckit's randomly failing on tempest tests 08:11
arxcruz|rucksince we have the program call today, i think it's better to promote it 08:11
arxcruz|ruckpojadhav|rover ^08:13
mariosarxcruz|ruck: k looking - do we at least have a bug about it ? 08:13
arxcruz|ruckmarios no, it's being failing for a while, we are skipping mostly of the time 08:14
arxcruz|ruckmarios it's random timeouts in random tempest test08:14
arxcruz|rucknot a specific one 08:14
arxcruz|ruckfor example Details: (FloatingIPsAssociationTestJSON:setUpClass) Server 71187b41-7ff4-4390-899c-37cc6980f81f failed to reach ACTIVE status and task state "None" within the required time (300 s). Server boot request ID: req-b108d8b0-2105-4bdf-8c75-ba3a22e3bb8d. Current status: BUILD. Current task state: spawning.08:14
ysandeepfolks o/ anyone knows if we have merged some related to whole_disk_images yesterday? I am hitting 'whole_disk_images' is undefined" today in downstream bm job.08:16
mariosarxcruz|ruck: k then you should probably file one (hasn't been green since 15th on periodic)08:16
mariosarxcruz|ruck: even if it is just "fails on various tempest tests" for now until we can figure something more specific out08:16
arxcruz|ruckmarios ok, meanwhile can you ack on the skip ?08:16
arxcruz|rucki'll open as soon as i checkk the other branches 08:16
mariosysandeep: i know rcastillo was doing something around centosci there but nto sure if something merged check quickstart/extras repos commits? 08:17
mariosarxcruz|ruck: ack08:17
arxcruz|ruckack as in review the patch :D08:17
mariosarxcruz|ruck: yea i am commenting... did you post testproject already ? same results? 08:18
arxcruz|ruckmarios i post a test project now 08:18
ysandeepmarios: ack, thanks!08:18
arxcruz|ruckhttps://review.rdoproject.org/r/c/testproject/+/3797308:18
arxcruz|ruckmarios oh, rlandy|out left a comment for us on hackmd08:20
arxcruz|ruckhttps://hackmd.io/wjL3Tpu8RhqiFWdKoGTOSw?both08:20
arxcruz|ruckmarios 08:20
arxcruz|ruck* wallaby c8  - https://review.rdoproject.org/r/c/testproject/+/35663 in rerun - missing only periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-wallaby. Ask amol to skip on promoter and promote this hash08:21
arxcruz|ruckjust notice now 08:21
arxcruz|ruck:) 08:21
mariosarxcruz|ruck: k can you also immediately post the followup to re-add this so we don't forget it? (you can workflow -1 that until the promotion happens for example)08:21
arxcruz|ruckmarios i can use the revert button once it get merged :) 08:22
mariosarxcruz|ruck: k just don't want it to be forgotten since you are switching shift tomorrow08:22
arxcruz|ruckmarios fear nothing, i'll do it :) 08:22
mariosquestion arxcruz|ruck pojadhav|rover did you guys split it upstream arxcruz|ruck and downstream pojadhav|rover ? 08:22
arxcruz|ruckmarios yes 08:23
arxcruz|rucki'm terrible with downstream stuff 08:23
mariosif so doesn't seem very efficient but of course it is entirely up to you08:23
mariosarxcruz|ruck: yeah but if it is very quiet downstream and pojadhav|rover is not doing something you can miss things upstream 08:23
arxcruz|ruckmarios well, as far as pojadhav|rover is happy i'm happy with this arrangement 08:23
mariosack fair enough arxcruz|ruck pojadhav|rover 08:23
arxcruz|ruckmarios she also help me on upstream :) 08:24
mariosack 08:24
arxcruz|ruckysandeep https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/40826 do you mind ? :) 08:24
ysandeeplooking08:27
ysandeeptbh, I would be confident to +w that if we had a bug.. But looking at last runs i see failures are random - node provisioning/ overcloud deploy and tempest.08:32
ysandeepyesterday run: only 1 test failed: https://logserver.rdoproject.org/63/35663/29/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-wallaby/09e3d2c/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz 08:32
ysandeepand we have that test in skip now08:33
*** jpena|off is now known as jpena08:37
* ysandeep checking c9 job status08:38
ysandeeparxcruz|ruck, +wed to clear promotion but please revert this https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/40826 as soon as you get wallaby promotion.08:46
arxcruz|ruckysandeep yup08:46
arxcruz|ruckfuck, timeout08:47
arxcruz|ruckakahat hey, can you manually remove fs35 from wallaby c8 and c9 in the promoter ?08:47
arxcruz|ruckthese tox jobs are timeout because of vexxhost mirror issue :( 08:48
pojadhav|rovermarios, yeah.. its up to you with your pair one with mutual understanding.. who will look into what things to make RR week easier.. once downstream monitoring done by me.. I am also looking all upstream component lines to help arxcruz|ruck... :)08:50
mariosack pojadhav|rover fair enough it is up to you to agree how to split - as long as things are covered 08:53
pojadhav|rovermarios, yep08:54
mariosarxcruz|ruck: did you manage to get onto the promoter? i see your key is in authorized_keys on there09:16
mariosarxcruz|ruck: i see you have node fails on the skip patch 09:17
mariosarxcruz|ruck: you should be able to do remove the job from criteria - let me know if you want to jump on a quick call and we can do it together 09:17
arxcruz|ruckmarios nope, let me check it 09:17
pojadhav|roverarxcruz|ruck, hey can we chat for a sec for program call updates ?09:17
arxcruz|ruckpojadhav|rover sure 09:17
pojadhav|roverarxcruz|ruck, https://meet.google.com/oza-izbd-mwq09:18
arxcruz|ruckpojadhav|rover give me 2 min for a coffee 09:18
pojadhav|roverarxcruz|ruck, sure09:18
arxcruz|ruckmarios can you join quickly https://meet.google.com/oza-izbd-mwq ?09:23
arxcruz|ruckre promoter09:23
mariosarxcruz|ruck: sure gimme a minute will join 09:24
*** ysandeep is now known as ysandeep|lunch09:35
*** chem is now known as Guest11810:25
arxcruz|ruckpojadhav|rover centos 8 wallaby promoted10:36
arxcruz|rucklet's pray 10:36
*** rlandy|out is now known as rlandy10:36
arxcruz|rucknow only missing centos 9 wallaby should run next 10:36
arxcruz|ruckthen i'll revert it 10:37
mariosarxcruz|ruck: and abandon your change then so we don't merge it later on by mistake :)10:37
arxcruz|ruckmarios yeah, doing that now 10:37
pojadhav|roverarxcruz|ruck, great !!10:38
rlandyarxcruz|ruck: pojadhav|rover: hey 10:38
rlandyarxcruz|ruck: pojadhav|rover: let's sync10:38
pojadhav|roverrlandy, yep10:38
pojadhav|roverrlandy, arxcruz|ruck : https://meet.google.com/enh-rpiw-iaw10:39
rlandyysandeep|lunch: since you reported you are done setting up rhos-17 on rhel 9, I assume you're no longer watching that?10:49
rlandyand we consider it a regular line to watch now?10:49
ysandeep|lunchrlandy: yes - I have informed team about that in scrum - i think you were on PTO that day10:51
*** ysandeep|lunch is now known as ysandeep10:51
rlandyok10:52
* pojadhav|rover needing short break.. brb11:14
*** dviroel|out is now known as dviroel11:19
soniya29arxcruz|ruck, kopecmartin, ysandeep, chandankumar, do you have anything to share/discuss?11:20
soniya29kopecmartin, ysandeep, chandankumar, ^^11:27
soniya29for tempest meeting?11:27
chandankumardviroel: https://review.opendev.org/c/openstack/tripleo-quickstart/+/83480211:29
chandankumarhttps://review.opendev.org/c/openstack/tripleo-quickstart/+/83405111:29
arxcruz|rucksoniya29 no, i'm on rr duty too 11:30
chandankumarpojadhav|rover: please have a look https://code.engineering.redhat.com/gerrit/c/openstack/rrcockpit/+/315574 at marios's comment11:30
chandankumarwhen free11:30
chandankumarhttps://review.rdoproject.org/r/c/config/+/4044311:31
ysandeepsoniya29: no11:32
arxcruz|ruckmaster promoted11:33
arxcruz|ruck\o/11:33
pojadhav|roverchandankumar, yeah.. doing ruck rovering thats why didnt look at it yet.. once RR finish will look at it.11:33
chandankumarpojadhav|rover: no problem, take your time :-)11:34
pojadhav|roverarxcruz|ruck, WOW ;)11:34
dviroelrlandy: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/834556/3/roles/extras-common/defaults/main.yml 11:34
kopecmartinsoniya: not really11:40
soniyakopecmartin: i am also busy with other stuffs11:40
kopecmartinalso, it seems i have a conflict with another meeting11:40
soniyaysandeep, arxcruz|ruck, chandankumar, kopecmartin, rlandy, let's cancel today's meeting?11:40
chandankumarsoniya: fine by me11:44
soniyachandankumar, ack11:44
rlandyjm1: confirming the HP consoles are updated right?11:49
rlandyI though rcastillo did those?11:49
rlandyysandeep: ^^?11:50
ysandeepI couldn't login on HP servers (tried enve) - After entering password I redirected to same page and I see following error: "iLO Self-Test reports a problem with: Embedded Flash/SD-CARD" on HP server 11:56
rlandyysandeep: ok - sending12:02
ysandeepMay be caused after firware upgrade?12:02
ysandeephttps://community.spiceworks.com/topic/2295331-hpe-ilo4-embedded-flash-error12:02
rlandypossibly12:02
ysandeepI believe there was an issue with early revisions of the ILO firmware that was later corrected12:02
ysandeep^^ quote from above article12:02
rlandyrcastillo: ^^ pls look when you get in12:04
rlandypojadhav|rover: pls check the network component on master/wallaby12:08
rlandylagging there12:08
pojadhav|roverrlandy, ack12:08
rlandypojadhav|rover: pls also testproject tripleo component for wallaby c812:13
rlandynot the whole line12:13
pojadhav|roverrlandy, ok12:13
rlandywe need to keep on top of those pls12:13
rlandypojadhav|rover: 6 days out12:13
pojadhav|roversure12:13
arxcruz|ruckrlandy 2 days ago it fails because of the containers weren't anymore on rdo registry12:19
rlandyarxcruz|ruck: ack12:19
pojadhav|roverrlandy, tripleo component for wallaby C8 testrun : https://review.rdoproject.org/zuul/status#3627412:20
rlandypojadhav|rover: pls check the other lines as well12:21
rlandyall of them12:21
pojadhav|roverrlandy, checking12:21
rlandypojadhav|rover: and keep any lines that are lagging on the hackmd for the next ruck rovers12:22
pojadhav|roverrlandy, sure12:23
jm1rlandy: rcastillo updated the hp servers, yes12:25
jm1ysandeep: which sever is affected?12:26
ysandeepjm1: I will send details in a pm 12:27
rlandyopenstack-component-compute also flat out failing12:27
rlandypojadhav|rover: train is out on a bunch of components as well12:29
rlandypojadhav|rover: good report12:31
pojadhav|roverrlandy, thank you :)12:31
jm1rlandy, ysandeep: the flash error does not affect html5 console. some ILOs show this type of errors but we cant fix them. since everything works as expected and servers are probably out of "warranty" anyway, we just ignore it12:33
ysandeepjm1, I can login from firefox browser, Earlier was on chrome.. thanks for checking!12:34
ysandeeprlandy, fyi.. 12:34
chandankumarmarios: thanks for proposing the patch for making the options job name generic, in RDO side there are few jobs which have definitions can we also make them generic in tripleo-ci itself for example sc712:34
jm1rcastillo: no issue with hp ilo's, no worry ;) see ysandeep's message above12:35
jm1ysandeep: i am able to login with chromium and firefox 12:37
ysandeepjm1: ack, probably a chrome thing then12:37
marioschandankumar: sure... its lined up behind the reparenting so it will be a while12:49
marioschandankumar: i have posted the rdo side too (see commit message it links there) but posting update we will have to keep the original ones until we merge the rdo patch 12:49
pojadhav|roverrlandy, fyi we got green run for sc04 for osp17 rhel-9 : https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status#20987412:51
rlandypojadhav|rover: great12:51
rlandyhmmm ... did we change something? https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-content-provider&skip=012:52
rlandy2022-03-23 11:59:12.183658 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:47): sudo -y '--exclude=python2*' install python3-setuptools python3-requests python3-urllib3 python3-PyYAML12:52
rlandy2022-03-23 11:59:12.187118 | primary | sudo: invalid option -- 'y'12:52
rlandyarxcruz|ruck: ^^12:52
rlandychandankumar: ^6?12:53
arxcruz|ruckrlandy i need to check i think is missing a sudo dnf -y 12:55
rlandyarxcruz|ruck: can you patch pls13:02
*** amoralej is now known as amoralej|lunch13:06
ysandeeprlandy, arxcruz|ruck interesting: 2022-03-23 11:54:48.700529 | localhost | Distro: Ubuntu 20.0413:16
ysandeephttps://dc0059b2deec140455ec-8e6063eece8c96bdec38e25d6079d8b4.ssl.cf5.rackcdn.com/834051/2/gate/tripleo-ci-centos-9-content-provider/bf287b9/job-output.txt13:16
ysandeeprlandy, arxcruz|ruck wrong nodeset?13:16
ysandeepwe are getting ubuntu instead of cs913:17
arxcruz|ruckysandeep 2022-03-23 11:54:48.700720 | localhost | Label: centos-9-stream 13:19
arxcruz|ruck2022-03-23 11:54:48.700237 | localhost | # Node Information13:19
arxcruz|ruck2022-03-23 11:54:48.700317 | localhost | Inventory Hostname: primary13:19
arxcruz|ruck2022-03-23 11:54:48.700386 | localhost | Hostname: ubuntu-focal-iweb-mtl01-002884861713:19
arxcruz|ruck2022-03-23 11:54:48.700453 | localhost | Username: zuul13:19
arxcruz|ruck2022-03-23 11:54:48.700529 | localhost | Distro: Ubuntu 20.0413:19
arxcruz|ruck2022-03-23 11:54:48.700651 | localhost | Provider: iweb-mtl0113:19
arxcruz|ruck2022-03-23 11:54:48.700720 | localhost | Label: centos-9-stream13:19
arxcruz|ruck2022-03-23 11:54:48.700784 | localhost | Interface IP: 198.72.124.13013:19
arxcruz|ruckhmmmm13:20
ysandeepyes, label is right but we are getting wrong node 13:20
ysandeepi think we should report to #opendev13:20
arxcruz|ruckfungi hi, we are getting a ubuntu node on centos-9-stream nodeset did something change?13:21
arxcruz|ruckhttps://dc0059b2deec140455ec-8e6063eece8c96bdec38e25d6079d8b4.ssl.cf5.rackcdn.com/834051/2/gate/tripleo-ci-centos-9-content-provider/bf287b9/job-output.txt13:21
arxcruz|ruckops, wrong channel 13:21
ysandeeparxcruz|ruck: ++ #openstack-infra even better :D13:29
arxcruz|ruckysandeep usually fungi is very helpful :) 13:29
ysandeeptrue, he is very helpful indeed. :D13:30
marioswell spotted ysandeep thats a nasty one 13:34
ysandeepyeah that's totally weird :) seeing for first time13:35
rlandyysandeep: arxcruz|ruck: sorry was in meeting - looks like you sorted it13:39
arxcruz|ruckrlandy we are discussing now on #o-infra with fungi 13:39
rlandygreat13:40
rlandychandankumar: hey ..13:46
rlandychandankumar: bogdan mentioned yesterday that the content provider jobs were not bringing in changes from the gating repo13:47
rlandychandankumar: you familiar with ^^?13:47
rlandywe should mount all repos for image build13:47
rlandycan you confirm??13:47
*** dasm|off is now known as dasm13:58
dasmo/13:58
* pojadhav|rover stepping out for hr... when back will continue to monitor upstream component lines..14:06
rlandypojadhav|rover: rhos-17 on rhel9 should promote14:06
*** amoralej|lunch is now known as amoralej14:06
pojadhav|roverrlandy, yep.. clear run this time 14:07
pojadhav|roverwill check after an hr for its promoted or not..14:08
jm1rcastillo: please update this patch, seems like it has not been merged so far and now it has merge conflicts :/ https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4002614:56
mariosarxcruz|ruck: i joined so i can catchup a bit on the cix i am taking next shift ;)15:32
rlandypojadhav|rover: woohoo https://osp-trunk.hosted.upshift.rdu2.redhat.com/rhel9-osp17/15:32
arxcruz|ruckmarios ok15:32
rlandypromoted rhle-915:32
arxcruz|ruckrlandy akahat btw, i rollback the fs035 changes i did on the promoter15:33
rlandyarxcruz|ruck: thank you15:33
arxcruz|ruckakahat if you can, take a look if i did everything right :) 15:33
arxcruz|ruckthe promoter service is up and running though 15:33
arxcruz|ruckthere's a tmux there 15:33
rlandyarxcruz|ruck: going to dequeue and requeue stabe315:34
rlandystable315:34
* dviroel lunch15:36
*** dviroel is now known as dviroel|lunch15:36
rlandydasm: reading hardware prov g-chat15:39
rlandyare you all set to move forward?15:39
mariosarxcruz|ruck: is there no cix for fs35 wallaby from this morning? 15:40
arxcruz|ruckmarios nope15:40
arxcruz|ruck?15:41
arxcruz|ruckops15:41
mariosarxcruz|ruck: i thought you were going to file a bug since the job was not green since 15th... 15:41
arxcruz|ruckrlandy should we create a bug for random failures we are seeing on fs035? 15:41
rlandyarxcruz|ruck: if we have something to log15:41
rlandyother than vexx is unstable15:41
arxcruz|ruckmarios rlandy it's a mix between vexx and random timeouts on random tempest tests15:42
mariosarxcruz|ruck: so no2 runs were the same thing? 15:42
arxcruz|ruckmarios sometimes is vexx mirror issue 15:42
rlandywe have the mirror issue open15:42
arxcruz|rucksometimes some tempest failure that is just timeout 15:42
mariosk ... :/ would still be good to capture somewhere regardless that this job was not green for over a week but... i mean all jobs had vexx mirror issues15:42
mariosbut this one has been down since 15th ... 15:42
rlandyit looks like there is still some migration work to be done later today15:42
arxcruz|ruckmarios there' s the vexx, and this job was running the web-download test that was failing, it was skipped only yesterday 15:43
arxcruz|ruckand also random timeouts in tempest 15:43
dasmrlandy: i believe so15:46
* pojadhav|rover back..15:55
pojadhav|roverrlandy, WOW ;)15:55
rlandydasm: ok - great - let us know at the meeting tomorrow how things stand 15:56
dasmack15:56
arxcruz|ruckbrb16:04
mariosso tl;dr for cix is we don't know until vx finishes the upgrade rlandy arxcruz|ruck 16:04
mariosthat has been giong for well over a week now though right 16:04
rlandymarios: ack - will update ruck/rovers at thurs scrum meeting16:05
rlandyawaiting rest of migration at 2pm est16:05
marios rlandy: sure just wanted to get a feel of what the cix board is like before taking the shift16:06
* ysandeep shutdown sequence... 16:07
rlandymarios: ack - understand16:07
*** ysandeep is now known as ysandeep|out16:09
arxcruz|ruckmarios it's not so bad today 16:14
arxcruz|ruckmarios so, this is for you :D  https://www.youtube.com/watch?v=79DijItQXMM 16:21
rlandymarios: arxcruz|ruck and pojadhav|rover got all lines promoted within the last two days16:22
rlandyso you're not picking up at a bad point16:23
rlandythey got a worse start16:23
mariosrlandy: well fantastic then :)16:24
mariosarxcruz|ruck: thank you 16:24
arxcruz|ruckmarios you're welcome :D 16:24
arxcruz|ruckmarios check the youtube video :P 16:24
mariosarxcruz|ruck: yes unfortunately i already did 16:25
mariosarxcruz|ruck: :D16:25
arxcruz|rucklol16:25
dasmnow I want to watch Moana...16:25
dasm;)16:25
pojadhav|roverarxcruz|ruck, this issue got resolved right ? "msg": "The conditional check 'push_registry in [\"quay.rdoproject.org\"]' failed. The error was: error while evaluating conditional (push_registry in [\"quay.rdoproject.org\"]): 'push_registry' is undefined\n\nThe error appears to be in '/var/lib/zuul/builds/15e5feca73794f2489ef208439ccc48d/trusted/project_0/review.rdoproject.org/config/playbooks/tripleo-rdo-base/container-login.yaml': line 19, column 16:30
pojadhav|rover11, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n      block:\n        - name: Set vars for quay login\n          ^ here\n"16:30
pojadhav|roverarxcruz|ruck, i am facing it again :( https://logserver.rdoproject.org/82/33582/21/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/15e5fec/job-output.txt16:30
arxcruz|ruckchandankumar ^16:31
pojadhav|roverarxcruz|ruck, chandankumar : sorry.. old logs.. my bad16:34
*** dviroel|lunch is now known as dviroel16:46
*** marios is now known as marios|out16:55
rlandylunch  - brb17:08
* dasm running some errands. bbl17:36
*** jpena is now known as jpena|off17:40
*** amoralej is now known as amoralej|off17:42
chandankumarrlandy: will take a look at content provider tomorrow19:16
chandankumarrlandy: dviroel can we get these patches merged? https://review.rdoproject.org/r/c/config/+/40822 and https://review.opendev.org/c/openstack/tripleo-ci/+/83163119:17
chandankumarIt will complete the container build reparenting19:18
rlandychandankumar; thank you19:18
dviroelchandankumar: i will take a look19:18
chandankumarrlandy: dviroel thanks!19:18
*** dviroel is now known as dviroel|afk20:59
rlandychandankumar: dviroel|afk and I +2'ed both - pls merge in your morning21:26
*** rlandy is now known as rlandy|out21:55
dasmwe seem to have som progress with uefi ipxe thanks to Steve.22:32
dasmI'm gonna check that tomorrow's morning.22:33
dasmLocally, it seems promising.22:33
dasmnew job started: https://review.rdoproject.org/r/c/testproject/+/4033223:04
dasmwill check results later23:04
* dasm dasm|afk23:04
*** dviroel|afk is now known as dviroel\23:44
*** dviroel\ is now known as dviroel23:44

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!