Wednesday, 2022-07-13

« Tuesday, 2022-07-12 Index Thursday, 2022-07-14 »
ysandeepmaybe we can do retro and planning both on thursday - we don't have many new items for planning this time.00:00
*** ysandeep is now known as ysandeep|afk00:09
dviroel|roveryeah, would be a good idea00:10
* dviroel|rover out00:10
*** dviroel|rover is now known as dviroel|out00:10
*** ysandeep|afk is now known as ysandeep01:16
*** ysandeep is now known as ysandeep|afk03:19
*** ysandeep|afk is now known as ysandeep04:44
bhagyashrisysandeep, ack05:47
bhagyashrisi will cancel today's retro and will shift tomorrow 05:47
ysandeepbhagyashris: thanks!06:43
*** ysandeep is now known as ysandeep|afk06:44
*** soniya is now known as soniya|ruck06:48
jm1o/07:06
*** amoralej|off is now known as amoralej07:19
*** ysandeep|afk is now known as ysandeep07:40
jm1ysandeep: hello :) time for a brief chatß07:49
*** ysandeep is now known as ysandeep|lunch08:25
*** anbanerj is now known as frenzy_friday09:17
*** soniya|ruck is now known as soniya|ruck|lunch09:41
*** soniya|ruck|lunch is now known as soniya|ruck10:09
* soniya|ruck will be back in 15 minutes10:10
*** soniya|ruck is now known as soniya|ruck|afk10:11
frenzy_fridaydasm|off hey, the ER bot has started commenting on patches now :D https://review.opendev.org/c/openstack/openstack-helm/+/848832 It is not making much sense. Next I will be updating the queries so that it makes sense10:12
frenzy_fridayexample: https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/84957810:13
*** rlandy|out is now known as rlandy10:26
rlandysoniya|ruck|afk: hello10:27
rlandyI'm updating the program doc10:27
rlandywe can syn when dviroel is in10:27
*** ysandeep|lunch is now known as ysandeep10:40
rlandysoniya|ruck|afk: dviroel is starting late today - will ping you to sync10:43
* jm1 offline for a while to install additional ram11:01
rlandyfrenzy_friday; https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001&skip=0 - looking better today11:05
rlandysoniya|ruck|afk; we need to try promote wallaby today and missing master components11:06
frenzy_fridayrlandy, cool :D !!11:07
*** soniya|ruck|afk is now known as soniya|ruck11:07
soniya|ruckrlandy, ack11:10
rlandysoniya|ruck: we can sync after review time - will ping11:14
soniya|ruckrlandy, okay11:14
rlandybefore program call11:15
rlandythen we need to look at cix11:15
*** rlandy is now known as rlandy|rover11:15
marioshello11:16
mariosreview time now11:16
mariosjoin us11:16
mariosit will be amazin11:16
soniya|ruckkopecmartin, chandankumar, arxcruz, rlandy, please add/edit today's agenda for tempest review meeting :- https://hackmd.io/xWJLuBHGSI2YGX7UlkwRbg11:30
rlandy|roversoniya|ruck: let's sync before program call https://meet.google.com/iox-medr-vax11:43
soniya|ruckrlandy|rover, give me 2 minutes11:43
*** dviroel is now known as dviroel|rover12:12
* dviroel|rover on duty12:12
*** rlandy|rover is now known as rlandy12:23
*** amoralej is now known as amoralej|lunch12:29
dviroel|roverrlandy: soniya|ruck: ready to sync12:40
rlandydviroel|rover: https://meet.google.com/rmz-jnjw-hgj?pli=1&authuser=012:42
rlandysoniya|ruck: ^^ if you want12:42
soniya|ruckrlandy, dviroel|rover, will join in 5 minutes12:46
*** ysandeep is now known as ysandeep|afk12:59
soniya|ruckkopecmartin, ysandeep|afk, rlandy, tempest review meeting?13:01
kopecmartinjoining13:01
soniya|ruckrlandy, ^^13:02
*** amoralej|lunch is now known as amoralej13:29
*** ysandeep|afk is now known as ysandeep13:31
jm1marios: tried to answer your questions in https://review.opendev.org/c/openstack/tripleo-ci/+/849500 :)13:46
mariosthanks jm1 will revisit 13:59
*** dasm|off is now known as dasm14:29
dasmo/14:29
dasmysandeep: thanks for pinging bhagyashris and changing retro!14:30
ysandeepdasm: thanks for highlighting that we have a conflict :)14:31
dasm~o~14:32
rlandydviroel|rover: soniya|ruck: chandankumar: ha - checkout the stability of this run: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-internal-wallaby&skip=014:38
rlandyand this: 14:38
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-internal-wallaby&skip=014:38
soniya|ruckrlandy, ack14:39
chandankumarrlandy: job success rate looks good except node_failures and retry 14:40
rlandychandankumar: ack14:40
rlandybetter than upstream14:40
chandankumaryes14:40
* rlandy checks master14:40
chandankumarrlandy: Do we want to include it in criteria ?14:40
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-internal-master&skip=014:41
rlandydviroel|rover: ^^ maybe we add a train job here14:41
chandankumarrlandy: master is not so good, it has mix of pass and fail14:41
rlandyto see if fs001 is better14:42
rlandychandankumar: fs035 is not working: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-internal-wallaby&skip=014:47
rlandyI don;t think it ever was14:47
dviroel|roverrlandy: chandankumar soniya|ruck: i will investigate the internal erros, we may have multiple bugs on these jobs, but the one that we are hitting more in upstream is always a DB connection error - which may be infra related14:48
dviroel|roverit is a good idea to have fs001-train on internal, yes14:48
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-internal-wallaby&skip=014:49
rlandythat was passing14:49
rlandydviroel|rover: will try add a train job there14:50
rlandylet's see what we get14:50
rlandyfs00114:50
dviroel|roverack tks14:50
rlandywill ping for review14:50
dviroel|roverok14:50
-opendevstatus- NOTICE: Due to an incident in our hosting provider, the tarballs.opendev.org site (and possibly other sites served from static.opendev.org) is offline while we attempt recovery14:51
* dviroel|rover lunch14:52
*** dviroel|rover is now known as dviroel|rover|biab14:53
rlandydviroel|rover|biab: it's there: https://code.engineering.redhat.com/gerrit/gitweb?p=openstack/tripleo-ci-internal-jobs.git;a=blob;f=zuul.d/ovb-rdo.yaml;h=f147867a62294339ec3edd3e3791147291e233f4;hb=HEAD#l5214:53
* rlandy testproject kicks it14:53
chandankumarrlandy: regarding fs035 we talked last time14:54
chandankumarto split it into standalone jobs14:54
chandankumardoes someone picked it up?14:54
dasmmarios: thanks for +W on skiplist fix.14:57
rlandychandankumar: not yet - pls raise at planning14:58
chandankumarrlandy: sure14:58
mariosack np dasm was done during reviews call 14:59
reviewbotDo you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks.14:59
dasmmarios: still you got a call on this :)15:00
mariosdasm: ack np i was already +2 anyway 15:00
dasmnow, we can rekick the pipeline and watch how the world is burning15:01
dasm;)15:01
*** soniya is now known as soniya|ruck15:01
dasmdviroel|rover|biab: o/ can i kick off "openstack-periodic-weekend" pipeline? I don't want to slam the zuul15:02
soniya|rucksecurity master component got promoted :)15:03
dasmsoniya|ruck: the same question to you ^15:03
* jm1 taking a break now and letting memtest verify my ram upgrade15:04
*** ysandeep is now known as ysandeep|out15:04
soniya|ruckdasm, i am not so sure about it, may be rlandy can answer this?15:05
rlandydviroel|rover|biab: train tests triggered in internal15:05
rlandysoniya|ruck: what's the question?15:06
dasmrlandy: can i kick off periodic-weekend pipeline?15:06
dasmwe've merged the change to run "skipped" tests15:06
dasm**only** skipped15:06
rlandydasm: they are all single node?15:06
dasmyes15:07
rlandydasm: yeah - ok - go4it15:08
dasmk, thx15:08
dasmit's running15:10
soniya|ruckrlandy, cs9 fs01 tripleo wallaby fresh run shows 'Failed to delete port with name or ID '171053d0-d2a2-40dc-aa89-c06773b16ce6': No Port found for 171053d0-d2a2-40dc-aa89-c06773b16ce6'15:15
soniya|ruckdviroel|rover|biab, ^^15:16
soniya|ruckrlandy, dviroel|rover|biab, updated hackmd15:27
* soniya|ruck leaving for the day15:27
rlandythanks15:32
chandankumarsee ya!15:33
rlandydviroel|rover|biab: hmmm ... gate retry_limit15:34
rlandy2022-07-13 14:39:42.961676 | ubuntu-focal | E: Failed to fetch https://mirror.iad3.inmotion.opendev.org/ubuntu/dists/focal/main/binary-amd64/Packages  403  Forbidden [IP: 173.231.253.126 443]15:34
rlandy2022-07-13 14:39:42.961716 | ubuntu-focal | E: Failed to fetch https://mirror.iad3.inmotion.opendev.org/ubuntu/dists/focal-updates/main/binary-amd64/Packages  403  Forbidden [IP: 173.231.253.126 443]15:34
rlandy2022-07-13 14:39:42.961722 | ubuntu-focal | E: Failed to fetch https://mirror.iad3.inmotion.opendev.org/ubuntu/dists/focal-backports/universe/binary-amd64/Packages  403  Forbidden [IP: 173.231.253.126 443]15:34
rlandy2022-07-13 14:39:42.961726 | ubuntu-focal | E: Failed to fetch https://mirror.iad3.inmotion.opendev.org/ubuntu/dists/focal-security/main/binary-amd64/Packages  403  Forbidden [IP: 173.231.253.126 443]15:34
rlandymaybe we didn't may another bill?15:35
*** marios is now known as marios|out15:44
rlandylunch  - brb15:47
*** dviroel|rover|biab is now known as dviroel|rover16:11
rlandydviroel|rover: current wallaby hash only out fs001 and fs035 fs035 in still running - will rekick 0116:23
dviroel|roverrlandy: great, thanks for rerunning those16:24
dasm10:12:51    frenzy_friday | dasm|off hey, the ER bot has started commenting on patches now :D16:40
dasmoh, frenzy is not around.16:40
dasmeither way - thanks :)16:41
* dviroel__ isp issues16:41
* dviroel__ run an errand, be back in a few minutes16:42
*** dviroel__ is now known as dviroel|rover|biab16:42
*** amoralej is now known as amoralej|off16:47
dasmrlandy: o/ yesterday we talked about ovb master branch. You mentioned about running fs001 and not fs039. Can you repeat what you were saying? I didn't take notes about that.17:14
rlandydasm: yeah - in a bit17:14
rlandyjust submitting patch17:14
dasmk, no worries17:14
dasmbtw. > tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039 runs only > branches: ^(stable/(train|wallaby)).*$ hence when i tried testing it with master branch, it didn't kick off17:15
dasmd'oh17:15
* dasm is digging through deeper to learn about branches... is it branch per repo? or is it branch per backport? do we have backports? hmmm...17:17
dasmor is it branch for openstack?17:17
dasmhmmm, but why? >name: tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train and > branches: ^master$ override-checkout: "stable/train"17:19
*** dviroel|rover|biab is now known as dviroel|rover17:21
rlandydviroel|rover: pls review : https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/419740 Add container login role with dlrn playbook17:24
rlandydviroel|rover: back to updating 16.2 base image17:24
rlandytesting out the content provider and container build w/o upload now17:24
dviroel|roversure, i will review17:24
rlandydviroel|rover: train fs001 passes fine on internal18:09
rlandydasm: wrt ovb ...18:09
rlandyfor w c8 - fs039 is not in use18:10
rlandyit is for c918:10
rlandyfs001 in w c8 is used18:10
rlandywoohoo rhos-17 on rhel-8 should promote now18:11
dviroel|roverrlandy: oh, thats a interesting result18:12
dasmhm18:12
rlandydviroel|rover: in fact fs035 passes there as well18:13
dasmi see it being not in use, because it's not kicking off, but not sure what should18:13
dviroel|roveroh18:15
rlandydviroel|rover: testing all 16.2 tests with container builds - hoping this works now18:17
dviroel|roverme too :)18:18
dviroel|roverrlandy: baremetal nodes on internal have more vcpu, 8 vs 4 on vexx18:33
dviroel|roverrlandy: nhicher openned a ticket for a new flavor on vexx18:33
rlandydviroel|rover: ok18:42
rlandyso maybe that's the thing18:42
jm1we do not have review time this evening.. dang18:42
rlandydviroel|rover; let's hope - would be an easier fix18:42
rlandyjm1: no we did review time this morning18:43
rlandyyou can raise it at at scrum/planning tomorrow18:43
jm1rlandy: just saw it on gcal. my patches werent ready then. but maybe you could give the first downstream patch a +w ;)18:44
rlandyok - will look18:44
rlandydviroel|rover: what do you want to do about the train hash: https://code.engineering.redhat.com/gerrit/c/testproject/+/41863918:44
rlandysee fs035 and fs001 passed there?18:44
dviroel|roverrlandy: interesting, what if instead of just removing fs001 and fs035 from criteria, we replace by the internal job, and revert after promotion?18:47
rlandydviroel|rover: by all means18:48
dviroel|rovercriteria patch is up, waiting zuul18:59
rlandyk19:02
dviroel|roverrlandy: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/4398719:22
rlandydviroel|rover: voted19:22
dviroel|roverlets merge?19:23
rlandydviroel|rover: sure19:26
rlandydviroel|rover: also rekicked internal fs001 wallaby19:27
rlandyonly one missing to promo19:27
rlandyfs00119:27
dviroel|roverrlandy: fs001-train running with a new flavor on vexx19:29
rlandydviroel|rover++ nice19:29
rlandylmk19:29
dviroel|roversure19:30
rlandydviroel|rover: change in base image also looking good19:30
rlandyhave content-provider in rerun19:30
rlandyall the rest passed here: https://code.engineering.redhat.com/gerrit/c/testproject/+/20029519:30
rlandydviroel|rover: will bring it for review time tomorrow19:31
rlandyincluding your patch for v1 registries19:31
dviroel|roverack19:32
dviroel|roverstarted fs001-train with xlarge flavor again, setting previous hash, current one is still building images19:33
dviroel|roverrlandy: i don't see my patch as depends-on on your testproject19:36
dviroel|roverrlandy: nevermind19:37
dviroel|roveryou rebased on top of it19:37
rlandyjust updated jira card19:37
rlandyadding to review list now19:37
rlandydone19:41
rlandywoohoo - finally ... tripleo-ci-rhel-8-content-provider-rhos-16.2testprojectmastercheck418635,520:22
rlandy1 hr 11 mins 11 secs2022-07-13 19:04:14SUCCESS20:22
rlandyhttps://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-scenario010-standalone&skip=020:22
rlandygate blocker?20:23
rlandy2022-07-13 20:00:16.289526 | fa163e58-70ed-ce91-2afa-000000003890 |      FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_4 | standalone | error={"changed": false, "msg": "Failed containers: nova_libvirt_init_secret"}20:24
rlandyhttps://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-scenario010-ovn-provider-standalone&skip=020:24
rlandydviroel|rover: ^^ think we have a new gate blocker20:24
rlandyhttps://a9f1aef221b9e8d1cf76-922433d163de5a07cac84d974d42345f.ssl.cf1.rackcdn.com/849688/2/check/tripleo-ci-centos-9-scenario010-ovn-provider-standalone/e93e550/logs/undercloud/var/log/extra/podman/containers/nova_libvirt_init_secret/stdout.log20:25
rlandyError: /etc/ceph/ceph.conf contained an empty fsid definition20:26
rlandylogging20:26
* dviroel|rover looking20:26
rlandydviroel|rover: https://bugs.launchpad.net/tripleo/+bug/198163420:32
rlandynotifying openstack-discuss20:32
rlandywe'll need guilio yo look at this if not immediately obvious20:32
dviroel|roverrlandy: ack20:34
rlandyhttps://opendev.org/openstack/tripleo-ansible/commit/5630a227943e282e2861d212e150fc1c4eaac6d920:42
rlandymaybe20:42
rlandydviroel|rover: wdty? worth a revert?20:45
rlandyto test20:46
dviroel|roverlooking at the logs, last success one has fsid on ceph.conf20:47
rlandyhttps://review.opendev.org/c/openstack/tripleo-ansible/+/849572 - no scenario010 tests ran20:49
dviroel|rover2022-07-13 02:12:54,242 7fecf0f43740 INFO Generating new minimal ceph.conf20:52
dviroel|rovercephadm log ^20:52
dviroel|roverrlandy: we should that revert20:53
rlandydviroel|rover: ok - let's try20:53
rlandydviroel|rover: https://review.opendev.org/c/openstack/tripleo-ansible/+/84973220:54
rlandylet's depends-on test that for scenario01020:54
rlandygetting dnm patch - sec20:55
* dviroel|rover biab20:57
* dviroel|rover back21:35
* rcastillo leaving for today22:04
* dasm is leaving too22:08
dasmo/22:08
*** dasm is now known as dasm|off22:08
dviroel|roverC8 train criteria changed on revert process22:57
dviroel|roversince: current-tripleo/2022-07-13 20:5522:58
dviroel|rovertrain ^22:58
* dviroel|rover afk22:58
*** dviroel|rover is now known as dviroel|rover|Afk22:58
*** dviroel|rover|Afk is now known as dviroel|rover23:57
« Tuesday, 2022-07-12 Index Thursday, 2022-07-14 »

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!