Friday, 2019-10-04

*** aakarsh has joined #oooq00:02
*** jtomasek has quit IRC00:30
*** dsneddon_ has quit IRC00:31
*** jbadiapa has joined #oooq00:33
*** dsneddon_ has joined #oooq00:36
*** jtomasek has joined #oooq00:42
*** dsneddon_ has quit IRC00:51
*** dsneddon_ has joined #oooq01:01
*** panda|off has quit IRC01:12
*** panda has joined #oooq01:12
*** dsneddon_ has quit IRC01:13
*** rfolco has quit IRC01:19
*** rfolco has joined #oooq01:27
*** dsneddon_ has joined #oooq01:48
recheck[gh-ansible] v2.9.0rc2 → https://github.com/ansible/ansible/releases/tag/v2.9.0rc202:02
*** dsneddon_ has quit IRC02:34
*** ykarel|away has joined #oooq02:36
*** rfolco has quit IRC02:45
*** dsneddon_ has joined #oooq02:51
*** dsneddon_ has quit IRC02:56
*** dsneddon_ has joined #oooq03:06
*** dsneddon_ has quit IRC03:18
*** dsneddon_ has joined #oooq03:22
*** dsneddon_ has quit IRC03:27
*** dsneddon_ has joined #oooq03:36
*** dsneddon_ has quit IRC03:56
*** ykarel|away has quit IRC04:05
*** gouthamr has quit IRC04:15
*** gouthamr has joined #oooq04:16
*** dsneddon_ has joined #oooq04:16
*** dsneddon_ has quit IRC04:22
*** ykarel|away has joined #oooq04:25
*** aakarsh has quit IRC04:28
*** ykarel|away is now known as ykarel04:31
*** surpatil has joined #oooq04:44
*** ykarel is now known as ykarel|afk04:50
*** dsneddon_ has joined #oooq04:51
*** jbadiapa has quit IRC04:52
*** soniya29 has joined #oooq04:57
*** dsneddon_ has quit IRC05:07
*** ratailor has joined #oooq05:13
*** raukadah is now known as chandankumar05:13
*** marios has joined #oooq05:27
*** rlandy|bbl is now known as rlandy05:43
*** dsneddon_ has joined #oooq05:44
*** jfrancoa has joined #oooq05:57
*** ykarel|afk is now known as ykarel06:00
*** skramaja has joined #oooq06:37
*** udesale has joined #oooq06:46
*** bogdando has joined #oooq07:09
*** tosky has joined #oooq07:18
*** dsneddon_ has quit IRC07:19
*** amoralej|off is now known as amoralej07:25
chandankumarmarios: please have a look at this https://review.opendev.org/#/c/685784/1 needed for tobiko stuff07:35
*** jpena|off is now known as jpena07:38
*** dsneddon_ has joined #oooq07:40
mariosack chandankumar07:42
*** dsneddon_ has quit IRC07:45
zbrmorning!07:50
chandankumarmarios: thanks!07:53
pandazzzz07:56
pandamarios: I have a doctor appointment in 2 hours08:01
mariospanda: ack - will you be back later? otherwise we can talk monday?08:07
mariospanda: i am available any time you can talk, including now or in 1 hour or whatever works08:08
*** holser has joined #oooq08:08
*** dsneddon_ has joined #oooq08:12
pandamarios: monday will be too late, let's do it in 15 minutes08:13
mariospanda: ack sending the hangout email now08:13
pandamarios: I'll also be back after the doctor appointmant, for all day today, and tomorrow, and sunday, I'll never top working on this ... O_o08:13
*** dsneddon_ has quit IRC08:17
zbrpanda: go panda!08:19
pandazbr: https://en.wikipedia.org/wiki/Panda!_Go,_Panda!08:20
*** akahat has joined #oooq08:21
*** zbr is now known as zbr|ruck08:23
*** soniya29 has quit IRC08:24
*** derekh has joined #oooq08:37
*** soniya29 has joined #oooq08:40
*** dsneddon_ has joined #oooq08:50
soniya29arxcruz|ruck, kopecmartin, weshay, If it is okay with everyone then can we prepond evening meeting at 6.00 PM? Actually, We have Garba night event in the evening.08:54
*** fmount is now known as fmount|off08:54
soniya29s/prepond/prepone08:58
arxcruz|rucksoniya29: fine by me09:01
*** akahat has quit IRC09:11
*** slaweq has joined #oooq09:18
slaweqhi09:18
slaweqmay I have a question about quickstart?09:18
slaweqI'm trying now to deploy manually oooq with ml2/ovs neutron backend09:18
pandaslaweq: you may have all the questions you want.09:18
slaweqI used command like:09:18
slaweq./quickstart.sh --clean --teardown all --tags all --requirements $REQUIREMENTS --no-clone --release master -I --environment ~/tripleo-heat-templates/environments/services/neutron-ovs.yaml09:18
pandaslaweq: the answers on the other hand ....09:18
slaweqbut it was deployed with ovn instead09:18
*** ccamacho has joined #oooq09:19
slaweqdo You know how I should override default neutron backend in oooq?09:19
*** rfolco has joined #oooq09:23
pandaslaweq: --environment defines the general CI environment. you can try passing --extra-vars network_args="-e ~/tripleo-heat-templates/environments/services/neutron-ovs.yaml" and network_args will be passed to overcloud command09:23
pandaslaweq: but it may not be neough, default deployment may be standalone, and overcloud commands are not touched there09:24
slaweqpanda: ok, I will try that, thx09:25
*** panda is now known as panda|bbl09:25
*** jbadiapa has joined #oooq09:42
*** dsneddon_ has quit IRC09:45
*** jfrancoa has quit IRC09:50
*** jfrancoa has joined #oooq10:06
*** dsneddon_ has joined #oooq10:16
*** saneax has joined #oooq10:23
chandankumarsoniya29: surpatil please update this card https://tree.taiga.io/project/tripleo-ci-board/task/1305?kanban-status=144727510:30
*** saneax has quit IRC10:32
*** panda|bbl is now known as panda10:39
*** soniya29 has quit IRC10:42
*** jaosorior has joined #oooq10:54
*** dsneddon_ has quit IRC10:56
*** jpena is now known as jpena|lunch11:00
*** soniya29 has joined #oooq11:08
*** jbadiapa has quit IRC11:09
*** zbr|ruck is now known as zbr|lunch11:15
chandankumarmarios: arxcruz|ruck weshay Here I present you the passing fs01 os_temepst https://logs.rdoproject.org/00/673400/42/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/de3cdd2/logs/stestr_results.html on ovb11:22
*** dsneddon_ has joined #oooq11:26
*** matbu has quit IRC11:34
*** matbu has joined #oooq11:35
*** panda is now known as panda|eat11:43
*** ccamacho has quit IRC11:45
mariosnice chandankumar11:52
marioswhat was it11:52
*** EmilienM has quit IRC11:53
*** EmilienM has joined #oooq11:55
*** ratailor has quit IRC11:59
*** ratailor has joined #oooq11:59
arxcruz|ruckchandankumar: :O12:00
weshay0/12:00
weshayarxcruz|ruck, welcome back12:00
*** jpena|lunch is now known as jpena12:00
arxcruz|ruckchandankumar: so the issue was tempest_cidr: '10.0.0.0/24' ?12:01
arxcruz|ruckbecause i tested that configuration, but with tempest_cidr 192.168.0.0/2412:01
arxcruz|ruckweshay: hey boss12:02
chandankumararxcruz|ruck: we also need to include deployinput conf file12:04
chandankumarotherwise it will not worl12:04
chandankumar*work12:04
chandankumararxcruz|ruck: do we want to run from stein onwards to os_tempest?12:05
arxcruz|ruckchandankumar: I'm up to destroy validate-tempest12:05
arxcruz|rucktear down12:05
chandankumararxcruz|ruck: ok12:05
*** udesale has quit IRC12:16
*** udesale has joined #oooq12:16
chandankumararxcruz|ruck: if the stein passed, we can merge fs01 and continue working on moving all jobs to os_tempest12:20
arxcruz|ruckok12:20
chandankumarexcept with few exception octavia and ironic that we will figure out later12:21
*** dsneddon_ has quit IRC12:23
*** rfolco has quit IRC12:24
*** rfolco has joined #oooq12:25
ykarelweshay, marios can u please check https://review.opendev.org/#/c/686522/12:30
ykarelto fix log collection in virthost phase1 jobs12:30
chandankumarkopecmartin: weshay meeting time12:30
ykareljobs failing there, but no log12:30
chandankumarweshay: we are waiting for you.12:33
weshaychandankumar, start w/o me please12:33
weshayin another mtg12:33
chandankumarweshay: sure12:33
*** rlandy has joined #oooq12:34
*** panda|eat is now known as panda12:37
chandankumarkopecmartin: arxcruz|ruck https://opendev.org/x/ospurge12:42
ykarelchandankumar, https://trunk-primary.rdoproject.org/api-centos-train/api/civotes_detail.html?commit_hash=72487d804ef4c64d16889b1b474252f6963a18c2&distro_hash=3f633a877cc44c0fa774e06dbf99c7db29419b2f most of the jobs passed for train, good to promote?12:49
ykarelrlandy, ^^12:49
ykareland setup the pipeline for train12:50
weshayykarel, let's get fs001 fixed12:50
rlandylooking12:50
ykarelweshay, fs001 any known issue there?12:50
chandankumarrlandy: https://review.rdoproject.org/r/#/c/22581/ please remove -2 from his12:51
rlandyykarel pipeline is setup12:51
chandankumarthis12:51
*** soniya29 has quit IRC12:51
chandankumarwe need to merge that12:51
ykarelrlandy, yes you put -2 there12:51
rlandyykarel: chandankumar: -2 removed12:51
rlandyI wil w+ it12:51
ykarelrlandy, ack go for it12:52
rlandyykarel: I have a patch for phase 1 - ci-centos jobs ...12:52
ykarelrlandy, wrt timeout?12:52
rlandyykarel: no wrt train ... https://review.rdoproject.org/r/#/c/22582/12:53
*** dsneddon_ has joined #oooq12:53
ykarelrlandy, yup i reviewed12:53
weshayrlandy, ya.. you were investigating our timeout settings on ovb yesterday I think12:53
rlandychandankumar: should I split that patch and take out just the phase1 stuff?12:53
chandankumarrlandy: keep it as it is12:54
rlandychandankumar: ok - so that can go after promotions run12:55
rlandychandankumar: anything else required for train ... weshay said no check12:55
rlandyweshay: wrt timeout - let me check bm job timing ... sec12:56
*** udesale has quit IRC12:56
rlandymaster ran and passed yesterday12:56
chandankumarrlandy: currently no, but we will put the check patch from our side12:57
rlandyweshay: 2 hours and 20 mins ... https://sf.hosted.upshift.rdu2.redhat.com/logs/97/182597/2/check/periodic-tripleo-ci-centos-7-bm_envA-3ctlr_1comp-featureset001-master/07a1dd9/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz12:58
chandankumarwith -w12:58
rlandyreally 2 hours and 10 mins12:59
rlandybut it's a long run12:59
arxcruz|ruckchandankumar: https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/rrcockpit/files/telegraf/zuulv3_job_builds.py12:59
rlandywe never overran 2 hours before12:59
rlandyykarel: ^^ fyi - was checking b against ovb deploy time increases12:59
rlandywe have similar on bm12:59
rlandyincreased deploy time there as well13:00
ykarelrlandy, yes same we observed in phase 1 ci.centos:- 0:38:23.94713:00
ykareltaken by container image prepare13:00
ykarelso it will never finish in 90 minute13:01
rlandygot stein running now13:01
ykarelrlandy, stein has similar behavior13:01
rlandyand master on a second env13:01
*** rlandy is now known as rlandy|mtg13:02
arxcruz|rucksurpatil: https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/rrcockpit/files/telegraf/zuulv3_job_builds.py13:06
surpatilok13:07
*** ratailor has quit IRC13:23
*** zbr|lunch is now known as zbr|ruck13:25
arxcruz|ruckweshay: 1-1 ?13:30
arxcruz|ruckbj or gmeet ?13:30
weshaysec13:31
*** SurajPatil has joined #oooq13:34
weshaysorry folks.. long call13:34
*** surpatil has quit IRC13:36
*** dsneddon_ has quit IRC13:36
weshayarxcruz|ruck, https://meet.google.com/zwn-wheu-fqg13:37
zbr|ruckweshay: let me know what i can help with.13:41
pandarfolco: you shold probably rename molecule-tripleo-common to promotion-tripleo-common-integration, and make it trigger in tripleo-common13:41
rlandy|mtgpanda: we never spoke about cix card13:41
pandarlandy|mtg: I rejoined13:42
zbr|ruckpanda: better to keep molecule in its name for maintenance, but it does not have to start with it.13:42
rlandy|mtgok - rejoining13:42
toskyshould I wait for a tripleo promotion in train t have tripleo-ci-centos7-scenario003-standalone passing?13:42
pandarlandy|mtg: not sure if you can let me in ...13:42
pandazbr|ruck: fine13:42
toskythe derived job in sahara is failing and I will need to backport a patch soon13:42
toskyso I have to decide whether to disable that tripleo job or wait13:43
pandazbr|ruck: I just want to be clear it's a integration job between the promotion code and tripleo-common13:43
rfolcozbr|ruck, panda molecule-promotion-tripleo-common-integration is a LONG name, any objections13:43
rfolco?13:43
zbr|rucktripleo-common-promotion-molecule13:43
zbr|ruckin the pas I wanted to use "mol" but some people argued against not being clean, probably now they will be fine with it.13:44
pandaooo-c-prot-mole-integr13:44
rfolcohaha13:44
zbr|ruckmaybe i should propose renaming molecule project to h2o13:45
zbr|ruck... or h2so4 -- depends on who's using it13:45
rfolco3O-c-pro-mole-int13:46
zbr|ruckpanda: what's the status with promoter changes? many more incoming? asking because I want to merge https://review.rdoproject.org/r/#/c/22900/13:47
*** amoralej is now known as amoralej|lunch13:50
mariospanda: rlandy|mtg rfolco 2019-10-04 13:46:37,482 5545 ERROR    promoter Exception when calling api_promote_post: <class 'dlrnapi_client.rest.ApiException'>  so it got passed the images see logs at http://logs.rdoproject.org/72/22772/14/check/tripleo-ci-promotion-staging/9d7f49d/logs/promoter_logs/centos7_master.log13:51
marios2019-10-04 13:46:36,900 5545 INFO     promoter INFO: /home/zuul/ci-config/ci-scripts/promote-images.sh succeded updating file:///tmp/promoter-staging/overcloud_images/centos7/master/rdo_trunk/tripleo-ci-staging-promoted to point to file:///tmp/promoter-staging/overcloud_images/centos7/master/rdo_trunk/1c67b1ab8c6fe273d4e175a14f0df5d3cbbd0edc_8170b86813:51
pandamarios: \o/13:51
mariospanda: rfolco rlandy|mtg going to remove WIP please review in your own time maybe we can merge by early next week thanks https://review.rdoproject.org/r/#/c/22772/13:52
zbr|rucki hope docker inc rots in hell of corporate underworld.... https://github.com/docker/compose/pull/690013:54
chandankumarrlandy|mtg: we need to merge this https://review.rdoproject.org/r/#/c/22582/ for promotion13:57
chandankumarplease change the promotion count to 513:57
weshayzbr|ruck, arxcruz|ruck one of you please add a row for train here :) https://code.engineering.redhat.com/gerrit/#/q/project:rhosp-dashboards13:59
weshaythat will fill out http://rhos-release.virt.bos.redhat.com:3030/rhosp13:59
*** dsneddon_ has joined #oooq14:07
pandamarios: rfolco I know why it's failing, credentials are wrong14:09
pandamarios: but yes we are ready to merge your patch now14:09
*** aakarsh has joined #oooq14:18
pandarlandy|mtg: https://trunk-primary.rdoproject.org/api-centos-stein/api/civotes_detail.html?ci_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-stein14:21
*** surpatil has joined #oooq14:24
*** SurajPatil has quit IRC14:26
rfolcopanda, how does your setup know that registries should run source_registry and target_registries ?14:27
rfolcopanda, staging_environment.py setup --components registries14:27
weshaypanda, when you have a sec.. did you see my email re: the promotion steps and it's updating dlrn before containers14:29
pandaweshay: me and ronelle are looking at that right now14:30
pandawell, ronelle and me mainly14:30
rlandy|mtgweshay: we are typing on the LP bug14:30
rlandy|mtghttps://bugs.launchpad.net/tripleo/+bug/184666214:30
openstackLaunchpad bug 1846662 in tripleo "dlrn_promoter is updating links in dlrn_trunk prior to all the containers being pushed" [Critical,Triaged]14:30
rlandy|mtgit's not consistent14:30
pandawe have a gift for you, it's called pandora's box14:30
*** skramaja has quit IRC14:30
rlandy|mtgwe see your once off reference14:30
rlandy|mtgor panda's box14:31
pandarfolco: the setup creates the registries with info from the config file stage-config.yaml14:31
rfolcoaaah14:31
rfolcopanda, https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/tests/staging-setup/stage-config.yaml14:31
rfolcopanda, and why do we also need the vars in playbook ?14:31
*** rlandy|mtg is now known as rlandy14:32
chandankumarrlandy|mtg: I am updating this review https://review.rdoproject.org/r/#/c/22582/ to take get promotion on 5 hashes if ok?14:32
rlandychandankumar: sorry - was in meeting14:32
rlandylooking14:32
pandarfolco: which var in which playbook ?14:32
chandankumarit is the last review everyone needs promotion14:32
rlandychandankumar: was this comment for me ? <chandankumar> please change the promotion count to 5?14:32
rlandyand emit releases14:33
rlandythat has to merge14:33
chandankumarrlandy: for emit release to passing, we need promotion14:33
chandankumarrlandy: yes14:33
rlandywhat is the status there?14:33
chandankumarrlandy: the comment is for you14:33
rfolcopanda, https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/promoter/molecule/container-push/playbook.yml#L24-L3614:33
chandankumarrlandy: the emit release review is on hold until all tripleo cores review it14:33
* rlandy looks for promotion count14:33
rlandychandankumar: help me out ^^ where is that?14:34
chandankumarrlandy: https://review.rdoproject.org/r/#/c/22582/6/ci-scripts/dlrnapi_promoter/config/CentOS-7/train.ini@1014:34
pandamarios: can you fix the linters ?14:34
chandankumarrlandy: latest_hashes_count: 1014:35
rlandyoh - ok14:35
pandamarios: otherwise it's good to go14:35
rlandypanda: marios: pls check emit releases patch14:35
*** jbadiapa has joined #oooq14:35
rlandyblocking train14:35
rlandychandankumar's patch14:35
pandaah yeah14:35
rlandyweshay wants all cores to review14:35
*** amoralej|lunch is now known as amoralej14:35
pandaI tried to find taht yesterday, chandankumar link ?14:36
chandankumarpanda: one min14:36
rlandychandankumar: I already looked at it - so will let others take a shot14:36
rlandysee the upgrades jobs output14:36
chandankumarpanda: marios weshay rfolco zbr|ruck arxcruz|ruck https://review.opendev.org/68426014:36
pandamarios: commented the patch14:38
mariospanda: looking thanks14:38
chandankumarplease please please review14:38
pandachandankumar: reviewing now14:38
chandankumarmarios: thank you, we sorted out os_tempest fs01 issue14:38
rlandychandankumar: https://review.rdoproject.org/r/#/c/22582/ updated with latest count14:39
rlandyanything else - otherwise that can go14:39
ykarelrlandy, what about promoting based on current results14:39
chandankumarrlandy: nope14:39
rlandypanda: marios: rfolco: fyi ... https://review.rdoproject.org/r/#/c/22582/ - promoter additions in there14:40
chandankumarykarel: do you want to decrease the count?14:40
rlandyykarel: if we are ready to - merge the patch14:40
rlandythe promoter will take care of it14:40
ykarelchandankumar, nope, that should not help14:40
chandankumarykarel: ok14:40
ykarelfor current situation both 5 and 10 are same for us14:41
ykarelrlandy, https://trunk-primary.rdoproject.org/api-centos-train/api/civotes_detail.html?commit_hash=72487d804ef4c64d16889b1b474252f6963a18c2&distro_hash=3f633a877cc44c0fa774e06dbf99c7db29419b2f14:41
ykarelnot all jobs passed ^^, so promotion will not happen until promotion criteria is adjusted14:41
chandankumarykarel: can you comment on jobs, we can comment those and get the promotion?14:42
chandankumarykarel: please update the patch to get the promotion14:42
ykarelchandankumar, okk doing14:42
rlandyykarel: so -  my first thought would be to push the patch with the criteria in place and add a second patch we can revert with changes - but that is fine14:42
ykarelrlandy, +1 for that14:42
ykarelthat would be much better14:43
rlandyykarel: ok - so let's put in a second patch on top and merge this one14:43
ykarelrlandy, ok for me14:44
*** skramaja has joined #oooq14:45
rlandyok w+'ing14:46
pandachandankumar: reviewd ok!14:47
rlandyok14:47
chandankumarpanda: weshay rlandy marios zbr|ruck arxcruz|ruck rfolco https://blogs.rdoproject.org/2019/10/rdo-centos-stream/ for centos8 in RDO14:47
chandankumarfeel free to read when free14:47
chandankumarpanda: ^^14:48
ykarelyup i saw that14:48
rlandycool14:48
chandankumarpanda: thanks!14:49
rlandyykarel: sorry, you're editing the criteria or I am?14:49
* rlandy lost track here14:49
ykarelrlandy, you can edit, i commented on patch14:49
weshay chandankumar ya.. we have a long road before that happens :)14:50
weshaybut.. it will be awesome when it does.. if it's stable14:50
rlandyok - adding pactch14:51
weshayrlandy, ykarel are we not going to wait for fs001?14:52
ykarelweshay, same question, what's issue with fs00114:52
ykarelany new issue?14:52
weshayit didn't pass :)14:52
weshaylet's adjust timeouts and rerun14:52
rlandysame timeout as master14:52
weshayya.. let's get the timeouts updated14:53
ykarelweshay, ok, since it's initial promotion, and train jobs broken upstream, good to initial promotion14:53
weshayykarel, rlandy I'm the only one w/ access to the promoter atm :)14:53
ykarelmostly main jobs passed14:53
chandankumarweshay: :-)14:54
weshayOct 07 - Oct 1114:54
weshayR-114:54
weshayFinal RCs and intermediary releases14:54
chandankumarykarel: does fs01 train broken?14:54
chandankumaror is it just queens?14:54
weshaytimes out14:54
weshayqueens promoted14:54
chandankumar\o/14:54
rlandyweshay: ykarel: I am putting n the patch with fs001 still active - you two can edit/merge/upload it as you both see fit14:55
weshayykarel++14:55
rlandyjust doing the leg work here14:55
ykarelchandankumar, fs001 not broken, it's the same issue as master/stein, it times out intermittently14:55
weshayI saw the timeout still set to 90 miin14:55
weshayneeds to be bumped14:55
weshayarxcruz|ruck, zbr|ruck ^14:56
*** dsneddon_ has quit IRC14:56
*** skramaja has quit IRC14:57
chandankumarHappy weekend #oooq, see ya on monday14:57
zbr|ruckok, i was replying to the post14:57
*** chandankumar is now known as raukadah14:57
weshayraukadah, take care :)14:58
raukadahweshay: :-)14:58
raukadahrlandy: thank you, we are done with train promotion patches on time :-)14:58
weshayarxcruz|ruck, skip list for train https://logs.rdoproject.org/29/22529/12/check/periodic-tripleo-ci-centos-7-standalone-full-tempest-train/7771dc6/logs/stestr_results.html14:58
rlandyraukadah: yay us :)14:58
arxcruz|ruckweshay: is there a lp already ?14:59
weshaydoubt it15:00
ykarelweshay, fs020 passed though15:00
ykarelso good to check with that also for full tempest standalone15:00
arxcruz|ruckweshay: ykarel I believe it's a randon issue, the error was timeout while wait down status15:00
* raukadah will be making a unskip job for full tempest soon15:00
rlandyykarel: https://review.rdoproject.org/r/22941 Adjust train criteria for initial promotion15:01
rlandyyou can decide on fs00115:01
ykarelok for me to go without fs001 :) as other similar jobs passed, and train jobs upstream are not working15:02
ykarelgood to have promotion before weekend too :)15:03
ykarelso weekend runs we can seem more movement15:03
ykarelboth related to updates jobs and phase115:03
weshayrlandy,  /me notes.. updates to the promotion server are frozen15:04
weshayit's not going to promote15:04
rlandyweshay: ack - understood15:04
ykarel:(15:04
rlandygetting work in place15:04
* rlandy gets to break the promoter from two different projects :)15:05
weshayarxcruz|ruck, zbr|ruck let's get the time outs fixed15:05
weshayin ovb15:05
ykarelweshay, rlandy can u review https://review.opendev.org/#/c/686522/15:07
ykarellog collection phase 115:07
zbr|ruckweshay: up timeout from 3.5h to 4h for all ovb?15:08
arxcruz|ruckweshay: increase only for fs001 ?15:08
arxcruz|ruckokay, it seems zbr|ruck is already on it :P15:08
*** arxcruz|ruck is now known as arxcruz|rover15:08
weshayarxcruz|rover, might as well do it for all15:08
zbr|rucki am ready to make the change, once I have a) LP ticket asking for it b) info reg which job if only 001 or its parent, tripleo-ci-base-ovb-rdo15:09
zbr|rucki am inclined to do the parent, as this is where the timeout is current set15:09
rlandyykarel: so I am not sure about  https://review.opendev.org/#/c/686522/ - not sure if this is used elsewhere15:10
ykarelrlandy, elsewhere means?15:10
rlandywhere 127.0.0.2 will break15:10
ykarelrlandy, so localhost is not allowed now15:11
rlandyoutside of ci.centos15:11
ykarelno where it's allowed15:11
ykarelrlandy, https://review.opendev.org/#/c/672075/9/quickstart.sh15:11
ykarelso need to use 127.0.0.215:12
weshayykarel, where is that happening? I had a change on that already15:12
weshayykarel, yes.. you are correct15:12
ykarelweshay, your change moved forward, but now collect logs is failing15:12
rlandyok - then15:12
ykarelweshay, rlandy see https://ci.centos.org/job/tripleo-quickstart-promote-master-current-tripleo-delorean-minimal/603/consoleText15:13
ykareland look for ERROR: Please use 127.0.0.2 to deploy to localhost15:13
weshayhttps://review.opendev.org/#/c/685090/15:13
ykarelyup that was partial fix15:13
weshayykarel, ++15:14
* weshay looks at ur review15:14
weshayykarel, wf15:15
weshaythanks for catching that15:15
ykarelack Thanks15:15
*** ykarel is now known as ykarel|afk15:15
zbr|ruckweshay: i need a LP ticket re timeout to reference in my change.15:18
zbr|ruckbacking & documenting purposes15:19
weshayzbr|ruck, ok.. write one up.. find the timing out fs001 jobs in train, stein, etc15:19
zbr|ruckok15:19
weshayzbr|ruck, https://trunk-primary.rdoproject.org/api-centos-train/api/civotes_detail.html?commit_hash=72487d804ef4c64d16889b1b474252f6963a18c2&distro_hash=3f633a877cc44c0fa774e06dbf99c7db29419b2f15:20
weshaypanda, rlandy  re: containers uploaded after the dlrn_trunk promotion15:23
weshaypanda, rlandy in containers push..   can we add a test to check and display the hash in dlrn_trunk?15:23
rlandyweshay: we have a check above it for dlrn promoter15:24
weshaynot sure if we'd want to go as far as rewinding the dlrn_trunk promotion.. but I would like to see a unit test covering this15:24
rlandypanda: ^^ maybe we extend that15:24
zbr|ruckhttps://review.rdoproject.org/r/#/c/22943/15:24
weshayit's pretty dangerous to promote dlrn_trunk first..15:24
weshayzbr|ruck, k.. that addresses the job.. there is also a timeout set to 90 min on the deploy15:25
pandaweshay: what's the check ?15:26
weshayzbr|ruck, rlandy this is hardcoded? https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-deploy/defaults/main.yml#L3015:27
pandaweshay: I meand what's the failure criteria for this check, and the success criteria ?15:27
weshayzbr|ruck, rlandy afaict.. the job is not timing out.. the deployment is15:27
weshaypanda, let's chat for a sec15:27
rlandyweshay: ack15:27
zbr|ruckweshay: is in defaults, so we can pass it15:27
weshayfigure this out15:27
rlandythat is correct15:27
rlandyits' the deploy timeout15:28
zbr|ruckso I would not call it hardcoded15:28
weshayur right15:28
weshay:)15:28
weshaypanda, https://meet.google.com/cnt-smtm-hkj15:28
zbr|ruckweshay: if I add it to vars in job definition is ok?15:28
rlandyzbr|ruck,: you can override it in settings15:28
rlandyzbr|ruck: no15:28
rlandyyou would need either a fs override or to add it in one of the settings file15:29
rlandythe ovb settings file is easiest15:29
rlandythe job definition settings are not passed through to toci-quickstart unless they are featureset_override15:29
*** kopecmartin is now known as kopecmartin|off15:30
mariospanda: v17 should address your comments thanks (whenever you next have time) https://review.rdoproject.org/r/#/c/22772/15:31
zbr|ruckrlandy: so ovb-internal.yml ?15:32
rlandyzbr|ruck: getting15:32
*** dsneddon_ has joined #oooq15:32
rlandyzbr|ruck: weshay: my suggestion is to add the timeout here ... https://github.com/openstack/tripleo-ci/blob/master/toci-quickstart/config/testenv/ovb.yml15:33
rlandyzbr|ruck: weshay: now it is possible that the whole job may timeout once we give the deployment enough time to complete - but you will see that in the logs15:35
rlandyon bm, I increased both  - in that order15:35
*** jaosorior has quit IRC15:41
zbr|ruckrlandy: https://review.opendev.org/#/c/686750/ ?15:43
jbadiapahi all15:43
jbadiapaWe are having some issue with a patch and I would like to reproduce the patch locally15:44
rlandyzbr|ruck: ack - will +2 - will w+ if we see the deploy time increase properly in tests15:44
jbadiapawell, we're bassically trying to add the molecule test to the tripleo-upgrades15:45
weshayzbr|ruck++15:45
zbr|ruckjbadiapa: depends on the job, can you find a "reproducer" folder among long?15:45
zbr|ruck(logs)15:45
jbadiapazbr|ruck, the main point is that we are trying to build the "test"  https://zuul.opendev.org/t/openstack/build/2b69dad75122434b8c046bafd307f54515:46
* rlandy needs to run out fora few - brb15:47
raukadahweshay: rlandy https://review.opendev.org/#/c/686149/ is good to go15:47
raukadahneeded for tobiko15:47
*** saneax has joined #oooq15:47
jbadiapazbr|ruck, and we havent done anything to add the replicate script on it15:47
rlandyraukadah: w+15:47
raukadahrlandy: thanks!15:47
weshaytripleo-ci-rhel-8-scenario001-standalone-rdoSUCCESS in 1h 11m 17s15:47
weshaytripleo-ci-rhel-8-scenario002-standalone-rdoSUCCESS in 1h 06m 43s15:47
weshaylook at that15:47
weshayzbr|ruck, one more request..15:48
weshayzbr|ruck, rhel-8 scenario003 is passing in periodic..15:48
weshaylet's add it to check please15:48
zbr|ruckjbadiapa: brb, i don't know about this molecule job but I know about molecule, should be easy to reproduce.15:49
rlandyzbr|ruck: will be back in a few if you need me to review anything else15:49
*** rlandy is now known as rlandy|brb15:49
zbr|ruckhttps://review.rdoproject.org/r/#/c/22944/15:53
weshayzbr|ruck, rock wf.. and then add it check as voting :)15:53
zbr|rucki am not aware of any place with this job cofigured as non voting15:55
pandadammit15:56
zbr|ruckha! found one.15:56
*** SurajPatil has joined #oooq15:58
zbr|ruckweshay: https://review.rdoproject.org/r/#/c/22945/16:00
*** surpatil has quit IRC16:01
*** marios has quit IRC16:01
*** tosky has quit IRC16:05
*** ykarel|afk is now known as ykarel16:07
pandarfolco: merged marios patch16:10
pandarfolco: now I'm working on the finalization (the remaining parts)16:10
pandarfolco: I'll take care of the file triggering as part of the finalization16:11
rfolcopanda, ok I am still trying to understand where do I get the list of containers.... stage-config, stage-info... too many places16:11
rfolcofrom stage-config, containers['images-suffix']16:12
rfolcohttps://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/tests/staging-setup/stage-config.yaml#L11-L1916:12
*** SurajPatil has quit IRC16:12
rfolcofrom stage-info, would need to parse...16:12
rfolcocontainers:16:12
rfolco- tripleomaster/centos-binary-base:1c67b1ab8c6fe273d4e175a14f0df5d3cbbd0edc_8170b86816:12
pandarfolco: stage-config is the source for the staging setup env. That is used in combination with the dlrn db fixtures to create the staging environment. stage_info.yaml contais the effect of the creation16:12
rfolcopanda, molecule test creates parsed-containers txt file16:13
pandarfolco: generally stage-config should be used only by the stagin environment, stag_info is what the rest of the test should look at16:13
rfolcoI need container names only16:13
pandarfolco: like "base" "openstack-base" ?16:14
pandarfolco: or centos-binary-base ?16:14
rfolcopanda, need base openstack-base only16:14
rfolcofor image prepare16:14
rfolcoI used the parsed file for molecule16:14
pandarfolco: so the image_suffixes ?16:15
rfolco    containers_file: "../container-push/parsed_containers-{{ distro_name }}-{{ distro_version }}-{{ release }}.txt"16:15
pandarfolco: the parsed file should be producesd also in integration test16:15
rfolco          with_items: "{{ lookup('file', containers_file ).splitlines() }}"16:15
pandais it not ?16:15
rfolcopanda, I think you create it on container-push molecule test16:15
pandarfolco: not me, it's the container-push playbook that creates it16:16
pandarfolco: it's part of the workflow16:16
rfolcopanda, so i'll just use it16:16
rfolcok16:16
pandarfolco: as part of the test instead I use a pattern file to grep out most of the names16:16
pandarfolco: yes, you should be able to16:16
rfolcook good16:17
rfolcopanda, target_registries is a bad name... for tripleo-common test docker.io is the source... so I would rename it to promote_registries16:18
rfolcothere is a also a check for type, if its source or target... I did not quite understand that code16:18
pandarfolco: publication_registries ?16:18
rfolcoanyways, will keep dups for now16:18
rfolcopanda, could be16:19
pandarfolco: ok, enqeueuing in the finalization work16:19
weshayrfolco, back16:19
rfolcopanda, cool16:19
rfolcoweshay, joining16:19
*** holser has quit IRC16:19
*** jbadiapa has quit IRC16:21
*** dsneddon_ has quit IRC16:22
weshayzbr|ruck, this is the periodic.. should vote.. thank you.. https://review.rdoproject.org/r/#/c/22945/1/zuul.d/standalone-jobs.yaml16:22
weshayzbr|ruck, need the check job now :)16:22
weshay3rd party check.. like 1 && 216:23
*** jfrancoa has quit IRC16:23
*** ykarel has quit IRC16:24
*** saneax has quit IRC16:27
pandazbr|ruck: did you merge the linter change ?16:29
zbr|ruckyeah16:31
*** rlandy|brb is now known as rlandy16:31
pandazbr|ruck: you just blocked a promotion change to merge.16:32
weshayshould have two +2's16:33
zbr|ruckpanda: links, you made me curious. and yes, no self-merges on my side.16:34
pandazbr|ruck:  https://review.rdoproject.org/r/22772 PS17 passed checks, then your patch merges, failed gates. 20 minutes delay other two fixes to merge this again.16:35
zbr|ruckpanda: like french say: c'est la vie -- it had to rebase these changes <10 times as some people were keep adding code without even reviewing it.16:36
zbr|rucki was afraid that i really broke something16:37
zbr|ruckat least these take 10min to run, not like >4h16:37
*** zbr|ruck has quit IRC16:39
*** dsneddon_ has joined #oooq16:41
*** dsneddon_ has quit IRC16:46
*** bogdando has quit IRC16:47
pandayou know what else french say ?16:47
panda....16:48
pandanothing works now, another 30 minutes to understand what to fix.16:48
pandafor an out of sprint task not even tracked.16:49
pandathat need to go in right now.16:49
*** jfrancoa has joined #oooq16:49
*** zbr has joined #oooq16:50
*** ykarel has joined #oooq16:53
*** dsneddon_ has joined #oooq16:54
zbrweshay: am i correct, this is to be abandoned https://review.opendev.org/#/c/686192/ ?16:57
weshayzbr, hrm.. not sure.. we should probably chat about that16:58
zbrnow?16:58
weshayturning off login in the vars is not a long term option16:58
weshaysec.. in 1-1-16:58
*** dsneddon_ has quit IRC17:02
weshayzbr, if you have a sec, if not it can wait till next week https://meet.google.com/etf-xwxz-kwa17:04
*** derekh has quit IRC17:06
*** jpena is now known as jpena|off17:08
*** chem has joined #oooq17:15
weshayhttp://logs.rdoproject.org/92/686192/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch/31693cd/logs/undercloud/etc/sysconfig/17:16
weshayzbr, ^17:16
*** dsneddon_ has joined #oooq17:22
*** jtomasek has quit IRC17:31
*** dsneddon_ has quit IRC17:32
*** dsneddon_ has joined #oooq17:36
rlandynice ... https://review.rdoproject.org/r/#/c/22772/ merging :)17:54
*** ykarel is now known as ykarel|away17:56
*** amoralej is now known as amoralej|off17:57
pandawoot woot woot woot18:00
pandahttp://logs.rdoproject.org/40/22940/7/check/tripleo-ci-promotion-staging/c9e1413/logs/promoter_logs/centos7_master.log18:00
pandarlandy: rfolco ^18:00
rlandy2019-10-04 17:42:54,171 5591 INFO     promoter FINISHED promotion process18:01
rlandypanda++18:01
rlandy2019-10-04 17:42:57.557977 | rdo-centos-7 | AttributeError: 'Namespace' object has no attribute 'stage_info_path'18:02
pandait's not over18:03
pandabut we're closer and we have basic funcinality18:03
rlandypanda: is the staging info file anywhere I can see it yet?18:04
pandarlandy: no18:04
pandarlandy: adding to the things to do next18:04
rlandypanda: ack - ok18:05
rlandypanda: should I change the test in the same patch or a different one?18:08
pandarlandy: 22940 ? I'm going to work on that a lot still18:09
rlandypanda: ok - so I am going to edit the test on a patch based off of that18:09
rlandy 'Namespace' object has no attribute 'stage_info_path'18:10
rlandy^^ fix that etc.18:10
rlandyenable log testing18:10
*** rfolco is now known as rfolco|bbl18:12
rlandypanda: https://review.rdoproject.org/r/#/c/22940/ - see comment on typo for path18:22
pandarlandy: thaks new PS incoming18:24
rlandyugh - nova's baaaaaaaack18:25
rlandypanda: one more comment ... https://review.rdoproject.org/r/#/c/22940/18:37
rlandyI can fix things bur probably easier not to mess in your active patch18:37
rlandyfixing in my patch anyways18:39
rlandynot blocking me18:40
*** chem has quit IRC19:12
rlandypanda: just fyi ... adding changes to https://review.rdoproject.org/r/#/c/22958/19:13
pandarlandy: ok19:15
rlandyformulation of target regs are unclear atm19:16
pandarlandy: new patcset fixed error in tox jobs19:16
rlandywill rebase19:16
pandarlandy: ?19:16
rlandypanda: see change to staging -info19:16
rlandyin post for collect logs19:17
pandarlandy: which line19:17
rlandycommented19:17
*** jfrancoa has quit IRC19:18
rlandyneed to see what stage info looks like19:18
rlandyto see target host19:18
rlandyI think I changed it correctly - we will see19:18
pandarlandy: ok, I think that part will continue on your patch, it makes sens since it's needed for your test19:19
rlandyack - fixing pieces of that now19:19
rlandypanda: we're getting there19:31
pandarlandy: I think we can merge 22940  .... I thought I would add more and more stuff but after seeing the "FINISHED promotion" message my spirit is lifted and energies are slowly fading19:31
*** ykarel|away has quit IRC19:31
rlandypanda: ack - fine by me - will +219:31
rlandyI will work in my current patch fixing stuff19:32
pandaweshay: wehn did you want to provision the new promoter server and do some real-life testing ? get something on the calendar as soon as possibile so we know that this works.19:32
rlandyfeel free to pick it up where I leave off19:32
pandarlandy: I will jump from time to time tonight to see if you need help but I probably won't do anything else19:32
rlandypanda: get some rest19:32
rlandypanda: you will be on on monday before I arrive19:33
pandarlandy: ack19:33
rlandypanda: besides - it's good for me to try get this right on my own :)19:33
rlandypanda look - http://logs.rdoproject.org/58/22958/7/check/tripleo-ci-promotion-staging/fe92582/logs/stage-info.yaml19:34
rlandywoohoo19:34
rlandyyay - happiness is ... not debugging in the dark19:34
pandarlandy: \o/19:34
rlandy2019-10-04 19:30:00.434841 | rdo-centos-7 |   File "/home/zuul/ci-config/ci-scripts/dlrnapi_promoter/tests/staging-setup/promoter_test.py", line 97, in query_container_registry_promotion19:35
rlandy2019-10-04 19:30:00.434905 | rdo-centos-7 |     assert missing_images == [], "Images are missing"19:35
rlandy2019-10-04 19:30:00.434947 | rdo-centos-7 | AssertionError: Images are missing19:35
rlandy2019-10-04 19:30:00.801914 | rdo-centos-7 | ERROR19:35
rlandy^^ which means the first test passed19:35
rlandyand the second is close19:35
*** saneax has joined #oooq19:36
rlandypanda: have a good weekend and thanks for the help19:37
pandarlandy: commits 0 and 1 shuold probalby be tripleo-ci-staging-promoted and previous-tripleo-ci-staging-promoted19:37
pandarlandy: all right, have fun then :)19:37
*** panda is now known as panda|off19:38
rlandyah19:38
*** zbr has quit IRC19:46
*** saneax has quit IRC20:01
*** EmilienM is now known as EvilienM20:18
*** EvilienM is now known as containerizes_hi20:19
*** containerizes_hi is now known as containerized20:19
*** containerized is now known as EvilienM20:19
rlandyrfolco|bbl: ping ...20:38
rlandy019-10-04 20:31:02.510274 | rdo-centos-7 | DEBUG:promoter-staging:Pushing container tripleomaster/centos-binary-neutron-server:1c67b1ab8c6fe273d4e175a14f0df5d3cbbd0edc_8170b868_ppc64le to localhost:600020:38
rlandy2019-10-04 20:31:02.510396 | rdo-centos-7 | DEBUG:promoter-staging:Pushing container tripleomaster/centos-binary-neutron-server:1c67b1ab8c6fe273d4e175a14f0df5d3cbbd0edc_8170b868_x86_64 to localhost:600020:38
rlandy2019-10-04 20:31:02.517078 | rdo-centos-7 | DEBUG:urllib3.connectionpool:http://localhost:None "POST /v1.35/images/sha256:f7d1bb53b2ef862dd569d4818f27c3135fc8f82045f9627eb651a394ee39ac8d/tag?repo=localhost%3A6000%2Ftripleomaster%2Fcentos-binary-neutron-server&tag=1c67b1ab8c6fe273d4e175a14f0df5d3cbbd0edc_8170b868_x86_64&force=0 HTTP/1.1" 201 020:38
rlandy^^ shouldn't that be 6500?20:38
rlandynvm20:38
*** slaweq has quit IRC20:59
*** holser has joined #oooq21:17
*** aakarsh has quit IRC21:17
*** holser has quit IRC21:27
*** derekh has joined #oooq21:29
*** derekh has quit IRC21:29
*** tosky has joined #oooq21:31
*** rlandy has quit IRC21:40
*** jbadiapa has joined #oooq21:50
*** EvilienM is now known as EmilienM21:52
*** rfolco|bbl has quit IRC22:54
*** aakarsh has joined #oooq23:05
*** slaweq has joined #oooq23:11
*** slaweq has quit IRC23:16
*** tosky has quit IRC23:39

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!