Friday, 2020-07-17

*** ysandeep|away is now known as ysandeep00:04
ysandeeprlandy|ruck|afk, around ?00:04
ysandeeprlandy|ruck|afk, tripleo component was expected to fail, I added a reference in RR notes about it yesterday00:05
ysandeeprlandy|ruck|afk, To fix we need stevedore3.1.0(which is in client component), Because stevedore and heat are in different components, so need to trick the promotions to get both component promotion (clients/tripleo)together (may be by manual promotion/relax criteria of these component)00:06
*** saneax has quit IRC00:08
rlandy|ruck|afkysandeep: sec - on phone00:21
rlandy|ruck|afkok - I'll take care of it00:21
*** saneax has joined #oooq00:38
*** rlandy|ruck|afk is now known as rlandy|ruck01:13
rlandy|ruckysandeep: hi - back01:13
rlandy|ruckysandeep: let me see what's blocking the client promotion01:14
rlandy|ruckor maybe it's juts a time thing until clients up and down promotes01:14
rlandy|ruckysandeep: I'll assign the bug to me to make sure we get the right versions in01:15
rlandy|ruckthanks for looking into it01:15
ysandeeprlandy|ruck, thanks!01:15
ysandeeprlandy|ruck, mmmm.. i still see stevedore.2.0.1 in client consistent https://trunk.rdoproject.org/centos8-master/component/clients/consistent/01:16
rlandy|ruckysandeep: rhos-17 promoted today01:16
rlandy|ruckwith octavia01:16
ysandeeprlandy|ruck, thanks! great news01:17
ysandeepabout stevedore.3.0.1 , yatin mentioned this about 3.0.1 :_01:17
ysandeep~~~~01:17
ysandeep<ykarel> ysandeep|rover, yes, so once heat patch merges, tripleo component will start failing01:17
ysandeep<ykarel> for that we need to build stevedore3.1.001:17
ysandeep<ykarel> after building it will be in clients component01:17
ysandeep<ykarel> would need https://review.rdoproject.org/r/#/c/28529/3..4/tags/victoria-uc.yml included01:17
ysandeep<ykarel> it will fail without heat, so need to get it merged to get build01:17
ysandeep~~~01:17
rlandy|ruckhttp://rhos-release.virt.bos.redhat.com:3030/rhosp - that cleaned up01:18
ysandeeprlandy|ruck, wow all green :)01:18
rlandy|ruckysandeep: ok - so let's keep watching clients01:19
rlandy|ruckwhen we see the right version, we make sure to promote01:19
rlandy|ruckalso scenario010 failed in the master line01:19
ysandeeprlandy|ruck, yup01:19
rlandy|ruckso the python-tempestconf fix has not gotten in yet01:19
ysandeeprlandy|ruck, let me see where its stuck01:20
rlandy|ruckysandeep: so you still have access to the tmate? in case you need to restart promoter?01:20
rlandy|ruckdo01:20
ysandeeprlandy|ruck, i have one tmate.. but i am not sure its the old one or new one01:21
rlandy|ruckprobably the old one01:21
rlandy|rucknew one is going fine01:22
ysandeepokay01:22
rlandy|ruckI just need to rerun fs001 so there is a hash on master to promote when chandan wants to test his changes01:29
rlandy|ruckI may just run it on rdocloud01:29
rlandy|ruckvexxhost still fails a ton01:29
rlandy|ruckoh -k - master kicked again ...01:36
rlandy|ruckysandeep: if fs001 fails, pls rerun ( even if you have to use rdocloud) for ^^01:37
ysandeeprlandy|ruck, testproject ?01:38
rlandy|ruckysandeep: yep - it may pass01:40
rlandy|ruckyou never know01:40
rlandy|ruckmiracles happen01:40
rlandy|ruckstill running01:40
ysandeeprlandy|ruck, could you please pass me your test patch, i will keep an eye01:40
rlandy|ruckysandeep: I was going to kick a test patch but then the hash rev'ed because master kicked again  ...01:41
rlandy|ruckif you need to rekick fs001 ... https://review.rdoproject.org/r/#/c/21683/01:42
ysandeeprlandy|ruck, thanks! will keep an eye on main line for now01:42
rlandy|ruckysandeep++ thanks01:43
ysandeeprlandy|ruck, could you please merge https://code.engineering.redhat.com/gerrit/#/c/206448/01:43
ysandeeptestrun was green yesterday01:43
ysandeepperiodic-tripleo-ci-rhel-8-multinode-1ctlr-featureset010-rhos-17 https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/f5942977eae6429dabdafabb0f54dc6b : SUCCESS in 1h 38m 48s01:44
rlandy|ruckyep01:44
rlandy|rucksurprise!01:44
ysandeeprlandy++ thanks!01:44
rlandy|ruckysandeep: need anything else before I log off?01:45
ysandeeprlandy|ruck, no I think I am good, thanks!01:46
ysandeeprlandy|ruck, have a great evening01:46
rlandy|ruckysandeep: thanks - see you later01:46
*** rlandy|ruck has quit IRC01:46
*** ysandeep is now known as ysandeep|away01:49
*** apetrich has quit IRC02:14
*** Goneri has quit IRC02:41
*** rfolco has quit IRC02:41
*** saneax has quit IRC03:18
*** skramaja has joined #oooq03:58
*** ykarel has joined #oooq04:52
*** udesale has joined #oooq04:57
*** ratailor has joined #oooq05:29
*** soniya29 has joined #oooq05:36
*** ysandeep|away is now known as ysandeep06:09
*** amoralej|off is now known as amoralej06:19
*** ysandeep is now known as ysandeep|rover06:24
*** marios has joined #oooq06:47
ysandeep|roverbhagyashris, I have left a comment on https://review.rdoproject.org/r/#/c/28282/06:47
bhagyashrisysandeep|rover, ack06:48
bhagyashristhanks earlier you point out me but i forgot06:49
bhagyashrisysandeep|rover, i will do the needful06:49
ysandeep|roverbhagyashris, thank you :)06:49
ysandeep|roverbhagyashris, we can -w till you make cherrypick that patch06:50
ysandeep|roverohh you already did, sry06:50
bhagyashrisysandeep|rover, yup :)06:53
zbr|ruckpanda: if you can have a brief look at https://review.rdoproject.org/r/#/c/28553/ it would be great, just to confirm/infirm my comments.07:13
*** udesale has quit IRC07:17
chandankumarzbr|ruck: marios https://review.rdoproject.org/r/28549 and https://review.rdoproject.org/r/28565 when free07:19
mariosack chandankumar07:20
*** ykarel_ has joined #oooq07:31
*** ykarel has quit IRC07:33
*** tosky has joined #oooq07:33
*** ykarel_ is now known as ykarel07:43
*** udesale has joined #oooq07:57
*** jbadiapa has joined #oooq08:18
*** derekh has joined #oooq08:30
pandazbr|ruck: commented08:31
pandamarios: would you do the honors ? https://review.rdoproject.org/r/2762608:31
*** apetrich has joined #oooq08:33
mariospanda: make me08:38
pandamarios: don't make me go there punk08:38
*** ysandeep|rover is now known as ysandeep|lunch08:41
marioso/ folks please I need votes on a cherrypick when you have time thank you https://review.opendev.org/#/c/741383/09:13
*** dtantsur|afk is now known as dtantsur09:19
*** apetrich has quit IRC09:26
*** ysandeep|lunch is now known as ysandeep|rover09:27
*** bogdando has joined #oooq09:28
*** marios has quit IRC09:38
ysandeep|roverzbr|ruck,  fyi.. I will push a test patch to try to get tripleo/clients components promoted to fix https://bugs.launchpad.net/tripleo/+bug/1887856. Details are here http://paste.openstack.org/show/796040/09:50
openstackLaunchpad bug 1887856 in tripleo "centos-8 master tripleo component tests are failing with "ModuleNotFoundError: No module named 'blazarclient'"" [Critical,In progress] - Assigned to Ronelle Landy (rlandy)09:50
akahatysandeep|rover, ping, something wrong with fs39-train job.09:59
akahathttps://review.rdoproject.org/r/#/c/27523/09:59
akahathttps://logserver.rdoproject.org/23/27523/2/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train/1321dd7/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz09:59
ysandeep|roverakahat, master failing too?10:00
akahatysandeep|rover, not sure about master.10:00
ysandeep|roveryes master failing too https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master10:01
*** udesale_ has joined #oooq10:01
akahatysandeep|rover, yes.. same errors.10:02
ysandeep|roverakahat, lately there were some issues with ovb jobs.. I will run this through ronelle(she will be in a few hours), will open a new bz if required10:03
akahatysandeep|rover, great. thanks :)10:04
*** udesale has quit IRC10:04
ysandeep|roverakahat, you can separate minion and ovb job in different patchs.. and get the minion job merged if you want10:04
akahatysandeep|rover, ack.10:05
chandankumararxcruz: \o/10:07
*** dtantsur is now known as dtantsur|brb10:07
chandankumararxcruz: please have a look at this patch https://review.rdoproject.org/r/#/c/28543/10:08
chandankumarnot sure why skip tests jobs are passing and it is not picking the skiplist10:08
chandankumarzbr|ruck: https://review.rdoproject.org/r/#/c/28565/10:10
zbr|ruckdone10:11
chandankumarzbr|ruck: -w, for cherry picking on promoter?10:15
*** sshnaidm|afk is now known as sshnaidm|off10:16
zbr|rucktbh,... i am kinda tired of cherry picking stuff on the promoter10:16
zbr|ruckhow about making a small change that allows custom local config, so we can avoid cherry picks?10:16
zbr|rucklike looking for custom config in ~/.config/promoter before using in-repo values? so we avoid the need to have a dirty-git on it?10:17
*** apetrich has joined #oooq10:18
chandankumarzbr|ruck: nice idea, but what about new panda's work on new config change to yaml?10:20
zbr|rucks***10:20
zbr|ruckchandankumar: go do the change, i have a tmate still open anyway.10:21
chandankumarzbr|ruck: let's wait for rlandy to come, in another 1 h and 30 mins10:21
chandankumarwe have two changes to cherry-pick10:21
zbr|ruckchandankumar: give https://review.rdoproject.org/r/#/c/28571/ a kick10:23
panda.....10:24
chandankumarysandeep|rover: fyi https://review.rdoproject.org/r/#/c/28571/ merging this, if see any breakage let zbr|ruck, me and rfolco know10:27
arxcruzchandankumar: i'll check, give me a few minutes, i'm not feeling well since yesterday :/10:27
ysandeep|roverykarel, ^^10:27
chandankumararxcruz: oh, get well soon & take rest man10:27
ysandeep|roverchandankumar, ack10:28
chandankumararxcruz: kopecmartin soniya29 akahat https://hackmd.io/fIOKlEBHQfeTZjZmrUaEYQ?view#2020-07-17 feel free to add or remove10:29
*** ykarel_ has joined #oooq10:31
ykarel_ysandeep|rover, ?10:32
ykarel_28571 ?10:32
*** ykarel has quit IRC10:32
ysandeep|roverykarel_, yes fyi..10:34
ykarel_ack10:34
*** ykarel_ is now known as ykarel10:38
*** marios has joined #oooq10:42
ysandeep|roverchandankumar, thanks for your review on my patch, regarding can we also propose a patch to remove c7 train based container build jobs?  --> i will but I heard wes said he want to convey by ML to announce the removal of c7 and introduction of c8 jobs first.. will check with him on monday.10:55
chandankumarysandeep|rover: sure, thanks for the ML part :-)10:56
ysandeep|rover:)10:57
chandankumarzbr|ruck: boom new failure https://logserver.rdoproject.org/49/28549/6/check/tripleo-ci-promotion-staging-single-pipeline-centos-7/fddc028/job-output.txt11:04
chandankumarzbr|ruck: http://paste.openstack.org/show/796042/ looks like a connection issue11:05
chandankumarmarios: https://review.rdoproject.org/r/#/c/28549/ I have updated the commit msg I hope now it is clear11:05
marioschandankumar: ack will you add to master.ini and train here? I don't see in in latest https://review.rdoproject.org/r/#/c/28549/7/ci-scripts/dlrnapi_promoter/config/CentOS-8/master.ini11:07
*** ysandeep|rover is now known as ysandeep|afk11:11
chandankumarmarios: added for c8, for RHEL-8 it is not needed, as we have no promotion pipeline for RHEL-811:12
marioschandankumar: well, we will have internal rhel pipeline ... but ok for now11:13
marioschandankumar: but no openstack-base for master?11:13
chandankumarmarios: yes, there is no openstack-base for master11:13
marioschandankumar: ack11:14
chandankumarmarios: do you want to join the promoter cherry -pick code call?11:14
chandankumarsent the invite, if free, feel free to join that11:15
marioschandankumar: sure11:18
marioschandankumar: checking the invite11:18
mariosthanks chandankumar11:19
chandankumarmarios: :-)11:19
*** soniya29 has quit IRC11:30
*** ysandeep|afk is now known as ysandeep11:31
*** ysandeep is now known as ysandeep|rover11:34
*** apetrich has quit IRC11:34
*** soniya29 has joined #oooq11:45
*** Tengu has quit IRC11:48
*** Tengu has joined #oooq11:50
*** rlandy has joined #oooq11:56
*** rlandy is now known as rlandy|ruck11:56
rlandy|ruckchandankumar: hey ...11:56
chandankumarrlandy|ruck: Hello11:56
rlandy|ruckaccepted your meeting11:57
chandankumarrlandy|ruck: cool, thanks!11:57
rlandy|ruckbut I'm here now if you want to start earlier11:57
chandankumarrlandy|ruck: let's do it after an hour11:57
rlandy|ruckchandankumar: sure ...11:57
rlandy|ruckdo you know if there a hash to promote?11:57
chandankumarzbr|ruck: marios do we also need https://review.rdoproject.org/r/27626 to cherry pick?11:58
chandankumarrlandy|ruck: may be ysandeep|rover and zbr|ruck knows11:58
rlandy|ruckchandankumar: last run looks good11:58
rlandy|ruckwe're fine11:58
rlandy|ruckysandeep|rover: zbr|ruck: ^^ no worries11:58
marioschandankumar: so i am out of context... i don't know what we are cherrypicking to where? rdo promoter?11:59
rlandy|ruckhttps://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-main11:59
marioschandankumar: but that is switching to using the new qcow image promotion11:59
zbr|ruckas the cherry pick list keeps growing, i am going to ask panda: what is the impact of making config file overridable?11:59
chandankumarmarios: yes rdo promoter11:59
marioschandankumar: so yes i think we should have it why not11:59
marioschandankumar: if it is broken we should find out about it now :D11:59
chandankumarmarios: sure :-)11:59
ysandeep|roverrlandy|ruck, To fix https://bugs.launchpad.net/tripleo/+bug/1887856 - With help of test patch we tried to get tripleo/clients components promoted - https://review.rdoproject.org/r/2858612:00
openstackLaunchpad bug 1887856 in tripleo "centos-8 master tripleo component tests are failing with "ModuleNotFoundError: No module named 'blazarclient'"" [Critical,In progress] - Assigned to Ronelle Landy (rlandy)12:00
ysandeep|roverrlandy|ruck, stevedore3.1 reached tripleo-ci-testing12:01
ysandeep|roverrlandy|ruck, we triggered main line to get promotion12:01
rlandy|ruckysandeep|rover: that's great - will save us a headache12:01
rlandy|ruckperiodic-tripleo-ci-centos-8-scenario010-standalone-master12:01
zbr|ruckchandankumar: rlandy|ruck: small bit https://review.rdoproject.org/r/#/c/28590/12:02
rlandy|ruckysandeep|rover: hmm scenario010 still failing12:02
ysandeep|roverrlandy|ruck, ^^ periodic-tripleo-ci-centos-8-scenario010-standalone-master will solve once tempestconf reaches current tripleo right?12:02
rlandy|ruckysandeep|rover: tripleo-ci-testing12:03
rlandy|ruckI'm looking at main-line12:03
* rlandy|ruck checks gates12:03
rlandy|ruckany disasters there?12:03
* rlandy|ruck breathes - ok12:03
ysandeep|roverrlandy|ruck, then we need to take help from ocatavia guys.. its may be the original bug on which carlos patches are pending then12:06
ysandeep|roversc10 not in promotion criteria, should not block promotion for now12:06
rlandy|ruckysandeep|rover: it's not ... but its should be :)12:07
* rlandy|ruck wants to put it back12:07
rlandy|rucklet's check patch status12:07
ysandeep|roverrlandy|ruck, agree12:07
chandankumardpawlik2: https://review.rdoproject.org/r/#/c/28590/ please have a look at this patch12:07
*** rfolco has joined #oooq12:08
*** dtantsur|brb is now known as dtantsur12:10
rlandy|ruckysandeep|rover: anything else burning?12:12
rlandy|ruckrestarted ci,centos queens12:12
rlandy|ruckthat should clear12:12
ysandeep|roverrlandy|ruck, nah all good, just fyi.. https://review.rdoproject.org/r/#/c/28571/ incase it break something12:13
rlandy|ruckoh dear12:14
rlandy|ruckysandeep|rover: ok  we'll have to watch downstream on impact from that change12:15
rlandy|ruckrelease is osp_release there12:15
rlandy|ruckbut we may get by12:16
ysandeep|roverrlandy|ruck, fyi.. in downstream tripleo component seems also busted because of that stevedore3.1 thing .. we need to do the trick there as well once stevedore3.1 reaches clients12:18
*** ysandeep|rover is now known as ysandeep|coffee12:19
rfolcozbr|ruck, o/12:19
*** ratailor has quit IRC12:20
rfolcozbr|ruck, can I work on the other scenarios now ? promote_images, container_push, tripleo-common-integration ?12:20
rlandy|ruckysandeep|coffee: np - the changes will get imported downstream and yes we can rerun/bypass criteria if need be12:21
rfolcozbr|ruck, there is still some work to be done on promoter mol..12:21
zbr|ruckrfolco: i am sure there is, but now we are in much better shape.12:22
rfolcozbr|ruck, yep, thanks for your great help12:23
rfolcozbr|ruck, for example, provisioning playbook is not idempotent... I can list in hackmd all the improvements required in promoter mol12:24
zbr|rucksure, add them there and start fixing them, one at a time12:25
rfolcozbr|ruck, ok12:25
chandankumarzbr|ruck: do we also need to cherry pick rsync fix also?12:28
zbr|rucknope12:29
chandankumarok12:29
chandankumararxcruz: kopecmartin akahat soniya29 tempest meeting time12:30
zbr|ruckchandankumar: https://review.rdoproject.org/r/#/c/28549/8/ci-scripts/dlrnapi_promoter/config/CentOS-8/master.ini12:34
chandankumarmarios:  https://review.opendev.org/#/c/726773/ please have a look12:34
zbr|ruckif is ok, add +W and we will only pull master12:35
marioschandankumar: will do middle rebase conflict right now12:35
zbr|rucki need vote on https://review.rdoproject.org/r/#/c/28587/ to workflow it.12:36
*** ysandeep|coffee is now known as ysandeep|rover12:40
chandankumararxcruz: https://f8c6a5addd2917e07fe1-e912075c69482e8155cd5ddb4c09699c.ssl.cf1.rackcdn.com/726773/10/check/tripleo-ci-centos-8-undercloud-containers/dcba488/logs/reproducer-quickstart/reproducer-fresh-box-setup.sh12:41
chandankumarzbr|ruck: openstack-base is not needed12:44
chandankumarmarios: np, :-)12:45
chandankumarakahat: use this https://review.opendev.org/#/c/738488/ as a depends on fs062 patch12:48
chandankumarand other also12:48
akahatchandankumar, ack12:49
*** amoralej is now known as amoralej|lunch12:55
*** pojadhav is now known as pojadhav|brb12:59
chandankumarrlandy|ruck: meeting for cherry picking13:01
dpawlik2rlandy|ruck, chandankumar: could you check  https://review.rdoproject.org/r/#/c/28550/ ?13:07
*** TrevorV has joined #oooq13:12
*** amoralej|lunch is now known as amoralej13:21
*** pojadhav|brb is now known as pojadhav13:26
*** skramaja has quit IRC13:29
chandankumarrlandy|ruck: marios zbr|ruck ysandeep|rover thank you :-)13:43
marioschandankumar: heh  beat me to it13:44
mariosthanks rlandy|ruck zbr|ruck chandankumar ysandeep|rover13:44
rlandy|ruckchandankumar: marios: thanks for your help13:44
ysandeep|rovermarios, chandankumar rlandy|ruck zbr|ruck thank you, and happy weekend! bye o/13:44
rlandy|ruckysandeep|rover: bye - see you monday13:45
mariosrlandy|ruck: thanks for driving13:45
mariosysandeep|rover: have a nice w/e13:45
mariosok i think we're done with thanks now13:45
marios:D13:45
chandankumarysandeep|rover: happy weekend, see ya on monday!13:45
chandankumarwe finally deployed in production on Friday :D13:46
marioschandankumar: careful now13:47
marioschandankumar: we only *ran* it so far13:48
marioschandankumar: we don't know if it will stay running :D13:48
marioslet's see how that promotion goes13:48
chandankumarmarios: let's see :-)13:48
* rlandy|ruck runs away ...13:48
rlandy|rucklet's hope Friday afternoon we don't regret this13:48
chandankumarrlandy|ruck: marios https://review.rdoproject.org/r/#/c/28593/ paramiko change13:49
rlandy|ruckthanks13:49
chandankumarthanks zbr|ruck :-)13:50
*** marios has quit IRC13:53
*** marios has joined #oooq13:57
*** Goneri has joined #oooq13:57
*** ysandeep|rover is now known as ysandeep|away13:58
*** apetrich has joined #oooq14:08
rlandy|ruck2020-07-17 14:07:44,632 p=2024 u=centos n=ansible | localhost                  : ok=14   changed=8    unreachable=0    failed=114:09
rlandy|ruckwe have a failuer14:09
rlandy|ruckfailure14:10
rlandy|ruckzbr|ruck: ^^14:10
rlandy|ruckchandankumar: hmmm ... 2020-07-17 14:05:24,472 p=2024 u=centos n=ansible | failed: [localhost] (item=openstack-base) => {"ansible_loop_var": "item", "changed": false, "item": "openstack-base", "msg": "Error pulling image trunk.registry.rdoproject.org/tripleomaster/centos-binary-openstack-base:405d4eb1b2f7cb39424978225a9bacfc - 404 Client Error: Not Found (\"manifest for trunk.registry.rdoproject.org/tripleomaster/centos-binar14:10
rlandy|rucky-openstack-base:405d4eb1b2f7cb39424978225a9bacfc not found: manifest unknown: manifest unknown\")"}14:10
rlandy|ruckstill picking up14:10
zbr|ruckclueless14:11
* rlandy|ruck checks14:11
chandankumarrlandy|ruck: it means var from criteria file donot picked up14:11
chandankumarlet me check14:11
rlandy|ruckchandankumar: I'll put back the old change14:11
chandankumarrlandy|ruck: let me see14:12
chandankumarok14:12
rlandy|ruckchandankumar: k - let me know14:12
rlandy|ruckotherwise we can go back to the other change14:12
bogdandohola, PTAL https://review.opendev.org/#/c/696306/14:18
bogdandosince 11/2019 it's almost ready I hope! :D14:18
rlandy|ruckchandankumar: hey ...14:35
rlandy|ruckwhy don;t we just set https://review.rdoproject.org/r/#/c/28549/9/ci-scripts/container-push/roles/containers-promote/defaults/main.yml with releases?14:35
chandankumarrlandy|ruck: change coming14:36
rlandy|ruckchandankumar: ok14:36
* rlandy|ruck waits14:36
chandankumarrlandy|ruck: zbr|ruck https://review.rdoproject.org/r/#/c/28594/14:39
* rlandy|ruck looks14:40
rlandy|ruckchandankumar: left one comment on https://review.rdoproject.org/r/#/c/28594/14:43
rlandy|rucksince newworkflow will go forward14:43
chandankumarok14:44
chandankumarupdate coming14:44
rlandy|ruckchandankumar: then we can cherry-pick again :)14:45
chandankumarrlandy|ruck: done14:46
chandankumarrlandy|ruck: let the zuul gives +114:48
rlandy|rucklooking14:49
rlandy|ruckchandankumar: it will take a bit to get zuul +1 - but agree to wait of you have the time14:51
chandankumarrlandy|ruck: I am still around for 1 hr approx14:52
rlandy|ruckok14:52
ykarelrlandy|ruck, zbr|ruck is https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-master/db18d83/job-output.txt known?15:01
ykareli see similar failure in other job15:01
chandankumarykarel: I think we need to pin heat tempest plugin to a good known hash15:02
chandankumar?15:02
ykarelchandankumar, which commit broke it?15:02
* rlandy|ruck looks15:03
chandankumarthere is no change from ladt one month in heat tempest plugin15:03
chandankumarlet me try to reproduce it locally15:04
ykarelhttps://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-master/db18d83/logs/undercloud/var/log/containers/heat/heat_api.log.txt.gz15:05
chandankumarhttps://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/standalone-jobs.yaml#L520 -> sc02 - heat tempest plugin is coming telemetry tempest plugin15:07
chandankumarit broke at while listing tests15:08
chandankumarykarel: can you check did something got changed on heat client?15:12
ykarelchandankumar, ack looking15:14
chandankumarykarel: it is working fine on local system15:15
ykarelchandankumar, heatclient not changed15:15
ykarelit's 2.2.0 in currenttripleo https://trunk.rdoproject.org/centos8-master/component/clients/current-tripleo/15:15
ykareland same in promotion15:15
ykarelu seen above logs? heat-api15:16
chandankumarlet me check heat-api log15:16
ykareland heat engine https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-master/db18d83/logs/undercloud/var/log/containers/heat/heat-engine.log.txt.gz15:16
chandankumarykarel: oslo messageing ampqdriver15:17
ykarelalso see heat engine ^15:17
rlandy|ruckchandankumar: zuul -1'ed - we need to exclude staging15:18
chandankumarykarel: yes I think ysandeep|away filed abbug15:18
ykarelwith stevedore-3.1.0 and latest heat it should be there15:18
ykarelchandankumar, yes, but ^ it should have solved15:18
rlandy|ruckchandankumar: https://bugs.launchpad.net/tripleo/+bug/188785615:18
openstackLaunchpad bug 1887856 in tripleo "centos-8 master tripleo component tests are failing with "ModuleNotFoundError: No module named 'blazarclient'"" [Critical,In progress] - Assigned to Ronelle Landy (rlandy)15:18
chandankumarykarel: https://bugs.launchpad.net/tripleo/+bug/188785615:18
ykarelchandankumar, yes this one15:18
rlandy|ruckwe need to promote the component line15:18
ykarelrlandy|ruck, it was promoted, and still issue is coming ^15:19
rlandy|ruckykarel: if it's failing in tripleo-ci-testing then yes - current tripleo hasn't promoted yet15:19
rlandy|ruckthat's what we are working on now15:20
rlandy|ruckpromoting master15:20
ykarelwithout getting jobs pass?15:20
ykarelthose jobs should pass for promotion current-tripleo15:20
rlandy|ruckykarel: I assigned the bug to me so I can check the versions after promotion15:20
chandankumarrlandy|ruck: let me go back to my patch15:20
rlandy|ruckykarel: there is a problem on the promoter itself15:20
rlandy|ruckthere is a hash to promote15:21
ykarelpromotion will block gates too if failures are ignored i think15:21
rlandy|ruckykarel: scenario002 where you picked up the bug is in criteria15:21
*** amoralej is now known as amoralej|off15:21
ykarelhttps://trunk.rdoproject.org/api-centos8-master-uc/api/civotes_agg_detail.html?ref_hash=4eb890a0003fd6ee429cbda0f20cd81815:22
ykarelhmm i meant don't adjust criteria to promote15:23
rlandy|ruckykarel: if anthiny I'll adjust the criteria up - not down15:24
rlandy|ruckanything15:24
ykarelack good15:24
rlandy|ruckchandankumar: I think release is the wrong way to go here15:24
chandankumarrlandy|ruck: then ?15:24
rlandy|ruckchandankumar: does the promoter know about release??15:24
chandankumarrlandy|ruck: yes, it knows above env15:25
rlandy|ruckit knows about the release to promote15:25
rlandy|ruckrelease: "{{ lookup('env','RELEASE') }}"15:26
rlandy|ruckwhere is that set?15:26
rlandy|ruckchandankumar: https://meet.google.com/?hs=197&pli=1&authuser=015:27
chandankumarjoingin15:29
*** ykarel is now known as ykarel|away15:29
chandankumarrlandy|ruck: url wrong i think15:30
* marios out15:30
rlandy|ruckhttps://meet.google.com/owp-izmc-ufj15:30
rlandy|ruckchandankumar: ^^15:30
mariosrlandy|ruck: need me to join?15:30
rlandy|ruckthat work for you?15:30
mariosrlandy|ruck: i can stay few more mins if need?15:30
chandankumarrlandy|ruck: rejoined15:32
*** udesale_ has quit IRC15:37
*** bogdando has quit IRC15:49
*** derekh has quit IRC15:55
chandankumarrlandy|ruck: https://review.rdoproject.org/r/#/c/27626/15:59
*** ykarel|away has quit IRC16:07
*** marios is now known as marios|out16:12
*** dtantsur is now known as dtantsur|afk16:13
*** jbadiapa has quit IRC16:21
*** marios|out has quit IRC16:23
*** yolanda has quit IRC16:39
*** soniya29 has quit IRC17:18
*** owalsh has quit IRC17:21
rlandy|ruck"ceilometer-ipmi"], "msg": "Error pushing image docker.io/tripleomaster/centos-binary-ceilometer-ipmi: received unexpected HTTP status: 525 Origin SSL Handshake Error"}17:27
rlandy|ruck2020-07-17 16:26:40,586 p=18006 u=centos n=ansible | changed: [localhost] => (item=[u'405d4eb1b2f7cb39424978225a9bacfc', u'ceilometer-central']) => {"actions": ["Tagged ima17:27
*** owalsh has joined #oooq17:42
chandankumarrlandy|ruck: Does the container promotion worked?17:49
rlandy|ruckchandankumar: hey ...17:58
rlandy|ruckso yes and no17:58
rlandy|ruckwe got one container push failure17:58
rlandy|ruckwhich I think was juts a timeout17:58
rlandy|rucktrain is promoting now17:58
rlandy|ruckand I didn;t want to stop it17:58
chandankumarrlandy|ruck: ah ok17:59
rlandy|ruckchandankumar: I am going to restart the promoter after train is done17:59
rlandy|ruckso we don;t wait17:59
chandankumarrlandy|ruck: sure17:59
rlandy|ruckand try again17:59
rlandy|ruck"ceilometer-ipmi"], "msg": "Error pushing image docker.io/tripleomaster/centos-binary-ceilometer-ipmi: received unexpected HTTP status: 525 Origin SSL Handshake Error"}17:59
rlandy|rucklooks like a regular timeout18:00
rlandy|ruckI checked - the container is there ...18:00
rlandy|ruckhttps://trunk.registry.rdoproject.org:8443/oapi/v1/namespaces/tripleomaster/imagestreamtags/18:00
rlandy|ruckso I think it will work on the next run18:00
chandankumarrlandy|ruck: ok18:02
chandankumarjust drop me a message, if more needed18:02
chandankumarrlandy|ruck: Happy weekend, see ya :-)18:02
rlandy|ruckchandankumar: thanks but you've done so much18:02
*** chandankumar is now known as raukadah18:02
rlandy|rucktake some time off :)18:02
rlandy|ruck promoter Containers promote 'aggregate: 9ddf21901cfc43a2995e5ee1583a0296, commit: 304866417dd09bb9d3da238c4305d05b56a9bc56, distro: 9083591c4d8909e14d4339c47bd8e981169fd370, component: ui, timestamp: 1594958594' to current-tripleo: Successful promotion18:23
*** johnsom has quit IRC18:24
*** johnsom has joined #oooq18:25
rlandy|ruckzbr|ruck: you around?18:34
rlandy|ruckwe got a image promotion issue18:34
rlandy|rucknvm - think fixed it18:52
*** owalsh has quit IRC19:20
*** owalsh has joined #oooq19:29
*** TrevorV has quit IRC19:35
*** rlandy|ruck is now known as rlandy|ruck|mtg19:36
*** rlandy|ruck|mtg is now known as rlandy|ruck20:29
*** rlandy|ruck has quit IRC22:45
*** tosky_ has joined #oooq22:52
*** tosky has quit IRC22:53
*** tosky_ is now known as tosky23:01
*** rfolco has quit IRC23:16
*** tosky has quit IRC23:38
*** rfolco has joined #oooq23:41

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!