Friday, 2018-10-05

« Thursday, 2018-10-04 Index Saturday, 2018-10-06 »
*** hamzy_ has joined #oooq00:01
*** hamzy_ is now known as hamzy00:02
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario008-multinode-oooq-container, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario004-multinode-oooq-container, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario007-multinode- (4 more messages)00:51
*** agopi has joined #oooq01:57
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master, tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-scenario004 (3 more messages)02:51
*** agopi has quit IRC03:21
*** agopi has joined #oooq03:21
*** rlandy has quit IRC03:22
*** jaganathan has joined #oooq03:23
*** udesale has joined #oooq03:49
*** ykarel__ has joined #oooq04:15
*** ykarel__ is now known as ykarel04:34
*** holser_ has joined #oooq04:48
*** holser_ has quit IRC04:51
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-undercloud-upgrades, tripleo-ci-centos-7-scenario009-multinode-oooq, tripleo-ci- (1 more message)04:51
*** jbadiapa has quit IRC04:52
*** d0ugal has quit IRC04:52
*** jbadiapa has joined #oooq04:57
*** d0ugal has joined #oooq04:57
*** agopi has quit IRC05:03
*** ratailor has joined #oooq05:18
*** agopi has joined #oooq05:24
*** jtomasek has joined #oooq05:45
*** jfrancoa has joined #oooq06:03
quiquell|offsshnaidm: Are yoiu there sir ?06:08
*** udesale has quit IRC06:39
*** kopecmartin|scho is now known as kopecmartin|ruck06:41
*** quiquell|off is now known as quiquell|brb06:50
quiquell|brbsshnaidm: Running periodics at check https://review.rdoproject.org/r/#/c/16806/ :-)06:50
quiquell|brbsshnaidm: Let's see where we are06:51
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-undercloud-upgrades, tripleo-ci-centos-7-scenario009-multinode-oooq, tripleo-ci- (1 more message)06:51
*** udesale has joined #oooq06:53
*** quiquell|brb is now known as quiquell07:08
*** amoralej|off is now known as amoralej07:17
*** holser_ has joined #oooq07:18
*** skramaja has joined #oooq07:29
*** skramaja_ has joined #oooq07:38
*** skramaja has quit IRC07:38
*** bogdando has joined #oooq07:39
*** tosky has joined #oooq07:45
*** ykarel has quit IRC07:46
*** ykarel has joined #oooq07:46
bogdandohi. Can we send https://review.openstack.org/#/c/593103/ done?07:58
bogdandothere is https://review.openstack.org/#/c/576746/ awaiting for it07:58
*** panda|off is now known as panda08:40
pandaquiquell: how's going ?08:41
quiquellpanda: testing periodic at check with a little hack https://review.rdoproject.org/r/#/c/1680608:42
quiquellpanda: consistent-to-tripleo-ci-testing working fine08:43
quiquellpanda: build-containers working fine08:43
quiquellI think they don't are not false positives08:43
quiquellpanda: confirmed that depandants don't run before post-run08:43
quiquellpanda: post-run failing because we don't have the logs tripleo-ci change merged  :-( I am trying to priorize this08:44
quiquellpanda: I see weird stuff with branch_override08:44
*** ykarel_ has joined #oooq08:44
quiquellpanda: Have not check 'the check' job08:45
pandaquiquell: like what ?08:46
quiquellpanda: So branch_override is correctly set here https://logs.rdoproject.org/06/16806/4/check/periodic-tripleo-centos-7-rocky-containers-build/6c32228/08:46
pandaquiquell: if branch_override is not working, repo-seupt is isntalling the wrong repo, and we are building the wrong containers ...08:46
quiquellpanda: the opposite, is correclty set, but I don't know where08:46
quiquellpanda: It's using rocky as expected08:47
pandaquiquell: in the configuration08:47
quiquellpanda: where ?08:47
pandaquiquell: I added them to the job configuration08:47
pandain config08:47
pandafetching08:47
*** ykarel has quit IRC08:47
quiquellpanda: Ahh have to fetch..08:47
pandaquiquell: https://review.rdoproject.org/r/gitweb?p=config.git;a=blob;f=zuul.d/jobs.yaml;h=254a977a017f279ff9ea43a1c7eb8113ef21cccd;hb=refs/heads/master#l57608:49
*** ykarel_ is now known as ykarel|lunch08:51
quiquellpanda: I see it know, I was making the change :-)08:51
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-undercloud-upgrades, tripleo-ci-centos-7-scenario009-multinode-oooq, tripleo-ci- (1 more message)08:51
quiquellpanda: https://logs.rdoproject.org/06/16806/4/check/periodic-tripleo-centos-7-master-containers-build/3263767/job-output.txt.gz#_2018-10-05_08_08_09_63746608:52
quiquellpanda: ^ this is related to the fix a tripleo-ci not yet merged ?08:53
pandaquiquell: looks like it is08:54
quiquellpanda: going to add the branch_override for the periodics at rdo-jobs08:55
pandaquiquell: in all of them ?08:56
pandaquiquell: yeah, makes sense08:56
quiquellpanda: yep, do we want to remove ocata ?08:56
pandaquiquell: the problem with your test is that zuul.change is defined. If it is defined the log_path take a completely different shape, and something fails08:56
pandaquiquell: not yet08:56
quiquellpanda: I can undefine it08:57
quiquellpanda: in the hack08:57
quiquellpanda: so we don't have to merge config to test pipeline08:57
quiquellpanda: Humm I think I can't...08:58
pandaquiquell: it's better to understan where it fails08:58
quiquellpanda: At least we can check those preliminary jobs and the overrdie_branch and the dependant stuff08:58
pandaquiquell: yes. i don't think LOG PATH is important there, ti just makes the job fail everytime09:03
pandaquiquell: it's branch_override, not override branch09:03
pandaquiquell: don't make the wrong change there :)09:03
quiquellpanda: We can hack that to bypass post, so we can test the rest09:03
quiquellpanda: I am always making this kind of dislexic stuff09:04
quiquellpanda: https://review.rdoproject.org/r/1680809:04
quiquellpanda: rest of job pending, but you can start reviewing that I don't miss any09:04
*** chandankumar has joined #oooq09:06
chandankumarquiquell, marios|rover \o/09:09
quiquellchandankumar: o/09:11
*** agopi has quit IRC09:14
pandaquiquell: 1 comment09:16
quiquellpanda: Do we want to exercise queens and pike in the test job ?09:18
quiquellpanda: Or better test rocky first ?09:18
quiquellpanda: checking at rdo trunk rocky hashes looks like consistent-to-tripleo-ci-testing is working fine09:24
quiquellpanda: We are not using zuul_success at build-containers post09:26
quiquellpanda: It's by design ?09:26
pandano, it is by forgetting09:30
pandaquiquell: we decided to start with the 4h pipeline09:30
pandaquiquell: if there's an important error it will show up in master or rocky09:31
pandaquiquell: we'll enebla and analyze the other pipelines after we are sure the workflow works09:31
quiquellpanda: zuul_success https://review.rdoproject.org/r/1680909:31
pandaquiquell: do you know if dependent jobs are run if containres build fail in post ?09:34
pandaI don't know if post_failure means job failure.09:35
pandaIt's a waste to run all the other jobs if container build fails09:35
quiquellpanda: it means failure happend in the test09:35
quiquellpanda: One of the post has fail at the logs thing, and the dependant didn't trigger09:36
quiquellpanda: I think we are good there09:36
quiquellpanda: There is no parallelization between post and dependant run, but better wait for green at post and compare times09:37
pandaok09:37
pandaquiquell: commented the run09:37
pandaquiquell: the patch09:37
quiquellpanda: Humm why mark post as failure ? run has already fail09:38
quiquellpanda: The whole pipeline will stop09:38
pandaquiquell: you sure ?09:40
pandaquiquell: well at lest a debug message then09:40
quiquellpanda: failure run is failure job, but post will continue09:40
pandaquiquell: to explain why we were skipping09:40
quiquellpanda: ack debug09:40
*** gkadam has joined #oooq09:43
quiquellpanda: https://review.rdoproject.org/r/16809 changed09:45
quiquellpanda: Let's merge it to test it09:46
quiquellpanda: About check jobs, I think we need master and rocky at tq and tqe but we have master at tq and master and queens at tqe09:47
*** chandankumar has quit IRC09:48
quiquellsshnaidm: are you there ?09:49
bogdandossbarnea ssbarnea_ hi, answered https://review.openstack.org/#/c/593103/709:52
bogdandoI think you had missed my original reply to the same concern09:52
bogdandothe code in block, should reside in the block. We cannot make the user module returning the path w/o creating a user, right? And I need no create users when chrooted: true09:53
*** ykarel|lunch is now known as ykarel09:54
pandaquiquell: it's friday09:56
pandaquiquell: i think we'll need to ask other people09:56
pandaquiquell: or wait for rlandy09:56
quiquellpanda: Do you know who else has superpowers at config ?09:56
quiquellpanda: btw, why do we have the check jobs at config ? they don't use secrets09:56
pandaquiquell: the people at #rdo09:57
pandaquiquell: not sure, the translation I made was in rdo-jobs09:57
*** chandankumar has joined #oooq09:59
quiquellpanda: I think this is the translation we want https://review.rdoproject.org/r/1681010:00
arxcruzchandankumar: ping10:05
arxcruzchandankumar: can you please comment on https://tree.taiga.io/project/tripleo-ci-board/task/28?kanban-status=1447276 and confirm that python-tempestconf was merged on openstack/rpm-packages ?10:05
chandankumararxcruz, sure10:07
arxcruzty10:08
quiquellpanda: jpena helping us to merge stuff10:08
chandankumararxcruz, you can move the above card to done.10:10
arxcruzchandankumar: ty10:10
panda\o/10:13
chandankumarquiquell, /me has the access to config repo also10:16
chandankumar:-)10:16
quiquellcool10:17
*** udesale has quit IRC10:18
*** chandankumar has quit IRC10:26
*** sshnaidm is now known as sshnaidm|off10:35
*** jtomasek has quit IRC10:43
quiquellArgg fridays are slow10:47
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-undercloud-upgrades, tripleo-ci-centos-7-scenario009-multinode-oooq, tripleo-ci- (1 more message)10:51
quiquellpanda: we have a bug at tripleo-ci that will prevent to run periodics10:54
*** holser_ has quit IRC10:55
marios|roverquiquell: which one?10:57
marios|roverquiquell: (ironic?)10:57
quiquellmarios|rover: more prosaic https://review.openstack.org/60822510:57
quiquellmarios|rover: I mean new workflow periodics10:58
marios|roverquiquell: k10:58
quiquellpanda: The pile of reviews we need to merge is getting bigger :-(11:04
pandaquiquell: where's chandan ?11:04
pandaquiquell: I see it did some merge11:05
quiquellpanda: chandan has disconnect but he was here11:05
pandahe did some merging11:05
quiquellpanda: The branch_override11:06
quiquellpanda: we can fix that is something is missing11:07
*** jtomasek has joined #oooq11:07
*** dtantsur|afk is now known as dtantsur11:09
pandaquiquell: it was merged11:10
quiquellpanda: tripleo-ci change that we need https://review.openstack.org/#/c/608225/11:10
quiquellpanda: It will take time to merge11:11
pandaquiquell: and the space is giving problems ?11:11
quiquellpanda: yep, |bool evaluate it to false11:12
pandammhhh11:12
quiquell panda https://github.com/openstack-infra/tripleo-ci/blob/master/playbooks/tripleo-ci/templates/toci_gate_test.sh.j2#L11111:13
quiquellpanda: now my test patch is working as periodic11:13
quiquellpanda: And using correct release file promotion-testing-hash-rocky.yml11:14
quiquellSo we are good there11:14
quiquellpanda: Upstream is working fine as PERIODIC=011:16
quiquellpanda: Going for lunch now, all the reviews that need to be merge are here https://tree.taiga.io/project/tripleo-ci-board/task/106?kanban-status=144727511:18
quiquellpanda: Humm that's wrong card... we are testing, maybe you can organize that11:20
*** quiquell is now known as quiquell|lunch11:20
pandaquiquell|lunch: don't get crazy around cards11:32
pandaquiquell|lunch: we're doing way more than what is human possible there :)11:33
*** quiquell|lunch is now known as quiquell11:42
quiquellpanda: LOG_PATH is wrong at the test review, this is expected? even using the special role for it11:47
quiquellpanda: LOG_PATH=06/16806/9/check/periodic-tripleo-centos-7-rocky-containers-build/8c248a011:47
quiquellWe are hitting the limit of this test review I think11:48
quiquellWas good tool11:48
*** jtomasek has quit IRC11:48
pandabecause the container build is doing something strange, it expects the log dir to be there11:52
pandaquiquell: but it's not there, nobody created it11:52
quiquellpanda: what  do you  mean ?11:53
pandaquiquell: https://review.rdoproject.org/r/gitweb?p=config.git;a=blob;f=playbooks/tripleo-ci-periodic-base/containers-build.yaml;h=9bd1149e572434327d872dfbb8b1d451340a43db;hb=refs/heads/master#l2111:54
pandathis is what fails11:54
quiquellpanda: I know, but I don't know if with a periodic pipeline that don't have zuul.change ot it the role 'set-zuul-log-path-fact' is going to work11:55
quiquellpanda: https://github.com/openstack-infra/zuul-jobs/tree/master/roles/set-zuul-log-path-fact11:56
quiquellpanda: is always relative path11:56
pandaquiquell: the role is working, the problem is that someone, somewhere, is not mkdir'ing that path11:56
quiquellpanda: It has to be directly in the workspace ?11:57
*** holser_ has joined #oooq11:58
pandaquiquell: that's what I'm not sure about11:58
*** agopi has joined #oooq11:58
pandain other places in rdo-jobs the LOG_PATH is prefixed by BASE_LOG_PATH11:58
pandabut I can'f find BAS_LOG_PATH defined anywhere11:58
quiquellpanda: LOG_PATH has always being relative here https://review.openstack.org/#/c/608080/2/playbooks/tripleo-ci/run-v3.yaml11:59
pandaanyway the only place aI see taht uses LOG_PATH is the tempest reporting11:59
quiquellpanda: let's look at the old code from containers-build11:59
*** agopi has quit IRC12:00
*** jtomasek has joined #oooq12:00
*** jtomasek has quit IRC12:01
quiquellpanda: Ok, found the issue12:01
*** jtomasek has joined #oooq12:02
pandaquiquell: missing mkdir somewhere ?12:02
quiquellpanda: They were just creating $WORKSPACE/logs12:02
quiquellpanda: And copy kolla there12:02
quiquellpanda: legacy-periodic-tripleo-centos-7-master-containers-build/post.yaml12:02
quiquellpanda: going to leave it like legacy12:04
pandaquiquell: still don't see how LOG_PATH is involved12:05
quiquellpanda: legacy https://review.rdoproject.org/r/gitweb?p=config.git;a=blob;f=playbooks/legacy-periodic-tripleo-centos-7-master-containers-build/post.yaml;h=71d5b73fea28dbf63b53bd8a5dcda40068853570;hb=HEAD12:07
*** ratailor has quit IRC12:08
quiquellpanda: where do we  run post ?12:10
pandaquiquell: in the base12:11
quiquellpanda: what machine it runs I mean12:11
quiquellpanda: LOG_PATH should have being alread create by the undercloud install12:11
quiquellpanda: at /home/zuul/12:11
pandaquiquell: you sure ? I see run-v3 just exports the variable12:11
quiquellpanda: Maybe collect-logs12:12
pandaquiquell: collect logs happens in post12:12
*** udesale has joined #oooq12:12
pandaquiquell: after the containers build12:12
quiquellpanda: Humm let's just create the bloody path12:12
quiquellpanda: https://review.rdoproject.org/r/#/c/16813/12:14
pandamkdir -p $PATH; stab --bleeding $PATH12:14
quiquellpanda: Good name for a github project12:15
quiquellpuff we have so much reviews12:15
quiquellLet's merge the mkdir so we can still use the test review12:15
quiquellpanda: give a +2 https://review.rdoproject.org/r/#/c/16813/ I will ask jpena to merge it12:16
pandaquiquell: still can't understand how we are using it ..12:16
quiquellPuff I thikn tempest is screw :-(12:17
pandaquiquell: jpena has the quicker merge on the west12:17
quiquellpanda: I think we are using it wrong12:18
quiquellpanda: legacy code has a expliti post that collect stuff12:18
quiquellpanda: we don't12:18
quiquellpanda: we are going to miss stuff at log server12:18
quiquellpanda: but looks like collect-logs is running after this so maybe not12:18
quiquellpanda: Let's just merg and test...12:19
quiquellok mergd12:19
quiquellpanda: Le's review upload job... I am not sure we have cover everything there12:24
*** rlandy has joined #oooq12:25
quiquellrlandy: o/12:26
rlandyquiquell; hi there!12:26
rlandyhow's it going?12:26
quiquellrlandy: have being able to run periodics at check with a hack, found broken stuff already12:27
quiquellrlandy: stuff pending of merge https://tree.taiga.io/project/tripleo-ci-board/task/106?kanban-status=144727512:27
*** amoralej is now known as amoralej|lunch12:27
*** agopi has joined #oooq12:27
quiquellrlandy: test review https://review.rdoproject.org/r/#/c/16806/12:27
*** dtantsur is now known as dtantsur|brb12:27
rlandyreading through12:28
quiquellrlandy, panda: we have to fix upload job too, it's broken12:28
pandaquiquell: log ?12:28
quiquellpanda: Not executed, just reading the code12:29
quiquellpanda: It does the legacy way12:29
rlandyso interesting all this stuff worked fine before12:29
quiquellrlandy: Humm really ?12:30
quiquellrlandy: Runing new workflow at periodic mode changes a lot of stuff12:32
rlandyguess we never touched these paths before12:32
quiquellrlandy: All this legacy stuff depends a lot on devstack12:32
quiquellrlandy: Now it's gone12:32
rlandyquiquell: we can merge review.rdoproject.org/r/16809  if you address the two comments12:32
rlandylooks good otherwise12:32
quiquellrlandy: ack, btw we have check that if post fail dependants run is not executed, looks like it waits for the all the phases12:34
rlandyquiquell: left comment https://review.rdoproject.org/r/#/c/16810/ - see what you think12:36
*** holser_ has quit IRC12:37
rlandythis is a nice hack ... https://review.rdoproject.org/r/#/c/16806/12:38
rlandyquiquell++12:38
hubbot1rlandy: quiquell's karma is now 812:38
rlandyquiquell: so where are we now?12:38
rlandywhat is there is do atm?12:39
quiquellrlandy: We have to fix upload job12:40
quiquellrlandy: And make it similar to the promote-... and containers-builds12:41
*** trown|outtypewww is now known as trown12:41
quiquellrlandy: taking git repo from zuul and using workspace etc...12:41
* rlandy looks12:41
quiquellrlandy: funny thing upload job is already using zuul_success so maybe build containers at post is not that bad12:41
rlandywell upload kind of makes sense at post12:41
rlandybut then again so does conatiner build , I guess :)12:42
rlandyhttps://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/legacy-tripleo-ci-periodic-base-upload/post.yaml?12:43
quiquellrlandy: yep12:43
rlandy^^ you're referring to ^^file with upload?12:43
quiquellrlandy: yep that file12:44
quiquellrlandy: also release job var is missing12:44
rlandyok  - I can work on that12:44
quiquellrlandy: https://review.rdoproject.org/r/16809 ready12:44
quiquellrlandy: ack with https://review.rdoproject.org/r/#/c/16810/, will abandon the change12:46
rlandyquiquell: https://review.rdoproject.org/r/#/c/16809/ looks good to me - can merge if panda doesn't have any other objections12:47
quiquellrlandy: Any upstream change we need is going to take us time, there are some tempest issues :-(12:49
pandarlandy: none, your honor12:49
rlandydone12:50
rlandyquiquell: well lucky for your hack12:50
rlandyI was going to do something a lot worse12:50
quiquellrlandy: If has give us a lot of power this morning12:50
quiquellrlandy: Humm but there is one issue there... zuul.changes12:51
quiquellrlandy: But since we are not changing any RPMs project I suppose we are not promoting wrong stuff12:51
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci- (2 more messages)12:51
quiquellrlandy: I mean we don't promote tq or tqe or triple-ci12:51
rlandycorrect - not following the question though12:52
rlandyhmmm... I can't strike out items on https://tree.taiga.io/project/tripleo-ci-board/task/106?kanban-status=144727512:53
quiquellrlandy: going to prepare the upload thingy direclty in the real deal without proto12:53
rlandybut review 3 and  4 can be crossed out12:53
quiquellrlandy: en edit mode select test, a window will pop-up12:53
pandarfolco: is internal sf installed in Centos or RHEL ?12:53
rlandyoh stupid me- not signed in on FF12:54
quiquellrlandy: Nah it's suprised me, that weird pop-up12:54
quiquellrlandy: just remove the tq/tqe master/rocky we have a lot of noise already12:54
rlandymy fault - I can edit where I am logged in12:55
rlandysorry - just remove from where12:55
rlandycheck?12:55
*** holser_ has joined #oooq12:56
rlandyoh from the test job12:57
rlandyunderstood12:57
rlandyquiquell: you're putting in the upload change or I am?12:58
quiquellrlandy: Doing it, like my last stuff before drop13:00
rlandysure ok13:00
quiquellrlandy: what featureset do we need for upload jobs ? do we run a job there ?13:00
rlandy2313:00
quiquellrlandy: this is the part at 'config' https://review.rdoproject.org/r/1681513:01
rlandywait fs 00213:02
rlandyupload is for sure fs00213:05
*** toure|gone is now known as toure13:05
rlandyquiquell: one comment13:07
quiquellrlandy: we also need this https://review.rdoproject.org/r/1681613:09
rlandyis release defined at that point?13:10
quiquellrlandy: needed for the post13:10
rlandyquiquell: why not zuul.branch?13:10
quiquellrlandy: we don't have zuul.branch at periodics we use override_branch13:10
rlandyalso we would need to add release to all upload jobs13:10
quiquellrlandy: and only for non master13:10
rlandyoh right13:10
rlandybranch override13:11
rlandyok13:11
quiquellzuul.branch is going to be master or nothing don't know at periodic13:11
rlandyok13:11
rlandymerging that13:11
rlandyhttps://review.rdoproject.org/r/#/c/16815 ok as well13:12
rlandysame as legacy workflow13:12
quiquellrlandy: answered https://review.rdoproject.org/r/#/c/1681513:12
rlandyok let's try this13:12
*** ykarel has quit IRC13:12
quiquellrlandy: I mean we can add complex logic to discover release between branch_override and zuul.branch I think it does not worth it13:13
rlandyquiquell: ok - can accept that13:13
quiquellrlandy: So as allways we miss the upstream stuff at gates :-(13:14
quiquellrlandy: but we can exercise that with the Depends-On13:14
rlandyquiquell: your jobs are merged13:14
rlandyyou can rekick13:14
quiquellrlandy: wait... want to check if post works now with the mkdir -p13:15
quiquellrlandy: and the next jobs in the pipeline kicks in13:15
quiquellrlandy: btw, why don't we add a tripleo-ci-testing -> tripleo-ci at the end of the pipeline ?13:15
quiquellrlandy: So we don't need the promoter at least for this13:15
quiquellrlandy: we can alwasy change the criteria in the pipeline over 'config'13:16
rlandypromoter is commented out13:16
quiquellrlandy: Yah I mean normal working stuff, like why do we need promoter for this is zuul can do that for us13:16
rlandythe promoter pre-dated zuul-v313:17
quiquellrlandy: Maybe we can rething this, so we simplify promoter13:17
quiquellrethink13:17
rlandyyeah - possible13:17
pandaWe have a long line of improvements for the promoter ...13:17
quiquellpanda: with zuulv3 we don't need it for tripleo-ci promotion, still needed for phase1 and phase213:18
pandaquiquell: we don't ?13:18
quiquellpanda: nope, in the pipeline we can create dependencies with the criteria to promote13:19
pandaquiquell: not that easy13:19
quiquellpanda: and same we do consistent-tripleo-ci-testing job we do tripleo-ci-testing-to-tripleo-ci also the docker stuff13:19
pandaquiquell: the promoter script goes back in report history to find a suitable candidate13:19
*** vinaykns has joined #oooq13:19
quiquellpanda: This is not needed if it's done in demand in the pipeline13:20
pandathere's a bit more logic than "oh, everything passed"13:20
quiquellpanda: we just check old hashses if current ones are no good13:20
quiquellpanda: if zuul promote itself the latest good one is already promoted, maybe the docker part is more problematic13:20
pandaquiquell: yes, and that is par tof the problem. If the current job with the dependency fails for an unrelated reason, then thet hash is lost, there's no way to promote it a second time13:21
pandatry to promote it13:22
quiquellpanda: next run will do13:22
quiquellpanda: If there is new hash is good too no ?13:22
rlandyoh dear - failed gates tripleo-ci13:22
pandanooooo13:22
quiquellrlandy: yep... I think they are broken :-( I have ask for pirority but still pufff13:23
quiquellpanda: Bah sure you are right, I am always too naif with promoter13:23
quiquellDaman how much time it takes "Build and push images" ?13:23
pandaquiquell: we've been naif a lot, and we went through the same questions13:23
pandaquiquell: you're just lacking some history :)13:24
quiquellYep... is only that pipeline feel so natural13:24
rlandyready to recheck? does want post work with mkdir -p?13:25
pandaI'm about to leave, people next week you'll be the only one to defend the fort13:25
quiquellpanda: You will read from us in the news, or in the stock options of redhat :-)13:26
quiquellpanda: I mean bad news13:26
rlandypanda: enjoy13:26
quiquellrlandy: Taking too much time, going to force recheck with a change13:26
rlandyquiquell: ack13:27
rlandyquiquell: any other changes needed so far?13:27
quiquellrlandy: nope... just the fu... upstream ones13:28
rlandyyep - no much we can do about those13:28
quiquellrlandy: ok re-kicked13:28
rlandythanks13:29
quiquellrlandy: maybe we have to deactivate periodic pipeline will we use the testing putch13:29
quiquellpatch13:29
quiquellAlso we can even put queens and pike ther ein the testing patch if the others are ok13:29
rlandyquiquell: wrt deactivate pipeline ...13:30
rlandyyou mean review.rdoproject.org/r/16798?13:30
*** panda is now known as panda|off13:31
rlandymaybe should comment both out13:31
quiquellrlandy: we don't want to kick the test review and perioidc pipeline kicking them too13:31
quiquellrlandy: Yep comment both13:31
rlandyk - updating patch for your review13:31
quiquellrlandy: cool, I think the big pieces are in place, it will need some hammering around13:32
quiquellrlandy: Didn't have time to look at check jobs13:33
quiquellOk I drop now13:33
rlandyquiquell: no worries - I will deal with check13:33
*** skramaja_ has quit IRC13:34
* quiquell gives the magic wand to rlandy and disappear13:34
*** quiquell is now known as quiquell|off13:34
mariosanyone else know what quiquell|off meant on tripleo please?13:37
marios16:27 < quiquell> marios: gates broken at upgrades13:37
*** ykarel has joined #oooq13:38
mariosis it about https://bugs.launchpad.net/tripleo/+bug/1792527/comments/6 or something else13:38
openstackLaunchpad bug 1792527 in tripleo "Undercloud upgrade rocky -> master broken on ironic" [Critical,Fix released] - Assigned to Jiří Stránský (jistr)13:38
marioskopecmartin|ruck: any idea?13:38
panda|offmarios: the gates failed on some patches we are trying to merge13:39
rlandymarios: he is referring to the top failing job in the tripleo gate13:39
panda|offmarios: I suppose upgrades jobs failed13:39
*** amoralej|lunch is now known as amoralej13:39
rlandyyeah13:39
rlandyhttp://logs.openstack.org/80/608080/2/gate/tripleo-ci-centos-7-containerized-undercloud-upgrades/52824fe/13:39
*** ykarel has quit IRC13:39
*** ykarel_ has joined #oooq13:39
rlandypanda|off: you should be off!!!13:40
mariospanda|off: rlandy ack . so the problem with that one is i also see lots of successes in http://cistatus.tripleo.org and supposedly the patches from jistr should have fixed it anyway, well we are still not clear if the right tht/new/updated/master tht is used there.13:40
rlandypanda|off: btw merging https://review.rdoproject.org/r/#/c/16798/ to disable pipelines13:41
mariosjistr was also looking in tripleo earlier13:41
rlandymarios: ok - will keep a watch on it13:41
panda|offrlandy: how do you launch and test periodic ? Quiquell's patch  in check ?13:43
mariosrlandy: oh there are lots more fails than success at this point13:43
rlandypanda|off: ack13:43
rlandyactual periodic will die w/o tripleo-ci changes13:43
panda|offrlandy: ok13:43
rlandyif we want to keep periodic running, which was my initial thought,13:44
rlandywe would need that fake duplicated run-v313:44
rlandywhich I gather you guys ditched13:44
rlandymarios: yeah - we are desperate to merge a couple of change in tripleo-ci13:45
*** dtantsur|brb is now known as dtantsur13:48
panda|offand more will come13:52
panda|offI'm off! I'm off!13:52
rlandywow - all alone13:55
rlandypanda|off: wait, you can't go!!!!13:55
rlandymarios: pls ping me before you go14:07
rlandybecause I will be the only one left :(14:07
rlandyoh  ... ssbarnea is still here14:08
rlandyyay - another team mate14:08
mariosrlandy: ack :) i'm still looking into that undercloud-upgrade thing at least enough to write something into a new bug, i don't think we want to continue using https://bugs.launchpad.net/tripleo/+bug/179252714:08
openstackLaunchpad bug 1792527 in tripleo "Undercloud upgrade rocky -> master broken on ironic" [Critical,Fix released] - Assigned to Jiří Stránský (jistr)14:08
marioskopecmartin|ruck: around?14:08
kopecmartin|ruckmarios, yes14:08
*** ykarel_ is now known as ykarel14:09
marioskopecmartin|ruck: ^^^ fyi that undercloud-upgrade thing we discussed this morning ^^14:09
marioskopecmartin|ruck: i added a comment on that bug earlier but i think we need a new one (will file in a bit)14:09
kopecmartin|ruckso the patch did not fix it?14:10
marioskopecmartin|ruck: right this is the question. why/how are the tasks being executed there still failing14:10
marioskopecmartin|ruck: is it not using new enough tht?14:10
marioswhy?14:10
mariosno one found the answer so far14:10
kopecmartin|ruckah14:11
marioskopecmartin|ruck: because especially for the master version of that change it merged laste night so it will easily be in builds by now14:11
mariosrlandy: is weshay_pto ah i see yes he is :)14:11
rlandymarios: yeah - he has some pto to burn before it expires I guess14:12
rlandymarios; just fyi - you know promotions have stopped14:13
rlandywhile we debug periodic14:13
mariosrlandy: ack quiquell|off said earlier and wanred14:14
marioswarned14:14
rlandyso this may be a chicken and egg situation ...14:15
rlandywe need promotions to get changes and we need reviews in to get promotions14:15
rlandymarios: ^^14:15
mariosrlandy: ah i see. so looking at the tht in one of the failed runs i don't see the task ("query if xinetd is active") in http://logs.openstack.org/30/607830/2/check/tripleo-ci-centos-7-undercloud-upgrades/3603578/logs/undercloud/home/zuul/undercloud-ansible-OzhrFz/Undercloud/upgrade_tasks.yaml.txt.gz14:16
marioskopecmartin|ruck: ^14:16
mariosrlandy: i am looking for the fix for the issue https://review.openstack.org/#/c/602550/ here at master14:16
mariosrlandy: yeah might be that. the version of tht installed/used as per ^^^ is not new enough anyway14:17
mariosrlandy: so what do?14:17
rlandymarios: I really don't know14:17
rlandyany depends-on possibilities14:18
rlandymerged sep 1814:19
rlandyshould be ok14:19
rlandywe only stopped promotion yesterday14:19
rlandyafter there was a promotion the week before14:20
rlandyif we need additional changes, that is a problem14:20
rlandyrocky would fail in this case14:20
*** ykarel is now known as ykarel|afk14:23
rlandymarios: anything I can do to help?14:26
rlandyhmmm ...14:28
rlandylegacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master is passing now14:28
rlandybut tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master still fails deploy14:29
mariosrlandy: lemme write up the bug14:30
mariosrlandy: and then see where we're at not clear yet rlandy i mean what you can do14:30
rlandynew check jobs actually seem ok14:34
mariosrlandy: so this is the thing (didn't check now, you mean via cistatus.tripleo? ) earlier today, it looked like loong list of success and a few of these14:34
mariosrlandy: so we thought ok lets wait for rocky merge and it will be ok (backport of that fix i mean)14:35
mariosrlandy: but seems not14:35
rlandymarios:so maybe recheck is just the right thing to do?14:35
rlandythe gate is master though14:35
*** panda|off has quit IRC14:36
*** panda has joined #oooq14:37
*** udesale has quit IRC14:38
rlandyyep - seems like a fairly consistent failure14:48
rlandynow at least14:48
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci- (2 more messages)14:52
mariosrlandy: am unable to file the bug, i keep getting https://up1-vrutkovs.int.open.paas.redhat.com/#-4i_bxo74z9IUvFvnrVulw14:52
mariosrlandy: here is the text fwiw ill try again in a bit http://paste.openstack.org/raw/731592/14:53
marioskopecmartin|ruck: do you want to try copy/pasta to file that? ^^14:53
mariosor rlandy if you're up for it?14:54
mariosi even logged out/in so maybe is a temp lp issue14:54
rlandylooking14:55
rlandymarios: I'll try14:55
rlandywhat's the bug summary?14:56
mariosrlandy: thanks it the paste 17:53 < marios> rlandy: here is the text fwiw ill try again in a bit http://paste.openstack.org/raw/731592/14:56
rlandyyeah text has no summary - will make one14:57
mariosThe tripleo-ci-centos-7-undercloud-upgrades and containerized-undercloud-upgrades fails during undercloud upgrade in restart xinetd task14:57
mariossummary ^14:57
mariosrlandy: ^14:57
rlandymarios: https://bugs.launchpad.net/tripleo/+bug/179634114:59
openstackLaunchpad bug 1796341 in tripleo "The tripleo-ci-centos-7-undercloud-upgrades and containerized-undercloud-upgrades fails during undercloud upgrade in restart xinetd task" [Undecided,New]14:59
rlandycan you access that?14:59
mariosrlandy: thanks yeah14:59
rlandyotherwise I will add tags etc.14:59
rlandyk14:59
marioskopecmartin|ruck: fyi rlandy filed 17:59 < rlandy> marios: https://bugs.launchpad.net/tripleo/+bug/179634115:01
openstackLaunchpad bug 1796341 in tripleo "The tripleo-ci-centos-7-undercloud-upgrades and containerized-undercloud-upgrades fails during undercloud upgrade in restart xinetd task" [Undecided,New]15:01
mariosincase you see the earlier ping thanks :)15:01
*** jaganathan has quit IRC15:06
*** jfrancoa has quit IRC15:07
*** ykarel|afk is now known as ykarel|ptotillwe15:33
*** ykarel|ptotillwe is now known as ykarel|ptotwedne15:34
rascasoftrlandy, hey you around? I'm still playing around yesterday's rdocloud failure16:03
rlandyrascasoft: yeah - I am - sorry been heads down on the zuul-v3 transition16:03
rascasoftrlandy, no worries at all16:04
rascasoftrlandy, if you have time we can check it otherwise it can wait16:04
rlandyrascasoft: should have some time in a little bit - while the job runs16:04
rlandyjust need to put in a new review16:05
rlandywill ping you in about 20 mins16:05
marios|roverrlandy: /me off now16:07
marios|roverlooks like emilien will sort it today16:07
rlandymarios|rover: thanks16:07
rlandyyeah - watching that16:07
rlandyhave a nice weekend16:07
marios|roveru2 o/16:07
*** gkadam has quit IRC16:21
*** jtomasek has quit IRC16:21
*** bogdando has quit IRC16:26
*** trown is now known as trown|lunch16:30
rlandyrascasoft: hello16:43
rlandyready to work on your issue16:43
rascasoftrlandy, hey16:43
rlandyrascasoft: can you link the latest failure?16:44
rascasoftrlandy, I can make you login into the env and show you16:44
rlandysure16:45
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-3nodes-multinode, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci- (2 more messages)16:52
*** jtomasek has joined #oooq16:52
*** amoralej is now known as amoralej|off17:05
*** dtantsur is now known as dtantsur|afk17:18
*** vinaykns has quit IRC17:27
*** trown|lunch is now known as trown17:32
*** ykarel|ptotwedne has quit IRC17:33
*** holser_ has quit IRC17:42
*** dsneddon has quit IRC17:59
*** vinaykns has joined #oooq18:13
rlandyrfolco: need your vote one more time pls ... I forget the required-repo ...https://review.rdoproject.org/r/#/c/16821/18:51
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master, tripleo-ci-centos-7-undercloud-upgrades, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci- (2 more messages)18:52
rlandyjust so I am not merging code with no oversight18:52
rfolcorlandy, whats in ci-config that is required for that job ?18:52
rlandyhttps://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/tripleo-ci-periodic-base/containers-build.yaml#L1518:52
rlandyI didn't add the dlrn-secret18:53
rlandythat may come to bite me18:53
rlandyrfolco: ^^18:54
rfolcooh I see18:54
rlandythank you18:56
rlandywe may have to do this one more time :(18:56
rfolconp18:56
*** zul has quit IRC19:02
*** zul has joined #oooq19:03
rlandyrascasoft: any luck?19:49
*** dsneddon has joined #oooq19:57
*** agopi is now known as agopi|away19:59
rlandywoohoo - some progress20:32
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master, tripleo-ci-centos-7-undercloud-upgrades, tripleo-ci-centos-7-containerized-undercloud-upgrades, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades,  (2 more messages)20:52
*** vinaykns has quit IRC20:54
*** kopecmartin|ruck is now known as kopecmartin|off21:00
*** toure is now known as toure|gone21:15
*** holser_ has joined #oooq21:17
*** holser_ has quit IRC21:19
*** trown is now known as trown|outtypewww21:21
*** rlandy has quit IRC22:24
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-containerized-undercloud-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades,  (2 more messages)22:52
*** rfolco has quit IRC23:23
« Thursday, 2018-10-04 Index Saturday, 2018-10-06 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!