| *** tosky has quit IRC | 00:00 | |
| *** dmellado has quit IRC | 00:03 | |
| *** dmellado has joined #oooq | 00:04 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, (5 more messages) | 00:13 |
|---|---|---|
| *** sshnaidm is now known as sshnaidm|afk | 00:28 | |
| *** chem has quit IRC | 01:25 | |
| *** chem has joined #oooq | 01:25 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, (5 more messages) | 02:14 |
| *** dsneddon has quit IRC | 02:21 | |
| *** chem has quit IRC | 02:25 | |
| *** chem has joined #oooq | 02:26 | |
| *** dsneddon has joined #oooq | 02:46 | |
| *** ykarel has joined #oooq | 02:54 | |
| *** dsneddon has quit IRC | 02:57 | |
| *** rlandy|bbl is now known as rlandy | 03:06 | |
| *** dsneddon has joined #oooq | 03:07 | |
| rlandy | weshay|rover: anything need to be reverted/merged? | 03:09 |
| weshay|rover | rlandy I'm putting in a change to update the release file | 03:09 |
| rlandy | weshay|rover: k | 03:10 |
| weshay|rover | rlandy it's pretty out of date | 03:10 |
| weshay|rover | https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/Fedora-28/promotion-testing-hash-master.yml | 03:10 |
| weshay|rover | https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-fedora-28-standalone-master/5e38cbb/logs/undercloud/home/zuul/repo_setup.log.txt.gz#_2019-03-05_23_19_22 | 03:10 |
| weshay|rover | how was the show? | 03:10 |
| rlandy | weshay|rover: it was actually awesome | 03:10 |
| rlandy | I didn't expect much but it was very well done | 03:10 |
| *** dsneddon has quit IRC | 03:13 | |
| *** saneax has joined #oooq | 03:14 | |
| weshay|rover | nice | 03:14 |
| weshay|rover | https://review.openstack.org/641187 | 03:14 |
| weshay|rover | there is some error in the all the post jobs | 03:14 |
| weshay|rover | but I think it was maybe infra | 03:15 |
| weshay|rover | not sure | 03:15 |
| weshay|rover | https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-fedora-28-standalone-master/5e38cbb/job-output.txt.gz#_2019-03-05_23_19_25_808521 | 03:15 |
| * rlandy looks | 03:15 | |
| weshay|rover | this is the task prior to the fail http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/set-zuul-log-path-fact/tasks/main.yaml#n11 | 03:16 |
| rlandy | weshay|rover: both fedora and fedora28 resolve - weird | 03:17 |
| rlandy | https://trunk.rdoproject.org/fedora28/stable-base/latest/ | 03:17 |
| * ykarel was also looking at it | 03:17 | |
| rlandy | https://trunk.rdoproject.org/fedora/stable-base/latest/ | 03:17 |
| ykarel | rlandy, weshay|rover https://review.rdoproject.org/r/#/c/19083/4/playbooks/tripleo-ci-periodic-base/post.yaml@27 causing it | 03:17 |
| ykarel | actually /me was looking at centos ones | 03:18 |
| weshay|rover | ykarel my lanta.. you never sleep | 03:18 |
| weshay|rover | a MACHINE | 03:18 |
| weshay|rover | ykarel I think you are right | 03:18 |
| ykarel | woke early today as has to go out in some time | 03:19 |
| weshay|rover | is it the missing quotes? | 03:20 |
| rlandy | that chnage was after we switch to the role | 03:20 |
| ykarel | weshay|rover, indentation | 03:21 |
| weshay|rover | hrm.. being able to test config changes is becoming critical imho | 03:21 |
| weshay|rover | hrm.. http://yaml-online-parser.appspot.com/?yaml=-+hosts%3A+primary%3Atripleo-ovb-centos-7%0A++vars%3A%0A++++workspace%3A+%22%7B%7B+ansible_user_dir+%7D%7D%2Fworkspace%22%0A++++ci_config_repo%3A+%22%7B%7B+ansible_user_dir+%7D%7D%2F%7B%7B+zuul.projects%5B%27review.rdoproject.org%2Fconfig%27%5D.src_dir+%7D%7D%22%0A++tasks%3A%0A++++-+name%3A+Set+zuul | 03:23 |
| weshay|rover | -log-path+fact%0A++++++include_role%3A%0A++++++++name%3A+set-zuul-log-path-fact%0A++++-+shell%3A%0A++++++++cmd%3A+%7C%0A++++++++++source+%7B%7B+workspace+%7D%7D%2Fhash_info.sh%0A++++++++++%7B%25+if+nodes+is+defined+%25%7D%0A++++++++++export+TOCI_JOBTYPE%3D%22periodic-%7B%7B+environment_type+%7D%7D-%7B%7B+nodes+%7D%7D-featureset%7B%7B+featureset+%7D | 03:23 |
| weshay|rover | %7D%22%0A++++++++++%7B%25+else+%25%7D%0A++++++++++export+TOCI_JOBTYPE%3D%22periodic-%7B%7B+environment_type+%7D%7D-featureset%7B%7B+featureset+%7D%7D%22%0A++++++++++%7B%25+endif+%25%7D%0A++++++++++export+LOG_PATH%3D%22%7B%7B+zuul_log_path+%7D%7D%22%0A++++++++++export+SUCCESS%3D%22%7B%7B+zuul_success+%7C+bool+%7D%7D%22%0A++++++++++bash+-xe+%7B%7B+ci | 03:23 |
| weshay|rover | _config_repo+%7D%7D%2Fci-scripts%2Ftripleo-upstream%2Fdlrnapi_report.sh%0A++++++++++%23+Pass+also+the+new+naming+scheme+as+JOBTYPE+to+report+to+DLRN%0A++++++++++%23+In+this+way+each+job+will+report+success%2Ffailure+with+the+new+name%0A++++++++++%23+and+the+old+name%2C+and+we+can+use+one+or+the+other+in+the+promotion%0A++++++++++%23+criteria%0A++++ | 03:23 |
| weshay|rover | ++++++export+TOCI_JOBTYPE%3D%22%7B%7B+zuul.job+%7D%7D%22%0A++++++++++bash+-xe+%7B%7B+ci_config_repo+%7D%7D%2Fci-scripts%2Ftripleo-upstream%2Fdlrnapi_report.sh%0A++++++++chdir%3A+%27%7B%7B+workspace+%7D%7D%27%0A++++++++environment%3A+%7C%0A++++++++++%7B%7B+zuul+%7C+zuul_legacy_vars+%7C+combine(%7B%0A++++++++++++%27DLRNAPI_PASSWORD%27%3A+dlrnapi.pass | 03:23 |
| hubbot1 | weshay|rover: Error: "7D%22%0A++++++++++%7B%25+else+%25%7D%0A++++++++++export+TOCI_JOBTYPE%3D%22periodic-%7B%7B+environment_type+%7D%7D-featureset%7B%7B+featureset+%7D%7D%22%0A++++++++++%7B%25+endif+%25%7D%0A++++++++++export+LOG_PATH%3D%22%7B%7B+zuul_log_path+%7D%7D%22%0A++++++++++export+SUCCESS%3D%22%7B%7B+zuul_success+%7C+bool+%7D%7D%22%0A++++++++++bash+-xe+%7B%7B+ci" is not a valid command. | 03:23 |
| weshay|rover | word%2C%0A++++++++++++%27DLRNAPI_USERNAME%27%3A+dlrnapi_user%0A++++++++++++%7D)+%7D%7D&type=json | 03:23 |
| weshay|rover | oh man | 03:23 |
| rlandy | wow | 03:25 |
| weshay|rover | sorry | 03:25 |
| *** dsneddon has joined #oooq | 03:25 | |
| *** chem has quit IRC | 03:26 | |
| weshay|rover | rlandy we're looking at https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic | 03:26 |
| *** chem has joined #oooq | 03:26 | |
| weshay|rover | rlandy ykarel should we try all on one line w/ quotes until we can test it more effectively? | 03:28 |
| rlandy | weshay|rover: that would be my first bet | 03:28 |
| rlandy | make it pretty afterwards | 03:28 |
| rlandy | just make sure all three options work | 03:28 |
| ykarel | fixing indentation should be enough i think | 03:29 |
| ykarel | can run a standalone in test project to test | 03:29 |
| weshay|rover | ah tru | 03:29 |
| rlandy | weshay|rover: wrt https://review.openstack.org/#/c/641187/ I can +2 because both options work but fedora28 does resolve for me for both cases | 03:29 |
| ykarel | next periodic run is in 1 hr 40 minutes | 03:30 |
| weshay|rover | rlandy hrm.. maybe it was just the mirror? | 03:31 |
| weshay|rover | cache miss | 03:31 |
| weshay|rover | ? | 03:31 |
| rlandy | weshay|rover: yep - but the error was clear | 03:31 |
| weshay|rover | ykarel ok.. I give in.. the open braces should be under the colon? | 03:32 |
| weshay|rover | not sure why that mirror is just getting hit now | 03:32 |
| *** dsneddon has quit IRC | 03:32 | |
| ykarel | weshay|rover, no, weshay|rover if you see the patch it moved environment 2 space ahead | 03:33 |
| ykarel | which made it wrong ansible arg for shell module | 03:33 |
| weshay|rover | oh dang | 03:34 |
| weshay|rover | I see that now | 03:34 |
| weshay|rover | thanks for the eyes | 03:34 |
| ykarel | ya, see L 50 for comparison | 03:34 |
| weshay|rover | rlandy hang on for review | 03:34 |
| rlandy | hanging | 03:34 |
| rlandy | curl --silent http://mirror.regionone.rdo-cloud-tripleo.rdoproject.org:8080/rdo/fedora28/71/d9/71d9a66abe93049a91d433f805c045abe135303a_e6dde0f8/delorean.repo -S | 03:35 |
| rlandy | that's a mirror not NODEPOOL_RDO_PROXY | 03:35 |
| weshay|rover | rlandy https://review.rdoproject.org/r/19121 ykarel | 03:38 |
| ykarel | ack | 03:38 |
| rlandy | weshay|rover: do you want to run that with testproject before merge? | 03:39 |
| rlandy | or just merge because it can't be worse | 03:39 |
| ykarel | nope we need to get that merged, it's config | 03:39 |
| weshay|rover | rlandy naw.. the error is now obvious now that ykarel has spelled it out for me like abc | 03:40 |
| rlandy | k - waiting for zuul to vote | 03:40 |
| rlandy | wrt the mirror error ... | 03:40 |
| rlandy | if it's something like this: https://github.com/rdo-infra/rdo-jobs/blob/master/playbooks/run-distgit.yaml#L146 - it's hardcoded | 03:41 |
| weshay|rover | rlandy well.. that is just alfeado's jobs | 03:42 |
| rlandy | weshay|rover: indentation fix is workflowed | 03:42 |
| rlandy | yep - I know | 03:42 |
| weshay|rover | and that job is crazy | 03:42 |
| rlandy | just looking at where it picks up mirror | 03:42 |
| weshay|rover | rlandy you def.. fixed the pipeline though | 03:42 |
| weshay|rover | so | 03:42 |
| weshay|rover | rlandy++ | 03:43 |
| hubbot1 | weshay|rover: rlandy's karma is now 47 | 03:43 |
| weshay|rover | and | 03:43 |
| weshay|rover | ykarel++ | 03:43 |
| hubbot1 | weshay|rover: ykarel's karma is now 11 | 03:43 |
| rlandy | weshay|rover: indentation strikes again | 03:43 |
| rlandy | change is merged | 03:44 |
| *** dsneddon has joined #oooq | 03:44 | |
| *** skramaja has joined #oooq | 03:50 | |
| *** dsneddon has quit IRC | 03:53 | |
| *** dsneddon has joined #oooq | 04:00 | |
| *** dsneddon has quit IRC | 04:05 | |
| weshay|rover | ykarel btw.. containers-build-push did not ever hit the post_failure | 04:10 |
| weshay|rover | it was the other jobs | 04:10 |
| ykarel | weshay|rover, yes because that job don't post to dlrn | 04:11 |
| ykarel | report to dlrn | 04:11 |
| weshay|rover | ah kk | 04:12 |
| * rlandy out - will check in tomorrow morning | 04:12 | |
| *** rlandy has quit IRC | 04:12 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010 (5 more messages) | 04:14 |
| *** chem has quit IRC | 04:27 | |
| *** ratailor has joined #oooq | 04:27 | |
| *** chem has joined #oooq | 04:27 | |
| *** dsneddon has joined #oooq | 04:35 | |
| *** dsneddon has quit IRC | 04:40 | |
| ykarel | weshay|rover, so standalone at test project at another post failure: https://logs.rdoproject.org/17/19017/4/check/periodic-tripleo-ci-centos-7-scenario003-standalone-master/a0d3b88/job-output.txt.gz#_2019-03-06_04_26_05_752352 | 04:43 |
| ykarel | somehow workspace is wrong | 04:45 |
| ykarel | so good to try all in one line | 04:47 |
| *** dsneddon has joined #oooq | 04:54 | |
| *** raukadah is now known as chandankumar | 04:56 | |
| *** dsneddon has quit IRC | 04:59 | |
| *** dsneddon has joined #oooq | 05:27 | |
| *** dsneddon has quit IRC | 05:32 | |
| *** dsneddon has joined #oooq | 06:04 | |
| *** udesale has joined #oooq | 06:07 | |
| *** dsneddon has quit IRC | 06:09 | |
| *** gkadam has quit IRC | 06:13 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010 (5 more messages) | 06:14 |
| *** chem has quit IRC | 06:28 | |
| *** chem has joined #oooq | 06:29 | |
| *** jfrancoa has joined #oooq | 06:52 | |
| *** quiquell|off is now known as quiquell | 06:53 | |
| quiquell | ykarel: o/ | 06:54 |
| ykarel | o | 06:54 |
| ykarel | \o/ | 06:54 |
| ykarel | quiquell, you saw wes's mail | 06:57 |
| quiquell | ykarel++ | 07:03 |
| hubbot1 | quiquell: ykarel's karma is now 12 | 07:03 |
| quiquell | Eagle eye | 07:03 |
| ykarel | quiquell, post that i pushed one more patch suspecting your patch, but still we have issue | 07:04 |
| ykarel | in post | 07:04 |
| quiquell | I am the owner of that bug | 07:04 |
| *** dsneddon has joined #oooq | 07:04 | |
| quiquell | What's the issue now | 07:04 |
| ykarel | quiquell, https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-master/7e5e2de/job-output.txt.gz#_2019-03-06_06_28_02_924510 | 07:05 |
| ykarel | somehow WORKSPACE is not set, | 07:06 |
| *** dsneddon has quit IRC | 07:08 | |
| quiquell | Yka | 07:09 |
| marios | o/ folks | 07:09 |
| quiquell | ykarel: is it there if job passes ? | 07:09 |
| quiquell | marios: o/ | 07:10 |
| ykarel | quiquell, yes it's there | 07:10 |
| quiquell | They have pass configuration but were broke a dlrn report ( I broke stuff with my stupid BM things) | 07:10 |
| quiquell | ykarel: found and fix | 07:11 |
| quiquell | Now workspace is missing at dlrn reporting if job fails | 07:11 |
| ykarel | if job fails? | 07:13 |
| ykarel | job is passing, but dlrn report is failing at post | 07:13 |
| ykarel | when you asked ykarel: is it there if job passes ?, i answered yes for the runs where dlrn report was success | 07:13 |
| ykarel | quiquell, ^^ | 07:13 |
| quiquell | ykarel: I see undercloud install fail there | 07:14 |
| ykarel | quiquell, ok let me grab other passing job | 07:14 |
| quiquell | ykarel: ack missunderstood | 07:14 |
| ykarel | https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario003-standalone-master/59f0200/job-output.txt.gz | 07:15 |
| *** dsneddon has joined #oooq | 07:20 | |
| quiquell | ykarel: I see also the Add DLRNAPI_USERNAME in quotes | 07:21 |
| quiquell | ykarel: that's weird, the DLRN reporting role do the same and it's working | 07:21 |
| ykarel | quiquell, yes i pushed but it didn't fixed the issue | 07:21 |
| ykarel | quiquell, post that i pushed one more patch suspecting your patch, but still we have issue | 07:22 |
| quiquell | ykarel: btw I think we need to put some ansible-lint stuff at config project | 07:22 |
| ykarel | yup would be better | 07:22 |
| *** dsneddon has quit IRC | 07:25 | |
| *** chem has quit IRC | 07:29 | |
| *** chem has joined #oooq | 07:32 | |
| *** rascasoft has quit IRC | 07:36 | |
| *** rascasoft has joined #oooq | 07:37 | |
| *** ccamacho has joined #oooq | 07:41 | |
| *** dsneddon has joined #oooq | 07:44 | |
| quiquell | workspace is set by zuul at zuul_legacy_vars https://github.com/openstack-infra/zuul/blob/master/zuul/ansible/filter/zuul_filters.py#L37 | 07:48 |
| *** dsneddon has quit IRC | 07:48 | |
| ykarel | hmm but somehow it's not there currently | 07:49 |
| quiquell | weird thing is the role version of this is working fine at internal sf | 07:49 |
| quiquell | going to try again | 07:49 |
| ykarel | okk | 07:49 |
| quiquell | have to leave fo ra fiew thoug | 07:49 |
| quiquell | back in 30 minutos or so | 07:50 |
| *** quiquell is now known as quiquell|brb | 07:50 | |
| *** dsneddon has joined #oooq | 07:50 | |
| ykarel | ack | 07:51 |
| *** dtantsur|afk is now known as dtantsur | 08:03 | |
| *** dtantsur is now known as dtantsur|mtg | 08:03 | |
| *** kopecmartin|off is now known as kopecmartin | 08:05 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010 (5 more messages) | 08:14 |
| *** dsneddon has quit IRC | 08:14 | |
| *** gkadam has joined #oooq | 08:14 | |
| *** amoralej|off is now known as amoralej | 08:19 | |
| *** dsneddon has joined #oooq | 08:22 | |
| *** dsneddon has quit IRC | 08:26 | |
| *** quiquell|brb is now known as quiquell | 08:28 | |
| *** tosky has joined #oooq | 08:29 | |
| chandankumar | marios: quiquell sshnaidm|afk https://review.openstack.org/#/c/640089/ please have a look, thanks! when free! | 08:48 |
| chandankumar | tosky: https://review.openstack.org/#/c/640992/ please have a look, when free! | 08:49 |
| quiquell | ykarel: I see they harcode WORKSPACe at pre.yaml | 08:53 |
| quiquell | ykarel: Will just do that to unblock stuff | 08:53 |
| *** jpena|off is now known as jpena | 08:55 | |
| *** bogdando has joined #oooq | 08:58 | |
| *** dsneddon has joined #oooq | 08:59 | |
| ykarel | quiquell, hmm, also good to find what causing the issue as zuul_legacy_vars have WORKSPACE defined, it's weird | 09:02 |
| quiquell | ykarel: to unblock https://review.rdoproject.org/r/19123 | 09:03 |
| quiquell | ykarel: while we debug why is not passed | 09:03 |
| quiquell | ykarel: Have run manually the zuul_legacy_vars filters from source code and it's working fine | 09:03 |
| quiquell | ykarel: I see WORKSPACE | 09:03 |
| quiquell | ykarel: want to test it a sf zuul though | 09:04 |
| ykarel | quiquell, ack | 09:04 |
| ykarel | quiquell, merging | 09:05 |
| quiquell | ykarel: small test here https://review.rdoproject.org/r/19124 | 09:09 |
| quiquell | ykarel: do we re run periodics ? | 09:10 |
| ykarel | quiquell, running a standalone job, if that pass, we can reschedule periodic | 09:11 |
| quiquell | ack | 09:11 |
| ykarel | quiquell, testing in https://review.rdoproject.org/r/#/c/19017/ | 09:13 |
| quiquell | ykarel: I have test ansible-lint over broken indent and it detects it :-( | 09:19 |
| quiquell | ykarel: WORKSPACE works fine http://logs.rdoproject.org/24/19124/2/check/test-zuul-legacy-vars/7778afe/job-output.txt.gz | 09:21 |
| ykarel | quiquell, cool for catching, and yes it's weird WORKSPACE is detected, but job is failing | 09:21 |
| ykarel | cool for catching indentation with ansible-lint | 09:22 |
| quiquell | ykarel: it's also weird that is forced at pre.yaml | 09:22 |
| ykarel | hmm | 09:23 |
| quiquell | maybe there are some problems with the secrets | 09:24 |
| quiquell | and instead of ansible failure you get empty environment | 09:24 |
| quiquell | puff don't know | 09:24 |
| *** sshnaidm|afk has quit IRC | 09:26 | |
| quiquell | ykarel: damn I know what it is | 09:28 |
| ykarel | what is it? | 09:28 |
| *** chem has quit IRC | 09:30 | |
| quiquell | ykarel: https://review.rdoproject.org/r/19125 | 09:31 |
| quiquell | ykarel: ansible does not show the undefine variable issue | 09:32 |
| ykarel | quiquell, because of ^^, WORKSPACE is unset? | 09:33 |
| quiquell | ykarel: locally I get | 09:33 |
| quiquell | TASK [shell] ********************************************************************************************************* | 09:33 |
| quiquell | [WARNING]: could not parse environment value, skipping: [u"{{ zuul | zuul_legacy_vars | combine({'DLRNAPI_PASSWORD': | 09:33 |
| quiquell | dlrnapi.password,'DLRNAPI_USERNAME': dlrnapi_user}) }}"] | 09:33 |
| quiquell | ykarel: environment section is not executed | 09:33 |
| quiquell | Don't know why this warning does not appear at zuul | 09:33 |
| quiquell | I have to be more careful with this changes :-(((( | 09:34 |
| quiquell | quiquell-- | 09:34 |
| hubbot1 | quiquell: Error: You're not allowed to adjust your own karma. | 09:34 |
| ykarel | quiquell, hmm, /me can reproduce that too | 09:35 |
| quiquell | ykarel: runing at zuul to see if it appears | 09:35 |
| ykarel | quiquell, yup good to find why it's not complaining at zuul | 09:35 |
| quiquell | ykarel: yep it's important | 09:35 |
| quiquell | ykarel: also adding ansible-lint to config | 09:36 |
| ykarel | ack | 09:36 |
| quiquell | ykarel: problem with this is that config is ver delicate we cannot start linting all the stuff :-/ | 09:36 |
| *** ykarel is now known as ykarel|lunch | 09:36 | |
| *** derekh has joined #oooq | 09:37 | |
| quiquell | zbr: ping | 09:38 |
| quiquell | zbr: did you know how hard would be to add ansible-lint to config ? | 09:38 |
| quiquell | zbr: maybe you have already a patch around | 09:39 |
| quiquell | panda|ruck|off++ | 09:39 |
| hubbot1 | quiquell: panda|ruck|off's karma is now 1 | 09:39 |
| quiquell | panda|ruck++ | 09:39 |
| marios | man they really went for it last night with the build jobs i mean looks like they merged the world /me lost | 09:46 |
| marios | https://review.rdoproject.org/r/#/c/19066/ merged too so in theory now we are using the new container build everywhere in periodics... http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-fedora-28-master-containers-build-push/6ab9224/ http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build-push | 09:49 |
| marios | quiquell: what is the problem you and ykarel|lunch are discussing - related to that or some other job? | 09:49 |
| marios | quiquell: those are green from last night. still i would have liked to test before the switch (https://review.rdoproject.org/r/#/c/19066/ ) | 09:50 |
| quiquell | marios: well looks like I miss to add the dlrnapi_user to the post playbook | 09:55 |
| quiquell | marios: locally ansible complains but not at zuul :-(((( | 09:55 |
| marios | quiquell: which patch/job i mean where is this failing | 09:55 |
| quiquell | marios: also ansible-lint would have discover the issue with environment | 09:56 |
| quiquell | marios: at DLRN reporting | 09:56 |
| marios | quiquell: (yah saw some of the linting discusion flyin by but didn't catch what was failing exactly. | 09:56 |
| marios | quiquell: ah so it affects everything? | 09:56 |
| quiquell | marios: https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-master/7e5e2de/job-output.txt.gz#_2019-03-06_06_28_02_924510 | 09:56 |
| quiquell | marios: yep | 09:56 |
| marios | no but those ran this morning (I mean e.g. http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-fedora-28-master-containers-build-push/6ab9224/ | 09:56 |
| marios | 2019-03-06 06:12 | 09:56 |
| quiquell | marios: where do you see it green ? | 09:57 |
| quiquell | marios: we have put a workspace bypass but still dlrnapi_user is not there so dlrnreport will fail | 09:57 |
| marios | quiquell: from here it declared success https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic | 09:58 |
| marios | quiquell: re hash info i wonder if that is related. is one of the things that was merged last night https://review.rdoproject.org/r/#/c/19108/2/ci-scripts/tripleo-upstream/get-hash.sh | 09:59 |
| marios | quiquell: its the file which generates hash_info | 09:59 |
| marios | quiquell: but i don't see something yet there... | 09:59 |
| quiquell | marios: well it will be a post failure | 09:59 |
| quiquell | marios: I see a lot of POST_FAILURES | 10:00 |
| marios | quiquell: k | 10:00 |
| *** holser_ has joined #oooq | 10:01 | |
| ykarel|lunch | quiquell, the testproject job moved forward but failed reporting to DLRN | 10:02 |
| ykarel|lunch | so possibly your patch to fix DLRNAPI_USER will fix that too | 10:02 |
| ykarel|lunch | periodic is runnning currently, so let's merge that and cross fingers | 10:03 |
| quiquell | ykarel|lunch: yep WORKSPACE will not work we need the DLRNAPI_USERNAME there :-/ | 10:03 |
| quiquell | ykarel|lunch: puting in place ansible-lint at config project | 10:04 |
| ykarel|lunch | quiquell, +W | 10:04 |
| quiquell | ack let's see now :-((( | 10:05 |
| ykarel|lunch | quiquell, and for linting, just a heads up ansible-linting at rdo-jobs broke some jobs, so it's reverted | 10:05 |
| quiquell | yep... is dificult | 10:05 |
| quiquell | config is even worse | 10:05 |
| ykarel|lunch | so care should be taken when doing it in config | 10:05 |
| ykarel|lunch | yup | 10:05 |
| quiquell | I am going to put the review and try to reduce scope | 10:05 |
| quiquell | so we go little by little | 10:06 |
| * ykarel|lunch going for lunch now | 10:07 | |
| *** holser_ has quit IRC | 10:10 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, (5 more messages) | 10:14 |
| *** holser_ has joined #oooq | 10:15 | |
| quiquell | ykarel|lunch, marios: The linters https://review.rdoproject.org/r/#/c/19129/ | 10:23 |
| quiquell | I am not sure it can be dangeroous but this detect stuff like the identation problem at environment | 10:23 |
| quiquell | marios: still the configuration issue does not make sense :-/ | 10:25 |
| *** skramaja has quit IRC | 10:27 | |
| *** ykarel|lunch has quit IRC | 10:27 | |
| *** ykarel|lunch has joined #oooq | 10:28 | |
| marios | quiquell: ack will check the patch in a bit | 10:31 |
| marios | quiquell: so we don't know why post is failing? | 10:31 |
| quiquell | marios: yes we do | 10:31 |
| quiquell | marios: ansible var dlrnapi_user is missing | 10:31 |
| quiquell | marios: in the playbook | 10:32 |
| quiquell | marios: but that's not very clear from zuul logs | 10:32 |
| marios | quiquell: ah ok (from something that merged yesterday? ) | 10:32 |
| quiquell | marios: my stupid stuff about parameterize user for DLRN report role :-/ | 10:32 |
| quiquell | marios: fix a send to panda|ruck|off was not complete | 10:32 |
| quiquell | marios: was missing the dlrnapi_user | 10:32 |
| marios | quiquell: ah ok cool | 10:33 |
| quiquell | marios: ykarel|lunch is testing it at standalone will relaunch periodics if it works | 10:33 |
| quiquell | marios: But this is unrelated to config error at zuul | 10:33 |
| quiquell | marios: Still I don't know why it was not running at all not even enqueue | 10:33 |
| *** ykarel|lunch is now known as ykarel | 10:45 | |
| *** sshnaidm|afk has joined #oooq | 10:54 | |
| ykarel | marios, so seems now new error is related to new container build-push job | 10:56 |
| ykarel | failure log:- https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-master/f27242d/logs/undercloud/home/zuul/undercloud_install.log.txt.gz | 10:56 |
| *** udesale has quit IRC | 10:57 | |
| *** sshnaidm|afk has quit IRC | 10:57 | |
| marios | ykarel: looking | 11:00 |
| quiquell | ykarel: dlrn reporting looks like working now | 11:00 |
| marios | ykarel: i wasn't here but a lot of things were merged for that stuff last night ... ah not found image :/ | 11:01 |
| marios | 2019-03-06 10:47:55 | ImageNotFoundException: Not found image: docker://trunk.registry.rdoproject.org/tripleomaster/centos-binary-cron:01561d3ce52da677ef7e7c6e9618b16ef431af08_dd831c96 | 11:01 |
| ykarel | quiquell, yes reporting is working now:- https://trunk-primary.rdoproject.org/api-centos-master-uc/api/civotes_detail.html?commit_hash=01561d3ce52da677ef7e7c6e9618b16ef431af08&distro_hash=dd831c96abed64e66d3a66a7dcdf6b8228838bf1 | 11:01 |
| quiquell | well one less :-) | 11:02 |
| marios | we probably want to revert this https://review.rdoproject.org/r/#/c/19108/ i think (or maybe all of them but its a bit of amess | 11:02 |
| marios | like they pulled the trigger on it with https://review.rdoproject.org/r/#/c/19108/2/zuul.d/tripleo.yaml quiquell | 11:02 |
| marios | so all the jobs depends on the new containers build ykarel ^ | 11:02 |
| quiquell | marios: maybe we can do fast check about why container is not there | 11:04 |
| quiquell | marios: before reverting | 11:04 |
| quiquell | marios: running both jobs containers-build and containers-build-push could be a problem ? | 11:05 |
| quiquell | marios: like one rewriting the other or the like | 11:06 |
| quiquell | marios: I see both of them running https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/ | 11:06 |
| marios | quiquell: are they both running still? yeah i mean they both build and pushw ith same tag (another thing merged last night was the tag switch https://review.rdoproject.org/r/#/c/19066/ | 11:06 |
| marios | :/ | 11:06 |
| quiquell | marios: humm they are here both https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/ | 11:07 |
| marios | quiquell: but even so it doesn't explain missin container unless one of them builds it the other doesn't or something | 11:07 |
| quiquell | marios: well good to cleanup and discover | 11:07 |
| marios | quiquell: yeah ok should i go ahead or you on it? | 11:08 |
| ykarel | marios, quiquell i know why they are missing | 11:08 |
| marios | ykarel: will you share :) ? | 11:08 |
| ykarel | marios, ya | 11:08 |
| marios | ykarel: how much? | 11:08 |
| ykarel | marios, quiquell so old kolla push job used to push both tripleo-ci-testing and repo hash | 11:09 |
| ykarel | but new job shouldn't be doing that | 11:09 |
| ykarel | so standalone will be passing, because there we are directly consuming tripleo-ci-testing | 11:10 |
| ykarel | quiquell, rememeber tag_from_label thing | 11:10 |
| marios | ykarel: so we need to push containers twice? | 11:10 |
| quiquell | ykarel: I remmeber there is a bug around :-/ | 11:10 |
| marios | ykarel: like once with ooo-ci-testing and once with the hash of repo being used? | 11:10 |
| *** panda|ruck|off is now known as panda|ruck|flu | 11:10 | |
| panda|ruck|flu | 'morning | 11:10 |
| panda|ruck|flu | anything I missed this morning ? | 11:10 |
| marios | panda|ruck|flu: o/ everything is broken go back to bed | 11:11 |
| ykarel | marios, yes both should be pushed atleast seeing the fact how kolla build pushes | 11:11 |
| ykarel | tag_from_label: rdo_version | 11:11 |
| ykarel | ^^ setting looks for repo_hash | 11:11 |
| marios | panda|ruck|flu: some issue with the containers build currently being discussed (see mail from wes about the stuff that was merged there while we slept innocently) | 11:11 |
| panda|ruck|flu | marios: ok | 11:11 |
| quiquell | ykarel: so it'0s about changing the prepare-containers yaml files with the tag_from_label ? | 11:15 |
| quiquell | ykarel, marios: like here ? https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-master/f27242d/logs/undercloud/home/zuul/containers-prepare-parameter.yaml.txt.gz | 11:16 |
| ykarel | quiquell, yes, by default, tag_from_label is set to rdo_version, so it will always look for container tagged with FULL_HASH for tripleo-ci-testing | 11:17 |
| ykarel | and in standalone due to a bug, tag_from_label is set to null, so it fetches containers with tag: tripleo-ci-testing | 11:17 |
| ykarel | or whatever tag is set. | 11:17 |
| marios | ykarel: am trying to find the tag_from_label in ansible-role-rdo-kolla-build but don't find it | 11:18 |
| ykarel | marios, it's tripleo-common thing, ansible-role-rdo-kolla-build don't rely on it | 11:18 |
| marios | ykarel: found the version hash though for rdo version https://github.com/rdo-infra/ansible-role-rdo-kolla-build/blob/e2618e2cc179828c62e0ea11bd7b971cb38d8928/templates/template-overrides.j2#L6 | 11:18 |
| quiquell | ykarel: adding "tag: tripleo-ci-testing" | 11:18 |
| quiquell | ykarel: to those template would be enough ? | 11:19 |
| quiquell | ykarel: or tag_from_label will ignore that ? | 11:19 |
| marios | ykarel: quiquell yah see it now and its in the defaults | 11:19 |
| quiquell | marios: what default ? | 11:20 |
| marios | ./container-images/container_image_prepare_defaults.yaml i mean quiquell | 11:21 |
| marios | https://github.com/openstack/tripleo-common/blob/6f88e900a085ad68b679d32058a30d3e9196a769/container-images/container_image_prepare_defaults.yaml#L5 | 11:21 |
| ykarel | marios, so https://github.com/rdo-infra/ansible-role-rdo-kolla-build/blob/cc966cf8c3154e330fd4b6cd7fe7c520c4b2b1dc/tasks/tag.yml#L23-L24 pushes hash containers | 11:22 |
| marios | quiquell: ykarel so if we explicitly don't set it? or set to none? | 11:22 |
| marios | ykarel: thanks (have some good backround here https://tree.taiga.io/project/tripleo-ci-board/us/652 ) | 11:23 |
| quiquell | ykarel: so we have to push twice to have hash and tag, is that right ? | 11:23 |
| marios | quiquell: thats what rdo kolla build does i think | 11:24 |
| ykarel | quiquell, yes atleast to get job passing and new push to be compatible with tag_from_label | 11:24 |
| quiquell | ykarel: is not easier to change container-prep yaml templates ? or this is more dangerous ? | 11:24 |
| marios | quiquell: yeah i think that is more dangerous | 11:25 |
| marios | i would prefer not to change that cos it affects everything at least not without more thought/slowly | 11:25 |
| marios | but it seems like that ship sailed | 11:25 |
| panda|ruck|flu | quiquell: you added a patch for the post failure somewhere ? | 11:25 |
| quiquell | panda|ruck|flu: my fix from yesterday was not complete :-( | 11:25 |
| quiquell | panda|ruck|flu: also zuul output is not very helpful, reproduced it locally with a simple ansible call and it give you a warning | 11:26 |
| marios | quiquell: theres a lot of logina round that (tag_from_label) under tripleo_common | 11:26 |
| marios | s/logina/logic | 11:26 |
| quiquell | marios: ack let's repush then | 11:26 |
| quiquell | marios: yep is nightmare I have a bug to fix around that dind't have time though | 11:26 |
| marios | quiquell: k will try post something in a sec thanks ykarel | 11:26 |
| ykarel | ack | 11:26 |
| marios | quiquell: i think we should also disable the old job as you said we don't want both pushing anyway | 11:27 |
| marios | quiquell: i'll include in one | 11:27 |
| quiquell | marios: maybe removing also the old containers-build jobs from pipeline ? | 11:27 |
| marios | quiquell: right is what i just said ^ | 11:27 |
| quiquell | panda|ruck|flu: https://review.rdoproject.org/r/#/c/19125/ | 11:27 |
| panda|ruck|flu | quiquell: ok already merged. thanks | 11:28 |
| quiquell | panda|ruck|flu: I have also add a review with ansible-lint for tripleo stuff at config https://review.rdoproject.org/r/#/c/19129/ | 11:28 |
| quiquell | panda|ruck|flu: but I know this stuff is complicated | 11:28 |
| quiquell | panda|ruck|flu: would have catch the identation issue | 11:28 |
| ykarel | marios, quiquell but old containers build is not running currently, if that was running container not found issue would not be there | 11:29 |
| panda|ruck|flu | quiquell: but indentation issue was what was preventing zuul to trigger the jobs ? | 11:29 |
| quiquell | panda|ruck|flu: nope | 11:29 |
| quiquell | panda|ruck|flu: that is different I think, marios? ^ | 11:30 |
| marios | ykarel: quiquell we can't retag i mean, we are using kolla conf. so we aren't tagging manually with docker tag like https://github.com/rdo-infra/ansible-role-rdo-kolla-build/blob/master/tasks/tag.yml does | 11:30 |
| marios | quiquell: ykarel we will have to change the whole job/me not sure how we'll do the second tag yet | 11:30 |
| marios | poking anyway | 11:30 |
| *** chem has joined #oooq | 11:31 | |
| panda|ruck|flu | damn how would I like to help you guys ... I still have a lot of promotion blockers to follow | 11:33 |
| quiquell | panda|ruck|flu: Nah don't worry team work | 11:33 |
| marios | ykarel: quiquell ah so even current job does build & push, then retrieve and retag again https://github.com/rdo-infra/ansible-role-rdo-kolla-build/blob/e2618e2cc179828c62e0ea11bd7b971cb38d8928/tasks/main.yml#L79-L111 | 11:34 |
| marios | so we need the second step for retag ok doable i think | 11:35 |
| *** dsneddon has quit IRC | 11:35 | |
| ykarel | marios, ack | 11:36 |
| quiquell | marios: so it's just missing the repush is that it ? | 11:37 |
| quiquell | ahh no it does tag.yaml that also push too | 11:37 |
| marios | quiquell: yeah we need another step after the build finishes, retrieve and retag and repush | 11:37 |
| marios | quiquell: so it is indeed 2 push | 11:38 |
| quiquell | marios: tag.yaml is pushing already | 11:38 |
| marios | quiquell: first push with tripleo-ci-testing in https://github.com/rdo-infra/ansible-role-rdo-kolla-build/blob/e2618e2cc179828c62e0ea11bd7b971cb38d8928/tasks/main.yml#L81 | 11:38 |
| marios | quiquell: then push again with new tag https://github.com/rdo-infra/ansible-role-rdo-kolla-build/blob/e2618e2cc179828c62e0ea11bd7b971cb38d8928/tasks/main.yml#L103 | 11:38 |
| quiquell | marios: then why are we missing the second push ? | 11:39 |
| marios | quiquell: thats the old job i pointing to | 11:39 |
| marios | quiquell: new one only does the first step | 11:39 |
| quiquell | ahh ok | 11:39 |
| quiquell | marios: wehere is the stuff of new one ? | 11:39 |
| marios | quiquell: https://github.com/openstack-infra/tripleo-ci/blob/master/playbooks/tripleo-buildcontainers/run.yaml | 11:40 |
| quiquell | marios: so we generaate another kolla config and call again | 11:43 |
| marios | quiquell: could do that but it would rebuild i think instead we need to just retrieve and tag | 11:43 |
| marios | quiquell: like current job | 11:43 |
| marios | quiquell: will post something in bit doing then we can discuss specifics | 11:44 |
| marios | i mean by pointing to code review | 11:44 |
| quiquell | ack | 11:44 |
| *** ratailor has quit IRC | 11:54 | |
| *** sshnaidm|afk has joined #oooq | 12:01 | |
| *** holser_ is now known as holser|lunch | 12:02 | |
| *** dsneddon has joined #oooq | 12:06 | |
| *** dsneddon has quit IRC | 12:11 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010 (5 more messages) | 12:14 |
| chandankumar | arxcruz: Hello | 12:15 |
| chandankumar | arxcruz: skip list is working http://logs.openstack.org/87/641287/2/check/tripleo-ci-centos-7-standalone-os-tempest/f65a92b/logs/stestr_results.html | 12:15 |
| arxcruz | chandankumar: after 40 patches, of course it's working :P | 12:15 |
| chandankumar | arxcruz: https://review.openstack.org/#/c/641287/2 | 12:15 |
| chandankumar | arxcruz: thank you for working on that! It was one of the important pieces | 12:16 |
| chandankumar | arxcruz++ | 12:16 |
| hubbot1 | chandankumar: arxcruz's karma is now 14 | 12:16 |
| arxcruz | ;) | 12:16 |
| chandankumar | now I think we can move the job to voting | 12:16 |
| arxcruz | chandankumar: I already move the card to done, but would be nice if you can document there | 12:16 |
| arxcruz | chandankumar: if you haven't yet | 12:16 |
| kopecmartin | chandankumar, https://review.openstack.org/#/c/638272/ this contains the cinder discovery fix | 12:20 |
| kopecmartin | arxcruz, can you have a look at this? is it fine i removed the get_gatalog method? check latest comment https://review.openstack.org/#/c/638272/ | 12:20 |
| chandankumar | kopecmartin: tested here https://review.openstack.org/#/c/641287/2 | 12:21 |
| chandankumar | kopecmartin: it is working | 12:21 |
| kopecmartin | chandankumar, \o/ | 12:21 |
| chandankumar | kopecmartin: result is here http://logs.openstack.org/87/641287/2/check/tripleo-ci-centos-7-standalone-os-tempest/f65a92b/logs/stestr_results.html | 12:21 |
| arxcruz | kopecmartin: you're replacing get_catalog for get_codename right ? | 12:22 |
| kopecmartin | arxcruz, yes, the method was not used anywhere | 12:22 |
| arxcruz | kopecmartin: are you sure? I remember this was related to when we get the services from the catalog, but it wasn't accurate | 12:22 |
| arxcruz | i don't know, if it's working, and it's not being used anywhere, I am okay | 12:23 |
| kopecmartin | arxcruz, there was a plan for it , but i don't know which and then we moved different way in refactoring , i don't know | 12:23 |
| arxcruz | kopecmartin: can we do a recheck once the os_tempest job get merged so we can get that results as well ? | 12:23 |
| kopecmartin | i grepped the code and it's not used | 12:23 |
| kopecmartin | sure | 12:23 |
| arxcruz | kopecmartin: cool, os_tempest is our priority now, so changes to python-tempestconf must be tested there just in case :) | 12:24 |
| arxcruz | once the os_tempest job is merged and you recheck and everything pass i'll +2 | 12:24 |
| kopecmartin | arxcruz, that's reasonable | 12:24 |
| kopecmartin | shit, according the zuul, the job will fail :/ | 12:25 |
| quiquell | arxcruz: wre you able to run reproducer ? | 12:28 |
| panda|ruck|flu | I think I will need to reproduce a job. Every attempt to understand what's happening failed, and I need to understand why a command is failing | 12:29 |
| quiquell | panda|ruck|flu: what job ? | 12:31 |
| panda|ruck|flu | quiquell: featureset001 | 12:31 |
| quiquell | panda|ruck|flu: ovb ? | 12:31 |
| panda|ruck|flu | there's a mkfs tahta fails but its output is not logged anywhere | 12:31 |
| panda|ruck|flu | quiquell: yes | 12:31 |
| *** jpena is now known as jpena|lunch | 12:31 | |
| quiquell | panda|ruck|flu: reproducer script should work out of the box | 12:32 |
| quiquell | panda|ruck|flu: remover to do autohold | 12:32 |
| *** udesale has joined #oooq | 12:38 | |
| *** dsneddon has joined #oooq | 12:40 | |
| *** dsneddon has quit IRC | 12:44 | |
| weshay|rover | zbr http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-fedora-28-master-containers-build-push/bb86bb4/job-output.txt.gz | 12:45 |
| weshay|rover | marios ^ | 12:45 |
| weshay|rover | can you guys get the kolla patches fixed up? | 12:45 |
| marios | weshay|rover: trying to fix the issue i sent on email | 12:46 |
| marios | weshay|rover: will post something in a sec for that | 12:46 |
| marios | looking at the link | 12:46 |
| marios | weshay|rover: ah thats the kolla patch needs rebase? | 12:46 |
| marios | zbr: can you do that never seen that patch before | 12:47 |
| marios | quiquell: last problem i have right now is we don't use build id, and they retrieve the list with build id like here https://github.com/rdo-infra/ansible-role-rdo-kolla-build/blob/e2618e2cc179828c62e0ea11bd7b971cb38d8928/tasks/main.yml#L97 i mean the list that is retagged. maybe i'll just use the tag to filter :/ | 12:48 |
| marios | hm can't filter on tag ? | 12:50 |
| marios | no you can sorry | 12:53 |
| weshay|rover | marios I don't think zbr is here today | 13:02 |
| marios | quiquell: ykarel posted just now https://review.openstack.org/641348 | 13:03 |
| quiquell | marios: is not this the old code ? | 13:03 |
| marios | weshay|rover: ack ok i can have a look in a sec | 13:03 |
| marios | quiquell: yeah i am copying from the rdo kolla build and trying to do same | 13:04 |
| *** holser|lunch is now known as holser_ | 13:04 | |
| marios | quiquell: but main difference is there is no build_id | 13:04 |
| quiquell | marios: you are not going to use openstack container build stuff ? | 13:04 |
| marios | quiquell: so retrieveing by tag | 13:04 |
| marios | quiquell: we are using that what you mean. the container build stuff happens first then this retag | 13:04 |
| weshay|rover | panda|ruck|flu can we chat about the ordering of dlrn hash tagging? | 13:04 |
| weshay|rover | when you have a minute | 13:05 |
| quiquell | ahh ok "manual" retag | 13:05 |
| panda|ruck|flu | weshay|rover: after the program mmeting ? | 13:05 |
| weshay|rover | aye | 13:05 |
| *** dsneddon has joined #oooq | 13:05 | |
| panda|ruck|flu | weshay|rover: you reporting ? | 13:06 |
| marios | weshay|rover: why did we decide to pull the trigger on that btw https://review.rdoproject.org/r/#/c/19108/ and https://review.rdoproject.org/r/#/c/19066/ seems a bit dramatic | 13:06 |
| weshay|rover | panda|ruck|flu ya.. you can drop if you want | 13:06 |
| marios | i mean i thought we would do it this week 'sometime' :D | 13:06 |
| marios | weshay|rover: ^ | 13:06 |
| marios | weshay|rover: was a bit surprised this morning the world merged | 13:06 |
| marios | weshay|rover: hopefully once we fix this retagging thing should be ok. ont he other hand we wouldn't be fixing this issue if you didn't merge everything so... | 13:07 |
| weshay|rover | marios once I saw that the f28 job was building and pushing into the registry I realized we were closer than I thought | 13:07 |
| weshay|rover | marios which retagging issue? | 13:07 |
| marios | weshay|rover: yeah but especially this https://review.rdoproject.org/r/#/c/19066/4/zuul.d/jobs.yaml and the final switch https://review.rdoproject.org/r/#/c/19108/2/zuul.d/tripleo.yaml i wasn't sure we were ready for (i.e. actually push with tripleo-ci-testing and replace the existig job) | 13:08 |
| marios | weshay|rover: i replied on your email like https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-master/f27242d/logs/undercloud/home/zuul/undercloud_install.log.txt.gz 2019-03-06 10:47:55 | Exception: Not found image: docker://trunk.registry.rdoproject.org/tripleomaster/centos-binary-cron:01561d3ce52da677ef7e7c6e9618b16ef431af08_dd831c96 | 13:08 |
| marios | weshay|rover: ykarel spotted it and suggests its cos we aren't tagging with the version hash like the rdo kolla build was doing | 13:09 |
| weshay|rover | marios oh ya | 13:09 |
| marios | weshay|rover: i.e. tag_from_label | 13:09 |
| weshay|rover | but that has nothing to do w/ f28 and the changes we made last night | 13:09 |
| marios | weshay|rover: trying that with https://review.openstack.org/641348 | 13:09 |
| marios | weshay|rover: yeah like i said above, if you hadn't merged we wouldn't see this issue so soon we would have seen it later/next week | 13:10 |
| marios | so... | 13:10 |
| marios | ;D | 13:10 |
| marios | thanks? | 13:10 |
| weshay|rover | ya.. we were ready for those changes | 13:10 |
| quiquell | marios: Can we filter tripleo-ci-testing somehow ? | 13:10 |
| weshay|rover | so we see MORE | 13:10 |
| quiquell | marios: taking some ansible var or something ? | 13:10 |
| marios | quiquell: thats what im trying to do | 13:10 |
| marios | quiquell: https://review.openstack.org/#/c/641348/2/playbooks/tripleo-buildcontainers/run.yaml@92 | 13:11 |
| marios | --filter "reference=*/*:{{ push_tag }}" | 13:11 |
| *** dsneddon has quit IRC | 13:11 | |
| marios | quiquell: cos we don't have/set a build id. We could set it if i can work out how but we don't use a docker file there and kolla build conf doesn't take label afaics | 13:11 |
| marios | quiquell: in rdo kolla build they are generating a buildid like ./tasks/kolla.yml:129: build_id: "{{ lookup('pipe', 'date +%s') }}" | 13:12 |
| marios | quiquell: and they use that when they get the image list https://github.com/rdo-infra/ansible-role-rdo-kolla-build/blob/e2618e2cc179828c62e0ea11bd7b971cb38d8928/tasks/main.yml#L98 | 13:12 |
| quiquell | marios: do you have a running system or the like ? | 13:14 |
| marios | quiquell: no what you thinking? i hope to try that with a testproject & depends on | 13:16 |
| marios | quiquell: another difference there is https://review.openstack.org/#/c/641348/2/playbooks/tripleo-buildcontainers/run.yaml@82 | 13:16 |
| marios | quiquell: i am just using the installed delorean.repo here | 13:16 |
| marios | vs they were retrieveing it but https://github.com/rdo-infra/ansible-role-rdo-kolla-build/blob/e2618e2cc179828c62e0ea11bd7b971cb38d8928/tasks/repositories.yml#L25 | 13:17 |
| quiquell | marios: docker images |grep stable |awk '{ print $3 }' | 13:17 |
| quiquell | this ? | 13:17 |
| marios | quiquell: but i think it is ok cos by that point we already installed the delorean.repo | 13:18 |
| quiquell | but with corect tag | 13:18 |
| marios | quiquell: why not what i have with the filter reference is it wrong? | 13:18 |
| marios | quiquell: docker images --format "{{ '{{' }}.Repository{{ '}}' }}" \ --filter "reference=*/*:{{ push_tag }}" | 13:18 |
| quiquell | marios: --filter "reference='*/*:latest'" ? | 13:18 |
| marios | wher push_tag is like tripleo-ci-testing | 13:18 |
| marios | quiquell: err sorry i updated it to have pushtag | 13:19 |
| marios | quiquell: see v2 | 13:19 |
| quiquell | ahh ok | 13:19 |
| quiquell | I am just using latest at my machine and it's not working | 13:19 |
| marios | quiquell: yeah sorry its meant to be tha tag | 13:20 |
| quiquell | I mean | 13:20 |
| quiquell | this docker images --format "{{ .Repository }}" --filter "reference=*/*:latest" | 13:20 |
| quiquell | Have to show me like this docker images |grep stable |awk '{ print $3 }' | 13:21 |
| quiquell | or similar | 13:21 |
| quiquell | but is not | 13:21 |
| quiquell | I am talking about my local images here at my laptop | 13:21 |
| quiquell | but is more or less th esame | 13:21 |
| quiquell | containers I mean | 13:21 |
| marios | quiquell: 'latest' is meant to be the tag name | 13:21 |
| quiquell | marios: yep | 13:21 |
| weshay|rover | marios, panda|ruck|flu ya.. some how when looking at the older push script the tagging w/ the hash was missed | 13:21 |
| quiquell | marios: I have docker container with latest but they don't show with the filter | 13:21 |
| *** rlandy has joined #oooq | 13:21 | |
| marios | quiquell: can you try --filter "reference=*:latest" | 13:22 |
| quiquell | marios: working | 13:23 |
| quiquell | it does not work with stable though | 13:24 |
| marios | quiquell: nice thanks updating. | 13:24 |
| quiquell | but we don't care | 13:24 |
| quiquell | marios: btw it returns me the imag ename | 13:24 |
| quiquell | no the image id | 13:24 |
| quiquell | is that ok ? | 13:24 |
| marios | quiquell: that's what the --format "{{ '{{' }}.Repository{{ '}}' }}" is for | 13:24 |
| quiquell | marios: I know, but feels weird not using image id | 13:25 |
| quiquell | marios: na ok let's roll | 13:25 |
| marios | quiquell: to give the repo info does it work ok then ? that part is just lifted from the rdo kolla stuff | 13:25 |
| quiquell | marios: and see | 13:25 |
| *** amoralej is now known as amoralej|lunch | 13:26 | |
| *** sshnaidm|afk is now known as sshnaidm | 13:32 | |
| quiquell | sshnaidm: o/ looks like nested virt is all broken now at RDO | 13:32 |
| quiquell | sshnaidm: reproducer ci is screw | 13:32 |
| quiquell | sshnaidm: even increasing retries http://logs.rdoproject.org/47/19047/4/check/tripleo-ci-reproducer-fedora-28-libvirt/8d8bc74/job-output.txt.gz | 13:33 |
| sshnaidm | quiquell, I think everything is screw in rdo atm | 13:33 |
| quiquell | sshnaidm: thing increasing retries does not fix it | 13:33 |
| *** jpena|lunch is now known as jpena | 13:34 | |
| quiquell | sshnaidm: have to be something else with Get libvirt nodepool IP addresses | 13:34 |
| sshnaidm | quiquell, I mean it may be because of last outages and problems that we have there | 13:34 |
| quiquell | sshnaidm: force_tcg is not enough | 13:34 |
| quiquell | sshnaidm: kforde just confirmed the nested virt issue at all hypervisors | 13:34 |
| weshay|rover | panda|ruck|flu I good to go early if you want | 13:34 |
| weshay|rover | quiquell sshnaidm ya.. there was an email sent | 13:35 |
| weshay|rover | I'll forward | 13:35 |
| weshay|rover | sent | 13:35 |
| *** dsneddon has joined #oooq | 13:36 | |
| sshnaidm | quiquell, so what are our options? ci.centos? :D | 13:37 |
| marios | quiquell: fyi https://review.rdoproject.org/r/19131 testproject for the retag | 13:38 |
| sshnaidm | quiquell, let's try maybe to see console output of libvirt machine.. | 13:38 |
| panda|ruck|flu | weshay|rover: ok, joining | 13:39 |
| quiquell | sshnaidm: maybe put in place te testing teanant to exercise openstack | 13:39 |
| sshnaidm | quiquell, what do you mean? | 13:39 |
| weshay|rover | quiquell sshnaidm you guys may be able to utilize upshift internal | 13:39 |
| quiquell | weshay|rover: humm... | 13:39 |
| sshnaidm | weshay|rover, I don't think we can trigger jobs on internal cloud yet | 13:39 |
| quiquell | weshay|rover: can we make voting jobs at RDO from third party ? | 13:39 |
| weshay|rover | k | 13:39 |
| quiquell | like real votin g | 13:40 |
| sshnaidm | at least what I heard from apevec | 13:40 |
| quiquell | sshnaidm: we can | 13:40 |
| sshnaidm | quiquell, how? | 13:40 |
| weshay|rover | can you share w/ me what Alan said | 13:40 |
| quiquell | sshnaidm: we have whole zuul tenant for ourselfs | 13:40 |
| quiquell | sshnaidm: it's a matter to wait for changes at rdo review project | 13:40 |
| quiquell | sshnaidm: but don't know how to vote at rdo | 13:41 |
| quiquell | sshnaidm: like really vote | 13:41 |
| sshnaidm | quiquell, do you mean to setup a different zuul internally? | 13:41 |
| quiquell | sshnaidm: the internal sf | 13:41 |
| sshnaidm | quiquell, I meant registering of internal cloud in rdo sf | 13:41 |
| quiquell | sshnaidm: we can put job there to run libvirt | 13:41 |
| quiquell | sshnaidm: humm we are talking about two things | 13:41 |
| quiquell | sshnaidm: one is where to run libvirt | 13:41 |
| quiquell | sshnaidm: and second how to exercise openstack nodepool provider | 13:42 |
| *** dsneddon has quit IRC | 13:42 | |
| sshnaidm | quiquell, bluej? | 13:42 |
| quiquell | lunch first | 13:42 |
| *** quiquell is now known as quiquell|lunch | 13:42 | |
| sshnaidm | quiquell|lunch, ack | 13:43 |
| *** dsneddon has joined #oooq | 13:46 | |
| *** dsneddon has quit IRC | 13:54 | |
| *** quiquell|lunch is now known as quiquell | 13:55 | |
| quiquell | sshnaidm: ready | 13:55 |
| quiquell | sshnaidm: going to your blue | 13:55 |
| sshnaidm | quiquell, https://bluejeans.com/u/sshnaidm/ | 13:57 |
| marios | weshay|rover: i *think* rebased but first time i seen those so might be nits... :/ https://review.openstack.org/624838 https://review.openstack.org/632156 https://review.openstack.org/639219 | 14:00 |
| marios | weshay|rover: the merge conflict was there https://review.openstack.org/#/c/632156/21/docker/openstack-base/Dockerfile.j2 | 14:01 |
| marios | adding a comment | 14:01 |
| *** rfolco|pto is now known as rfolco|ruck | 14:02 | |
| panda|ruck|flu | rfolco|ruck: what are you doing here ? | 14:07 |
| rlandy | quiquell: weshay|rover: initial bm doc merged http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/doc/oooq-upstream-baremetal.rst until we decide what to do about upstream doc | 14:07 |
| rfolco|ruck | panda|ruck|flu, :) | 14:07 |
| rlandy | quiquell: you can go ahead and add to that/edit it | 14:07 |
| rfolco|ruck | panda|ruck|flu, carnival is over. What happens in carnival, stays in carnival. | 14:09 |
| rfolco|ruck | panda|ruck|flu, should we switch ? oh wait, you're sick! | 14:09 |
| weshay|rover | panda|ruck|flu http://logs.openstack.org/95/640895/1/gate/tripleo-ci-centos-7-standalone/8a77633/logs/stackviz/#/testrepository.subunit/test-details/tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops | 14:09 |
| weshay|rover | http://logs.openstack.org/80/639080/1/gate/tripleo-ci-centos-7-standalone/3e104bf/logs/stackviz/#/testrepository.subunit/test-details/tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_subnet_details | 14:09 |
| chandankumar | PTAL needs +w on this https://review.openstack.org/#/c/640089/ | 14:11 |
| rfolco|ruck | panda|ruck|flu, please ping me when you can sync | 14:13 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010 (5 more messages) | 14:14 |
| *** gkadam_ has joined #oooq | 14:16 | |
| *** vinaykns has joined #oooq | 14:17 | |
| *** gkadam has quit IRC | 14:19 | |
| weshay|rover | marios can I review this with you for a sec https://review.openstack.org/#/c/641348/1/playbooks/tripleo-buildcontainers/run.yaml | 14:19 |
| weshay|rover | - name: Retrieve list of built images # TODO build_id? | 14:20 |
| weshay|rover | shell: > | 14:20 |
| weshay|rover | docker images --format "{{ '{{' }}.Repository{{ '}}' }}" \ | 14:20 |
| weshay|rover | --filter "reference='*/*:latest'" | 14:20 |
| weshay|rover | register: built_images | 14:20 |
| *** rascasoft has quit IRC | 14:20 | |
| weshay|rover | is that querying the local images on the system.. or the registry/ | 14:20 |
| marios | weshay|rover: sure iam just grabbing some food cos didn't get a chance real quick gimme few? | 14:21 |
| weshay|rover | np | 14:21 |
| weshay|rover | rfolco|ruck are you back today? | 14:22 |
| quiquell | sshnaidm: I am thinking about opposite direction, what about moving reproducer role upstream and we use upstream nodepool ? | 14:23 |
| quiquell | sshnaidm: maybe nested libvirt has no issues there | 14:23 |
| weshay|rover | panda|ruck|flu third one.. diff tempest failure http://logs.openstack.org/85/633885/13/gate/tripleo-ci-centos-7-standalone/6954de8/logs/stackviz/#/testrepository.subunit | 14:23 |
| sshnaidm | quiquell, I think they don't have nested completely | 14:23 |
| weshay|rover | chandankumar tempest basic ops on standalone centos is starting to get flaky again | 14:24 |
| quiquell | sshnaidm: I suppose depends on the provider the job ends... | 14:24 |
| chandankumar | weshay|rover: will take a look! | 14:24 |
| sshnaidm | quiquell, I think we tried to run libvirt upstream, right weshay|rover ? | 14:24 |
| weshay|rover | 3 diff tests fail in the gate in the last 24hr | 14:24 |
| *** dsneddon has joined #oooq | 14:24 | |
| sshnaidm | quiquell, even so you can't choose provider afaik | 14:25 |
| weshay|rover | chandankumar for the pattern consult http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&panelId=61&fullscreen | 14:25 |
| weshay|rover | standalone jobs | 14:25 |
| quiquell | sshnaidm: ack | 14:25 |
| quiquell | Depends on hypervisors more or less | 14:25 |
| weshay|rover | meh.. http://logs.openstack.org/22/640722/2/gate/tripleo-ci-centos-7-undercloud-containers/7006d00/logs/undercloud/home/zuul/tempest.log.txt.gz | 14:26 |
| weshay|rover | and undercloud.. | 14:26 |
| chandankumar | does these errors common http://logs.openstack.org/85/633885/13/gate/tripleo-ci-centos-7-standalone/6954de8/logs/undercloud/var/log/extra/errors.txt.txt.gz ? | 14:27 |
| weshay|rover | can you rephrase the question | 14:28 |
| chandankumar | yeah these are common | 14:28 |
| chandankumar | I was comparing errors.txt file fo a passed /failed one | 14:28 |
| chandankumar | need to look somewhere else why it is timing out | 14:29 |
| *** amoralej|lunch is now known as amoralej | 14:29 | |
| *** dsneddon has quit IRC | 14:29 | |
| chandankumar | weshay|rover: 1:1 time | 14:31 |
| weshay|rover | chandankumar coming | 14:31 |
| sshnaidm | quiquell, let's try on vexxhost? | 14:31 |
| quiquell | sshnaidm: is special nodset or how do we use it ? | 14:33 |
| quiquell | sshnaidm: https://review.rdoproject.org/r/19047 | 14:33 |
| quiquell | sshnaidm: upstream-centos-7-vexxhost | 14:35 |
| quiquell | this ? | 14:35 |
| quiquell | humm we don't have fedora28 | 14:35 |
| sshnaidm | quiquell, https://review.rdoproject.org/r/#/c/19134/ | 14:35 |
| sshnaidm | quiquell, let's see centos | 14:35 |
| quiquell | ack | 14:36 |
| quiquell | goot call vexxhost though | 14:36 |
| quiquell | sshnaidm: about git review from RPMs centos need EPEL :-/ | 14:37 |
| quiquell | sshnaidm: and also having it it was not working | 14:38 |
| sshnaidm | quiquell, hmm.. | 14:39 |
| sshnaidm | quiquell, seem like I had epel preinstalled | 14:39 |
| sshnaidm | quiquell, so pip? | 14:39 |
| quiquell | sshnaidm: yep, but RPM fails still :-/ | 14:39 |
| quiquell | sshnaidm: git install --user git-review before doing the push | 14:40 |
| sshnaidm | quiquell, "pip install"? | 14:40 |
| quiquell | s/git/pip/ | 14:40 |
| marios | weshay|rover: o/ lemme know when you want to talk | 14:42 |
| *** dsneddon has joined #oooq | 14:42 | |
| *** rascasoft has joined #oooq | 14:43 | |
| quiquell | rlandy: we cannot parent from RDO jobs | 14:43 |
| quiquell | rfolco|ruck: I mean if we want to have clean config | 14:43 |
| quiquell | rlandy: ^ | 14:43 |
| quiquell | rlandy: I removed it from tenant project s/[job]/[]/ | 14:43 |
| quiquell | Do I remove it from doc ? | 14:44 |
| quiquell | damn maybe we will need stuff for OVB ? | 14:45 |
| rlandy | quiquell: ack - I am about to add a osp ovb job there | 14:46 |
| quiquell | rlandy: what stuff do we need for RDO for OVB ? | 14:46 |
| quiquell | maybe just roles ? | 14:46 |
| rlandy | quiquell: let's just leave the doc and parenting for now | 14:46 |
| quiquell | ack | 14:46 |
| rlandy | quiquell: I am getting our upshift tenant back into action | 14:46 |
| rlandy | let me see what shakes out there | 14:46 |
| quiquell | rlandy: no more NODE_FAILURES ? | 14:47 |
| rlandy | I will update you | 14:47 |
| *** dsneddon has quit IRC | 14:47 | |
| rlandy | quiquell: \o/ | 14:47 |
| quiquell | rlandy: doc is all good as I see it | 14:49 |
| quiquell | rlandy: I think it has the stuff needed to remember what was your life like when you come back from PTO :-) | 14:49 |
| quiquell | rlandy: so all good | 14:49 |
| quiquell | rlandy: make sense to add just a periodic that do reporting to DLRN just to execise stuff ? | 14:50 |
| quiquell | rlandy: without the DLRN trigger | 14:50 |
| *** rascasoft has quit IRC | 14:50 | |
| quiquell | rlandy: it will just add status to DLRN at some promotions | 14:50 |
| rlandy | quiquell: sure - we just need somewhere to start collecting doc for when we decide to enable the triggering | 14:51 |
| *** rascasoft has joined #oooq | 14:51 | |
| quiquell | rlandy: what do you mean ? | 14:51 |
| rlandy | quiquell: the original plan was to have this doc upstream | 14:51 |
| rlandy | which will be needed once we start reporting | 14:52 |
| rlandy | adding to the promotion criteria | 14:52 |
| rlandy | etc. | 14:52 |
| rlandy | right now, that doc is a dumping ground for us | 14:52 |
| quiquell | rlandy: I mean reporting but not adding to promotion criteria | 14:52 |
| rlandy | quiquell: we should discuss at tomorrow's scrum about ^^ | 14:52 |
| quiquell | ack | 14:52 |
| quiquell | I will do a handover about the dlrn trigger though | 14:53 |
| rlandy | what else needs to be done or we stop here for the moment | 14:53 |
| quiquell | I am on PTO two weeks | 14:53 |
| rlandy | quiquell: oh right - forgot that | 14:53 |
| rlandy | quiquell: pls doc whatever I left out | 14:53 |
| rlandy | so we can pick it up | 14:53 |
| quiquell | I will explain a little the DLRN trigger PoC in the doc we can remove it later | 14:53 |
| quiquell | is that ok ? | 14:53 |
| quiquell | or taiga task is enough ? | 14:54 |
| *** gkadam__ has joined #oooq | 15:00 | |
| weshay|rover | chandankumar https://trello.com/c/lp7wVIzA/36-propose-openstack-ansible-tempest-roles-and-playbooks-for-internal-ci-and-use | 15:00 |
| chandankumar | weshay|rover: thanks! | 15:02 |
| weshay|rover | np | 15:02 |
| * chandankumar is on pto tomorrow and day after tomorrow | 15:03 | |
| chandankumar | see ya | 15:03 |
| *** chandankumar is now known as chkumar|pto | 15:03 | |
| *** gkadam_ has quit IRC | 15:03 | |
| quiquell | marios: did you have issues running reproducer when you are in the VPN ? | 15:06 |
| rfolco|ruck | weshay|rover, yes I am back, carnival holiday ends wed lunch according to https://mojo.redhat.com/docs/DOC-1176015 | 15:08 |
| marios | quiquell: no didn't see that | 15:08 |
| rlandy | quiquell: ack | 15:08 |
| marios | quiquell: all my issues are in the taiga https://tree.taiga.io/project/tripleo-ci-board/task/765 ovb and 766 for libvirt | 15:09 |
| quiquell | ack | 15:09 |
| quiquell | my keys are not working now :-/ | 15:09 |
| quiquell | weird | 15:09 |
| sshnaidm | quiquell, seems like vexxhost is working.. | 15:09 |
| quiquell | \o/ | 15:09 |
| quiquell | sshnaidm: do you know if we can have a fedora28 image there ? | 15:10 |
| sshnaidm | quiquell, I think so | 15:10 |
| sshnaidm | quiquell, actually as I see fedora worked on rdo cloud too | 15:10 |
| quiquell | so maybe they have fix it | 15:10 |
| weshay|rover | quiquell did you patch the linter for jobs to catch indentation? | 15:11 |
| quiquell | weshay|rover: Have the review, but not merged | 15:11 |
| quiquell | since it's delicate | 15:11 |
| quiquell | weshay|rover: https://review.rdoproject.org/r/#/c/19129/ | 15:11 |
| weshay|rover | see it | 15:11 |
| weshay|rover | thanks | 15:11 |
| weshay|rover | quiquell++ | 15:11 |
| hubbot1 | weshay|rover: quiquell's karma is now 21 | 15:11 |
| quiquell | weshay|rover: have tested with broken problem and it was failing as expected | 15:12 |
| quiquell | sshnaidm: let's merge the vexhoost for centos | 15:12 |
| quiquell | sshnaidm: if fedora28 give us problems we move this too | 15:12 |
| *** dsneddon has joined #oooq | 15:13 | |
| *** agopi has joined #oooq | 15:14 | |
| sshnaidm | quiquell, let's change commit message at least :) | 15:18 |
| *** dsneddon has quit IRC | 15:18 | |
| *** agopi has quit IRC | 15:18 | |
| quiquell | sshnaidm: ack | 15:21 |
| quiquell | droping now | 15:21 |
| *** quiquell is now known as quiquell|off | 15:21 | |
| weshay|rover | marios re: the tagging | 15:23 |
| weshay|rover | is that patch deriving the container list from the local containers? | 15:23 |
| marios | weshay|rover: no its meant to be retrieving from the registry. like it tries to filter on tripleo-ci-testing with --filter "reference=*:{{ push_tag }}" https://review.openstack.org/#/c/641348/5/playbooks/tripleo-buildcontainers/run.yaml | 15:24 |
| marios | weshay|rover: i'm just trying to replicate what is in the rdo kolla build for th e retag | 15:24 |
| weshay|rover | marios so that's what I was afraid of | 15:24 |
| weshay|rover | I think that is problematic | 15:25 |
| weshay|rover | we have the list of built containers on the system... and that's what should be used | 15:25 |
| marios | weshay|rover: i.e. https://github.com/rdo-infra/ansible-role-rdo-kolla-build/blob/e2618e2cc179828c62e0ea11bd7b971cb38d8928/tasks/main.yml#L92 here is the rdo kolla buld | 15:25 |
| weshay|rover | marios ah so that was preexisting | 15:26 |
| marios | weshay|rover: we build and push with kolla already (and the push is happening as things are build) | 15:26 |
| weshay|rover | panda|ruck|flu ^ | 15:27 |
| marios | weshay|rover: so we can't tag twice there i.e. we already pushed stuff by the time the build containers part finishes | 15:27 |
| weshay|rover | can you review this with me | 15:27 |
| weshay|rover | panda|ruck|flu ^ | 15:27 |
| weshay|rover | panda|ruck|flu marios my concern is that we'll tag a container that we should not | 15:27 |
| marios | weshay|rover: yeah so one problem is that in rdo kolla build, they are retrieveing with a build id (which is something they set) like https://github.com/rdo-infra/ansible-role-rdo-kolla-build/blob/e2618e2cc179828c62e0ea11bd7b971cb38d8928/tasks/main.yml#L98 | 15:28 |
| marios | where build id is like ./tasks/kolla.yml:129: build_id: "{{ lookup('pipe', 'date +%s') }}" | 15:28 |
| marios | weshay|rover: but in our case i don't have such a thing to filter on so instead am using the tag | 15:28 |
| marios | weshay|rover: i.e. --filter "reference=*:{{ push_tag }}" | 15:29 |
| panda|ruck|flu | weshay|rover: which one sorry ? | 15:32 |
| rlandy | good bye jenkins ... jobs are officially deleted | 15:34 |
| weshay|rover | panda|ruck|flu /me gets | 15:34 |
| weshay|rover | panda|ruck|flu https://review.openstack.org/#/c/641348/1/playbooks/tripleo-buildcontainers/run.yaml | 15:35 |
| *** udesale has quit IRC | 15:37 | |
| marios | brb coffee refill | 15:37 |
| panda|ruck|flu | marios: are you pushing as tripleo-ci-testing and then retag with the actual hash ? | 15:39 |
| panda|ruck|flu | marios: I thing you should do the opposite | 15:39 |
| panda|ruck|flu | marios: so when you'll collect the images to retag, there will be no ambiguity | 15:40 |
| panda|ruck|flu | marios: you'll know that all the images you list will be for the hash you want to retag | 15:42 |
| weshay|rover | arxcruz kopecmartin ok... the tempest failures in the gate are now worrying me quite a bit | 15:43 |
| kopecmartin | weshay|rover, which ones? are they related to the fact that cinder is not discovered by python-tempestconf? | 15:44 |
| weshay|rover | standalone and undercloud jobs in http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&panelId=61&fullscreen and I'm watching the gate /me gets | 15:45 |
| weshay|rover | http://logs.openstack.org/59/641159/1/gate/tripleo-ci-centos-7-undercloud-containers/5155431/job-output.txt.gz | 15:45 |
| marios | panda|ruck|flu: yeah exactly build and tag/push with tripleo-ci-testing via the kolla build then retag with the hash | 15:46 |
| marios | panda|ruck|flu: but what you're saying makes sense except it neds some thought | 15:46 |
| marios | panda|ruck|flu: we only want to tag with the hash, when its periodic | 15:47 |
| arxcruz | weshay|rover: let me see | 15:47 |
| marios | panda|ruck|flu: well actually taggin in general only matters for periodic | 15:47 |
| marios | panda|ruck|flu: ok let me see if i can change it for that | 15:47 |
| arxcruz | if i recall correctly, this test was on skip list | 15:47 |
| arxcruz | tempest.api.network.test_ports.PortsIpV6TestJSON.test_update_port_with_two_security_groups_and_extra_attributes | 15:47 |
| *** dsneddon has joined #oooq | 15:48 | |
| *** dtantsur|mtg is now known as dtantsur|afk | 15:50 | |
| arxcruz | okay, it seems to be a bug | 15:50 |
| arxcruz | weshay|rover: i'll start work on that okay ? do you have a lp open already ? | 15:51 |
| weshay|rover | arxcruz thanks.. no bugs open yet on the most recent tempest failures panda|ruck|flu ^ | 15:51 |
| weshay|rover | thanks arxcruz! | 15:51 |
| arxcruz | weshay|rover: np, shall i open one? | 15:52 |
| arxcruz | weshay|rover: meanwhile, i'll also add this to skip list | 15:53 |
| weshay|rover | yes | 15:53 |
| panda|ruck|flu | arxcruz: nope no lp opened | 15:54 |
| weshay|rover | open as you wish.. arxcruz mark them alert | 15:54 |
| arxcruz | panda|ruck|flu: opening | 15:54 |
| weshay|rover | FYI.. meeting w/ jim re: IBM starting shortly | 15:54 |
| weshay|rover | all ^ | 15:55 |
| panda|ruck|flu | I already spoke with jim, everything's good. | 15:55 |
| arxcruz | panda|ruck|flu: weshay|rover https://bugs.launchpad.net/tripleo/+bug/1818860 | 15:56 |
| openstack | Launchpad bug 1818860 in tripleo "tempest.api.network.test_ports.PortsIpV6TestJSON.test_update_port_with_two_security_groups_and_extra_attributes failing with SecurityGroupNotFound" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 15:56 |
| sshnaidm | quiquell|off, now fedora started to fail, heh.. | 16:00 |
| rlandy | panda|ruck|flu: define good | 16:00 |
| panda|ruck|flu | rlandy: jim asked me to keep quiet. | 16:01 |
| rlandy | panda|ruck|flu: didn't know you two were so close | 16:01 |
| arxcruz | weshay|rover: panda|ruck|flu https://review.openstack.org/#/c/641429/ | 16:05 |
| weshay|rover | thanks arx | 16:06 |
| weshay|rover | look for some follow up emails from myself or panda|ruck|flu for tomorrow | 16:06 |
| arxcruz | that was the fastest review I got | 16:11 |
| *** saneax has quit IRC | 16:12 | |
| chkumar|pto | weshay|rover: https://review.openstack.org/#/c/640089/ if you are in mood of +w | 16:14 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010 (5 more messages) | 16:14 |
| * chkumar|pto needs to kill 1 hr time at airport! | 16:15 | |
| zbr | marios: quiquell|off i am back but I need some time to recover after >6h of driving. | 16:16 |
| zbr | i am trying not to fall asleep during the primetime... | 16:18 |
| marios | zbr: how did it go | 16:18 |
| marios | are you a citizen? :D | 16:18 |
| zbr | marios: they got my "biometrics", i should get a response in ~1-6mo, filed 60 documents. now I only need to wait. | 16:19 |
| marios | zbr: cool | 16:19 |
| *** jdennis has joined #oooq | 16:19 | |
| zbr | anyway, if you ever visit UK, you can safely drop Croydon from the places to visit. | 16:20 |
| zbr | quique way saying something about linting on some rdo hosted repo, but i do not know which one. glad to help there as we are little bit behind. | 16:21 |
| zbr | easy https://review.rdoproject.org/r/#/c/18631/ | 16:22 |
| chkumar|pto | zbr: how was the ride ? | 16:26 |
| zbr | boring | 16:27 |
| chkumar|pto | zbr: I travelled in a bus for 8 hours last month It was horrifying! | 16:27 |
| *** ccamacho has quit IRC | 16:27 | |
| chkumar|pto | zbr: which car you were driving? | 16:28 |
| weshay|rover | marios doy you want s/ "zuul is defined" to "zuul.pipeline is defined" | 16:29 |
| zbr | chkumar|pto: outlander phev,... well the battery got depleted in the first 40km. | 16:29 |
| * chkumar|pto google | 16:29 | |
| weshay|rover | marios when: '"value" in variable1' | 16:32 |
| weshay|rover | line 89 | 16:32 |
| marios | yah just talking with panda|ruck|flu about it in tripleo weshay|rover thanks | 16:36 |
| marios | doing | 16:36 |
| marios | panda|ruck|flu:++ | 16:38 |
| marios | panda|ruck|flu: ++ | 16:38 |
| marios | why karma bot ignores me | 16:38 |
| * marios shakes fist | 16:38 | |
| panda|ruck|flu | marios++ | 16:39 |
| hubbot1 | panda|ruck|flu: marios's karma is now 7 | 16:39 |
| marios | panda|ruck|flu++ | 16:40 |
| hubbot1 | marios: panda|ruck|flu's karma is now 1 | 16:40 |
| marios | panda++ | 16:40 |
| marios | k enough of that | 16:40 |
| panda|ruck|flu | enough love for today. | 16:41 |
| rlandy | zbr: 6 months is not bad - USA citizenship took me 13 years | 16:42 |
| *** gkadam__ has quit IRC | 16:42 | |
| marios | chkumar|pto: i once had to ride 5 hours next to panda it was best 5 hours of my life (pune->airport ;) ) | 16:43 |
| marios | chkumar|pto: (since we're talking about car rides) | 16:44 |
| zbr | rlandy: 6mo to decide, after you file and have the 6 years of residency (3 if you are married). | 16:44 |
| *** jfrancoa has quit IRC | 16:46 | |
| chkumar|pto | marios: what pune to aiport 5 hours! where you were staying? | 16:46 |
| chkumar|pto | in pune | 16:46 |
| chkumar|pto | marios: is this the westin hotel? | 16:47 |
| marios | chkumar|pto: we were driving to mumbai | 16:47 |
| marios | chkumar|pto: yeah hotel to mumbai | 16:47 |
| chkumar|pto | marios: oh, mumbai then it might take time with traffic it is 3 and 1/2 hrs | 16:48 |
| chkumar|pto | *without | 16:48 |
| marios | chkumar|pto: no man i am sure it was much more than that i did it twice :D | 16:48 |
| marios | once going and once coming | 16:48 |
| chkumar|pto | marios: panda|ruck|flu we can convince weshay|rover to bring down the whole team to pune! once again let's bring more fun! | 16:49 |
| weshay|rover | YES! | 16:49 |
| chkumar|pto | but it was awesome when I first time met marios and panda|ruck|flu in pune! | 16:50 |
| chkumar|pto | talked a lot like hell! | 16:50 |
| panda|ruck|flu | I deny that. Marios didn't exist for me before he joined the CI team. | 16:51 |
| chkumar|pto | let's bring the party to pune :-) | 16:51 |
| chkumar|pto | and if come pune! please spare one week we have lots of forts to track, if you are interested, suitable time in Aug and sept too much greenary! | 16:52 |
| weshay|rover | zbr please start on those jobs we discussed.. we're getting very close | 16:53 |
| marios | chkumar|pto: if you're not going to use your pto can i have it please? | 16:53 |
| chkumar|pto | and want to come to my home town then one more week, i.e. Bihar | 16:53 |
| chkumar|pto | marios: no, few mins left for board the flight! | 16:53 |
| marios | :D | 16:54 |
| marios | chkumar|pto: have fun mate | 16:54 |
| chkumar|pto | marios: thanks! | 16:54 |
| chkumar|pto | One thing I can stay proudly I work with the most awesome team where we break stuff and fix it a lot! | 16:54 |
| chkumar|pto | *say | 16:56 |
| *** panda|ruck|flu is now known as panda|ruck|off | 16:57 | |
| weshay|rover | sshnaidm sova down? | 16:58 |
| sshnaidm | weshay|rover, yes, rebooted by Kieran | 16:59 |
| chkumar|pto | time to flight out | 16:59 |
| chkumar|pto | see ya on monday! | 16:59 |
| weshay|rover | k | 16:59 |
| weshay|rover | l8r chkumar|pto | 16:59 |
| *** ccamacho has joined #oooq | 17:02 | |
| zbr | marios: weshay|rover do you know why we have the files section defined inside template twice (check and gate) instead of using it in the specific job definition? example scenario001. | 17:05 |
| zbr | seems a lot of duplication. are these also used with periodical? | 17:06 |
| zbr | this being the only reason that could explain it. | 17:06 |
| marios | zbr: you mean https://github.com/openstack-infra/tripleo-ci/blob/a646abc971e521ca4d44db940af51d13ded27237/zuul.d/standalone-jobs.yaml#L267 | 17:06 |
| marios | zbr: if we can define it once in the check and avoid it in the gate then sure we should remove it int he gate layout i thought they are indepenent. but otherwise yeah they should be same triggers right | 17:07 |
| marios | zbr: one thing we thought of is to use yaml refs as well since even across the scnarios they have some same files there | 17:08 |
| marios | zbr: but /me almost out today will check it more tomorrow lets fix that if its fixable | 17:08 |
| zbr | marios: exactly. mainly I can add the pattern to the job definition. i will make a poc change and see if it works. just wanted to ask. | 17:08 |
| marios | zbr: k | 17:09 |
| zbr | we have a LOT of repetition there | 17:09 |
| marios | zbr: yup | 17:09 |
| zbr | sadly I discovered recently that zuul is not able to expand all yaml anchors, only some of them. | 17:09 |
| marios | zbr: hm you might want to comment in https://review.openstack.org/#/c/639358/9 and the other patches in that topic | 17:11 |
| marios | theres a lot of yaml refs | 17:11 |
| *** ccamacho has quit IRC | 17:15 | |
| *** ccamacho has joined #oooq | 17:19 | |
| *** ykarel is now known as ykarel|away | 17:20 | |
| *** rlandy is now known as rlandy|brb | 17:21 | |
| rfolco|ruck | arxcruz, did you open a bug for this ? http://logs.openstack.org/59/641159/1/gate/tripleo-ci-centos-7-undercloud-containers/5155431/job-output.txt.gz#_2019-03-06_15_11_55_646483 | 17:25 |
| *** bogdando has quit IRC | 17:26 | |
| *** dsneddon has quit IRC | 17:33 | |
| *** kopecmartin is now known as kopecmartin|off | 17:33 | |
| weshay|rover | a extra cup of wine to the person who can point to where the project triggers are for fs001 | 17:37 |
| *** rlandy|brb is now known as rlandy | 17:38 | |
| rlandy | weshay|rover: what triggers do you need check or gate? | 17:39 |
| rlandy | or pipeline? | 17:39 |
| arxcruz | rfolco|ruck: nope | 17:39 |
| rfolco|ruck | arxcruz, I see multiple tempest issues | 17:40 |
| rfolco|ruck | in gate :( | 17:40 |
| weshay|rover | rlandy I want to trigger fs001 on another tripleo project | 17:40 |
| weshay|rover | but I'm lost in zuul atm | 17:41 |
| arxcruz | rfolco|ruck: i'm checking one regarding ipv6 | 17:41 |
| rfolco|ruck | arxcruz, that one I saw the bug, ok | 17:41 |
| arxcruz | rfolco|ruck: if you know others, let me know | 17:41 |
| weshay|rover | arxcruz rfolco|ruck let's get them added to the skip if we see more than one occurance of the specific test | 17:41 |
| weshay|rover | and take it from there | 17:41 |
| arxcruz | just finish the deploy to debug | 17:41 |
| rlandy | weshay|rover: so you may be looking for the templates ... https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/project-templates.yaml#L68 | 17:41 |
| rlandy | or see the branchless one | 17:41 |
| arxcruz | weshay|rover: rfolco|ruck if add on skip, please add with Related-Bug, not with Closes-Bug | 17:41 |
| rlandy | weshay|rover: like tht uses those | 17:42 |
| rfolco|ruck | arxcruz, like this one http://logs.openstack.org/21/640921/2/gate/tripleo-ci-centos-7-undercloud-containers/d97eeb1/logs/undercloud/home/zuul/tempest.log.txt.gz#_2019-03-06_17_05_19 | 17:42 |
| arxcruz | because closes bug closes the bug, even though the bug is still there | 17:42 |
| rlandy | or you may want something more specific like: | 17:42 |
| rfolco|ruck | weshay|rover, ^ will open a separate bug | 17:42 |
| arxcruz | rfolco|ruck: 2019-03-06 17:05:19 | Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. | 17:42 |
| weshay|rover | rlandy maybe I need to add it as a required project? | 17:42 |
| weshay|rover | rfolco|ruck ya.. each test has to be a seperate bug | 17:42 |
| rlandy | https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/tripleo.yaml#L340 | 17:42 |
| weshay|rover | or you will kill arx's skip list :) | 17:42 |
| rlandy | weshay|rover: required project is needed in zuul job definition | 17:43 |
| rlandy | not as a project trigger | 17:43 |
| rlandy | iiuc | 17:43 |
| arxcruz | rfolco|ruck: on cix call, we should ask for some nova guy to take a look | 17:43 |
| weshay|rover | rlandy ah.. ok.. thanks I think that's it | 17:43 |
| * weshay|rover will send a review up | 17:43 | |
| rlandy | most project use the templates | 17:43 |
| rlandy | weshay|rover: ^^ | 17:43 |
| weshay|rover | thanks | 17:43 |
| rfolco|ruck | arxcruz, hmm, let me make a quick root analysis first, thanks | 17:43 |
| weshay|rover | remember.. first put them in the skip | 17:44 |
| weshay|rover | then debug | 17:44 |
| arxcruz | and do not use closes-bug, uses related-bug | 17:45 |
| arxcruz | otherwise when the patch merge, it will mark the lp as fix released | 17:45 |
| rfolco|ruck | oh no, sova seems to be dead | 17:46 |
| rlandy | sshnaidm: just fyi in case you used https://rhos-ocp.infra.prod.upshift.eng.rdu2.redhat.com - removing instances there | 17:50 |
| sshnaidm | rlandy, sure | 17:50 |
| rlandy | our tenant is moved to https://rhos-d.infra.prod.upshift.rdu2.redhat.com | 17:50 |
| sshnaidm | ack | 17:50 |
| weshay|rover | rlandy panda|ruck|off fyi https://review.rdoproject.org/r/19139 | 17:51 |
| rlandy | syntax error | 17:51 |
| weshay|rover | rfolco|ruck the rdo admins are rebooting stuff | 17:51 |
| sshnaidm | rlandy, did all users/pass/tenant-id change? | 17:52 |
| rlandy | sshnaidm: you can log in with your kerb username/password | 17:52 |
| rlandy | but the service user we need for ci was not connected | 17:52 |
| rfolco|ruck | weshay|rover, ack | 17:52 |
| rlandy | kforde is putting that user back for us | 17:52 |
| rlandy | I am setting up the osp jobs there again | 17:52 |
| rlandy | ovb | 17:52 |
| sshnaidm | rlandy, ok, I'd like to use clouds.yaml still | 17:52 |
| rlandy | sshnaidm: is that a problem? | 17:53 |
| rlandy | you should be able to | 17:53 |
| sshnaidm | rlandy, yeah, should be fine | 17:53 |
| rlandy | it's just the user for ci that is missing | 17:53 |
| rlandy | so we don;t spew our kerb passwords anywhere | 17:53 |
| weshay|rover | marios you still around? | 17:54 |
| weshay|rover | http://logs.openstack.org/48/641348/8/check/tripleo-build-containers-centos-7/fdd11d5/job-output.txt.gz#_2019-03-06_17_46_06_946869 | 17:54 |
| sshnaidm | rlandy, did ovb work for you there in repro? | 17:54 |
| weshay|rover | marios | 17:54 |
| weshay|rover | - name: Retrieve the images and retag with version hash if periodic | 17:54 |
| weshay|rover | when: | 17:54 |
| weshay|rover | - zuul is defined | 17:54 |
| weshay|rover | - "'periodic' in zuul.pipeline" | 17:54 |
| weshay|rover | works ^ | 17:54 |
| rlandy | sshnaidm: will let you know :) | 17:54 |
| * weshay|rover updates review shorty | 17:54 | |
| sshnaidm | rlandy, ok) | 17:54 |
| weshay|rover | shortly | 17:54 |
| rlandy | just got started again | 17:54 |
| weshay|rover | marios updated the review | 17:57 |
| zbr | weshay|rover: is this what you asked for about f28 scenarios https://review.openstack.org/#/c/641447/ ? | 17:58 |
| weshay|rover | I'm looking | 17:59 |
| weshay|rover | zbr couple questions | 17:59 |
| weshay|rover | why are you changing centos jobs? | 17:59 |
| weshay|rover | I see | 18:00 |
| weshay|rover | to parent | 18:00 |
| zbr | weshay|rover: just moving the file sections which where duplicated in check and gate.... | 18:00 |
| *** ccamacho has quit IRC | 18:00 | |
| zbr | i can split the change, and fix the duplication first. | 18:00 |
| weshay|rover | zbr we need the jobs defined in the promotion pipeline | 18:00 |
| zbr | ouch... | 18:00 |
| weshay|rover | zbr you can leave what you've done | 18:01 |
| weshay|rover | but ya.. wrong placve | 18:01 |
| weshay|rover | palce | 18:01 |
| weshay|rover | place even | 18:01 |
| weshay|rover | zbr let's add a user story.. so it's clear | 18:02 |
| weshay|rover | zbr read the epic, I'll get started on the us | 18:02 |
| zbr | yep, user story would clearly help, as we can link to it. | 18:02 |
| rlandy | weshay|rover: 1-on-1 or skipping this week? | 18:03 |
| *** dsneddon has joined #oooq | 18:03 | |
| *** derekh has quit IRC | 18:03 | |
| weshay|rover | rlandy I'll ping u | 18:04 |
| *** jtomasek has quit IRC | 18:06 | |
| *** irclogbot_0 has joined #oooq | 18:10 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, (5 more messages) | 18:14 |
| weshay|rover | rlandy k.. ready | 18:17 |
| weshay|rover | sorry for the delay | 18:17 |
| rlandy | k - joining | 18:18 |
| *** jpena is now known as jpena|off | 18:19 | |
| *** amoralej is now known as amoralej|off | 18:26 | |
| *** irclogbot_0 has quit IRC | 18:36 | |
| *** holser_ has quit IRC | 18:45 | |
| sshnaidm | weshay|rover, rfolco|ruck sova is great again | 18:51 |
| rfolco|ruck | sshnaidm, thx sova's trump | 18:51 |
| weshay|rover | thanks! | 18:55 |
| sshnaidm | weshay|rover, can you re-review please? https://review.openstack.org/#/c/639670/ | 19:10 |
| *** ykarel_ has joined #oooq | 19:20 | |
| *** ykarel|away has quit IRC | 19:22 | |
| rlandy | weshay|rover: ok to merge https://review.rdoproject.org/r/#/c/19139/? | 19:23 |
| *** rfolco|ruck has quit IRC | 19:26 | |
| *** rfolco has joined #oooq | 19:27 | |
| weshay|rover | rlandy let er rip | 19:42 |
| rlandy | if you say so | 19:42 |
| rlandy | weshay|rover: fyi - trying all the playbooks - https://code.engineering.redhat.com/gerrit/#/c/164700/1/zuul.d/jobs.yaml - let's see what explodes | 19:53 |
| rlandy | now adding standalone | 19:53 |
| fultonj | which version of ansible should i be writing changes to tripleo-ci in? | 19:53 |
| fultonj | e.g. should i use with_toegher or loop ? https://docs.ansible.com/ansible/latest/user_guide/playbooks_loops.html#with-together | 19:54 |
| * fultonj guesses with_together and expects to find answer from CI for the CI | 19:58 | |
| *** tosky has quit IRC | 20:00 | |
| *** tosky has joined #oooq | 20:01 | |
| weshay|rover | fultonj you can see in the requirements in tripleo-quickstart | 20:01 |
| zbr | fultonj: 2.5/2.6 to be on the safe side. | 20:02 |
| weshay|rover | https://github.com/openstack/tripleo-quickstart/blob/master/requirements.txt | 20:02 |
| fultonj | thanks | 20:02 |
| zbr | fultonj: there is work on making zuul able to work with multiple versions of ansible, but is only a spec at the moment. | 20:03 |
| fultonj | zbr: that's cool | 20:04 |
| weshay|rover | oh my.. panda|ruck|off check this out.. http://logs.rdoproject.org/17/641417/2/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035/37cf680/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz | 20:09 |
| weshay|rover | look at | Waiting for messages on queue 'tripleo' with no timeout. | 20:10 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, (4 more messages) | 20:14 |
| weshay|rover | rlandy please review in your time https://review.rdoproject.org/r/#/c/19129/ | 20:15 |
| * rlandy looks | 20:16 | |
| rlandy | weshay|rover: two jobs running in https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status now - one testing full overcloud deploy on bm and one testing centos standalone (to see if standlone runs ok here) - then can switch to rhel8 nodes | 20:22 |
| weshay|rover | rlandy you are going to have to update the dns entries for the internal jobs | 21:04 |
| weshay|rover | https://sf.hosted.upshift.rdu2.redhat.com/logs/03/164703/1/check/tripleo-ci-standalone-upshift/ae23b1e/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-03-06_20_31_07 | 21:04 |
| weshay|rover | rlandy I think you need to add an internal dns server and I would also add 1.1.1.1 | 21:04 |
| weshay|rover | if that is not blocked internally | 21:05 |
| weshay|rover | but that that is what you hit there | 21:05 |
| weshay|rover | should be env updates | 21:05 |
| *** zbr|ssbarnea has joined #oooq | 21:07 | |
| *** irclogbot_0 has joined #oooq | 21:07 | |
| *** zbr has quit IRC | 21:10 | |
| rlandy | thanks | 21:25 |
| rlandy | weshay|rover: bm passed introspection ... https://sf.hosted.upshift.rdu2.redhat.com/logs/00/164700/1/check/periodic-tripleo-ci-centos-7-baremetal-3ctlr_1comp-featureset001-master/43ca856/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz | 21:27 |
| *** irclogbot_0 has quit IRC | 21:28 | |
| weshay|rover | ah.. rlandy I may need that as proof | 21:28 |
| rlandy | weshay|rover: well there it is in blank and white | 21:28 |
| weshay|rover | rlandy check out how messed up this is | 21:28 |
| weshay|rover | http://logs.rdoproject.org/17/641417/2/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035/37cf680/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz | 21:28 |
| weshay|rover | rlandy search for openstack overcloud node provide --all-manageable | 21:29 |
| rlandy | 2019-03-06 16:29:50 | + sudo kill 74415 | 21:32 |
| rlandy | 2019-03-06 16:29:50 | 1694 packets captured | 21:32 |
| rlandy | 2019-03-06 16:29:50 | 1785 packets received by filter | 21:32 |
| rlandy | 2019-03-06 16:29:50 | 43 packets dropped by kernel | 21:32 |
| rlandy | deploy failure https://sf.hosted.upshift.rdu2.redhat.com/logs/00/164700/1/check/periodic-tripleo-ci-centos-7-baremetal-3ctlr_1comp-featureset001-master/43ca856/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2019-03-06_20_45_43 | 21:33 |
| rlandy | dns and ntp | 21:36 |
| weshay|rover | rlandy do you have an ovb env that is up? | 21:47 |
| rlandy | weshay|rover: no - only the bm that stays up | 21:49 |
| rlandy | weshay|rover: can run one on my tenant | 21:49 |
| weshay|rover | rlandy can you run it through building images ? | 21:49 |
| weshay|rover | rlandy /me working w/ ianw in #rdo re the image building issues | 21:50 |
| rlandy | bm doesn't build - setting up ovb | 21:51 |
| *** fmount has quit IRC | 22:03 | |
| *** fmount has joined #oooq | 22:04 | |
| weshay|rover | rlandy re: http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/toci-quickstart/config/testenv/ovb.yml#n39 | 22:05 |
| weshay|rover | is that the right env file for rdo-ovb? | 22:05 |
| weshay|rover | seems like we're not wanting to build images in 3rd party | 22:05 |
| rlandy | the file is correct | 22:06 |
| rlandy | but we are building images | 22:07 |
| rlandy | there are more settings than that | 22:07 |
| weshay|rover | ug | 22:07 |
| rlandy | http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/toci-quickstart/config/testenv/ovb-rdocloud.yml | 22:08 |
| rlandy | but that doesn't say anything about building images | 22:08 |
| *** apetrich has quit IRC | 22:08 | |
| weshay|rover | rlandy right.. we have a diff there | 22:08 |
| weshay|rover | for that setting | 22:08 |
| rlandy | roles: | 22:10 |
| rlandy | - {role: fetch-images, | 22:11 |
| rlandy | when: not to_build|bool} | 22:11 |
| *** apetrich has joined #oooq | 22:11 | |
| * weshay|rover looks.. I think to_build is a fact | 22:11 | |
| weshay|rover | iirc | 22:11 |
| * rlandy gets local env started | 22:12 | |
| weshay|rover | so.. I know for sure we want to build the images in periodic http://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/tree/playbooks/to-build-or-not-to-build.yml#n49 | 22:12 |
| weshay|rover | I'm not 100% sure if that is the case for check | 22:12 |
| weshay|rover | I think you, me and sshnaidm would have to try to recall | 22:13 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010 (4 more messages) | 22:14 |
| sshnaidm | weshay|rover, we don't build images in check jobs except tripleo-common repo | 22:14 |
| sshnaidm | and all in default_projects_need_build_list | 22:14 |
| weshay|rover | sshnaidm I wonder if that got messed up | 22:15 |
| sshnaidm | weshay|rover, why? | 22:15 |
| rlandy | sshnaidm: is it doc'ed anywhere hos to hold nodes in reproducer? | 22:15 |
| weshay|rover | sshnaidm you don't happen to have an ovb env up do you? | 22:15 |
| rlandy | how | 22:15 |
| weshay|rover | ian is wondering if /tmp is too small | 22:15 |
| weshay|rover | sshnaidm /me trying to get a handle around https://bugs.launchpad.net/tripleo/+bug/1818305 | 22:15 |
| openstack | Launchpad bug 1818305 in tripleo "overcloud-full image fails to build calling mkfs -t xfs, exec sudo failed" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 22:15 |
| weshay|rover | sshnaidm building on a libvirt guest works fine, several times | 22:16 |
| sshnaidm | rlandy, ovb nodes or nodepool node? | 22:16 |
| rlandy | sshnaidm: both | 22:16 |
| sshnaidm | rlandy, not sure it's documented | 22:16 |
| weshay|rover | rlandy Gabriele had a patch on that | 22:16 |
| rlandy | zuul autohold ? | 22:16 |
| * rlandy starts job | 22:16 | |
| weshay|rover | no.. it was a hack to do it | 22:16 |
| sshnaidm | rlandy, docker-compose exec scheduler zuul autohold --tenant tripleo-ci-reproducer --job tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039 --reason debug --project test1 | 22:17 |
| sshnaidm | rlandy, just change the job name | 22:17 |
| weshay|rover | sshnaidm search in fs001 in sova.. for "overcloud image create" | 22:17 |
| sshnaidm | rlandy, but job needs to fail to make autohold to work | 22:17 |
| weshay|rover | rdo is a mess right now | 22:18 |
| rlandy | k - setting up | 22:18 |
| rlandy | then will hols | 22:18 |
| rlandy | hold | 22:18 |
| sshnaidm | weshay|rover, what is the problem though? | 22:19 |
| sshnaidm | weshay|rover, does image build where it shouldn't? | 22:19 |
| weshay|rover | it fails when it tries to build | 22:19 |
| *** irclogbot_0 has joined #oooq | 22:19 | |
| weshay|rover | jobs pass and sometimes fail's when it's skipped | 22:20 |
| weshay|rover | sshnaidm you see the trace in the bug? | 22:20 |
| weshay|rover | http://paste.openstack.org/show/747378/ | 22:20 |
| weshay|rover | sudo mkfs -t xfs -s size=4096 -L img-rootfs -m uuid=8fac897b-261b-4d0b-9364-52776b5f2616 -q /dev/loop0] | 22:20 |
| weshay|rover | fails | 22:20 |
| sshnaidm | weshay|rover, how is it related to build_or_not_to_build? | 22:22 |
| weshay|rover | sshnaidm my question for you and rlandy was WHEN should we be building overcloud images in check | 22:23 |
| weshay|rover | so if it's just tripleo-common, let me see if that is the case | 22:23 |
| sshnaidm | weshay|rover, tripleo-common and other repos from here: http://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/tree/playbooks/to-build-or-not-to-build.yml#n49 | 22:24 |
| weshay|rover | ya.. so far it does just look like tripleo-common | 22:24 |
| weshay|rover | I see one python-tripleoclient | 22:24 |
| weshay|rover | ya.. seems like those two repos which makes sense | 22:26 |
| weshay|rover | but does not explain WHY it's failing | 22:26 |
| sshnaidm | weshay|rover, worth to wait until rdo cloud is ok | 22:30 |
| weshay|rover | sshnaidm ya.. that actually is good advice | 22:31 |
| weshay|rover | sshnaidm thanks for popping on late | 22:31 |
| sshnaidm | np | 22:31 |
| *** irclogbot_0 has quit IRC | 22:34 | |
| *** irclogbot_0 has joined #oooq | 22:35 | |
| *** vinaykns has quit IRC | 22:36 | |
| rlandy | weshay|rover: hmmm reproducer ovb does not launch - you are correct | 22:37 |
| weshay|rover | rlandy I'm trying one now as well | 22:38 |
| rlandy | weshay|rover: check the depends on in the zuul.yaml | 22:38 |
| rlandy | if it is a periodic - that would need to be fixed | 22:38 |
| rlandy | I fixed mine but it still does not launch | 22:38 |
| weshay|rover | rfolco fyi ^ | 22:39 |
| rlandy | weshay|rover: oh - it did something now | 22:41 |
| weshay|rover | since i have you out of context https://review.openstack.org/#/c/640538/ | 22:41 |
| weshay|rover | rlandy OH | 22:41 |
| rlandy | k - will trust you on this one | 22:42 |
| weshay|rover | I ran a check ovb job | 22:42 |
| weshay|rover | you ran a periodic? | 22:42 |
| rlandy | I did | 22:43 |
| rlandy | keeps trying to launch and fails | 22:43 |
| rlandy | what did you try | 22:43 |
| * rlandy will edit launch file | 22:43 | |
| *** irclogbot_0 has quit IRC | 22:44 | |
| rlandy | trying non-periodic now | 22:45 |
| rlandy | maybe can't get node from rod-cloud | 22:46 |
| * rlandy needs to set up ovb on internal zuul | 22:47 | |
| weshay|rover | mine triggered | 22:49 |
| rlandy | how nice | 22:49 |
| rlandy | mine doesn't | 22:49 |
| rlandy | weshay|rover: maybe because you can use quota from ci | 22:50 |
| weshay|rover | rlandy got a node failure | 22:51 |
| weshay|rover | no.. I have change the tenant | 22:51 |
| rlandy | mine never launches | 22:53 |
| rlandy | probably no nodes | 22:53 |
| rlandy | weshay|rover: ok - I will work on setting up ovb on downstream so we have that alternative | 22:53 |
| weshay|rover | rlandy k.. cool would be nice to compare | 22:55 |
| rlandy | this is not workable | 22:58 |
| weshay|rover | ok.. my nodepool node is up | 23:01 |
| weshay|rover | and job is running | 23:01 |
| weshay|rover | I'm on the node | 23:01 |
| weshay|rover | rlandy which bit? | 23:01 |
| weshay|rover | is not workable? | 23:01 |
| rlandy | not having an nodes - but you got one | 23:02 |
| rlandy | good for you | 23:02 |
| rlandy | rechecking | 23:03 |
| weshay|rover | heh.. well my heat stack failed | 23:09 |
| weshay|rover | have to try again | 23:09 |
| weshay|rover | rdo is a piece | 23:09 |
| rlandy | wow - bad day for rdo cloud | 23:10 |
| *** rlandy is now known as rlandy|bbl | 23:24 | |
| *** rascasoft has quit IRC | 23:34 | |
| *** rascasoft has joined #oooq | 23:37 | |
| weshay|rover | TASK [ovb-manage : Find out UUID of instance with metadata URL] | 23:39 |
| weshay|rover | 2019-03-06 23:36:16.964720 | primary | Traceback (most recent call last): | 23:39 |
| weshay|rover | 2019-03-06 23:36:16.964859 | primary | File "<string>", line 1, in <module> | 23:39 |
| weshay|rover | 2019-03-06 23:36:16.964972 | primary | File "/usr/lib64/python2.7/json/__init__.py", line 290, in load | 23:39 |
| weshay|rover | 2019-03-06 23:36:16.965123 | primary | **kw) | 23:39 |
| weshay|rover | 2019-03-06 23:36:16.965245 | primary | File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads | 23:39 |
| weshay|rover | 2019-03-06 23:36:16.965308 | primary | return _default_decoder.decode(s) | 23:39 |
| weshay|rover | 2019-03-06 23:36:16.965406 | primary | File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode | 23:39 |
| weshay|rover | 2019-03-06 23:36:16.965674 | primary | obj, end = self.raw_decode(s, idx=_w(s, 0).end()) | 23:39 |
| weshay|rover | 2019-03-06 23:36:16.965784 | primary | File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode | 23:40 |
| weshay|rover | 2019-03-06 23:36:16.965877 | primary | raise ValueError("No JSON object could be decoded") | 23:40 |
| weshay|rover | 2019-03-06 23:36:16.965947 | primary | ValueError: No JSON object could be decoded | 23:40 |
| weshay|rover | NO STACKS FOR U | 23:41 |
| weshay|rover | I now feel good about my libvirt requirement :) /me looking at quiquell|off | 23:42 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!