| *** dsneddon has joined #oooq | 00:05 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, (3 more messages) | 00:19 |
|---|---|---|
| *** rlandy has quit IRC | 01:16 | |
| *** agopi has joined #oooq | 01:24 | |
| *** zbr has quit IRC | 01:34 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, (3 more messages) | 02:19 |
| *** skramaja has joined #oooq | 02:50 | |
| *** apetrich has quit IRC | 03:16 | |
| *** dsneddon has quit IRC | 03:43 | |
| *** dsneddon has joined #oooq | 04:08 | |
| *** dsneddon has quit IRC | 04:12 | |
| *** udesale has joined #oooq | 04:17 | |
| *** rfolco|ruck has quit IRC | 04:17 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo- (3 more messages) | 04:19 |
| *** ykarel|away has joined #oooq | 04:22 | |
| *** rfolco has joined #oooq | 04:24 | |
| *** rfolco has quit IRC | 04:30 | |
| *** rfolco has joined #oooq | 04:30 | |
| *** dsneddon has joined #oooq | 04:41 | |
| *** dsneddon has quit IRC | 04:45 | |
| *** dsneddon has joined #oooq | 05:11 | |
| *** dsneddon has quit IRC | 05:15 | |
| *** udesale has quit IRC | 05:16 | |
| *** udesale has joined #oooq | 05:17 | |
| *** jaosorior has joined #oooq | 05:22 | |
| *** dsneddon has joined #oooq | 05:26 | |
| *** dsneddon has quit IRC | 05:31 | |
| *** udesale has quit IRC | 05:51 | |
| *** dsneddon has joined #oooq | 06:05 | |
| *** dsneddon has quit IRC | 06:10 | |
| *** udesale has joined #oooq | 06:11 | |
| *** ykarel|away is now known as ykarel | 06:15 | |
| *** saneax has joined #oooq | 06:15 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo- (3 more messages) | 06:19 |
| *** dsneddon has joined #oooq | 06:20 | |
| *** dsneddon has quit IRC | 06:25 | |
| *** yolanda has quit IRC | 06:36 | |
| *** saneax has quit IRC | 06:53 | |
| *** saneax has joined #oooq | 06:53 | |
| *** dsneddon has joined #oooq | 06:58 | |
| *** kopecmartin|off is now known as kopecmartin | 07:07 | |
| *** yolanda has joined #oooq | 07:09 | |
| *** dsneddon has quit IRC | 07:11 | |
| *** rascasoft has joined #oooq | 07:36 | |
| *** dsneddon has joined #oooq | 07:45 | |
| *** gkadam has joined #oooq | 07:48 | |
| *** dsneddon has quit IRC | 07:50 | |
| *** apetrich has joined #oooq | 07:50 | |
| *** dsneddon has joined #oooq | 07:55 | |
| *** matbu has joined #oooq | 07:59 | |
| *** sshnaidm|afk has quit IRC | 07:59 | |
| *** jfrancoa has joined #oooq | 08:10 | |
| *** amoralej|off is now known as amoralej | 08:12 | |
| *** holser_ has joined #oooq | 08:15 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010 (3 more messages) | 08:19 |
| *** ccamacho has joined #oooq | 08:23 | |
| chandankumar | marios: Good morning :-) | 08:34 |
| chandankumar | marios: I just returned from this place https://en.wikipedia.org/wiki/Rajgir | 08:34 |
| marios | chandankumar: welcome back looks nice | 08:35 |
| chandankumar | marios: done with too much trekking this month!, need to plan for next month. | 08:37 |
| *** chem has joined #oooq | 08:48 | |
| *** jpena|off is now known as jpena | 08:49 | |
| *** bogdando has joined #oooq | 09:01 | |
| *** zbr has joined #oooq | 09:05 | |
| *** matbu has quit IRC | 09:10 | |
| *** dtantsur|afk is now known as dtantsur | 09:14 | |
| *** panda|rover|off is now known as panda|rover | 09:17 | |
| zbr | does anyome knows reason for No package tmpwatch available. ? | 09:20 |
| panda|rover | zbr: context ? | 09:26 |
| zbr | i think it was networking/mirror, i seen it in few builds but not in last. | 09:27 |
| zbr | panda|rover: see http://logs.openstack.org/17/642517/2/check/tripleo-ci-fedora-28-standalone/cea6a01/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-03-12_19_09_44 | 09:27 |
| zbr | panda|rover: take a look http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22No%20package%20tmpwatch%20available.%5C%22 -- se window to 48 hours | 09:28 |
| zbr | sadly the error is very opaque and we have no idea why it did not find it. | 09:29 |
| zbr | arxcruz: are you here? can we look at another tempest error? | 09:30 |
| *** rfolco is now known as rfolco|ruck | 09:32 | |
| *** holser_ has quit IRC | 09:32 | |
| *** derekh has joined #oooq | 09:35 | |
| *** sshnaidm|afk has joined #oooq | 09:36 | |
| panda|rover | ykarel: whay are we still checking ocata packages ? for the extended maintenance ? | 09:41 |
| ykarel | panda|rover, yes | 09:41 |
| ykarel | packages are still merging | 09:42 |
| ykarel | like https://trunk.rdoproject.org/centos7-queens/report.html | 09:42 |
| arxcruz | zbr: sure.. | 09:43 |
| zbr | arxcruz: https://bluejeans.com/2655417928 | 09:45 |
| arxcruz | zbr: i'm the only one there | 09:46 |
| arxcruz | chandankumar: https://github.com/openstack/python-tempestconf/commit/72f0edffb0ecb899e278137102915e58b7ddbe23 | 09:51 |
| arxcruz | chandankumar: we are seeing this error on fedora 28 job | 09:51 |
| arxcruz | wasn't it supposed to have a rpm to fedora as well ? | 09:51 |
| chandankumar | arxcruz: job link | 09:52 |
| arxcruz | chandankumar: http://logs.openstack.org/17/642517/3/check/tripleo-ci-fedora-28-standalone/5a671ae/logs/undercloud/home/zuul/tempest/tempest.html.gz | 09:52 |
| chandankumar | i mean log link | 09:52 |
| chandankumar | arxcruz: https://github.com/openstack/python-tempestconf/commit/f9b3c05 -> is installed on fedora job | 09:53 |
| chandankumar | arxcruz: above commit is not there | 09:53 |
| arxcruz | chandankumar: we need to update the package | 09:54 |
| chandankumar | arxcruz: we need to update the promotion hash for fedora 28 | 09:54 |
| chandankumar | I thought fedora 28 was automated? or is it still pinned at particular hash? | 09:54 |
| arxcruz | ¯\_(ツ)_/¯ | 09:54 |
| *** sshnaidm|afk has quit IRC | 09:56 | |
| panda|rover | rfolco|ruck: how are periodics ? | 09:58 |
| panda|rover | rfolco|ruck: program meeting today is at 12UTC | 09:59 |
| *** matbu has joined #oooq | 10:02 | |
| *** saneax has quit IRC | 10:07 | |
| zbr | panda|rover: rfolco|ruck FYI: https://bugs.launchpad.net/tripleo/+bug/1819877 | 10:07 |
| openstack | Launchpad bug 1819877 in tripleo "f28: tempest failure due to outdated python3-tempestconf" [High,Triaged] - Assigned to Sorin Sbarnea (ssbarnea) | 10:07 |
| rfolco|ruck | panda|rover, last time periodics were failing to build containers, need to check | 10:08 |
| rfolco|ruck | panda|rover, ok I am updating prog call doc | 10:08 |
| panda|rover | rfolco|ruck: see if weshay didn't already | 10:10 |
| chandankumar | zbr: panda|rover rfolco|ruck https://bugs.launchpad.net/tripleo/+bug/1819877 regarding this hash I think we need to update the f28 dlrn hash I am not why one package does not gets udpated as https://github.com/openstack/tempest/commit/bf97151 tempest is also very old | 10:10 |
| openstack | Launchpad bug 1819877 in tripleo "f28: tempest failure due to outdated python3-tempestconf" [High,Triaged] - Assigned to Sorin Sbarnea (ssbarnea) | 10:10 |
| rfolco|ruck | panda|rover, no we made a draft last night and noew I am updating it | 10:10 |
| chandankumar | *not sure | 10:11 |
| panda|rover | rfolco|ruck: ok | 10:12 |
| rfolco|ruck | panda|rover, hmm rocky just worked last run in periodic | 10:15 |
| chandankumar | arxcruz: regarding this bug https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-rocky-rdo_trunk-minimal-118/console.txt.gz | 10:17 |
| chandankumar | arxcruz: https://bugs.launchpad.net/tripleo/+bug/1819440 | 10:17 |
| openstack | Launchpad bug 1819440 in tripleo "phase 1 failing on No such file or directory: '/home/stack/tempest'" [Critical,In progress] - Assigned to Arx Cruz (arxcruz) | 10:17 |
| zbr | i did a "tripleo-repos current" on my f28 test machine and now I can see that the right version of python3-tempestconf gets in. | 10:17 |
| arxcruz | chandankumar: yes? | 10:17 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010 (3 more messages) | 10:19 |
| chandankumar | arxcruz: i think there is a permission issue the directory 2019-03-09 19:12:50.790 | Could not find or access '/var/lib/tempestdata/tempest' | 10:19 |
| panda|rover | again ? | 10:19 |
| arxcruz | chandankumar: yes... | 10:19 |
| arxcruz | chandankumar: problem is i can't reproduce it but i'm taking care of it | 10:20 |
| arxcruz | chandankumar: i found a lot of more issues | 10:20 |
| chandankumar | arxcruz: like? | 10:22 |
| arxcruz | chandankumar: if you're running standalone job | 10:23 |
| arxcruz | you can't run containerized tempest | 10:23 |
| arxcruz | because the logic there is messed | 10:23 |
| zbr | panda|rover: yep, i found the fedora28 rpm from @delorean is too old on http://logs.openstack.org/17/642517/3/check/tripleo-ci-fedora-28-standalone/5a671ae/logs/undercloud/var/log/extra/package-list-installed.txt.gz | 10:23 |
| arxcruz | also, the patch you send, wasn't being executed because we don't have one single job running containerized upstream with standalone | 10:23 |
| chandankumar | arxcruz: yes, it is known, that's why we are running using package, as it requires to copy bunch of more files and we need to fix the routing of os-cloud api endpoint i think | 10:24 |
| arxcruz | chandankumar: nope, pretty easy, i just fix in my local machine, and i have standalone running containerized | 10:24 |
| arxcruz | well, not pretty easy, i had to comment a lot of the wrong logic there | 10:25 |
| chandankumar | arxcruz: ah, it is, cool! | 10:25 |
| chandankumar | let me find the review where we switch to package for that | 10:25 |
| arxcruz | but just need to copy the .config file to the container and export OS_CLOUD | 10:25 |
| arxcruz | chandankumar: http://paste.openstack.org/show/747697/ | 10:25 |
| arxcruz | permission issue? | 10:26 |
| chandankumar | arxcruz: yes | 10:26 |
| chandankumar | arxcruz: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/tasks/tempest-results.yml#L5 might be this part is not working | 10:26 |
| arxcruz | chandankumar: check the paste bin i show you | 10:26 |
| arxcruz | it's changing the permission | 10:26 |
| arxcruz | and i did a manual run on ansible file module, and recurse is working properly | 10:27 |
| chandankumar | arxcruz: it might be due to this | 10:31 |
| chandankumar | arxcruz: https://review.openstack.org/#/c/605980/ | 10:31 |
| chandankumar | arxcruz: I think this change never backported to rocky release | 10:31 |
| arxcruz | panda|rover: https://ci.centos.org/job/tripleo-quickstart-promote-rocky-rdo_trunk-minimal/119/console | 10:31 |
| arxcruz | it's failing, but still running, how can we hold this machine ? | 10:31 |
| arxcruz | i really want to see the permissions for that directory | 10:32 |
| chandankumar | arxcruz: will i try and backport that patch and let's see what happens? | 10:34 |
| arxcruz | chandankumar: so, it's like the directory /var/lib/tempestdata doesn't exist? | 10:35 |
| chandankumar | arxcruz: before container run it was not there | 10:37 |
| chandankumar | arxcruz: https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-rocky-rdo_trunk-minimal-118/undercloud/home/stack/tempest.log.gz | 10:37 |
| chandankumar | arxcruz: 2019-03-09 18:55:52 | + '[' '!' -d /var/lib/tempestdata ']' | 10:37 |
| chandankumar | 2019-03-09 18:55:52 | + sudo mkdir -p /var/lib/tempestdata | 10:37 |
| chandankumar | as per this "secontext": "unconfined_u:object_r:var_lib_t:s0", it might be related to selinux issue so in master we created that volume and mounted that especially with tempest container using tht | 10:40 |
| chandankumar | with proper flag | 10:40 |
| rfolco|ruck | panda|rover, weird this didn't happen today https://bugs.launchpad.net/tripleo/+bug/1819766 | 10:41 |
| openstack | Launchpad bug 1819766 in tripleo "containers build fails to build ironic-inspector and octavia-base" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 10:41 |
| panda|rover | rfolco|ruck: why weird ? | 10:43 |
| rfolco|ruck | panda|rover, why containers failed to build and now just works ? | 10:43 |
| rfolco|ruck | panda|rover, magic? | 10:44 |
| *** dsneddon has quit IRC | 10:44 | |
| panda|rover | rfolco|ruck: ever heard of the butterfly effect ? this happens all the time, people fix things somewhere without knowing there's a bug | 10:49 |
| arxcruz | chandankumar: no need to backport, just create this directory in tempest-containers.yml | 10:51 |
| panda|rover | rfolco|ruck: maybe try to find out what was merged that fixed it | 10:51 |
| *** holser_ has joined #oooq | 11:01 | |
| rfolco|ruck | panda|rover, do you understand the conclusion of https://bugs.launchpad.net/tripleo/+bug/1819452 ? | 11:15 |
| openstack | Launchpad bug 1819452 in tripleo "f28: tempest fails Exhausted all hosts available for retrying build failures, due to hw_machine_type=x86_64=pc-i440fx-rhel7.6.0" [High,Triaged] | 11:15 |
| panda|rover | rfolco|ruck: maybe zbr: can explain a bit more | 11:16 |
| *** dsneddon has joined #oooq | 11:16 | |
| *** tosky has joined #oooq | 11:17 | |
| zbr | rfolco|ruck: yes, is a WIP which will take some time to fix: https://review.openstack.org/#/q/topic:nova-arch | 11:20 |
| zbr | we really need at least one of them merged to continue on f28. the rest are improvments, not blockers. | 11:20 |
| zbr | panda|rover: maybe you can help workflowing https://review.openstack.org/#/c/642517/ ? | 11:21 |
| zbr | is the least risky of all changes: changing only our default to "pc", no side effects on centos, and fixing f28. | 11:21 |
| *** dsneddon has quit IRC | 11:21 | |
| zbr | long run, we want to all use "q35" everywhere, probably removing our overrie. it was reported the q35 is better than pc, being newer. | 11:22 |
| *** gkadam_ has joined #oooq | 11:24 | |
| *** jpena is now known as jpena|lunch | 11:25 | |
| *** gkadam has quit IRC | 11:25 | |
| rfolco|ruck | zbr, this isn't machine type, which is a libvirt thing to get discovered ? why do we need to enforce it ? | 11:26 |
| *** sshnaidm|afk has joined #oooq | 11:28 | |
| zbr | rfolco|ruck: because you have multiple architectures, even on the same host, so you need to have a default to use. | 11:30 |
| zbr | i guess that the default could be detected at runtime but is likely to be a costly decision which may not make QE happy as causing unexpected behaviours in some cases. | 11:31 |
| rfolco|ruck | zbr, :) | 11:31 |
| zbr | i guess they preferred to contain the situation | 11:31 |
| zbr | probably for the same reason they added the hardcoded version in the first place, at that time nobody really considered the multi-distro as a reality. | 11:32 |
| chandankumar | sshnaidm|afk: Hello | 11:36 |
| chandankumar | sshnaidm|afk: can we remove wip for nova join tempest tests patch https://review.openstack.org/#/c/639324/ ? | 11:36 |
| chandankumar | arxcruz: where can I put this task https://review.openstack.org/#/c/642948/ os_tempest tripleo voting in taiga? | 11:38 |
| rfolco|ruck | panda|rover, fs035 still failing on pacemaker bug... https://bugs.launchpad.net/tripleo/+bug/1818994 | 11:40 |
| openstack | Launchpad bug 1818994 in tripleo "ovb jobs broken because pacemaker is unconfigured" [Critical,Triaged] - Assigned to Juan Antonio Osorio Robles (juan-osorio-robles) | 11:40 |
| panda|rover | rfolco|ruck: ok, let's focus on that then. It's still a mistral containers missing the tripleo-common ? | 11:40 |
| rfolco|ruck | panda|rover, see comments from ykarel... in spite of having the latest tripleo-common, this is not used | 11:41 |
| rfolco|ruck | the one used is from mistral container, which does NOT have the fix | 11:41 |
| marios | rfolco|ruck: panda|rover looks like ovb jobs are pulling with tripleo-ci-testing and not the hash | 11:41 |
| marios | so that might be related to what you're seeing? | 11:41 |
| rfolco|ruck | marios, I thought the tag chat was unrelated | 11:42 |
| marios | panda|rover: rfolco|ruck ykarel https://bugs.launchpad.net/tripleo/+bug/1819583 | 11:42 |
| openstack | Launchpad bug 1819583 in tripleo "periodic-... containers-build-push job skips retagging with tripleo-ci-testing" [Low,Opinion] - Assigned to Marios Andreou (marios-b) | 11:42 |
| marios | comment 4 | 11:42 |
| marios | rfolco|ruck: ah ok i thought you were saying that it is pulling an older version of some container without the fix (didn't catch/follow the rest ofyour chat) | 11:42 |
| marios | ykarel: panda|rover so anyway for that bug we need to fix the ovb jobs to use the hash instead of tripleo=ci-testing, rather than fixing the retag i think | 11:43 |
| panda|rover | ok first thing to understand is if kolla builds the container with the right tripleo-common package | 11:43 |
| panda|rover | marios: are you sure the ovb jobs are not using the tag ? | 11:43 |
| marios | panda|rover: well i checked and comented on the bug see pointer | 11:43 |
| marios | panda|rover: i found at least one which isn't | 11:43 |
| marios | periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/f5a872d/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 11:44 |
| panda|rover | well, yeah, so I think in the promotion-hash release file we probably specify the tripleo-ci-testing tag | 11:46 |
| marios | panda|rover: ack yeah this is what i was looking for. we should fix that, rather than adding retag just for these jobs | 11:47 |
| panda|rover | how were we retagging before ? | 11:48 |
| marios | panda|rover: build and push using hash, retag with tripleo-ci-testing | 11:48 |
| marios | but the retag didn't ever work because of yml vs yaml (that bug original descriptikon) | 11:49 |
| panda|rover | no I mean, in the previous method how come we had containers with two tags ? | 11:50 |
| *** dsneddon has joined #oooq | 11:50 | |
| marios | panda|rover: not sure. in rdo kolla build they were doing the opposite. build and push with tripleo-ci-testing. then retag with hash | 11:50 |
| marios | panda|rover: and during the deploys the hash is used (well except ovb as above) | 11:50 |
| panda|rover | marios: make the ovb job use the hash may prove difficult, we sould need to override the dhln_hash_tag somewhere in the job, and not use tripleo-ci-testing | 11:50 |
| marios | panda|rover: why would it be difficult | 11:51 |
| *** dsneddon has quit IRC | 11:55 | |
| *** vkapalav has joined #oooq | 11:55 | |
| panda|rover | yeah, we were not even pushing the hash as a tag | 11:56 |
| marios | panda|rover: ? | 11:56 |
| panda|rover | we just tagged with tripleo-ci-testing | 11:56 |
| marios | panda|rover: where? | 11:56 |
| marios | panda|rover: the push jobs are pushing with the tag | 11:56 |
| panda|rover | marios: in the old method | 11:56 |
| marios | panda|rover: well there's definitely a docker push with the tripleo-ci-testing sec lemme find it | 11:56 |
| panda|rover | we didn't care about the hash, because the other jobs were using tripleo-ci-testing | 11:57 |
| panda|rover | marios: yes I have it in build-containers-images.sh | 11:57 |
| marios | https://github.com/rdo-infra/ansible-role-rdo-kolla-build/blob/master/tasks/tag.yml | 11:57 |
| marios | panda|rover: yeah but this is the retag with push for the version _hash | 11:57 |
| marios | https://github.com/rdo-infra/ansible-role-rdo-kolla-build/blob/master/tasks/main.yml#L92 | 11:57 |
| *** sshnaidm|afk is now known as sshnaidm | 11:57 | |
| sshnaidm | chandankumar, it still fails, need to make it work before removing wip | 11:58 |
| chandankumar | sshnaidm: ack! | 11:58 |
| panda|rover | marios: care to chat ? | 11:59 |
| marios | panda|rover: so one of the files under config/release/ right? is what i'm looking for | 11:59 |
| marios | panda|rover: http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-master.yml | 11:59 |
| panda|rover | marios: yes the ones that start with promotion- | 11:59 |
| marios | panda|rover: how about i update that? ^ | 11:59 |
| panda|rover | marios: yes | 11:59 |
| marios | panda|rover: and see it works? | 11:59 |
| panda|rover | marios: ok, with what you plan to update it ? | 12:00 |
| marios | dlrn_hash_tag: tripleo-ci-testing | 12:00 |
| marios | instead make | 12:00 |
| marios | like delorean_hash_label: &promotion-testing-tag "{{ dlrn_hash|default(dlrn_hash_tag) }}" | 12:00 |
| marios | panda|rover: or something :D | 12:00 |
| marios | ? | 12:00 |
| ykarel | dlrn_hash need to be calculated | 12:00 |
| marios | panda|rover: we have it looke like? | 12:00 |
| marios | ykarel: ^ | 12:00 |
| panda|rover | marios: what ykarel said | 12:00 |
| marios | http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-master.yml | 12:00 |
| *** jpena|lunch is now known as jpena | 12:00 | |
| panda|rover | marios: at this point is probably faster to make the containers build push and tag only with tripleo-ci-testing | 12:01 |
| panda|rover | marios: as it was happening before | 12:01 |
| panda|rover | marios: there was no retagging before, just single tagging | 12:01 |
| panda|rover | with tripleo-ci-testing | 12:01 |
| weshay | zbr I need a lot more detail re: https://bugs.launchpad.net/tripleo/+bug/1819877 .. if the python3-tempestconf is out of date w/ centos can you show the rpm diff in jobs | 12:01 |
| openstack | Launchpad bug 1819877 in tripleo "f28: tempest failure due to outdated python3-tempestconf" [High,Invalid] - Assigned to Sorin Sbarnea (ssbarnea) | 12:01 |
| marios | panda|rover: but what about https://github.com/rdo-infra/ansible-role-rdo-kolla-build/blob/master/tasks/main.yml#L92 https://github.com/rdo-infra/ansible-role-rdo-kolla-build/blob/master/tasks/tag.yml | 12:02 |
| weshay | zbr also you used a check job for a promotion blocker | 12:02 |
| marios | panda|rover: docker push with the version hash ? | 12:02 |
| ykarel | panda|rover, earlier both tripleo-ci-testing and version hash were pushed, some jobs rely on version_hash and some on tripleo-ci-testing, that's bad, but it was like that | 12:02 |
| panda|rover | marios: do you think that's a retag ? | 12:02 |
| marios | weshay: (see comment 4 @ https://bugs.launchpad.net/tripleo/+bug/1819583 ) is what we are discussing | 12:02 |
| openstack | Launchpad bug 1819583 in tripleo "periodic-... containers-build-push job skips retagging with tripleo-ci-testing" [Low,Opinion] - Assigned to Marios Andreou (marios-b) | 12:02 |
| weshay | zbr let me know what you are seeing and I'll help | 12:02 |
| panda|rover | ykarel: then it was the rdo kolla ansible role that it was retagging ? | 12:02 |
| marios | panda|rover: looks like tag and push to me but is that wrong? | 12:02 |
| ykarel | panda|rover, yes | 12:02 |
| weshay | marios k | 12:02 |
| weshay | I saw you reopened that as low | 12:03 |
| * weshay reads scrollback | 12:03 | |
| panda|rover | an yep because with line 84 kolla was called with pushed | 12:03 |
| marios | weshay: ovb using tripleo-ci-testing no the hash. so we either fix that (in oooq/config/release bla) or add retag | 12:03 |
| panda|rover | so yeas. we ARE retagging, and we rely on BOTH tripleo-ci-testing AND hash | 12:04 |
| weshay | ya.. the dlrn_hash alwasy needs to be calculated | 12:04 |
| marios | weshay: well it might be back to critical cos it blocks ovb jobs | 12:04 |
| weshay | panda|rover where do we rely on hash_tag ? | 12:04 |
| weshay | that should be looked at | 12:04 |
| marios | panda|rover: so sound like we need to retag with tripleo-ci-testing. or at least tagt with both tripleo-ci-testing and has_tag | 12:04 |
| marios | we can make that happen with https://review.openstack.org/#/c/642662 i mean latest version is removing the tagging but previous ones were trying to make it work | 12:04 |
| weshay | marios panda|rover can you show me the line of code where the tag is required | 12:04 |
| marios | weshay: i cannot | 12:05 |
| weshay | marios can you show me the error | 12:05 |
| marios | weshay: i want to change http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-master.yml | 12:05 |
| marios | weshay: dlrn_hash_tag: tripleo-ci-testing | 12:05 |
| weshay | is there a job that is failing? | 12:05 |
| marios | weshay: but sounds like panda|rover has founda reason for needing the retag. in which case we make it work with https://review.openstack.org/#/c/642662 | 12:05 |
| panda|rover | weshay: https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-master.yml#L23 | 12:06 |
| weshay | panda|rover which is | 12:06 |
| panda|rover | weshay: promotion release file uses tripleo-ci-testing | 12:06 |
| marios | weshay: the error is here http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/f5a872d/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz (i mean from the bug comment, the ovb job is pulling with the tripleo-ci-testing 2019-03-13 01:05:31 | TASK [tripleo-container-tag : Pull 192.168.24.1:8787/tripleomaster/centos-binary-cinder- | 12:06 |
| weshay | panda|rover that's odd | 12:06 |
| weshay | panda|rover the promotion script of all things knows what hash we're using | 12:07 |
| panda|rover | weshay: I don't know if it calculates the hash then, but starts from there | 12:07 |
| weshay | panda|rover that seems like a bug in the promotion script | 12:07 |
| weshay | panda|rover can you show me | 12:07 |
| marios | panda|rover: retag should be very simple in theory (like https://review.openstack.org/#/c/642662/5/playbooks/tripleo-buildcontainers/run.yaml) but we ran into ansible version issues so we need to bump something | 12:08 |
| weshay | marios right.. which is what I'm trying to push off for now | 12:08 |
| marios | weshay: well we need to decide what is faster but also safe enough to do quickly. fixup promotion script and http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-master.yml | 12:09 |
| marios | vs https://review.openstack.org/#/c/642662/5/playbooks/tripleo-buildcontainers/run.yaml | 12:09 |
| weshay | I need to see why the tag is needed | 12:09 |
| weshay | and I say this for two reasons.. | 12:10 |
| weshay | 1. ansible version | 12:10 |
| weshay | 2. it takes time and has to run a long operation and can fail | 12:10 |
| weshay | so let's try to avoid the HAVE TO tag twice if possible in promotion | 12:10 |
| weshay | does that make sense? | 12:11 |
| marios | weshay: panda|rover posting review for http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-master.yml for now | 12:11 |
| marios | weshay: ack | 12:11 |
| marios | weshay: i mean | 12:11 |
| marios | depends how much more is needed besides that oneliner ^ | 12:11 |
| marios | doing | 12:11 |
| ykarel | weshay, +1, marios more is needed | 12:12 |
| ykarel | you need to calculate dlrn_hash | 12:12 |
| ykarel | see repo setup role, | 12:12 |
| ykarel | for more context | 12:12 |
| marios | ykarel: can't we use dlrn_hash_tag in that file already? | 12:12 |
| arxcruz | chandankumar: will you finish today ? because today is the sprint end | 12:12 |
| marios | docker_image_tag: *promotion-testing-tag | 12:12 |
| marios | delorean_hash_label: &promotion-testing-tag "{{ dlrn_hash|default(dlrn_hash_tag) }}" | 12:12 |
| chandankumar | arxcruz: it is done | 12:12 |
| marios | ykarel: is that wrong? http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-master.yml | 12:12 |
| ykarel | marios, nope | 12:12 |
| arxcruz | chandankumar: you can put in storyless i guess | 12:12 |
| chandankumar | arxcruz: you mean voting one na? | 12:12 |
| weshay | yes.. the dlrn hash can be calculated w/ dlrn trunk | 12:13 |
| marios | ykarel: so you mean tripleo-ci-testing delorean repo hash != dlrn_hash_tag in http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-master.yml | 12:13 |
| weshay | panda|rover are you looking up where it's needed in the promotion script | 12:13 |
| marios | weshay: so i should grab dlrn trunk ooo-ci-testing delorean.repo hash ? | 12:13 |
| ykarel | marios, dlrn_hash_tag = tripleo-ci-testing, you need to calulate dlrn_hash from that tag | 12:14 |
| ykarel | let me fetch link | 12:14 |
| marios | ykarel: ack yeah that's what i mean | 12:14 |
| marios | ykarel: like fetch the delrean repo for tripleo-ci-testing | 12:14 |
| weshay | ykarel right.. the name might be wrong in the release file and THAT IS an issue | 12:14 |
| marios | ykarel: and get the baseurl /hash from there | 12:14 |
| marios | ykarel: that what you meant? | 12:14 |
| weshay | http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-master.yml#n23 | 12:15 |
| zbr | weshay: why making https://bugs.launchpad.net/tripleo/+bug/1819877 incomplete? | 12:15 |
| openstack | Launchpad bug 1819877 in tripleo "f28: tempest failure due to outdated python3-tempestconf" [High,Invalid] - Assigned to Sorin Sbarnea (ssbarnea) | 12:15 |
| weshay | looks right to me | 12:15 |
| ykarel | marios, yes, see https://github.com/openstack/tripleo-quickstart/blob/master/roles/repo-setup/tasks/main.yml#L2 | 12:15 |
| ykarel | and the playbook | 12:16 |
| ykarel | so you need more that 1 line change | 12:16 |
| marios | ykarel: ack yeah we had to do this already for the buildocntainers job too i mean setup the tripleo-ci-testing repo for the periodics and then use the hash for the tagging | 12:16 |
| marios | ykarel: thanks | 12:16 |
| ykarel | panda|rover, rfolco|ruck is the fs010 promotion job timeout issue known? | 12:17 |
| ykarel | it's failing since 11th | 12:17 |
| ykarel | timing out | 12:17 |
| ykarel | i think ovb will hit that too once testing good containers | 12:17 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, (3 more messages) | 12:19 |
| panda|rover | sorry, I'm lost. too many variables called similarly that do similar thing. We have delorean_hash_label, docker_image_tag, dlrn_hash_tag, dlrn_hash_tag_newest. I see promotion release file have most defined, and I this point I have to look which one wins in the end. | 12:20 |
| weshay | panda|rover++ | 12:20 |
| hubbot1 | weshay: panda|rover's karma is now 5 | 12:20 |
| weshay | ya.. I had the same thought | 12:20 |
| weshay | was looking for it's use | 12:20 |
| *** rlandy has joined #oooq | 12:21 | |
| weshay | http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/roles/repo-setup/tasks/get-dlrn-hash.yml#n63 | 12:21 |
| marios | weshay: and it is being consumed as 'dlrn_hash' in delorean_hash_label: &promotion-testing-tag "{{ dlrn_hash|default(dlrn_hash_tag) }}" http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-master.yml#n6 | 12:22 |
| marios | weshay: (sidenote. since repos *are* setup OK here, why don't i just grab it directly from /etc/yum.repos.d as we do upstream, if i really need to work out the hash here | 12:23 |
| weshay | so I think some clean up in order w/ regards to the promotion release files | 12:23 |
| weshay | and to check if there is fall out | 12:23 |
| *** dsneddon has joined #oooq | 12:24 | |
| weshay | marios here's the flow... in check / gate and promotion jobs.. | 12:24 |
| weshay | in the release file... | 12:24 |
| weshay | 1. grab https://trunk.rdoproject.org/centos7-master/current-tripleo/delorean.repo | 12:25 |
| weshay | calculate the dlrn_hash | 12:25 |
| weshay | and use that hash to pull content .. whether that's from a container registry, image etc.. | 12:26 |
| rlandy | weshay: when you get a chance, pls send me the password for the service user. kforde asked to test it out | 12:26 |
| marios | weshay: ack but did you mean grab the tripleo-ci-testing delorean repo | 12:26 |
| marios | weshay: (for promotion to current tripleo) | 12:27 |
| *** dsneddon has quit IRC | 12:29 | |
| panda|rover | marios: there is now way to make the retag work with a lower ansible version ? weshay: Bringing the tag situation to what we had previously could be easier than to fix the rest | 12:29 |
| panda|rover | marios: sorry, I meant is there no way* | 12:29 |
| marios | panda|rover: well don't know yet just what is in https://review.openstack.org/#/c/642662/5/playbooks/tripleo-buildcontainers/run.yaml@94 | 12:30 |
| panda|rover | need to take 30 minutes to eat | 12:30 |
| *** panda|rover is now known as panda|rover|eat | 12:30 | |
| weshay | panda|rover I'm going to get some coffee.. then lets chat and squash this ^ | 12:30 |
| marios | panda|rover: no sorry | 12:30 |
| marios | no food for you | 12:30 |
| panda|rover|eat | :D | 12:30 |
| marios | thanks panda|rover|eat weshay ykarel | 12:31 |
| panda|rover|eat | I have my radical solution ... but it's for next cycle. | 12:32 |
| marios | EAT NOW | 12:32 |
| marios | or forever hold your peace | 12:32 |
| weshay | PANDA!!!! | 12:35 |
| weshay | lolz | 12:35 |
| weshay | ok.. ready | 12:36 |
| ykarel | ohhk | 12:37 |
| ykarel | now i got why only ovb is affected | 12:37 |
| ykarel | because get_build_id is not passed there | 12:37 |
| ykarel | s/get_build_id/get_build_command | 12:38 |
| *** amoralej is now known as amoralej|luncgh | 12:40 | |
| *** amoralej|luncgh is now known as amoralej|lunch | 12:40 | |
| weshay | anyone who wants to talk dlrn tags join my blue now please | 12:41 |
| * ykarel joining | 12:44 | |
| ykarel | i see no meeting in bluejeans | 12:44 |
| ykarel | Sorry, no meetings out here! | 12:44 |
| weshay | ykarel have 7 min https://bluejeans.com/u/whayutin/ | 12:53 |
| ykarel | weshay, ack | 12:54 |
| *** dsneddon has joined #oooq | 12:55 | |
| *** saneax has joined #oooq | 12:57 | |
| *** panda|rover|eat is now known as panda|rover | 13:00 | |
| *** dsneddon has quit IRC | 13:00 | |
| *** dsneddon has joined #oooq | 13:02 | |
| *** vkapalav has quit IRC | 13:02 | |
| panda|rover | rfolco|ruck: meeting ? | 13:03 |
| panda|rover | https://trello.com/b/0VFswmht/rdo-infra-retrospective?menu=filter&filter=label:UniSprint6 | 13:03 |
| rfolco|ruck | marios, quiquell, sshnaidm, weshay, panda, rlandy, arxcruz, mwhahaha, rfolco, chkumar, ssbarnea, kopecmartin | 13:05 |
| rfolco|ruck | retro mtg ^ | 13:05 |
| *** dsneddon has quit IRC | 13:10 | |
| *** fmount has quit IRC | 13:16 | |
| *** udesale has quit IRC | 13:16 | |
| *** udesale has joined #oooq | 13:17 | |
| *** fmount has joined #oooq | 13:17 | |
| *** tosky has quit IRC | 13:22 | |
| ykarel | panda|rover, so the bug that i mentioned re. fs010 timeout is probably caused by https://review.openstack.org/#/c/554950/17 | 13:24 |
| ykarel | mistralclient change, it started affecting recently because mistralclient released few days back | 13:25 |
| panda|rover | ykarel: ok, in retro meeting right now, looking later | 13:27 |
| ykarel | panda|rover, ack np | 13:27 |
| *** dsneddon has joined #oooq | 13:42 | |
| *** dsneddon has quit IRC | 13:47 | |
| *** amoralej|lunch is now known as amoralej | 13:49 | |
| *** dsneddon has joined #oooq | 13:55 | |
| marios | zbr synergy++ | 13:56 |
| *** skramaja has quit IRC | 13:57 | |
| *** vkapalav has joined #oooq | 13:57 | |
| panda|rover | marios: zbr well it depends also on you, you're part of the component of the synergy | 13:58 |
| *** dsneddon has quit IRC | 14:02 | |
| *** gkadam_ has quit IRC | 14:02 | |
| *** dsneddon has joined #oooq | 14:04 | |
| *** dsneddon has quit IRC | 14:09 | |
| *** irclogbot_0 has quit IRC | 14:09 | |
| *** irclogbot_0 has joined #oooq | 14:12 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, (3 more messages) | 14:19 |
| *** irclogbot_0 has quit IRC | 14:25 | |
| *** irclogbot_0 has joined #oooq | 14:27 | |
| *** jtomasek has quit IRC | 14:38 | |
| *** dsneddon has joined #oooq | 14:40 | |
| *** agopi has quit IRC | 14:42 | |
| *** agopi has joined #oooq | 14:42 | |
| *** dsneddon has quit IRC | 14:45 | |
| *** dsneddon has joined #oooq | 14:54 | |
| arxcruz | chandankumar: go ahead and backport the patch to rocky | 14:59 |
| arxcruz | it really needs that | 14:59 |
| chandankumar | arxcruz: ok, will do that. | 14:59 |
| arxcruz | even creating the directory | 15:00 |
| arxcruz | it seems it fails | 15:00 |
| arxcruz | the directory is there | 15:00 |
| arxcruz | I'll keep my patch to fix the container | 15:00 |
| rlandy | zbr: pls post the link you use to access registry so I can see if I have access now | 15:05 |
| zbr | rlandy: http://console.registry.rdoproject.org/ | 15:09 |
| rlandy | zbr: thanks | 15:10 |
| *** rfolco|ruck is now known as rfolco | 15:10 | |
| weshay | marios actually hit me up at the bottom of the hour | 15:11 |
| weshay | I have a managers mtg | 15:11 |
| chandankumar | arxcruz: panda|rover sshnaidm https://review.openstack.org/#/c/642948/ please have a look when free! | 15:15 |
| marios | weshay: ack | 15:15 |
| marios | panda|rover: rfolco for hand over wher are you folks tracking status please (etherpad/whatever) | 15:23 |
| marios | sshnaidm: ^ | 15:23 |
| panda|rover | chandankumar: voted with a comment | 15:24 |
| marios | sshnaidm: panda|rover rfolco maybe we can have a call tomorrow EU noon? rfolco does it work for you? for hand over i mean? but would be nice to have something to start prepping with like etherpad/trello wherver you are tracking your status | 15:24 |
| marios | i can setup a call if folks agree | 15:24 |
| panda|rover | marios: just grab the first bug you see. | 15:26 |
| weshay | marios zbr rfolco and everyone else FYI https://docs.google.com/document/d/15WOk-ZwiDI4f3dwqDEJbKXpRiY61gFfIKBFH4CgZ8-s/edit | 15:28 |
| marios | weshay: ack | 15:29 |
| weshay | fyi.. marios ready join my blue | 15:30 |
| weshay | rlandy btw.. my rhel reinstall failed on storage I'll pick it back up today | 15:31 |
| rlandy | weshay: ack thanks | 15:31 |
| panda|rover | weshay: mind if I join if you're going to talk re: (not) retagging ? | 15:31 |
| rlandy | arxcruz: hey - need to borrow your tempest brain - probably simple question ... | 15:32 |
| weshay | panda|rover hold for a sec.. have 1 private thing first | 15:32 |
| marios | weshay: joining | 15:33 |
| chandankumar | panda|rover: how to add os_tempest job to cockpit? since it is a standalone job i think it will be automatically pulled | 15:33 |
| panda|rover | weshay: ok | 15:33 |
| rlandy | arxcruz: https://sf.hosted.upshift.rdu2.redhat.com/logs/22/165122/4/check/periodic-tripleo-ci-centos-7-baremetal-3ctlr_1comp-featureset001-rocky/39bc5d2/job-output.txt.gz#_2019-03-13_01_34_44_313962 | 15:33 |
| weshay | marios? | 15:33 |
| weshay | https://bluejeans.com/u/whayutin/ | 15:33 |
| panda|rover | chandankumar: what makes you think that ? | 15:33 |
| marios | weshay: 17:33 < marios> weshay: joining | 15:33 |
| rlandy | ^^ tempest passes but somewhere I am missing whatever setup/var defines where the results need to go | 15:33 |
| panda|rover | chandankumar: I really don't know the code that pulls job in the cockpit .. | 15:34 |
| chandankumar | panda|rover: https://github.com/rdo-infra/ci-config/blob/a29f879793915c21cf4617158e4b782ead130cb6/ci-scripts/infra-setup/roles/rrcockpit/files/telegraf/zuulv3_job_builds.py#L12 | 15:34 |
| arxcruz | rlandy: chandankumar is working on a patch for it | 15:35 |
| rlandy | chandankumar: arxcruz: thanks | 15:35 |
| panda|rover | chandankumar: wow, spotted a typo on line 13 | 15:35 |
| chandankumar | panda|rover: nice catch, let me fix it | 15:36 |
| *** irclogbot_0 has quit IRC | 15:36 | |
| rlandy | arxcruz: ^^ side note ... that is bm - if you ever want to run baremetal specific tempest tests | 15:36 |
| *** ykarel is now known as ykarel|away | 15:37 | |
| rlandy | weshay: just fyi ... https://review.rdoproject.org/r/#/c/19351/ - ovb timeout patch | 15:38 |
| *** irclogbot_0 has joined #oooq | 15:39 | |
| marios | panda|rover: do you wana join weshay bluejeans | 15:40 |
| marios | talk about the retag | 15:40 |
| marios | panda|rover: it will be fun | 15:40 |
| chandankumar | panda|rover: https://review.rdoproject.org/r/19355 | 15:40 |
| chandankumar | sshnaidm: thanks, &commented :-) | 15:41 |
| *** chandankumar is now known as raukadah | 15:47 | |
| *** irclogbot_0 has quit IRC | 15:49 | |
| *** irclogbot_0 has joined #oooq | 15:50 | |
| *** irclogbot_0 has quit IRC | 15:52 | |
| *** irclogbot_0 has joined #oooq | 15:56 | |
| *** holser_ has quit IRC | 15:58 | |
| *** holser_ has joined #oooq | 16:01 | |
| *** dsneddon has quit IRC | 16:04 | |
| *** jtomasek has joined #oooq | 16:16 | |
| *** trown is now known as trown|lunch | 16:18 | |
| *** dsneddon has joined #oooq | 16:19 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, tripleo-ci-centos-7-scenario010-multinode-oooq-container @ https://review.openstack.org/604298, (3 more messages) | 16:19 |
| sshnaidm | marios, https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build-push/9085374/ara-report/ | 16:19 |
| sshnaidm | marios, s/http/https/ | 16:19 |
| sshnaidm | marios, some browsers redirect to https, some don't.. not sure what is the problem there | 16:20 |
| *** dsneddon has quit IRC | 16:23 | |
| rlandy | sshnaidm: OVB ivp6 question ... if https://github.com/openstack/tripleo-quickstart/blob/master/roles/common/defaults/main.yml#L154 switch cases for ipv6, why does https://github.com/openstack/tripleo-quickstart-extras/blob/master/config/environments/ovb-common.yml#L36 remain ipv4. (considering fs035) | 16:24 |
| * rlandy hit a problem with https://review.openstack.org/#/c/642924 | 16:25 | |
| sshnaidm | rlandy, I think it never worked with external ipv6 network | 16:25 |
| sshnaidm | rlandy, maybe because we didn't have ipv6 routing in networks | 16:26 |
| rfolco | rlandy, panda|rover sshnaidm may I have one more +2 for this quick patch https://review.openstack.org/#/c/641808 ? | 16:26 |
| rlandy | sshnaidm: we do now though? | 16:26 |
| weshay | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/undercloud-deploy/templates/containers-prepare-parameter.yaml.j2 | 16:27 |
| weshay | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/standalone/tasks/main.yml#L67 | 16:27 |
| sshnaidm | rlandy, not sure it's enough, to make nat on ipv6 doesn't make sense, so need to be configured a little bit different | 16:27 |
| sshnaidm | rlandy, afaik we don't have ipv6 routing on rdo-cloud though.. | 16:28 |
| *** saneax has quit IRC | 16:28 | |
| *** dsneddon has joined #oooq | 16:28 | |
| rlandy | sshnaidm: I can get around the hardcoding in https://review.openstack.org/#/c/642924/1/roles/undercloud-deploy/templates/hieradata-overrides-t-h-t-undercloud.yaml.j2 - but not sure it's the right thing to do | 16:28 |
| rlandy | having discussion with hjensas on #tripleo | 16:29 |
| rlandy | <hjensas> rlandy: yes, that looks correct. And the ip6tables nat rules are created. | 16:29 |
| sshnaidm | rlandy, and what is undercloud_external_network_cidr ? | 16:29 |
| rlandy | sshnaidm: it resolve to ipv6 at that point | 16:29 |
| rlandy | from ttps://github.com/openstack/tripleo-quickstart/blob/master/roles/common/defaults/main.yml#L154 | 16:29 |
| sshnaidm | rlandy, external network should be routable | 16:30 |
| rlandy | sshnaidm: I can use another var - get by it somehow in CI - if using ipv4 network is the correct thing to do now in OVB rdocloud | 16:31 |
| sshnaidm | rlandy, we always do this | 16:32 |
| sshnaidm | rlandy, I mean external ipv4 net | 16:32 |
| *** dsneddon has quit IRC | 16:34 | |
| rlandy | variable overrides are confusing | 16:34 |
| rlandy | undercloud_external_network_cidr: 10.0.0.0/24 in https://github.com/openstack/tripleo-quickstart-extras/blob/master/config/environments/ovb-common.yml#L36 with not switch casing | 16:35 |
| *** dsneddon has joined #oooq | 16:35 | |
| *** holser_ has quit IRC | 16:41 | |
| *** ykarel|away is now known as ykarel | 16:41 | |
| *** udesale has quit IRC | 16:42 | |
| zbr | fyi, pyyaml 5.1 was released few minutes ago. | 16:43 |
| *** bogdando has quit IRC | 16:49 | |
| *** ykarel is now known as ykarel|away | 16:51 | |
| amoralej | an oooq job is failing in rdo | 17:02 |
| amoralej | http://logs.rdoproject.org/53/19353/2/check/distgit-tripleo-master-testing-centos-7-standalone-featureset052/dfb7d37/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 17:02 |
| amoralej | 2019-03-13 16:52:34 | Exception: 500 Server Error: Server Error for url: http://mirror.regionone.rdo-cloud-tripleo.rdoproject.org:8081/registry-1.docker/v2/ | 17:02 |
| amoralej | http://mirror.regionone.rdo-cloud-tripleo.rdoproject.org:8081/registry-1.docker/v2/ is returning 500 | 17:02 |
| amoralej | is this infra issue? | 17:03 |
| amoralej | panda|rover, ^ | 17:03 |
| amoralej | or it's using wrong url for some reason? | 17:03 |
| panda|rover | amoralej: the deafult has been switched to registry v2 | 17:06 |
| *** trown|lunch is now known as trown | 17:07 | |
| *** dtantsur is now known as dtantsur|afk | 17:15 | |
| *** kopecmartin is now known as kopecmartin|off | 17:23 | |
| *** jfrancoa has quit IRC | 17:27 | |
| *** ccamacho has quit IRC | 17:29 | |
| *** chem has quit IRC | 17:36 | |
| *** jpena is now known as jpena|off | 17:42 | |
| panda|rover | amoralej: so I could not save the distgit and rdoinfo jobs probably | 17:45 |
| panda|rover | amoralej: dammit, I thought I anticipated enough this time | 17:46 |
| amoralej | wdym? | 17:46 |
| amoralej | do we need some fix? | 17:46 |
| panda|rover | amoralej: yes this was caused by this review https://review.openstack.org/642808 | 17:46 |
| panda|rover | amoralej: all the jobs are now using the registry protocol v2 to pull containers images | 17:46 |
| panda|rover | the distgit are probably setting an equivalent variable somewhere ? | 17:47 |
| amoralej | but that review is not merged yet | 17:47 |
| panda|rover | amoralej: mmhh well I see v2/ in the proxy url | 17:48 |
| panda|rover | amoralej: so it's related | 17:48 |
| panda|rover | amoralej: I think jpena|off had to change something in the proxy configuration to make this work | 17:49 |
| panda|rover | maybe fixing for protocol 2 has broken protocol 1 | 17:49 |
| amoralej | ok, i'll merge that distgit patch | 17:49 |
| amoralej | as it's simple enough and oooq job is non voting | 17:49 |
| panda|rover | we should also check if protocol v1 is broken everywhere at this point | 17:50 |
| *** panda|rover is now known as panda|rover|off | 17:57 | |
| *** derekh has quit IRC | 18:00 | |
| *** amoralej is now known as amoralej|off | 18:08 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-multinode-1ctlr-featureset010, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, (3 more messages) | 18:19 |
| *** ykarel|away has quit IRC | 18:35 | |
| *** jtomasek has quit IRC | 18:48 | |
| rlandy | weshay: hello! | 20:06 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/51/165051/18/check/tripleo-ci-rhel-7-standalone-rhos-14/95f88cb/logs/undercloud/etc/yum.repos.d/rhos-release-rhel-7.6.repo.txt.gz | 20:06 |
| rlandy | ^^ [rhelosp-rhel-7.6-extras] is enabled | 20:06 |
| rlandy | we need another repo? | 20:06 |
| rlandy | that would enable rhos-14 standalone in ci | 20:07 |
| weshay | rlandy hello! | 20:07 |
| * weshay looks | 20:07 | |
| weshay | rlandy hrm.. don't see git in extras http://pulp.dist.prod.ext.phx2.redhat.com/content/dist/rhel/server/7/7Server/x86_64/extras/os/Packages/g/ | 20:08 |
| weshay | maybe optional | 20:09 |
| weshay | rlandy ya. we'll need to enable http://pulp.dist.prod.ext.phx2.redhat.com/content/dist/rhel/server/7/7Server/x86_64/optional/os/Packages/g/ | 20:10 |
| weshay | enable optional | 20:10 |
| * rlandy looks for optional | 20:10 | |
| rlandy | [rhelosp-rhel-7.6-server-opt]? | 20:11 |
| weshay | ya | 20:12 |
| weshay | [rhelosp-rhel-7.6-server-opt] | 20:12 |
| weshay | name=Red Hat Enterprise Linux $releasever - $basearch - Server Optional | 20:12 |
| weshay | baseurl=http://pulp.dist.prod.ext.phx2.redhat.com/content/dist/rhel/server/7/7Server/$basearch/optional/os/ | 20:12 |
| weshay | skip_if_unavailable=1 | 20:12 |
| weshay | enabled=0 | 20:12 |
| weshay | gpgcheck=0 | 20:12 |
| weshay | yum config enable rhelosp-rhel-7.6-server-opt | 20:13 |
| weshay | or something like that | 20:13 |
| rlandy | yum-config-manager --enable rhelosp-rhel-7.6-server-opt | 20:14 |
| rlandy | in http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/config/release/rhos-14.yml#n36 | 20:15 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-multinode-1ctlr-featureset010, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-scenario010 (4 more messages) | 20:19 |
| weshay | rlandy ya.. I don't see it in that file atm, but ya.. that's the place | 20:23 |
| rlandy | cool | 20:23 |
| rlandy | let's try it | 20:23 |
| weshay | rlandy can you help an idiot out https://review.openstack.org/#/c/643157/ | 20:33 |
| rlandy | weshay: commented | 20:34 |
| weshay | ha.. | 20:35 |
| weshay | geez | 20:35 |
| weshay | I think visual studio did that to me | 20:35 |
| weshay | dang it | 20:35 |
| weshay | rlandy ok fixed | 20:36 |
| rlandy | weshay: yes - look right - let's see if the logs agree | 20:37 |
| weshay | ah man | 20:37 |
| weshay | :) | 20:37 |
| rlandy | what now? | 20:38 |
| weshay | rlandy can we pull the trigger on https://review.openstack.org/#/c/642578/ | 20:38 |
| * rlandy checks logs output on that sec | 20:39 | |
| rlandy | weshay: we need to update doc ... https://logs.rdoproject.org/78/642578/3/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-pike-branch/61cdb1e/logs/README-reproducer-zuul-based-quickstart.html | 20:42 |
| rlandy | Check the top-level logs directory for a tar file reproducer-zuul-based-quickstart.tar. | 20:42 |
| weshay | ah k | 20:43 |
| weshay | will do so now | 20:43 |
| weshay | this is odd.. tangent | 20:43 |
| weshay | rlandy you should stop talking to me | 20:43 |
| weshay | http://logs.openstack.org/95/643095/2/check/tripleo-ci-centos-7-scenario003-standalone/0c97241/logs/undercloud/home/zuul/containers-prepare-parameters.yaml.txt.gz | 20:43 |
| weshay | vs | 20:43 |
| weshay | http://logs.rdoproject.org/95/643095/2/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/35f14d7/logs/undercloud/home/zuul/containers-prepare-parameter.yaml.txt.gz | 20:43 |
| * rlandy says nothing - stops talking to weshay | 20:44 | |
| *** weshay is now known as other_guys | 20:46 | |
| *** other_guys is now known as other_guy | 20:46 | |
| other_guy | rlandy this line? wget the reproducer-zuul-based-quickstart.tar file. | 20:47 |
| other_guy | just add the dir? | 20:47 |
| rlandy | yes - looking through - a few more comments | 20:47 |
| rlandy | sec | 20:48 |
| rlandy | other_guy: doc updates posted in comments | 20:55 |
| rlandy | rest looks good | 20:55 |
| rlandy | can still wget file and tar it | 20:55 |
| other_guy | rlandy thanks | 20:55 |
| *** other_guy is now known as weshay | 20:55 | |
| rlandy | the rest is the same | 20:55 |
| *** fultonj has quit IRC | 20:59 | |
| *** fultonj has joined #oooq | 21:01 | |
| weshay | rlandy one more https://review.openstack.org/#/c/642866/ | 21:08 |
| *** trown is now known as trown|outtypewww | 21:10 | |
| weshay | rlandy aslo can we just merge this please https://review.rdoproject.org/r/#/c/19351/ | 21:12 |
| rlandy | looking | 21:13 |
| rlandy | https://review.rdoproject.org/r/#/c/19351/ - merged | 21:13 |
| rlandy | w+1 at least | 21:13 |
| rlandy | weshay: you sure about https://review.openstack.org/#/c/642866/ - I had those options in the doc originally and you told me to take them out | 21:14 |
| rlandy | as they could get out of date | 21:15 |
| weshay | rlandy well.. I was trying to explain something to hrybacki and it would have been nice to have it their in our conversation | 21:15 |
| weshay | ya.. they could get out of date, but I think they would be helpful | 21:15 |
| weshay | also it's not as messy now that you can expand / contract the doc | 21:15 |
| * rlandy just checks tests failure to be sure | 21:16 | |
| rlandy | "overcloud_deploy_result": "failed" | 21:16 |
| rlandy | - ok | 21:16 |
| rlandy | weshay: k - got my +2 on https://review.openstack.org/#/c/642866/ | 21:18 |
| rlandy | weshay: do you want me to proposed you as core on rdo repos? | 21:18 |
| weshay | rlandy I get dinged for being core.. so I don't think so | 21:22 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/51/165051/19/check/tripleo-ci-rhel-7-standalone-rhos-14/ff5941e/job-output.txt.gz#_2019-03-13_16_44_40_679591 | 21:25 |
| rlandy | enabling opt didn't help :( | 21:26 |
| rlandy | weshay: ^^ | 21:26 |
| * rlandy looks for other options | 21:26 | |
| * weshay looks | 21:26 | |
| weshay | hrm.. but I saw gcc there | 21:26 |
| weshay | ah MOCK | 21:27 |
| weshay | I remember MOCK | 21:27 |
| weshay | dam you mock | 21:27 |
| rlandy | yes in deed | 21:29 |
| weshay | rlandy chatting in rhos-release | 21:38 |
| weshay | re: mock | 21:38 |
| weshay | rlandy sorry.. rhos-delivery | 21:38 |
| weshay | rlandy to unblock you | 21:39 |
| weshay | let's chat for a sec and tmate | 21:39 |
| weshay | https://bluejeans.com/u/whayutin/ | 21:40 |
| rlandy | was just reading back | 21:43 |
| rlandy | joining blue | 21:43 |
| weshay | rlandy https://sf.hosted.upshift.rdu2.redhat.com/logs/51/165051/19/check/tripleo-ci-rhel-7-standalone-rhos-14/ff5941e/job-output.txt.gz#_2019-03-13_16_44_40_679591 | 21:46 |
| *** tosky has joined #oooq | 21:56 | |
| *** vkapalav has quit IRC | 22:15 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-multinode-1ctlr-featureset010, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-fedora-28-standalone, (4 more messages) | 22:19 |
| weshay | rlandy 15:39 <@mburns> weshay: i'm coming to the conclusion that it's not shipped | 22:21 |
| weshay | rlandy ya.. multinode and ovb look similiar | 22:22 |
| rlandy | weshay: k - that makes sense | 22:25 |
| *** holser_ has joined #oooq | 22:40 | |
| rlandy | weshay: progress - failing on standlaone deploy ... will investigate ... https://sf.hosted.upshift.rdu2.redhat.com/logs/51/165051/20/check/tripleo-ci-rhel-7-standalone-rhos-14/15a0d92/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 22:40 |
| weshay | rlandy https://sf.hosted.upshift.rdu2.redhat.com/logs/51/165051/20/check/tripleo-ci-rhel-7-standalone-rhos-14/15a0d92/logs/undercloud/home/zuul/containers-prepare-parameters.yaml.txt.gz | 22:46 |
| weshay | is why | 22:46 |
| weshay | I think | 22:46 |
| weshay | maybe not | 22:46 |
| weshay | meh | 22:46 |
| weshay | rlandy if you can get a node up... try running | 22:47 |
| weshay | sudo /usr/bin/tripleo-container-image-prepare --roles-file /tmp/ansible.UNT2u9-role-data --environment-file /tmp/ansible.NIU6Bm-prepare-param --cleanup partial 2> tripleo-container-image-prepare.log | 22:47 |
| rlandy | yep - just putting in the rocky review stuff | 22:51 |
| *** saneax has joined #oooq | 23:02 | |
| *** saneax has quit IRC | 23:02 | |
| rlandy | weshay: https://github.com/openstack-infra/tripleo-ci/blob/master/zuul.d/standalone-jobs.yaml#L10 should run for stable/rocky | 23:03 |
| weshay | thanks | 23:03 |
| rlandy | do we need an addition job def? | 23:03 |
| * weshay looks to see if it's running | 23:03 | |
| * rlandy will make sure internal does the same | 23:04 | |
| weshay | rlandy http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-standalone&branch=stable%2Frocky | 23:04 |
| weshay | ya.. it's working | 23:04 |
| weshay | ugh.. maybe I remember that | 23:05 |
| weshay | too many $thing | 23:05 |
| weshay | s | 23:05 |
| weshay | ok | 23:05 |
| weshay | rlandy last updated 9 months ago | 23:05 |
| weshay | so.. ya we must have had it working in rocky cycle | 23:05 |
| rlandy | weshay: ack - ok - need to go help sister with hectic times - back later | 23:06 |
| *** rlandy is now known as rlandy|bbl | 23:06 | |
| weshay | k k.. no worries at all ur even more ahead of schedule now | 23:07 |
| *** rascasoft has quit IRC | 23:09 | |
| *** tosky has quit IRC | 23:23 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!