| *** rnoriega_ has joined #oooq | 00:28 | |
| *** weshay_ has joined #oooq | 00:28 | |
| *** rasca_ has joined #oooq | 00:28 | |
| *** _atoth has joined #oooq | 00:29 | |
| *** lhinds|out has joined #oooq | 00:29 | |
| *** pliu_ has joined #oooq | 00:29 | |
| *** rasca has quit IRC | 00:30 | |
| *** pliu has quit IRC | 00:30 | |
| *** rnoriega has quit IRC | 00:31 | |
| hubbot | FAILING CHECK JOBS on master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/567224 | 00:31 |
|---|---|---|
| *** myoung|off has quit IRC | 00:31 | |
| *** jjoyce has quit IRC | 00:31 | |
| *** jschluet has joined #oooq | 00:32 | |
| *** weshay has quit IRC | 00:32 | |
| *** jschlueter has quit IRC | 00:32 | |
| *** lhinds has quit IRC | 00:32 | |
| *** faceman has quit IRC | 00:32 | |
| *** saneax has quit IRC | 00:33 | |
| *** jjoyce has joined #oooq | 00:33 | |
| *** myoung has joined #oooq | 00:33 | |
| *** rasca has joined #oooq | 00:42 | |
| *** rasca_ has quit IRC | 00:42 | |
| *** rnoriega has joined #oooq | 00:43 | |
| *** myoung_ has joined #oooq | 00:43 | |
| *** weshay has joined #oooq | 00:43 | |
| *** fuzzball81 has joined #oooq | 00:43 | |
| *** pliu_ has quit IRC | 00:43 | |
| *** _atoth has quit IRC | 00:44 | |
| *** lhinds|out has quit IRC | 00:44 | |
| *** weshay_ has quit IRC | 00:44 | |
| *** rnoriega_ has quit IRC | 00:44 | |
| *** myoung has quit IRC | 00:44 | |
| *** jschluet has quit IRC | 00:44 | |
| *** jjoyce has quit IRC | 00:44 | |
| *** pliu has joined #oooq | 00:46 | |
| *** jschlueter has joined #oooq | 00:46 | |
| *** lhinds has joined #oooq | 00:47 | |
| *** faceman has joined #oooq | 00:49 | |
| *** jbadiapa has joined #oooq | 01:30 | |
| *** yolanda_ has joined #oooq | 01:56 | |
| *** yolanda_ has quit IRC | 01:57 | |
| *** yolanda_ has joined #oooq | 01:58 | |
| *** yolanda has quit IRC | 01:59 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/567224, master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/560445 | 02:31 |
| *** udesale has joined #oooq | 03:44 | |
| *** links has joined #oooq | 04:02 | |
| *** rlandy|rover|bbl is now known as rlandy|rover | 04:10 | |
| *** jbadiapa has quit IRC | 04:12 | |
| hubbot | FAILING CHECK JOBS on master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/567224 | 04:31 |
| *** udesale has quit IRC | 04:56 | |
| *** udesale has joined #oooq | 04:57 | |
| *** udesale has quit IRC | 04:57 | |
| *** udesale has joined #oooq | 04:58 | |
| *** jaganathan has joined #oooq | 05:03 | |
| *** pgadiya has joined #oooq | 05:11 | |
| *** pgadiya has quit IRC | 05:11 | |
| *** saneax has joined #oooq | 05:26 | |
| *** alee_ has quit IRC | 05:31 | |
| *** alee_ has joined #oooq | 05:31 | |
| *** links has quit IRC | 05:33 | |
| *** alee_ has quit IRC | 05:36 | |
| *** quiquell|off is now known as quiquell | 05:38 | |
| *** sshnaidm|brq has joined #oooq | 05:42 | |
| *** sshnaidm|brq has quit IRC | 05:47 | |
| *** links has joined #oooq | 05:49 | |
| *** jaosorior has joined #oooq | 05:52 | |
| *** holser__ has joined #oooq | 06:23 | |
| *** ccamacho has joined #oooq | 06:29 | |
| hubbot | FAILING CHECK JOBS on master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/567224 | 06:31 |
| *** brault has joined #oooq | 06:33 | |
| *** brault has quit IRC | 06:37 | |
| *** links has quit IRC | 06:44 | |
| *** jaosorior has quit IRC | 06:47 | |
| *** alee_ has joined #oooq | 06:50 | |
| *** ykarel|training has joined #oooq | 06:50 | |
| *** bogdando has joined #oooq | 06:54 | |
| *** brault has joined #oooq | 06:58 | |
| *** zoli|gone is now known as zoli | 07:01 | |
| *** links has joined #oooq | 07:03 | |
| *** jbadiapa has joined #oooq | 07:09 | |
| *** links has quit IRC | 07:12 | |
| *** kopecmartin has joined #oooq | 07:13 | |
| *** brault has quit IRC | 07:14 | |
| *** ratailor has joined #oooq | 07:16 | |
| *** jbadiapa has quit IRC | 07:17 | |
| *** jbadiapa has joined #oooq | 07:17 | |
| *** gkadam has joined #oooq | 07:18 | |
| *** sshnaidm|brq has joined #oooq | 07:21 | |
| *** tosky has joined #oooq | 07:28 | |
| *** links has joined #oooq | 07:33 | |
| *** skramaja has joined #oooq | 07:37 | |
| *** amoralej|off is now known as amoralej | 07:41 | |
| *** skramaja_ has joined #oooq | 07:42 | |
| *** skramaja has quit IRC | 07:42 | |
| quiquell | sshnaidm|brq: You there, I have a question regarding build-test-packages | 07:51 |
| *** brault has joined #oooq | 07:57 | |
| *** brault has quit IRC | 07:57 | |
| *** brault has joined #oooq | 07:58 | |
| arxcruz|ruck | chandankumar: around ? | 08:08 |
| chandankumar | arxcruz|ruck: yup | 08:09 |
| arxcruz|ruck | chandankumar: https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/519da2f/tempest.html.gz | 08:09 |
| *** brault has quit IRC | 08:13 | |
| chandankumar | arxcruz|ruck: it is a bug, due to volume v3 migration | 08:14 |
| sshnaidm|brq | quiquell, will ping you little bit later | 08:15 |
| chandankumar | arxcruz|ruck: https://github.com/openstack/barbican-tempest-plugin/blob/master/barbican_tempest_plugin/tests/scenario/manager.py#L82 | 08:15 |
| arxcruz|ruck | chandankumar: do we have a patch already or shall i open a bug or what ? | 08:15 |
| chandankumar | arxcruz|ruck: I am creating a patch | 08:15 |
| quiquell | sshnaidm|brq: Ok, no hurries. | 08:15 |
| arxcruz|ruck | chandankumar: what's the reveiw ? | 08:16 |
| sshnaidm|brq | quiquell, yep, shoot | 08:23 |
| quiquell | sshnaidm|brq: I am with this card https://trello.com/c/6tcD7ilr/778-injecting-zuul-changes-at-various-points-in-the-job-workflow | 08:24 |
| quiquell | Going to test this https://review.openstack.org/572694 | 08:25 |
| quiquell | Adding release to the gating repo tarball name | 08:25 |
| quiquell | Also maybe adding release to the compressed_gating_repo fact | 08:25 |
| quiquell | sshnaidm|brq: The concrete question is why compressed_gating_repo fact is cacheble ? | 08:26 |
| sshnaidm|brq | quiquell, well, I think we did all "set_fact" cacheable by default, without special treating of compressed_gating_repo. We actually need cacheable fact if we are going to use this fact in different playbook | 08:28 |
| chandankumar | arxcruz|ruck: https://review.openstack.org/572697 | 08:28 |
| sshnaidm|brq | quiquell, why do we need to rename it..? Don't we just install the latest package by default? | 08:29 |
| quiquell | sshnaidm|brq: The thing is the gating repo generated at undercloud playbook doesn't make senes for undercloud-upgrade playbook | 08:29 |
| quiquell | is for the n -> n + 1 scenario | 08:29 |
| sshnaidm|brq | quiquell, right, but we can drop all built packages there, right? | 08:29 |
| sshnaidm|brq | quiquell, I mean it shouldn't hurt | 08:29 |
| quiquell | We were talking about archiving instead of dropping for debug purposes | 08:30 |
| sshnaidm|brq | quiquell, it will build the latest "n+1" always because it has higher number | 08:30 |
| quiquell | But I was thinkin that adding the release to the gating repo is not a bad option | 08:30 |
| quiquell | no need for achive and you know what stable release has been used | 08:30 |
| arxcruz|ruck | chandankumar: -1 :) | 08:31 |
| hubbot | FAILING CHECK JOBS on master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/567224 | 08:32 |
| sshnaidm|brq | quiquell, ok, seems like I missed this talk, not really understand the purpose of that.. | 08:32 |
| quiquell | sshnaidm|brq: Will test and maybe go back to you with more specifics | 08:32 |
| arxcruz|ruck | quiquell: dhclient.conf file is big, that's the reason i didn't show up the content :) | 08:38 |
| *** brault has joined #oooq | 08:39 | |
| quiquell | arxcruz|ruck: Ahh ok, what do you think about the block ? | 08:40 |
| quiquell | Is good to reuse conditions | 08:40 |
| arxcruz|ruck | quiquell: bad idea | 08:40 |
| arxcruz|ruck | because i need to send another patch set | 08:40 |
| arxcruz|ruck | lol | 08:40 |
| quiquell | arxcruz|ruck: But my beer is waitting | 08:40 |
| arxcruz|ruck | that's a good incentive | 08:40 |
| *** jfrancoa has joined #oooq | 08:41 | |
| arxcruz|ruck | chandankumar: https://bugs.launchpad.net/tripleo/+bug/1775353 | 08:42 |
| openstack | Launchpad bug 1775353 in tripleo "barbican_tempest_plugin.tests.scenario.test_volume_encryption failing on periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master" [Critical,Triaged] - Assigned to chandan kumar (chkumar246) | 08:42 |
| arxcruz|ruck | chandankumar: also add this bug on your patch ok ? | 08:42 |
| *** jfrancoa has quit IRC | 08:43 | |
| arxcruz|ruck | quiquell: hey, do you know where's the promotion criteria file ? | 08:56 |
| arxcruz|ruck | i don't remember | 08:56 |
| quiquell | arxcruz|ruck: https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/dlrnapi_promoter/config | 08:57 |
| arxcruz|ruck | quiquell: so periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-queens-upload isn't part of the promotion criteria right ? | 08:58 |
| quiquell | arxcruz|ruck: If it's not there, then is not | 08:58 |
| chandankumar | arxcruz|ruck: ack | 09:01 |
| arxcruz|ruck | chandankumar: i also assign the bug for you, since you are working on that :P | 09:23 |
| *** gkadam has quit IRC | 09:25 | |
| arxcruz|ruck | sshnaidm|brq: http://cistatus.tripleo.org/phase1/ and phase2 not found (not sure if you change the name though) | 09:26 |
| quiquell | arxcruz|ruck, sshnaidm|brq: Do you recall this ? error: (25, 'Inappropriate ioctl for device') | 09:26 |
| arxcruz|ruck | quiquell: no, where is having this failure? shall i open a bug ? | 09:26 |
| quiquell | I am running a reproducer script at my fedora | 09:26 |
| quiquell | fedora 28 | 09:26 |
| quiquell | new fresh installed | 09:26 |
| *** zoli is now known as zoli|lunch | 09:33 | |
| *** links has quit IRC | 09:39 | |
| *** links has joined #oooq | 09:57 | |
| *** jaganathan has quit IRC | 10:02 | |
| *** jaganathan has joined #oooq | 10:03 | |
| *** jaganathan has quit IRC | 10:08 | |
| *** jaosorior has joined #oooq | 10:08 | |
| *** links has quit IRC | 10:16 | |
| *** jaganathan has joined #oooq | 10:26 | |
| quiquell | github is down ? | 10:27 |
| chandankumar | arxcruz|ruck: quiquell please have a look why http://logs.openstack.org/94/572094/2/check/tripleo-ci-centos-7-containers-multinode/5826b69/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-06-05_13_38_32 git clone is not working | 10:29 |
| *** links has joined #oooq | 10:29 | |
| quiquell | chandankumar: github is down | 10:29 |
| quiquell | chandankumar: is back online | 10:30 |
| chandankumar | quiquell: it is clonning from git.openstack.org | 10:30 |
| chandankumar | but not sure why it is not getting clonned | 10:30 |
| chandankumar | https://review.openstack.org/572347 | 10:30 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/567224, master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/560445 | 10:32 |
| quiquell | chandankumar, arxcruz|ruck: Do we have a policy on using github.com ? | 10:34 |
| sshnaidm|brq | github is M$ now, so expect for downtime :)) | 10:50 |
| *** zoli|lunch is now known as zoli | 10:54 | |
| *** dtantsur|afk is now known as dtantsur | 10:54 | |
| quiquell | sshnaidm|brq: github Me | 10:54 |
| quiquell | sshnaidm|brq: Have some unit test for the https://review.rdoproject.org/r/#/c/13604/ | 11:00 |
| *** yolanda_ has quit IRC | 11:00 | |
| quiquell | Don't know to do, upload them to the patch or wait for gabriel | 11:00 |
| *** yolanda_ has joined #oooq | 11:00 | |
| sshnaidm|brq | quiquell, you can make a following up patch | 11:01 |
| sshnaidm|brq | quiquell, rebased on this | 11:01 |
| quiquell | sshnaidm|brq: Also the fixes ? | 11:01 |
| sshnaidm|brq | quiquell, no, fixes should be in original patch | 11:01 |
| arxcruz|ruck | quiquell: no idea | 11:07 |
| quiquell | sshnaidm|brq: unit tests https://review.rdoproject.org/r/#/c/14084/ | 11:12 |
| sshnaidm|brq | quiquell++ | 11:17 |
| hubbot | sshnaidm|brq: quiquell's karma is now 0 | 11:17 |
| sshnaidm|brq | quiquell++ | 11:17 |
| hubbot | sshnaidm|brq: quiquell's karma is now 1 | 11:17 |
| sshnaidm|brq | quiquell, we need to create a job with these tests | 11:18 |
| sshnaidm|brq | quiquell, and gate promotion svript | 11:18 |
| sshnaidm|brq | quiquell, not urgent, of course | 11:18 |
| *** brault has quit IRC | 11:19 | |
| quiquell | sshnaidm|brq: Sure, let's wait for panda, so we organize | 11:19 |
| ykarel|training | chandankumar, shouldn't that be python-tempestconf? | 11:27 |
| ykarel|training | commented on the patch | 11:28 |
| *** dtantsur is now known as dtantsur|brb | 11:31 | |
| *** amoralej is now known as amoralej|lunch | 11:41 | |
| *** gkadam has joined #oooq | 11:49 | |
| rlandy|rover | rasca: hello ... https://logs.rdoproject.org/55/572155/2/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/Zde65c6b738de4399a1737179c04c7bbb/delorean_logs/07/fb/07fbbe2a9448b9c6ab8449f1a6c7f8777a06dce4_dev/build.log.txt.gz - build is failing a unit test | 12:02 |
| *** rfolco_ is now known as rfolco | 12:04 | |
| *** brault has joined #oooq | 12:04 | |
| weshay | arxcruz|ruck, you on the program call? | 12:05 |
| arxcruz|ruck | weshay: joinig | 12:05 |
| weshay | arxcruz|ruck, ugh | 12:07 |
| arxcruz|ruck | weshay: lol | 12:07 |
| weshay | arxcruz|ruck, don't let the colorblind guy work the colors | 12:07 |
| arxcruz|ruck | weshay: ctrl + z ? | 12:07 |
| arxcruz|ruck | weshay: let me do it | 12:08 |
| arxcruz|ruck | lol | 12:08 |
| arxcruz|ruck | weshay: done | 12:08 |
| weshay | fak... I have problems | 12:08 |
| weshay | :) | 12:08 |
| arxcruz|ruck | weshay: I'm working on the source problem you mentioned yesterday ;) | 12:08 |
| arxcruz|ruck | bug opened also | 12:09 |
| weshay | sorry which one? | 12:09 |
| arxcruz|ruck | weshay: https://bugs.launchpad.net/tripleo/+bug/1775365 | 12:09 |
| openstack | Launchpad bug 1775365 in tripleo "periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-queens-upload job is failing to fetch overcloud node names" [High,In progress] - Assigned to Arx Cruz (arxcruz) | 12:10 |
| arxcruz|ruck | weshay: in sova is showing as source fail | 12:10 |
| arxcruz|ruck | you mention yesterday, but i was sleeping already hehe | 12:10 |
| weshay | arxcruz|ruck, rlandy|rover I think we can delete ocata rdo phase 2 | 12:12 |
| arxcruz|ruck | weshay: so, last cix meeting alex told us to keep it until eol upstream | 12:12 |
| *** quiquell is now known as quiquell|lunch | 12:12 | |
| weshay | arxcruz|ruck, upstream doesn't care much about rdo2 | 12:12 |
| arxcruz|ruck | weshay: that's why we put it back to the dashboard | 12:12 |
| arxcruz|ruck | weshay: but rdo yes | 12:12 |
| arxcruz|ruck | rdoproject | 12:13 |
| rlandy|rover | weshay: arxcruz|ruck: I thought we asked about newton, - also ocata? | 12:16 |
| arxcruz|ruck | rlandy|rover: it was ocata | 12:16 |
| arxcruz|ruck | like, osp is done for ocata, but rdoproject no | 12:16 |
| arxcruz|ruck | so we keep for rdoproject | 12:16 |
| rlandy|rover | I know it went yellow ysterday | 12:16 |
| arxcruz|ruck | that was my understanding | 12:16 |
| arxcruz|ruck | rlandy|rover: not really, i need to talk with myoung_ because on http://rhos-release.virt.bos.redhat.com:3030/rhosp is showing hash date 05/31 | 12:17 |
| rlandy|rover | weshay: following the conversation with steve baker, there are some changes we need to discuss wrt the reproducer | 12:17 |
| arxcruz|ruck | but on https://dashboards.rdoproject.org/ocata the last repo is from 06-01 | 12:17 |
| rlandy|rover | I was going to look at that yesterday | 12:18 |
| rlandy|rover | let me look - maybe we can fix that | 12:18 |
| arxcruz|ruck | i check the code for the rhos-release dashboard , but i don't see anything wrong | 12:19 |
| rlandy|rover | looks largely green | 12:20 |
| quiquell|lunch | Is someone using rdo-ci-fx2-01-s4.v101.rdoci.lab.eng.rdu.redhat.com ? | 12:20 |
| *** trown|outtypewww is now known as trown | 12:20 | |
| rlandy|rover | depends on promotion criteria | 12:20 |
| arxcruz|ruck | rlandy|rover: which one ? | 12:21 |
| rlandy|rover | quiquell|lunch: ss says it's still yours | 12:21 |
| rlandy|rover | arxcruz|ruck: env B | 12:21 |
| quiquell|lunch | rlandy|rover: ss ? | 12:22 |
| rlandy|rover | quiquell|lunch: we keep a spread sheet with owners | 12:22 |
| rlandy|rover | tripleo-quickstart-ocata-rdo_trunk-baremetal-dell_fc430_envB-single_nic_vlans | 12:22 |
| rlandy|rover | still in promo cr | 12:23 |
| rlandy|rover | we can kick it for now | 12:23 |
| rlandy|rover | https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-quickstart-ocata-rdo_trunk-baremetal-dell_fc430_envB-single_nic_vlans/212/console | 12:24 |
| rlandy|rover | looking at what failed here | 12:24 |
| arxcruz|ruck | okay, so, now i'm not understanding, i thought dashboards.rdoproject.org/ocata was in sync with rhos-release dashboard | 12:24 |
| rlandy|rover | 2018-06-05 19:50:21 | Failed to contact the endpoint at http://10.12.149.10:35357 for discovery. Fallback to using that endpoint as the base url. | 12:24 |
| rlandy|rover | 2018-06-05 19:50:21 | Unable to establish connection to http://10.12.149.10:35357/OS-KSADM/roles: HTTPConnectionPool(host='10.12.149.10', port=35357): Max retries exceeded with url: /OS-KSADM/roles (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f02b4e6fc90>: Failed to establish a new connection: [Errno 111] Connection refused',)) | 12:24 |
| arxcruz|ruck | and on d.rdop is showing latest hash from 06-01 | 12:24 |
| weshay | arxcruz|ruck, rlandy|rover fyi https://review.rdoproject.org/jenkins/job/periodic-tripleo-centos-7-master-containers-build/980/console | 12:25 |
| arxcruz|ruck | and rhos-release 05-31 | 12:25 |
| rlandy|rover | 2018-06-06 12:17:26,784 7905 INFO promoter Skipping promotion of current-tripleo to current-tripleo-rdo, missing successful jobs: ['tripleo-quickstart-promote-ocata-rdo_trunk-minimal'] | 12:25 |
| rlandy|rover | that's one level up | 12:25 |
| weshay | https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-centos-7-master-containers-build/8e1b7ab/undercloud/home/jenkins/undercloud_install.log.txt.gz#_2018-06-06_04_34_29 | 12:25 |
| arxcruz|ruck | weshay: i already open a bug for this | 12:26 |
| weshay | rockin | 12:26 |
| arxcruz|ruck | weshay: https://bugs.launchpad.net/tripleo/+bug/1775199 | 12:26 |
| openstack | Launchpad bug 1775199 in tripleo "Failure in tripleo undercloud install no such option collect_timing in keystone_authtoken group" [Critical,Triaged] | 12:26 |
| weshay | see it now | 12:26 |
| weshay | thanks arxcruz|ruck | 12:26 |
| weshay | arxcruz|ruck, I removed the quickstart tag | 12:27 |
| arxcruz|ruck | weshay: k | 12:27 |
| weshay | quickstart is for bugs caused by quickstart | 12:27 |
| arxcruz|ruck | ok | 12:27 |
| rlandy|rover | arxcruz|ruck: I ma rekicking https://ci.centos.org/job/tripleo-quickstart-promote-ocata-rdo_trunk-minimal/ | 12:28 |
| rlandy|rover | that is where ocata i held up | 12:28 |
| arxcruz|ruck | rlandy|rover: ack | 12:28 |
| rlandy|rover | we can change the criteria and yet phase 2 through (remove B and put it back) E passed | 12:29 |
| rlandy|rover | weshay: ^^ ? | 12:29 |
| rlandy|rover | ok? | 12:29 |
| weshay | rlandy|rover, oh ya | 12:29 |
| weshay | let's do that | 12:29 |
| rlandy|rover | weshay: really - this is the killer 2018-06-06 12:17:26,784 7905 INFO promoter Skipping promotion of current-tripleo to current-tripleo-rdo, missing successful jobs: ['tripleo-quickstart-promote-ocata-rdo_trunk-minimal'] | 12:30 |
| rlandy|rover | 2018-06-06 12:17:26,785 7905 INFO promoter FINISHED promotion process | 12:30 |
| rlandy|rover | phase 1 job is very unstable | 12:30 |
| weshay | rlandy|rover nice.. look at that https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/all%20top%20level%20multijobs/job/rdo-promote-ocata-rdo_trunk/ | 12:30 |
| * weshay updates criteria | 12:30 | |
| rlandy|rover | weshay: give me one sec pls | 12:30 |
| rlandy|rover | just checking that that really si blocking phase2 | 12:31 |
| rlandy|rover | and we just arn't blocked ahead | 12:31 |
| rlandy|rover | 2018-06-06 12:17:01,282 7905 INFO promoter Skipping promotion of current-tripleo-rdo to current-tripleo-rdo-internal, missing successful jobs: ['tripleo-quickstart-ocata-rdo_trunk-baremetal-dell_fc430_envb-single_nic_vlans'] | 12:31 |
| rlandy|rover | weshay: ^^ ok - change the criteria | 12:31 |
| rlandy|rover | and I'll put it back after we promote | 12:31 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/567224, master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/560445 | 12:32 |
| weshay | rlandy|rover, ya.. doing it now | 12:32 |
| rlandy|rover | cool | 12:32 |
| *** links has quit IRC | 12:33 | |
| rlandy|rover | ok - let's deal with newton phase 1 - now | 12:33 |
| weshay | https://review.rdoproject.org/r/#/c/14085/ | 12:34 |
| weshay | 17:40:31 "job_id": "tripleo-quickstart-ocata-rdo_trunk-baremetal-hp_dl360_envd-single_nic_vlans", | 12:34 |
| weshay | 17:40:31 "notes": "buildinfo,phase=rdophase2 id=https://trunk.rdoproject.org/centos7-ocata/ab/4e/ab4eecdac47f47f685b6af17ff2ceab0dab73b5e_6969aa09,major=ocata,minor=rdo_trunk 1528220422", | 12:34 |
| weshay | 17:40:31 "success": true, | 12:34 |
| weshay | using lowercase d | 12:35 |
| rlandy|rover | weshay: merged | 12:37 |
| rlandy|rover | let's clean up one more thing .. | 12:37 |
| rlandy|rover | arxcruz|ruck: weshay: ok - newton build failure ... phase 1 - since April .. | 12:38 |
| rlandy|rover | https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-newton-delorean-build-images-1122/console.txt.gz | 12:38 |
| rlandy|rover | task path: /home/jenkins/workspace/tripleo-quickstart-promote-newton-delorean-build-images/tripleo-quickstart/roles/image-build/tasks/main.yml:61 | 12:38 |
| rlandy|rover | I don't see a clear error | 12:39 |
| weshay | WOW https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/all%20top%20level%20multijobs/job/rdo-promote-pike-rdo_trunk/ | 12:44 |
| *** amoralej|lunch is now known as amoralej | 12:44 | |
| *** atoth has joined #oooq | 12:46 | |
| rlandy|rover | weshay: we updated all the slaves and ff branches last week | 12:47 |
| weshay | rlandy|rover, what were the slaves updated to? | 12:50 |
| rlandy|rover | weshay: rhel 7.4 to rhel 7.5 | 12:50 |
| weshay | those are the virthosts | 12:50 |
| rlandy|rover | yep | 12:50 |
| rlandy|rover | the undercloud is a vm | 12:50 |
| rlandy|rover | on those virthosts | 12:51 |
| weshay | what were the slaves updated to? | 12:51 |
| rlandy|rover | rhel 7.4 to rhel 7.5 | 12:51 |
| weshay | we don't have slaves on rhel | 12:51 |
| rlandy|rover | apologies, the undercloud boxes | 12:51 |
| rlandy|rover | not the slaves | 12:51 |
| * weshay can't wait to get most of our work out of jenkins | 12:52 | |
| rlandy|rover | we don't reprovision those per job any more | 12:52 |
| weshay | we never have | 12:52 |
| rlandy|rover | ok ... image build ... where is the error I am missing??? | 12:53 |
| weshay | is that question to me? | 12:53 |
| rlandy|rover | Throw error msg if image build is failing is skipped | 12:53 |
| rlandy|rover | to anyone who knows | 12:54 |
| weshay | rlandy|rover, show me.. on blue | 12:54 |
| rlandy|rover | https://ci.centos.org/job/tripleo-quickstart-promote-newton-delorean-build-images/1123/console | 12:54 |
| rlandy|rover | ok - joining blue | 12:54 |
| * weshay looks | 12:54 | |
| *** quiquell|lunch is now known as quiquell | 12:55 | |
| weshay | ssbarnea, https://review.openstack.org/#/c/572733/1/scripts/website/planet.config.ini | 13:07 |
| myoung_ | morning o/, caught up to this mornings scrollback | 13:15 |
| myoung_ | weshay: ack rdo2 BM virthosts were updated last week --> rhel 7.5, we didn't touch the jenkins slaves | 13:16 |
| myoung_ | weshay: they were failing mostly on libvirt package dependencies, as well as the pinned 7.4 kernel, during prep-internal-rhel | 13:16 |
| weshay | ssbarnea, https://etherpad.openstack.org/p/tripleo-ci-squad-meeting | 13:16 |
| myoung_ | arxcruz|ruck: regarding the difference between how rhos-release dashboard and promotions work / are different, the rhos-release dashboard if I recall correctly uses the timestamp on the .repo file - when the repo was *created* as the source date...not when it was *promoted* | 13:17 |
| myoung_ | ^^ that is why dashboards.rdoproject.org/{{release}} lists both promotion and creation date for each promoted hash | 13:18 |
| *** jaosorior has quit IRC | 13:20 | |
| arxcruz|ruck | myoung_: ok | 13:20 |
| myoung_ | weshay, arxcruz|ruck, rlandy|rover - last week while you were out in CIX meeting we had a fairly lengthy discussion in that forum around how long to keep rdo1/rdo2 for ocata (specifically) online - I was suggesting we turn the lights off to limit the teams resource suckage (human, electrical, emotional :)) - the consensus and specific ask from Alex was that we leave them in place (and reported on) until ~ 2018-08-27 (https://releases. | 13:20 |
| myoung_ | openstack.org). So I/we re-patched the rhos-release dashboard to include ocata (it had been removed as the OSP variant based on it is a short term and is now EOL) | 13:20 |
| arxcruz|ruck | myoung_: yes, that's what i remember | 13:21 |
| rlandy|rover | myoung_: ack - we just confirmed we are keeping ocata | 13:21 |
| myoung_ | (re: osp11 those tiles are obviously gone for good and we should likely nuke those jobs) | 13:21 |
| rlandy|rover | I mixed it up with newton | 13:21 |
| rlandy|rover | my fault | 13:21 |
| *** myoung_ is now known as myoung | 13:21 | |
| weshay | rlandy|rover, ok.. /me looking again | 13:25 |
| myoung | rlandy|rover, weshay, I think we should do a patch like this https://code.engineering.redhat.com/gerrit/#/c/139138/ and stop building newton images alltogther | 13:25 |
| myoung | ^^ removed ocata and pike | 13:26 |
| * myoung focuses on sprint things, HTH if needed but otherwise ignoring | 13:26 | |
| rlandy|rover | weshay: blue again? | 13:26 |
| * myoung has a patch locally he's not messing with that rips ALL of the debris out, let him know if he should post it later | 13:26 | |
| myoung | ^^ no more building anything | 13:26 |
| weshay | rlandy|rover, sure | 13:26 |
| rlandy|rover | k | 13:27 |
| *** udesale has quit IRC | 13:32 | |
| myoung | weshay: we had an action item from monday scrum to update one or more cards with detail / etc. If you have a few minutes this morning we could bang it out together. I'm available this morning to chat. Guessng it's 5-10 tops. | 13:39 |
| quiquell | rlandy|rover, arxcruz|ruck: I see massive failing in the gates | 13:49 |
| quiquell | 7 fails in 30 minutes | 13:49 |
| quiquell | http://38.145.34.131:3000/d/pgdr_WVmk/ruck-rover?orgId=1&panelId=55&fullscreen | 13:50 |
| rlandy|rover | arxcruz|ruck: can you look - opening a ticket to ci.centos | 13:51 |
| arxcruz|ruck | rlandy|rover: quiquell 3 patches with some job failing, all of them upgrades and non voting | 13:52 |
| arxcruz|ruck | quiquell: https://review.rdoproject.org/zuul/ | 13:52 |
| arxcruz|ruck | quiquell: is your board counting non-voting jobs ? | 13:53 |
| quiquell | arxcruz|ruck: It shows upstream problems | 13:53 |
| quiquell | upstream tripleo jobs | 13:54 |
| arxcruz|ruck | quiquell: yeah, but can you point me the jobs failing? because in zuul it's showing 7 failures, but it's upgrades job and they are non-voting | 13:55 |
| quiquell | arxcruz|ruck: http://38.145.34.131:3000/d/pgdr_WVmk/ruck-rover?orgId=1&panelId=61&fullscreen | 13:56 |
| quiquell | You can click in the logs | 13:56 |
| quiquell | And in the Patch too to open the gerrit | 13:56 |
| arxcruz|ruck | quiquell: ok | 13:57 |
| hrybacki | weshay: sorry, wast off yesterday afternoon | 13:57 |
| rlandy|rover | ha - we're logging again :) | 13:58 |
| *** dtantsur|brb is now known as dtantsur | 13:58 | |
| arxcruz|ruck | quiquell: so, some of them was because github was down | 13:59 |
| arxcruz|ruck | one was because of timeout | 13:59 |
| arxcruz|ruck | so nothing we can do, github now is m$ problems :P | 14:00 |
| myoung | quiquell: I really like this presentation (stacked bars) for builds over time. | 14:01 |
| myoung | http://38.145.34.131:3000/d/pgdr_WVmk/ruck-rover?orgId=1&panelId=55&fullscreen | 14:01 |
| rlandy|rover | arxcruz|ruck: closing https://bugs.launchpad.net/tripleo/+bug/1775205 | 14:03 |
| openstack | Launchpad bug 1775205 in tripleo "Failing jobs are no longer getting the output from gate_test_oooq.sh script" [Critical,Triaged] - Assigned to Ronelle Landy (rlandy) | 14:03 |
| arxcruz|ruck | rlandy|rover: did infra fix it? \o/ | 14:03 |
| rlandy|rover | arxcruz|ruck: check the logs | 14:03 |
| arxcruz|ruck | i believe you :) | 14:03 |
| rlandy|rover | http://logs.openstack.org/97/571897/2/gate/tripleo-ci-centos-7-containerized-undercloud-upgrades/fd15958/job-output.txt.gz | 14:04 |
| hrybacki | using OOOQ reproducer to test a fix for a CI blocker (this is a first for me) -- fingers crossed! | 14:06 |
| myoung | hrybacki: early adopter! it's pretty cool. | 14:09 |
| myoung | hrybacki: ovb or libvirt? | 14:09 |
| hrybacki | myoung: ovb | 14:10 |
| hrybacki | myoung: tbh I wasn't even sure how to test my proposed fix against the failing periodic job. This seemed like an obvious test route | 14:10 |
| * myoung nods | 14:12 | |
| *** dtrainor has joined #oooq | 14:14 | |
| *** dtrainor has quit IRC | 14:14 | |
| quiquell | myoung: sshnaidm|brq is the creator of the bars chart is super awesome | 14:20 |
| quiquell | arxcruz|ruck: cool, was checking if the alarm was legit. | 14:21 |
| *** quiquell is now known as quiquell|off | 14:22 | |
| *** skramaja_ has quit IRC | 14:25 | |
| rlandy|rover | weshay: back in 2016, I applied to add my keys to the jumpbox to access ci.centos nodes. this is the slave we get assigned ... rdo-ci-cloudslave05 we get assigned. do we have instructions somewhere of how to access the jumphost, it's been a while and I don't think I ever actually landed up doing it | 14:28 |
| weshay | rlandy|rover, that infra to ci.centos was wiped | 14:29 |
| rlandy|rover | oh well - was worth a shot | 14:29 |
| rlandy|rover | ok - here we go with the request | 14:30 |
| *** moguimar_ has joined #oooq | 14:31 | |
| *** moguimar has quit IRC | 14:31 | |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/567224, master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/560445 | 14:32 |
| weshay | rfolco, panda ok.. so after chatting w/ some folks.. we will not be building osp containers/images in the internal sf | 14:40 |
| weshay | rfolco, panda rather consuming what has already been built | 14:40 |
| weshay | hrybacki, ping me when you have a sec | 14:41 |
| *** moguimar_ has quit IRC | 14:53 | |
| weshay | myoung, please make sure the bz triage for the tempest squad is on the agenda for tomorrow | 15:00 |
| myoung | weshay: ack. I've set up the ci squad's agenda for tomorrow, doing tempest etherpad shortly. | 15:01 |
| myoung | weshay: do you want me to increase the tempest squad scrum duration? If we're going thru that many bugs it's going to take > 30 | 15:02 |
| rfolco | weshay, k | 15:02 |
| weshay | rfolco, rdo on rhel is still valid | 15:04 |
| amoralej | is there any known blocker for master promotion? | 15:05 |
| amoralej | we are planning next test day | 15:05 |
| amoralej | so undercloud install is failing with | 15:07 |
| amoralej | APIException: no such option collect_timing in group [keystone_authtoken] | 15:08 |
| rlandy|rover | arxcruz|ruck: ^^ | 15:08 |
| hrybacki | o/ weshay | 15:08 |
| rlandy|rover | amoralej: ^^ yeah - we have a bug logged for that | 15:08 |
| amoralej | https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-centos-7-master-containers-build/8a1e0c1/undercloud/home/jenkins/undercloud_install.log.txt.gz | 15:08 |
| arxcruz|ruck | rlandy|rover: already have a bug opened for that | 15:08 |
| amoralej | is it escalated? | 15:08 |
| arxcruz|ruck | amoralej: yes, and hrybacki is working on that | 15:08 |
| weshay | hrybacki, there's my man | 15:08 |
| amoralej | ok, thanks | 15:08 |
| *** ratailor has quit IRC | 15:08 | |
| rlandy|rover | amoralej: that is all we know if atm - before that master was promoting well | 15:09 |
| rlandy|rover | if there is something caught when the build steps completes and tests start, we won't know atm | 15:09 |
| amoralej | yeah, thanks rlandy|rover | 15:09 |
| hrybacki | arxcruz|ruck: so question is: how do I verify my patch does or does not fix that broken job | 15:17 |
| arxcruz|ruck | hrybacki: adding a depends on ? | 15:18 |
| hrybacki | arxcruz|ruck: ? | 15:18 |
| arxcruz|ruck | hrybacki: where's your patch? was it merged already ? | 15:19 |
| hrybacki | arxcruz|ruck: it's a standalone patch: https://review.openstack.org/#/c/572773/ | 15:19 |
| hrybacki | arxcruz|ruck: no, need to verify CI before merging | 15:19 |
| hrybacki | but since the failure was noticed on a periodic job I'm not sure how to re-trigger it | 15:19 |
| hrybacki | https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-centos-7-master-containers-build/14ef616/undercloud/home/jenkins/undercloud_install.log.txt.gz#_2018-06-05_12_35_03 | 15:20 |
| arxcruz|ruck | hrybacki: you can check on rdocloud with the reproducer, i think you need to set an gerrit_changes variable to your patch | 15:20 |
| arxcruz|ruck | sshnaidm|brq: around? do you know how to do it ? | 15:20 |
| arxcruz|ruck | hrybacki: the periodic job will only get your patch once it get merged though | 15:21 |
| hrybacki | arxcruz|ruck: okay, so reproducer is the only way to verify failures found in periodic jobs? (just trying to understand) | 15:21 |
| arxcruz|ruck | hrybacki: you can submit a dummy patch on tq/tqe with a depends on to your patch also but it will not run as periodic though | 15:22 |
| hrybacki | some ci is better than no CI :) lemme try that | 15:22 |
| *** moguimar has joined #oooq | 15:23 | |
| hrybacki | arxcruz|ruck: sshnaidm|brq Depends-on: <commit hash> or <change-id> ? | 15:26 |
| arxcruz|ruck | hrybacki: both works, but now it's the full url for change id | 15:26 |
| arxcruz|ruck | https://review.o.o/#/12345 | 15:27 |
| rlandy|rover | quiquell|off: arxcruz|ruck: what happened with those failing jobs quiquell|off reported? need to look into them? | 15:28 |
| hrybacki | thanks arxcruz|ruck testing on https://review.openstack.org/572809 | 15:29 |
| arxcruz|ruck | rlandy|rover: mostly was because github was down this morning | 15:29 |
| rlandy|rover | arxcruz|ruck: ok | 15:29 |
| arxcruz|ruck | rlandy|rover: two was timeout | 15:29 |
| arxcruz|ruck | rlandy|rover: so don't worry | 15:29 |
| rlandy|rover | arxcruz|ruck: good - one less thing to tackle :) | 15:29 |
| arxcruz|ruck | yup | 15:29 |
| rlandy|rover | need to work on the reproducer | 15:29 |
| rlandy|rover | but will try fix rdo2 for a bit first | 15:30 |
| *** alee_ has quit IRC | 15:33 | |
| rlandy|rover | hrybacki: reading back ... if you want to try your change with a periodic job, you can take the reproducer from that job and add your ZUUL_CHANGES | 15:33 |
| *** jbadiapa has quit IRC | 15:33 | |
| rlandy|rover | sorry if I am answering something you already know | 15:33 |
| *** saneax has quit IRC | 15:35 | |
| arxcruz|ruck | rlandy|rover: no, you answer better than me, i though it was gerrit_changes | 15:36 |
| arxcruz|ruck | the variable | 15:36 |
| rlandy|rover | no you need the ZUUL_CHANGES | 15:36 |
| rlandy|rover | you can get that from the console log of any job that ran with that change | 15:36 |
| *** sshnaidm|brq has quit IRC | 15:37 | |
| rlandy|rover | arxcruz|ruck: just fyi ... to fix the newton image build failure in phase 1 ... https://bugs.launchpad.net/tripleo/+bug/1775416 - tripleo-quickstart-promote-newton-delorean-build-images is failing to scp images - "/home/jenkins/.ssh/rdo-ci-public.pem: No such file or directory" | 15:37 |
| openstack | Launchpad bug 1775416 in tripleo "tripleo-quickstart-promote-newton-delorean-build-images is failing to scp images - "/home/jenkins/.ssh/rdo-ci-public.pem: No such file or directory"" [High,Triaged] - Assigned to Ronelle Landy (rlandy) | 15:37 |
| hrybacki | rlandy|rover: no worries -- I think I've got it running properly now | 15:37 |
| rlandy|rover | cool | 15:38 |
| hrybacki | thank you :) | 15:38 |
| hrybacki | weshay: o/ I've got ~20 before my next mtg. Wanna chat? | 15:38 |
| arxcruz|ruck | rlandy|rover++ | 15:38 |
| hubbot | arxcruz|ruck: rlandy|rover's karma is now 5 | 15:38 |
| *** bogdando has quit IRC | 15:38 | |
| *** ykarel|training has quit IRC | 15:39 | |
| weshay | hrybacki, ya.. getting off blue in a minute | 15:41 |
| myoung | weshay, chandankumar, kopecmartin, arxcruz|ruck, please note change to tempest squad scrum meeting tomorrow (duration and content). Details --> https://etherpad.openstack.org/p/tripleo-tempest-squad-meeting | 15:44 |
| hrybacki | weshay: ack, join https://bluejeans.com/u/hrybacki/ anytime before 1100 mountain time | 15:46 |
| weshay | hrybacki, ok.. ready | 15:47 |
| weshay | rfolco, I'm updating the sf doc | 16:08 |
| *** trown is now known as trown|lunch | 16:09 | |
| *** zoli is now known as zoli|gone | 16:12 | |
| *** zoli|gone is now known as zoli | 16:12 | |
| weshay | rfolco, working on the goals | 16:13 |
| chandankumar | weshay: myoung http://logs.openstack.org/94/572094/2/check/tripleo-ci-centos-7-containers-multinode/3bf57de/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-06-06_14_38_13 | 16:13 |
| rfolco | weshay, yes, need to answer some whys | 16:13 |
| chandankumar | please have a look at this one i donot know why git clone tempestconf is not working | 16:13 |
| chandankumar | weshay: myoung related reviews https://review.openstack.org/#/c/572094/ | 16:14 |
| *** gkadam has quit IRC | 16:15 | |
| weshay | rfolco, did you get my pings re: the upgrade check job? | 16:16 |
| weshay | rfolco, talking to a few folks.. we're interesting in merging the non-voting job as is.. even if it's failing | 16:16 |
| rfolco | weshay, I am confused | 16:18 |
| rfolco | weshay, with zuul added to docker group, looks like it worked https://review.openstack.org/#/c/572187/ | 16:18 |
| rfolco | weshay, are you saying my workaround did not fix it ? | 16:19 |
| rfolco | weshay, my workaround --> https://review.openstack.org/#/c/572187/5/playbooks/tripleo-ci/run.yaml | 16:19 |
| weshay | rfolco, it's not the right place | 16:19 |
| weshay | for that fix | 16:19 |
| rfolco | I know | 16:19 |
| rfolco | but apparently that was the fix (to be done somewhere else) | 16:20 |
| weshay | rfolco, well maybe I need help | 16:20 |
| weshay | rfolco, was that fix for the prep-containers issue, console or something else? | 16:20 |
| rfolco | ok, I'll ditch the hack | 16:21 |
| weshay | in run.yaml | 16:21 |
| rfolco | weshay, the fix is for https://bugs.launchpad.net/tripleo/+bug/1775209 | 16:21 |
| openstack | Launchpad bug 1775209 in tripleo "docker permission denied on upgrade prep containers" [Critical,Triaged] - Assigned to Steve Baker (steve-stevebaker) | 16:21 |
| weshay | rfolco, you could just use a diff patch | 16:21 |
| weshay | rfolco, ok.. ya | 16:21 |
| weshay | rfolco, in the bug.. https://review.openstack.org/#/c/572663/ | 16:22 |
| weshay | is posted | 16:22 |
| weshay | rfolco, probably good to comment in that review, the bug, and if you know you have it fixed.. then I would say add a new review | 16:22 |
| weshay | rfolco, I still see the error in your job though http://logs.openstack.org/87/572187/5/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades/2a0b9fb/logs/undercloud/home/zuul/upgrade_overcloud_prep_containers.log.txt.gz | 16:23 |
| rfolco | weshay, multitasking mistakes. Apologize. | 16:24 |
| weshay | all FYI https://review.rdoproject.org/r/#/c/13951/ | 16:25 |
| rfolco | weshay, you are right, I am just removing the hack | 16:25 |
| *** bogdando has joined #oooq | 16:26 | |
| *** alee has joined #oooq | 16:26 | |
| hubbot | FAILING CHECK JOBS on master: tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/567224 | 16:32 |
| *** apetrich has quit IRC | 16:34 | |
| *** apetrich has joined #oooq | 16:35 | |
| weshay | rlandy|rover, arxcruz|ruck ruh roh | 16:48 |
| weshay | rlandy|rover, arxcruz|ruck whole bunch of gate failures | 16:49 |
| weshay | rlandy|rover, arxcruz|ruck http://38.145.34.131:3000/d/pgdr_WVmk/ruck-rover?orgId=1 | 16:49 |
| *** myoung is now known as myoung|lunch | 16:50 | |
| weshay | rlandy|rover, arxcruz|ruck must be infra related http://logs.openstack.org/97/571897/2/gate/tripleo-ci-centos-7-scenario002-multinode-oooq-container/f84eb5d/logs/ | 16:50 |
| rlandy|rover | looking | 16:51 |
| weshay | rlandy|rover, fyi.. 2018-06-05 17:36:58Zuul job-output.txt files were incomplete if at any point the job stopped producing logs for more than 5 seconds. This happened due to a timeout in the log streaming daemon. This has been fixed and the zuul executors have been restarted. Jobs running after now should have complete logs. | 16:52 |
| weshay | rlandy|rover, unrelated to gate failures | 16:52 |
| weshay | but this is the dealio http://38.145.34.131:3000/d/pgdr_WVmk/ruck-rover?orgId=1 | 16:52 |
| rlandy|rover | weshay: ok - looking through more carefully - | 16:53 |
| rlandy|rover | the ones failing all on the same job worry me less | 16:54 |
| *** brault has quit IRC | 16:55 | |
| rlandy|rover | and I already closed the bug about the log streaming going awol | 16:55 |
| rlandy|rover | RPC failed; result=35, HTTP code = 0", "fatal: The remote end hung up unexpectedly"], "stdout": "Cloning into '/home/zuul/DLRN/rdoinfo'...\n", "stdout_lines": ["Cloning into '/home/zuul/DLRN/rdoinfo'..."]} | 16:56 |
| rlandy|rover | sagi already has a recheck on that job | 16:57 |
| weshay | k | 17:00 |
| *** kopecmartin has quit IRC | 17:02 | |
| rlandy|rover | weshay: what do you trust sova or new dashboard - they don't seem to correlate exactly | 17:03 |
| *** trown|lunch is now known as trown | 17:03 | |
| rlandy|rover | unless the timezones are taken differently | 17:03 |
| rlandy|rover | they don't match up | 17:04 |
| rlandy|rover | the times look was behind | 17:07 |
| chandankumar | weshay: rlandy|rover where overcloud_container_prep happens? in TQE or TQ or tht | 17:08 |
| rlandy|rover | chandankumar: https://github.com/openstack/tripleo-quickstart-extras/tree/master/roles/overcloud-prep-containers | 17:09 |
| rlandy|rover | ^^ you talking about that? | 17:09 |
| rlandy|rover | not sure what "happens means" | 17:09 |
| arxcruz|ruck | weshay: those failures are related to github down this morning | 17:10 |
| arxcruz|ruck | rlandy|rover: ^ | 17:10 |
| chandankumar | rlandy|rover: I need to update the tempest container when there is any package related to it gets updated? | 17:11 |
| chandankumar | rlandy|rover: I am not sure where to make changes | 17:11 |
| weshay | k | 17:11 |
| arxcruz|ruck | weshay: this was the first outage after M$ aquisition ;) | 17:11 |
| weshay | lolz | 17:11 |
| rlandy|rover | it hasn't even happened yet | 17:12 |
| arxcruz|ruck | rlandy|rover: reflections already :P | 17:12 |
| *** bogdando has quit IRC | 17:12 | |
| arxcruz|ruck | funny fact for people moving to gitlab: | 17:12 |
| arxcruz|ruck | whois $(dig +short gitlab\.com) | egrep -A3 OrgName | 17:12 |
| rlandy|rover | chandankumar: so I am not sure if I am answering your question ... but when CI runs, we call two roles | 17:13 |
| weshay | arxcruz|ruck, ya.. was going to look at that this morning | 17:13 |
| weshay | arxcruz|ruck, /me wonders if the dos attacks spiked after the announcement | 17:13 |
| rlandy|rover | the build-test role that builds from ZUUL_CHANGES and then the prep-containers role picks up the latest repos | 17:14 |
| rlandy|rover | does a yum update and updates containers | 17:14 |
| rlandy|rover | getting script | 17:14 |
| arxcruz|ruck | weshay: well, there were a huge migration of repos from github to gitlab, that might made github went down, afterall, it increase the trafic on both sides | 17:14 |
| rlandy|rover | we call https://pypi.org/project/container-check/ | 17:15 |
| rlandy|rover | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-prep-containers/templates/overcloud-prep-containers.sh.j2#L107 | 17:16 |
| rlandy|rover | whether that addresses you concern, I am not sure | 17:16 |
| rlandy|rover | so iow, if a new repo is create by build-test-packages, yum update and container-check should pick it up | 17:17 |
| chandankumar | container-update.yaml from where it is comign | 17:17 |
| rlandy|rover | let's go look at a job for an example - that get built | 17:18 |
| chandankumar | arxcruz|ruck: https://review.openstack.org/#/c/572697/ -> barbican merged | 17:19 |
| arxcruz|ruck | chandankumar: I saw, waiting for jobs on our side to confirm, then i'll close the bug :) | 17:19 |
| chandankumar | arxcruz|ruck: need to fix our tempest image job | 17:20 |
| chandankumar | rlandy|rover: in tripleo-common tempest container is there | 17:23 |
| chandankumar | rlandy|rover: so may be I need to take a look what wrong is happening there | 17:24 |
| rlandy|rover | chandankumar: are you looking at a particular job? | 17:24 |
| chandankumar | rlandy|rover: since these containers are used on undercloud and how it is related and updated there | 17:24 |
| chandankumar | rlandy|rover: https://review.openstack.org/#/c/570524/ -> http://logs.openstack.org/24/570524/2/check/tripleo-ci-centos-7-containers-multinode/0f326e8/ | 17:25 |
| rlandy|rover | I don;t see it in the jobs I looked at | 17:25 |
| rlandy|rover | containers-default-parameters.yaml | 17:28 |
| chandankumar | rlandy|rover: I am not seeing container is getting | 17:28 |
| chandankumar | updated | 17:28 |
| chandankumar | weshay: myoung|lunch regarding this thread [openstack-program] Closing remaining OSP11 bugs tempest have 1 bug | 17:30 |
| chandankumar | we cannot update that bug because ocata internal tempest repo get messed up due to certification bug issue | 17:31 |
| chandankumar | I am thinking to close ot | 17:31 |
| chandankumar | it | 17:31 |
| *** holser__ has quit IRC | 17:32 | |
| weshay | arxcruz|ruck, or chandankumar could use your input on a keystone only tempest config in https://review.openstack.org/572830 | 17:35 |
| chandankumar | weshay: since standalone is supported from rocky na? | 17:36 |
| weshay | chandankumar, it's just landing now | 17:38 |
| weshay | for upstream | 17:38 |
| weshay | not downstream | 17:38 |
| chandankumar | weshay: so i would remove till queens related part | 17:38 |
| weshay | chandankumar, not 100% sure which section you are talking about.. please point it out in the review | 17:40 |
| chandankumar | weshay: left few comment | 17:40 |
| chandankumar | weshay: where can i find standalone.yaml? | 17:43 |
| weshay | in tht | 17:44 |
| weshay | https://github.com/openstack/tripleo-heat-templates/blob/master/roles/Standalone.yaml | 17:44 |
| chandankumar | weshay: cool, we are good there | 17:45 |
| chandankumar | weshay: I will try tomorrow | 17:45 |
| weshay | chandankumar, :) | 17:45 |
| arxcruz|ruck | chandankumar: weshay sorry, miss the fun, was having dinner | 17:45 |
| chandankumar | arxcruz|ruck: weshay https://review.openstack.org/#/c/572094/ -> http://logs.openstack.org/94/572094/2/check/tripleo-ci-centos-7-containers-multinode/3bf57de/ | 17:46 |
| chandankumar | any help on this one is appreciated | 17:46 |
| weshay | k | 17:47 |
| weshay | 2018-06-06 14:38:13 | ls: cannot access /home/zuul/python-tempestconf: No such file or directory | 17:48 |
| * weshay looks at the code | 17:48 | |
| chandankumar | weshay: donot know why clone is not working | 17:49 |
| weshay | umm m$ | 17:49 |
| weshay | ? | 17:49 |
| chandankumar | weshay: we are using git.openstack.org there | 17:50 |
| weshay | I saw | 17:50 |
| weshay | that is the right thing.. | 17:50 |
| weshay | github.com should not be in our code anywhere.. and that was true 6months ago | 17:50 |
| weshay | really just joking around though | 17:50 |
| chandankumar | weshay: people are saying lots of things related to accquastion on twitter | 17:50 |
| weshay | meh | 17:50 |
| chandankumar | it is fun to watch | 17:50 |
| arxcruz|ruck | chandankumar: i'll take a look on this tomorrow, now it's heroes of the storm time :D | 17:51 |
| weshay | ya.. maybe I'm not seeing it.. but I don't see how m$ has any more power to muc around w/ opensource w/ github on their servers | 17:51 |
| chandankumar | arxcruz|ruck: weshay rlandy|rover quiquell|off myoung|lunch panda https://dgplug.org/summertraining18/ if someone wants to get started in open source feel free to join | 17:52 |
| chandankumar | feel free to share it | 17:52 |
| weshay | http://logs.openstack.org/94/572094/2/check/tripleo-ci-centos-7-containers-multinode/3bf57de/job-output.txt.gz#_2018-06-06_14_36_25_465132 | 17:53 |
| chandankumar | yup task runned but there is no dir there | 17:54 |
| rlandy|rover | chandankumar: cool - thanks for the link | 17:54 |
| *** amoralej is now known as amoralej|off | 17:54 | |
| weshay | chandankumar-- | 17:55 |
| hubbot | weshay: chandankumar's karma is now 3 | 17:55 |
| weshay | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/tasks/tempest-venv.yml#L18 | 17:55 |
| chandankumar | weshay: check the dependent patch | 17:55 |
| weshay | ah k | 17:56 |
| weshay | chandankumar, 404 https://git.openstack.org/openstack/python-tempestconf | 17:56 |
| chandankumar | weshay: tempest have worked | 17:56 |
| chandankumar | weshay: do git clome locally it will worl | 17:57 |
| weshay | arxcruz|ruck, we should add a linter to tq/tqe to flag github | 17:57 |
| weshay | chandankumar, ya.. same for eme | 17:58 |
| weshay | for me | 17:58 |
| weshay | chandankumar, I wonder if it's been populted in the infra mirrors | 17:58 |
| chandankumar | weshay: do we need a flag tp use zuul local mirror | 17:59 |
| chandankumar | ? | 17:59 |
| weshay | no.. but I think it's a proxy.. so a job has to try once at least | 17:59 |
| weshay | not sure how many times you've tried this | 18:00 |
| rlandy|rover | chandankumar: so looking at where tempest is missing ... there are two source that get passed to openstack overcloud container image prepare ... https://github.com/openstack/tripleo-heat-templates/blob/master/environments/docker.yaml and docker.io/tripleoqueens | 18:02 |
| rlandy|rover | and .. package for update ... which considers the gating_repo | 18:03 |
| chandankumar | rlandy|rover: let me add a patch there in tht | 18:03 |
| rlandy|rover | %gatestatus | 18:05 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/567224 | 18:05 |
| chandankumar | !gatestatus | 18:07 |
| openstack | chandankumar: Error: "gatestatus" is not a valid command. | 18:07 |
| rlandy|rover | with a % | 18:08 |
| chandankumar | rlandy|rover: yup | 18:08 |
| rlandy|rover | weshay: any other gate/check failures you are concerned about - looking at http://cistatus.tripleo.org | 18:10 |
| weshay | nope.. just the ones from the dashboard | 18:10 |
| weshay | I saw the flood, wasn't sure why | 18:11 |
| weshay | arxcruz|ruck, I wasn't kidding w/ lint failures for finding github though | 18:11 |
| weshay | rlandy|rover, ya.. 82% today | 18:11 |
| weshay | http://cistatus.tripleo.org/gates/ | 18:11 |
| rlandy|rover | the dashboard doesn't seem to sync for me | 18:11 |
| weshay | hopefully that will come back up now | 18:11 |
| rlandy|rover | the failures/times etc | 18:12 |
| rlandy|rover | but whatever | 18:12 |
| weshay | rlandy|rover, sync? | 18:12 |
| weshay | want to show me? | 18:12 |
| rlandy|rover | times don't match up | 18:12 |
| weshay | oh | 18:12 |
| rlandy|rover | 2018-06-06 11:48:46 | 18:12 |
| rlandy|rover | what timezone is that? | 18:12 |
| weshay | rlandy|rover, well.. each one has a link to the log | 18:12 |
| weshay | not sure re: timezone | 18:12 |
| rlandy|rover | yeah but there is more recent info | 18:13 |
| rlandy|rover | sova is showing results from 16:00+ | 18:13 |
| weshay | rlandy|rover, ur saying http://38.145.34.131:3000/d/pgdr_WVmk/ruck-rover?orgId=1 does not have the most recent? | 18:13 |
| rlandy|rover | would seem | 18:13 |
| weshay | sova only pulls data.. once per hour | 18:13 |
| rlandy|rover | plus even if I look back, the results time don't correlate | 18:13 |
| weshay | so imho.. for the latest info I go to http://38.145.34.131:3000/d/pgdr_WVmk/ruck-rover?orgId=1 | 18:14 |
| weshay | and for the trend I go to sova | 18:14 |
| rlandy|rover | then it's missing the latest failures | 18:15 |
| rlandy|rover | but really we have other stuff to chase atm | 18:15 |
| chandankumar | rlandy|rover: weshay arxcruz|ruck https://review.openstack.org/#/c/572858/ | 18:16 |
| *** myoung|lunch is now known as myoung | 18:18 | |
| rlandy|rover | we should see tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates kick but may want to test with an actual depends-on tempest change | 18:18 |
| myoung | oooo..i just discovered the shortcut cheat sheet for http://38.145.34.131:3000/d/pgdr_WVmk/ruck-rover?orgId=1 --> hit '?' | 18:22 |
| myoung | chandankumar: re: that course starting next week...neat! thx | 18:23 |
| rlandy|rover | and we're back with phase 1 "no valid hosts fun" https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-ocata-rdo_trunk-minimal-366/undercloud/home/stack/overcloud_deploy.log.gz | 18:25 |
| chandankumar | myoung: it is free of cost, the attendee need a linux box with decent internet connection irc client and hunger to learn, the training will end in 3 montgs with some your own coll foss projects | 18:26 |
| chandankumar | and with lots of virtual friends | 18:26 |
| rlandy|rover | rfolco: weshay: left a comment on https://review.openstack.org/#/c/572187 - of it's urgent to vote, let me know | 18:29 |
| *** dtantsur is now known as dtantsur|afk | 18:30 | |
| rlandy|rover | if | 18:30 |
| rlandy|rover | or if you never expect a pass | 18:30 |
| weshay | rlandy|rover, it was passing, but now it's not | 18:30 |
| weshay | see my comment | 18:30 |
| weshay | we'd like to get in check, so it's more visible | 18:30 |
| hubbot | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/567224 | 18:32 |
| rlandy|rover | container - probably unrelated | 18:33 |
| weshay | rfolco, how is this going? https://trello.com/c/flI683EI/774-ci-job-create-job-37-work-on-queens-and-calls-tripleo-upgrade-updates-workflow | 18:34 |
| weshay | myoung, have you started testing this https://trello.com/c/6tcD7ilr/778-injecting-zuul-changes-at-various-points-in-the-job-workflow ? | 18:35 |
| rlandy|rover | rasca: still around? | 18:35 |
| weshay | trown, we need a QE on https://trello.com/c/ZlaiOAZ2/780-change-the-emit-release-script-as-necessary | 18:36 |
| rfolco | weshay, I believe it is the missing user/group (zuul --> docker), steve is assigned to this bug | 18:36 |
| weshay | rfolco, no.. no | 18:36 |
| weshay | 37 is update | 18:37 |
| weshay | not upgrade bud | 18:37 |
| weshay | rfolco, get into those zuul changes w/ quiquell|off | 18:37 |
| trown | weshay: ya... quiquell|off has done some reviews and is familiar with the script | 18:37 |
| weshay | fak | 18:37 |
| rfolco | aaah | 18:37 |
| weshay | he's got too many cards | 18:37 |
| rfolco | meh, crossed stations again | 18:37 |
| weshay | rfolco, let's go brotha | 18:37 |
| rfolco | getting close to 40 | 18:38 |
| rfolco | its a syndrom | 18:38 |
| weshay | rfolco, ha ha.. don't claim old man w/ me | 18:38 |
| rfolco | haha | 18:38 |
| rfolco | ok ok my bad | 18:38 |
| rfolco | we should be good, just double checking | 18:38 |
| myoung | weshay: last night I put out what (I think?) are the cases to cover to test for https://trello.com/c/6tcD7ilr/778-injecting-zuul-changes-at-various-points-in-the-job-workflow, was hoping to catch quiquell|off before he was off to confirm but didn't get him today | 18:38 |
| weshay | myoung, you need quiquell|off to get started? | 18:39 |
| weshay | why | 18:39 |
| myoung | weshay: been walking thru https://trello.com/c/ZlaiOAZ2/780-change-the-emit-release-script-as-necessary patches as well (about to +1 on them, wanted to understand more fully the actual script as it's related) | 18:39 |
| weshay | myoung, if you are working on it.. you think you can assign yourself as qe? | 18:40 |
| myoung | weshay: don't need to wait to get started, we had talked about the test plan for that card and cases and just wanted to sync / confirm I had captured things correctly | 18:40 |
| weshay | as there is none atm | 18:40 |
| weshay | myoung, meh | 18:40 |
| weshay | let's review your card | 18:40 |
| myoung | weshay: sure | 18:40 |
| myoung | re: QE for https://trello.com/c/ZlaiOAZ2/780-change-the-emit-release-script-as-necessary | 18:40 |
| weshay | myoung, re: https://trello.com/c/6tcD7ilr/778-injecting-zuul-changes-at-various-points-in-the-job-workflow | 18:41 |
| weshay | I think some of these items are available to be poked at right now | 18:41 |
| myoung | weshay: ack | 18:41 |
| weshay | multiple patches from the same release | 18:41 |
| weshay | the from / to | 18:41 |
| rlandy|rover | whoever was on the pomoter server tmux, switched back to the script execution | 18:42 |
| rlandy|rover | watching it | 18:42 |
| weshay | rlandy|rover, k | 18:42 |
| rlandy|rover | weshay: sorry if you were busy there | 18:43 |
| weshay | nope.. not on it | 18:43 |
| weshay | rlandy|rover, fyi.. we have tripleo-ci-centos-7-scenario008-multinode-oooq-container in non-voting check | 18:46 |
| weshay | rlandy|rover, we should add that to promotion pipeline ( need to check branches ) | 18:46 |
| weshay | rlandy|rover, so.. added to promotion pipeline, but not criteria until it's voting | 18:47 |
| weshay | make sense? | 18:47 |
| weshay | trown, can you think of any better ways to do this? https://review.openstack.org/#/c/572736/ | 18:48 |
| rlandy|rover | weshay: add to promotion pipeline but not to promotion criteria? ok - can submit that review | 18:49 |
| weshay | thanks | 18:49 |
| weshay | rfolco, can you join me in blue | 19:01 |
| rfolco | weshay, yep | 19:01 |
| myoung | weshay: just got my head around https://review.openstack.org/#/c/572736 and your comment, other than needing to be able to run this multiple times (for multiple patches) and the issue of needing to differentiate the previous repo tar.gz, are there other issues? or were referring to your -1 comment? | 19:01 |
| trown | weshay: I am confused by that patch | 19:24 |
| trown | weshay: if it exists... we copy it, but then immediately delete the copy? | 19:28 |
| rfolco | weshay, install and target releases are the same (queens), is it right? isn't an upgrade job ? | 19:29 |
| *** atoth has quit IRC | 19:32 | |
| weshay | trown, ya.. I need to discuss the issue w/ someone | 19:37 |
| weshay | trown, so here's what we're concerned about | 19:37 |
| weshay | queens -> master upgrade | 19:37 |
| weshay | trown, meh.. can I do this on blue? | 19:38 |
| weshay | it's a lot to type | 19:38 |
| weshay | I think I have the issue in the vard | 19:38 |
| weshay | card | 19:38 |
| weshay | after discussing this w/ @quiquellorente we need to be sure to clean/archive rpm's in the gating repo that != the current branch. | 19:38 |
| weshay | In the example of a queens -> master upgrade w/ zuul changes from both queens and master. | 19:38 |
| weshay | In the workflow, when we're at master, we'll have a queens patch built and available in the gating repo that will get installed on top of master rpms due to the higher epoch of dlrn built rpms. | 19:38 |
| weshay | Solution: clean and archive any rpms built for branches != $release | 19:38 |
| weshay | trown, ^ | 19:38 |
| weshay | the patch built for queens.. could get installed on the master run.. it's in the gating.repo | 19:39 |
| arxcruz|ruck | weshay: rlandy|rover trown panda https://review.openstack.org/#/c/572709/ this fix a promotion blocker | 19:40 |
| weshay | nice arxcruz|ruck | 19:42 |
| rlandy|rover | looking | 19:43 |
| weshay | trown, make sense? | 19:43 |
| trown | weshay: the issue makes sense but still not the patch... | 19:45 |
| weshay | trown, so this role runs multiple times | 19:46 |
| weshay | if it finds a previously built gating repo, it's archived for the logs | 19:47 |
| weshay | and then removed, so a fresh gating repo w/ only patches from the current branch built | 19:47 |
| weshay | arxcruz|ruck, are we seeing the problem only in promotions? https://review.rdoproject.org/jenkins/job/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch/ | 19:48 |
| arxcruz|ruck | weshay: well, in any patch running queens jobs | 19:49 |
| arxcruz|ruck | weshay: for some reason, only queens run that task | 19:49 |
| weshay | arxcruz|ruck, hrm | 19:49 |
| weshay | arxcruz|ruck, ya.. but the history on fs001 queens is good | 19:50 |
| weshay | it's not failing | 19:50 |
| arxcruz|ruck | weshay: hmmm, but still, it's failing in the promotion :/ | 19:51 |
| arxcruz|ruck | in my patch fail on queens | 19:52 |
| weshay | arxcruz|ruck, ya.. k k.. | 19:52 |
| arxcruz|ruck | so checks that run queens job it will fail | 19:52 |
| weshay | arxcruz|ruck, if your patch works in https://review.rdoproject.org/jenkins/job/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch/2178/console | 19:52 |
| weshay | then we're still good | 19:52 |
| weshay | arxcruz|ruck, weird that it's only running on queens | 19:53 |
| arxcruz|ruck | weshay: weird now is on this job, the task was skipped | 19:53 |
| weshay | rlandy|rover, reminder re: pto cal | 20:03 |
| *** holser__ has joined #oooq | 20:04 | |
| rlandy|rover | weshay: I added it to RHOS PTO/Travel - did I miss a diff pto cal? | 20:04 |
| weshay | ah see it now | 20:05 |
| weshay | approved, thanks | 20:05 |
| rlandy|rover | cool - thanks | 20:05 |
| rlandy|rover | weshay: btw added for your review https://review.rdoproject.org/r/#/c/14093/ - just looking at what config-check is failing on | 20:06 |
| trown | weshay: ok, I have looked through a bunch of the CI on https://review.openstack.org/572736 only to realize it doesnt actually do anything on existing jobs... but I am very unconvinced it will do anything on n->n+1 jobs too | 20:25 |
| trown | I commented as such on the patch... not sure how/if it has been tested | 20:25 |
| weshay | trown, thanks.. let me know if you think of a better way to prevent the rpm getting installed where it should not | 20:30 |
| trown | weshay: I think we should just be disabling it | 20:31 |
| trown | then it is still archived | 20:31 |
| hubbot | All check jobs are working fine on stable/ocata, stable/ocata, master, stable/queens. | 20:32 |
| weshay | trown, if we just disable it. it will get re-enabled in the upgrade step | 20:33 |
| weshay | trown, not sure if we're communicating about the potential issue | 20:33 |
| trown | weshay: I do understand the issue... probably not communicating the idea well though | 20:34 |
| weshay | k k | 20:34 |
| weshay | trown, only one yum repo is created for any zuul changes right? | 20:34 |
| weshay | so if we disable it.. then build the next patch on $other branch.. I don't see how that would help | 20:35 |
| trown | weshay: we need to have the release of the change somehow available to the install-built-repo role, and then only enable the repo if it matches the release we are currently on | 20:35 |
| weshay | trown, sure.. but if you have both a queens and master zuul change | 20:35 |
| weshay | what then? | 20:35 |
| weshay | I could see us making one yum repo per zuul change | 20:36 |
| weshay | and naming them such.. w/ the hash.. so they can be easily disabled | 20:36 |
| trown | ya i dont see how the current patch would ever address changes on multiple branches | 20:36 |
| trown | that would just be broken without splitting at least one repo per branch | 20:37 |
| weshay | well.. it's not great, but I think works in that... if you had a queens and master zuul change | 20:37 |
| weshay | when upgrade starts.. the gating.repo is cleared.. build-test-packages runs and the master patch is built and queens is not | 20:37 |
| weshay | so it works | 20:38 |
| trown | hmm ya... so all we really need is for https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/install-built-repo/templates/install-built-repo.sh.j2#L8-L15 to have the release var in it | 20:38 |
| trown | and then we only enable the repos for the release we are on | 20:39 |
| weshay | trown, hrm.. well if we changed line 11 to be | 20:39 |
| weshay | file:///opt/queens_gating_repo | 20:40 |
| trown | in any case... I think the solution should be adding more information to the dlrn roles rather than taking away information from the CI artifacts | 20:40 |
| weshay | then we could just disable it | 20:40 |
| weshay | ok | 20:40 |
| trown | right | 20:40 |
| weshay | trown, I'm a little paranoid of this.. because if the queens patch did install on master | 20:41 |
| trown | we have the release when the role runs... we can just add it in | 20:41 |
| weshay | that would be really hard to debug if it caused problems | 20:41 |
| trown | ya | 20:41 |
| weshay | so.. I think what we're saying is | 20:41 |
| weshay | anytime we create or use gating_repo it needs to have the branch name on it | 20:41 |
| trown | ya I think that is the best way to have it segregated | 20:41 |
| weshay | and the role needs to disable all non-$release gating repos prior to starting if they exist | 20:42 |
| weshay | trown, ok.. thanks for talking through that | 20:42 |
| trown | ya, probably just disable all *gating repos, then enable the one we want | 20:42 |
| weshay | ya | 20:42 |
| trown | then we have all the repos without any "back ups" | 20:43 |
| trown | could probably even add other information to the repos, like what playbook it is from | 20:43 |
| trown | but release would be enough for the issue I think | 20:44 |
| rlandy|rover | fyi ... config-check error is an infra thing | 20:44 |
| * myoung is catching up on scrollback from trown/weshay, he was nose down making patches to try to excercise this in CI from upgrade job | 20:45 | |
| * myoung just caught up | 20:47 | |
| myoung | trown, weshay, I like having discreet repos with release+hash encoded into them (but disabled) so all the artifacts are there and digestable/diagnosable | 20:47 |
| trown | I will try to get undercloud n->n+1 upgrade job posted, so we can see what any solution actually does | 20:50 |
| weshay | ok.. comments posted to https://review.openstack.org/#/c/572736/2 | 20:52 |
| weshay | myoung, just the release | 20:53 |
| weshay | not the hash | 20:53 |
| *** trown is now known as trown|outtypewww | 20:57 | |
| myoung | weshay: ack, thanks for the detail in gerrit | 20:58 |
| myoung | weshay: will rebase and/or reparent the patches I just finished making when we address your comments | 20:59 |
| myoung | for now just letting the jobs run as it's still useful data to have later tonight/tomorrow | 20:59 |
| * myoung prepares to depart to pick up his teenager, and bring her --> https://goo.gl/images/rwAwt4 | 21:00 | |
| *** holser___ has joined #oooq | 21:02 | |
| *** holser__ has quit IRC | 21:06 | |
| *** myoung is now known as myoung|off | 21:06 | |
| rlandy|rover | arxcruz|ruck: still around? | 21:28 |
| rlandy|rover | probably the middle of the night there :) | 21:28 |
| *** holser___ has quit IRC | 21:32 | |
| rlandy|rover | weshay: ok - rdo infra problem with config check was fixed ... https://review.rdoproject.org/r/#/c/14093/ is ready | 21:59 |
| hubbot | All check jobs are working fine on stable/ocata, stable/ocata, master, stable/queens. | 22:32 |
| *** rlandy|rover is now known as rlandy|rover|bbl | 22:37 | |
| *** tosky has quit IRC | 23:31 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!