| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci-centos-7-scenario009-multinode-oooq @ https://review.openstack.org/560445, (1 more message) | 00:52 |
|---|---|---|
| *** rlandy has quit IRC | 00:58 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci-centos-7-scenario009 (1 more message) | 02:52 |
| *** jaganathan has joined #oooq | 03:17 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci-centos-7-scenario009 (1 more message) | 04:52 |
| *** apetrich has joined #oooq | 05:06 | |
| *** jtomasek has joined #oooq | 05:17 | |
| *** chem has quit IRC | 05:29 | |
| *** jfrancoa has joined #oooq | 05:33 | |
| *** quiquell|off is now known as quiquell | 05:46 | |
| quiquell | Good morning here too | 05:48 |
| quiquell | sshnaidm: Good morning sir | 05:50 |
| quiquell | sshnaidm: I think this is wrong jobs https://review.rdoproject.org/r/#/c/16827 | 05:52 |
| *** ratailor has joined #oooq | 05:53 | |
| *** ratailor has quit IRC | 06:28 | |
| *** ratailor has joined #oooq | 06:35 | |
| *** quiquell is now known as quiquell|brb | 06:37 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci-centos-7-scenario009 (1 more message) | 06:53 |
| *** kopecmartin|off is now known as kopecmartin|ruck | 07:02 | |
| *** quiquell|brb is now known as quiquell | 07:09 | |
| *** tosky has joined #oooq | 07:13 | |
| *** jtomasek has quit IRC | 07:16 | |
| *** jtomasek has joined #oooq | 07:17 | |
| *** bogdando has joined #oooq | 07:29 | |
| bogdando | o/ please merge https://review.openstack.org/#/c/593103/ | 07:31 |
| bogdando | it is needed by https://review.openstack.org/#/c/576746/ | 07:31 |
| *** chandankumar has joined #oooq | 07:40 | |
| *** skramaja has joined #oooq | 07:52 | |
| quiquell | ssbarnea_: I am playing around with the f28 | 07:57 |
| quiquell | ssbarnea_: Have upload stuff about yaml to tripleo-ci | 07:57 |
| ssbarnea_ | quiquell: thanks! i back and ready to continue on f28 task now. | 07:59 |
| quiquell | ssbarnea_: Going to remove some Depends-On already merged | 08:00 |
| quiquell | ssbarnea_: Will help you while I am waitting for promotion testings | 08:00 |
| ssbarnea_ | quiquell: yep, good idea, i already did some. there is still a problem that is kinda serious related to py3 rpm packages | 08:01 |
| quiquell | ssbarnea_: Yep, I have a fedora28 at my laptop, even installing python-yaml shyaml is not working | 08:01 |
| quiquell | ssbarnea_: pip install works tough | 08:01 |
| ssbarnea_ | fc28 is a py3 first OS, and most py packages have different names. | 08:01 |
| quiquell | ssbarnea_: Humm I am using python2 | 08:02 |
| *** skramaja_ has joined #oooq | 08:02 | |
| quiquell | ssbarnea_: I am looking at the job here https://review.openstack.org/#/c/595374/ | 08:02 |
| quiquell | ssbarnea_: is that ok ? | 08:02 |
| ssbarnea_ | my original approach was to use "yum python*-foo" but is wrong, as this may easily install 2,3,4 different python distros! | 08:02 |
| *** skramaja has quit IRC | 08:02 | |
| quiquell | ssbarnea_: Can we go with pip install shyaml ? | 08:03 |
| ssbarnea_ | quiquell: only as a temp measure until we address the rpm distrib, but overall yes. feel free to put a TODO item to move to rpm when possible. | 08:04 |
| quiquell | ssbarnea_: Going to check at my fs28 laptop | 08:04 |
| *** amoralej|off is now known as amoralej | 08:04 | |
| quiquell | ssbarnea_: shyaml has this shebag !/usr/bin/python3 -s | 08:05 |
| quiquell | This is the issue we are forced to install python3-yaml | 08:05 |
| ssbarnea_ | quiquell: the shebang is part of the installer pip, so it will use the same interpreter | 08:05 |
| quiquell | ssbarnea_: I installed with dnf | 08:06 |
| ssbarnea_ | quiquell but if you want to enforce py3 do : "python3 -m pip install ... " which will assure it forces use of py3. | 08:06 |
| tosky | can anyone please check this change to tempestmail configuration where all affected people added +1 already? https://review.openstack.org/#/c/590298/ | 08:06 |
| quiquell | ssbarnea_: I have a python2 fedora28 but it install shyaml that enforces python3 looks like shyaml is a python3 only thing | 08:07 |
| ssbarnea_ | quiquell: correct me if I am wrong, but default cloud image comes only with py3 instaled by default and our expectation is to use py3 and not to install py3, right? | 08:07 |
| ssbarnea_ | this means that you should use a clean image for testing, not one with both pythons. | 08:08 |
| quiquell | ssbarnea_: Puff no clue, I only know that shyaml has a python3 shebag | 08:08 |
| quiquell | ssbarnea_: installed with dnf install -y python-shyaml | 08:08 |
| ssbarnea_ | quiquell: i have a vm running on rdo, if you want. | 08:08 |
| quiquell | ssbarnea_: Let's make this work, we can choose proper image later | 08:09 |
| quiquell | ssbarnea_: Damn I want to put standalone at every openstack project so ruck/rover is not so painful | 08:09 |
| quiquell | ssbarnea_: Looks like this f28 can help with that ? | 08:09 |
| ssbarnea_ | quiquell: i need to work on dealing with the bug where tq(e) is downgraded on CI, this prevented me to test some jobs. | 08:10 |
| ssbarnea_ | yep | 08:10 |
| quiquell | ssbarnea_: what bug ? | 08:11 |
| ssbarnea_ | https://bugs.launchpad.net/tripleo/+bug/1796048 | 08:11 |
| openstack | Launchpad bug 1796048 in tripleo "quiskstart.sh installs downgraded version of tripleo-quickstart-extras" [High,In progress] - Assigned to Sorin Sbarnea (ssbarnea) | 08:11 |
| quiquell | sshnaidm: Don't know why is this failing at merge https://review.rdoproject.org/r/#/c/16806/ | 08:14 |
| quiquell | ssbarnea_: What's the relation of the bug and standalone-f28 = | 08:20 |
| quiquell | ? | 08:20 |
| ssbarnea_ | quiquell i got some changes that failed because they ended using tripleo code that was like ~150 changesets old | 08:21 |
| quiquell | ssbarnea_: What changes do you have the review ? | 08:21 |
| ssbarnea_ | in fact this bug seems to randomly affects lots of jobs and is very hard to spot as you need to dig the logs to figure it out. in fact Wes spotted it first. | 08:22 |
| quiquell | ssbarnea_: This is the review ? | 08:23 |
| quiquell | https://review.openstack.org/#/c/606944/ | 08:24 |
| quiquell | ssbarnea_: but the issue are the other jobs | 08:24 |
| quiquell | They don't do the uninstalling | 08:25 |
| ssbarnea_ | quiquell in am not sure about the unintalling, btw, do you want to BJ about it, it may prove faster | 08:25 |
| quiquell | ssbarnea_: Sure bored of waiting promotions | 08:26 |
| ssbarnea_ | give me few mins to get a coffee | 08:26 |
| quiquell | ssbarnea_: before you go do we need to merge this https://review.openstack.org/#/c/606944/ ? | 08:27 |
| *** fhubik has joined #oooq | 08:27 | |
| ssbarnea_ | yes, but only once we get it passing. | 08:28 |
| quiquell | ok, so bug is not a stopper | 08:28 |
| ssbarnea_ | this assures that we succesfully moved the config to their new location | 08:28 |
| arxcruz | sshnaidm: around? any chance you know where the upload-logs role is for http://logs.rdoproject.org/19/605419/10/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens-branch/c22c166/job-output.txt.gz | 08:28 |
| ssbarnea_ | i am hacking my f28 test machine by removing py2 and assuring that yum cannot install it, just to be be able to test | 08:29 |
| quiquell | ssbarnea_: ack | 08:29 |
| ssbarnea_ | this is going to become a big PITA, we no longer have "python" or "pip" commands available, making any scripts more complex to get the right interpreter. | 08:33 |
| quiquell | ssbarnea_: where ? | 08:34 |
| panda | quiquell: how's going ? | 08:39 |
| quiquell | panda: good morning | 08:39 |
| quiquell | panda: we have promotion-blockers for OVB, but I saw mutlinodes working :-) | 08:39 |
| quiquell | panda: Wiped it out after a recheck :-( so waitting for test review to finish | 08:39 |
| panda | quiquell: so they are failing in a known way | 08:40 |
| quiquell | panda: Also compared check jobs, I think they are doing the stuff | 08:40 |
| quiquell | panda: yep | 08:40 |
| quiquell | panda: Added Depends-On on the upstream fix https://review.rdoproject.org/r/#/c/16806/ | 08:41 |
| quiquell | panda: Also added a test patch to run queens/pike periodics | 08:41 |
| quiquell | panda: HUmm exception now :-( | 08:41 |
| panda | quiquell: we are testing only with your patch or we are spinning one periodic in pipeline a day at least ? | 08:42 |
| sshnaidm | quiquell, can we leave only debug patch there? I'd like to see if it starts at all | 08:42 |
| quiquell | panda: Just the patch, not periodics yet | 08:42 |
| sshnaidm | quiquell, I mean in dependencies | 08:42 |
| quiquell | sshnaidm: What do you mean ? | 08:43 |
| quiquell | We have an exception at periodic-tripleo-centos-7-master-containers-build periodic-tripleo-centos-7-master-containers-build | 08:43 |
| sshnaidm | quiquell, in https://review.rdoproject.org/r/#/c/16806/ to leave only dependency https://review.openstack.org/#/c/608166/ | 08:43 |
| sshnaidm | quiquell, did it work in patch 17? I don't see results.. | 08:44 |
| quiquell | sshnaidm: we need the Depends-On to run as periodics | 08:44 |
| quiquell | sshnaidm: I saw 'multinode' jobs passing, but I added a Depends-On to the fix of OVB | 08:44 |
| quiquell | sshnaidm: And results gone | 08:44 |
| sshnaidm | quiquell, you don't need https://review.openstack.org/#/c/608589/ | 08:45 |
| quiquell | sshnaidm: Cannot find them at zuul builds | 08:45 |
| quiquell | sshnaidm: why ? | 08:45 |
| *** chem has joined #oooq | 08:45 | |
| sshnaidm | quiquell, let's go step by step and be sure it starts and finishes.. let's put features aside for now | 08:45 |
| ssbarnea_ | quiquell: out of curiosity: should this be swapped, prefering dnf when installed? | 08:45 |
| ssbarnea_ | https://review.openstack.org/#/c/595374/35/playbooks/tripleo-ci/templates/oooq_common_functions.sh.j2 | 08:45 |
| sshnaidm | quiquell, yesterday it couldn't start at all because of various issues.. | 08:46 |
| quiquell | sshnaidm: Ok, let's remove the Depends-On, this morning was very greeny | 08:46 |
| quiquell | sshnaidm: Except for OVB | 08:46 |
| sshnaidm | quiquell, great, let's see it happens again, also we'll check that OVB reports its failure | 08:47 |
| quiquell | sshnaidm: done | 08:47 |
| sshnaidm | quiquell, cool, thanks | 08:47 |
| quiquell | sshnaidm: was reporting, In fact I found it there | 08:47 |
| sshnaidm | quiquell, it's good news! | 08:48 |
| quiquell | sshnaidm: Think we are near | 08:49 |
| quiquell | sshnaidm: But we have to check we don't have false possitives | 08:49 |
| sshnaidm | quiquell, well, yeah, and all other stuff like right branches, repos, etc | 08:50 |
| quiquell | sshnaidm, panda: Passing multinode https://logs.rdoproject.org/06/16806/17/check/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-master/aaeb043/job-output.txt.gz | 08:50 |
| quiquell | sshnaidm: If we check at logs dir we can get the latest | 08:50 |
| sshnaidm | cool, so last try worked yesterday | 08:51 |
| *** chandankumar has quit IRC | 08:51 | |
| quiquell | sshnaidm: weird thing zuul don't report it | 08:52 |
| quiquell | sshnaidm: but we have the log dir | 08:52 |
| quiquell | sshnaidm: Zuul mark them as SKIPPED even if they have pass | 08:52 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci-centos-7-scenario009 (1 more message) | 08:53 |
| sshnaidm | quiquell, maybe Sf folks know why.. | 08:53 |
| marios|rover | kopecmartin|ruck: fyi https://bugs.launchpad.net/tripleo/+bug/1796639 quiquell cc were you asking about queens earlier? undercloud-upgrades job has issue fyi if that is what you meant | 08:54 |
| openstack | Launchpad bug 1796639 in tripleo "[stable/queens] The tripleo-ci-centos-7-undercloud-upgrades is failing on queens for No such file or directory: '/etc/heat/policy.json" [High,Triaged] - Assigned to Marios Andreou (marios-b) | 08:54 |
| quiquell | marios|rover: But that's different from the bandini fix ' | 08:54 |
| quiquell | ? | 08:54 |
| marios|rover | quiquell: yeah i know oh i see you were asking if that issue was affecting queens | 08:55 |
| marios|rover | quiquell: i thought you were asking about state of queens gate | 08:55 |
| quiquell | marios|rover: nope if the issue affect queens | 08:55 |
| marios|rover | kopecmartin|ruck: but otherwise gates look ok ? \o/ | 08:56 |
| kopecmartin|ruck | marios|rover, yes they do, i was really surprised in the morning :D | 08:57 |
| *** jrist has joined #oooq | 08:57 | |
| marios|rover | kopecmartin|ruck: sssshhh :) lets not say it too often ;) | 08:57 |
| ssbarnea_ | i need some clarifications regarding f28 expectations for openstack: should we run using py3 and avoid py3 completly or we do a ....mix. | 09:00 |
| panda | ssbarnea_: as far as I understand pyt3 will be the only one available in f28, rhel8 and centos8 | 09:01 |
| panda | ssbarnea_: there's no mix, there's only zuul. | 09:01 |
| *** jrist has quit IRC | 09:02 | |
| ssbarnea_ | panda: my impression was the same, py3 only. zuul is an interesting question because based on the readme it does some messing, hopefully it can be conficed to use py3 instead on f28 node (willfindout soon) | 09:03 |
| panda | ssbarnea_: the second part of the answer was just a quote, ignore it | 09:04 |
| ssbarnea_ | to assure we don't mess things up we should disable python2* in /etc/dnf.conf as first step. | 09:04 |
| ssbarnea_ | panda: at least it should help us stop accidents fast, where some install command gets py2, because on f28 there is py2 as optional. | 09:05 |
| panda | ssbarnea_: uninstall --purge | 09:05 |
| ssbarnea_ | panda: uninstall does not prevent future install by some other random task. for example I discovered that "mc" installs python2 ... clearly a bug in the package nut i am sure is not the onlyone. | 09:06 |
| quiquell | panda: I looked into cnew OVB check job, they look ok, maybe we can start migrating them | 09:08 |
| quiquell | panda: I know it's not the plan | 09:08 |
| panda | quiquell: the plan is to migrate everythign we can, so we can understand in this week how much we are breaking | 09:11 |
| *** jrist has joined #oooq | 09:13 | |
| quiquell | panda: So we activate more jobs like this https://review.rdoproject.org/r/#/c/16781/5/zuul.d/tripleo.yaml ? | 09:14 |
| quiquell | panda: Or remove the legacy counterpart of working ones ? | 09:14 |
| quiquell | panda: and make new ones voting ? | 09:15 |
| panda | quiquell: they can't vote anyway | 09:16 |
| panda | qnd having both legacy and new ay be too much for rdocloud | 09:16 |
| quiquell | panda: What do you mean they can't vote ? we cannot set vote: true ? | 09:17 |
| panda | quiquell: 3rd party CI from OVB can't really vote on the job, patches can be merged even if all jobsin rdocloud fail | 09:19 |
| panda | quiquell: usually people take the result into consideration, but they are not blocked by them | 09:19 |
| quiquell | panda: Ahh ok, yep, so we can start to remove legacy of working ones and continue with next ones ? | 09:21 |
| panda | quiquell: I'd think so yes. | 09:21 |
| ssbarnea_ | quiquell: the mess is getting bigger: while -requests and -urllib3 are installable using python(2|3), yaml does not as the old rpm is named "python-yaml" and new one "python3-yaml",with no "2" alias available. | 09:39 |
| ssbarnea_ | quiquell: so we need to start adding conditionals for package names for each distro as there is no clear pattern to use. | 09:39 |
| ssbarnea_ | and we cannot use wildcards | 09:39 |
| *** sshnaidm is now known as sshnaidm|lnch | 09:40 | |
| panda | ssbarnea_: is there any way you can work with packaging team to solve these ? | 09:41 |
| ssbarnea_ | panda: yes and no, i did this in the past and in some cases it worked (takes weeks) and in some cases it may not as they would avoid changing an old version due to regression risks | 09:43 |
| ssbarnea_ | also we do not have time: so I would find a local workaround so we can continue work (and likely find other issues) and raise a ticket for this. once/if we had it fixed by packaging, we can remove our workaround. sounds ok? | 09:44 |
| ssbarnea_ | panda quiquell : take a look at what I need to run: https://gist.github.com/ssbarnea/860c50ee73f5018ea3b2a92475401ccb maybe you have an idea on how to avoid extra conditionals | 09:47 |
| ssbarnea_ | look that python-yaml does not have a "2" | 09:47 |
| *** dtantsur|afk is now known as dtantsur | 09:49 | |
| *** jfrancoa has quit IRC | 09:52 | |
| *** jfrancoa has joined #oooq | 09:57 | |
| *** sshnaidm|lnch is now known as sshnaidm | 09:59 | |
| *** jtomasek has quit IRC | 10:09 | |
| quiquell | sshnaidm, panda: Have to leave, will connect back | 10:20 |
| *** quiquell is now known as quiquell|brb | 10:20 | |
| panda | quiquell|brb: good luck | 10:20 |
| quiquell|brb | panda: thanks | 10:20 |
| *** quiquell|brb is now known as quiquell | 10:21 | |
| quiquell | not yet | 10:21 |
| *** ssbarnea_ has quit IRC | 10:24 | |
| quiquell | ssbarnea_: Why are we using quickstart.sh for the job ? | 10:24 |
| quiquell | ups has gone | 10:24 |
| *** jaosorior has joined #oooq | 10:26 | |
| *** jtomasek has joined #oooq | 10:28 | |
| quiquell | panda: This make sense https://review.rdoproject.org/r/16841 ? | 10:35 |
| quiquell | panda: Merge it and then fix | 10:35 |
| panda | quiquell: fieatureset001 is one of the jobs we tested the most right ? | 10:37 |
| quiquell | panda: Yep | 10:38 |
| panda | quiquell: | 10:38 |
| quiquell | panda: Since this is not voting, we can merge and try to fix as it goes, the strategy was more or less that | 10:38 |
| panda | quiquell: yep | 10:41 |
| *** gvrangan has joined #oooq | 10:41 | |
| quiquell | panda: ok let's do it then | 10:45 |
| quiquell | sshnaidm: what do you think ? | 10:45 |
| sshnaidm | quiquell, if one job passed, I think worth to merge it | 10:47 |
| quiquell | sshnaidm: we talking about replacing all that we have covered at openstack-check pipeline | 10:49 |
| quiquell | sshnaidm: and deal with them | 10:49 |
| quiquell | sshnaidm: more or lease remove legacy- | 10:49 |
| quiquell | sshnaidm: except experimental and -tht | 10:49 |
| sshnaidm | quiquell, do we have one new job passed? | 10:49 |
| sshnaidm | quiquell, in some of test patches? | 10:50 |
| quiquell | sshnaidm: Not in check, | 10:50 |
| sshnaidm | quiquell, "check"? | 10:51 |
| quiquell | sshnaidm: We talking about just fix them in production | 10:51 |
| quiquell | sshnaidm: openstack-check | 10:51 |
| quiquell | sshnaidm, panda: ok now I leave see you later | 10:51 |
| sshnaidm | quiquell, do we have job passed in check pipeline? | 10:51 |
| quiquell | sshnaidm: master and queens ovb fs001 | 10:51 |
| quiquell | for tq and tqe | 10:51 |
| *** fhubik has left #oooq | 10:51 | |
| sshnaidm | quiquell, cool, then let's do it.. openstack-check we can't test afaik anyway | 10:51 |
| quiquell | sshnaidm: ok, let's merge this then https://review.rdoproject.org/r/16841 | 10:52 |
| quiquell | panda: ^ ok ? | 10:52 |
| *** quiquell is now known as quiquell|brb | 10:53 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, tripleo-ci-centos-7-scenario009-multinode-oooq @ https://review.openstack.org/560445, stable/queens: (1 more message) | 10:53 |
| panda | quiquell: already +1ed | 10:53 |
| sshnaidm | quiquell|brb, panda why only tripleo.yaml? We have also jobs in projects.yaml | 10:59 |
| panda | sshnaidm: he probably didn't want to migrate everything at once. | 11:07 |
| *** gvrangan has quit IRC | 11:08 | |
| *** ratailor has quit IRC | 11:20 | |
| sshnaidm | panda, so we leave half legacy and half not? | 11:27 |
| panda | sshnaidm: I didn't say it was a ggod decision. So, for what I imagine, there will be some period in which, if something fails of goes wrong, we'll want to understand if it's the new workflow or not. As long as we have at least one job to compere things to, we'll be fine. It doesn't have to be half, we'll probably need one to compare | 11:30 |
| sshnaidm | panda, the change is merged, now need to keep eye on jobs | 11:44 |
| sshnaidm | and I see still legacy-* jobs are queued | 11:45 |
| *** jrist has quit IRC | 11:49 | |
| *** jrist has joined #oooq | 11:50 | |
| *** jrist has quit IRC | 11:54 | |
| *** jrist has joined #oooq | 11:58 | |
| *** rlandy has joined #oooq | 12:04 | |
| rlandy | sshnaidm: hi - looks like you got it working. | 12:09 |
| sshnaidm | rlandy, yeah, the final patch fixed it :) | 12:17 |
| rlandy | oh the workspace | 12:18 |
| rlandy | nice | 12:18 |
| *** trown|outtypewww is now known as trown | 12:18 | |
| *** weshay_pto is now known as weshay | 12:19 | |
| weshay | marios|rover, kopecmartin|ruck how is the gate doing? | 12:19 |
| *** quiquell|brb is now known as quiquell | 12:20 | |
| quiquell | sshnaidm: What jobs are missing ? | 12:21 |
| weshay | marios|rover, we really at 0 fails in 24hrs? and 85% pass | 12:21 |
| sshnaidm | quiquell, I see now that both legacy and non-legacy are executed.. | 12:21 |
| quiquell | sshnaidm: bug we have merge the template | 12:22 |
| quiquell | s/bug/but/ | 12:22 |
| quiquell | sshnaidm: I don't see the test patch running https://review.rdoproject.org/r/#/c/16806/ | 12:23 |
| quiquell | sshnaidm: nah I see it now | 12:24 |
| rlandy | quiquell: hi - ok if I take this card https://tree.taiga.io/project/tripleo-ci-board/task/107?kanban-status=1447275? I am putting in reviews to undo the stuff I did to avoid promotions | 12:25 |
| quiquell | rlandy: Totally they ir kind of good now | 12:25 |
| quiquell | rlandy: did we have a run of the -upload ? | 12:26 |
| sshnaidm | quiquell, heh, I see what a problem.. | 12:26 |
| quiquell | sshnaidm: what is ? | 12:26 |
| rlandy | I'll check | 12:26 |
| marios|rover | weshay: mostly ok but queens borked undercloud job https://bugs.launchpad.net/tripleo/+bug/1796639 | 12:28 |
| openstack | Launchpad bug 1796639 in tripleo "[stable/queens] The tripleo-ci-centos-7-undercloud-upgrades is failing on queens for No such file or directory: '/etc/heat/policy.json" [High,Triaged] - Assigned to Marios Andreou (marios-b) | 12:28 |
| rlandy | quiquell: not yet | 12:28 |
| quiquell | rlandy: I have put in place a test pat ch for queens/pike too | 12:28 |
| rlandy | quiquell: yeah - I see that | 12:28 |
| rlandy | and to covert the rest | 12:28 |
| rlandy | quiquell: which is good - the check jobs did fine | 12:29 |
| quiquell | rlandy: We are in good position to start transition | 12:31 |
| rlandy | quiquell: sshnaidm I don;t see upload run yet ... | 12:31 |
| quiquell | rlandy: are you going to also activate openstack-periodic24hr ? | 12:32 |
| sshnaidm | rlandy, afaik there is problem with ovb at all now | 12:32 |
| quiquell | sshnaidm: Upstream is broken at OVB | 12:32 |
| rlandy | quiquell; sshnaidm: I am not activating anything yet ... just putting in reviews for when we are ready to do so | 12:33 |
| rlandy | I move the pipelines to run once per month | 12:33 |
| rlandy | and commented promoter | 12:33 |
| rlandy | I am undoing that | 12:33 |
| rlandy | also - would like to see upload run once first | 12:33 |
| sshnaidm | quiquell, in all branches? | 12:33 |
| quiquell | sshnaidm: master and rocky for sure the others could be too | 12:34 |
| quiquell | sshnaidm: That's why I add a Depends-On t his morning with the fix | 12:34 |
| quiquell | ssbarnea: Some of the requirements from tq is not meet at fedora http://logs.openstack.org/74/595374/36/check/tripleo-ci-fedora-28-standalone/18e8d7d/job-output.txt.gz | 12:38 |
| quiquell | sshnaidm: ahhh No match for argument: /usr/bin/virtualenv | 12:38 |
| quiquell | path is different at fedora | 12:39 |
| sshnaidm | most likely it's virtualenv3 or kinda | 12:40 |
| quiquell | sshnaidm: pufff ok | 12:41 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, tripleo-ci- (2 more messages) | 12:53 |
| agopi | rlandy, quiquell https://review.openstack.org/#/c/604355/ seems to have worked. pl review when you get a chance. Thanks | 12:53 |
| weshay | marios|rover, kopecmartin|ruck fyi https://tree.taiga.io/project/tripleo-ci-board/issue/127?kanban-status=2027733 | 12:53 |
| quiquell | agopi: That's the most secured ansible task I have ever seen with a "when" :-) | 12:55 |
| marios|rover | quiquell: can you check https://trello.com/c/ocIzDwCI/750-cixlp1793293tripleociproa-pike-promotion-failing-trying-to-use-ipv6-to-get-buildlogs what's the action there? | 12:55 |
| agopi | haha just triple making sure quiquell :) | 12:56 |
| marios|rover | weshay: ack | 12:56 |
| quiquell | marios|rover: Preventing action was to merge the review to fix release files, it's already merged so it's already prevented | 12:56 |
| marios|rover | quiquell: can you please add some not/comment there please :)... haha but it looked like you had somethign else in mind? | 12:57 |
| marios|rover | quiquell: or perhaps it was misunderstanding (just came up on call) | 12:57 |
| weshay | ok.. folks.. | 12:59 |
| weshay | scrum in 1 min | 12:59 |
| weshay | hrm.. I don't have his passcode | 12:59 |
| weshay | move to my blue | 12:59 |
| weshay | https://bluejeans.com/u/whayutin/ | 12:59 |
| weshay | sshnaidm, rlandy quiquell ssbarnea | 13:00 |
| sshnaidm | quiquell, ^^ | 13:01 |
| quiquell | weshay: I am on the meeting | 13:01 |
| quiquell | moderator is missing | 13:01 |
| weshay | quiquell, https://bluejeans.com/u/whayutin/ | 13:02 |
| weshay | move there please | 13:02 |
| weshay | ssbarnea, you around? | 13:02 |
| marios|rover | weshay: whats' the right gerrit id so i can spam you for reviews please | 13:11 |
| *** toure|gone is now known as toure | 13:15 | |
| *** vinaykns has joined #oooq | 13:36 | |
| *** agopi is now known as agopi|brb | 13:36 | |
| *** zul has quit IRC | 13:38 | |
| *** jaosorior has quit IRC | 13:39 | |
| *** agopi|brb has quit IRC | 13:41 | |
| rlandy | quiquell: ok with upload? ... I think so | 13:55 |
| quiquell | rlandy: didn't check the logs yet, I suppose we have to be sure that is running at proper DLRN hash and the like | 13:56 |
| weshay | kopecmartin|ruck, you available? | 13:56 |
| rlandy | https://images.rdoproject.org/queens/rdo_trunk/tripleo-ci-testing/ https://images.rdoproject.org/queens/rdo_trunk/71add308632b60fbe43ea44e9576358ad98a974c_bb32139d/ | 13:57 |
| rlandy | quiquell: ^^ | 13:57 |
| quiquell | rlandy: The we are all good, I think | 13:58 |
| kopecmartin|ruck | weshay, I am | 13:59 |
| quiquell | rlandy: current was 71add308632b60fbe43ea44e9576358ad98a974c before the run | 13:59 |
| quiquell | rlandy: https://logs.rdoproject.org/37/16837/1/check/periodic-tripleo-centos-7-queens-promote-consistent-to-tripleo-ci-testing/7a926c3/job-output.txt.gz#_2018-10-08_09_03_34_812043 | 13:59 |
| quiquell | We are good | 13:59 |
| quiquell | 71add308632b60fbe43ea44e9576358ad98a974c_bb32139d to be exact | 13:59 |
| rlandy | yep - all there | 13:59 |
| rlandy | nice to know something worked without debug :) | 14:00 |
| rlandy | sshnaidm: quiquell: I'm ok if you want to merge the hardening patch and run test - since we have seen upload work | 14:01 |
| sshnaidm | rlandy, yeah, let's try it | 14:01 |
| rlandy | I can merge that | 14:01 |
| quiquell | rlandy: yep let's put all together so we have a proper fireworks | 14:01 |
| sshnaidm | rlandy, great | 14:01 |
| rlandy | sshnaidm: we didn;t remove the scripts from ci-config yet? | 14:03 |
| rlandy | just added to config | 14:03 |
| sshnaidm | rlandy, no, didn't remove | 14:05 |
| rlandy | sshnaidm: that's fine - if this works, we can remove | 14:05 |
| rlandy | or add it as part of the clean up work | 14:05 |
| sshnaidm | quiquell, rerunning test.. | 14:06 |
| *** agopi|brb has joined #oooq | 14:08 | |
| *** agopi|brb is now known as agopi | 14:09 | |
| rlandy | sshnaidm: quiquell: are we done with migrating check jobs? | 14:09 |
| rlandy | https://tree.taiga.io/project/tripleo-ci-board/task/114?kanban-status=1447275 | 14:10 |
| arxcruz | weshay: https://review.openstack.org/#/c/605419/ fix stackviz | 14:10 |
| sshnaidm | rlandy, seems so | 14:10 |
| rlandy | k - moving card then | 14:10 |
| quiquell | sshnaidm: Do we need to move upgrades to experimental ? | 14:10 |
| sshnaidm | quiquell, yeah | 14:11 |
| sshnaidm | quiquell, making patch? | 14:11 |
| quiquell | sshnaidm: doing | 14:11 |
| quiquell | sshnaidm, rlandy: upgrades are parth of the promotion pipeline | 14:12 |
| rlandy | sshnaidm: sova updates :) | 14:12 |
| quiquell | sshnaidm, rlandy: Do I keep them there ? | 14:12 |
| rlandy | we need to track new jobs | 14:13 |
| rlandy | quiquell: I guess so | 14:13 |
| rlandy | quiquell: we can't change the promotion criteria | 14:14 |
| rlandy | quiquell: fs037 seems to be passing | 14:15 |
| rlandy | we can ask the upgrades team to confirm that we have not missed something | 14:15 |
| rlandy | although that is an update not an upgrade | 14:15 |
| quiquell | rlandy: https://review.rdoproject.org/r/16846 | 14:16 |
| rlandy | hmmm ... we have not seen an upgrade run anywhere | 14:17 |
| rlandy | marios|rover: using your upgrades hat, any concern with ^^ | 14:18 |
| quiquell | damn we where missing one job tripleo-ci-centos-7-containers-multinode-upgrades-pike not defined | 14:18 |
| rlandy | quiquell: I'm ok with that - let's ping sofer to give him a heads up | 14:18 |
| rlandy | chem: ^^ | 14:22 |
| chem | rlandy: hi, what's up ? | 14:22 |
| weshay | marios|rover, http://38.145.34.55/rocky.log | 14:24 |
| rlandy | chem: just a notification of https://review.rdoproject.org/r/16846 - we are moving the upgrades jobs to the new format | 14:24 |
| rlandy | no more legacy work | 14:25 |
| *** dtrainor has joined #oooq | 14:26 | |
| chem | rlandy: hum, does this mean zuulv3 or is this something else ? | 14:27 |
| rlandy | chem: yes - zuulv3 | 14:27 |
| chem | rlandy: ack, thanks for the heads up, let's see if toci_gate_test.sh.j2 matches the old one :) | 14:28 |
| rlandy | chem; including those jobs in the promotion pipeline | 14:28 |
| rlandy | chem: yep - we'll need someone from the upgrades team to confirm we have not missed/changed anything | 14:28 |
| rlandy | for example ... we currently have a test periodic job that reported results for fs037 ... https://logs.rdoproject.org/37/16837/1/check/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-queens/0aed4b0/ | 14:29 |
| panda | arxcruz: ping ping ping | 14:31 |
| panda | arxcruz: ping -f | 14:31 |
| arxcruz | panda: pong pong pong | 14:32 |
| panda | arxcruz: I have the same question for different issues | 14:32 |
| panda | arxcruz: more or less | 14:32 |
| arxcruz | panda: can you contextualize ? | 14:33 |
| chem | rlandy: oh, looks like the update happened, was quick (16min) and successful | 14:33 |
| panda | arxcruz: yes | 14:33 |
| panda | arxcruz: is issue #91 a duplicate of user story #22 ? | 14:33 |
| panda | arxcruz: what's the status of #77, and who's assigned to #27 ? | 14:34 |
| panda | rlandy: I created #18 as an example of this sprint, are you actively working on it ? | 14:35 |
| rlandy | chem: thanks - looking if we have an upgrades job to review | 14:35 |
| quiquell | rlandy: Adding missing ones to multinode-jobs | 14:35 |
| arxcruz | panda: hold on, let me check | 14:35 |
| rlandy | quiquell: thanks - I'll review and merge when you have it ready | 14:36 |
| rlandy | panda: 18? | 14:36 |
| weshay | arxcruz, thanks | 14:36 |
| weshay | arxcruz, chandan is at a conference? | 14:36 |
| rlandy | panda: oh - browbeat | 14:36 |
| panda | rlandy: https://tree.taiga.io/project/tripleo-ci-board/issue/18?kanban-status=2027733 | 14:36 |
| arxcruz | weshay: i think he is | 14:36 |
| weshay | k | 14:37 |
| weshay | thanks | 14:37 |
| arxcruz | panda: issue #91 isn't related to user story #22, although it could be | 14:37 |
| *** jaganathan has quit IRC | 14:37 | |
| rlandy | panda: yeah - I can probably do the last work and close that out now | 14:37 |
| marios | rlandy: was in a call with weshay reading back | 14:38 |
| rlandy | panda: it could be part of the mass migration | 14:38 |
| rlandy | I'll work with agopi today to finish that | 14:38 |
| panda | rlandy: do you think you'll have time considering all other tasks your'e working on ? | 14:38 |
| arxcruz | panda: #77 i just put a comment | 14:38 |
| rlandy | marios: hi - was chatting with chem and quiquell about the migration of upgrades jobs to zuulv3 ... | 14:38 |
| marios | rlandy: well what do you mean about 'concern' :) hopefully the jobs are green, otherwise shouldn't make a difference to the workflow or what the jobs do? | 14:39 |
| marios | rlandy: i mean the migration | 14:39 |
| arxcruz | panda: #27 should be assigned to kopecmartin|ruck | 14:39 |
| rlandy | marios: just that we should check at least one job of upgrades and make sure it's not a false positive | 14:39 |
| rlandy | and we have not missed something | 14:39 |
| marios | rlandy: ack is there a test posted? (where can we check) | 14:40 |
| rlandy | panda: the job definition is already there | 14:40 |
| rlandy | marios: chem just checked fs037 from the promotion pipeline | 14:41 |
| rlandy | marios: I was looking if there was an upgrades job in the pipeline to check as well | 14:41 |
| rlandy | updates seemed ok | 14:41 |
| panda | rlandy: ok, if you're working it as part of the US #113, is it ok to create a task there and remove the issue ? | 14:41 |
| rlandy | panda: ack | 14:41 |
| agopi | rlandy, don't quite follow the convo but do let me know how i can help :) | 14:41 |
| panda | arxcruz: assigned to kopecmartin|ruck | 14:42 |
| rlandy | agopi: we are done with converting check jobs to zuul ve | 14:42 |
| rlandy | v3 | 14:42 |
| kopecmartin|ruck | panda, arxcruz thanks, it's practically done, needs only polishing | 14:42 |
| rlandy | agopi: as such ... wrt browbeat ... | 14:42 |
| panda | rlandy: agopi https://tree.taiga.io/project/tripleo-ci-board/task/133?kanban-status=1447274 | 14:43 |
| panda | kopecmartin|ruck: can you update the card ? | 14:43 |
| rlandy | https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/ovb-jobs.yaml#L382 | 14:43 |
| kopecmartin|ruck | panda, sure | 14:43 |
| panda | kopecmartin|ruck: https://tree.taiga.io/project/tripleo-ci-board/issue/27?kanban-status=2027733 | 14:43 |
| rlandy | agopi: panda: ^^ | 14:43 |
| quiquell | rlandy: zuul-check passed https://review.rdoproject.org/r/#/c/16846/ | 14:43 |
| panda | kopecmartin|ruck: just report what you said here, it's ok | 14:43 |
| quiquell | rlandy: I drop know, read you tomorrow | 14:43 |
| *** quiquell is now known as quiquell|off | 14:44 | |
| agopi | ack rlandy and https://review.openstack.org/#/c/604355/9 is ready and i verified both the v3 job and legacy job running fine | 14:44 |
| agopi | once i get a review from any of you, will merge it | 14:44 |
| rlandy | agopi: we need to change the trigger to run only th ev3 job | 14:45 |
| rlandy | agopi: I see only the legacy jobs run | 14:45 |
| rlandy | where is the v3 job log? | 14:46 |
| agopi | okay rlandy https://logs.rdoproject.org/10/16310/5/check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053-master/c5c3cf1/ | 14:46 |
| rlandy | agopi: ah good - ok - I +1'ed youre review | 14:47 |
| rlandy | checking trigger | 14:47 |
| rlandy | agopi: ^^ | 14:47 |
| agopi | ack rlandy i'll merge it then. Pl point me to how I can update triggers and will do so. | 14:48 |
| rlandy | agopi: https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/tripleo.yaml#L719 | 14:49 |
| rlandy | ^^ the new job should be running if you change ansible/oooq/browbeat-minimal.yml | 14:49 |
| rlandy | agopi: can you confirm that? | 14:49 |
| agopi | ack rlandy but the trigger for legacy job? i thought we wanted to change so only v3 job triggers | 14:50 |
| agopi | okay gotcha | 14:50 |
| agopi | nvm | 14:50 |
| agopi | yes i'll test it | 14:50 |
| rlandy | agopi: yep that is done | 14:50 |
| rlandy | agopi: I am going to put in one last change ... to trigger that test on *any* browbeat change - not just the one file. ok? | 14:51 |
| agopi | yes that works rlandy :) | 14:52 |
| agopi | i think its triggering | 14:52 |
| agopi | i see it here queued up https://review.rdoproject.org/zuul/status.html | 14:52 |
| agopi | thanks rlandy | 14:52 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, tripleo-ci- (2 more messages) | 14:53 |
| rlandy | weshay: rook: agopi: ^^ are we done with the browbeat work if the vuul-v3 job runs on any browbeat change? or do we want to add it to check elsewhere? tae? | 14:53 |
| rlandy | tqe | 14:53 |
| rlandy | panda: ^^ fyi | 14:53 |
| agopi | I think I already have a review doing same rlandy | 14:54 |
| agopi | https://review.rdoproject.org/r/#/c/16276/2/zuul.d/tripleo.yaml | 14:54 |
| agopi | need to update the legacy | 14:54 |
| agopi | remove the legacy | 14:54 |
| agopi | will do it | 14:54 |
| rlandy | agopi: ack, pls update to use non-legacy and we can merge that once we see the v3 job pass on a browbeat change | 14:55 |
| *** vinaykns has quit IRC | 14:55 | |
| agopi | yes rlandy on it | 14:55 |
| rlandy | agopi: thanks | 14:56 |
| rook | agopi rlandy ++ | 14:56 |
| rook | thanks yall :D | 14:56 |
| panda | rook: without the space before the ++ | 15:01 |
| rook | agopi++ | 15:01 |
| hubbot1 | rook: agopi's karma is now 2 | 15:01 |
| rook | rlandy++ | 15:01 |
| hubbot1 | rook: rlandy's karma is now 25 | 15:01 |
| rook | panda++ | 15:01 |
| hubbot1 | rook: panda's karma is now 4 | 15:01 |
| rook | :D | 15:01 |
| marios|rover | weshay: do you know where those 'weirdo' jobs are runnin gon https://trello.com/c/YSwkeNRM/765-poiscenario0022nd-puppet-run-failing-at-starting-cinder-backup-service | 15:03 |
| weshay | marios|rover, look for the job w/ weirdo in their name in rdo phase 1 | 15:04 |
| weshay | marios|rover, weirdo jobs wrap the upstream puppet integration jobs | 15:04 |
| marios|rover | weshay: thanks | 15:04 |
| weshay | so that they test w/ rpm | 15:04 |
| marios|rover | http://cistatus.tripleo.org/phase1/ no weirdo ? | 15:05 |
| marios|rover | :) | 15:05 |
| marios|rover | weshay: ^ is that where you meant i don't see wirdo | 15:05 |
| marios|rover | weirdo | 15:05 |
| weshay | https://ci.centos.org/view/rdo/view/promotion-pipeline/ | 15:07 |
| weshay | https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-master-current-tripleo/ | 15:07 |
| marios|rover | weshay: ah thanks | 15:07 |
| weshay | quiquell|off, dang | 15:08 |
| weshay | ah.. now I see a gate failure in cockpit.. nvrmind | 15:08 |
| weshay | quiquell|off, /me was wondering what bugs you are fixing | 15:08 |
| weshay | marios|rover, ugh.. rdo 3rd party jobs are at 20% pass rate | 15:09 |
| weshay | my lanta | 15:09 |
| marios|rover | https://review.openstack.org/#/c/560445/ weshay looks like fs 1/35 are that ssl thing http://logs.rdoproject.org/45/560445/160/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/4f4cd02/job-output.txt.gz | 15:14 |
| weshay | marios|rover++ | 15:14 |
| hubbot1 | weshay: marios|rover's karma is now 1 | 15:14 |
| marios|rover | weshay: and fs1 pike is 8 mins/transient issue ? | 15:15 |
| weshay | marios|rover, fix is https://review.openstack.org/#/c/608589/ | 15:16 |
| weshay | ya | 15:16 |
| marios|rover | weshay: ok cool, looking at rdo failing check in grafana it looks like they are all fs 1/35 suspect all the same | 15:16 |
| * weshay looking | 15:16 | |
| agopi | rlandy, https://review.openstack.org/#/c/608699/ triggered a fs53 job, saw it in queue so looks like https://review.rdoproject.org/r/#/c/16276/4 is good to go | 15:17 |
| weshay | pike error is | 15:18 |
| weshay | mError: Failed to apply catalog: Execution of '/bin/openstack role list --quiet --format csv' returned 1: Unable to establish connection to http://192.168.24.1:35357/v3/roles?: HTTPConnectionPool(host='192.168.24.1', port=35357): Max retries exceeded with url: /v3/roles (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f3ea4f43390>: Failed to establish a new connection: [Errno 111] Connection | 15:18 |
| weshay | refused',)) (tried 36, for a total of 170 secon | 15:18 |
| weshay | might be transient | 15:18 |
| weshay | I can waive pike failures | 15:18 |
| weshay | marios|rover, bah.. fs01 failed in prep-images | 15:20 |
| weshay | 35 passed tempest | 15:20 |
| weshay | we'll need to recheck rdo | 15:20 |
| weshay | marios|rover, ah.. nevermind | 15:21 |
| weshay | marios|rover, it's in the gate | 15:21 |
| weshay | lolz | 15:21 |
| weshay | heh.. and it failed in the gate | 15:22 |
| weshay | on the undercloud install | 15:22 |
| weshay | lolz | 15:22 |
| marios|rover | :( | 15:23 |
| *** panda is now known as panda|off | 15:23 | |
| weshay | sorry.. meant http://logs.openstack.org/89/608589/5/gate/tripleo-ci-centos-7-scenario004-multinode-oooq-container/5d46fce/job-output.txt.gz#_2018-10-08_14_00_14_922486 | 15:23 |
| rlandy | agopi: weshay: pls see my comment on browbeat test triggering https://review.rdoproject.org/r/#/c/16276 - wrt tq/tqe/tripleo-ci ... what changes should we trigger browbeat test on? all files? just a subset? | 15:24 |
| weshay | rlandy, let's err on the side of overtesting for browbeat and ha | 15:24 |
| weshay | imho | 15:24 |
| rlandy | weshay: agopi: ^^ ok -so pls just add irrelevant file as in tests above and we should be ok | 15:25 |
| agopi | ack rlandy | 15:26 |
| agopi | will do it | 15:26 |
| agopi | sorry | 15:26 |
| weshay | marios|rover, I'll babysit this patch | 15:26 |
| weshay | marios|rover, I think we should kill the gate job, and recheck it | 15:27 |
| weshay | as it's DOA | 15:27 |
| weshay | wdyt? | 15:27 |
| marios|rover | weshay: is that a new issue "Failed to parse dlrn hash" 18:23 < weshay> sorry.. meant | 15:27 |
| marios|rover | http://logs.openstack.org/89/608589/5/gate/tripleo-ci-centos-7-scenario004-multinode-oooq-container/5d46fce/job-output.txt.gz#_2018-10-08_14_00_14_922486 | 15:27 |
| marios|rover | weshay: ack no objections | 15:27 |
| weshay | marios|rover, most likely because of a hiccup on http://mirror.us-west-1.packethost.openstack.org:8080/rdo/centos7/ | 15:29 |
| weshay | hard to test that though | 15:29 |
| *** agopi is now known as agopi|lunch | 15:32 | |
| weshay | panda|off, ping me tomorrow.. we need to catch pu | 15:34 |
| weshay | up | 15:34 |
| marios|rover | thanks weshay | 15:35 |
| *** skramaja_ has quit IRC | 15:45 | |
| *** sshnaidm is now known as sshnaidm|afk | 15:51 | |
| *** ssbarnea|bkp2 has joined #oooq | 15:54 | |
| rlandy | sshnaidm|afk: panda|off: weshay: I am going to wait for results of fs050 from the current test run to see how upgrades behaves. If that looks ok - should we merge https://review.rdoproject.org/r/#/c/16846/? | 15:54 |
| rlandy | other concerns? | 15:54 |
| *** jfrancoa has quit IRC | 15:55 | |
| weshay | rlandy, I think undercloud update has been stable, undercloud updgrade I think is non-voting | 16:00 |
| weshay | rlandy, if it's non-voting upstream the promotion job results can be waived as long as it's not a zuul config issue | 16:00 |
| weshay | rlandy, make sense to you? | 16:00 |
| rlandy | weshay: only concern is if it's an issue on our side | 16:06 |
| rlandy | zuul vars | 16:06 |
| rlandy | release work | 16:06 |
| rlandy | weshay: fs037 was looked at by upgrades team already | 16:07 |
| weshay | rlandy++ | 16:07 |
| hubbot1 | weshay: rlandy's karma is now 26 | 16:07 |
| *** jrist has quit IRC | 16:07 | |
| *** agopi|lunch is now known as agopi | 16:11 | |
| *** jrist has joined #oooq | 16:12 | |
| *** jrist has quit IRC | 16:17 | |
| weshay | ssbarnea, you back? | 16:17 |
| *** bogdando has quit IRC | 16:19 | |
| *** dtantsur is now known as dtantsur|afk | 16:20 | |
| *** panda|off has quit IRC | 16:22 | |
| *** panda has joined #oooq | 16:23 | |
| agopi | rlandy, I've updated https://review.rdoproject.org/r/#/c/16276/ pl review when you get a chance and lmk any other changes needed. | 16:27 |
| agopi | thanks :) | 16:27 |
| rlandy | looking | 16:41 |
| *** sshnaidm|afk is now known as sshnaidm | 16:48 | |
| sshnaidm | rlandy, yeah, I think we can merge https://review.rdoproject.org/r/#/c/16846 | 16:48 |
| rlandy | sshnaidm; I am just watching https://review.rdoproject.org/zuul/stream.html?uuid=468d4332096a44679fd19b0cbf4f622b&logfile=console.log | 16:49 |
| rlandy | and then I'll merge | 16:50 |
| sshnaidm | rlandy, it passed | 16:50 |
| sshnaidm | rlandy, oops.. seems like failed | 16:50 |
| rlandy | +(./toci_quickstart.sh:150): echo 'Playbook run of multinode-validate.yml passed successfully' | 16:51 |
| rlandy | if the logs look ok to upgrades team, I'll merge | 16:52 |
| rlandy | master failed overcloud deploy | 16:53 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, tripleo-ci- (2 more messages) | 16:53 |
| rlandy | 2018-10-08 16:39:06 | Error: The following files were not found: /home/zuul/inject-trust-anchor.yaml | 16:54 |
| weshay | rlandy, that may be a current issue | 16:56 |
| sshnaidm | rlandy, yeah, I see "success": true, but "REPORTING FAILURE TO DLRN API".. weird | 16:56 |
| weshay | rlandy, https://bugs.launchpad.net/tripleo/+bug/1796626 | 16:56 |
| openstack | Launchpad bug 1796626 in tripleo "OVB - overcloud-ssl fail with KeyError - breaks legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master and featureset035" [High,In progress] - Assigned to Michele Baldessari (michele) | 16:56 |
| sshnaidm | rlandy, oh, I see.. [[ True = \t\r\u\e ]], hehe | 16:57 |
| agopi | rlandy, job failed but probably because of rdo cloud i'm guessing? https://logs.rdoproject.org/92/608692/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053-master/2de812b/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-10-08_16_14_56 | 17:01 |
| *** trown is now known as trown|lunch | 17:02 | |
| rlandy | will look in a sec | 17:04 |
| agopi | ack rlandy thanks :) | 17:04 |
| *** apetrich has quit IRC | 17:11 | |
| rlandy | agopi: yep - ironic error | 17:14 |
| rlandy | https://logs.rdoproject.org/06/16806/20/check/periodic-tripleo-ci-centos-7-singlenode-featureset050-upgrades-rocky/468d433/ | 17:14 |
| rlandy | sshnaidm: ok now we have a log to compare ... https://logs.rdoproject.org/06/16806/20/check/periodic-tripleo-ci-centos-7-singlenode-featureset050-upgrades-rocky/468d433/ | 17:14 |
| agopi | ack rlandy hit recheck lets see if it works. | 17:18 |
| rlandy | chem: we have an upgrades job that ran and passed ... https://logs.rdoproject.org/06/16806/20/check/periodic-tripleo-ci-centos-7-singlenode-featureset050-upgrades-rocky/468d433/ - looks like it completed the upgrade ... https://logs.rdoproject.org/06/16806/20/check/periodic-tripleo-ci-centos-7-singlenode-featureset050-upgrades-rocky/468d433/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz | 17:19 |
| rlandy | sshnaidm: looks ok afaict ... will merge the experimental patch unless any objections | 17:20 |
| rlandy | not your experimental job, the upgrades one | 17:21 |
| sshnaidm | rlandy, yeah, I think we can merge experimental too: https://review.rdoproject.org/r/#/c/16842/ | 17:29 |
| *** ssbarnea_ has joined #oooq | 17:29 | |
| *** ssbarnea|bkp2 has quit IRC | 17:30 | |
| rlandy | sshnaidm; k - merging https://review.rdoproject.org/r/#/c/16842 | 17:32 |
| *** sshnaidm is now known as sshnaidm|afk | 17:32 | |
| *** apetrich has joined #oooq | 17:47 | |
| weshay | need reviews on https://review.rdoproject.org/r/#/c/15978/7 | 17:53 |
| rlandy | looking | 17:56 |
| rlandy | weshay: ^^ why still jenkins/jobs? | 17:59 |
| weshay | rlandy, it's ppc | 18:00 |
| weshay | in ci.centos | 18:00 |
| weshay | weee | 18:00 |
| rlandy | weshay - oh gee - ok - don;t know much about that - not sure I'm the best reviewer - but if you want it merged I can do that | 18:01 |
| rlandy | weshay: also ... I'm merging https://review.rdoproject.org/r/#/c/16846 (upgrades) .. fs050 passed - posted results above for chem to look at. He may be out already - can revert if it's an issue | 18:02 |
| weshay | k. | 18:03 |
| weshay | no worries.. just getting it in channel | 18:04 |
| weshay | rlandy, nice on upgrades | 18:04 |
| *** trown|lunch is now known as trown | 18:11 | |
| *** panda has quit IRC | 18:26 | |
| *** panda has joined #oooq | 18:28 | |
| *** kopecmartin|ruck is now known as kopecmartin|off | 18:45 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, tripleo-ci- (2 more messages) | 18:53 |
| rlandy | sshnaidm|afk: weshay: requesting review for https://github.com/sshnaidm/sova/pull/45. thanks | 18:58 |
| weshay | panda, fyi https://tree.taiga.io/project/morucci-software-factory/us/1570 | 18:58 |
| weshay | nice | 18:59 |
| weshay | rlandy, hey.. what's the diff between $job and $job-branch | 19:00 |
| weshay | they are both check right? | 19:00 |
| rlandy | one is check on a change on the same branch and the other on master | 19:01 |
| sshnaidm|afk | rlandy, thanks, merged | 19:01 |
| rlandy | not sure if I am answering that per your questions | 19:01 |
| rlandy | sshnaidm|afk: sure - I never realized how advanced sova was - thanks for keeping that up all this time | 19:02 |
| weshay | rlandy, ah.. makes sense | 19:02 |
| weshay | thakns | 19:02 |
| rlandy | sshnaidm|afk: I'm going to start work on https://tree.taiga.io/project/tripleo-ci-board/task/131?kanban-status=1447274 and put in a review to update the trigger and test. If there is more to the card, let m e know | 19:16 |
| weshay | rlandy, let me know if you need anything | 19:19 |
| rlandy | weshay: I think we're ok - the card I with -tht jobs ^^ is the only outstanding card I see | 19:20 |
| rlandy | browbeat added triggers should be ok to merge shortly | 19:20 |
| rlandy | we are waiting for final results on the hardening job | 19:21 |
| rlandy | weshay: unless I missed something? I think that is it until we re-enable the dlrn reporting | 19:21 |
| weshay | very nice :) | 19:22 |
| rlandy | and promoter | 19:22 |
| rlandy | weshay: do you recall what the 'redefine need' is in https://tree.taiga.io/project/tripleo-ci-board/task/131?kanban-status=1447274? I can update the trigger and test it | 19:28 |
| rlandy | not sure what the rest is about | 19:28 |
| * weshay looks | 19:29 | |
| weshay | rlandy, so sshnaidm|afk has the back story on the jobs they were created for juan and the tls folks | 19:30 |
| rlandy | maybe I should just leave that? | 19:30 |
| rlandy | I see the trigger | 19:30 |
| rlandy | https://github.com/rdo-infra/review.rdoproject.org-config/blob/156196921ff84106ccdf814449a3e588f273d261/zuul.d/tripleo.yaml#L738 | 19:30 |
| weshay | we discussed it this morning.. I think we need / want sshnaidm|afk to add more info there just to be sure we don't do the wrong thing w/ the jobs | 19:30 |
| weshay | the conversation was about migrating the tht jobs, or updating the triggers to execute the check jobs against what is required for tls | 19:31 |
| rlandy | yeah - I remember the discussion but never heard the complete decision | 19:31 |
| * weshay looks | 19:31 | |
| rlandy | I found the trigger | 19:32 |
| weshay | see it | 19:32 |
| weshay | I see any extra vars there passed to the tht job | 19:32 |
| weshay | just kicking off ipsec.*$ and the config | 19:32 |
| rlandy | yea | 19:32 |
| weshay | sitll better to ask Sagi in the task | 19:32 |
| rlandy | and panda and I compared the legacy jobs | 19:32 |
| rlandy | not much diff | 19:32 |
| *** jtomasek has quit IRC | 20:13 | |
| *** toure is now known as toure|biab | 20:23 | |
| rlandy | agopi: browbeat is looking good :) | 20:43 |
| rlandy | https://review.rdoproject.org/zuul/stream.html?uuid=4251ea93bc5745c0988f275ebe791151&logfile=console.log | 20:43 |
| agopi | yes rlandy :) | 20:45 |
| rlandy | agopi; merging your change | 20:46 |
| agopi | also I've hit recheck on https://review.openstack.org/#/c/608699/1 an hour back but can't see it in the queue | 20:46 |
| rlandy | weshay: ^^ finally :) | 20:46 |
| agopi | just curios as to what might've happened | 20:46 |
| * weshay looks | 20:46 | |
| weshay | rlandy, which queue? | 20:46 |
| weshay | agopi, which queue? rdo zuul or upstream zuul | 20:47 |
| rlandy | agopi: I see the results not sure what you are referring to | 20:47 |
| agopi | ack rlandy the above case fs53 job failed because https://review.openstack.org/#/c/604355/ wasnt merged | 20:47 |
| agopi | the results are from before i hit recheck rlandy | 20:47 |
| rlandy | weshay: was pointing at the fact that we are merging the browbeat trigger | 20:47 |
| weshay | +1 | 20:47 |
| weshay | agopi, it's running upstram | 20:47 |
| weshay | check-rdo | 20:47 |
| weshay | to refire ovb jobs | 20:48 |
| rlandy | I see a recheck | 20:48 |
| rlandy | Patch Set 1: recheck | 20:48 |
| rlandy | 3:24 PM | 20:48 |
| rlandy | Aakarsh | 20:48 |
| rlandy | Patch Set 1: recheck | 20:48 |
| rlandy | 3:24 PM | 20:48 |
| rlandy | RDO Third Party CI | 20:48 |
| rlandy | 3:35 PM | 20:48 |
| rlandy | agopi: hold on - I'll merge and then you can run again | 20:49 |
| weshay | fyi.. rlandy some what related https://tree.taiga.io/project/morucci-software-factory/us/1570 | 20:49 |
| weshay | infra has a few things to do to open up the capacity on rdo sf | 20:49 |
| agopi | weshay, i'm looking here https://review.rdoproject.org/zuul/status.html and i don't see any queue: openstack/tripleo-quickstart for https://review.openstack.org/#/c/608699/ | 20:50 |
| agopi | ack rlandy | 20:50 |
| weshay | agopi, in rdo sf | 20:50 |
| weshay | ya | 20:50 |
| rlandy | weshay: interesting | 20:50 |
| weshay | agopi, check-rdo | 20:51 |
| rlandy | merge in progress ... | 20:51 |
| agopi | weird it popped now weshay | 20:52 |
| agopi | do you see this | 20:52 |
| agopi | https://review.rdoproject.org/zuul/status.html | 20:52 |
| weshay | agopi, ya.. I add check-rdo | 20:52 |
| weshay | yes | 20:52 |
| agopi | oh okay gotcha | 20:52 |
| agopi | sorry | 20:52 |
| agopi | now it makes sense | 20:52 |
| weshay | rlandy, does recheck kick both? | 20:52 |
| weshay | hrm.. it should zuul.d/upstream.yaml:21: comment: (?i)^(Patch Set [0-9]+:)?( [\w\\+-]*)*(\n\n)?\s*(recheck|check-rdo|verify-rdo) | 20:52 |
| weshay | agopi, don't knjow | 20:53 |
| rlandy | agopi: merge complete - edit your job to remove depends and run again | 20:53 |
| weshay | recheck should have done it | 20:53 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, tripleo-ci- (2 more messages) | 20:53 |
| agopi | recheck kicked RDO CI for sure though weshay from what i saw | 20:53 |
| rlandy | weshay: yes | 20:53 |
| rlandy | recheck kicks both | 20:53 |
| weshay | maybe check-rdo kicks it harder | 20:53 |
| weshay | stupid infra | 20:53 |
| agopi | ack rlandy | 20:53 |
| rlandy | weshay: agopi: sometimes changes to dependent patches collide | 20:53 |
| rlandy | and the job doesn't run - I've seen that happen | 20:54 |
| weshay | never cross the streams | 20:54 |
| rlandy | not sure what happened here but recheck shoudl kick both | 20:54 |
| agopi | ooh okay rlandy gotcha anyway I updated commit and removed depends on looks good as I see fs53 queued up | 20:55 |
| rlandy | good | 20:55 |
| agopi | thanks rlandy, weshay :) | 20:55 |
| rlandy | weshay: I *think* we are ready to turn periodic pipeline back on - maybe waiting for a morning to do that is better? | 20:57 |
| rlandy | hardening looks ok to me | 20:57 |
| weshay | rlandy, better to have the jobs run overnight | 20:58 |
| weshay | imho | 20:58 |
| rlandy | the only task I see left is tht and periodic does not rely on it | 20:59 |
| *** trown is now known as trown|outtypewww | 20:59 | |
| rlandy | weshay: looking at results here ... https://review.rdoproject.org/r/#/c/16806/ | 20:59 |
| rlandy | the errors are either tls or ironic related | 21:00 |
| rlandy | same we see in check | 21:00 |
| rlandy | weshay: if you're ok with that ... pls vote on reviews in https://tree.taiga.io/project/tripleo-ci-board/task/107?kanban-status=1447275 | 21:01 |
| rlandy | and we can merge those to re-enable | 21:02 |
| rlandy | could re-enable reporting afterwards | 21:02 |
| weshay | rlandy, ya.. we need a promotion on master | 21:02 |
| rlandy | ie: promotions first reporting later | 21:02 |
| rlandy | weshay: ok | 21:02 |
| weshay | rlandy, unless that test patch is using a recent hash | 21:02 |
| weshay | and is recreating tripleo-ci-testing | 21:02 |
| weshay | rlandy, still it's not an infra / zuul issue | 21:03 |
| weshay | so we're good | 21:03 |
| rlandy | it is | 21:03 |
| rlandy | weshay: k - pls vote and we'll see what shakes out | 21:03 |
| rlandy | just so I don't merge my own code w/o anyone's ok | 21:04 |
| weshay | rlandy, +1 on https://review.rdoproject.org/r/#/c/16806/ | 21:06 |
| weshay | what else? | 21:06 |
| rlandy | review.rdoproject.org/r/16843 Restore promotions to run every day | 21:06 |
| rlandy | review.rdoproject.org/r/#/c/16845 Revert "Temp comment out promotions run in RDO Cloud" | 21:06 |
| rlandy | weshay: https://review.rdoproject.org/r/#/c/16806/ never merges | 21:06 |
| rlandy | just a test job | 21:06 |
| rlandy | ^^ those two are needed to restore | 21:07 |
| rlandy | weshay: thanks - periodic pipeline should kick in 43 mins | 21:17 |
| rlandy | re-enabling dlrn | 21:18 |
| rlandy | weshay: k - we've pulled the trigger now ... https://review.rdoproject.org/r/#/c/16845/ | 21:23 |
| weshay | woot | 21:23 |
| * rlandy runs for cover | 21:24 | |
| *** agopi has quit IRC | 21:42 | |
| *** agopi has joined #oooq | 22:05 | |
| weshay | zuul ui is down | 22:27 |
| weshay | curl http://zuul.openstack.org/status | 22:27 |
| rlandy | well periodic jobs kicked | 22:45 |
| rlandy | weshay: ^^ fyi | 22:45 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master, tripleo-ci- (2 more messages) | 22:53 |
| *** tosky has quit IRC | 23:03 | |
| *** rlandy is now known as rlandy|bbl | 23:11 | |
| *** quiquell|off has quit IRC | 23:27 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!