| *** dtrainor has joined #oooq | 00:07 | |
| *** Goneri has joined #oooq | 00:15 | |
| *** Goneri has quit IRC | 00:37 | |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000 (1 more message) | 01:02 |
|---|---|---|
| *** agopi is now known as agopi|afk | 02:01 | |
| agopi|afk | rlandy, https://review.openstack.org/#/c/583011/ has been merged. | 02:07 |
| rlandy | agopi|afk; great - we'll try it tomorrow | 02:24 |
| agopi|afk | rlandy++ | 02:27 |
| hubbot` | agopi|afk: rlandy's karma is now 13 | 02:27 |
| rlandy | and passing 7-3 job - yay https://review.openstack.org/#/c/581376/ | 02:28 |
| rlandy | weshay: ^^ | 02:28 |
| *** rlandy has quit IRC | 02:28 | |
| weshay | ah nice | 02:29 |
| weshay | rlandy | 02:29 |
| *** weshay has quit IRC | 02:30 | |
| *** pliu has quit IRC | 02:30 | |
| *** jschlueter has quit IRC | 02:30 | |
| *** pliu has joined #oooq | 02:31 | |
| *** rasca has quit IRC | 02:31 | |
| *** rnoriega has quit IRC | 02:31 | |
| *** rasca has joined #oooq | 02:32 | |
| *** rnoriega has joined #oooq | 02:32 | |
| *** weshay has joined #oooq | 02:32 | |
| *** jschlueter has joined #oooq | 02:34 | |
| *** brault has quit IRC | 02:53 | |
| *** brault has joined #oooq | 02:56 | |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000 (1 more message) | 03:02 |
| *** skramaja has joined #oooq | 03:07 | |
| *** jaganathan has joined #oooq | 03:11 | |
| *** skramaja_ has joined #oooq | 03:40 | |
| *** skramaja has quit IRC | 03:41 | |
| *** saneax has joined #oooq | 03:43 | |
| *** sshnaidm|bbl has quit IRC | 03:49 | |
| *** udesale has joined #oooq | 03:50 | |
| *** sshnaidm|bbl has joined #oooq | 03:50 | |
| *** sshnaidm|bbl has quit IRC | 03:56 | |
| *** sshnaidm|bbl has joined #oooq | 03:57 | |
| *** ykarel|away has joined #oooq | 04:00 | |
| *** vinaykns has quit IRC | 04:02 | |
| *** vinaykns has joined #oooq | 04:03 | |
| *** vinaykns has quit IRC | 04:15 | |
| *** vinaykns has joined #oooq | 04:15 | |
| *** brault has quit IRC | 04:21 | |
| *** brault has joined #oooq | 04:22 | |
| *** ykarel|away is now known as ykarel | 04:29 | |
| *** skramaja_ is now known as skramaja | 04:49 | |
| *** vinaykns has quit IRC | 04:53 | |
| *** vinaykns has joined #oooq | 04:54 | |
| *** holser_ has joined #oooq | 04:57 | |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000 (1 more message) | 05:02 |
| *** ykarel has quit IRC | 05:15 | |
| *** links has joined #oooq | 05:33 | |
| *** udesale_ has joined #oooq | 05:35 | |
| *** udesale has quit IRC | 05:35 | |
| *** udesale_ has quit IRC | 05:40 | |
| *** ratailor has joined #oooq | 05:45 | |
| *** quiquell|off is now known as quiquell | 05:45 | |
| *** ykarel has joined #oooq | 05:46 | |
| *** vinaykns has quit IRC | 05:49 | |
| *** jfrancoa has joined #oooq | 06:02 | |
| *** udesale_ has joined #oooq | 06:11 | |
| *** bogdando has joined #oooq | 06:14 | |
| chkumar|ruck | %gatestatus | 06:44 |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000 (1 more message) | 06:44 |
| *** kopecmartin has joined #oooq | 06:45 | |
| *** tesseract has joined #oooq | 06:48 | |
| *** holser_ has quit IRC | 06:48 | |
| bogdando | PTAL https://review.openstack.org/#/c/575003/ | 06:52 |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000 (1 more message) | 07:02 |
| *** pgadiya has joined #oooq | 07:03 | |
| *** pgadiya has quit IRC | 07:03 | |
| quiquell | chkumar|ruck: This is legit ? http://38.145.34.131:3000/d/pgdr_WVmk/cockpit?orgId=1&from=1531638398226&to=1531811198226&var-launchpad_tags=alert&var-promotion_names=current-tripleo&var-promotion_names=current-tripleo-rdo&var-promotion_names=current-tripleo-rdo-testing&var-releases=master&var-releases=queens&var-releases=pike&var-releases=ocata&panelId=104&fullscreen | 07:06 |
| *** ccamacho has joined #oooq | 07:09 | |
| quiquell | chkumar|ruck: Ok I see NODE_FAILURE, looks legit and massive | 07:11 |
| chkumar|ruck | quiquell: https://bugs.launchpad.net/tripleo/+bug/1781395 | 07:17 |
| openstack | Launchpad bug 1781395 in tripleo "[stable/queens] multiple times NODE_FAILURE on legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens job against noop check job" [Critical,Fix released] | 07:17 |
| quiquell | chkumar|ruck: legit, thanks | 07:17 |
| ykarel | chkumar|ruck, master promotion jobs are also faiiling | 07:17 |
| ykarel | undercloud install failing | 07:18 |
| *** zoli is now known as zoli|wfh | 07:24 | |
| *** zoli|wfh is now known as zoli | 07:25 | |
| *** tosky has joined #oooq | 07:38 | |
| chkumar|ruck | ykarel: https://bugs.launchpad.net/tripleo/+bug/1780091 | 07:42 |
| openstack | Launchpad bug 1780091 in tripleo "containerized undercloud deployment failed on periodic jobs" [Critical,Triaged] | 07:42 |
| chkumar|ruck | ykarel: it is the same issue related to NTP | 07:42 |
| chkumar|ruck | ykarel: the job runned on 14th July | 07:42 |
| chkumar|ruck | ykarel: https://review.openstack.org/#/c/582733/ landed on 15th | 07:44 |
| chkumar|ruck | ykarel: I am not seeing any recent failures | 07:46 |
| *** florianf has joined #oooq | 07:46 | |
| *** florianf has joined #oooq | 07:47 | |
| *** rfolco__ has joined #oooq | 07:47 | |
| *** rfolco_ has quit IRC | 07:50 | |
| *** amoralej has joined #oooq | 07:55 | |
| quiquell | chkumar|ruck, sshnaidm|bbl: rr-cockpit will be down for maintenance :-P | 07:56 |
| *** sshnaidm|bbl is now known as sshnaidm|rover | 07:56 | |
| sshnaidm|rover | quiquell, ok :) | 07:56 |
| chkumar|ruck | quiquell: ack | 07:56 |
| ykarel | chkumar|ruck, https://trunk-primary.rdoproject.org/api-centos-master-uc/api/civotes_detail.html?commit_hash=19e28d0cd340c97f3232070bba06392d162448c2&distro_hash=50cc52eb89c0f7ec1bb15a73cfb115976516de93 | 07:58 |
| ssbarnea1 | morning! I wanted to ask you to consider adding me to your reviews for two reasons: i want to help you and that is also a good opportunity for me to learn more about what is happening. | 08:01 |
| quiquell | ssbarnea1: Agree with that, will do, sorry mate. | 08:02 |
| quiquell | Want to create a script with all the members and add them with git review | 08:02 |
| *** holser_ has joined #oooq | 08:23 | |
| *** holser_ has quit IRC | 08:28 | |
| *** holser_ has joined #oooq | 08:29 | |
| *** gkadam has joined #oooq | 08:41 | |
| *** sanjayu_ has joined #oooq | 09:00 | |
| *** saneax has quit IRC | 09:01 | |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000 (1 more message) | 09:02 |
| *** sanjayu_ has quit IRC | 09:03 | |
| arxcruz | panda: sshnaidm|rover https://review.openstack.org/#/c/570884/ when you guys have time :) | 09:18 |
| *** dsneddon_ has quit IRC | 09:22 | |
| *** ykarel has quit IRC | 09:23 | |
| *** saneax has joined #oooq | 09:33 | |
| *** sanjayu_ has joined #oooq | 09:35 | |
| *** saneax has quit IRC | 09:38 | |
| *** dtantsur|afk is now known as dtantsur | 09:39 | |
| *** Goneri has joined #oooq | 09:40 | |
| *** ykarel has joined #oooq | 09:44 | |
| *** ykarel is now known as ykarel|away | 09:48 | |
| *** chem has joined #oooq | 09:53 | |
| *** Goneri has quit IRC | 09:54 | |
| *** ykarel|away has quit IRC | 09:57 | |
| quiquell | sshnaidm|rover: Added comments on the role in deploy rrcockpit | 10:08 |
| quiquell | panda, marios, sshnaidm|rover: for sprint16 featureset as string, https://review.openstack.org/#/c/583022/ | 10:12 |
| rasca | hey sshnaidm|rover hi! I am following this review https://review.openstack.org/#/c/538952 gating, and it failed on one RDO CI job: https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-gate-newton-delorean-quick-basic-5594/console.txt.gz but I don't think it's related, do you? | 10:25 |
| sshnaidm|rover | rasca, yeah, you can ignore it | 10:27 |
| rasca | sshnaidm|rover, but will it have a weight on the overall approval process? | 10:28 |
| sshnaidm|rover | we need to remove this newton job.. | 10:28 |
| sshnaidm|rover | rasca, nope, RDO CI doesn't vote | 10:28 |
| rasca | sshnaidm|rover, yeah right, thanks | 10:28 |
| sshnaidm|rover | rasca, do you have ansible lint in ha-utils role repo? | 10:31 |
| marios | ack quiquell | 10:35 |
| quiquell | panda, marios: I think this one is starting to be mergeable https://review.openstack.org/#/c/582885/ | 10:38 |
| quiquell | what do you think ? | 10:38 |
| *** zoli is now known as zoli|lunch | 10:41 | |
| sshnaidm|rover | chkumar|ruck, can you please take a look why legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset021-master fails here? https://review.openstack.org/#/c/577039/ | 10:57 |
| chkumar|ruck | sshnaidm|rover: ERROR: SSLError: : resources.Compute<nested_stack>.resources.0<https://192.168.24.2:13808/v1/AUTH_7a6168b9590e49ab9aca07679e3c8430/overcloud/puppet/compute-role.yaml>.resources.NovaCompute: : SSL exception connecting to https://192.168.24.3:8774/v2.1/: ("bad handshake: Error([('SSL routines', 'SSL23_GET_SERVER_HELLO', 'unknown protocol')],)",) | 11:01 |
| chkumar|ruck | sshnaidm|rover: known issue | 11:01 |
| chkumar|ruck | sshnaidm|rover: https://bugs.launchpad.net/tripleo/+bug/1781541 | 11:01 |
| openstack | Launchpad bug 1781541 in tripleo "[master][promotion][RDO phase1] Creating overcloud Heat stack failed giving Error([('SSL routines', 'SSL23_GET_SERVER_HELLO', 'unknown protocol')]" [Critical,Fix released] - Assigned to wes hayutin (weshayutin) | 11:01 |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000 (1 more message) | 11:02 |
| sshnaidm|rover | chkumar|ruck, ack | 11:03 |
| chkumar|ruck | sshnaidm|rover: as per ykarel fs21 got affected | 11:03 |
| chkumar|ruck | sshnaidm|rover: I am putting a patch for fs21 | 11:05 |
| rasca | sshnaidm|rover, not at the moment, but I would like to add it | 11:07 |
| *** udesale_ has quit IRC | 11:12 | |
| *** atoth has joined #oooq | 11:14 | |
| *** d0ugal has quit IRC | 11:15 | |
| *** d0ugal has joined #oooq | 11:18 | |
| *** d0ugal has quit IRC | 11:18 | |
| *** d0ugal has joined #oooq | 11:18 | |
| *** zoli|lunch is now known as zoli | 11:52 | |
| *** zoli is now known as zoli|wfh | 11:53 | |
| *** ratailor has quit IRC | 12:01 | |
| *** agopi|afk has quit IRC | 12:02 | |
| *** amoralej is now known as amoralej|lunch | 12:09 | |
| *** rfolco__ is now known as rfolco | 12:18 | |
| weshay | chkumar|ruck, is fs21 running? | 12:24 |
| chkumar|ruck | weshay: https://review.rdoproject.org/zuul3/job.html?job_name=legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset021-master runns against upstream tempest | 12:26 |
| chkumar|ruck | https://review.rdoproject.org/zuul3/builds.html?job_name=legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset021-master | 12:26 |
| *** rlandy has joined #oooq | 12:28 | |
| rlandy | marios: hello - thanks for voting on the 3 node stuff | 12:29 |
| rlandy | the debug was a little confusing but came down to this: | 12:29 |
| rlandy | https://github.com/openstack/tripleo-quickstart/blob/master/config/nodes/2ctlr.yml | 12:30 |
| marios | rlandy: sorry i completely missed it in the mornign reviews run and we just discussed it on a call with quique so i had another pass justnow | 12:30 |
| rlandy | marios: no worries - my concern about merging this is that we just got a lucky ordering on this test run. but I guess we can revert if it is no consistent | 12:31 |
| rlandy | marios: weshay: you guys ok with w+1 and seeing how this goes or should we recheck this a few times to be sure? | 12:32 |
| marios | rlandy: i started typing that (do you want to recheck it here a few times) | 12:32 |
| marios | and wasn't sure it was sane :) | 12:32 |
| rlandy | marios: yeah - let's do that | 12:32 |
| marios | of course we could merge and try get a few runs with other reviews too | 12:32 |
| marios | :) | 12:32 |
| marios | but it might break so lets do the former | 12:32 |
| rlandy | let's not take down ci, shall we | 12:32 |
| marios | right | 12:32 |
| marios | ! | 12:32 |
| rlandy | k, rechecking | 12:33 |
| *** amoralej|lunch has quit IRC | 12:33 | |
| *** agopi|afk has joined #oooq | 12:34 | |
| *** agopi|afk is now known as agopi | 12:34 | |
| quiquell | panda: Are you there ? | 12:39 |
| rlandy | quiquell: panda said he was training | 12:40 |
| weshay | ssbarnea1, ok.. you have a minute | 12:40 |
| quiquell | rlandy: Ahh I remember now, ok | 12:40 |
| weshay | rlandy, which patch? | 12:40 |
| quiquell | rfolco: Do you have a minute ? | 12:40 |
| rfolco | quiquell, sure | 12:40 |
| rlandy | weshay; no worries - I already rechecked it - the 3 node one | 12:41 |
| quiquell | rfolco: bj ? | 12:42 |
| rfolco | quiquell, ok, give me 1 min | 12:43 |
| *** udesale has joined #oooq | 12:43 | |
| quiquell | rfolco: https://bluejeans.com/7891065232 | 12:45 |
| weshay | quiquell, ssbarnea1 you know you are too tired when.... | 12:48 |
| weshay | you think it's Wednesday on a Tuesday | 12:48 |
| weshay | wtf | 12:48 |
| weshay | sshnaidm|rover, chkumar|ruck let's sync up on ruck / rover | 12:49 |
| weshay | when you guys have a minute | 12:49 |
| weshay | sshnaidm|rover, chkumar|ruck I think this will fix fs16/17 in master promotion https://review.openstack.org/#/c/581183/ | 12:51 |
| sshnaidm|rover | weshay, no, it's fixed by other patch | 12:52 |
| chkumar|ruck | weshay: we are testing here https://review.rdoproject.org/r/#/c/13943/ | 12:55 |
| *** tosky has quit IRC | 12:56 | |
| *** tosky has joined #oooq | 12:56 | |
| chkumar|ruck | sshnaidm|rover: weshay bj link? | 12:56 |
| weshay | sshnaidm|rover, chkumar|ruck we haven't synced yet | 12:57 |
| weshay | would be good to figure this stuff out before tomorrow | 12:57 |
| weshay | https://bluejeans.com/whayutin | 12:57 |
| hubbot` | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode- (1 more message) | 13:02 |
| quiquell | weshay: You need some PTO | 13:27 |
| weshay | quiquell, srsly | 13:29 |
| weshay | sshnaidm|rover, https://review.rdoproject.org/r/14865 | 13:29 |
| *** amoralej has joined #oooq | 13:32 | |
| quiquell | weshay, marios, rfolco, rlandy: I think this is the next patch to merge for the sprint16 https://review.openstack.org/#/c/582885/ | 13:40 |
| quiquell | It stores at a common place the como ansible variables used by jobs | 13:40 |
| marios | quiquell: ack i will have another pass ;) | 13:41 |
| quiquell | marios: thanks | 13:41 |
| quiquell | sshnaidm|rover, chkumar|ruck, weshay: rrcockpit is up again at http://38.145.34.131 | 13:42 |
| quiquell | It's starting from fresh data, so not much buils gathered | 13:43 |
| weshay | quiquell, admin / ruckrover | 13:43 |
| weshay | quiquell, don't see the dashboard | 13:44 |
| weshay | I see the grafana home dashboard | 13:44 |
| quiquell | weshay: http://38.145.34.131/d/GmBOsEdmk/cockpit?orgId=1 | 13:45 |
| *** skramaja has quit IRC | 13:46 | |
| weshay | quiquell++ | 13:49 |
| hubbot` | weshay: quiquell's karma is now 3 | 13:49 |
| rasca | weshay, sshnaidm|rover, rlandy, I'm testing the three BM/HA/OVB reviews in my personal tenant | 13:49 |
| weshay | rasca, if browbeat beats you to upstream.. I'm going to tackle you | 13:50 |
| rasca | weshay, sshnaidm|rover, rlandy, I don't have yet results, but the tests is without any external intervention | 13:50 |
| rasca | weshay, you're going to try to tackle me, that's a slight but significant difference | 13:51 |
| weshay | rasca, mixed rules.. rugby vs. footy | 13:51 |
| rasca | weshay, I'll be the one standing up, in any case | 13:52 |
| rasca | I trust myself | 13:52 |
| weshay | :) | 13:52 |
| rasca | :D | 13:52 |
| *** vinaykns has joined #oooq | 13:53 | |
| rasca | weshay, so I hit "Systemd start for certmonger failed" during undercloud installation, I remember this one is a well known issue | 13:59 |
| rlandy | agopi: going through the failure in browbeat now - commenting on the review | 14:00 |
| agopi | rlandy, went through your comment | 14:01 |
| agopi | do you want me to create a playbook folder in the ansible/oooq and put browbeat-minimal.yml ? | 14:01 |
| agopi | im not sure if im following correctly | 14:01 |
| rlandy | agopi: yes - but that is not the real problem | 14:02 |
| rlandy | still debugging | 14:02 |
| rlandy | will ping you when I know exactly | 14:02 |
| rlandy | also | 14:02 |
| rlandy | wrt fs053 | 14:02 |
| rlandy | we need to move some of those settings out | 14:02 |
| rlandy | will explain in a bit | 14:02 |
| agopi | ack rlandy . | 14:03 |
| *** sshnaidm|rover is now known as sshnaidm|afk | 14:06 | |
| rasca | rlandy, since I'm basically hitting https://bugzilla.redhat.com/show_bug.cgi?id=1569122 and I'm not so ovb familiar, does the ovb provision reboot the undercloud at some point or not? | 14:12 |
| openstack | bugzilla.redhat.com bug 1569122 in dbus "Undercloud installation fails with "Execution of '/bin/getcert list' returned 1: Error org.freedesktop.DBus.Error.TimedOut"" [High,New] - Assigned to dking | 14:12 |
| rlandy | rasca: I also hit an undercloud install failure on OVB for rhos-13 ... haven't had time to look into to it much... | 14:15 |
| myoung | chkumar|ruck, could you please update https://etherpad.openstack.org/p/tripleo-ci-squad-meeting @L54 with any additional notes for current CI status? I've already updated etherpad with the #'s for this week | 14:15 |
| rlandy | ovb provision itself should not reboot | 14:15 |
| myoung | chkumar|ruck: (e.g. ongoing investigation / debug, or other issues of note) | 14:15 |
| rlandy | rasca: just need to submit a change for browbeat and then I | 14:16 |
| rlandy | will look into it more | 14:16 |
| rasca | rlandy, the problem I hit was on master | 14:16 |
| rlandy | rhel or centos? | 14:16 |
| rlandy | rdocloud centos | 14:16 |
| rlandy | the bug says rhel | 14:16 |
| rlandy | give me 10 | 14:17 |
| rasca | rlandy, no worries, btw it is CentOS and it's a plain deployment made to test the three patches I've worked on today | 14:19 |
| rlandy | rasca: let's chat after the community meeting | 14:31 |
| weshay | hrm.. having trouble joining | 14:32 |
| weshay | sshnaidm|afk, https://review.openstack.org/#/c/583264/1/ci-scripts/basic.sh | 14:33 |
| weshay | rlandy, panda https://review.openstack.org/#/c/583264/1/ci-scripts/basic.sh | 14:33 |
| *** quiquell is now known as quiquell|off | 14:35 | |
| *** d0ugal has quit IRC | 14:37 | |
| *** d0ugal has joined #oooq | 14:45 | |
| *** sanjay__u has quit IRC | 14:48 | |
| *** bogdando has quit IRC | 14:48 | |
| *** sshnaidm|afk is now known as sshnaidm|rover | 14:54 | |
| hubbot` | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode- (1 more message) | 15:02 |
| *** d0ugal has quit IRC | 15:03 | |
| chkumar|ruck | myoung: done | 15:07 |
| *** ccamacho has quit IRC | 15:07 | |
| myoung | chkumar|ruck: thx | 15:08 |
| *** florianf has quit IRC | 15:09 | |
| *** florianf has joined #oooq | 15:11 | |
| *** d0ugal has joined #oooq | 15:15 | |
| *** dsneddon has joined #oooq | 15:16 | |
| rfolco | rlandy, add to required-projects like we do for tq tqe and tu | 15:20 |
| rlandy | rfolco: https://softwarefactory-project.io/r/#/c/13019/1/zuul/rdo.yaml? | 15:21 |
| rfolco | rlandy, no, https://github.com/openstack-infra/tripleo-ci/blob/master/zuul.d/base.yaml#L23 | 15:23 |
| *** jaganathan has quit IRC | 15:25 | |
| rfolco | rlandy, zuul clones required-projects to src/git.openstack.org.... | 15:25 |
| *** d0ugal has quit IRC | 15:26 | |
| rfolco | rlandy, https://zuul-ci.org/docs/zuul/user/jobs.html#git-repositories | 15:27 |
| *** myoung is now known as myoung|biaf | 15:27 | |
| rlandy | looking | 15:28 |
| *** sanjayu_ has quit IRC | 15:28 | |
| rfolco | repos should be listed in 'projects' to let zuul resolve job inheritance, etc | 15:35 |
| rfolco | but if you are going to use the repo in the job, should be in required-projects | 15:35 |
| rfolco | legacy-dsvm-base had more projects in 'required-projects' as they were required for devstack based deploys: https://github.com/openstack-infra/openstack-zuul-jobs/blob/41ea3948eae1f7e30ef7bea87443d14536fb446e/zuul.d/jobs.yaml#L862 | 15:36 |
| rlandy | rfolco: this is ovb though | 15:38 |
| rlandy | which runs legacy | 15:38 |
| rlandy | https://github.com/openstack-infra/tripleo-ci/blob/master/zuul.d/base.yaml#L11 wrong job | 15:39 |
| rlandy | ah | 15:40 |
| rlandy | but I need to make the change in multinode to get the gates to pass | 15:40 |
| rfolco | add 'bopenstack/browbeat' to required-projects for the job that runs ovb (or its parent) | 15:42 |
| rfolco | rlandy, I guess its tripleo-ci-dsvm ? | 15:43 |
| rfolco | https://github.com/openstack-infra/tripleo-ci/blob/master/zuul.d/base.yaml#L57 | 15:43 |
| rlandy | rfolco: I need to add it where you suggested to get multinode to pass with the change that is needed for ovb | 15:45 |
| rlandy | but ovb is doing ok with that change | 15:45 |
| rfolco | rlandy, logs for ovb job happy ? | 15:48 |
| rlandy | still running | 15:48 |
| rlandy | https://review.rdoproject.org/zuul3/status.html | 15:48 |
| rfolco | rlandy, are you sure ovb executes run-v3 workflow ? | 15:50 |
| chkumar|ruck | sshnaidm|rover: please have a look at this one https://review.openstack.org/#/c/582503/ tmux one | 15:50 |
| chkumar|ruck | thanks! | 15:50 |
| rfolco | brb hungry | 15:51 |
| rlandy | rfolco: looking for exact parent of those jobs | 15:51 |
| rfolco | rlandy, adding browbeat to required-projects doesn't hurt other jobs... maybe you can add to the multinode abstract job layer if the legacy one. If the new one, add to the tripleo-ci-base. | 15:52 |
| rlandy | rfolco:problem is they all have their own run.yaml | 15:52 |
| rlandy | https://github.com/rdo-infra/rdo-jobs/tree/master/playbooks/legacy | 15:52 |
| rlandy | https://github.com/rdo-infra/rdo-jobs/blob/master/playbooks/legacy/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/run.yaml | 15:53 |
| rlandy | ^^ example | 15:53 |
| rfolco | I see coz it was automatically converted | 15:54 |
| rfolco | rlandy, grabbing food while I think on something | 15:55 |
| *** tesseract has quit IRC | 15:57 | |
| *** d0ugal has joined #oooq | 15:57 | |
| *** gkadam has quit IRC | 15:59 | |
| rlandy | https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/jobs.yaml | 16:02 |
| rlandy | legacy-dsm-base | 16:02 |
| rlandy | rfolco: ^^ | 16:02 |
| weshay | sshnaidm|rover, interesting https://code.engineering.redhat.com/gerrit/#/c/143982/ is not picked up in the internal jobs | 16:04 |
| weshay | https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/periodic-master-rdo_trunk-featureset020-1ctlr_1comp_64gb/ws/config/environments/oooq-internal.yml/*view*/ | 16:04 |
| weshay | it checkouted out the latest | 16:06 |
| weshay | https://thirdparty.logs.rdoproject.org/jenkins-periodic-master-rdo_trunk-featureset020-1ctlr_1comp_64gb-102/console.txt.gz#_2018-07-17_11_35_22_195 | 16:06 |
| weshay | http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/commit/ | 16:07 |
| rfolco | rlandy, yep it should be :) | 16:07 |
| rlandy | rfolco: set putting up a new review for you to look at | 16:07 |
| rlandy | sec | 16:07 |
| panda | rlandy: for the 3nodes is that the only change you had to make ? because insertafter: EOF is the default behaviour .. | 16:09 |
| rlandy | panda: yep | 16:12 |
| rlandy | the debug was involved | 16:12 |
| rlandy | but the answer was simple | 16:13 |
| weshay | jenkins :( https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/periodic-master-rdo_trunk-featureset020-1ctlr_1comp_64gb/ws/config/environments/oooq-internal.yml/*view*/ | 16:13 |
| rlandy | panda: https://github.com/openstack/tripleo-quickstart/blob/master/config/nodes/2ctlr.yml | 16:13 |
| rlandy | ^^ is the problem | 16:13 |
| rlandy | the ordering in /etc/nodepool/sub_nodes_private is specific | 16:13 |
| rlandy | otherwise it does not match : "{{ hostvars[groups['overcloud'][1]].ansible_hostname }}" | 16:14 |
| *** vinaykns has quit IRC | 16:16 | |
| panda | rlandy: https://docs.ansible.com/ansible/2.5/modules/lineinfile_module.html. insertafter is EOF by default | 16:16 |
| rlandy | panda: I know | 16:17 |
| rlandy | but consistent failure before | 16:17 |
| *** vinaykns has joined #oooq | 16:17 | |
| rlandy | twice passing after | 16:17 |
| rlandy | we run in a loop | 16:17 |
| *** d0ugal has quit IRC | 16:18 | |
| *** myoung|biaf is now known as myoung | 16:19 | |
| panda | this is not coding, this is sorcery | 16:19 |
| panda | you're a wizard rlandy | 16:20 |
| rlandy | panda: well - let's see how long it lasts | 16:20 |
| rlandy | maybe we just got lucky with the ordering on the last two rounds | 16:20 |
| panda | meerge it before it changes idea | 16:21 |
| rlandy | panda: if it starts to fail again, at least we know where the error is | 16:21 |
| rlandy | we can hack something to force the order if need be | 16:21 |
| panda | mmhh, maybe that's why this job is so unstable, it was put on nonvoting back and forth | 16:21 |
| *** vinaykns has left #oooq | 16:22 | |
| rlandy | panda: that would make sense | 16:22 |
| weshay | rlandy, sorry to distract you.. | 16:22 |
| weshay | however.. /me wonders how much of a lift it would be | 16:22 |
| weshay | to adjust ovb jobs to match space requirements | 16:22 |
| weshay | https://logs.rdoproject.org/17/582917/3/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/c9cf398/logs/undercloud/home/zuul/undercloud_install.log.txt.gz | 16:22 |
| rlandy | weshay: so we have two options ... | 16:23 |
| rlandy | we could use another flavor | 16:24 |
| rlandy | downstream we add a volume and boot from volume | 16:24 |
| *** links has quit IRC | 16:24 | |
| weshay | Tengu, fyi ^ | 16:24 |
| rlandy | or we would have to play with partitions | 16:24 |
| rlandy | tbh, I'd have to look at how that disk is partitioned to start with | 16:25 |
| rlandy | but the undercloud is s nodepool node | 16:25 |
| weshay | rlandy, k.. let me ask Tengu what other things he may be adding and how critical it may be to have a set of upstream jobs meeting these requirements | 16:25 |
| rlandy | we can look/ask infra ow that node is built | 16:25 |
| weshay | rlandy, Tengu it doesn't seem that critical | 16:25 |
| rlandy | weshay: ^^ | 16:25 |
| * weshay gives up on jenkins | 16:26 | |
| rlandy | we can get nodepool to make the node with diff partitions | 16:27 |
| weshay | rlandy, hrm.. upstream seems like too much w/o consulting w/ infra | 16:28 |
| weshay | rlandy, ovb is a maybe | 16:28 |
| weshay | it' | 16:28 |
| weshay | it's pretty close to matching now | 16:28 |
| weshay | so close in fact.. maybe we can get Tengu to change the requirements | 16:28 |
| rlandy | weshay: even with ovb, the undercloud node is from nodepool | 16:28 |
| weshay | from 60 -> 55 | 16:28 |
| rlandy | in rdocloud | 16:28 |
| *** udesale has quit IRC | 16:29 | |
| weshay | ya.. but from rdo nodepool | 16:29 |
| weshay | our resources | 16:29 |
| weshay | not infra | 16:29 |
| weshay | s | 16:29 |
| rlandy | correct | 16:29 |
| rlandy | that is where we can request the change | 16:29 |
| weshay | upstream 2018-07-17 10:40:44 | Message: The available space on the root partition is 33.1 GB, but it should be at least 60 GB. | 16:29 |
| weshay | 2018-07-17 10:40:44 | | 16:29 |
| rlandy | forget that one | 16:29 |
| rlandy | yep rdo infra | 16:29 |
| weshay | rlandy, ah k | 16:29 |
| rlandy | ask them to up the node partition | 16:29 |
| rlandy | we have no control there | 16:29 |
| *** d0ugal has joined #oooq | 16:30 | |
| agopi | weshay: ping | 16:38 |
| weshay | agopi, ellooooo | 16:40 |
| agopi | somethign weird happened with jenkins | 16:40 |
| agopi | the jobs were aborted by wesley it says. | 16:41 |
| weshay | agopi, jenkins sucks | 16:41 |
| weshay | agopi, yup | 16:41 |
| weshay | agopi, maintenance reboot | 16:41 |
| weshay | sorry | 16:41 |
| agopi | oh okay cool, np just wanted to let you know | 16:41 |
| *** kopecmartin has quit IRC | 16:42 | |
| weshay | agopi, sorry man. it's not pulling the right files in from git | 16:42 |
| weshay | very annoying | 16:42 |
| agopi | oh yes, i've noticed some more weird stuff wrt jenkins. | 16:44 |
| agopi | i'll put everything in an etherpad/doc along with the problems that ive noticed with perfci | 16:44 |
| agopi | and myoung rlandy you and I cna look at it | 16:44 |
| agopi | whenever everyone has a bit of time. will let yall know when i get doen with it | 16:45 |
| *** agopi is now known as agopi|lunch | 16:47 | |
| *** amoralej is now known as amoralej|off | 16:48 | |
| weshay | myoung, are we allowed to ssh into the internal jenkins? | 16:50 |
| *** amoralej|off has quit IRC | 16:52 | |
| myoung | weshay: afaik yes, but only as the jenkins user, we don't have root access or we lose the SLA | 16:55 |
| myoung | weshay: but i need to verify...in the past I've found it faster to make a custom job on the master node, where you can run/do whatever you want...with a log :) | 16:55 |
| myoung | weshay: agopi|lunch reached out a short while ago, and I asked him to put details into an etherpad and schedule a short session with rlandy/you/me/whomever else | 16:56 |
| rlandy | myoung: yes sorry | 16:57 |
| rlandy | been a little distracted | 16:57 |
| rlandy | will respond in a bit | 16:57 |
| myoung | rlandy: no worries, just wanted to do it in the open - i think agopi|lunch proposed 4pm quick chat | 16:57 |
| *** agopi|lunch is now known as agopi | 16:58 | |
| agopi | once i get everything into etherpad, we cna have a look at it, and we cna do the quickchat even tomorrow. | 16:59 |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000 (1 more message) | 17:02 |
| *** dtantsur is now known as dtantsur|afk | 17:12 | |
| *** atoth has quit IRC | 17:12 | |
| *** dtrainor has quit IRC | 17:15 | |
| myoung | agopi: ack | 17:16 |
| myoung | holey moley every connected device just freaked out with a weather statement...massive thunderstorm just hit. lights flickered lol. | 17:17 |
| *** dtrainor has joined #oooq | 17:17 | |
| rlandy | really? nothing here | 17:18 |
| Tengu | weshay: talking about me? :) | 17:24 |
| weshay | Tengu, ya you :) | 17:24 |
| Tengu | so basically the validation causes some issues right? I don't know from where the 60G requirement comes. | 17:24 |
| weshay | Tengu, it doesn't cause an issue.. we've just never met those requirements in CI | 17:24 |
| Tengu | weshay: :) | 17:24 |
| weshay | Tengu, my suggestion is to propose a 55gb min disk.. vs 60 | 17:25 |
| weshay | then at least your code could be CI'd in ovb | 17:25 |
| Tengu | well, we might even get other problems, as I'll activate almost any validations from the tripleo-validations. | 17:25 |
| Tengu | weshay: I can ask the Validation team for the source of this 60G. | 17:25 |
| weshay | Tengu, ovb may pass some of those | 17:25 |
| Tengu | I'm pretty sure even 40 would be fine. | 17:25 |
| weshay | ha.. 2 bucks says they don't know | 17:25 |
| weshay | Tengu, even still we only have 30gb upstream | 17:26 |
| Tengu | pretty sure as well ;). | 17:26 |
| Tengu | weshay: fact is, there are wome things that WILL take space on a productive undercloud : container registry. | 17:26 |
| Tengu | I'm pretty sure the 60G comes from that fact. | 17:26 |
| Tengu | i.e. "ask for enough space in order to avoid future issues with the registry + packages". Seems pretty legit. | 17:27 |
| weshay | Tengu, right.. so couldn't you make it configurable for ci purposes? | 17:29 |
| weshay | Tengu, bend the world mate | 17:29 |
| weshay | it's software | 17:29 |
| *** trown is now known as trown|lunch | 17:29 | |
| Tengu | weshay: well. This might be done, in fact. | 17:30 |
| weshay | Tengu, ok... please check | 17:30 |
| weshay | Tengu, doesn't make sense to use more resources in ci if we don't have to | 17:30 |
| Tengu | weshay: I can check that tomorrow. Et can actually use the "--extra-thingy-foo" from ansible-playbook allowing to override the 60G via the command line. | 17:30 |
| weshay | woot | 17:30 |
| Tengu | weshay: and we might use some environment variables passed from the CI shell… could we? | 17:30 |
| * weshay fucking hates containers atm | 17:31 | |
| Tengu | we can do something like that. meaning the deplo would looks like CI_DISK_SPACE_FOO_BAR=30 openstack undercloud install --use-heat | 17:32 |
| Tengu | or something like that | 17:32 |
| Tengu | more or less | 17:32 |
| Tengu | weshay: will give it a try tomorrow on my lab. | 17:32 |
| Tengu | myoung: -^^^ | 17:32 |
| weshay | Tengu++ thanks | 17:32 |
| hubbot` | weshay: Tengu's karma is now 1 | 17:32 |
| Tengu | you might be interested by this nice solution maybe :). | 17:33 |
| Tengu | weshay: and we might use the same… "hack" for the memory check - I'm pretty sure we don't have the required 16G in the CI | 17:33 |
| weshay | Tengu, really the whole validations workflow should use the same thing for CI | 17:33 |
| weshay | Tengu, create a profile for CI that works.. | 17:34 |
| weshay | maybe w/ some known negative tests | 17:34 |
| Tengu | weshay: ok, maybe the best way would be to give a configuration file. | 17:34 |
| weshay | aye | 17:34 |
| Tengu | and we take the values from there, so that you just inject that config file for validations. | 17:34 |
| Tengu | already thought about it when I saw the issue in the CI | 17:34 |
| Tengu | :) | 17:34 |
| Tengu | will add this capability in some way. | 17:34 |
| Tengu | weshay: as you saw in the #tripleo meeting, I'll work a lot on validations, so we might work together ;) | 17:35 |
| Tengu | weshay: maybe the undercloud.conf might have those ci_[diskspace|memory|others] directly. | 17:37 |
| Tengu | that way, we only have one file, and basta. anyway. Check that tomorrow. :) | 17:37 |
| weshay | Tengu, that would be nice.. however.. I don't think you want to do it in a way that customers could just change | 17:37 |
| weshay | Tengu, you need an easter egg feature | 17:38 |
| weshay | sshnaidm|rover, chkumar|ruck gate is reseting on container updates in the undercloud install afaict | 17:38 |
| Tengu | weshay: I'll call the Easter Rabbit, he's a friend :) | 17:38 |
| weshay | lolz.. I show up to parties in that suite and everyone runs.. /me has no idea why | 17:39 |
| weshay | I'm just a friendly rabbit | 17:39 |
| Tengu | :) | 17:39 |
| Tengu | so yeah. that's a nice project :) | 17:40 |
| Tengu | and we will get a proper way to validate the env, and some nice way to actually NOT fail in the CI :). I'll also create the BP and spec tomorrow, so that we can work it a bit more. | 17:40 |
| Tengu | so, now, I'm really off :) | 17:41 |
| Tengu | see you! | 17:41 |
| *** zoli|wfh is now known as zoli|gone | 17:44 | |
| weshay | :) | 17:44 |
| weshay | Thanks Tengu | 17:44 |
| weshay | rlandy, what could I do for 5-10 minutes of ironic debug help | 17:46 |
| weshay | I think I need to get something other than timedout | 17:46 |
| weshay | rlandy, nvrmind... ovb is now failing on introspection | 17:49 |
| weshay | thank goodness | 17:49 |
| *** myoung is now known as myoung|lunch | 17:50 | |
| weshay | jrist, https://bugs.launchpad.net/tripleo/+bug/1782211 | 17:56 |
| openstack | Launchpad bug 1782211 in tripleo "ironic locks and times out the tripleo deployment" [Critical,Triaged] | 17:56 |
| jrist | booP! | 17:58 |
| rlandy | weshay: sorry - lunch - you still want me to look at ironic introspection | 18:00 |
| jrist | yes please | 18:04 |
| jrist | :) | 18:04 |
| rlandy | state is "clean wait" - hmmm | 18:05 |
| rlandy | ok - backtracking on the logs | 18:06 |
| rlandy | https://logs.rdoproject.org/76/581376/7/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/14d80f8/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz works | 18:12 |
| *** atoth has joined #oooq | 18:13 | |
| rlandy | sshnaidm|rover; where are we tracking periodic jobs now? | 18:21 |
| rlandy | http://cistatus.tripleo.org/promotion/ empty | 18:22 |
| sshnaidm|rover | rlandy, yeah, need to update it after zuulv3 switch.. currently I use promotion log: http://38.145.34.55/ | 18:29 |
| rlandy | https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/8776c03/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz | 18:29 |
| rlandy | ^^ works | 18:30 |
| rlandy | sshnaidm|rover: thanks 0 I was looking to compare passes vs failures on logs | 18:30 |
| rlandy | just using this now: https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/ | 18:31 |
| *** trown|lunch is now known as trown | 18:31 | |
| rlandy | https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/cb21587/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz | 18:31 |
| rlandy | fails | 18:31 |
| rlandy | https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/3365d1e/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz | 18:32 |
| rlandy | passes | 18:32 |
| rlandy | so we have a change from 2018-07-17 02:56 to 2018-07-17 13:47 | 18:32 |
| *** sshnaidm|rover has quit IRC | 18:37 | |
| *** sshnaidm has joined #oooq | 18:39 | |
| *** sshnaidm is now known as sshnaidm|rover | 18:44 | |
| *** myoung|lunch is now known as myoung | 18:47 | |
| rlandy | DriverLoadError: Driver, hardware type or interface ilo-pxe could not be loaded. | 18:52 |
| rlandy | Jul 17 00:58:35 undercloud.localdomain dockerd-current[31662]: 2018-07-17 00:58:35.981 7 ERROR oslo_service.service [req-07e64ab8-d320-4787-922d-758177264119 - - - - -] Error starting thread.: DriverLoadError: Driver, hardware type or interface ilo-pxe could not be loaded. Reason: [Errno 13] Permission denied: '/var/lib/ironic/httpboot/boot.ipxe' - shows on passing one as well | 18:54 |
| rlandy | weshay: hi | 19:02 |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode-oooq, legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000 (1 more message) | 19:02 |
| rlandy | weshay: do you only hit the above error in libvirt reproducer with the same hash? | 19:02 |
| *** panda is now known as panda|off | 19:03 | |
| weshay | rlandy, which error? | 19:03 |
| rlandy | error about pxe_ilo I think is unrelatde | 19:03 |
| weshay | <rlandy> DriverLoadError: Driver, hardware type or interface ilo-pxe could not be loaded. | 19:03 |
| rlandy | unrelated | 19:03 |
| weshay | rlandy, https://bugs.launchpad.net/tripleo/+bug/1782211 | 19:03 |
| openstack | Launchpad bug 1782211 in tripleo "ironic locks and times out the tripleo deployment" [Critical,Triaged] | 19:03 |
| panda|off | rfolco: I'm off sf is all yours | 19:03 |
| weshay | rlandy, it's the same error | 19:03 |
| rlandy | "clean wait", | 19:03 |
| weshay | different driver | 19:03 |
| rlandy | ^^ that is the failure | 19:03 |
| weshay | ya | 19:03 |
| rlandy | so in the bug you pasted two logs | 19:03 |
| weshay | ya | 19:04 |
| rlandy | the one reporting ilo shows up in passing jobs | 19:04 |
| rlandy | the clean wait thing is what is killing it | 19:04 |
| weshay | oh dam | 19:04 |
| rlandy | https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/ | 19:04 |
| rlandy | compare the two logs from 07/17 | 19:04 |
| weshay | k | 19:05 |
| rlandy | 3365d1e/ | 19:05 |
| rlandy | and cb21587/ | 19:05 |
| rfolco | panda|off, thx | 19:05 |
| rlandy | weshay: when you run your libvirt reproducer | 19:05 |
| rlandy | is it the same dlrn hash? | 19:05 |
| weshay | rlandy, this has been happening for the last number of hashes | 19:05 |
| rlandy | ironic versions look the same in both jobs | 19:05 |
| weshay | at least in libvirt | 19:05 |
| rlandy | but in ovb | 19:05 |
| weshay | rlandy, not w/ libvirt reproducer | 19:05 |
| weshay | but w/ quickstart.sh | 19:06 |
| rlandy | the previous promotion job on 01/17 passes | 19:06 |
| rlandy | so ... | 19:06 |
| rlandy | you can increase the clean timeout | 19:06 |
| weshay | hrm | 19:06 |
| weshay | k | 19:06 |
| rlandy | is it really is ***just** a timeout | 19:06 |
| rlandy | but idk | 19:06 |
| * rlandy gets | 19:06 | |
| weshay | rlandy, I was wanting to just try to boot the nodes by hand | 19:06 |
| weshay | openstack baremetal node $foo commands | 19:06 |
| rlandy | good luck with that | 19:07 |
| weshay | I hate you ironic | 19:07 |
| rlandy | once you hit the clean thing, run for the hills | 19:07 |
| rlandy | I don;t hate ironic | 19:07 |
| weshay | rlandy, let me get a repro up in rdo | 19:07 |
| weshay | rlandy, last time I hit this.. | 19:07 |
| rlandy | weshay: sec - let me get you the setting | 19:07 |
| weshay | rlandy, it was pretty nice to watch the console off rdo-cloud | 19:08 |
| weshay | and run introspection | 19:08 |
| rlandy | dtantsur|afk keeps asking us to capture that | 19:08 |
| weshay | rlandy, capture what? | 19:08 |
| rlandy | weshay: this has some nice info: https://access.redhat.com/solutions/3349081 | 19:09 |
| weshay | the console? | 19:09 |
| rlandy | capture the bmc console | 19:09 |
| rlandy | ironic.conf #clean_callback_timeout = 1800 | 19:09 |
| rlandy | in fact, we have a setting to control cleaning | 19:11 |
| weshay | rlandy, I'm on a libvirt system now w/ the issue | 19:11 |
| rlandy | weshay: can I access it? | 19:11 |
| rlandy | https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset001.yml#L46 | 19:11 |
| rlandy | ^^ can set that to false | 19:11 |
| weshay | # disable timeout. (integer value) | 19:11 |
| weshay | #clean_callback_timeout = 1800 | 19:11 |
| weshay | it's not set | 19:11 |
| rlandy | right | 19:12 |
| weshay | rlandy, let's jump on blue and tmate | 19:12 |
| rlandy | joining | 19:12 |
| *** tosky has quit IRC | 19:20 | |
| *** jfrancoa has quit IRC | 20:00 | |
| rfolco | rlandy, does this need +W again? https://review.openstack.org/#/c/583022/ | 20:16 |
| rlandy | rfolco: no - failure s non-voting | 20:17 |
| *** brault has quit IRC | 20:17 | |
| rfolco | rlandy, failed on gate job, I did recheck but I believe it needs to +W again to merge | 20:18 |
| *** brault has joined #oooq | 20:18 | |
| rlandy | gate needs to pass | 20:18 |
| rlandy | check passed | 20:18 |
| rlandy | Workflow | 20:18 |
| rlandy | +1 Marios Andreou is still there | 20:18 |
| rfolco | there is no regate :) | 20:18 |
| rfolco | marios, still around ? https://review.openstack.org/#/c/583022/ would you +W again please ? | 20:19 |
| myoung | agopi: did you still want to chat today? | 20:22 |
| agopi | hey myoung | 20:22 |
| agopi | i rekicked some jobs to see | 20:22 |
| agopi | meanwhile | 20:22 |
| myoung | ok | 20:22 |
| agopi | https://etherpad.openstack.org/p/perfci_jenkins_derailment | 20:22 |
| * myoung looks | 20:23 | |
| agopi | myoung, weshay rlandy ^ | 20:23 |
| myoung | agopi: i just commented | 20:24 |
| agopi | yes myoung but it still didnt get triggered on 6/13 | 20:24 |
| agopi | thats when it was actually updated | 20:25 |
| weshay | agopi, I fixed 2 | 20:25 |
| weshay | agopi, https://review.openstack.org/#/c/581789/ | 20:25 |
| weshay | agopi, read through that.. and please focus on opening lp's and bringing the lp's to our attention | 20:26 |
| agopi | weshay++ | 20:26 |
| hubbot` | agopi: weshay's karma is now 6 | 20:26 |
| agopi | ack weshay | 20:26 |
| weshay | agopi, | 20:26 |
| weshay | 00:00:18.888 Collecting cmd2==0.8.5 (from -r requirements.txt (line 1)) | 20:26 |
| weshay | 00:00:18.890 Could not find a version that satisfies the requirement cmd2==0.8.5 (from -r requirements.txt (line 1)) (from versions: ) | 20:26 |
| weshay | 00:00:18.891 No matching distribution found for cmd2==0.8.5 (from -r requirements.txt (line 1)) | 20:26 |
| weshay | from the console | 20:26 |
| weshay | it's very clear it's a pip issue | 20:26 |
| agopi | okay weshay, it failed on a different version later on Collecting functools32; python_version == "2.7" (from jsonschema<3,>=0.7->warlock<2,>=1.2.0->python-glanceclient>=2.8.0->python-openstackclient->-r requirements.txt (line 11)) | 20:28 |
| agopi | 16:08:53 Could not find a version that satisfies the requirement functools32; python_version == "2.7" (from jsonschema<3,>=0.7->warlock<2,>=1.2.0->python-glanceclient>=2.8.0->python-openstackclient->-r requirements.txt (line 11)) (from versions: ) | 20:28 |
| weshay | agopi, sure.. this is part of the suckage of running internally | 20:28 |
| agopi | also myoung https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/user/agopi/my-views/view/Browbeat_view/job/rdo-promote-pike-rdo_trunk-performance/ check this | 20:28 |
| weshay | agopi, if you put up a pip mirror that could be avoided | 20:28 |
| agopi | you'll notice it never built off the 06/13 | 20:28 |
| weshay | rlandy, these ovb jobs are building images | 20:29 |
| weshay | should they not be pulling the images from the image server? | 20:29 |
| weshay | oh because it's periodic | 20:30 |
| weshay | recreate | 20:30 |
| weshay | nvrmind | 20:30 |
| myoung | agopi: hrm...looking at the URL poll - that is strange... | 20:35 |
| myoung | something's wrong | 20:35 |
| *** brault has quit IRC | 20:37 | |
| rlandy | sorry - just in the middle of an rdo-jobs edit/commit | 20:37 |
| rlandy | rfolco: this required-projects is a nightmare ... https://review.rdoproject.org/r/#/c/14808/6/zuul.d/zuul-legacy-jobs.yaml | 20:40 |
| rlandy | overrides on every job | 20:40 |
| rfolco | rlandy, isn't there a parent job you can put this ? | 20:41 |
| rfolco | oh jeez hundreds of jobs | 20:41 |
| rlandy | rfolco: what will that help if the job itself overrides it?? | 20:41 |
| rlandy | oh yeah | 20:41 |
| rlandy | let's see what it does | 20:41 |
| rlandy | ugh - now another merge issue | 20:42 |
| rlandy | this doc file will kill me | 20:42 |
| rfolco | rlandy, like a abstract parent job just to include browbeat as required-project | 20:42 |
| rfolco | hmm legacy-dsvm-base is the upstream one in openstack-zuul-jobs, right? | 20:43 |
| rlandy | rfolco :correct | 20:44 |
| rlandy | but if the child job override that setting, won't matter | 20:45 |
| rfolco | rlandy, yeah I was just trying to reduce the amount of times you copy that config. Not easy. If you change parent, will have the same issue. | 20:48 |
| rlandy | nightmare!!!!! | 20:48 |
| agopi | rlandy: wrt https://review.openstack.org/#/c/581484/5..6/config/general_config/featureset053.yml where can we put them then? | 20:49 |
| *** florianf has quit IRC | 20:50 | |
| rlandy | agopi: in the rdocloud settings | 20:50 |
| rlandy | we can work on that once we have a running job | 20:50 |
| rlandy | sorry - not ignoring you | 20:50 |
| rlandy | just a lot of infra work to get this to pass | 20:50 |
| rlandy | I just want to see one set of jobs run | 20:50 |
| agopi | okay rlandy, lmk how i can make myself useful. | 20:51 |
| rlandy | agopi: will ping you when we have the infra set | 20:51 |
| agopi | ack rlandy | 20:52 |
| weshay | rlandy, do you recall if your are making some decision to run the ovs vxlan setup | 20:56 |
| weshay | based on undercloud-setup? | 20:56 |
| weshay | http://logs.openstack.org/90/581790/3/check/tripleo-ci-centos-7-scenario008-multinode-oooq-container/14f2619/logs/undercloud/home/zuul/undercloud_install.log.txt.gz | 20:56 |
| *** d0ugal has quit IRC | 20:58 | |
| agopi | akc myoung just read your message | 20:59 |
| *** d0ugal has joined #oooq | 21:00 | |
| *** d0ugal has quit IRC | 21:00 | |
| *** d0ugal has joined #oooq | 21:00 | |
| *** jtomasek has quit IRC | 21:01 | |
| hubbot` | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode- (1 more message) | 21:02 |
| myoung | agopi, weshay: will track this as a ruck/rover trello card | 21:04 |
| myoung | or is LP better? | 21:04 |
| weshay | myoung, what? | 21:04 |
| weshay | agopi's stuff | 21:04 |
| myoung | https://etherpad.openstack.org/p/perfci_jenkins_derailment | 21:04 |
| myoung | the URL trigger wierdness | 21:05 |
| weshay | myoung, that's not on us | 21:05 |
| myoung | k | 21:05 |
| myoung | ack | 21:05 |
| weshay | agopi, you have access to the jenkins server? | 21:05 |
| *** jtomasek has joined #oooq | 21:05 | |
| weshay | rlandy, 2018-07-17 20:40:55 | Introspection completed. | 21:06 |
| weshay | 2018-07-17 20:40:55 | + openstack overcloud node provide --all-manageable | 21:06 |
| rlandy | weshay: not sure what you are referring to | 21:06 |
| rlandy | some decision to run the ovs vxlan setup | 21:06 |
| rlandy | weshay: great - re; introspection | 21:06 |
| myoung | agopi: left notes in the etherpad | 21:06 |
| rlandy | for ovb we may then be on the border for clean up timeout | 21:06 |
| weshay | rlandy, I think this.. https://review.openstack.org/#/c/579161/ | 21:07 |
| weshay | rlandy, so until that lands.. the vxlan setup is still executed by undercloud-setup right? | 21:07 |
| weshay | ya | 21:07 |
| rlandy | oh for the reproducer | 21:07 |
| weshay | rlandy, no.. upstream | 21:07 |
| weshay | not the repro | 21:07 |
| weshay | rlandy, /me goes to #tripleo | 21:08 |
| rlandy | upstream has already merged | 21:08 |
| agopi | weshay, yes i've acess to the jenkins to create/update/manage jobs if that's what you're asking about. | 21:10 |
| weshay | agopi, ok.. | 21:11 |
| weshay | agopi, we are not jenkins support, we can help you but it's not on us to resolve | 21:11 |
| weshay | agopi, I'm not sure what is wrong w/ the trigger | 21:11 |
| weshay | agopi, however I *think* something is wrong w/ the time settings | 21:12 |
| weshay | as I see some patches not getting checked out appropriately | 21:12 |
| weshay | agopi, this jenkins is supported by IT, rhos-ops | 21:12 |
| weshay | agopi, it is fully supported in fact.. | 21:12 |
| agopi | weshay, ack, i'll raise a ticket there. | 21:12 |
| weshay | myoung, can point you at the process for opening tickets | 21:13 |
| myoung | weshay, ack. agopi putting into the etherpad | 21:16 |
| agopi | myoung++ | 21:17 |
| hubbot` | agopi: myoung's karma is now 2 | 21:17 |
| agopi | i'm looking at other pipelines | 21:17 |
| myoung | agopi: https://etherpad.openstack.org/p/perfci_jenkins_derailment @L69 (tldr: one.engineering.company.com) | 21:19 |
| weshay | myoung, can you put in a request to have ntp checked out on the box | 21:19 |
| weshay | that shit is busted afaict | 21:19 |
| myoung | weshay: siure | 21:19 |
| myoung | weshay: sure | 21:19 |
| agopi | thanks myoung | 21:19 |
| weshay | myoung, that would obviously affect agopi's triggers too | 21:19 |
| myoung | weshay: is there a bug/issue we've been investigating already I can/should link to the report (apart from what agopi is seeing) | 21:20 |
| myoung | ? | 21:20 |
| weshay | fak man.. it's terrible.. so.. make a bogus change in a comment to a settings file in tripleo-environments:/config/foo, merge the change | 21:20 |
| weshay | myoung, watch how jenkins pulls the right hash for the job.. w/ the change.. and watch how the change is not in the workspace | 21:21 |
| weshay | afaict.. our clock is about 30min off | 21:21 |
| weshay | heh.. but it's not | 21:22 |
| weshay | :( | 21:22 |
| weshay | something is off | 21:22 |
| weshay | [rhos-ci@dhcp-12-53-28 workspace]$ date | 21:22 |
| weshay | Tue Jul 17 21:21:57 UTC 2018 | 21:22 |
| weshay | [rhos-ci@dhcp-12-53-28 workspace]$ | 21:22 |
| weshay | agopi, myoung at anyrate.. something is wrong w/ jenkins | 21:24 |
| weshay | and it's up to support to fix iyt | 21:24 |
| weshay | it | 21:24 |
| agopi | ack weshay | 21:30 |
| *** holser_ has quit IRC | 21:53 | |
| *** apetrich has quit IRC | 22:08 | |
| *** apetrich has joined #oooq | 22:12 | |
| *** holser_ has joined #oooq | 22:15 | |
| *** myoung is now known as myoung|off | 22:31 | |
| *** holser_ has quit IRC | 22:31 | |
| *** jtomasek has quit IRC | 22:36 | |
| hubbot` | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario009-multinode- (1 more message) | 23:02 |
| *** tosky has joined #oooq | 23:09 | |
| *** rlandy has quit IRC | 23:24 | |
| *** agopi is now known as agopi|off | 23:36 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!