| hubbot1 | FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/rocky: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7 (2 more messages) | 01:46 |
|---|---|---|
| *** ykarel has joined #oooq | 03:31 | |
| *** udesale has joined #oooq | 03:34 | |
| *** skramaja has joined #oooq | 03:37 | |
| hubbot1 | FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/rocky: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7 (2 more messages) | 03:46 |
| *** udesale has quit IRC | 04:03 | |
| *** udesale has joined #oooq | 04:24 | |
| *** udesale has quit IRC | 04:25 | |
| *** udesale has joined #oooq | 04:25 | |
| *** ykarel has quit IRC | 05:22 | |
| *** jbadiapa has joined #oooq | 05:25 | |
| *** quique|rover|off is now known as quiquell|rover | 05:30 | |
| *** jtomasek has joined #oooq | 05:34 | |
| chkumar|ruck | quiquell|rover: Good morning :-) | 05:42 |
| chkumar|ruck | quiquell|rover: I am taking a look at periodic jobs | 05:42 |
| *** ykarel has joined #oooq | 05:42 | |
| quiquell|rover | chkumar|ruck: good morning | 05:44 |
| *** jtomasek has quit IRC | 05:44 | |
| quiquell|rover | still no promotions :-/ | 05:44 |
| quiquell|rover | This is impossible | 05:44 |
| quiquell|rover | chkumar|ruck: jobs are running now | 05:45 |
| *** jtomasek has joined #oooq | 05:45 | |
| hubbot1 | FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/rocky: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7 (2 more messages) | 05:46 |
| chkumar|ruck | quiquell|rover: fs019 is failing on master and rocky with overcloud deploy step | 05:49 |
| quiquell|rover | chkumar|ruck: Yep, on friday we saw the problem | 05:51 |
| quiquell|rover | chkumar|ruck: Didn't have time to find a soltuion | 05:51 |
| chkumar|ruck | quiquell|rover: do we have open bug for fs-19 | 05:51 |
| quiquell|rover | chkumar|ruck: Let's check queens | 05:52 |
| quiquell|rover | chkumar|ruck: yep | 05:52 |
| quiquell|rover | https://bugs.launchpad.net/tripleo/+bug/1792296 | 05:52 |
| openstack | Launchpad bug 1792296 in tripleo "Overcloude deploy error:Timed out waiting for messages from Execution" [Critical,Triaged] - Assigned to Quique Llorente (quiquell) | 05:52 |
| quiquell|rover | chkumar|ruck: I will find a solution today | 05:52 |
| quiquell|rover | chkumar|ruck: They has post a solution | 05:53 |
| quiquell|rover | chkumar|ruck: We are also affected at master/rocky by this (is intermitent) | 05:53 |
| quiquell|rover | https://bugs.launchpad.net/tripleo/+bug/1792343 | 05:54 |
| openstack | Launchpad bug 1792343 in tripleo "[tripleo] rocky baremetal deployment fails with jq: error: Could not open file /var/lib/heat-config/deployed/<id>.notify.json" [Critical,Triaged] - Assigned to Quique Llorente (quiquell) | 05:54 |
| chkumar|ruck | quiquell|rover: so there are two bugs affecting fs019 ? | 05:54 |
| chkumar|ruck | for master and rocky | 05:54 |
| quiquell|rover | chkumar|ruck: fs019 only affected by the first one | 05:55 |
| quiquell|rover | chkumar|ruck: The other is intermitent issue we can see at different featuresets | 05:55 |
| quiquell|rover | In case you find it as ruck | 05:55 |
| *** saneax has joined #oooq | 05:55 | |
| *** quiquell|rover has quit IRC | 05:56 | |
| chkumar|ruck | quiquell|rover: fs017 tempest telemtry issue https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/90fd211/logs/undercloud/home/zuul/tempest.log.txt.gz | 05:56 |
| *** quiquell has joined #oooq | 05:56 | |
| chkumar|ruck | filing a bz | 05:56 |
| *** quiquell is now known as quiquell|rover | 05:56 | |
| quiquell|rover | chkumar|ruck: What bz ? | 05:56 |
| chkumar|ruck | quiquell|rover: https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/90fd211/logs/undercloud/home/zuul/tempest.log.txt.gz | 05:57 |
| chkumar|ruck | quiquell|rover: telemetry tests failure | 05:57 |
| quiquell|rover | chkumar|ruck: Yep, ykarel told me so on friday | 05:57 |
| quiquell|rover | chkumar|ruck: But Don't know why we didn't have a queens promotion | 05:57 |
| chkumar|ruck | ykarel: do we have fix up for telemetry issue from ceilometer side | 05:57 |
| quiquell|rover | chkumar|ruck: All is suppose to be in place | 05:57 |
| quiquell|rover | chkumar|ruck: fs001 queens has fail :-/ | 05:58 |
| chkumar|ruck | quiquell|rover: fs01 failed on rocky also checking | 05:59 |
| ykarel | chkumar|ruck, yes, sileht proposed upstream | 05:59 |
| ykarel | chkumar|ruck, is bug created for that issue ^^ | 05:59 |
| chkumar|ruck | ykarel: do we have a bug? | 05:59 |
| quiquell|rover | chkumar|ruck: Ok, open a bug if necessary, going to fix the master fs019 issue | 06:00 |
| ykarel | chkumar|ruck, no idea | 06:00 |
| chkumar|ruck | quiquell|rover: ok opening a bz | 06:00 |
| ykarel | chkumar|ruck, prposed fix was https://review.openstack.org/#/c/602625/ but it's failing in ci | 06:00 |
| chkumar|ruck | ykarel: quiquell|rover https://review.openstack.org/#/c/602625/ andhttps://review.openstack.org/#/c/602624/1 | 06:00 |
| chkumar|ruck | two fixes | 06:01 |
| *** quiquell|rover is now known as quique|rover | 06:01 | |
| chkumar|ruck | ykarel: it might also needs fixes in puppet-ceilometer? | 06:01 |
| chkumar|ruck | https://review.openstack.org/#/c/602624/1 | 06:01 |
| *** quique|rover is now known as quiquell|rover | 06:01 | |
| quiquell|rover | chkumar|ruck: Add them to the etherpad to know what we have to focus on today | 06:02 |
| ykarel | chkumar|ruck, not sure about that | 06:02 |
| chkumar|ruck | quiquell|rover: sure | 06:02 |
| ykarel | but good to create lp and set promotion-blocker | 06:02 |
| *** holser|ptg has joined #oooq | 06:03 | |
| ykarel | chkumar|ruck, many tests failed, not sure if ur recheck will help there | 06:03 |
| ykarel | on the ceilometer patch | 06:03 |
| ykarel | unit tests needs to be fixed there | 06:04 |
| *** holser|p_ has joined #oooq | 06:05 | |
| *** jfrancoa has joined #oooq | 06:05 | |
| *** holser|ptg has quit IRC | 06:05 | |
| chkumar|ruck | ykarel: quiquell|rover https://bugs.launchpad.net/tripleo/+bug/1792862 | 06:08 |
| openstack | Launchpad bug 1792862 in tripleo "[master] Telemetry Tempest integration tests failed giving Unable to complete operation on subnet b13c76ec-c851-48d0-91a9-245a2fdcad9b: One or more ports have an IP allocation from this subnet." [Critical,Triaged] - Assigned to Mehdi Abaakouk (sileht) | 06:08 |
| quiquell|rover | chkumar|ruck: ack | 06:08 |
| *** jaganathan has joined #oooq | 06:11 | |
| ykarel | chkumar|ruck, updated ceilometer patch: https://review.openstack.org/#/c/602624/, now should be ok | 06:16 |
| *** rnoriega has quit IRC | 06:17 | |
| *** apetrich has joined #oooq | 06:18 | |
| chkumar|ruck | quiquell|rover: on fs01 for queens and rocky I am seeing two issues related to ironic | 06:18 |
| chkumar|ruck | quiquell|rover: https://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/59f5f7c/logs/undercloud/var/log/ironic/app.log.txt.gz#_2018-09-16_06_30_14_147 | 06:18 |
| chkumar|ruck | quiquell|rover: https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky/4820d62/logs/undercloud/var/log/extra/docker/containers/ironic_pxe_http/log/ironic/ironic-conductor.log.txt.gz#_2018-09-17_02_06_28_644 | 06:19 |
| chkumar|ruck | quiquell|rover: opening two seperate bugs for both | 06:20 |
| *** rnoriega has joined #oooq | 06:20 | |
| quiquell|rover | chkumar|ruck: Feels related but ok | 06:20 |
| *** quiquell|rover is now known as quique|rover|brb | 06:37 | |
| *** udesale has quit IRC | 06:38 | |
| *** udesale has joined #oooq | 06:40 | |
| *** udesale has quit IRC | 06:45 | |
| *** saneax has quit IRC | 07:04 | |
| *** quique|rover|brb is now known as quiquell|rover | 07:07 | |
| *** holser|p_ is now known as holser_ | 07:24 | |
| *** ykarel is now known as ykarel|lunch | 07:45 | |
| hubbot1 | FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/rocky: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7 (2 more messages) | 07:46 |
| *** kopecmartin has joined #oooq | 07:52 | |
| quiquell|rover | chkumar|ruck: queens fs001 has pass now https://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/2f0399c/ | 08:07 |
| chkumar|ruck | quiquell|rover: for queens no jobs are blocking now i think | 08:09 |
| chkumar|ruck | for promotion | 08:09 |
| quiquell|rover | chkumar|ruck: fs020 has to finish but looks good | 08:09 |
| quiquell|rover | chkumar|ruck: Going to cry if we have a queens promotion | 08:10 |
| chkumar|ruck | quiquell|rover: yes same here | 08:10 |
| quiquell|rover | chkumar|ruck: Going to close this https://bugs.launchpad.net/tripleo/+bug/1792872 | 08:14 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Quique Llorente (quiquell) | 08:14 |
| chkumar|ruck | quiquell|rover: ack | 08:17 |
| quiquell|rover | chkumar|ruck: let's wait, maybe it's rdo infra issue | 08:18 |
| chkumar|ruck | ack | 08:23 |
| *** udesale has joined #oooq | 08:25 | |
| *** udesale has quit IRC | 08:27 | |
| *** udesale has joined #oooq | 08:28 | |
| *** udesale has quit IRC | 08:29 | |
| *** udesale has joined #oooq | 08:29 | |
| *** udesale has quit IRC | 08:29 | |
| *** udesale has joined #oooq | 08:30 | |
| *** bogdando has joined #oooq | 08:32 | |
| *** saneax has joined #oooq | 08:35 | |
| *** tosky has joined #oooq | 08:41 | |
| ssbarnea | hi! I am back! | 08:46 |
| *** udesale has quit IRC | 08:47 | |
| chkumar|ruck | quiquell|rover: I think we can add this also http://grafana.openstack.org/d/ACtl1JSmz/afs?orgId=1 in ruck rover dashboard for afs mirror status | 08:49 |
| *** udesale has joined #oooq | 08:55 | |
| quiquell|rover | chkumar|ruck: Agree | 08:56 |
| chkumar|ruck | quiquell|rover: I will check with fedora guys how they track the mirror sync status | 09:00 |
| *** udesale has quit IRC | 09:01 | |
| quiquell|rover | chkumar|ruck: The only thin we need from this dashboard is the mirror.centos time or do we also need fedora's ? | 09:02 |
| quiquell|rover | chkumar|ruck: queens fs020 pass :-), we have to see a promotion | 09:09 |
| quiquell|rover | chkumar|ruck: Yippiii!!! http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&panelId=117&fullscreen | 09:10 |
| quiquell|rover | after near a month :-) | 09:10 |
| *** ykarel|lunch is now known as ykarel | 09:12 | |
| quiquell|rover | chkumar|ruck: What are we missing for pike ? https://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-pike/c396a8a/logs/undercloud/home/zuul/tempest.log.txt.gz | 09:13 |
| quiquell|rover | chkumar|ruck: fs019 is still failing at SharesCephFSTest | 09:14 |
| quiquell|rover | chkumar|ruck: fs050 is broken too at master, going to open lp and dig into it | 09:21 |
| ssbarnea | quiquell|rover: chkumar|ruck : i am still catching up with emails but if there is anything I can help with, feel free to ping me. | 09:28 |
| quiquell|rover | ssbarnea: Welcome back, we have being on-fire all the week, didn't even have time to check UUID issue :-/ | 09:30 |
| ssbarnea | quiquell|rover: sure. i am sure that catching up will take me a lot of time. | 09:31 |
| chkumar|ruck | quiquell|rover: checking fs050 | 09:31 |
| chkumar|ruck | quiquell|rover: fs019 for manila na? | 09:32 |
| quiquell|rover | chkumar|ruck: Looks like wrong ansible version | 09:32 |
| quiquell|rover | chkumar|ruck: what do you mean ? | 09:32 |
| chkumar|ruck | quiquell|rover: sorry ignore | 09:32 |
| chkumar|ruck | quiquell|rover: for fs019 manila pike https://review.openstack.org/601522 this review will do the job | 09:33 |
| quiquell|rover | chkumar|ruck: huff still not merged :-( | 09:35 |
| quiquell|rover | chkumar|ruck: I remember big backport needed for pike : | 09:35 |
| chkumar|ruck | quiquell|rover: yes it is still a workaround | 09:35 |
| quiquell|rover | chkumar|ruck: Let's try to merge stuff now that people are coming back | 09:35 |
| chkumar|ruck | quiquell|rover: is it a known issue http://logs.openstack.org/22/601522/4/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/afb75fd/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz#_2018-09-16_11_16_55 | 09:37 |
| chkumar|ruck | ? | 09:37 |
| *** udesale has joined #oooq | 09:38 | |
| hubbot1 | FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/rocky: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7 (2 more messages) | 09:46 |
| *** arxcruz|ptg is now known as arxcruz | 10:11 | |
| *** kopecmartin has quit IRC | 10:15 | |
| *** dtantsur|afk is now known as dtantsur | 10:18 | |
| *** quiquell|rover is now known as quique|rover|afk | 10:19 | |
| quique|rover|afk | jrist: unit test ar passing locally... | 10:25 |
| ssbarnea | quique|rover|afk: regarding UID, bdobrelya found another way to avoid it, so if we merge his patch, I would abandon mine. https://review.openstack.org/602015 | 10:27 |
| quique|rover|afk | ssbarnea: Cool thanks | 10:28 |
| ssbarnea | at least his patch is easier to review and lower risks | 10:28 |
| quique|rover|afk | ssbarnea: LÃlet's wait for results, I diit out of nowhere | 10:29 |
| *** jfrancoa has quit IRC | 10:35 | |
| *** jfrancoa has joined #oooq | 10:36 | |
| *** quique|rover|afk is now known as quiquell|rover | 10:49 | |
| *** panda has joined #oooq | 10:58 | |
| quiquell|rover | chkumar|ruck: Can force run featureset019 at RDO ' | 11:07 |
| quiquell|rover | ? | 11:07 |
| quiquell|rover | chkumar|ruck: You have do something similar before for rocky and irnonic issue | 11:07 |
| *** kopecmartin has joined #oooq | 11:15 | |
| chkumar|ruck | quiquell|rover: let me try that | 11:18 |
| chkumar|ruck | quiquell|rover: fs019 is running here https://review.rdoproject.org/zuul/stream.html?uuid=506957ea0b7a44f6a63a17387625b129&logfile=console.log | 11:18 |
| chkumar|ruck | quiquell|rover *fs001 | 11:19 |
| chkumar|ruck | quiquell|rover: once it finishes I will trigger fs019 | 11:19 |
| *** kopecmartin has quit IRC | 11:19 | |
| quiquell|rover | chkumar|ruck: ack | 11:19 |
| chkumar|ruck | quiquell|rover: fs019 for rocky? | 11:23 |
| quiquell|rover | fs019 for master, to check that mirror is ok now | 11:24 |
| chkumar|ruck | quiquell|rover: ack | 11:24 |
| quiquell|rover | chkumar|ruck: thanks so much | 11:24 |
| quiquell|rover | chkumar|ruck: update CIX new / triage with the telemetry issue | 11:32 |
| chkumar|ruck | quiquell|rover: yup sure | 11:32 |
| chkumar|ruck | quiquell|rover: what to do two ironic related bugs | 11:32 |
| chkumar|ruck | quiquell|rover: fs01 rocky passed https://logs.rdoproject.org/43/13943/29/check/legacy-periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky/506957e/ | 11:33 |
| quiquell|rover | chkumar|ruck: Puff don't know, let's just say that they are related to RDO infra issues | 11:34 |
| quiquell|rover | chkumar|ruck: And that we are so lucky :-/ | 11:34 |
| quiquell|rover | chkumar|ruck: They will say what to do with those | 11:34 |
| *** kopecmartin has joined #oooq | 11:34 | |
| chkumar|ruck | quiquell|rover: do we have any upstream review to link with fso19? | 11:34 |
| chkumar|ruck | which is not yet merged? | 11:34 |
| quiquell|rover | chkumar|ruck: https://review.openstack.org/603054 | 11:36 |
| chkumar|ruck | quiquell|rover: https://review.rdoproject.org/r/#/c/13943/ added here | 11:39 |
| chkumar|ruck | let'see how it goes | 11:39 |
| quiquell|rover | chkumar|ruck: yep, can you add this to the launchpad or CIX card ? | 11:40 |
| arxcruz | ykarel: chkumar|ruck https://review.rdoproject.org/r/#/c/16216/ | 11:42 |
| arxcruz | \o/ | 11:42 |
| quiquell|rover | chkumar|ruck: I not able to go to CIX can you cover it ? | 11:42 |
| chkumar|ruck | quiquell|rover: yup | 11:42 |
| quiquell|rover | chkumar|ruck: Will try to join as soon I finish here | 11:42 |
| chkumar|ruck | quiquell|rover: do we have bz for mirror issue? | 11:42 |
| quiquell|rover | Yep | 11:43 |
| quiquell|rover | Is the old one https://bugs.launchpad.net/tripleo/+bug/1792296 | 11:43 |
| openstack | Launchpad bug 1792296 in tripleo "Overcloude deploy error:Timed out waiting for messages from Execution" [Critical,In progress] - Assigned to Quique Llorente (quiquell) | 11:43 |
| chkumar|ruck | quiquell|rover: added rdo review here | 11:44 |
| quiquell|rover | chkumar|ruck: I think I have fill in all the stuff neded at the CIX card | 11:45 |
| quiquell|rover | chkumar|ruck: Check them to see if we have averything I have like 15 minutes or so | 11:45 |
| hubbot1 | FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/rocky: tripleo-ci- (2 more messages) | 11:46 |
| chkumar|ruck | quiquell|rover: what is the duplicate card of this one https://trello.com/c/lxhDNvX4/744-cixlp1792296tripleociproa-overcloude-deploy-errortimed-out-waiting-for-messages-from-execution ? | 11:46 |
| quiquell|rover | chkumar|ruck: https://trello.com/c/zKpzUWCY/740-cixlptripleociproa-overcloude-deploy-errortimed-out-waiting-for-messages-from-execution | 11:47 |
| *** zul has joined #oooq | 11:48 | |
| *** kopecmartin has quit IRC | 11:51 | |
| quiquell|rover | ssbarnea: you were not already back ? | 11:51 |
| quiquell|rover | ssbarnea: ups old chat history, sorry :-) | 11:52 |
| *** rfolco has joined #oooq | 12:00 | |
| *** panda has quit IRC | 12:00 | |
| weshay | ssbarnea, welcome back | 12:03 |
| ssbarnea | weshay: thanks! | 12:04 |
| quiquell|rover | chkumar|ruck: I am able to make it to CIX | 12:04 |
| *** trown|outtypewww is now known as trown | 12:06 | |
| weshay | chkumar|ruck, question to you.. https://trello.com/c/OpWmOkAZ/742-cixlp1792862tripleociproa-master-telemetry-tempest-integration-tests-failed-giving-unable-to-complete-operation-on-subnet-b13c76 | 12:09 |
| weshay | chkumar|ruck, I don't know anything about the bug/fix but wondering why the issue was not caught by devstack... maybe the devstack tempest was updated? | 12:10 |
| chkumar|ruck | weshay: it is caught in POI | 12:10 |
| weshay | ah k | 12:11 |
| chkumar|ruck | weshay: quiquell|rover https://review.rdoproject.org/r/13943 | 12:12 |
| chkumar|ruck | here we are testing the failed jobs | 12:12 |
| ssbarnea | is sagi still away this week? | 12:12 |
| *** skramaja has quit IRC | 12:13 | |
| weshay | chkumar|ruck, quiquell|rover queens promoted.. that's great :) | 12:18 |
| quiquell|rover | weshay: We were watching it with pop corns | 12:21 |
| chkumar|ruck | weshay: https://review.rdoproject.org/etherpad/p/ruckrover-sprint19 | 12:28 |
| weshay | chkumar|ruck, quiquell|rover great job.. help real quick find the card that is associated w/ fs019 failing in rocky | 12:30 |
| chkumar|ruck | weshay: quiquell|rover fs019 for rocky is passing https://review.rdoproject.org/zuul/builds.html?job_name=legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-queens | 12:31 |
| chkumar|ruck | sorry | 12:31 |
| quiquell|rover | weshay: This week was a pain in the ass | 12:31 |
| chkumar|ruck | sorry wrong info | 12:32 |
| weshay | quiquell|rover, chkumar|ruck ya.. I know .. thank you for staying on guard :) | 12:32 |
| quiquell|rover | weshay: Can we skipp scalation ? | 12:32 |
| weshay | quiquell|rover, chkumar|ruck ya.. I'll cover you guys | 12:32 |
| quiquell|rover | cool thanks | 12:33 |
| *** rlandy has joined #oooq | 12:33 | |
| *** quiquell|rover is now known as quique|rover|lch | 12:33 | |
| * chkumar|ruck headed home | 12:34 | |
| quique|rover|lch | ssbarnea: This rings any bell ? https://ci.centos.org/job/tripleo-quickstart-gate-master-delorean-quick-basic/6964/consoleFull | 12:34 |
| weshay | quique|rover|lch, you posted a queens job | 12:35 |
| weshay | for rocky :P | 12:35 |
| quique|rover|lch | weshay: where ? | 12:36 |
| weshay | <chkumar|ruck> weshay: quiquell|rover fs019 for rocky is passing https://review.rdoproject.org/zuul/builds.html?job_name=legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-queens | 12:36 |
| arxcruz | weshay: fs019 is scenario004 right ? | 12:36 |
| weshay | aye | 12:37 |
| arxcruz | weshay: chkumar|ruck i talked with gouthamr last week, he will send the patches to fix it | 12:37 |
| arxcruz | if it's related to manila | 12:37 |
| quique|rover|lch | chkumar|ruck: Do you have around the review you did to force run rocky job ? | 12:37 |
| ssbarnea | quique|rover|lch: chkumar|ruck : https://review.openstack.org/#/c/596841/ -- maybe a +W on it? | 12:38 |
| weshay | quique|rover|lch, chkumar|ruck so fs019 / scen004 https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-rocky/3f03ed5/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 12:43 |
| * weshay doesn't see the error yet | 12:44 | |
| weshay | it's a puppet error | 12:44 |
| weshay | just a firewall timeout? https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-rocky/3f03ed5/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-09-17_09_03_21 | 12:46 |
| weshay | The WebSocket timed out before the Workflow completed. | 12:46 |
| *** udesale has quit IRC | 12:54 | |
| *** kopecmartin has joined #oooq | 12:55 | |
| *** kopecmartin has quit IRC | 12:55 | |
| *** kopecmartin has joined #oooq | 12:55 | |
| *** kopecmartin has quit IRC | 12:56 | |
| *** kopecmartin has joined #oooq | 12:56 | |
| marios | scrum? | 13:01 |
| weshay | let's do it! | 13:01 |
| weshay | ssbarnea, rlandy rfolco mtg | 13:02 |
| *** toure|gone is now known as toure | 13:03 | |
| marios | ssbarnea: congrats ! | 13:03 |
| *** quique|rover|lch is now known as quiquell|rover | 13:05 | |
| *** thrash|g0ne is now known as thrash | 13:12 | |
| quiquell|rover | ssbarnea: What happend ? | 13:13 |
| quiquell|rover | marios: ? | 13:13 |
| ssbarnea | quiquell|rover: I got married last week. | 13:15 |
| *** panda has joined #oooq | 13:16 | |
| quiquell|rover | ssbarnea: Ahh congratulations !!! | 13:16 |
| ykarel | ssbarnea, Congrats!! | 13:17 |
| weshay | panda, rlandy https://review.openstack.org/#/c/589448/33 | 13:22 |
| *** vinaykns has joined #oooq | 13:23 | |
| arxcruz | ssbarnea: congrats!!! | 13:28 |
| rfolco | ssbarnea, best wishes, congrats! | 13:31 |
| tosky | oh, congratulations! | 13:32 |
| quiquell|rover | panda, rlandy: We need to put this at te-broker https://review.openstack.org/#/c/588488 | 13:34 |
| quiquell|rover | panda, rlandy: We have miss it after the last reconstruction | 13:34 |
| rlandy | quiquell|rover: I'm with sagi on this .. "did you test it live on te-broker? worth to do it before we approve it" | 13:35 |
| rlandy | hard to approve unless we know it works | 13:35 |
| rlandy | we would need a live test :( | 13:36 |
| rlandy | the way te-broker works, | 13:36 |
| rlandy | we would have to edit on the instance itself | 13:36 |
| quiquell|rover | rlandy: So I apply the patch directly at te-broker ? | 13:36 |
| quiquell|rover | and try it ? | 13:36 |
| quiquell|rover | Or I start my own te-broker ? | 13:36 |
| rlandy | quiquell|rover: well, usually we would make the edit in /opt/stack on the actual te-broker instance | 13:37 |
| rlandy | and see if it works | 13:37 |
| rlandy | revert of it does not | 13:37 |
| quiquell|rover | rlandy: will do | 13:37 |
| rlandy | te-broker gets refreshed from master every nigh | 13:37 |
| rlandy | night | 13:37 |
| quiquell|rover | rlandy: The review looks good ? I have add the UTF part | 13:37 |
| rlandy | quiquell|rover: afaict | 13:37 |
| rlandy | panda would be good to ask | 13:38 |
| rlandy | he write the bmc log code | 13:38 |
| rlandy | if you can live test it, I can +2 it | 13:38 |
| quiquell|rover | rlandy: is panda around ? | 13:38 |
| rlandy | quiquell|rover: irc says he is | 13:38 |
| quiquell|rover | will try to apply the change | 13:39 |
| rlandy | but you can't always believe what you see on line these days :) | 13:39 |
| quiquell|rover | rlandy: I like this way :-) realtime status is a sh... | 13:39 |
| panda | I'm still coping with the effect of constant air conditioning at PTG | 13:39 |
| ssbarnea | chkumar|ruck: I seen IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem__' (HTTP 500) on http://logs.rdoproject.org/15/602015/1/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/f45dce8/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz - is this known issue? | 13:40 |
| quiquell|rover | panda: Are you throwing blood from your ears ? | 13:40 |
| quiquell|rover | ssbarnea: Yep, RDO infra issue | 13:40 |
| weshay | quiquell|rover, panda http://38.145.33.166/console-logs/ | 13:40 |
| quiquell|rover | panda: Going to apply your te-broker patch https://review.openstack.org/#/c/588488 to te-broker directly | 13:41 |
| quiquell|rover | weshay: we are talking about it | 13:41 |
| weshay | has that code been removed or you are waiting to redeploy it based on the review | 13:41 |
| weshay | quiquell|rover, ya | 13:41 |
| panda | quiquell|rover: weshay the code should already be there | 13:41 |
| quiquell|rover | weshay: te-broker was re-constructed last week, so we lose that | 13:41 |
| weshay | quiquell|rover, ya | 13:41 |
| quiquell|rover | panda: So I apply to live te-broker to test it ? | 13:42 |
| rlandy | why was that code not in the automatic re-apply? | 13:42 |
| rlandy | panda: ^^ th elive apply was my suggestion | 13:42 |
| quiquell|rover | rlandy: Does not have to be merged ? | 13:42 |
| rlandy | quiquell|rover: it would have to be merge to re-apply every night with the refresh | 13:42 |
| rlandy | to just test it, you can manually apply | 13:43 |
| rlandy | and see what explodes | 13:43 |
| quiquell|rover | rlandy: so how did it work before, because the review was not merged by the logs were there | 13:43 |
| quiquell|rover | s/by/but/ | 13:43 |
| quiquell|rover | ok liked to work at redhat, going to break te-broker | 13:43 |
| rlandy | quiquell|rover: idk- that is what I was asking above | 13:44 |
| panda | I applied the code live on te broker some days ago, but now that te-broken has been provisioned with our scripts, the tripleo-ci repo gets refreshed every 10 minutes, so to test any local modification we need to suspend the automatic refresh | 13:44 |
| rlandy | how was that not in the rebuilt te-broker to begin with? | 13:44 |
| panda | quiquell|rover: ^ | 13:44 |
| rlandy | of we rebuild from our code base | 13:44 |
| quiquell|rover | panda: How do I suspend automatic refresh, the ansible crontab line ? | 13:45 |
| panda | quiquell|rover: root crontab | 13:45 |
| panda | */5 * * * * flock -n -E 2 /var/lock/ansible-pull /usr/local/bin/ansible_pull.sh | 13:45 |
| quiquell|rover | panda: ack | 13:46 |
| hubbot1 | FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/rocky: tripleo-ci- (2 more messages) | 13:46 |
| panda | mmmhhh | 13:46 |
| panda | wait | 13:46 |
| quiquell|rover | done | 13:46 |
| quiquell|rover | :-// | 13:46 |
| panda | yep, ok | 13:47 |
| panda | that's it | 13:47 |
| quiquell|rover | panda: also the dashboard-ci part was missing https://review.rdoproject.org/r/#/c/16284/ | 13:48 |
| weshay | panda, rlandy you guys looking at that patch? | 13:51 |
| weshay | https://logs.rdoproject.org/48/589448/33/openstack-check/legacy-tripleo-ci-centos-7-container-to-container-upgrades-master/459ae33/job-output.txt.gz#_2018-09-13_14_34_26_330777 | 13:51 |
| * weshay wondering why 2018-09-13 14:34:26.330777 | primary | TASK [undercloud-setup : Run the TripleO-CI VXLAN networking script] *********** | 13:51 | |
| weshay | is running on an undercloud upgrade job | 13:51 |
| panda | weshay: is some of the task running, or they are just all skipped ? | 13:52 |
| weshay | panda, it's running | 13:53 |
| weshay | https://logs.rdoproject.org/48/589448/33/openstack-check/legacy-tripleo-ci-centos-7-container-to-container-upgrades-master/459ae33/logs/undercloud/home/zuul/vxlan_networking.sh.log.txt.gz | 13:53 |
| rlandy | yes | 13:53 |
| rlandy | that vxlan networking taks has been broke forever | 13:53 |
| quiquell|rover | Humm rfolco you are looking something regarding this ? | 13:53 |
| quiquell|rover | Wait imean jfrancoa | 13:53 |
| quiquell|rover | jfrancoa is debugging vxlan networking issues with upgrades at RDO | 13:54 |
| rlandy | I remember asking ssbarnea about this error a couple weeks ago | 13:56 |
| rlandy | there was a related bug I think | 13:56 |
| quiquell|rover | rlandy: ykarel say something about version of ovs | 13:56 |
| rlandy | quiquell|rover: truth is, we should not need that at all with reparenting | 13:56 |
| jfrancoa | quiquell|rover: weshay hey, yes..I'm trying to find the source of the problem and I'm somehow close. the vxlan_networking fails constantly in the upgrades jobs because we're not setting up the repos for the subnodes, only in the undercloud. So when it tries to install ovs in the subnode-1, it's a really old version and the script breaks | 13:57 |
| weshay | jfrancoa, rlandy panda is that job even valid? | 13:57 |
| ykarel | quiquell|rover, rlandy afair repos were not configured on overcloud nodes, so ovs was installed from centos base, | 13:57 |
| weshay | that is a multinode container upgrade job | 13:57 |
| rlandy | the script doesn't pass on the undercloud either | 13:57 |
| weshay | jfrancoa, which is duped by fs051 isn't it? | 13:58 |
| jfrancoa | weshay: well, it should..it's a multinode containers upgrade with several services enabled, but it has been failing for very long | 13:58 |
| *** agopi is now known as agopi|afk|ncsu|c | 13:58 | |
| *** agopi|afk|ncsu|c is now known as agopi|afk|car-fa | 13:58 | |
| *** agopi|afk|car-fa is now known as agopi|afk | 13:58 | |
| rlandy | https://logs.rdoproject.org/48/589448/33/openstack-check/legacy-tripleo-ci-centos-7-container-to-container-upgrades-master/459ae33/logs/undercloud/home/zuul/vxlan_networking.sh.log.txt.gz#_2018-09-13_14_34_30 | 13:58 |
| jfrancoa | weshay: fs051 is keystone only, which is not very realistic. it's good to test the workflow, but we were having this other one running in RDO to test the upgrade enabling other services | 13:58 |
| weshay | jfrancoa, we have a new plan of attack w/ regards to update/upgrade testing | 13:59 |
| weshay | marios, had the blueprint approved | 13:59 |
| jfrancoa | rlandy: exactly, it you see any other job which runs it (and it's not a mixed_upgrade job), it would install 2.0.9 version | 13:59 |
| weshay | jfrancoa, my thought is to nuke that job | 13:59 |
| rlandy | jfrancoa: I'd like to nuke this altogether | 14:00 |
| rlandy | not the job, the vxlan networking | 14:00 |
| jfrancoa | weshay: holser_ mentioned something about that, it seems there were some conversations about updates/upgrades CI jobs in the PTG | 14:00 |
| rlandy | it's only a legacy thing | 14:00 |
| weshay | jfrancoa, ya.. the vxlan script is legacy.. | 14:00 |
| holser_ | jfrancoa ... wait for chem | 14:00 |
| chkumar|ruck | quiquell|rover: https://review.rdoproject.org/r/#/c/13943/ | 14:00 |
| weshay | that job is not providing value.. ----> NUKE | 14:01 |
| chkumar|ruck | quiquell|rover: https://review.rdoproject.org/zuul/stream.html?uuid=7dee50d4a27c41de81350d1dc14df9d0&logfile=console.log | 14:01 |
| jfrancoa | weshay: anyway, the same is happening in the ffu one, which does provide value | 14:01 |
| weshay | rlandy, rfolco do we have any tht patches on top of https://review.openstack.org/#/c/589448/33 to test? | 14:02 |
| weshay | jfrancoa, point me at that job please | 14:02 |
| quiquell|rover | chkumar|ruck: Have already pass introspection, so we are good | 14:02 |
| jfrancoa | weshay: rlandy: so, we can get rid entirely of this vxlan_networking thing? I tried disabling it in the featurset, but it didnt work (probably multinode-rdo took precendence for the variables) https://review.openstack.org/#/c/600816/ | 14:02 |
| chkumar|ruck | weshay: fs019 rocky I am already testing it there in rdo noop jobs | 14:02 |
| weshay | chkumar|ruck, perfect.. thanks | 14:02 |
| jfrancoa | weshay: https://logs.rdoproject.org/16/563616/12/openstack-check/legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset036-oc-ffu-queens/811195e/ | 14:03 |
| quiquell|rover | We are not launching RDO CI at rocky THT :-/ | 14:03 |
| jfrancoa | weshay: it's a little bit old, but I guess it's still happening, if it's happening in the other RDO upgrades jobs | 14:03 |
| rlandy | weshay: not that I know of - I have tht patches on top of my work to test - can probably reuse | 14:04 |
| quiquell|rover | panda: Do I have to do anything else ? I still don't see any logs here http://38.145.33.166/console-logs/ | 14:04 |
| rlandy | jfrancoa: not quite yet | 14:04 |
| weshay | jfrancoa, that job last ran on 9/4 | 14:04 |
| weshay | jfrancoa, it last passed on 7/11 | 14:04 |
| panda | quiquell|rover: CONSOLE_LOG_PATH is missing | 14:04 |
| chkumar|ruck | ssbarnea: this is the bug https://bugs.launchpad.net/tripleo/+bug/1792872 | 14:05 |
| openstack | Launchpad bug 1792872 in tripleo "[queens] overcloud prepare image failed by giving IronicAction.node.set_provision_state failed: 'NoneType' object has no attribute '__getitem_" [Critical,Triaged] - Assigned to Quique Llorente (quiquell) | 14:05 |
| rlandy | jfrancoa: we are desperately trying to get rid of legacy. the non-legacy stuff uses a different role to set bridge | 14:05 |
| marios | weshay: sorry was on the phone checking | 14:05 |
| rlandy | for the moment, we still need it | 14:05 |
| panda | quiquell|rover: then you'll have to wait for a test env to be destroyed | 14:05 |
| marios | weshay: which blueprint? | 14:05 |
| weshay | marios, standalone upgrade | 14:05 |
| chkumar|ruck | arxcruz: manila issue is handled , only pike and ocata is blocked due to undercloud upgrade job failure | 14:05 |
| marios | weshay: didn't know we had one ;) | 14:05 |
| chkumar|ruck | also we need to work on removing ocata jobs | 14:06 |
| marios | weshay: i just checked the spec i need to update it https://review.openstack.org/#/c/579854/ some last comments and remove the WIP | 14:06 |
| weshay | marios, jfrancoa imho.. we kill 3rd party upgrade jobs, focus on blueprint work | 14:06 |
| marios | we can land it this week weshay | 14:06 |
| weshay | +1 | 14:06 |
| weshay | rlandy, panda jfrancoa for the context of the patch.. I'm ignoring the non-voting upgrade job | 14:07 |
| rlandy | weshay:ack - I left a similar comment on the patch | 14:07 |
| rlandy | when I reviewed it before | 14:07 |
| jfrancoa | weshay: you mean marios' blueprint? and the idea would be enabling these legacy jobs in the standalone deployment? In my opinion, it would be helpful having a FFU job upstream, as well as a closer-to-reality upgrades job | 14:08 |
| quiquell|rover | panda: /var/www/html/tebroker/console-logs/ this is the path ? | 14:08 |
| panda | quiquell|rover: yes | 14:08 |
| weshay | jfrancoa, /me firing up blue jeans for a few minutes to discuss upgrades w/ jfrancoa | 14:08 |
| weshay | anyone is welcome | 14:08 |
| weshay | https://bluejeans.com/u/whayutin/ | 14:09 |
| weshay | jfrancoa, have a sec? | 14:09 |
| rlandy | quiquell|rover: weshay: for https://review.openstack.org/#/c/589448/33 - the reviews listed in the comment, can they be abandoned? | 14:09 |
| jfrancoa | weshay: joining | 14:09 |
| rlandy | it says this review includes them | 14:09 |
| weshay | rlandy, the reviews? | 14:09 |
| weshay | in the commit msg? | 14:09 |
| rlandy | This is include the following reviews from [1]: - https://review.openstack.org/#/c/585714 - https://review.openstack.org/#/c/587340 - https://review.openstack.org/#/c/587360 - https://review.openstack.org/#/c/587344 | 14:09 |
| weshay | please update that.. rlandy :) | 14:09 |
| weshay | thanks | 14:09 |
| rlandy | weshay: well, I am not sure what the update should be - which is why I am asking | 14:10 |
| rlandy | do we abandon those revuews | 14:10 |
| rlandy | reviews | 14:10 |
| rlandy | and remove them from the comment? | 14:10 |
| rlandy | or actually consider them? | 14:10 |
| quiquell|rover | rlandy: I think so, we are not going to go back to them, and I put back the comments on them in the all-in-one | 14:11 |
| quiquell|rover | rlandy: drop them | 14:11 |
| rlandy | quiquell|rover: fine, I will abandon them and update the commit message | 14:12 |
| rlandy | weshay: ^^ | 14:12 |
| quiquell|rover | rlandy: btw, we are forcing vxlan at common.yaml, don't know if this is going to interfiere with future RDO jobs running new workflow | 14:12 |
| rlandy | quiquell|rover: what do you mean by forcing? | 14:13 |
| rlandy | the new workflow set that to false | 14:13 |
| rlandy | ie;running our role | 14:13 |
| quiquell|rover | rlandy: and also multinode-rdocloud.yaml is putting them to true, so don't know wich one take preference | 14:14 |
| rlandy | quiquell|rover: right now, in rdo cloud,we still run this vxlan networking role | 14:15 |
| rlandy | when we reparent we should not - and rely on the one from zuul-jobs I think | 14:15 |
| rlandy | we will see that when we reparent | 14:15 |
| quiquell|rover | rlandy: ok, then we will ahve to remove those false from anywhere | 14:16 |
| quiquell|rover | panda: console-logs working now | 14:19 |
| panda | quiquell|rover: ok, you want to patch the review to include CONSOLE_LOG_PATH ? | 14:19 |
| quiquell|rover | panda: Have already add it to the review | 14:19 |
| quiquell|rover | panda: I think the review have all the stuff now, take a look | 14:20 |
| quiquell|rover | ok drop now | 14:21 |
| rlandy | quiquell|rover: can we vote on ^^ yet? | 14:21 |
| rlandy | te-broker tested? | 14:21 |
| *** quiquell|rover is now known as quique|rover|off | 14:22 | |
| quique|rover|off | rlandy: working fine | 14:22 |
| quique|rover|off | ykarel: is happy now | 14:22 |
| quique|rover|off | http://38.145.33.166/console-logs/ | 14:22 |
| ykarel | :) | 14:22 |
| rlandy | with live patch? | 14:22 |
| quique|rover|off | rlandy: yep | 14:22 |
| rlandy | ok - voting | 14:22 |
| rlandy | http://logs.openstack.org/45/602445/2/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/ba720d5/job-output.txt.gz#_2018-09-17_13_45_42_733863 | 14:27 |
| rlandy | contaierized_upgrades is going to fail removal job | 14:27 |
| *** dtrainor has joined #oooq | 14:30 | |
| rlandy | http://logs.openstack.org/45/602445/2/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/ba720d5/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz#_2018-09-17_13_45_40 | 14:31 |
| rlandy | ^^ worth rechecking? | 14:31 |
| rlandy | jfrancoa: ^^? | 14:33 |
| *** ykarel is now known as ykarel|away | 14:33 | |
| rlandy | tripleo-ci-centos-7-containerized-undercloud-upgradesSUCCESS in 1h 43m 18s - passed in last run | 14:34 |
| weshay | rlandy, panda have a master and pike tht test patch on 589448 | 14:51 |
| weshay | visible in https://review.openstack.org/#/q/topic:toci_jobtype+(status:open+OR+status:merged) | 14:51 |
| weshay | panda, please vote before the end of your day | 14:51 |
| *** ykarel|away has quit IRC | 14:53 | |
| chkumar|ruck | is someoone working on this https://ci.centos.org/job/tripleo-quickstart-gate-master-delorean-quick-basic/7017/ to make it passing? | 14:55 |
| weshay | chkumar|ruck, https://ci.centos.org/job/tripleo-quickstart-gate-master-delorean-quick-basic/ | 14:56 |
| weshay | chkumar|ruck, pip errors? https://ci.centos.org/job/tripleo-quickstart-gate-master-delorean-quick-basic/7049/console | 14:57 |
| weshay | or clone | 14:57 |
| chkumar|ruck | weshay: chekcing | 14:59 |
| arxcruz | weshay: chkumar|ruck will we have scrum today ? | 15:01 |
| chkumar|ruck | arxcruz: yes, | 15:02 |
| arxcruz | k | 15:02 |
| weshay | arxcruz, chkumar|ruck ya | 15:02 |
| weshay | panda, opt/stack/new? | 15:03 |
| weshay | panda, ?? | 15:04 |
| panda | weshay: sorry, what's the context ? | 15:05 |
| weshay | panda, your patch https://review.openstack.org/#/c/602673/ | 15:06 |
| chkumar|ruck | weshay: will check tomorrow | 15:06 |
| weshay | chkumar|ruck, k | 15:06 |
| weshay | panda, where is that file logged? | 15:06 |
| weshay | zuul-variables.yaml | 15:07 |
| panda | dumpend in tripleo_root | 15:07 |
| weshay | panda, show me the log | 15:07 |
| panda | weshay: the file is not logged. And yes probably tripleo_root is not the right location to put that file | 15:08 |
| panda | weshay: I'll move it to local_working_dir | 15:09 |
| weshay | panda, either way.. just log it | 15:11 |
| weshay | but agree.. local_working_dir | 15:11 |
| *** thrash has quit IRC | 15:20 | |
| rlandy | ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500" | 15:28 |
| rlandy | ugh this again - in pike :( | 15:28 |
| rlandy | weshay: tripleo-ci-centos-7-containerized-undercloud-upgrades killed us in https://review.openstack.org/#/c/602445/ - is there a way to recheck while it's busy with the current run? | 15:30 |
| rlandy | my recheck did not take | 15:30 |
| arxcruz | chkumar|ruck: weshay: which bj ? | 15:30 |
| chkumar|ruck | arxcruz: https://bluejeans.com/7050859455/https://bluejeans.com/7050859455/ | 15:30 |
| chkumar|ruck | https://bluejeans.com/7050859455/ | 15:30 |
| chkumar|ruck | kopecmartin: weshay ^^ | 15:31 |
| weshay | joining | 15:31 |
| weshay | sory | 15:31 |
| chkumar|ruck | weshay: fs019 rokcy faild https://logs.rdoproject.org/43/13943/31/check/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-rocky/7dee50d/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-09-17_14_15_31 | 15:32 |
| hubbot1 | FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/rocky: tripleo-ci- (2 more messages) | 15:46 |
| bogdando | weshay, ssbarnea: PTAL https://review.openstack.org/#/c/576746/ | 15:48 |
| bogdando | weshay: one more https://review.openstack.org/#/c/592996/ | 15:49 |
| bogdando | aaand https://review.openstack.org/#/c/595265/ | 15:49 |
| bogdando | there is also https://review.openstack.org/#/c/595566/ which boosts provisioning by 100% or so | 15:50 |
| bogdando | not for CI tho, just for users | 15:50 |
| *** vinaykns has quit IRC | 15:55 | |
| *** saneax has quit IRC | 15:56 | |
| *** ykarel has joined #oooq | 15:59 | |
| weshay | rlandy, ok.. caught up w/ you.. running recheck again should rekick it from scratch | 16:11 |
| weshay | rlandy, I just did it.. watching | 16:11 |
| rlandy | no dice when I did it | 16:12 |
| weshay | hrm.. k | 16:12 |
| rlandy | but maybe it likes you better | 16:12 |
| *** marios is now known as marios|hometime | 16:12 | |
| rlandy | weshay: tbh, the actual failure worries me more | 16:12 |
| rlandy | it passed on the last run | 16:12 |
| rlandy | so I doubt it's related | 16:12 |
| *** marios|hometime is now known as marios | 16:13 | |
| weshay | rlandy, hrm.. I think tripleo-ci-centos-7-containerized-undercloud-upgrades may need to go nv | 16:13 |
| weshay | quique|rover|off, chkumar|ruck fyi ^ | 16:13 |
| weshay | rlandy, oh.. wait | 16:13 |
| weshay | we have a patch on that | 16:13 |
| weshay | sec | 16:13 |
| rlandy | oh great | 16:13 |
| weshay | how r the gate jobs at 97% | 16:13 |
| *** holser_ has quit IRC | 16:14 | |
| rlandy | statistics ?!? | 16:15 |
| rlandy | can you in patch - can I depend on it? | 16:16 |
| weshay | rlandy, https://review.openstack.org/#/c/602550/ | 16:16 |
| weshay | https://review.openstack.org/#/c/603026/ | 16:16 |
| weshay | and finally https://review.openstack.org/#/c/590774/ | 16:16 |
| weshay | rlandy, in reverse order | 16:16 |
| rlandy | hmmm ... well not sure it's worth depending on those pathces | 16:21 |
| rlandy | they may not merge | 16:21 |
| weshay | ya | 16:24 |
| rlandy | but if I don;t ... the test will fail | 16:27 |
| rlandy | worth the recheck? | 16:27 |
| rlandy | weshay: ack - ok I think we need to go non-voting here until these patches merge. I can put in the change for your review | 16:28 |
| weshay | rlandy, you can dep on the tht change | 16:28 |
| weshay | that will merge | 16:28 |
| rlandy | is that enough? | 16:28 |
| rlandy | besides other jobs will fail here | 16:29 |
| weshay | rlandy, the job passes there | 16:29 |
| rlandy | I see | 16:29 |
| weshay | rlandy, worth a shot imho | 16:29 |
| * rlandy see how close those gates are to passing | 16:30 | |
| rlandy | hmmm ... indeterminate | 16:30 |
| *** jfrancoa has quit IRC | 16:30 | |
| rlandy | ok - let's try the depends | 16:30 |
| rlandy | here we go again | 16:32 |
| weshay | bogdando, when did vars_prompt become a thing | 16:32 |
| weshay | pretty cool | 16:32 |
| bogdando | indeed | 16:32 |
| weshay | panda, rlandy check this out https://docs.ansible.com/ansible/2.6/user_guide/playbooks_prompts.html | 16:33 |
| bogdando | weshay: did you manage to deploy f28 with quickstart libvirt? | 16:33 |
| bogdando | I think I did | 16:33 |
| bogdando | from the cloud image | 16:33 |
| weshay | bogdando, I've gotten to the python-tripleoclient | 16:33 |
| bogdando | with https://review.openstack.org/#/c/603069/ | 16:34 |
| weshay | bogdando, which cloud image did you use? | 16:34 |
| bogdando | weshay: https://github.com/bogdando/oooq-warp/blob/3e6dc629df0d3e6a3f99a164d4e836abe2b1e6a1/vars/fedora28.yaml#L10 | 16:35 |
| rlandy | kind of cool - not sure we ever ask the users for their input though | 16:35 |
| weshay | bogdando, I wish there was idempotent command for that | 16:35 |
| rlandy | reproducer maybe | 16:35 |
| weshay | rlandy, reproduceer | 16:35 |
| weshay | maybe | 16:35 |
| weshay | :) | 16:35 |
| bogdando | weshay: I think the whole thing is wrapped with idempotency | 16:35 |
| rlandy | lol - agreed | 16:35 |
| weshay | not sure if that is > than bash | 16:35 |
| weshay | bogdando, bah | 16:35 |
| bogdando | like then you run provision based on tags/skip | 16:35 |
| weshay | bogdando, on my list today | 16:36 |
| bogdando | or images expiring rules | 16:36 |
| bogdando | IIUC | 16:36 |
| bogdando | so basically that section never runs on a mutated qcow2 | 16:36 |
| bogdando | it's either fully skipped, or always applies from the scratch | 16:36 |
| bogdando | (IIUC :D) | 16:37 |
| bogdando | so it's fully atomic with other virt-customize commands around | 16:37 |
| weshay | bogdando, is there some doc on https://review.openstack.org/#/c/595566/3/roles/modify-image/tasks/libguestfs.yml | 16:38 |
| weshay | force_tcg | 16:38 |
| * weshay is googling | 16:38 | |
| weshay | got it http://libguestfs.org/guestfs.3.html#force_tcg | 16:38 |
| bogdando | weshay: that thing WFM | 16:39 |
| bogdando | I put some benchmarking results into comments | 16:39 |
| bogdando | see 23. Aug 14:29 | 16:39 |
| bogdando | oops, not really, it's 15:01 weshay | 16:40 |
| *** bogdando has quit IRC | 16:48 | |
| *** ykarel is now known as ykarel|away | 16:50 | |
| *** kopecmartin has quit IRC | 16:51 | |
| *** dtantsur is now known as dtantsur|afk | 16:54 | |
| weshay | rlandy, so https://trello.com/c/TVsZ3Ut6/877-clean-up-rdo-sf-legacy-code-after-zuulv3-migration is up 2 date and accurate ? | 17:10 |
| *** vinaykns has joined #oooq | 17:10 | |
| weshay | panda, this needs QE https://trello.com/c/JN4UIjWv/854-replace-zuulchanges-handling-in-build-test-packages-role | 17:10 |
| *** trown is now known as trown|lunch | 17:18 | |
| rlandy | weshay: well, depends if we consider panda's patch to be part of it | 17:25 |
| rlandy | the lagacy zuul vars | 17:25 |
| rlandy | I see it has its own card though | 17:25 |
| *** dsneddon has quit IRC | 17:30 | |
| *** dsneddon has joined #oooq | 17:30 | |
| weshay | playbooks/tripleo-ci/vars/common.yaml | 17:34 |
| weshay | Line 23: | 17:34 |
| weshay | toci_gate_test.sh.j2 L130 uses nodes_file | 17:34 |
| weshay | while | 17:34 |
| weshay | toci_quickstart.sh.j2 L125 uses nodes_args | 17:34 |
| weshay | nodes_args includes "--extra-vars" to make the logic around string manipulation easier and keep consistent with the rest of extra-vars... | 17:34 |
| weshay | https://review.openstack.org/#/c/589448 | 17:35 |
| weshay | rlandy, fyi https://review.openstack.org/#/c/603187/1/zuul.d/undercloud-jobs.yaml | 17:36 |
| rlandy | yes thank you | 17:36 |
| *** tbarron_ is now known as tbarron | 17:41 | |
| hubbot1 | FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/rocky: tripleo-ci- (2 more messages) | 17:46 |
| rlandy | weshay: you mentioned that the container image prepare command changed. Is that merged/updated in CI yet? | 18:24 |
| weshay | rlandy, oh ya | 18:24 |
| weshay | rlandy, there have been some changes that happened w/o widespread knowledge | 18:24 |
| weshay | rlandy, read through that image-modify role | 18:25 |
| rlandy | weshay: still working on rhos-14 in the background | 18:25 |
| weshay | k | 18:25 |
| rlandy | looking - I think I have a few fixes | 18:25 |
| *** trown|lunch is now known as trown | 18:32 | |
| rlandy | hmmm ... don;t see any changes in CI | 18:41 |
| rlandy | ugh - no dice on depends-on THT review | 18:51 |
| rlandy | rebased on nv job | 18:53 |
| *** ykarel|away has quit IRC | 18:53 | |
| weshay | rlandy, /me shows u | 19:01 |
| weshay | sec | 19:01 |
| weshay | rlandy, https://github.com/openstack/ansible-role-tripleo-modify-image | 19:02 |
| weshay | https://review.openstack.org/#/q/status:open+project:openstack/tripleo-docs+branch:master+topic:bp/container-prepare-workflow | 19:02 |
| weshay | rlandy, and ALL https://review.openstack.org/#/q/status:merged+topic:bp/container-prepare-workflow | 19:03 |
| *** zul has quit IRC | 19:08 | |
| *** jtomasek has quit IRC | 19:17 | |
| *** jtomasek has joined #oooq | 19:20 | |
| rlandy | oh I see | 19:21 |
| *** zul has joined #oooq | 19:25 | |
| rfolco | tripleo-ci-centos-7-containerized-undercloud-upgrades is problematic weshay ? | 19:28 |
| rlandy | rfolco: ack | 19:29 |
| rlandy | <weshay> rlandy, fyi https://review.openstack.org/#/c/603187/1/zuul.d/undercloud-jobs.yaml | 19:29 |
| weshay | sec | 19:30 |
| rfolco | rlandy, thanks | 19:30 |
| rlandy | mirror vexxhost??? | 19:31 |
| rlandy | http://logs.openstack.org/45/602445/5/check/tripleo-ci-centos-7-containers-multinode/c01eace/logs/subnode-2/home/zuul/repo_setup.log.txt.gz#_2018-09-17_19_22_13 | 19:32 |
| rlandy | rfolco: weshay: ^^ fyi | 19:33 |
| rlandy | failing repo setup | 19:33 |
| rlandy | checking iwth sf-dfg | 19:35 |
| *** jtomasek has quit IRC | 19:37 | |
| *** jtomasek has joined #oooq | 19:40 | |
| weshay | rlandy, I just wrote up a blocking bug | 19:41 |
| weshay | rlandy, /me decides to speak w/ bugs | 19:41 |
| weshay | rlandy, vs.. english | 19:41 |
| rlandy | weshay: probably a wise switch ... english communication got no response for me | 19:43 |
| rlandy | oh my gosh, will we ever get this patch through ????? | 19:44 |
| weshay | rlandy, we might as well chat on blue | 19:45 |
| weshay | I have a question for you as well | 19:46 |
| hubbot1 | FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7 (2 more messages) | 19:46 |
| rlandy | ok - joining blue | 19:46 |
| weshay | rlandy, https://bugs.launchpad.net/tripleo/+bug/1793006 | 19:53 |
| openstack | Launchpad bug 1793006 in tripleo "yum repo failures in upstream mirrors" [Critical,Triaged] | 19:53 |
| *** jtomasek has quit IRC | 20:22 | |
| *** zul has quit IRC | 20:32 | |
| *** agopi|afk is now known as agopi | 21:06 | |
| *** trown is now known as trown|outtypewww | 21:07 | |
| hubbot1 | FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7 (2 more messages) | 21:46 |
| rlandy | weshay: hi | 21:56 |
| weshay | rlandy, hey | 21:56 |
| rlandy | weshay: https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-quickstart-rhos-14-passed-p1-baremetal-hp_dl360_envE-single_nic_vlans/7/console got as far as the tempest run - so we are getting there ... | 21:56 |
| rlandy | also | 21:56 |
| * rlandy copies ... | 21:56 | |
| weshay | wow | 21:56 |
| weshay | rlandy, oh.. it was trying to setup tempest on the container | 21:57 |
| weshay | Error: image rhosp14/centos-binary-tempest:current-tripleo not found | 21:57 |
| rlandy | https://paste.fedoraproject.org/paste/wLXlXMjIhL0CjM49dlP-JQ | 21:57 |
| rlandy | ^^ we need something like that for the changes | 21:57 |
| weshay | I wonder if internal is building that yet | 21:57 |
| rlandy | will test out and update | 21:57 |
| rlandy | I will ask | 21:58 |
| rlandy | anyways - will work on the change thing | 22:00 |
| rlandy | but that's the logic | 22:00 |
| rlandy | rerunning rhos-14 with tempest settings changes | 22:20 |
| *** vinaykns has quit IRC | 22:34 | |
| *** rlandy is now known as rlandy|bbl | 22:50 | |
| *** tosky has quit IRC | 23:23 | |
| hubbot1 | FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/602248, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens, tripleo-ci-centos-7-scenario004-multinode-oooq-container @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7 (2 more messages) | 23:46 |
| *** dsneddon has quit IRC | 23:52 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!