| myoung|ruck | rlandy: just checked in on osp13, we're well into deploy then create_failed | 00:03 |
|---|---|---|
| myoung|ruck | rlandy: kicking rhos-12 as well to see if we can make that work for now as well | 00:03 |
| myoung|ruck | and will queue up a few overnight...i would at least like to see same failure twice :) | 00:03 |
| * myoung|ruck watches https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/computer/rdo-manager-slave_rdo-ci-fx2-02-s2-whayutin-debug tonight | 00:05 | |
| rlandy | myoung|ruck: ok - good progress | 00:08 |
| rlandy | getting there for env E as well | 00:08 |
| rlandy | I hvae one more fix - just thinking how to incorporate it in the current settings | 00:09 |
| rlandy | bbl to do this | 00:09 |
| *** rlandy is now known as rlandy|bbl | 00:09 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/560445 | 00:15 |
| *** jtomasek has joined #oooq | 00:33 | |
| *** d0ugal_ has quit IRC | 00:33 | |
| *** d0ugal_ has joined #oooq | 00:40 | |
| *** d0ugal_ has quit IRC | 01:59 | |
| *** d0ugal_ has joined #oooq | 02:08 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/560445 | 02:15 |
| *** jtomasek has quit IRC | 03:39 | |
| *** matbu has joined #oooq | 04:03 | |
| *** saneax-_-|AFK has quit IRC | 04:08 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-containerized-undercloud-upgrades @ https://review.openstack.org/560445 | 04:15 |
| *** udesale has joined #oooq | 04:35 | |
| *** links has joined #oooq | 04:39 | |
| *** marios has joined #oooq | 04:41 | |
| *** jaganathan has quit IRC | 04:45 | |
| *** jaganathan has joined #oooq | 04:45 | |
| *** ykarel has joined #oooq | 04:47 | |
| *** ratailor has joined #oooq | 04:54 | |
| myoung|ruck | rlandy|bbl: looks like rhos-12 failed with OverLimit: resources.baremetal_volume: VolumeSizeExceedsAvailableQuota: Requested volume or snapshot exceeds allowed gigabytes quota. Requested 50G, quota is 1000G and 1000G has been consumed. (HTTP 413) (Request-ID: req-5e3f0c80-d5e2-4b4a-95e4-2fe6a46b3713), need to clean up tenant methinks. | 04:59 |
| myoung|ruck | rlandy|bbl: rhos-13 replicated same failure during OC deploy, so at least we're consistent there. will look to clean tenant and continue tomorrow | 04:59 |
| myoung|ruck | sshnaidm|off: ^fyi ^ | 05:00 |
| myoung|ruck | rlandy|bbl, sshnaidm|off: cleaned tenant and running again. | 05:05 |
| * myoung|ruck will now transmogrify into a sleeping person. zzzz | 05:06 | |
| *** gvrangan has joined #oooq | 05:09 | |
| *** gvrangan has quit IRC | 05:14 | |
| *** jfrancoa has joined #oooq | 05:20 | |
| *** ratailor has quit IRC | 05:36 | |
| *** ratailor has joined #oooq | 05:36 | |
| *** gvrangan has joined #oooq | 05:44 | |
| *** ratailor has quit IRC | 05:50 | |
| *** yolanda has joined #oooq | 05:50 | |
| *** ratailor has joined #oooq | 05:53 | |
| *** links has quit IRC | 06:04 | |
| *** quiquell|off is now known as quiquell | 06:05 | |
| quiquell | myoung|ruck: You awake ? | 06:05 |
| quiquell | Go t o bed | 06:05 |
| *** yolanda has quit IRC | 06:05 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-containerized-undercloud-upgrades @ https://review.openstack.org/560445 | 06:15 |
| *** udesale_ has joined #oooq | 06:16 | |
| *** ykarel_ has joined #oooq | 06:17 | |
| *** udesale has quit IRC | 06:18 | |
| *** rlandy|bbl has quit IRC | 06:18 | |
| *** yolanda has joined #oooq | 06:19 | |
| *** ykarel has quit IRC | 06:19 | |
| *** links has joined #oooq | 06:20 | |
| *** kopecmartin has joined #oooq | 06:28 | |
| *** jbadiapa_ has quit IRC | 06:32 | |
| *** _jbadiapa has joined #oooq | 06:32 | |
| *** holser__ has joined #oooq | 06:44 | |
| *** udesale__ has joined #oooq | 06:44 | |
| *** udesale_ has quit IRC | 06:46 | |
| *** udesale__ has quit IRC | 06:52 | |
| *** udesale__ has joined #oooq | 06:52 | |
| *** ykarel__ has joined #oooq | 06:53 | |
| *** ykarel_ has quit IRC | 06:56 | |
| *** ykarel__ is now known as ykarel|mtg | 07:01 | |
| *** udesale_ has joined #oooq | 07:01 | |
| *** udesale__ has quit IRC | 07:04 | |
| *** saneax has joined #oooq | 07:06 | |
| *** saneax has quit IRC | 07:08 | |
| *** saneax has joined #oooq | 07:09 | |
| *** udesale_ has quit IRC | 07:11 | |
| *** udesale has joined #oooq | 07:12 | |
| *** udesale has quit IRC | 07:12 | |
| *** udesale has joined #oooq | 07:12 | |
| *** jaosorior has quit IRC | 07:18 | |
| *** jaosorior has joined #oooq | 07:19 | |
| *** tesseract has joined #oooq | 07:20 | |
| *** _jbadiapa has quit IRC | 07:23 | |
| *** bogdando has joined #oooq | 07:30 | |
| *** _jbadiapa has joined #oooq | 07:38 | |
| *** amoralej|off is now known as amoralej | 07:42 | |
| *** udesale_ has joined #oooq | 07:51 | |
| *** udesale has quit IRC | 07:54 | |
| *** links has quit IRC | 08:01 | |
| *** d0ugal_ has quit IRC | 08:08 | |
| *** d0ugal has joined #oooq | 08:08 | |
| *** marios has quit IRC | 08:13 | |
| *** marios has joined #oooq | 08:14 | |
| *** links has joined #oooq | 08:15 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-containerized-undercloud-upgrades @ https://review.openstack.org/560445 | 08:15 |
| *** gkadam has joined #oooq | 08:17 | |
| quiquell | Hi need some cores love here https://review.openstack.org/#/c/566565/ | 08:37 |
| *** ykarel|mtg is now known as ykarel | 08:39 | |
| *** skramaja has joined #oooq | 08:50 | |
| *** sshnaidm|off is now known as sshnaidm|rover | 08:52 | |
| quiquell | sshnaidm|rover: Good morning | 08:53 |
| quiquell | Sprint13 review https://review.openstack.org/560445 | 08:54 |
| sshnaidm|rover | quiquell, morning, looking | 08:54 |
| quiquell | ^ thanks ! | 08:54 |
| sshnaidm|rover | quiquell, seems like it's gate checking :) | 08:55 |
| quiquell | already ! | 08:55 |
| quiquell | cool | 08:55 |
| quiquell | Wait, wrong patch | 08:55 |
| quiquell | :-/ | 08:55 |
| quiquell | sshnaidm|rover: This is it https://review.openstack.org/#/c/566565/ | 08:55 |
| quiquell | Sorry | 08:55 |
| sshnaidm|rover | quiquell, hmm.. how does it have -1 from weshay_pto | 08:57 |
| quiquell | sshnaidm|rover: Was old -1 | 08:57 |
| quiquell | PS 11 | 08:57 |
| quiquell | Now the logs appear | 08:57 |
| quiquell | This is at PS 15 | 08:57 |
| sshnaidm|rover | quiquell, why are these featuresets chosen? | 09:15 |
| quiquell | 37 is the keystone only update | 09:15 |
| quiquell | (give me a minute to elaborate) | 09:16 |
| quiquell | Ok so | 09:17 |
| quiquell | We have choose to apply it on the simplest scenarios | 09:17 |
| quiquell | fs037 (keystone only update) | 09:17 |
| quiquell | fs050 (containerized undercloud upgrade) | 09:17 |
| quiquell | They just install the undercloud | 09:18 |
| quiquell | fs010 and fs011 to ensure backward compatibility with mixed upgrades | 09:19 |
| sshnaidm|rover | quiquell, thanks, +w | 09:28 |
| quiquell | sshnaidm|rover: Awesome, thanks | 09:29 |
| *** zoli is now known as zoli|lunch | 09:30 | |
| *** udesale__ has joined #oooq | 09:31 | |
| *** yolanda has quit IRC | 09:32 | |
| *** udesale has joined #oooq | 09:32 | |
| *** udesale_ has quit IRC | 09:33 | |
| *** udesale__ has quit IRC | 09:36 | |
| *** yolanda has joined #oooq | 09:45 | |
| quiquell | sshnaidm|rover: Added changed to the ruck/rovert alarms | 09:51 |
| quiquell | http://38.145.34.131:3000/d/pgdr_WVmk/ruck-rover?orgId=1&panelId=24&tab=alert | 09:51 |
| quiquell | We have some alarms already | 09:52 |
| sshnaidm|rover | quiquell, great | 09:53 |
| quiquell | Need to link to sova somehow | 09:53 |
| sshnaidm|rover | quiquell, preparing now script with queues.. did you use "status?json" also in 3d party CI? https://review.rdoproject.org/zuul/ | 09:54 |
| quiquell | for upsream zuul.openstack.org/api/status | 09:54 |
| quiquell | for RDO | 09:54 |
| quiquell | review.rdoproject.org/zuul/status.json | 09:55 |
| quiquell | sshnaidm|rover: The script it's runnign now pushes the queue to influxdb | 09:57 |
| quiquell | We have to merge the two | 09:58 |
| quiquell | And read influxdb sova and grafana | 09:58 |
| quiquell | So we don't duplicate | 09:58 |
| quiquell | What do you think | 09:58 |
| * quiquell need to go back to sprint13 duties | 09:58 | |
| sshnaidm|rover | quiquell, need some time to test it, and then we'll sync | 10:00 |
| quiquell | sshnaidm|rover: Sound like a plan | 10:00 |
| quiquell | sshnaidm|rover: The one running now is here: https://review.rdoproject.org/r/#/c/13797/14/ci-scripts/ruck-rover-cockpit/zuul_queues_status.py | 10:01 |
| sshnaidm|rover | quiquell, afaiu the output from script should be just lines in influx format? | 10:02 |
| quiquell | Yep, using the module.exec with type influxdb in telegraf | 10:03 |
| quiquell | You can check the telegraf.conf | 10:03 |
| quiquell | Maybe we can just refactor zuul_queues_status.py with the stuff you are missing | 10:04 |
| sshnaidm|rover | quiquell, ok.. lemme play with that a little | 10:07 |
| quiquell | Ok, is super simple to use telegraf | 10:07 |
| hubbot | FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-ci-centos-7-containerized-undercloud-upgrades @ https://review.openstack.org/560445 | 10:15 |
| quiquell | sshnaidm|rover: sprint13 https://review.openstack.org/#/c/569958 | 10:16 |
| *** udesale has quit IRC | 10:40 | |
| *** zoli|lunch is now known as zoli | 10:40 | |
| sshnaidm|rover | myoung|ruck, ping me when you're online | 10:44 |
| quiquell | sshnaidm|rover: Do you know if we have any problem with zuul_changes ? | 10:56 |
| quiquell | I am running a reproducer and it's empty | 10:56 |
| sshnaidm|rover | quiquell, reproducer of which job? | 10:56 |
| quiquell | Looks like zuul_changes: "{{ lookup('env', 'ZUUL_CHANGES') }}" It's not working | 10:56 |
| quiquell | sshnaidm|rover: http://logs.openstack.org/65/566565/11/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/3bae60f/logs/ | 10:57 |
| quiquell | Ok, we are missing an export | 10:58 |
| sshnaidm|rover | quiquell, damn... we set : ${ZUUL_CHANGES:="openstack-infra/tripleo-ci:master:refs/changes/65/566565/11"} but no export | 10:58 |
| sshnaidm|rover | quiquell, exactly | 10:58 |
| quiquell | Ok, going to fix | 10:58 |
| sshnaidm|rover | quiquell, thanks! | 10:58 |
| quiquell | "Let's make the reproducer great again" | 10:59 |
| quiquell | :-) | 10:59 |
| quiquell | As Wes Trump would say | 10:59 |
| sshnaidm|rover | quiquell, seems like it was the reason for https://review.openstack.org/#/c/569902/ | 10:59 |
| quiquell | Hummm | 11:00 |
| quiquell | I like more using ansible zuul_changes | 11:00 |
| quiquell | more explicit | 11:00 |
| quiquell | But it's not what zuul does | 11:00 |
| quiquell | We have to reproduce zuul | 11:01 |
| sshnaidm|rover | quiquell, yeah, I think export will be enough | 11:01 |
| quiquell | sshnaidm|rover: What else do we need to export ? | 11:05 |
| sshnaidm|rover | quiquell, just looked..seems like nothing more | 11:06 |
| quiquell | ok | 11:06 |
| myoung|ruck | sshnaidm|rover: have a moment to BJ? i have about 10 min then need to leave to take kids to school | 11:12 |
| sshnaidm|rover | myoung|ruck, yep | 11:12 |
| myoung|ruck | sshnaidm|rover: bluejeans.com/matyoung | 11:12 |
| quiquell | sshnaidm|rover: https://review.openstack.org/#/c/570167/ the ZUUL_CHANGES | 11:16 |
| quiquell | myoung|ruck: I cannot write stuf af retro trello | 11:16 |
| sshnaidm|rover | myoung|ruck, if you need time, let's sync later, not urgent | 11:16 |
| sshnaidm|rover | myoung|ruck, everything is pretty good for now | 11:17 |
| *** dtantsur has joined #oooq | 11:18 | |
| *** dtantsur has quit IRC | 11:18 | |
| *** dtantsur has joined #oooq | 11:19 | |
| sshnaidm|rover | myoung|ruck, so what is problem with queens rdo phase 2? | 11:19 |
| myoung|ruck | sshnaidm|rover: sorry for bluejeans fun | 11:21 |
| sshnaidm|rover | myoung|ruck, no worries, let's switch to text mode | 11:21 |
| myoung|ruck | sshnaidm|rover: issue with RDO2 is the BM jobs for queens have likely some small piece of missing configuration, envB, envD, envE started going red...likely a backport missing on stable/queens, or a CP came into stable/queens and our BM configs has drifted | 11:22 |
| sshnaidm|rover | myoung|ruck, "CP"? | 11:22 |
| myoung|ruck | sshnaidm|rover: cherrypick | 11:24 |
| myoung|ruck | sshnaidm|rover: however we have removed the older BM from promotion criteria....for the 2 that DO vote now to promote rdo2, we already have a pass on https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/oooq-queens-rdo_trunk-bmu-haa16-lab-float_nic_with_vlans/48/ | 11:25 |
| sshnaidm|rover | myoung|ruck, so... should we get promotion then? | 11:25 |
| myoung|ruck | yes, later today after https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/rdo-promote-queens-rdo_trunk/86 causes fs020 to run | 11:26 |
| myoung|ruck | sshnaidm|rover: so rdo2: queens should promote https://trunk.rdoproject.org/centos7-queens/4a/7a/4a7aa0c37c83d9a6259cf5ccad674aa24a4ad5bf_f0f5884b later today | 11:26 |
| myoung|ruck | so status for program meeting is yellow (now) but should flip back to green later today | 11:27 |
| sshnaidm|rover | myoung|ruck, ack | 11:28 |
| myoung|ruck | quiquell: added you to retro board | 11:30 |
| myoung|ruck | https://trello.com/b/0VFswmht/rdo-infra-retrospective | 11:30 |
| * myoung|ruck will be back online in 1 hr | 11:30 | |
| *** quiquell is now known as quiquell|lunch | 11:37 | |
| *** marios has quit IRC | 11:39 | |
| *** marios has joined #oooq | 11:41 | |
| quiquell|lunch | trown|outtypewww: https://review.openstack.org/#/c/570167/, problem with reproducer | 11:44 |
| *** amoralej is now known as amoralej|lunch | 11:52 | |
| *** skramaja has quit IRC | 12:01 | |
| quiquell|lunch | sshnaidm|rover: Make sense to create a job that runs the reproducer ? | 12:04 |
| quiquell|lunch | To check that we don't break the reproducer ? | 12:04 |
| hubbot | FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/560445 | 12:15 |
| *** udesale has joined #oooq | 12:17 | |
| *** trown|outtypewww is now known as trown | 12:27 | |
| *** rlandy has joined #oooq | 12:35 | |
| *** quiquell|lunch is now known as quiquell | 12:35 | |
| chandankumar | myoung|ruck: can we get this merged https://review.rdoproject.org/r/#/c/13851/ ? | 12:41 |
| *** ratailor has quit IRC | 12:45 | |
| myoung|ruck | chandankumar: lgtm +1, I'm not a core on that repo. trown, rlandy ^^ | 12:49 |
| myoung|ruck | panda|off: ^^ | 12:49 |
| chandankumar | rlandy: myoung|ruck trown thanks :-) | 12:53 |
| myoung|ruck | all: good morning! sprint 13 retrospective meeting in 5m. | 12:56 |
| *** myoung|ruck is now known as myoung | 12:56 | |
| *** amoralej|lunch is now known as amoralej | 12:59 | |
| myoung | all: meeting invite is off... https://bluejeans.com/7050859455 is correct link | 12:59 |
| *** tcw has quit IRC | 13:13 | |
| *** tcw has joined #oooq | 13:13 | |
| *** yolanda has quit IRC | 13:17 | |
| *** yolanda has joined #oooq | 13:19 | |
| *** gvrangan has quit IRC | 13:21 | |
| *** ykarel is now known as ykarel|afk | 13:36 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/560445 | 14:15 |
| *** matbu has quit IRC | 14:20 | |
| sshnaidm|rover | myoung, wanna me to cover CIX call? | 14:21 |
| myoung | sshnaidm|rover: yes please | 14:21 |
| myoung | sshnaidm|rover: but if you want to blow it off to stay in retro that's fine too | 14:21 |
| myoung | sshnaidm|rover: i'm torn...this is our retrospective and one of the more important places where we as a team can discuss/improve | 14:21 |
| sshnaidm|rover | myoung, I'll join there | 14:21 |
| *** zoli is now known as zoli|brb | 14:22 | |
| *** matbu has joined #oooq | 14:22 | |
| *** saneax has quit IRC | 14:28 | |
| myoung | chandankumar: we lost ya... | 14:45 |
| rasca | hey myoung I'm experiencing failures everywhere in RDOPhase2, due to "AssertionError: tripleo-ha-utils .dist-info directory not found" do you have any hint on this | 14:48 |
| rasca | ? | 14:48 |
| myoung | rasca: can talk in a bit...finishing retro with team | 14:48 |
| myoung | rasca: guessing is related to recent patch that changed to tripleo-ha-utils (vs. redhat-openstack/oldreponame) | 14:54 |
| myoung | in code.eng | 14:54 |
| myoung | rasca: link? | 14:54 |
| rasca | uhm | 14:55 |
| rasca | but I checked it | 14:55 |
| rasca | it worked before | 14:55 |
| rasca | that's strange | 14:55 |
| myoung | rasca: aye...will be in retro for ~35 more mins | 14:55 |
| *** zoli|brb is now known as zoli | 15:02 | |
| *** matbu has quit IRC | 15:07 | |
| *** bogdando has quit IRC | 15:30 | |
| *** quiquell is now known as quiquell|off | 15:34 | |
| *** matbu has joined #oooq | 15:40 | |
| arxcruz | /nick arxcruz|rover | 15:42 |
| *** arxcruz is now known as arxcruz|rover | 15:42 | |
| *** rlandy is now known as rlandy|rover | 15:42 | |
| myoung | rook: are you aware of the pile of queued up jobs on rhos-dev-jenkins for browbeat? is agopi around? | 15:45 |
| myoung | rlandy|rover: ^^, arxcruz|rover ^^ | 15:45 |
| myoung | bunch have been waiting 13 days, we probably just need a timeout on those jobs but wanted to make sure it was on a radar for perf/browbeat folk | 15:46 |
| panda|off | myoung: do you have time for bj ? | 15:55 |
| myoung | panda|off: sure | 15:55 |
| myoung | now? | 15:55 |
| * myoung is updating https://review.rdoproject.org/etherpad/p/ruckrover-sprint14 for handoff and can talk now | 15:56 | |
| panda|off | myoung: I'm already in your room | 15:56 |
| myoung | panda|off: ack, incoming | 15:58 |
| *** marios has quit IRC | 16:01 | |
| *** kopecmartin has quit IRC | 16:03 | |
| *** arxcruz|rover is now known as arxcruz|ruck | 16:03 | |
| arxcruz|ruck | myoung: done | 16:03 |
| *** udesale has quit IRC | 16:07 | |
| *** zoli is now known as zoli|gone | 16:11 | |
| *** zoli|gone is now known as zoli | 16:11 | |
| chandankumar | arxcruz|ruck: myoung tomorrow i will be in bangalore | 16:12 |
| chandankumar | who is the rover? | 16:12 |
| hubbot | FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode @ https://review.openstack.org/560445 | 16:15 |
| *** jfrancoa has quit IRC | 16:15 | |
| *** jfrancoa has joined #oooq | 16:16 | |
| *** links has quit IRC | 16:16 | |
| arxcruz|ruck | chandankumar: rlandy|rover | 16:19 |
| *** ykarel|afk has quit IRC | 16:24 | |
| chandankumar | arxcruz|ruck: cool! | 16:27 |
| *** jfrancoa has quit IRC | 16:27 | |
| arxcruz|ruck | time for gow now | 16:27 |
| rlandy|rover | arxcruz|ruck: I think we changed roles :) | 16:34 |
| * rlandy|rover thought I was supposed to be ruck | 16:35 | |
| rlandy|rover | anyways ... | 16:35 |
| chandankumar | good night guys see ya tomorrow :-) | 16:38 |
| rlandy|rover | arxcruz|ruck: sorry - I messed up with my nick | 16:45 |
| rlandy|rover | myoung: meet at 1pm? | 16:45 |
| myoung | rlandy|rover: i've been with panda|off since we ended retro...could we make it 1:30? | 16:48 |
| *** sshnaidm|rover is now known as sshnaidm|off | 16:52 | |
| *** holser__ has quit IRC | 16:57 | |
| *** dtantsur is now known as dtantsur|afk | 16:59 | |
| *** trown is now known as trown|lunch | 17:00 | |
| rlandy|rover | sure | 17:07 |
| *** amoralej is now known as amoralej|off | 17:12 | |
| rlandy|rover | sshnaidm|off: myoung: not sure what the issue is exactly with https://ci.centos.org/job/tripleo-quickstart-promote-ocata-rdo_trunk-minimal/341/console | 17:34 |
| *** matbu has quit IRC | 17:35 | |
| *** tesseract has quit IRC | 17:38 | |
| myoung | hrm...looks like it hangs forever... | 17:40 |
| myoung | 2018-05-23 16:30:48 | Started Mistral Workflow tripleo.baremetal.v1.introspect_manageable_nodes. Execution ID: 8c6b629d-e94f-426f-8a18-75d456f5eaea | 17:40 |
| myoung | 2018-05-23 16:30:48 | Waiting for introspection to finish... | 17:40 |
| myoung | ^^ https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-ocata-rdo_trunk-minimal-341/undercloud/home/stack/overcloud_prep_images.log.gz | 17:40 |
| myoung | rlandy WOOT rhos-12 ovb gate job is green for the first time in a long time: https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tq-gate-rhos-12-ci-rhos-ovb-minimal-pacemaker-public-bond/917/ | 17:50 |
| *** hamzy has joined #oooq | 17:56 | |
| rlandy|rover | this makes no sense | 17:56 |
| myoung | rlandy|rover: back...can talk now if you have time still | 17:57 |
| rlandy|rover | myoung: yes | 17:57 |
| rlandy|rover | myoung: bj? | 17:58 |
| rlandy|rover | myoung: in your bj | 17:59 |
| myoung | incoming | 17:59 |
| *** matbu has joined #oooq | 17:59 | |
| *** jtomasek has joined #oooq | 18:00 | |
| *** jtomasek has quit IRC | 18:04 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/560445 | 18:15 |
| *** trown|lunch is now known as trown | 18:17 | |
| *** jtomasek has joined #oooq | 18:22 | |
| myoung | rlandy|rover: http://logs.openstack.org/36/568236/2/gate/tripleo-ci-centos-7-scenario007-multinode-oooq-container/45b33c4/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz#_2018-05-23_16_47_11 | 18:23 |
| *** matbu has quit IRC | 18:25 | |
| *** jtomasek has quit IRC | 18:26 | |
| *** jtomasek has joined #oooq | 18:34 | |
| *** hamzy has quit IRC | 18:35 | |
| *** hamzy has joined #oooq | 18:44 | |
| *** jtomasek has quit IRC | 18:46 | |
| *** matbu has joined #oooq | 19:02 | |
| *** rlandy|rover is now known as rlandy|rover|bia | 19:04 | |
| *** rlandy|rover|bia is now known as rlandy|rvr|biab | 19:04 | |
| *** jaganathan has quit IRC | 19:06 | |
| *** matbu has quit IRC | 19:08 | |
| *** gkadam has quit IRC | 19:13 | |
| *** jaganathan has joined #oooq | 19:14 | |
| *** trown is now known as trown|brb | 19:22 | |
| *** hamzy has quit IRC | 19:30 | |
| *** trown|brb is now known as trown | 19:48 | |
| rook | myoung: i thought he disabled the jobs prior to leaving | 20:02 |
| rook | he is coming back | 20:02 |
| rook | but he doesn't start until july | 20:03 |
| rook | looking at https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/user/agopi/my-views/view/Browbeat_view/ now | 20:03 |
| rook | it isn't even loading | 20:04 |
| rook | grand. | 20:04 |
| rook | rhos-11 seems fine | 20:07 |
| *** rlandy|rvr|biab is now known as rlandy|rover | 20:08 | |
| *** holser__ has joined #oooq | 20:11 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/560445 | 20:15 |
| *** hamzy has joined #oooq | 20:22 | |
| *** trown is now known as trown|outtypewww | 20:26 | |
| rlandy|rover | myoung: pushed change to envE - rekicking | 20:52 |
| rlandy|rover | myoung: what's with ""https://thirdparty.logs.rdoproject.org/jenkins-periodic-master-rdo_trunk-ovb-minimal-pacemaker-multiple-nics-183/ovb_create_stack.log | 21:11 |
| rlandy|rover | ci-rhos.centralci | 21:12 |
| rlandy|rover | can't tell where that is | 21:13 |
| myoung | rlandy|rover: afaict it's https://ci-rhos.centralci.eng.rdu2.redhat.com, and it's trying to find , CentOS-7-x86_64-GenericCloud-released-latest, of which there are 2...one raw one qcow2 (with same name)...which is dubious. I think we should update this to centos7 1804, which doesn't exist there (yet) | 21:25 |
| rlandy|rover | myoung: hmmm ... that requires updates | 21:26 |
| myoung | rlandy|rover: it's coming from the RDO2 master job (https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/periodic-master-rdo_trunk-ovb-minimal-pacemaker-multiple-nics) | 21:26 |
| myoung | i don't remember adding this | 21:26 |
| rlandy|rover | panda added it I think | 21:27 |
| rlandy|rover | looking at the job now | 21:27 |
| myoung | and afaict last success is never...i can make a ticket with centralCI to add 1804, in the meantime I wonder if the duplicate images (with same name but different guids) is the problem? | 21:27 |
| myoung | that was on the back burner all sprint and I didn't get to dig into it yet | 21:27 |
| myoung | (and it's not part of promoter criteria) | 21:28 |
| * myoung tries something | 21:29 | |
| rlandy|rover | why does the other job not trip over that? | 21:30 |
| rlandy|rover | myoung: ^^ | 21:32 |
| rlandy|rover | master: CentOS-7-x86_64-GenericCloud-released-latest | 21:33 |
| rlandy|rover | we have used that forever | 21:33 |
| *** matbu has joined #oooq | 21:38 | |
| *** holser__ has quit IRC | 21:39 | |
| rlandy|rover | oh I see why | 21:39 |
| myoung | rlandy I have 1804_02 uploading to glance now | 21:41 |
| myoung | patch incomiing | 21:41 |
| myoung | incoming | 21:41 |
| rlandy|rover | myoung: it may fix it - but I don't love the idea | 21:42 |
| rlandy|rover | tha requires constant update | 21:42 |
| myoung | wierdly the internal rhos tenant doesn't support URL's so had to download / upload it. | 21:42 |
| myoung | I don't love it either...but right now it's busted and this is a quick test | 21:42 |
| myoung | also the "latest" duplicated image name (one raw, one qcow2) are based on 2 diffferent composes, both of which are ancient | 21:43 |
| myoung | details in commit...sec...have a patch inna jiff | 21:43 |
| * myoung watches the progress bar reach 95% | 21:43 | |
| rlandy|rover | myoung: I think the right answer is to get two diff names | 21:44 |
| rlandy|rover | CentOS-7-x86_64-GenericCloud-released-latest-qcow | 21:44 |
| rlandy|rover | CentOS-7-x86_64-GenericCloud-released-latest-raw | 21:44 |
| * rlandy|rover puts in a ticket | 21:45 | |
| myoung | rlandy|rover: aye...but also neither of those images is close to current either | 21:48 |
| rlandy|rover | part of the request | 21:48 |
| myoung | the image create is hanging | 21:48 |
| myoung | and failed with a nondescript error | 21:48 |
| myoung | will make patch us 1607, latest qcow2 up there now...it'll just update longer. | 21:49 |
| myoung | rlandy|rover: https://code.engineering.redhat.com/gerrit/139666 Update master centos image to 1607 (unique) | 21:51 |
| rlandy|rover | myoung: why not switch them all? | 21:55 |
| rlandy|rover | can't find the stupid ticket category :( | 21:55 |
| myoung | rlandy|rover: afaik there are no other rdo ovb periodics, and figured we would get unique names sorted. just wanted to merge that, kick job, see what happens | 21:58 |
| * myoung looks for categories...there's a generic one | 21:59 | |
| rlandy|rover | myoung: https://redhat.service-now.com/pnt?id=ticket&table=x_redha_pnt_devops_table&sys_id=740980c1dbde1f80a16700b5ca9619b6 | 21:59 |
| rlandy|rover | myoung: ok - go4it | 22:00 |
| rlandy|rover | now we have a proper ticket | 22:00 |
| myoung | rlandy|rover: ack, just added gory/verbose details to igt | 22:02 |
| myoung | thanks! | 22:02 |
| *** matbu has quit IRC | 22:02 | |
| rlandy|rover | myoung: watching this ... https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-quickstart-master-rdo_trunk-baremetal-hp_dl360_envE-single_nic_vlans/128/console | 22:03 |
| rlandy|rover | if that passes, we have a way forward | 22:03 |
| myoung | rlandy|rover: nice | 22:04 |
| myoung | rlandy|rover: i just retried the periodic ovb master, looks like on the older node only 2 executors | 22:04 |
| myoung | the other one we stood up for gate_oooq_ovb pool (https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/computer/rdo-manager-17)is borked and offline | 22:04 |
| myoung | wierd | 22:04 |
| * myoung looks at it some more | 22:05 | |
| myoung | rdomgr17-hp-dl320g8-10.lab.eng.brq.redhat.com is nonresponsive. going to reboot it | 22:05 |
| rlandy|rover | ok | 22:06 |
| myoung | oof...ya entire machine is offline hp-dl320g8-10.lab.eng.brq.redhat.com | 22:08 |
| myoung | power state is on...just gonna push the reset button | 22:09 |
| * myoung mutters curses about iLO and boots up a windows machine to access the console | 22:11 | |
| myoung | brb | 22:11 |
| hubbot | FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-3nodes-multinode, tripleo-quickstart-extras-gate-newton-delorean-full-minimal @ https://review.openstack.org/560445 | 22:15 |
| *** hamzy has quit IRC | 22:25 | |
| rlandy|rover | myoung; got to go out fora few hours - will be back later to deal with failing jobs | 22:26 |
| myoung | rlandy|rover: rebooted host, had to manually start the slave VM for 17. https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/computer/rdo-manager-17/log is back online | 22:26 |
| *** rlandy|rover is now known as rlandy|rover|bbl | 22:26 | |
| *** matbu has joined #oooq | 22:37 | |
| *** dtrainor has joined #oooq | 22:51 | |
| *** dtrainor has quit IRC | 22:55 | |
| *** matbu has quit IRC | 23:10 | |
| *** hamzy has joined #oooq | 23:10 | |
| myoung | rlandy|rover|bbl: we're hitting https://bugs.launchpad.net/glance/+bug/1697728, and while I disagree with glance team's assessment of the issue...giving https://code.engineering.redhat.com/gerrit/#/c/139669 a go... | 23:40 |
| openstack | Launchpad bug 1697728 in Glance " openstack-glance: Adding images with the same name should not be allowed." [Undecided,Opinion] | 23:40 |
| myoung | rlandy|rover|bbl: details in gerrit...TLDR, swapped for id's, jobs running to validate centos + rhel 7.5 images (ref'd by id) for ci-rhos | 23:46 |
| myoung | https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/periodic-master-rdo_trunk-ovb-minimal-pacemaker-multiple-nics/186 | 23:46 |
| myoung | https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tq-gate-rhos-12-ci-rhos-ovb-minimal-pacemaker-public-bond/922 | 23:46 |
| *** myoung is now known as myoung|off | 23:46 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!