| *** dsneddon has joined #oooq | 00:45 | |
| *** dsneddon has quit IRC | 00:49 | |
| *** altlogbot_3 has quit IRC | 00:50 | |
| *** altlogbot_0 has joined #oooq | 00:53 | |
| *** altlogbot_0 has quit IRC | 00:54 | |
| *** altlogbot_3 has joined #oooq | 00:54 | |
| *** dsneddon has joined #oooq | 01:46 | |
| *** dsneddon has quit IRC | 01:50 | |
| *** dsneddon has joined #oooq | 02:46 | |
| *** dsneddon has quit IRC | 02:51 | |
| *** tcw has joined #oooq | 03:34 | |
| *** tcw has joined #oooq | 03:34 | |
| *** dsneddon has joined #oooq | 03:47 | |
| *** dsneddon has quit IRC | 03:52 | |
| *** udesale has joined #oooq | 03:56 | |
| *** jaosorior has joined #oooq | 04:43 | |
| *** dsneddon has joined #oooq | 04:48 | |
| *** jaosorior has quit IRC | 04:52 | |
| *** dsneddon has quit IRC | 04:53 | |
| *** jaosorior has joined #oooq | 05:00 | |
| *** ratailor has joined #oooq | 05:03 | |
| *** ykarel has joined #oooq | 05:13 | |
| *** quiquell|off is now known as quiquell | 05:17 | |
| *** marios has joined #oooq | 05:34 | |
| *** ksambor has joined #oooq | 05:36 | |
| *** udesale has quit IRC | 05:45 | |
| *** udesale has joined #oooq | 05:45 | |
| *** udesale has quit IRC | 05:49 | |
| *** udesale has joined #oooq | 05:49 | |
| *** dsneddon has joined #oooq | 05:49 | |
| *** ksambor has quit IRC | 05:52 | |
| *** dsneddon has quit IRC | 05:54 | |
| *** kopecmartin|off is now known as kopecmartin | 05:55 | |
| *** ksambor has joined #oooq | 06:06 | |
| *** holser_ has joined #oooq | 06:14 | |
| *** saneax has joined #oooq | 06:24 | |
| *** udesale has quit IRC | 06:47 | |
| *** udesale has joined #oooq | 06:47 | |
| *** dsneddon has joined #oooq | 06:50 | |
| *** dsneddon has quit IRC | 06:54 | |
| *** dsneddon has joined #oooq | 07:00 | |
| *** ykarel is now known as ykarel|lunch | 07:01 | |
| marios | o/ morning oooq | 07:04 |
|---|---|---|
| marios | sup | 07:04 |
| marios | quiquell: o/ | 07:05 |
| marios | ;) | 07:05 |
| quiquell | marios: who are you ? | 07:06 |
| marios | :( | 07:07 |
| quiquell | ip | 07:07 |
| marios | quiquell: "now you're just somebody that I used to know" | 07:08 |
| *** apetrich has joined #oooq | 07:10 | |
| quiquell | XD | 07:11 |
| *** sanjayu__ has quit IRC | 07:13 | |
| *** amoralej|off is now known as amoralej | 07:16 | |
| *** tesseract has joined #oooq | 07:17 | |
| *** ccamacho has joined #oooq | 07:20 | |
| *** bogdando has joined #oooq | 07:28 | |
| *** jbadiapa has joined #oooq | 07:37 | |
| *** jpena|off is now known as jpena | 07:44 | |
| *** ykarel|lunch is now known as ykarel | 07:48 | |
| *** dsneddon has quit IRC | 08:03 | |
| *** matbu has joined #oooq | 08:03 | |
| marios | needs reviews please https://review.opendev.org/#/c/653440/ | 08:13 |
| marios | part of https://tree.taiga.io/project/tripleo-ci-board/task/990 | 08:14 |
| *** dsneddon has joined #oooq | 08:15 | |
| panda | marios: cool user story bro. | 08:16 |
| marios | panda: :D | 08:16 |
| marios | panda: if you like it so much you can qe it ;) | 08:17 |
| *** dsneddon has quit IRC | 08:20 | |
| *** d0ugal has quit IRC | 08:47 | |
| *** derekh has joined #oooq | 08:48 | |
| *** ksambor has quit IRC | 08:52 | |
| *** chem has joined #oooq | 08:53 | |
| *** skramaja has joined #oooq | 08:55 | |
| *** d0ugal has joined #oooq | 09:03 | |
| *** ksambor has joined #oooq | 09:06 | |
| *** dtantsur has joined #oooq | 09:10 | |
| *** dsneddon has joined #oooq | 09:16 | |
| *** dsneddon has quit IRC | 09:21 | |
| *** dsneddon has joined #oooq | 09:30 | |
| *** holser_ is now known as holser|lunch | 10:15 | |
| *** dsneddon has quit IRC | 10:33 | |
| *** ykarel is now known as ykarel|afk | 10:39 | |
| marios | needs reviews please https://review.opendev.org/#/c/653440/ part of https://tree.taiga.io/project/tripleo-ci-board/task/990 | 11:00 |
| panda | marios: so my review is not enough ? | 11:05 |
| panda | tch | 11:05 |
| marios | panda: sorry there were no votes so i didnt scroll down far enough :) | 11:07 |
| marios | panda: thanks for review checking+updating | 11:08 |
| marios | panda: replied when you get a chance again thanks https://review.opendev.org/#/c/653440/3/zuul.d/base.yaml@171 | 11:13 |
| marios | panda: if you like i can reparent it and remove some of vars actualy just leave the playbooks override. however the general idea is that we have a base for standalone-upgrade periodic jobs to parent from, as the standlone periodic jobs parent on the tripleo-ci-base-standalone | 11:14 |
| *** udesale has quit IRC | 11:17 | |
| *** jpena is now known as jpena|lunch | 11:33 | |
| panda | marios: Sorry I wasn't clear. in crating the new base, you added some vas like undercloud, environment_infra, and an additional tag to tags: I wanted to know why you needed this additional stuff that the old hierarchy didn't need | 11:35 |
| marios | panda: ack but th eold hierarchy does need it | 11:36 |
| *** holser|lunch is now known as holser_ | 11:36 | |
| marios | panda: see notes under "Background, links on inheritance path" https://tree.taiga.io/project/tripleo-ci-board/task/990?kanban-status=1447276 | 11:37 |
| *** ykarel|afk is now known as ykarel | 11:39 | |
| marios | panda: e.g. periodic-tripleo-ci-centos-7-standalone-master --> tripleo-ci-base-standalone-periodic --> tripleo-ci-base-standalone-rdo --> tripleo-ci-base-standalone which is the one above the new one in https://review.opendev.org/#/c/653440/3/zuul.d/base.yaml | 11:39 |
| marios | panda: does it help? | 11:40 |
| marios | panda: so in the old hierarchy the periodics are parenting onto https://review.opendev.org/#/c/653440/3/zuul.d/base.yaml@142 | 11:40 |
| marios | panda: which has all those vars | 11:40 |
| marios | panda: and this job is all of that & standalone role, and then also the standalone-upgrade role | 11:41 |
| marios | panda: so we could make it a sub-job which is what i wrote in my comment https://review.opendev.org/#/c/653440/3/zuul.d/base.yaml | 11:41 |
| panda | marios: ok, thanks, is the job supposed to pass yet ? | 11:54 |
| marios | panda: the upstream master standalone-upgrade is not, but the stein should be with ~2hr runtime http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-standalone-upgrade-stein | 11:58 |
| marios | panda: however we can't see the periodic in https://review.rdoproject.org/r/#/c/20241/ until th depends on merge (cos it is a trusted project) but even that depends on the review we're discussing | 11:59 |
| marios | panda: see notes at "reviews" in https://tree.taiga.io/project/tripleo-ci-board/task/990?kanban-status=1447276 under "periodic" | 12:00 |
| *** dsneddon has joined #oooq | 12:00 | |
| *** rfolco|ruck has joined #oooq | 12:11 | |
| weshay | rfolco|ruck sorry I'm late.. I'm a little sick today | 12:11 |
| rfolco|ruck | weshay, do you still want to do bug triage ? | 12:13 |
| *** rlandy has joined #oooq | 12:17 | |
| *** quiquell is now known as quiquell|lunch | 12:25 | |
| rlandy | arxcruz: hi - trying to fix this error ... https://sf.hosted.upshift.rdu2.redhat.com/logs/15/165815/46/check/periodic-tripleo-ci-centos-7-bm_envD-3ctlr_1comp-featureset001-master/cfdca84/job-output.txt.gz#_2019-05-10_23_04_20_043833 | 12:27 |
| rlandy | ^^ happens when I build images | 12:27 |
| rlandy | I se some similar bug report for the reproducer | 12:28 |
| weshay | rfolco|ruck naw | 12:28 |
| rfolco|ruck | weshay, chandan is on pto, right | 12:28 |
| weshay | rfolco|ruck ya | 12:29 |
| weshay | let's save all the bugs for him :) | 12:29 |
| rlandy | arxcruz: what am I missing? | 12:29 |
| arxcruz | rlandy:checking | 12:31 |
| rlandy | arxcruz: there is also this ... https://sf.hosted.upshift.rdu2.redhat.com/logs/15/165815/46/check/periodic-tripleo-ci-centos-7-bm_envD-3ctlr_1comp-featureset001-master/cfdca84/logs/undercloud/home/zuul/tempest.log.txt.gz#_2019-05-10_23_04_16 | 12:31 |
| arxcruz | rlandy: hmmmm this was from some time ago, i remember weshay shows me how to fix it | 12:31 |
| arxcruz | weshay: do you remember what we did whit that message saying /home/zuul/tempest were missing ? | 12:32 |
| *** saneax has quit IRC | 12:32 | |
| rlandy | I see this as a fix for the reproducer bug: https://review.rdoproject.org/r/#/c/20212/3/tasks/libvirt/prepare.yaml | 12:33 |
| weshay | arxcruz hey.. that happened twice I think.. | 12:33 |
| weshay | arxcruz /me trying to remember | 12:34 |
| rlandy | since we are building the images maybe they are not included | 12:34 |
| weshay | rlandy what's not missing.. the tempest dir? | 12:34 |
| *** ratailor has quit IRC | 12:34 | |
| weshay | cirros image? | 12:34 |
| rlandy | weshay: this was your bug: https://bugs.launchpad.net/tripleo/+bug/1824243 and https://bugs.launchpad.net/tripleo/+bug/1824241 | 12:34 |
| openstack | Launchpad bug 1824243 in tripleo "cirros image missing from libvirt reproducer" [High,Fix released] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 12:35 |
| openstack | Launchpad bug 1824243 in tripleo "duplicate for #1824241 cirros image missing from libvirt reproducer" [High,Fix released] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 12:35 |
| weshay | ya.. so ya.. I can see baseos hittng the same thing | 12:35 |
| rlandy | weshay: ack .. https://sf.hosted.upshift.rdu2.redhat.com/logs/15/165815/46/check/periodic-tripleo-ci-centos-7-bm_envD-3ctlr_1comp-featureset001-master/cfdca84/logs/undercloud/home/zuul/tempest.log.txt.gz#_2019-05-10_23_04_16 | 12:35 |
| rlandy | consistent | 12:35 |
| rlandy | so I must be missing some tempest setup | 12:35 |
| rlandy | Error reading manifest 1c3646705592e5dca1c55a29f25e5239e719c35f_85df50c7-updated-20190510205604 in 10.9.122.1:8787/tripleomaster/centos-binary-tempest: error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p>The requested URL /v2/tripleomaster/centos-binary-te | 12:36 |
| rlandy | mpest/manifests/1c3646705592e5dca1c55a29f25e5239e719c35f_85df50c7-updated-20190510205604 was not found on this server.</p>\n</body></html>\n"" | 12:36 |
| rlandy | ^^ last issue to close out baseos | 12:36 |
| arxcruz | rlandy:https://trello.com/c/AR3jrrTo/904-cixlp1819440tripleociproa-phase-1-failing-on-no-such-file-or-directory-home-stack-tempest | 12:37 |
| weshay | rlandy oh.. that's diff issue | 12:38 |
| rlandy | we can get on the undercloud and poke around | 12:39 |
| weshay | rlandy hrm.. and this is centos rdo deployment | 12:39 |
| rlandy | weshay: ack | 12:39 |
| weshay | rlandy tempest pulls the container after the deployment | 12:39 |
| rlandy | ok - and? | 12:39 |
| weshay | rlandy meh.. I guess it failed to pull | 12:40 |
| rlandy | weshay: consistently though | 12:41 |
| arxcruz | rlandy: https://review.opendev.org/#/c/644578/ this was the fix | 12:41 |
| * rlandy gets on the box and tries again manually | 12:41 | |
| arxcruz | for the tempest not found | 12:41 |
| weshay | rlandy run the tempest playbook from the undercloud | 12:41 |
| ykarel | rlandy, issue is tag(build) vs hash(being pulled) | 12:41 |
| ykarel | for tempest container | 12:41 |
| weshay | ya.. the container is getting updated | 12:42 |
| ykarel | current-tripleo-updated-20190510205604 vs 1c3646705592e5dca1c55a29f25e5239e719c35f_85df50c7-updated-20190510205604 | 12:42 |
| *** apetrich has quit IRC | 12:42 | |
| rlandy | ykarel: hmmm that's a good point - ovb is working somehow | 12:42 |
| weshay | rlandy same has? | 12:44 |
| weshay | hash | 12:44 |
| rlandy | weshay: check all the above | 12:44 |
| rlandy | checking | 12:44 |
| weshay | I think the tempest container + dlrn hash | 12:44 |
| weshay | does not exist on the undercloud registry | 12:44 |
| rlandy | we are definitely trying o use dlrn hash now | 12:44 |
| weshay | rlandy so somehow we are passing the update value in ovb | 12:45 |
| weshay | but not baremetal | 12:45 |
| weshay | this is a playbook / var issue | 12:45 |
| ykarel | so somewhere container_build_id is mismatched | 12:45 |
| weshay | aye | 12:45 |
| rlandy | arxcruz: the logs does show it has https://review.opendev.org/#/c/644578/3/roles/validate-tempest/tasks/tempest-results.yml fix | 12:45 |
| rlandy | same fs so it can't be there | 12:46 |
| rlandy | has to be in playbook or env settings | 12:46 |
| arxcruz | hmmm | 12:46 |
| arxcruz | rlandy:is that possible to log into the machine to see what's going on ? well, if wasn't able to pull the tempest container, then the file will not exist anyway | 12:47 |
| rlandy | arxcruz: so - yeah we can get on the machine | 12:47 |
| rlandy | just checking the ovb files to compare | 12:47 |
| *** apetrich has joined #oooq | 12:48 | |
| rlandy | arxcruz: weshay: ykarel: ovb has these settings bm is missing: | 12:53 |
| rlandy | # Use cirros cached image | 12:53 |
| rlandy | tempest_test_image_path: /opt/cache/files/cirros-0.3.5-x86_64-disk.img | 12:53 |
| rlandy | tempest_track_resources: false | 12:53 |
| ykarel | ^^ seems unrelated to issue though | 12:54 |
| rlandy | and container-wise: | 12:54 |
| rlandy | undercloud_docker_registry_mirror: "{{ lookup('env','NODEPOOL_DOCKER_REGISTRY_V2_PROXY') }}" | 12:54 |
| rlandy | dlrn_use_local_mirrors: true | 12:54 |
| weshay | rlandy let's look at the logs together | 12:54 |
| weshay | first find the value of container_build_id to ykarel's point | 12:54 |
| ykarel | also why current-tripleo is coming, as it's periodic job | 12:55 |
| ykarel | so it should be tripleo-ci-testing or the hash tripleo-ci-testing refers | 12:55 |
| weshay | ykarel bm is not hooked in yet | 12:56 |
| weshay | to the pipeline | 12:56 |
| *** rlandy_ has joined #oooq | 12:58 | |
| rlandy_ | sorry - was disconnected | 12:59 |
| *** dmellado_ has joined #oooq | 12:59 | |
| *** rlandy has quit IRC | 13:00 | |
| *** dmellado has quit IRC | 13:00 | |
| *** dmellado_ is now known as dmellado | 13:00 | |
| *** rlandy_ is now known as rlandy | 13:02 | |
| *** Goneri has joined #oooq | 13:02 | |
| ykarel | weshay, ack | 13:02 |
| *** dsneddon has quit IRC | 13:03 | |
| marios | SCRUM | 13:03 |
| marios | NOW | 13:03 |
| marios | JOIN | 13:03 |
| rfolco|ruck | marios, quiquell, sshnaidm, weshay, panda, rlandy, arxcruz, mwhahaha, rfolco, chkumar, zbr, kopecmartin | 13:04 |
| rfolco|ruck | ping scrum | 13:04 |
| rfolco|ruck | last call | 13:04 |
| marios | your luggage WILL be removed! | 13:05 |
| *** udesale has joined #oooq | 13:10 | |
| *** vinaykns has joined #oooq | 13:13 | |
| *** udesale has quit IRC | 13:18 | |
| *** udesale has joined #oooq | 13:18 | |
| *** aakarsh has joined #oooq | 13:43 | |
| *** jpena|lunch is now known as jpena | 13:43 | |
| rlandy | arxcruz; see this ovb log ... looks similar to me ... http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/b8fe4c8/logs/undercloud/home/zuul/tempest.log.txt.gz | 13:45 |
| weshay | rlandy hey | 13:45 |
| weshay | jump back on blue | 13:46 |
| rlandy | weshay: k - joining | 13:46 |
| rlandy | back on blue | 13:47 |
| panda | marios: rebased the upstream change | 13:47 |
| marios | panda: thanks will check again tomorrow's review run | 13:51 |
| weshay | musical blue https://bluejeans.com/u/whayutin | 13:51 |
| rlandy | going there | 13:53 |
| panda | marios: the what ? | 13:53 |
| marios | panda: wut | 13:53 |
| panda | marios: wat > | 13:56 |
| rlandy | http://logs.rdoproject.org/54/657554/4/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/e4b9494/logs/undercloud/home/zuul/tempest.log.txt.gz | 13:57 |
| rlandy | weshay: ^^ | 13:57 |
| marios | panda: que-wut | 13:57 |
| arxcruz | rlandy: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/defaults/main.yml#L28 | 14:00 |
| arxcruz | rlandy: sorry https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/defaults/main.yml#L35 | 14:03 |
| rfolco|ruck | rlandy, do you usually point browbeat bug to someone ? https://bugs.launchpad.net/tripleo/+bug/1828559 | 14:05 |
| openstack | Launchpad bug 1828559 in tripleo "fs053: Bad admin creds while checking rally deployment" [Critical,Triaged] | 14:05 |
| rlandy | rfolco|ruck: troy wilson | 14:05 |
| rfolco|ruck | rlandy, thanks | 14:06 |
| rfolco|ruck | rlandy, tcw? | 14:08 |
| rlandy | rfolco|ruck: ack | 14:08 |
| rfolco|ruck | k | 14:08 |
| rfolco|ruck | tcw, hey, fs053 job is consistently failing on bad admin creds error, can you please have a look and get any insights ? https://bugs.launchpad.net/tripleo/+bug/1828559 thanks. | 14:09 |
| openstack | Launchpad bug 1828559 in tripleo "fs053: Bad admin creds while checking rally deployment" [Critical,Triaged] | 14:09 |
| *** ratailor has joined #oooq | 14:10 | |
| marios | rlandy: rfolco|ruck weshay panda arxcruz kopecmartin chkumar|pto needs review if you have time thanks https://review.rdoproject.org/r/#/c/20277/ | 14:10 |
| panda | marios: I'll add it to my next sprint queue early morning 10 minutes routine.. | 14:11 |
| marios | panda: ack :D | 14:13 |
| weshay | panda raised the issue in prod chain, looks like we'll probably see some changes. not sure when | 14:13 |
| marios | speaking of next sprint... weshay i tried the manual step through today ... quickly fails for tripleoclient (beaker and quickvm) notes at https://tree.taiga.io/project/tripleo-ci-board/task/1040 | 14:14 |
| weshay | panda fyi https://trello.com/c/zWOq4gcK/51-rhos-release-for-rdo-trunk-repos-both-up-down-stream | 14:14 |
| marios | weshay: not sure what next steps are there... i'd expect its something a lot of people should be hitting if they're trying osp15/rhel8 deploys but i dind't find any related bz at https://bugzilla.redhat.com/buglist.cgi?quicksearch=%22requires%20openstack-tripleo-validations%2C%20but%20none%20of%20the%20providers%20can%20be%20installed%22 | 14:15 |
| marios | weshay: sounds like reldel issue (rpm missing deps?) | 14:15 |
| marios | weshay: anyway its a next sprint issue just fyi | 14:15 |
| *** Vorrtex has joined #oooq | 14:22 | |
| panda | weshay: do you want to add this to the that card ? https://code.engineering.redhat.com/gerrit/#/c/169235/ | 14:23 |
| panda | marios: do you plan to put those job ina project somewhere ? | 14:24 |
| *** ratailor has quit IRC | 14:24 | |
| panda | marios: nope they are bases. | 14:24 |
| ykarel | marios, seems you hitting issue due to rhel8-1 image | 14:36 |
| marios | ykarel: thanks you mean at https://tree.taiga.io/project/tripleo-ci-board/task/1040?kanban-status=1447274 | 14:37 |
| ykarel | marios, yes | 14:37 |
| marios | ykarel: grateful for any pointers you can add comment there ^^ if you feel like it :) | 14:37 |
| ykarel | can u try with rhel 8 image | 14:37 |
| ykarel | see bandini's mail too | 14:37 |
| *** skramaja has quit IRC | 14:37 | |
| marios | ykarel: ah ok i will check that (bandini email i mean) | 14:38 |
| marios | ykarel: thanks | 14:38 |
| ykarel | okk it's send to rhos-infrared | 14:38 |
| ykarel | marios, http://post-office.corp.redhat.com/archives/rhos-infrared/2019-May/msg00046.html | 14:39 |
| ykarel | his failing at tooling but u are hitting same | 14:39 |
| ykarel | let me check from which repo ansible comes | 14:40 |
| ykarel | can u quickly check, output of yum repolist | 14:41 |
| ykarel | seems some repo is missing | 14:41 |
| marios | ykarel: thank you | 14:47 |
| marios | noted | 14:47 |
| *** quiquell|lunch is now known as quiquell|off | 14:49 | |
| *** dsneddon has joined #oooq | 15:00 | |
| *** dsneddon has quit IRC | 15:05 | |
| * marios shutdown sequqnce | 15:13 | |
| *** ykarel is now known as ykarel|away | 15:19 | |
| *** tcw has quit IRC | 15:29 | |
| *** yolanda_ has quit IRC | 15:34 | |
| *** yolanda_ has joined #oooq | 15:37 | |
| *** tcw has joined #oooq | 15:38 | |
| rlandy | weshay: arxcruz: so here is the confusion ... https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/container-prep/tasks/main.yml#L15 | 15:49 |
| rlandy | bm tests are resolving to tripleo-ci-testing | 15:49 |
| rlandy | and ovb periodic is resolving to the dlrn_hash_tag | 15:50 |
| weshay | ah | 15:50 |
| rlandy | 2019-05-13 02:20:50.429242 | primary | TASK [container-prep : echo container_build_id] ******************************** | 15:51 |
| rlandy | 2019-05-13 02:20:50.441090 | primary | Monday 13 May 2019 02:20:50 +0000 (0:00:00.057) 0:00:21.384 ************ | 15:51 |
| rlandy | 2019-05-13 02:20:50.526962 | primary | ok: [undercloud] => { | 15:51 |
| rlandy | 2019-05-13 02:20:50.527104 | primary | "container_build_id": "59c27442d8e88c45b48e2cb98e163356e63dd091_0129a5fa" | 15:51 |
| rlandy | 2019-05-13 02:20:50.527125 | primary | } | 15:51 |
| rlandy | ^^ ovb | 15:51 |
| *** ykarel|away has quit IRC | 15:51 | |
| *** marios has quit IRC | 15:52 | |
| rlandy | 2019-05-13 15:18:18.158041 | primary | TASK [container-prep : echo container_build_id] ******************************** | 15:53 |
| rlandy | 2019-05-13 15:18:18.195191 | primary | Monday 13 May 2019 15:18:18 +0000 (0:00:00.185) 0:00:01.308 ************ | 15:53 |
| rlandy | 2019-05-13 15:18:18.269443 | primary | ok: [undercloud] => { | 15:53 |
| rlandy | 2019-05-13 15:18:18.269541 | primary | "container_build_id": "tripleo-ci-testing" | 15:53 |
| rlandy | 2019-05-13 15:18:18.269556 | primary | } | 15:53 |
| rlandy | ^^ bm | 15:53 |
| rlandy | the container image tag of the two is different | 15:53 |
| rlandy | but if you look at tempest ... | 15:53 |
| rlandy | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/defaults/main.yml#L37 | 15:54 |
| rlandy | {{ container_build_id }}{{ update_containers_append_tag }} | 15:55 |
| rlandy | so it should follow container_build_id | 15:55 |
| rlandy | somewhere that may get updated | 15:55 |
| rlandy | watching what happens with overcloud container prep | 15:56 |
| rlandy | but with prepare_overcloud_containers that gets reset ... | 15:58 |
| rlandy | 2019-05-10 21:40:15.418366 | primary | TASK [container-prep : echo container_build_id] ******************************** | 15:58 |
| rlandy | 2019-05-10 21:40:15.455711 | primary | Friday 10 May 2019 21:40:15 +0000 (0:00:00.099) 0:00:34.679 ************ | 15:58 |
| rlandy | 2019-05-10 21:40:15.537516 | primary | ok: [undercloud] => { | 15:58 |
| rlandy | 2019-05-10 21:40:15.537638 | primary | "container_build_id": "1c3646705592e5dca1c55a29f25e5239e719c35f_85df50c7" | 15:58 |
| rlandy | 2019-05-10 21:40:15.537651 | primary | } | 15:58 |
| weshay | rlandy are you should prepare_overcloud_containers is in play? | 16:01 |
| * weshay sees if I have your log open | 16:01 | |
| rlandy | weshay: ack | 16:01 |
| rlandy | somewhere between undercloud and overcloud prep containers container_build_id is being reset | 16:01 |
| weshay | rlandy https://sf.hosted.upshift.rdu2.redhat.com/logs/15/165815/46/check/periodic-tripleo-ci-centos-7-bm_envD-3ctlr_1comp-featureset001-master/cfdca84/job-output.txt.gz#_2019-05-10_21_40_15_537638 | 16:03 |
| weshay | rlandy I bet that step is skipped in ovb | 16:03 |
| rlandy | weshay: well in ovb - both times that resolves to the dlrn hash | 16:03 |
| weshay | rlandy now that tripleo-common handles all that.. we don't run it for the overcloud | 16:04 |
| rlandy | in bm, undercloud resolves to tripleo-ci-testing | 16:04 |
| rlandy | overcloud resolves to hash | 16:04 |
| weshay | rlandy you want another pair of eyes? | 16:04 |
| weshay | ideally we're using the "updated" tag / container | 16:05 |
| rlandy | weshay: looking a bit more - will ping for help if I don;t track the exact line | 16:05 |
| rlandy | I see now the diff | 16:05 |
| weshay | k cool | 16:05 |
| *** aakarsh has quit IRC | 16:13 | |
| *** bogdando has quit IRC | 16:14 | |
| rlandy | ovb is correct - it does the process twice - once undercloud, once overcloud but it's consistent | 16:15 |
| rlandy | bm does the process twice but the answer is diff undercloud and overcloud | 16:15 |
| *** ykarel has joined #oooq | 16:15 | |
| weshay | rlandy ya.. but I don't think we need to call it twice anymore | 16:17 |
| rlandy | we still do according to logs | 16:17 |
| weshay | at least in any containerized undercloud deployment | 16:17 |
| weshay | rfolco|ruck need help w/ anything? | 16:19 |
| rfolco|ruck | weshay, no, I am working on container build check job improvements and then will check browbeat fs053 | 16:19 |
| rfolco|ruck | weshay, unless you have any other priorities | 16:19 |
| weshay | k | 16:19 |
| weshay | rfolco|ruck only other thing would be the ansible bump | 16:20 |
| *** tesseract has quit IRC | 16:20 | |
| rfolco|ruck | weshay, right, need to find a way to replace tags.... this is more complicated | 16:20 |
| weshay | replace tags just for include_tasks | 16:21 |
| weshay | not generally | 16:21 |
| rfolco|ruck | yes | 16:21 |
| weshay | rfolco|ruck vars, set facts can replace tags | 16:22 |
| weshay | w/ coupled w/ when | 16:22 |
| *** dtantsur is now known as dtantsur|afk | 16:23 | |
| rfolco|ruck | weshay, ok I'll test something like that in a bit | 16:24 |
| *** ykarel is now known as ykarel|away | 16:24 | |
| rlandy | ha - it's the order when repo_setup is called | 16:27 |
| weshay | rlandy repo_setup resets the var probably right? | 16:29 |
| rlandy | container_prep is called twice | 16:30 |
| rlandy | weshay: ack | 16:30 |
| rlandy | it sets it to dlrn_hash | 16:30 |
| weshay | aye | 16:30 |
| rlandy | that's the diff | 16:30 |
| rlandy | if you look at ovb - it runs that three times | 16:30 |
| rlandy | setup_undercloud, deploy_undercloud and overcloud | 16:30 |
| rlandy | bm does that twice | 16:30 |
| * rlandy checks why | 16:31 | |
| rfolco|ruck | weshay, what if I just move the tags into the task that is being included with include_task: tasks.yaml... and in tasks.yaml I set the tags | 16:31 |
| weshay | I'd have to see it to understand tbh | 16:33 |
| rfolco|ruck | weshay, ok, let me see if this works first, its just a thought, I might be neglecting something. | 16:33 |
| rfolco|ruck | weshay, also, please see this one when you get a chance https://review.opendev.org/#/c/658449/ | 16:34 |
| rlandy | ok - one more playbook | 16:40 |
| *** matbu has quit IRC | 16:40 | |
| weshay | rfolco|ruck I've opened two bugs on the queens promotion.. | 16:45 |
| rfolco|ruck | weshay, ack | 16:51 |
| *** holser_ has quit IRC | 16:52 | |
| *** matbu has joined #oooq | 16:57 | |
| *** udesale has quit IRC | 16:59 | |
| *** dsneddon has joined #oooq | 17:01 | |
| *** dsneddon has quit IRC | 17:06 | |
| *** jpena is now known as jpena|off | 17:08 | |
| *** aakarsh has joined #oooq | 17:14 | |
| *** jbadiapa has quit IRC | 17:17 | |
| *** kopecmartin is now known as kopecmartin|off | 17:18 | |
| *** dsneddon has joined #oooq | 17:20 | |
| *** tcw has quit IRC | 17:23 | |
| *** tcw has joined #oooq | 17:24 | |
| *** ykarel|away has quit IRC | 17:32 | |
| *** derekh has quit IRC | 17:37 | |
| rlandy | ha - better | 17:53 |
| *** Vorrtex has quit IRC | 18:24 | |
| *** amoralej is now known as amoralej|off | 18:33 | |
| weshay | rfolco|ruck we need to get rocky standalone in a zuul template or running against tht, tc, python-tripleoclient | 18:41 |
| weshay | rfolco|ruck perhaps I'll document in a bug? | 18:42 |
| weshay | https://review.opendev.org/#/q/topic:rocky-tempest+(status:open+OR+status:merged) | 18:42 |
| rfolco|ruck | weshay, looks like arxcruz is doing it here.. https://review.opendev.org/#/c/657589/2/zuul.d/standalone-jobs.yaml | 18:43 |
| rfolco|ruck | weshay, do you need me to add rocky standalone job separated ? | 18:43 |
| weshay | rfolco|ruck that doesn't get the job done though | 18:43 |
| rfolco|ruck | L10 branches: ^(?!stable/(newton|ocata|pike|queens|rocky)).*$ | 18:44 |
| weshay | the rocky version != master and the other branches | 18:44 |
| weshay | rfolco|ruck right.. ! rocky | 18:44 |
| weshay | it will fail on rocky | 18:44 |
| rfolco|ruck | k k | 18:44 |
| weshay | name: tripleo-ci-centos-7-standalone-rocky | 18:44 |
| weshay | works on rocky | 18:44 |
| weshay | rfolco|ruck I think we could patch https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/standalone-jobs.yaml, add the rocky version to the template | 18:46 |
| weshay | and we'd be done | 18:46 |
| rfolco|ruck | weshay, yep, its already there on https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/standalone-jobs.yaml#L29 | 18:47 |
| weshay | vs.. defining it on each repo as I tested | 18:47 |
| weshay | rfolco|ruck ya.. I guess that has the wrong parent | 18:48 |
| weshay | needs to parent to: line 41 on https://review.opendev.org/#/c/657589/2/zuul.d/standalone-jobs.yaml | 18:48 |
| weshay | afaict | 18:48 |
| weshay | rlandy please review https://review.opendev.org/#/c/657589/ in your time w/ context here ^ | 18:49 |
| weshay | rfolco|ruck rlandy noting rocky has use_os_tempest: false | 18:50 |
| rfolco|ruck | weshay, I am a bit confused yet, the one you pointed out is branchless... runs on master | 18:50 |
| weshay | that's the diff | 18:50 |
| rlandy | looking | 18:51 |
| rfolco|ruck | I think you want a real stable/branch job that runs against rocky and set os_tempest as off | 18:51 |
| weshay | that is what arxcruz's commit does rfolco|ruck | 18:51 |
| weshay | rfolco|ruck it's just not part of a template | 18:51 |
| weshay | because the name changed | 18:51 |
| rfolco|ruck | weshay, ok, let me check the layout | 18:52 |
| rlandy | interesting because we should juts be able to override the values | 18:53 |
| weshay | rlandy for just rocky? | 18:53 |
| rlandy | weshay: yes - featureset_ovveride | 18:54 |
| rlandy | guess it's a matter of style | 18:54 |
| rlandy | you would gain a little less duplication | 18:54 |
| weshay | rlandy I don't think that's right.. | 18:55 |
| weshay | it's already using a featureset override | 18:55 |
| weshay | and turning os_tempest on for stein, train | 18:55 |
| weshay | can we use the same job name and turn it off for stable/rocky? | 18:56 |
| rlandy | weshay: yep - that's how all override jobs work - parent job has a var - child job changes it ... what I am not sure about is how it works with featureset_override | 18:56 |
| rlandy | I'd have to think about this | 18:56 |
| rlandy | so you woudl have a rocky job | 18:56 |
| rlandy | correct | 18:56 |
| weshay | so the only thing that needs to change there... | 18:57 |
| weshay | would be line 41 | 18:57 |
| rlandy | but parent off tripleo-ci-centos-7-standalone | 18:57 |
| weshay | rename it to tripleo-ci-centos-7-standalone | 18:57 |
| rlandy | so you save some stuff in duplication | 18:57 |
| rfolco|ruck | there is no template for standalone | 18:57 |
| weshay | rlandy and you have have a parent/child w/ the same name no problem? | 18:57 |
| rfolco|ruck | only for standalone-scenarios | 18:57 |
| rlandy | child is called tripleo-ci-centos-7-standalone-rocky | 18:58 |
| weshay | rlandy currently yes | 18:58 |
| rlandy | weshay: the way it is would work | 18:58 |
| rlandy | but wondering why we just don't override only what we need | 18:59 |
| rlandy | maybe that is the only way to do this cleanly? | 18:59 |
| rfolco|ruck | I agree with you rlandy | 18:59 |
| * rlandy is thinking out a loud | 18:59 | |
| * rlandy thinks how featureset_override would work | 18:59 | |
| weshay | I don't think I've seen a parent/child w/ the same name | 18:59 |
| rfolco|ruck | branches is going to be overriden | 19:00 |
| rfolco|ruck | so you can parent off tripleo-ci-centos-7-standalone | 19:00 |
| rlandy | why would parent/child have the same name? | 19:00 |
| rfolco|ruck | and just override os_tempest as off | 19:00 |
| rlandy | ^^ not quite | 19:00 |
| rlandy | there is mote | 19:00 |
| rlandy | more to it than thar | 19:00 |
| rlandy | that | 19:00 |
| weshay | ya.. I think the other vars don't need to be overriden | 19:00 |
| weshay | just the | 19:00 |
| weshay | branches: | 19:00 |
| weshay | and featureset_override: use_os_tempest | 19:01 |
| rlandy | you need run_tempest to be true | 19:01 |
| rfolco|ruck | yes branches and os_tempest | 19:01 |
| rfolco|ruck | what else? | 19:01 |
| *** Vorrtex has joined #oooq | 19:01 | |
| rlandy | so ... using a pastebin ... | 19:01 |
| rfolco|ruck | ok rlandy | 19:01 |
| weshay | the other vars are not the crux of the issue atm | 19:01 |
| rfolco|ruck | unlessss.... he does not want to run the tests in whitelist | 19:02 |
| weshay | rfolco|ruck funny enough the standalone job ( non-scenario ) is in the multinode template https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/multinode-jobs.yaml#L19 | 19:03 |
| rfolco|ruck | lolz | 19:03 |
| weshay | so we could add standalone rocky to that template | 19:03 |
| weshay | and have this triggering the way we want | 19:03 |
| rlandy | https://paste.fedoraproject.org/paste/5Uz-iUTt7jFo3uQimDlzaQ | 19:03 |
| rfolco|ruck | yes just like we did for upgrade-stein for example | 19:04 |
| rfolco|ruck | rlandy, looking | 19:04 |
| rlandy | idk - we would have to test that | 19:04 |
| weshay | rlandy ya.. that is fine | 19:04 |
| rlandy | devil is in the details on these var definition | 19:04 |
| weshay | it's the triggering that is important atm | 19:04 |
| rlandy | weshay: I'll comment for arxcruz to review | 19:04 |
| rlandy | he can reject it | 19:04 |
| weshay | rlandy if it has a new name, it only triggers against tripleo-ci | 19:04 |
| weshay | you guys are getting hung up on the wrong thing | 19:05 |
| weshay | but anyhooo | 19:05 |
| rlandy | weshay: unless we add it to templates | 19:05 |
| rlandy | weshay: what is the right thing to hang up on? | 19:06 |
| rfolco|ruck | weshay, just trying to avoid duplication | 19:06 |
| rfolco|ruck | we understood you want to run against triple oprojects (template) | 19:06 |
| weshay | rfolco|ruck doesn't matter if your code is clean if the job doesn't trigger :) | 19:06 |
| rfolco|ruck | weshay, we are on the same page, I think rlandy and I were optimizing the parenting thing | 19:07 |
| * weshay updated the review | 19:07 | |
| rlandy | triggering is a diff matter | 19:07 |
| rlandy | we need to add this to templates | 19:08 |
| weshay | I think this will work.. /me runs w/ a test patch https://review.opendev.org/#/c/657589/3/zuul.d/multinode-jobs.yaml | 19:08 |
| weshay | k.. test review up | 19:10 |
| rlandy | weshay: ack that will work | 19:10 |
| rlandy | just thought we were reviewing the structure of the parenting before | 19:11 |
| weshay | thanks | 19:11 |
| weshay | rlandy while I have you distracted https://review.opendev.org/#/c/658449/ | 19:16 |
| rlandy | woohoo ... set: | 19:16 |
| rlandy | tag: "5722f5f9947dbd2dabd76f60bf75203e85dbe145_04eed6 | 19:16 |
| rlandy | looking | 19:16 |
| weshay | good :) | 19:16 |
| weshay | if that's the hash you are and the droids are looking for | 19:17 |
| weshay | https://review.opendev.org/#/c/645357 | 19:22 |
| rlandy | ack - how did we forget that | 19:27 |
| rlandy | or not notice it before? | 19:27 |
| weshay | rlandy I don't think our team uses quickstart.sh much | 19:28 |
| weshay | and centos.ci sucks | 19:28 |
| weshay | and we haven't gotten very far w/ molecule | 19:28 |
| weshay | I can go on | 19:29 |
| rlandy | no need | 19:34 |
| rlandy | I get it | 19:34 |
| weshay | rlandy https://review.rdoproject.org/r/#/c/20277/ | 19:39 |
| rlandy | weshay: ^^ ok to merge that? | 19:43 |
| weshay | rlandy I *think* so, I am realizing my zuul skills are duuul. afaict that doesn't actually add any jobs to the promotion pipelines | 19:44 |
| weshay | just setting up for it | 19:44 |
| rlandy | weshay: ack | 19:44 |
| rlandy | just job definition | 19:44 |
| weshay | ack to merge then | 19:45 |
| rlandy | to add jobs to promotion pipeline, you would edit https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/tripleo.yaml | 19:46 |
| rlandy | done | 19:46 |
| arxcruz | rlandy:? | 19:46 |
| arxcruz | which review ? | 19:46 |
| *** dtrainor has quit IRC | 19:47 | |
| rlandy | arxcruz: sorry what? problem solved for bm | 19:48 |
| *** dtrainor has joined #oooq | 19:49 | |
| weshay | arxcruz fyi.. I've updated https://review.rdoproject.org/r/#/c/20590/ | 19:49 |
| weshay | arxcruz only thing I couldn't do was get that ssh command on multple lines to save my life | 19:49 |
| arxcruz | weshay: just saw it :) | 19:50 |
| *** panda has quit IRC | 20:19 | |
| *** panda has joined #oooq | 20:21 | |
| rlandy | weshay: can I bother you for unrelated rhos-14 error? | 20:35 |
| weshay | rlandy aye | 20:36 |
| weshay | rlandy deployment failure, in the stack creation? | 20:36 |
| rlandy | weshay: k ... been chasing this one for a while - works fine in rocky | 20:36 |
| rlandy | weshay: so here is the error ... https://sf.hosted.upshift.rdu2.redhat.com/logs/periodic/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/tripleo-ci-rhel-7-standalone-rhos-14/02d6eff/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-05-12_20_14_22 | 20:37 |
| rlandy | and here is the fix ( in rocky branch) | 20:37 |
| rlandy | https://github.com/openstack/tripleo-common/blob/stable/rocky/container-images/overcloud_containers.yaml.j2#L111 | 20:37 |
| rlandy | weshay: ^^ that should have been fixed in tripleo-common 9.3 | 20:37 |
| rlandy | rhos-14 has triple-common 9.5 | 20:38 |
| rlandy | yet the same error | 20:38 |
| * rlandy is stumped as to how | 20:38 | |
| weshay | rlandy looking and poking around | 20:39 |
| weshay | few | 20:39 |
| rlandy | sure - no rush - been borked for weeks | 20:40 |
| *** Vorrtex has quit IRC | 20:42 | |
| rlandy | ha - browbeat job is preping images for overcloud :) | 20:48 |
| rlandy | progress | 20:48 |
| *** Vorrtex has joined #oooq | 20:50 | |
| rlandy | ugh ... noo!!! now bm jobs runs too long | 20:58 |
| *** derekh has joined #oooq | 20:59 | |
| weshay | rlandy pretty sure we need to see the hieradata for this | 21:02 |
| rlandy | weshay: the rhos-14 stuff? | 21:02 |
| weshay | ya | 21:03 |
| weshay | rlandy I can't tell what variables the standalone has.. /me walking though http://codesearch.openstack.org/?q=DockerCinderConfigImage&i=nope&files=&repos= | 21:03 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/periodic/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/tripleo-ci-rhel-7-standalone-rhos-14/02d6eff/logs/undercloud/home/zuul/containers-prepare-parameters.yaml.txt.gz | 21:03 |
| rlandy | ^^ other than that,no | 21:03 |
| weshay | rlandy so.. | 21:07 |
| weshay | I think we're lacking some logging in the standalone | 21:07 |
| weshay | so I can bring up a rocky version if you bring up a osp-14 version | 21:07 |
| weshay | I've been looking through the cherry-picks to code.eng don't see much there | 21:08 |
| rlandy | weshay: I have the rocky version logs | 21:08 |
| weshay | we need more than what's logged is my point | 21:08 |
| rlandy | oh you mean a full run | 21:08 |
| weshay | we need two live systems | 21:08 |
| weshay | rocky and osp-14 | 21:08 |
| rlandy | ok. let me repovision my testbox | 21:08 |
| weshay | rlandy why do you reprovision? | 21:08 |
| weshay | just curious | 21:09 |
| rlandy | because it has rhel-8 on it now | 21:09 |
| rlandy | for reproducer testing | 21:09 |
| weshay | aye | 21:09 |
| weshay | rlandy ok.. so I'll bring up rocky, you bring up 14 | 21:10 |
| weshay | I'll find the missing bit's we should log, and if we're lucky we'll see where osp is missing a thing | 21:10 |
| rlandy | rel eng claims all the patches I asked about are there | 21:11 |
| weshay | rlandy I wonder if this is the issue https://review.opendev.org/#/c/644912/1/container-images/overcloud_containers.yaml.j2 | 21:11 |
| rlandy | and a basic file compare gives me nothing | 21:11 |
| weshay | rlandy how much trouble would it be to throw up a fs010 containers-multinode on osp-14 | 21:12 |
| weshay | there were some change to rocky to get the standalone going | 21:12 |
| rlandy | weshay: creating the job is the easy part - keeping it passing takes resources :) | 21:13 |
| weshay | rlandy ya.. if keeping osp running w/ the past phase 2 tag on it | 21:13 |
| weshay | becomes a burden we need to look into why | 21:13 |
| rlandy | weshay: ok - so we could lock the tag | 21:14 |
| rlandy | but I think that is besides the point | 21:14 |
| weshay | no no | 21:14 |
| rlandy | weshay: this is really the first major issue, I could not debug easily | 21:14 |
| weshay | it should always pass minus some really weird exceptions | 21:14 |
| rlandy | mostly, it's been fine | 21:14 |
| weshay | standalone for osp-14 has been mostly fine? | 21:14 |
| weshay | just curious | 21:14 |
| weshay | because if that's the case I'm going down the wrong path looking at a couple patches here | 21:15 |
| rlandy | yeah - before this error | 21:15 |
| rlandy | which arrived in a puddle end of april | 21:15 |
| *** jtomasek has quit IRC | 21:17 | |
| *** aakarsh has quit IRC | 21:37 | |
| *** Goneri has quit IRC | 21:38 | |
| *** aakarsh has joined #oooq | 22:03 | |
| *** ksambor has quit IRC | 22:06 | |
| *** vinaykns has quit IRC | 22:41 | |
| *** saneax has joined #oooq | 23:05 | |
| *** rlandy is now known as rlandy|bbl | 23:25 | |
| *** derekh has quit IRC | 23:30 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!