| *** agopi has quit IRC | 00:34 | |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ (1 more message) | 01:00 |
|---|---|---|
| *** rnoriega has quit IRC | 01:02 | |
| *** pliu has quit IRC | 01:04 | |
| *** lhinds has quit IRC | 01:04 | |
| *** faceman has quit IRC | 01:05 | |
| *** agopi has joined #oooq | 01:12 | |
| *** pliu has joined #oooq | 01:16 | |
| *** rnoriega has joined #oooq | 01:16 | |
| *** faceman has joined #oooq | 01:17 | |
| *** lhinds has joined #oooq | 01:19 | |
| *** rlandy|bbl has quit IRC | 02:30 | |
| *** skramaja has joined #oooq | 02:46 | |
| *** skramaja has quit IRC | 02:51 | |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ (1 more message) | 03:00 |
| *** sshnaidm|bbl has quit IRC | 03:16 | |
| *** sshnaidm|bbl has joined #oooq | 03:29 | |
| *** udesale has joined #oooq | 03:49 | |
| *** ykarel|away has joined #oooq | 03:56 | |
| *** ykarel|away is now known as ykarel | 04:00 | |
| *** ykarel is now known as ykarel|afk | 04:31 | |
| *** ykarel|afk has quit IRC | 04:35 | |
| *** kopecmartin has joined #oooq | 04:56 | |
| *** saneax has joined #oooq | 04:56 | |
| *** sanjayu_ has joined #oooq | 04:58 | |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ (1 more message) | 05:00 |
| *** saneax has quit IRC | 05:01 | |
| *** links has joined #oooq | 05:03 | |
| *** ykarel|afk has joined #oooq | 05:03 | |
| *** hamzy has quit IRC | 05:17 | |
| *** ratailor has joined #oooq | 05:18 | |
| *** gvrangan has joined #oooq | 05:30 | |
| *** hamzy has joined #oooq | 05:40 | |
| *** holser_ has joined #oooq | 05:44 | |
| *** ykarel|afk is now known as ykarel | 05:44 | |
| chkumar|ruck | %gatestatus | 05:46 |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ (1 more message) | 05:46 |
| *** anande has joined #oooq | 05:55 | |
| *** anande has quit IRC | 05:57 | |
| *** brault has quit IRC | 05:57 | |
| *** anande has joined #oooq | 05:58 | |
| *** skramaja has joined #oooq | 05:58 | |
| *** jtomasek has quit IRC | 06:01 | |
| *** quiquell has joined #oooq | 06:16 | |
| *** florianf has joined #oooq | 06:25 | |
| *** hamzy has quit IRC | 06:27 | |
| *** hamzy_ has joined #oooq | 06:27 | |
| *** brault has joined #oooq | 06:31 | |
| *** udesale has quit IRC | 06:40 | |
| *** udesale has joined #oooq | 06:42 | |
| *** jfrancoa has joined #oooq | 06:44 | |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ (1 more message) | 07:00 |
| *** ccamacho has joined #oooq | 07:02 | |
| *** bogdando has joined #oooq | 07:03 | |
| *** tesseract has joined #oooq | 07:12 | |
| *** gkadam has joined #oooq | 07:20 | |
| *** sanjayu__ has joined #oooq | 07:25 | |
| *** sanjayu_ has quit IRC | 07:28 | |
| *** amoralej|off is now known as amoralej | 07:34 | |
| *** tosky has joined #oooq | 07:37 | |
| *** holser_ has quit IRC | 07:43 | |
| *** ykarel is now known as ykarel|lunch | 07:49 | |
| *** anande has quit IRC | 07:57 | |
| *** holser_ has joined #oooq | 07:57 | |
| *** sanjayu_ has joined #oooq | 08:25 | |
| *** sanjayu__ has quit IRC | 08:27 | |
| quiquell | chkumar|ruck: Do you know is this is related to any bug ? http://logs.openstack.org/62/578462/11/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/abedb95/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz#_2018-07-10_13_52_24 | 08:31 |
| chkumar|ruck | quiquell: not seen this issue, | 08:33 |
| quiquell | chkumar|ruck: Ok, just confirmed that it's legit | 08:34 |
| *** panda|off is now known as panda | 08:40 | |
| quiquell | panda: Good morning | 08:46 |
| quiquell | panda: Do you have some time | 08:46 |
| panda | quiquell: for you, never! | 08:46 |
| panda | quiquell: 'sup ? | 08:46 |
| *** quiquell is now known as alfread | 08:47 | |
| alfread | And for me ? | 08:47 |
| *** alfread is now known as quiquell | 08:47 | |
| *** ykarel|lunch is now known as ykarel | 08:47 | |
| quiquell | panda: just to understand the nodepool thing, maybe you go bj | 08:48 |
| panda | I like to go bj | 08:49 |
| quiquell | panda: Very fashionable | 08:49 |
| *** gvrangan has quit IRC | 08:50 | |
| *** jtomasek has joined #oooq | 08:50 | |
| panda | quiquell: I went bj on my cj | 08:50 |
| panda | quiquell: but I'm wearing my pj | 08:51 |
| quiquell | panda: bpj would be just perfect | 08:51 |
| panda | call me lil' panda | 08:51 |
| quiquell | ok | 08:51 |
| quiquell | panda: I am at your room | 08:53 |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ (1 more message) | 09:00 |
| *** gvrangan has joined #oooq | 09:02 | |
| *** gvrangan has quit IRC | 09:03 | |
| *** gvrangan has joined #oooq | 09:04 | |
| *** gvrangan has quit IRC | 09:05 | |
| *** gvrangan has joined #oooq | 09:06 | |
| *** gvrangan has quit IRC | 09:07 | |
| *** gvrangan has joined #oooq | 09:08 | |
| *** Goneri has joined #oooq | 09:09 | |
| *** gvrangan_odl has joined #oooq | 09:39 | |
| *** gvrangan_odl has quit IRC | 09:40 | |
| *** gvrangan_odl has joined #oooq | 09:41 | |
| *** gvrangan has quit IRC | 09:41 | |
| *** gvrangan_odl has quit IRC | 09:42 | |
| *** gvrangan_odl has joined #oooq | 09:43 | |
| *** gvrangan_odl has quit IRC | 09:44 | |
| *** gvrangan_odl has joined #oooq | 09:44 | |
| *** sshnaidm|bbl is now known as sshnaidm|rover | 09:45 | |
| quiquell | panda: I don't think oooq_common_functions or common_functions need to be jinja templates | 10:37 |
| quiquell | panda: Don't see the benefits | 10:37 |
| quiquell | panda: They will just be replaced in case we ansiblelize it | 10:38 |
| panda | quiquell: replaced with what ? | 10:40 |
| panda | quiquell: do they use environment variables that can be passed by zuul ? | 10:40 |
| quiquell | panda: Don't think so | 10:40 |
| panda | quiquell: or they are just functions with arguments ? | 10:40 |
| quiquell | panda: the function we use don't benefict from the zuul variables | 10:41 |
| quiquell | panda: Looking here http://logs.openstack.org/31/581331/6/check/tripleo-ci-centos-7-undercloud-containers/7497cd0/job-output.txt.gz | 10:41 |
| panda | quiquell: good, two down, three bash script to go | 10:41 |
| quiquell | panda: more than that | 10:44 |
| quiquell | panda: we use a function at common, that is only use by toci | 10:44 |
| quiquell | panda: we can move it to oooq_common_functions.sh | 10:44 |
| quiquell | echo_vars_to_deploy_env_oooq | 10:44 |
| quiquell | and we remove the common_functions.sh dependency | 10:44 |
| panda | quiquell: I hope that function goes completely away after replacing the bootstrap part | 10:45 |
| panda | ah, hm | 10:45 |
| panda | maybe not that one | 10:45 |
| panda | quiquell: this should have been done in the previous sprint :( but yeah, I like that, move stuff around and remove dependencies | 10:46 |
| quiquell | panda: in the oooq_common_functions we use 'is_featureset' and run_with_timeout | 10:46 |
| quiquell | collect_logs | 10:47 |
| quiquell | panda: We do a patch for this so we can merge it | 10:47 |
| panda | quiquell: ep | 10:47 |
| *** Guest67898 is now known as rook | 10:48 | |
| quiquell | panda: https://review.openstack.org/#/c/581668/ | 10:54 |
| hubbot` | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ (1 more message) | 11:00 |
| quiquell | panda: We can just do the change in the jinja variable, so we don't need to merge that patch | 11:05 |
| quiquell | in the jinja templtes | 11:06 |
| panda | quiquell: also, as I suspected, that funtion is mainly useful for the node bootstrap | 11:07 |
| panda | quiquell: we have a card to get rid of everything there and its dependencies | 11:07 |
| quiquell | panda: also the is_featureset will be removed soon | 11:07 |
| quiquell | panda: what card ? | 11:08 |
| panda | quiquell: I think we can just remove that function as a consequence of translating bootstrap to ansible | 11:08 |
| *** atoth has quit IRC | 11:09 | |
| quiquell | so we just don't do a jinja template of common_functions and remove the dependency | 11:09 |
| quiquell | when we finis the bootstrap card ? | 11:09 |
| panda | quiquell: I would think so | 11:09 |
| panda | quiquell: deploy.env is really used just by tripleo.sh | 11:10 |
| quiquell | ok cool so we just have the is_featurset, run_timeout and collect_logs | 11:10 |
| quiquell | and is_featureset will be removed later on | 11:10 |
| panda | marios: ^ | 11:12 |
| panda | quiquell: I think we can merge your patch anyway | 11:13 |
| quiquell | panda: Ok, let's see if it doesn't breaks anything | 11:14 |
| panda | quiquell: I would even remoe the file | 11:14 |
| panda | quiquell: so we test also if removing it breaks anything | 11:14 |
| quiquell | panda: common_functions.sh is used | 11:14 |
| quiquell | panda: tripleo.sh and deploy.sh... maybe not that much | 11:15 |
| panda | quiquell: common_functions.sh is used ? where ? | 11:17 |
| quiquell | panda: tripleo.sh and deploy.sh | 11:18 |
| panda | ah well in tripleo.sh | 11:18 |
| panda | yea, we can't remove it | 11:18 |
| quiquell | panda: Sure we don't use deploy.sh at downstream ? | 11:18 |
| quiquell | panda: Missed one subnodes_scp_deploy_env | 11:22 |
| panda | quiquell:if we do, we should drop this sprint topic and and move everything to oooq downstream | 11:22 |
| quiquell | used only at toci too | 11:22 |
| quiquell | panda: ok will try to remove | 11:22 |
| panda | quiquell: no, you were right | 11:23 |
| panda | quiquell: nevermind my last phrase | 11:24 |
| panda | aaaaaaaa | 11:24 |
| quiquell | panda: What do you mean ? | 11:25 |
| panda | quiquell: we can't remove common_functions.sh yet | 11:26 |
| quiquell | panda: what is missing ? | 11:26 |
| panda | quiquell: is used by tripleo.sh | 11:26 |
| panda | quiquell: and we use tripleo.sh for bootstrapping | 11:26 |
| panda | quiquell: so ok, for now we can just move the function and remove the sourcing | 11:26 |
| panda | but not the file | 11:27 |
| quiquell | panda: yep | 11:27 |
| quiquell | panda: We will se in the future | 11:27 |
| *** udesale has quit IRC | 11:29 | |
| marios | panda: but we will remove tripleo.sh | 11:32 |
| marios | panda: i mean that is a card for this sprint remove the bootstraping fro tripleo.sh | 11:32 |
| panda | marios: at the end of the sprint, yes | 11:32 |
| panda | marios: not right at this moment | 11:32 |
| marios | (sorry didn't follow the rest of the conversation) | 11:32 |
| quiquell | panda: We cannot remove tripleo.sh | 11:33 |
| marios | panda: k i'm just finishing scoping it out as first iteration gonna update my review in a moment and we can discuss hte card on thursday? | 11:33 |
| quiquell | panda, marios: We have the legacy jobs running the legacy toci_gate_test.sh | 11:33 |
| marios | panda: not sure what 'later in the sprint' means though given the rigid design/implement/test structure ... maybe you mean 'next sprint' ;) | 11:34 |
| marios | panda: either way the scoping and design is still a good thing to do for now anwyay | 11:34 |
| marios | panda: quiquell so this bootstrap subnodes is only done if we have > 1 node? i mean, otherwise etc/nodepool/sub_nodes_private is empty | 11:35 |
| marios | panda: quiquell i base that 'only when we have > 1 node' on https://github.com/openstack-infra/tripleo-ci/blob/83f0c56bf7e852e2a2a99b467b1131ee95125a11/scripts/tripleo.sh#L1477 | 11:35 |
| marios | panda: quiquell so all the stuff, repo setup, remove packages, setup ceph loop device. it is all done only on subnodes? | 11:35 |
| marios | panda: quiquell this confuses me. what is configuring the primary node then ? | 11:36 |
| *** EvilienM is now known as EmilienM | 11:36 | |
| marios | panda: quiquell same with the repo setup here https://github.com/openstack-infra/tripleo-ci/blob/83f0c56bf7e852e2a2a99b467b1131ee95125a11/scripts/tripleo.sh#L1549 | 11:37 |
| marios | quiquell: panda i.e. only on the subnodes at /etc/nodepool/sub_nodes_private | 11:37 |
| chkumar|ruck | sshnaidm|rover: How I can run a particular featureset on rdocloud? | 11:37 |
| marios | z/win 7 | 11:38 |
| quiquell | marios: Let me check, I have to learn this stuff too | 11:39 |
| panda | quiquell: it's probably because the bootstrap is the replacement for the introspection part. | 11:40 |
| panda | marios: ^ | 11:40 |
| panda | marios: and we don't need to do this in the undercloud because the undercloud install takes care of it | 11:40 |
| marios | panda: ack that makes sense | 11:41 |
| marios | quiquell: panda thanks | 11:41 |
| panda | quiquell: marios anyway, the card that replaces tripleo.sh with ansible is parallel to the migration to zuulv3. I hope we can add it directly in all the workflows | 11:41 |
| panda | and so removing tripleo.sh from all the workflows | 11:42 |
| quiquell | panda: You mean that we are going to be able to reuse that role even in the legacy jobs ? | 11:44 |
| panda | quiquell: I really hope so | 11:44 |
| panda | quiquell: otherwise we are going to be able to use it only after we translate toci_gate_test.sh to ansible | 11:45 |
| quiquell | ok | 11:45 |
| *** rfolco has joined #oooq | 11:55 | |
| *** gvrangan_odl has quit IRC | 11:55 | |
| *** panda is now known as panda|lunch | 11:57 | |
| ssbarnea | hi! can someone give me some hints regarding the checks? I have a trivial change that failed on few jobs that apparently are totally unrelated and I don't know what I should do. The change is not urgent but I want to learn on how to deal with these. See https://review.openstack.org/#/c/581012/ | 11:57 |
| quiquell | ssbarnea: The failure is unrelated | 11:58 |
| ssbarnea | if the solution is just to wait, and recheck the next day I don't mind but when I see ~25 jobs I feel bit a doing a DDOS by adding "recheck" comment. | 11:59 |
| quiquell | ssbarnea: You can put a comment "recheck" in the review, it will re trigger CI | 11:59 |
| panda|lunch | ssbarnea: you need all voting jobs to pass. So if you think it's unrelated you have to wait and recheck | 11:59 |
| quiquell | ssbarnea: and also ask ruck/rover about it, maybe you have discover a intermitent issue | 11:59 |
| panda|lunch | ssbarnea: the only thing you can do alternatively is trying to understand what's going , on, see it the failure is associated with a bug and recheck only when it's solved | 12:00 |
| ssbarnea | does "recheck" retriggers all or just the failed ones? what is the recommended recheck interval? | 12:00 |
| panda|lunch | ssbarnea: but there's not guarantee that something else broke in the meantime | 12:00 |
| panda|lunch | ssbarnea: all | 12:00 |
| ssbarnea | wow, that's why I was afraid to use "recheck". i can only imagine how many kWh does such word costs :D | 12:01 |
| quiquell | ssbarnea: it retrigger all of them, inverval is whenever you need to feel les anxious | 12:01 |
| panda|lunch | ssbarnea: retriggering only the failing jobs is something long discussed, and I think it's still deemed to risky in some occasion | 12:04 |
| panda|lunch | ssbarnea: just avoid rechecking when you just see FAILURE in gerrit, and avoid recheck flood. For the rest, there's not much we can do | 12:05 |
| ssbarnea | yep it can be risky but also useful, there is a way to make it safe, avoid adding the vote when is a partial recheck. | 12:05 |
| panda|lunch | then it's pointless | 12:05 |
| panda|lunch | because you'll need a partial recheck and a full recheck anyway to make it pass | 12:06 |
| panda|lunch | never trust somehing written by someone whose mouth is full of pizza anyway. | 12:06 |
| panda|lunch | I'll continue the lunch | 12:06 |
| ssbarnea | not really because i seen 5 failed jobs: i could work fixing these with minimal CI load and when I am confident, i can try the big one. Anyway is far too soon to have "ideas", i just want to understand the workflow. | 12:07 |
| ssbarnea | bon apetit! this reminds me that I should do the same. | 12:07 |
| weshay | sshnaidm|rover, chkumar|ruck queens is promoting right now right? | 12:08 |
| weshay | is there an issue w/ the images or containers? | 12:08 |
| weshay | same w/ master | 12:08 |
| weshay | sshnaidm|rover, chkumar|ruck | 12:12 |
| weshay | 2018-07-11 10:38:10,848 28706 ERROR promoter + RELEASE=queens | 12:12 |
| weshay | + PROMOTED_HASH=f84ce61bb1e7f9cd62c73cc8c14f01644989c279_1346e217 | 12:12 |
| weshay | + LINK_NAME=current-tripleo | 12:12 |
| weshay | + sftp_command 'rm /var/www/html/images/queens/rdo_trunk/previous-current-tripleo' | 12:12 |
| weshay | + echo 'rm /var/www/html/images/queens/rdo_trunk/previous-current-tripleo' | 12:12 |
| weshay | + sftp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null uploader@images.rdoproject.org | 12:12 |
| *** atoth has joined #oooq | 12:12 | |
| weshay | Warning: Permanently added 'images.rdoproject.org,38.145.33.168' (ECDSA) to the list of known hosts. | 12:13 |
| weshay | Permission denied (publickey,gssapi-keyex,gssapi-with-mic). | 12:13 |
| weshay | Couldn't read packet: Connection reset by peer | 12:13 |
| sshnaidm|rover | weshay, 5 min | 12:13 |
| weshay | k | 12:13 |
| *** amoralej is now known as amoralej|lunch | 12:20 | |
| *** agopi has quit IRC | 12:21 | |
| sshnaidm|rover | weshay, ok, now with you | 12:22 |
| sshnaidm|rover | weshay, it's issue with revoked ssh key yesterday, already fixed, waiting for apevec to put the new key in uploader | 12:23 |
| weshay | sshnaidm|rover, cool.. so the images for queens and master are stored on the promoter and once the uploader is updated the promotion server will run through successfully? | 12:25 |
| sshnaidm|rover | weshay, the problem was in uploading images from jobs | 12:26 |
| sshnaidm|rover | weshay, I talk about images, not containers | 12:26 |
| weshay | sshnaidm|rover, ok.. so I saw that go green last night | 12:26 |
| weshay | sshnaidm|rover, yes.. I know re: images | 12:26 |
| weshay | was the green job a false positive? | 12:26 |
| weshay | ah.. refreshing my understanding of the image promote | 12:27 |
| weshay | so the image never made it to the file server? | 12:28 |
| chkumar|ruck | weshay: sshnaidm|rover: https://bugs.launchpad.net/tripleo/+bug/1780726/comments/8 | 12:30 |
| openstack | Launchpad bug 1780726 in tripleo "[master][queens][pike][ocata] Base periodic jobs are broken after zuul v3 migration " [Critical,Fix released] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 12:30 |
| *** rlandy has joined #oooq | 12:30 | |
| *** udesale has joined #oooq | 12:31 | |
| weshay | sshnaidm|rover, so the sad thing will be if we can't promote that hash because the overcloud image couldn't have been saved | 12:32 |
| weshay | know what I mean? | 12:32 |
| sshnaidm|rover | weshay, which job exactly was green? | 12:32 |
| weshay | sshnaidm|rover, all the periodic promotoin jobs for queens and master | 12:32 |
| sshnaidm|rover | weshay, if it's false positive, we need to fix it | 12:33 |
| sshnaidm|rover | weshay, including upload job? | 12:33 |
| * weshay looks | 12:34 | |
| ykarel | sshnaidm|rover, weshay yes:- https://trunk.rdoproject.org/api-centos-queens/api/civotes_detail.html?commit_hash=f84ce61bb1e7f9cd62c73cc8c14f01644989c279&distro_hash=1346e2175b5ec58f1fabb08ded575146837f7116 | 12:34 |
| weshay | sshnaidm|rover, you saw the screenshot from my email ya? | 12:34 |
| *** udesale_ has joined #oooq | 12:35 | |
| *** udesale_ has quit IRC | 12:36 | |
| *** udesale has quit IRC | 12:36 | |
| weshay | ykarel, nice | 12:36 |
| sshnaidm|rover | weshay, this is weird.. rsync failed with 255, we have "set -e", but it still passed | 12:36 |
| weshay | quiquell, fyi.. the promotions section of http://38.145.34.131:3000/d/pgdr_WVmk/cockpit?orgId=1 seems out of date | 12:38 |
| quiquell | weshay: Let me check | 12:39 |
| weshay | panda|lunch, you want to 1-1 early? | 12:39 |
| quiquell | weshay: Yep, promoter is promoting | 12:39 |
| weshay | quiquell, here's a cool link to build from the cockpit.. https://trunk.rdoproject.org/api-centos-queens/api/civotes_detail.html?commit_hash=f84ce61bb1e7f9cd62c73cc8c14f01644989c279&distro_hash=1346e2175b5ec58f1fabb08ded575146837f7116 | 12:41 |
| quiquell | weshay: I don't see promotions here https://dashboards.rdoproject.org/master | 12:42 |
| quiquell | weshay: Something is not right, promoter is promoting but they don't appear at DLRN | 12:42 |
| marios | panda|lunch: sshnaidm|rover wdyt folks please when you next get some reviews time thanks https://review.openstack.org/#/c/578081/11/roles/create-reproducer-script/templates/reproducer-quickstart.sh.j2 | 12:42 |
| weshay | quiquell, the overcloud image promotion is failing | 12:43 |
| quiquell | weshay: Ok, then the promotion dashboard is ok | 12:43 |
| quiquell | There is no master promotions from like 5 days or so | 12:43 |
| weshay | quiquell, the latest data I see in ruck/rover cockpit is from the 9th | 12:43 |
| weshay | quiquell, ah.. SORT | 12:44 |
| weshay | fail | 12:44 |
| weshay | see it | 12:44 |
| weshay | quiquell, lolz | 12:44 |
| weshay | I'll fix that in the settings | 12:44 |
| quiquell | weshay: but ther is something weird, promoter shows activity... | 12:44 |
| quiquell | weshay: It's like promoting something that doesn't yet promotes | 12:44 |
| quiquell | weshay: Ahh the containers, it promotes the containers but not the DLRN | 12:45 |
| weshay | 2018-07-11 10:38:10,848 28706 ERROR promoter + RELEASE=queens | 12:45 |
| weshay | + PROMOTED_HASH=f84ce61bb1e7f9cd62c73cc8c14f01644989c279_1346e217 | 12:45 |
| weshay | + LINK_NAME=current-tripleo | 12:45 |
| weshay | + sftp_command 'rm /var/www/html/images/queens/rdo_trunk/previous-current-tripleo' | 12:45 |
| weshay | + echo 'rm /var/www/html/images/queens/rdo_trunk/previous-current-tripleo' | 12:45 |
| weshay | + sftp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null uploader@images.rdoproject.org | 12:45 |
| weshay | Warning: Permanently added 'images.rdoproject.org,38.145.33.168' (ECDSA) to the list of known hosts. | 12:45 |
| weshay | Permission denied (publickey,gssapi-keyex,gssapi-with-mic). | 12:45 |
| weshay | Couldn't read packet: Connection reset by peer | 12:45 |
| marios | weshay: wdyt do you want to keep this one? https://review.openstack.org/#/c/578768/2/quickstart.sh vote when you have a chance please | 12:46 |
| marios | rlandy: wdyt please https://review.openstack.org/#/c/579587/3/roles/create-reproducer-script/templates/reproducer-quickstart.sh.j2 when you next have reviews time thanks | 12:46 |
| *** radez has joined #oooq | 12:47 | |
| weshay | marios, checking it out | 12:49 |
| radez | hey folks I'm trying to run oooq on my laptop to provision a remote server unsing a baremetal delpoyment, quickstart throws this as soon as I execute | 12:49 |
| radez | http://paste.openstack.org/show/725557/ | 12:50 |
| radez | my command looks like this: http://paste.openstack.org/show/725558/ | 12:50 |
| *** panda|lunch is now known as panda | 12:53 | |
| *** trown|outtypewww is now known as trown | 12:55 | |
| panda | weshay: 5 minutes before only, but ready | 12:55 |
| rlandy | radez: hello | 12:55 |
| rlandy | radez: let's step back - from your laptop, you're trying to run quickstart and hitting install errors ... | 12:56 |
| rlandy | to deploy onto baremetal hardware ... | 12:57 |
| rlandy | error are unrelated to the baremetal ... | 12:57 |
| rlandy | if you run as non-root user and create a create a venv or not (from your laptop) and you try run quickstart, can you paste the error from that point? | 12:58 |
| rlandy | we'll start from there | 12:58 |
| radez | rlandy: that's what that paste is above, no venv as a non root user | 12:58 |
| radez | from my laptop | 12:59 |
| quiquell | sshnaidm|rover: We are buck in business with the ARA oc http://logs.openstack.org/62/5784 | 12:59 |
| quiquell | http://logs.openstack.org/62/578462/12/check/tripleo-ci-centos-7-scenario004-multinode-oooq-container/9a8b1d9/logs/ara_oooq_oc/ | 12:59 |
| quiquell | I mean ^ | 12:59 |
| rlandy | radez: this is from your laptop ( as non-root) http://paste.openstack.org/show/725557/ - I thought it was on the virthost itself - sorry, I am confused what is referenced as the server here | 13:00 |
| sshnaidm|rover | quiquell, cool | 13:00 |
| sshnaidm|rover | quiquell, although I see 008 is failing.. | 13:00 |
| hubbot` | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ (1 more message) | 13:00 |
| quiquell | sshnaidm|rover: ack, will check | 13:00 |
| sshnaidm|rover | quiquell, but I need to fix the patch in tripleo-common also, slagle has issues there | 13:01 |
| *** skramaja has quit IRC | 13:01 | |
| quiquell | sshnaidm|rover: Added a small fix there, to make it work. | 13:01 |
| radez | rlandy: that one is from my laptop, I've been doing both from laptop and from server, I'm probably making this more complicated that it needs to be :) | 13:01 |
| *** openstack has joined #oooq | 13:02 | |
| sshnaidm|rover | quiquell, btw, pep8 fails | 13:04 |
| radez | rlandy: I think I may have figured this, I think I have to wactually export the variables in bash, sec lemme try it | 13:05 |
| sshnaidm|rover | quiquell, oh, it was failing for me too, ok.. | 13:05 |
| quiquell | sshnaidm|rover: Didn't check just trying to make it run again. | 13:05 |
| radez | rlandy: yea that was is, I guess quickstart won't let me pass in env vars in the command, they need to actually be exported, ok I'm moving, I'm sure I'll bug you again before long :) | 13:06 |
| sshnaidm|rover | quiquell, I think you can hold on it right now, I'm in the middle of changing it, but need to decide which way we are going with it | 13:06 |
| sshnaidm|rover | quiquell, so it may change a lot.. | 13:07 |
| quiquell | sshnaidm|rover: ack, all yours. | 13:07 |
| rlandy | radez:cool - happy you figured it out - | 13:07 |
| quiquell | rfolco: We are going to try to merge this https://review.openstack.org/#/c/581331/ | 13:07 |
| quiquell | rfolco: maybe you can reparent with it | 13:08 |
| rfolco | quiquell, I am looking now... I just think you comment in the wrong card. The j2 templates have different cards, not the same as jobtype var | 13:09 |
| quiquell | rfolco: I mean, for your jobtype review to play around | 13:09 |
| quiquell | rfolco: You can reparent that to https://review.openstack.org/#/c/581331/ so yon focus only in the variable thing | 13:09 |
| rlandy | quiquell: is your test machine working now? | 13:09 |
| rlandy | quiquell: can you reprovision it? | 13:10 |
| rlandy | access it from beaker? | 13:10 |
| rlandy | we are having issues with marios machine | 13:10 |
| quiquell | rlandy: Was not able to use beaker with it | 13:10 |
| quiquell | rlandy: You can take it, I am using RDO cloud now | 13:10 |
| rlandy | quiquell: can you reprovision it at all? | 13:11 |
| quiquell | rlandy: Nope from beaker no | 13:11 |
| rlandy | quiquell: I don;t need it - I am just surveying what we have | 13:11 |
| * rlandy gets that ticket | 13:11 | |
| quiquell | rlandy: Last time you reprovisioned it for me | 13:11 |
| rlandy | quiquell: I know - that is not sustainable - we have to get this fixed | 13:12 |
| quiquell | rlandy: I can check back with beaker | 13:12 |
| rlandy | quiquell: no worries | 13:12 |
| quiquell | rlandy: We didn't have any issue last time you reprovisioned it | 13:13 |
| *** amoralej|lunch is now known as amoralej | 13:13 | |
| rlandy | marios: I am adding you to a ticket so you can read what happened | 13:23 |
| marios | rlandy: thanks very much for chasing that up | 13:24 |
| marios | rlandy: i'm also using rdo cloud for now and have another beaker box so not hugely urgent but yeah would be good to be able to reprovision that at will :) | 13:24 |
| rlandy | marios: you tried to reprovision with beaker to centos7.4? 7.5? | 13:24 |
| marios | 7.4 rlandy | 13:25 |
| rlandy | marios: yep - if you look at the ticket - that is the reported problem | 13:25 |
| marios | rlandy: ah i see | 13:27 |
| marios | rlandy: so not a new problem at all | 13:27 |
| rlandy | marios: no - but it's not really acceptable either - will chat with weshay about what to do long term | 13:27 |
| rlandy | marios: can we try one more time- can you reprovision through beaker with centos 7.2 | 13:29 |
| rlandy | I will watch the console | 13:29 |
| rlandy | in the other cases, I could get to the pxe boot menu | 13:29 |
| marios | rlandy: sure sec | 13:31 |
| weshay | arxcruz, ready | 13:31 |
| marios | rlandy: done | 13:31 |
| arxcruz | ok | 13:32 |
| weshay | sshnaidm|rover, chkumar|ruck let me know when the image upload issue is resolved | 13:32 |
| rlandy | ok - ou got power control | 13:33 |
| rlandy | you | 13:33 |
| quiquell | rfolco, panda: The deploy_type using the jinja templates | 13:35 |
| quiquell | It does not remove TOCI_JOBTYPE, so everything else is working | 13:35 |
| rlandy | quiquell: can I borrow your machine - but to check the boot options? | 13:35 |
| rlandy | I will need to reboot it | 13:35 |
| quiquell | rlandy: All yours | 13:35 |
| rlandy | which one did you have? | 13:36 |
| quiquell | rfolco, panda: https://review.openstack.org/#/c/581746/ | 13:36 |
| *** agopi has joined #oooq | 13:37 | |
| rlandy | marios: no pxe menu here - ok so that it a clear problem ... comparing with quiquell's machine | 13:39 |
| *** ratailor has quit IRC | 13:39 | |
| marios | rlandy: ok, i can do that i mean compare the beaker settings | 13:39 |
| marios | rlandy: gimme addresses in pvt | 13:40 |
| rlandy | marios: boot settings | 13:40 |
| panda | quiquell: mmmh, deploy_type is ambiguous, you've removed featureset from the loop even if you don't pass it as variable, and you cant' use $deploy_type in bash, {{ deploy_type }} is what you want to use in the template | 13:42 |
| quiquell | panda: Damn wait mixed two works at my laptop | 13:42 |
| panda | quiquell: try to fast forward instead | 13:43 |
| panda | quiquell: is it n + 1 or n - 1 work | 13:43 |
| quiquell | panda: keep featureset as it is | 13:44 |
| quiquell | panda: So we see that we can work only en deploy_type for example | 13:45 |
| radez | rlandy: my undercloud install is still failing, but it doesn't appear that the ss hkeys fron the server are being installed on the undercloud so I can get into it, how to I get the undercloud install log to debug? | 13:46 |
| quiquell | panda: fixed now, brain fart the $deploy_type | 13:48 |
| panda | oh, now I get what prouces code smell, brain farts! | 13:48 |
| quiquell | panda: Yep :-) | 13:49 |
| rlandy | radez: you're on the undercloud hardware box but not on the vm? or you're already on the vm? | 13:49 |
| rlandy | the undercloud install logs are in /home/stack on the vm | 13:50 |
| rlandy | to reach the vm, | 13:50 |
| rlandy | in your workspace, where you ran from, do you see a file called ssh.config.ansible? | 13:50 |
| quiquell | panda: What was the hosts to run stuff in the ansible executor ? | 13:51 |
| quiquell | localhost ? | 13:51 |
| rlandy | undercloud_install.log should be on the vm in /home/stack | 13:51 |
| radez | rlandy: yea I see ssh config asible file | 13:51 |
| panda | quiquell: yes | 13:51 |
| rlandy | radez: can you try ssh -F ssh.config.ansible undercloud | 13:51 |
| radez | rlandy: yup that got me in, thx | 13:52 |
| rlandy | radez: cool - see /home/stack - your log is there | 13:52 |
| arxcruz | panda: rlandy weshay sshnaidm|rover can we have https://review.openstack.org/#/c/580384/ merged? | 13:56 |
| radez | rlandy: error there is that it can't contact pool.ntp.org but I'm using the env template that you gave me that defines ntp-server | 13:57 |
| rlandy | radez: defines it for the overcloud deploy | 13:58 |
| rlandy | this is the undercloud install complaining | 13:58 |
| radez | rlandy: ah, gotcha, is there a place to pass ntp server for the undercloud? | 13:58 |
| radez | yes it's the undrcloud install thats' complaingin | 13:59 |
| rlandy | probably - I'd have to look but that is the first time I have seen that error | 13:59 |
| rlandy | undercloud not being able to reach pool.ntp.org | 14:00 |
| rlandy | do you have outside access at all? | 14:00 |
| rlandy | from that box? | 14:00 |
| radez | I do, but I think ntp is blocked to ntp.org from some labs | 14:00 |
| rlandy | oh, I see ... | 14:00 |
| radez | rlandy: I'll look up the docs and see if I can find the option | 14:01 |
| rlandy | radez: I think it would be in undercloud.conf | 14:01 |
| panda | arxcruz: commented, small nit | 14:01 |
| rlandy | checking | 14:01 |
| panda | arxcruz: sorry not a nit | 14:01 |
| panda | arxcruz: just minimal suggestion | 14:02 |
| panda | minor | 14:02 |
| rlandy | step_undercloud_ntp: true | 14:02 |
| radez | rlandy: I'm no passing in undecloud.conf, maybe there's a var in env template that would modify it? | 14:02 |
| rlandy | so that must be possible | 14:02 |
| rlandy | tripleo does | 14:02 |
| rlandy | radez: I am looking at rasca's work | 14:03 |
| rlandy | he has machines in some lab somewhere and I see he has an option to set ntp - looking at that | 14:04 |
| rasca | rlandy, yep it is part of the baremetal-undercloud role | 14:04 |
| rlandy | rasca: can your baremetal machines also not reach pool.ntp.org? | 14:04 |
| radez | I'm trying undercloud_ntp_servers in the env tpml | 14:05 |
| radez | do I need to clean or can I just rerun quickstart rlandy ? | 14:06 |
| rlandy | radez: quickstart run itself will clear the vm and recreate it | 14:06 |
| rlandy | clear will clear your own workspace | 14:07 |
| rlandy | clean | 14:07 |
| radez | kk, thx | 14:07 |
| rlandy | if you add a new option, it's better to start clean | 14:07 |
| rasca | rlandy, the ntp part was made to support any ntp server | 14:12 |
| *** holser_ has quit IRC | 14:14 | |
| *** holser_ has joined #oooq | 14:16 | |
| rlandy | marios: woohoo | 14:17 |
| rlandy | I got a pxe boot menu now | 14:17 |
| marios | rlandy: cool what did it?! | 14:17 |
| rlandy | marios; I switched the boot order | 14:18 |
| marios | rlandy:++ | 14:18 |
| marios | thank you | 14:18 |
| rlandy | marios: can you try another reprovision from beaker - centos 7.2? | 14:19 |
| rlandy | if that works, we'll upgrade | 14:19 |
| *** ykarel has quit IRC | 14:19 | |
| marios | rlandy: sure sec (why 7.2 is it more likely to boot, sure np i can yum upgrade it) | 14:19 |
| rlandy | marios: see the ticket - we know that one works | 14:19 |
| quiquell | Droping now, read you tomorrow folks | 14:20 |
| rlandy | we are debugging one at a time here | 14:20 |
| *** quiquell is now known as quiquell|off | 14:20 | |
| marios | rlandy: k just done | 14:20 |
| rlandy | thanks - watching console | 14:20 |
| rlandy | marios: ok - I can't tell from the console exactly what is going on | 14:25 |
| rlandy | if your reprovision fails, we ac redo it from drac | 14:25 |
| weshay | sshnaidm|rover, 2018-07-11 14:04:53,854 11945 INFO promoter Promoting the container images for dlrn hash e0c5c24e3bca25349894b8055589d88a16f4b894 on master to current-tripleo | 14:29 |
| weshay | http://38.145.34.55/master.log | 14:29 |
| rlandy | sshnaidm|rover: are you able to reprovision through beaker or do you use drac? | 14:33 |
| sshnaidm|rover | rlandy, didn't try it recently | 14:34 |
| rlandy | sshnaidm|rover: when you tried last? | 14:34 |
| sshnaidm|rover | rlandy, last time it was about 2 months ago | 14:34 |
| rlandy | beaker? | 14:34 |
| sshnaidm|rover | rlandy, from beaker and it succeeded | 14:34 |
| sshnaidm|rover | rlandy, to 7.2 | 14:34 |
| rlandy | sshnaidm|rover: and your machine is rdo-ci-fx2-02-s7? | 14:35 |
| rlandy | ok | 14:35 |
| rlandy | marios;sis your first reprovision to centos 7.2 work this morning? before I switched the boot order? | 14:36 |
| rlandy | did | 14:36 |
| rlandy | I think it might have - right now I see the boot menu from f12 but I don;t see anything going on now | 14:37 |
| sshnaidm|rover | rlandy, everything after 7.2 will fail | 14:37 |
| rlandy | sshnaidm|rover: yep I know | 14:37 |
| rlandy | need to complain about that when I remember to | 14:38 |
| sshnaidm|rover | rlandy, btw, I tried from beaker website, because beaker client seemed not working (not sure though) | 14:38 |
| rlandy | using the website as well | 14:38 |
| rlandy | ok .. | 14:39 |
| rlandy | marios: ping me before you go - I think we can work this out | 14:39 |
| amoralej | i'm finding an issue related to missing deps in heat horizon plugin in puppet promotion jobs, are you seeing something similar? | 14:40 |
| amoralej | error is that httpd fails to start | 14:40 |
| *** ykarel has joined #oooq | 14:42 | |
| *** kopecmartin has quit IRC | 14:48 | |
| marios | rlandy: o/ | 14:52 |
| marios | rlandy: sec | 14:52 |
| hubbot` | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ (1 more message) | 15:00 |
| *** holser_ has quit IRC | 15:02 | |
| arxcruz | lucas-afk: still afk man? :P | 15:09 |
| panda | arxcruz: he's off this week | 15:09 |
| *** florianf has quit IRC | 15:09 | |
| arxcruz | (╯°□°）╯︵ ┻━┻ | 15:09 |
| *** holser_ has joined #oooq | 15:11 | |
| *** florianf has joined #oooq | 15:14 | |
| *** holser_ has quit IRC | 15:15 | |
| *** holser_ has joined #oooq | 15:15 | |
| *** links has quit IRC | 15:16 | |
| *** holser_ has quit IRC | 15:16 | |
| *** bogdando has quit IRC | 15:16 | |
| *** holser_ has joined #oooq | 15:17 | |
| *** holser_ has quit IRC | 15:17 | |
| *** holser_ has joined #oooq | 15:18 | |
| weshay | rlandy, did you need me for something? | 15:20 |
| rlandy | marios: ok - you should be all set to log in now | 15:20 |
| rlandy | weshay: did I ping you? | 15:21 |
| rlandy | weshay: but now that you ask, pls can you look at the three reviews related to perf work | 15:21 |
| weshay | rlandy, k | 15:22 |
| rlandy | and let me know if you agree with those | 15:22 |
| rlandy | if so, I will ping the browbeat folks to continue | 15:22 |
| rlandy | panda: marios: sshnaidm|rover: could I get another review on https://review.openstack.org/#/c/566155/? | 15:24 |
| *** kopecmartin has joined #oooq | 15:31 | |
| radez | rlandy, got the ntp thing fixed, it's just ntp_servre in the env template. Now I'm getting an error that ansible can't ocnnect as root to the virthost. The ssh.config.ansible doesn't have a config for it, if i ssh to the virthost as root from my laptop where I'm running oooq I can loging fine | 15:32 |
| panda | rlandy: weshay I alwways forgot to mention, I didn't make any plans on when to migrate standalon jobs to zuulv3. If we think they are ready, we should add them to the list | 15:33 |
| rlandy | radez: does your user on the virthost have paswordless sudo? | 15:37 |
| rlandy | ie: the user in one the virthost in ssh.config.ansible | 15:37 |
| rlandy | if you can ssh root@$VIRTHOST ansible should be able to do the same | 15:39 |
| rlandy | no password | 15:39 |
| rlandy | we'd have to look at which exact task is failing | 15:39 |
| rlandy | panda: agreed - those should be able to move | 15:40 |
| rlandy | I am just rebase/merging the 007/008 review - in merge conflict atm | 15:41 |
| agopi | rlandy++ | 15:41 |
| hubbot` | agopi: rlandy's karma is now 10 | 15:41 |
| agopi | just saw your commits | 15:41 |
| agopi | was away for NHO last 2 days. | 15:41 |
| rlandy | agopi: we have work to do there | 15:41 |
| rlandy | I need to minimal playbook to run | 15:41 |
| rlandy | but those tree reviews should be the basic layout | 15:41 |
| rlandy | three | 15:41 |
| agopi | okay rlandy | 15:43 |
| rlandy | agopi: will chat with you more in a but - just in the middle of a review merge mess atm | 15:43 |
| rlandy | bit | 15:43 |
| agopi | sure thing! | 15:43 |
| *** ykarel is now known as ykarel|away | 15:46 | |
| rlandy | sshnaidm|rover: panda: marios: need core reviews on this again pls (sorry - merge conflict) https://review.openstack.org/#/c/581116/ | 15:48 |
| *** tcw has quit IRC | 15:51 | |
| *** tcw has joined #oooq | 15:51 | |
| *** tcw1 has joined #oooq | 15:52 | |
| *** tcw has quit IRC | 15:52 | |
| *** sanjayu_ has quit IRC | 15:54 | |
| marios | rlandy: sorry was on a call checking now | 16:00 |
| *** tesseract has quit IRC | 16:01 | |
| weshay | panda, yes please re: standalone | 16:02 |
| *** jfrancoa has quit IRC | 16:04 | |
| *** tcw1 has quit IRC | 16:05 | |
| marios | rlandy: ack | 16:07 |
| rlandy | marios: cool - let's go from there - let me know if you have other issues | 16:07 |
| marios | rlandy: did you reprovision? | 16:08 |
| marios | rlandy: or should i sorry i missed something in backchat maybe | 16:08 |
| rlandy | marios: no again | 16:08 |
| marios | you rebooted | 16:08 |
| marios | rlandy: i think was the last | 16:08 |
| rlandy | I just rebooted in the system to recomfirm boot order | 16:08 |
| marios | rlandy: ok | 16:09 |
| weshay | marios++ | 16:09 |
| hubbot` | weshay: marios's karma is now 1 | 16:09 |
| weshay | marios++ | 16:09 |
| hubbot` | weshay: marios's karma is now 2 | 16:09 |
| marios | weshay: ha | 16:09 |
| marios | what did i do | 16:09 |
| weshay | hehe.. that worked | 16:09 |
| marios | weshay: wow really? the () | 16:09 |
| weshay | ya.. | 16:09 |
| weshay | No matching distribution found for foowes==0.6.5 (from -r requirements.txt (line 2)) | 16:10 |
| weshay | python setup.py install failed | 16:10 |
| marios | well i still wonder why it was necessary | 16:10 |
| weshay | f.. a | 16:10 |
| marios | weshay: so might be worth git blaming i can check in the morning if you like | 16:10 |
| weshay | I see it | 16:10 |
| weshay | 2yrs old from lars | 16:10 |
| marios | k | 16:10 |
| *** tcw has joined #oooq | 16:10 | |
| panda | tcw: your nick remins me of a scrubs episode. | 16:11 |
| *** panda is now known as panda|off | 16:11 | |
| marios | rlandy: pvt | 16:11 |
| *** tcw has quit IRC | 16:12 | |
| *** tcw has joined #oooq | 16:14 | |
| *** ykarel|away is now known as ykarel | 16:18 | |
| *** Goneri has quit IRC | 16:19 | |
| weshay | marios, https://review.openstack.org/581789 | 16:22 |
| marios | rlandy++ | 16:24 |
| hubbot` | marios: rlandy's karma is now 11 | 16:24 |
| marios | rlandy: thanks | 16:24 |
| weshay | rlandy, fyi https://review.rdoproject.org/r/#/c/14780/5 | 16:25 |
| marios | weshay: thanks i will enjoy it over morning coffe if thats ok | 16:25 |
| marios | weshay: gonna call it there | 16:25 |
| weshay | marios, ya man | 16:25 |
| weshay | thanks for the help | 16:25 |
| marios | weshay: haha 'help' np ;) | 16:25 |
| *** florianf has quit IRC | 16:34 | |
| weshay | panda|off, rlandy this may block using the upstream infra role for multinode.. not sure .. https://bugs.launchpad.net/tripleo/+bug/1781255 | 16:47 |
| openstack | Launchpad bug 1781255 in tripleo ""Error: centos-release-ceph-luminous conflicts with centos-release-ceph-jewel-1.0-1.el7.centos.noarch" [Undecided,New] | 16:47 |
| weshay | maybe we need to update the centos image we're using for the recreate scripot | 16:47 |
| weshay | script | 16:47 |
| rlandy | reading back | 16:49 |
| weshay | rlandy, comments in https://review.openstack.org/#/c/579161/11 | 16:49 |
| * weshay wonders if you hit that too now | 16:49 | |
| rlandy | thought that was fixed ;( | 16:49 |
| weshay | rlandy, it IS in tripleo | 16:49 |
| weshay | and quickstart | 16:49 |
| weshay | this is infra bs again | 16:49 |
| rlandy | oh shoot me now | 16:50 |
| rlandy | no - I haven't hit it - have not run that in a while | 16:50 |
| weshay | rlandy, http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/multi-node-bridge/tasks/common.yaml#n10 | 16:50 |
| weshay | ya.. FAK | 16:50 |
| weshay | rlandy, I think Guilio may need to fix that | 16:50 |
| rlandy | merged yesterday | 16:51 |
| rlandy | weshay: why didn't we see that before?? | 16:52 |
| rlandy | we tested this | 16:52 |
| weshay | rlandy, so.. question.. where are we pulling the centos image | 16:55 |
| weshay | ? | 16:55 |
| weshay | maybe that is now out of date | 16:55 |
| *** ykarel has quit IRC | 16:57 | |
| rlandy | weshay: could be - I don't know what we pull exactly - looking | 16:59 |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ (1 more message) | 17:00 |
| weshay | rlandy overcloud_full? | 17:02 |
| weshay | \/var/lib/oooq-images | 17:02 |
| *** amoralej is now known as amoralej|off | 17:05 | |
| weshay | rlandy, included: /var/tmp/ara/tripleo-quickstart/roles/fetch-images/tasks/fetch.yml for 127.0.0.2 | 17:05 |
| weshay | TASK [fetch-images : include] *************************************************************************************************************************************************************************************** | 17:06 |
| weshay | included: /var/tmp/ara/tripleo-quickstart/roles/fetch-images/tasks/fetch.yml for 127.0.0.2 | 17:06 |
| weshay | TASK [fetch-images : image name] ************************************************************************************************************************************************************************************ | 17:06 |
| weshay | ok: [127.0.0.2] => { | 17:06 |
| weshay | "msg": "checking for image centos" | 17:06 |
| weshay | } | 17:06 |
| weshay | TASK [fetch-images : set local variables] *************************************************************************************************************************************************************************** | 17:06 |
| weshay | ok: [127.0.0.2] | 17:06 |
| weshay | TASK [fetch-images : Check if we have a latest image] *************************************************************************************************************************************************************** | 17:06 |
| *** holser_ has quit IRC | 17:06 | |
| weshay | ok: [127.0.0.2] | 17:06 |
| weshay | [root@localhost oooq-images]# locate centos | grep qcow2 | 17:07 |
| weshay | \/var/cache/tripleo-quickstart/images/queens/latest-centos.qcow2 | 17:07 |
| rlandy | is that a queens job? | 17:07 |
| rlandy | or a master job? | 17:08 |
| *** trown is now known as trown|lunch | 17:08 | |
| rlandy | \/var/cache/tripleo-quickstart/images/queens/latest-centos.qcow | 17:08 |
| rlandy | latest of what? | 17:09 |
| weshay | curl -sfL -C- -o _centos.qcow2 https://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud-1802.qcow2 | 17:10 |
| *** sshnaidm|rover is now known as sshnaidm|bbl | 17:12 | |
| weshay | rlandy, https://github.com/openstack/tripleo-quickstart/blob/master/config/environments/baseos_centos_libvirt.yml#L5 | 17:12 |
| radez | rlandy: hey, sry got pulled away from my discussion with you earlier. I'm still stuck at logging in as root to the virt host, it doesn't look like it's trying to sudo: http://paste.openstack.org/show/725590/ | 17:13 |
| radez | rlandy: if I do ssh -F ssh.config.ansible root@virthost I can't get in, but if I leave off the configuration file it works | 17:14 |
| rlandy | ok - and you can ssh root@ansible-fx2-1.tripleo.lab.eng.rdu2.redhat.com from the machine running quickstart without a password | 17:15 |
| rlandy | mostly our ci machines are set up that we set the passwprd for the non-root user and then copy that to root | 17:17 |
| rlandy | so root and the non-root login are the same | 17:17 |
| rlandy | you are correct that the ssh file will not help if that case is not there | 17:18 |
| rlandy | https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-quickstart-master-rdo_trunk-baremetal-dell_fc430_envB-single_nic_vlans/ws/ssh.config.ansible/*view*/ | 17:19 |
| rlandy | does not expect a root case | 17:19 |
| *** chkumar|ruck is now known as chandankumar | 17:29 | |
| radez | rlandy: hm, so I wonder why it's trying to use root? | 17:29 |
| rlandy | because it's deleting a user | 17:30 |
| radez | rlandy: why do you have passwords setup? is it not all ssh key setup? | 17:30 |
| rlandy | we don't - it is ssh | 17:30 |
| rlandy | the same keys on both | 17:30 |
| radez | oh ok, I see so maybe I need to add the pub key from the oooq direcotry to the root user on the virhost? | 17:31 |
| rlandy | the command is correct - has this happened once, more than? | 17:31 |
| rlandy | yeah - I think so | 17:33 |
| radez | hm, tried that and it didn't help, yea it's happened only once so far, maybe I'll kick off another deploy and see if I can recreate it | 17:34 |
| *** sshnaidm|bbl is now known as sshnaidm|rover | 17:45 | |
| *** kopecmartin has quit IRC | 17:57 | |
| sshnaidm|rover | rlandy, panda|off, weshay please take a look at patch for oooq to install additional roles via cli: https://review.openstack.org/#/c/576816/ I had a talk with downstream folks, it will help us to use same roles both upstream and downstream, also will be good for external roles like ops-tools etc | 17:59 |
| weshay | cool thanks | 18:03 |
| weshay | sshnaidm|rover, https://review.openstack.org/#/c/581789/ | 18:04 |
| *** agopi is now known as agopi|lunch | 18:07 | |
| weshay | rlandy, oh well https://github.com/openstack/tripleo-quickstart/blob/master/roles/libvirt/setup/overcloud/tasks/vars/libvirt_nodepool_vars.yml | 18:08 |
| weshay | not sure why we decided to suddenly burry vars and config | 18:13 |
| rlandy | bury? | 18:18 |
| rlandy | always been there | 18:18 |
| *** sshnaidm|rover is now known as sshnaidm|off | 18:19 | |
| weshay | rlandy, why there though? | 18:20 |
| weshay | please point out one other config file we have like that | 18:20 |
| weshay | rlandy, is that all for now? https://review.openstack.org/#/q/topic:browbeat-check+(status:open+OR+status:merged) | 18:22 |
| rlandy | I guess because of libvirt-nodepool | 18:22 |
| rlandy | weshay: and https://review.rdoproject.org/r/#/c/14772/1 | 18:23 |
| rlandy | ^^ that is the one where I am not sure | 18:23 |
| rlandy | I commented out stuff so we would not mistakenly run anything | 18:24 |
| weshay | rlandy, hrm.. what is your concern w/ running? | 18:25 |
| rlandy | like what to run when we change browbeat | 18:25 |
| rlandy | is browbeat in the right file | 18:25 |
| rlandy | tripleo or upstream? | 18:25 |
| rlandy | I would need to duplicate all the irrelevant files | 18:26 |
| weshay | rlandy, let's just assume it's our job for now | 18:26 |
| weshay | part of the tripleo jobs | 18:26 |
| rlandy | ok so it's in the right file at least | 18:27 |
| weshay | rlandy, you can't hurt anything w/ third party | 18:27 |
| rlandy | weshay, can I quote you on that???? | 18:27 |
| rlandy | you can hurt third party infra | 18:27 |
| rlandy | resources | 18:27 |
| weshay | ya.. you can | 18:28 |
| weshay | :) | 18:28 |
| rlandy | https://review.rdoproject.org/r/#/c/14772/1/zuul/upstream.yaml | 18:28 |
| rlandy | how about the addition to experimental? | 18:28 |
| rlandy | I only added it to check | 18:28 |
| rlandy | anyways, pls comment on that review and I'll complete it | 18:28 |
| weshay | rlandy, it should not be there in check | 18:28 |
| rlandy | then what we need is the minimal playbook from perf team | 18:29 |
| agopi|lunch | rlandy, weshay lmk how i can be of help. | 18:29 |
| weshay | actually | 18:29 |
| weshay | that is the right place | 18:29 |
| weshay | or not.. | 18:29 |
| weshay | rlandy, show me where that runs? | 18:29 |
| rlandy | agopi|lunch: we would need https://review.openstack.org/#/c/581488/1/playbooks/baremetal-quickstart-extras.yml | 18:29 |
| weshay | doesn't run anywhere | 18:30 |
| rlandy | baremetal-full-browbeat-minimal.yml | 18:30 |
| agopi|lunch | rlandy, im guessing something like this | 18:30 |
| agopi|lunch | https://github.com/openstack/browbeat/blob/master/ansible/oooq/baremetal-virt-undercloud-int-browbeat.yml | 18:30 |
| rlandy | https://github.com/openstack/browbeat/blob/master/ansible/oooq/baremetal-virt-undercloud-int-browbeat.yml#L4 | 18:30 |
| rlandy | ^^ not needed | 18:30 |
| rfolco | rlandy, can I help with standalone parent ? | 18:31 |
| weshay | rlandy, where does tripleo-ovb-check trigger? | 18:31 |
| rfolco | rlandy, any secrets or disclaimers ? | 18:31 |
| rlandy | rfolco: sure - panda has it unassigned | 18:31 |
| weshay | I've never seen all those jobs run | 18:31 |
| rlandy | weshay: ... it's listed in a bunch of places :) | 18:31 |
| weshay | rlandy, never seen that run | 18:32 |
| rlandy | agopi|lunch: also not that - we need something that can run post deploy | 18:32 |
| weshay | rlandy, I think that you think that is going to run on every patch | 18:32 |
| weshay | to every repo | 18:32 |
| *** trown|lunch is now known as trown | 18:32 | |
| rlandy | weshay: fine, I'll remove it from there | 18:33 |
| rlandy | weshay: are the other jobs definitions/inclusions correct? | 18:33 |
| rlandy | agopi|lunch: make sense? | 18:34 |
| rlandy | your playbook would get included after tempest | 18:34 |
| weshay | rlandy, you know I'm no expert :) | 18:34 |
| weshay | will have to dive into that for a bit | 18:35 |
| rlandy | also all those irrelevant files | 18:35 |
| rlandy | duplicated everywhere | 18:35 |
| rlandy | rfolco: you can look at my 3/4 node reparent job | 18:35 |
| rlandy | which panda has a comment on I was just about to fix ... | 18:36 |
| rfolco | rlandy, this looks much simpler since it is single node, isn't it ? why it was left behind :) ? | 18:36 |
| rlandy | rfolco: idk - I just work here | 18:36 |
| rfolco | haha | 18:36 |
| rlandy | hold on getting patch | 18:37 |
| rfolco | rlandy, may the luck be with me | 18:37 |
| rlandy | well - this work is not so bad - so, if the odds were ever in your favor... | 18:37 |
| rlandy | rfolco: https://review.openstack.org/#/c/581376/ | 18:38 |
| rlandy | single node nodeset is already there | 18:38 |
| rfolco | rlandy, yeah should be a 5 min change, 2h testing, and more 15 min to somebody verify it. | 18:41 |
| rlandy | ok | 18:42 |
| rlandy | weshay: so you want to go through those job additions together? then I can complete this work | 18:42 |
| weshay | rlandy, sure | 18:43 |
| weshay | rlandy, blue? | 18:44 |
| rlandy | yeah - probably easier | 18:44 |
| agopi|lunch | rlandy, so just to confirm you need a playbook from us, that will just setup the undercloud for browbeat and then run browbeat. | 18:50 |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ (1 more message) | 19:00 |
| *** agopi|lunch is now known as agopi | 19:01 | |
| agopi | rook, ^ | 19:05 |
| *** atoth has quit IRC | 19:14 | |
| *** gkadam has quit IRC | 19:54 | |
| rlandy | git.openstack.org/openstack/browbeat | 20:07 |
| rlandy | weshay: ^^ does not exit | 20:08 |
| rlandy | github.com/openstack/browbeat does | 20:08 |
| weshay | rlandy, http://git.openstack.org/cgit/openstack/browbeat | 20:09 |
| weshay | http://git.openstack.org/cgit/openstack/tripleo-quickstart | 20:09 |
| rlandy | hmmm - error says it does not exit | 20:11 |
| rlandy | exist | 20:11 |
| rlandy | why??? | 20:11 |
| rlandy | The project "git.openstack.org/openstack/browbeat" was not found. All | 20:11 |
| rlandy | projects referenced within a Zuul configuration must first be added to | 20:11 |
| rlandy | the main configuration file by the Zuul administrator. | 20:11 |
| rlandy | documentation time | 20:15 |
| rlandy | untrusted project? | 20:18 |
| *** holser_ has joined #oooq | 20:21 | |
| hubbot` | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ (1 more message) | 21:00 |
| *** trown is now known as trown|outtypewww | 21:03 | |
| weshay | rlandy, fyi +2 workflowed https://review.openstack.org/#/c/581116/2 | 21:05 |
| weshay | panda|off, ^ | 21:06 |
| rlandy | thank you | 21:07 |
| rlandy | ugh - need to fix other review .. better do that now | 21:07 |
| *** ccamacho has quit IRC | 21:19 | |
| *** jtomasek has quit IRC | 21:20 | |
| rlandy | panda|off: https://review.openstack.org/#/c/581376/ updated | 21:22 |
| *** agopi is now known as agopi|off | 21:39 | |
| *** holser_ has quit IRC | 21:42 | |
| *** agopi|off has quit IRC | 21:44 | |
| *** brault has quit IRC | 21:45 | |
| *** brault has joined #oooq | 22:00 | |
| *** brault has quit IRC | 22:05 | |
| *** agopi has joined #oooq | 22:19 | |
| *** holser_ has joined #oooq | 22:23 | |
| *** agopi has quit IRC | 22:32 | |
| hubbot` | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ (1 more message) | 23:00 |
| *** rlandy has quit IRC | 23:10 | |
| *** tosky has quit IRC | 23:35 | |
| *** holser_ has quit IRC | 23:37 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!