| *** tosky has quit IRC | 00:22 | |
| *** rfolco has joined #oooq | 01:07 | |
| *** d0ugal has quit IRC | 01:24 | |
| *** d0ugal has joined #oooq | 01:35 | |
| *** rfolco has quit IRC | 02:50 | |
| *** rlandy has quit IRC | 03:30 | |
| *** udesale has joined #oooq | 04:03 | |
| *** ykarel|away is now known as ykarel | 04:06 | |
| *** dsneddon has quit IRC | 04:23 | |
| *** skramaja has joined #oooq | 04:55 | |
| *** epoojad1 has joined #oooq | 05:03 | |
| *** epoojad1 has quit IRC | 05:04 | |
| *** saneax has joined #oooq | 05:08 | |
| *** yolanda has quit IRC | 05:24 | |
| *** raukadah is now known as chkumar|rover | 05:37 | |
| *** udesale_ has joined #oooq | 05:45 | |
| *** udesale has quit IRC | 05:48 | |
| *** ratailor has joined #oooq | 05:55 | |
| *** marios has joined #oooq | 06:03 | |
| *** soniya29 has joined #oooq | 06:34 | |
| *** udesale_ has quit IRC | 06:45 | |
| *** udesale has joined #oooq | 06:53 | |
| * chkumar|rover headed wework | 07:32 | |
| *** ykarel is now known as ykarel|lunch | 07:37 | |
| marios | back later | 07:45 |
|---|---|---|
| *** marios has quit IRC | 07:45 | |
| *** dtantsur|afk is now known as dtantsur | 07:56 | |
| *** beagles has quit IRC | 08:06 | |
| chkumar|rover | owalsh, hello | 08:14 |
| chkumar|rover | owalsh, please have a look at inventory issue for this review https://review.opendev.org/#/c/704919/ | 08:15 |
| *** b3nt_pin has joined #oooq | 08:16 | |
| *** amoralej|off is now known as amoralej | 08:24 | |
| *** tesseract has joined #oooq | 08:27 | |
| *** dsneddon has joined #oooq | 08:28 | |
| chkumar|rover | cgoncalves, Hello | 08:30 |
| chkumar|rover | cgoncalves, Anything we can do on this bug https://bugs.launchpad.net/tripleo/+bug/1861685 ? | 08:30 |
| openstack | Launchpad bug 1861685 in tripleo "scenario10 tempest random tempest failures in check / gate, cloud related" [High,Triaged] | 08:30 |
| *** b3nt_pin has quit IRC | 08:30 | |
| cgoncalves | chkumar|rover, hey. so the failure was because Nova took too long to boot a VM. it is not an Octavia issue per se but impact its tests | 08:32 |
| *** dsneddon has quit IRC | 08:33 | |
| cgoncalves | chkumar|rover, my suggestion is to set libvirt type=kvm, cpu_mode=host-passthrough whenever possible (OVH, vexxhost, limestone, fortnebula) and fall back to libvirt type=qemu when not (rackspace) | 08:33 |
| cgoncalves | devstack does this today and there is a patch to further improve it: https://review.opendev.org/#/c/703324/ | 08:34 |
| cgoncalves | a follow-up patch in octavia side is https://review.opendev.org/#/c/702921/ | 08:35 |
| cgoncalves | more context in the commit message | 08:35 |
| chkumar|rover | cgoncalves, great, I will take a look ont hat | 08:35 |
| cgoncalves | chkumar|rover, these two patches are for performance improvements. we'd still need to check what happened in Nova that made the VM not boot up in like +20 minutes | 08:37 |
| *** dsneddon has joined #oooq | 08:38 | |
| chkumar|rover | arxcruz, Hello | 08:38 |
| chkumar|rover | arxcruz, Can we make a proper commit for this patch https://review.opendev.org/#/c/704948/ it works for me | 08:39 |
| *** tosky has joined #oooq | 08:40 | |
| *** ykarel|lunch is now known as ykarel | 08:43 | |
| *** bogdando has joined #oooq | 08:43 | |
| arxcruz | chkumar|rover: hey, what you mean? only this commit? because this is not setting the flavor | 08:47 |
| *** jpena|off is now known as jpena | 08:47 | |
| chkumar|rover | arxcruz, yes updating the commit, but increasing the concurrency solving my timeout issue | 08:48 |
| arxcruz | chkumar|rover: without change the flavor ? | 08:48 |
| arxcruz | chkumar|rover: https://logserver.rdoproject.org/09/24709/10/check/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/b38124d/job-output.txt i'm getting failures... :/ | 08:49 |
| chkumar|rover | arxcruz, do you have the flavor change handy? | 08:49 |
| chkumar|rover | ah no logs | 08:49 |
| arxcruz | chkumar|rover: https://review.rdoproject.org/r/#/c/24709/ | 08:49 |
| *** chem has quit IRC | 08:57 | |
| chkumar|rover | arxcruz, thanks, will take a look | 08:57 |
| arxcruz | chkumar|rover: I can make a proper patch, if without the flavor works, fine | 09:00 |
| *** chem has joined #oooq | 09:00 | |
| owalsh | chkumar|rover: hi, you mean the POST_FAILUREs? | 09:03 |
| owalsh | chkumar|rover: how can that be inventory related, e.g the buildimage jobs failed too | 09:03 |
| arxcruz | chkumar|rover: https://review.opendev.org/#/c/704948/ updated properly | 09:05 |
| owalsh | chkumar|rover: ... and everything except ovb passed on the previous check-rdo | 09:07 |
| owalsh | chkumar|rover: and https://review.opendev.org/704962 passed. I'm guessing we need https://review.opendev.org/704740 to merge first | 09:10 |
| *** apetrich has joined #oooq | 09:11 | |
| owalsh | ... nah, it's already merged. Something borked with log collection? | 09:12 |
| *** jfrancoa has joined #oooq | 09:18 | |
| chkumar|rover | owalsh, looking | 09:41 |
| owalsh | chkumar|rover: the recheck is looking better so far - buildimage jobs are good anyway | 09:42 |
| chkumar|rover | owalsh, weshay|ruck was saying something got broken on rhel8 side | 09:43 |
| owalsh | chkumar|rover: by that patch? | 09:43 |
| *** derekh has joined #oooq | 09:45 | |
| chkumar|rover | owalsh, yes, https://code.engineering.redhat.com/gerrit/#/c/189436/ | 09:47 |
| ykarel | owalsh, There was issue related to log server due to which POST_FAILURE was happening, it's now fixed ~ 2 hour back | 09:48 |
| chkumar|rover | owalsh, https://review.opendev.org/#/c/704919/ check last comment | 09:48 |
| owalsh | chkumar|rover: check-rdo is already running | 09:49 |
| chkumar|rover | owalsh, ah, sorry thanks! | 09:49 |
| owalsh | ykarel: ack, thanks | 09:49 |
| chkumar|rover | ykarel, this fix https://softwarefactory-project.io/r/#/c/17425/ not yet merged? | 09:50 |
| chkumar|rover | zbr, arxcruz https://review.opendev.org/#/q/topic:unskipvolume+(status:open+OR+status:merged) when free, thanks! | 09:51 |
| ykarel | chkumar|rover, Fix is done on system, ^^ is preventive for future | 09:54 |
| chkumar|rover | ykarel, ack | 09:54 |
| arxcruz | chkumar|rover: zbr https://review.opendev.org/#/c/704762/ | 10:02 |
| arxcruz | kopecmartin: around? | 10:02 |
| *** jaosorior has quit IRC | 10:08 | |
| kopecmartin | arxcruz: yes | 10:08 |
| *** ykarel is now known as ykarel|afk | 10:10 | |
| *** marios has joined #oooq | 10:22 | |
| zbr | arxcruz: why moving ara tasks under a folder? | 10:23 |
| arxcruz | zbr: To not mess with the initial structure, then it's easy to know that all ara related yaml files are there | 10:24 |
| zbr | i do not see as a benefit, imho looks like an unnecessary complication on the structure. creating a new level for two files. if we would have >20 task files, or >4-5 ara specific task files it could have make sense, but with only 2, not much. | 10:25 |
| arxcruz | zbr: we don't know in the future | 10:26 |
| zbr | also `ara/ara_` is not DRY | 10:26 |
| zbr | lets do it when the real need arises. | 10:26 |
| zbr | there are only 8 files in that folder. | 10:27 |
| zbr | in fact is much worse, now we have ara code in 2 different folders. | 10:29 |
| arxcruz | zbr: we already following this pattern with collect/\ | 10:29 |
| zbr | well,... nobody askes when was done and collect has 5 files, not two. big difference. | 10:30 |
| zbr | i would say that about collect, i am not sure if is good/bad. | 10:30 |
| arxcruz | i really hate the mess that these task can became if we start to create more files | 10:32 |
| arxcruz | but i want to have the patch merged, so, if that pleases you i'll change | 10:32 |
| zbr | i do also have other arguments against using folders for tasks, makes more problematic to move tasks between them, due to relative-path issue. | 10:32 |
| zbr | with a flat stucture you do not expect surprises when you move one task from a file to another | 10:32 |
| zbr | arxcruz: wait for others to comment, i don't want to impose this on you. i am curios what panda, marios or sshnaidm|afk have to say. | 10:33 |
| zbr | but the old story "if aint broke, don't fix it" :D | 10:34 |
| arxcruz | zbr: this is infrared requirement | 10:34 |
| chkumar|rover | welcome to the world of python3 | 10:34 |
| *** ykarel|afk is now known as ykarel | 10:35 | |
| zbr | sorry but they are welcomed to change ir, but I more inclined to follow official ansible guidelines, or books. | 10:44 |
| *** dtantsur is now known as dtantsur|afk | 10:54 | |
| arxcruz | zbr: i don't see add an option to not run a set of tasks not follow guidelines | 10:56 |
| arxcruz | the structure was my choice | 10:57 |
| arxcruz | if the cores think it should not be in this way, and use flat, i don't care, as far as I get my job done | 10:57 |
| marios | weshay|ruck: panda: i thought we already merged that (wait so is promotion blocked since last week then?) https://review.rdoproject.org/r/#/c/24750/ | 10:59 |
| marios | weshay|ruck: panda: i rechecked now (it has +A but was blocked on gate) ... https://review.rdoproject.org/r/#/c/24771 merged | 10:59 |
| marios | chkumar|rover: is master promotion blocked still then ? no its not... i se it promoted 2nd master and yesterday train | 11:00 |
| chkumar|rover | marios, yes | 11:00 |
| chkumar|rover | marios, fs020 timeout | 11:00 |
| marios | chkumar|rover: yeah i was referring to th emanifest push problem | 11:00 |
| marios | chkumar|rover: that onehttps://bugs.launchpad.net/tripleo/+bug/1861342 | 11:00 |
| openstack | Launchpad bug 1861342 in tripleo "tripleo-ci promotion failing on "pull ppc64le tagged containers"" [Critical,Fix released] - Assigned to Marios Andreou (marios-b) | 11:00 |
| marios | chkumar|rover: and last week wes posted that https://review.rdoproject.org/r/#/c/24750/ which i thought had merged, but it did not | 11:01 |
| chkumar|rover | ah post failure node failure | 11:02 |
| marios | chkumar|rover: panda: actually we don't want to merge it now | 11:02 |
| chkumar|rover | sorry looking into that | 11:02 |
| *** holser has joined #oooq | 11:02 | |
| marios | panda: i -2 that weshay|ruck https://review.rdoproject.org/r/#/c/24750/ we should not need to disable it (but i'm still confused about how we're promoting maybe weshay did something on the promoter itself) | 11:02 |
| *** udesale has quit IRC | 11:04 | |
| marios | panda: chkumar|rover: yeah to answer my question it looks like weshay|ruck did it manually on the promoter | 11:07 |
| marios | [centos@promoter ci-config]$ git diff | 11:07 |
| marios | diff --git a/ci-scripts/dlrnapi_promoter/config/CentOS-7/master.ini b/ci-scripts/dlrnapi_promoter/config/C | 11:07 |
| marios | +manifest_push: false | 11:07 |
| marios | tsk tsk tsk | 11:07 |
| * marios nods dissaprovingly | 11:07 | |
| marios | chkumar|rover: weshay|ruck: so whenever you're ready please change that on promoter so we can check if https://review.rdoproject.org/r/#/c/24771 fixes the problem for that bug | 11:08 |
| *** apetrich has quit IRC | 11:14 | |
| *** tosky has quit IRC | 11:14 | |
| *** ratailor has quit IRC | 11:14 | |
| *** saneax has quit IRC | 11:14 | |
| *** skramaja has quit IRC | 11:14 | |
| *** irclogbot_3 has quit IRC | 11:14 | |
| *** jschlueter has quit IRC | 11:14 | |
| *** dtantsur|afk has quit IRC | 11:14 | |
| *** holser has quit IRC | 11:14 | |
| *** zbr has quit IRC | 11:14 | |
| *** irclogbot_2 has joined #oooq | 11:16 | |
| *** openstackstatus has quit IRC | 11:16 | |
| chkumar|rover | marios, I thought promoter was running code from master | 11:17 |
| *** dtantsur|afk has joined #oooq | 11:17 | |
| *** fuzzball81 has quit IRC | 11:18 | |
| *** dmellado has quit IRC | 11:18 | |
| *** tristanC has quit IRC | 11:18 | |
| *** arxcruz has quit IRC | 11:18 | |
| *** apetrich has joined #oooq | 11:18 | |
| *** tosky has joined #oooq | 11:18 | |
| *** ratailor has joined #oooq | 11:18 | |
| *** saneax has joined #oooq | 11:18 | |
| *** skramaja has joined #oooq | 11:18 | |
| *** jschlueter has joined #oooq | 11:18 | |
| *** holser has joined #oooq | 11:19 | |
| *** zbr has joined #oooq | 11:19 | |
| panda | chkumar|rover: marios yep, that's what I'm investigating | 11:19 |
| marios | chkumar|rover: maybe weshay|ruck deactivated the auto sync too don't know | 11:19 |
| panda | chkumar|rover: marios it's waird that a local change is not overwritten | 11:19 |
| panda | marios: no he didn't | 11:19 |
| marios | panda: k weird then have no idea how that was promoting... | 11:19 |
| marios | panda: i mean he actually made the change in the promoter... and if auto sync was running... ? /me confused | 11:20 |
| panda | marios: I'm investigating, maybe it's related to the lymlink | 11:20 |
| marios | panda: ack ok | 11:20 |
| marios | brb | 11:20 |
| *** marios has quit IRC | 11:20 | |
| *** dmellado has joined #oooq | 11:21 | |
| *** fuzzball81 has joined #oooq | 11:21 | |
| *** tesseract has quit IRC | 11:22 | |
| *** tristanC has joined #oooq | 11:23 | |
| *** tesseract has joined #oooq | 11:24 | |
| *** arxcruz has joined #oooq | 11:26 | |
| *** ChanServ sets mode: +o arxcruz | 11:26 | |
| chkumar|rover | cgoncalves, regarding octavia 10 failure, may be adding this https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/ci/environments/scenario007-multinode-containers.yaml#L77 to host_passthrough in tht standalone scenario 10 will work? | 11:27 |
| chkumar|rover | but it is very hard to distinguish based on cloud | 11:28 |
| *** dsneddon has quit IRC | 11:29 | |
| *** sshnaidm|afk is now known as sshnaidm | 11:31 | |
| cgoncalves | chkumar|rover, right, something has to check if KVM is available first and configure nova accordingly | 11:41 |
| chkumar|rover | sending a patch | 11:42 |
| cgoncalves | it would be a nice config detection to have regardless of octavia job or not. any other tripleo scenario would also benefit from it by running jobs faster | 11:42 |
| cgoncalves | to give an example, non-KVM devstack octavia jobs take ~1h45 whereas KVM enabled jobs take ~1:15 | 11:43 |
| chkumar|rover | cgoncalves, I think that needs to handle on tripleo ci side | 11:44 |
| cgoncalves | chkumar|rover, not necessarily. it could be somewhere in oooq | 11:44 |
| cgoncalves | so that it can also benefit local standalone deployments | 11:44 |
| chkumar|rover | cgoncalves, yes, make sense, if this works https://review.opendev.org/#/c/705638/ I can parameterize on standalone deployment oooq side | 11:45 |
| cgoncalves | chkumar|rover, it will fail if job is scheduled to a RAX nodepool instance | 11:46 |
| cgoncalves | see https://github.com/openstack/devstack/blob/master/lib/nova#L253-L267 | 11:46 |
| chkumar|rover | cgoncalves, good, I think i got the right place to fix and parameterize it https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/standalone/templates/standalone_config.yaml.j2#L32 | 11:49 |
| chkumar|rover | standalone_libvirt_type | 11:49 |
| cgoncalves | chkumar|rover, parameterize and set automatically, yes? | 11:50 |
| chkumar|rover | cgoncalves, yes | 11:50 |
| cgoncalves | chkumar|rover, cpu mode also needs to be set to host-passthrough | 11:50 |
| chkumar|rover | cgoncalves, may be we can borrow the logic from devstack and customize it here? | 11:51 |
| cgoncalves | that's what I have been hitting all along ;) | 11:51 |
| cgoncalves | *hinting | 11:52 |
| *** tesseract has quit IRC | 11:56 | |
| cgoncalves | chkumar|rover, please keep me posted and ping me anytime if you need help | 12:04 |
| chkumar|rover | cgoncalves, sure | 12:04 |
| cgoncalves | hopefully this will have a positive impact on every job that creates VMs on top :) | 12:05 |
| *** zbr is now known as zbr|lunch | 12:06 | |
| chkumar|rover | sshnaidm, Hello | 12:11 |
| sshnaidm | chkumar|rover, hi | 12:11 |
| chkumar|rover | sshnaidm, fs01 rhel8 check job is timing out https://logserver.rdoproject.org/43/702143/23/openstack-check/tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001/13596cf/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz | 12:11 |
| chkumar|rover | sshnaidm, while looking at https://logserver.rdoproject.org/43/702143/23/openstack-check/tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001/13596cf/logs/undercloud/var/log/extra/errors.txt.txt | 12:11 |
| chkumar|rover | sshnaidm, and comparing to fs01 centos 7 2020-02-04 00:51:24.458 ERROR /var/log/containers/ironic/ironic-conductor.log: 8 ERROR ironic.conductor.task_manager [req-f9882ebd-4684-41cb-9507-218727478173 - - - - -] Node 72527767-36f3-4fc5-b10f-188e17b9a0f4 moved to provision state "clean failed" from state "clean wait"; target provision state is "available" | 12:12 |
| chkumar|rover | is appearing odd | 12:12 |
| chkumar|rover | while looking at ironic-conductor log https://logserver.rdoproject.org/43/702143/23/openstack-check/tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001/13596cf/logs/undercloud/var/log/containers/ironic/ironic-conductor.log.txt.gz see this error Invalid completion code received: Invalid command not sure it is related please have a look | 12:12 |
| sshnaidm | chkumar|rover, cleaning failed when tripleo was "providing" them | 12:13 |
| *** amoralej is now known as amoralej|lunch | 12:13 | |
| sshnaidm | chkumar|rover, I see errors like "Error response 0xc1 from Get PICMG Properities" | 12:14 |
| chkumar|rover | sshnaidm, what is causing that | 12:15 |
| sshnaidm | chkumar|rover, this question is for Ironic folks | 12:15 |
| sshnaidm | no idea what that means | 12:15 |
| chkumar|rover | sshnaidm, let me open a bug we can take it from there | 12:15 |
| sshnaidm | chkumar|rover, yeah | 12:15 |
| *** marios has joined #oooq | 12:20 | |
| panda | marios: chkumar|rover | 12:21 |
| panda | Feb 03 00:36:36 promoter dlrn-promoter-service.sh[15735]: error: unable to unlink old 'ci-scripts/dlrnapi_promoter/config/CentOS-7/master.ini' (Operation not permitted) | 12:21 |
| panda | Feb 03 00:36:36 promoter dlrn-promoter-service.sh[15735]: fatal: Could not reset index file to revision 'origin/master'. | 12:21 |
| weshay|ruck | panda, ah.. let me fix that | 12:21 |
| marios | panda: so we were saved by permissions issue ;) | 12:22 |
| weshay|ruck | panda, ok.. should be fixed now | 12:25 |
| marios | weshay|ruck: back to running master ? | 12:26 |
| *** tesseract has joined #oooq | 12:26 | |
| weshay|ruck | marios, running master w/ multiarch you mean | 12:27 |
| weshay|ruck | ? | 12:27 |
| marios | weshay|ruck: yeah and push enabld | 12:27 |
| marios | weshay|ruck: noticd earlier that https://review.rdoproject.org/r/#/c/24750/ wasn't merged and checked on promoter you made the edit manually there | 12:28 |
| marios | weshay|ruck: but then we were confused cos it should be autsyncing | 12:28 |
| marios | (from master) | 12:29 |
| *** tesseract has quit IRC | 12:29 | |
| marios | weshay|ruck: looks like it was an issue with the symlink per panda ping 14:21 < panda> Feb 03 00:36:36 promoter dlrn-promoter-service.sh[15735]: error: unable to unlink old | 12:29 |
| weshay|ruck | ya.. I chattr'd the file | 12:29 |
| weshay|ruck | now it's writable | 12:30 |
| marios | weshay|ruck: oh i see that's one way around the autosync ;) | 12:30 |
| weshay|ruck | :) | 12:30 |
| *** jpena is now known as jpena|lunch | 12:31 | |
| *** rfolco has joined #oooq | 12:31 | |
| *** tesseract has joined #oooq | 12:31 | |
| chkumar|rover | weshay|ruck, sshnaidm bug logged for fs01 rhel8 time out https://bugs.launchpad.net/tripleo/+bug/1861802 | 12:32 |
| openstack | Launchpad bug 1861802 in tripleo "overcloud prepare image step time out with moved to provision state "clean failed" from state "clean wait"; on rhel8 fs01 master" [High,Confirmed] | 12:32 |
| weshay|ruck | chkumar|rover++ | 12:33 |
| weshay|ruck | chkumar|rover, I think we can turn clean off | 12:33 |
| chkumar|rover | weshay|ruck, but how? | 12:36 |
| chkumar|rover | brb | 12:36 |
| *** tesseract has quit IRC | 12:37 | |
| *** tesseract has joined #oooq | 12:38 | |
| sshnaidm | weshay|ruck, "providing" contains node clean iirc | 12:40 |
| sshnaidm | and iirc it's part of mistral workflow to provide.. exactly going to convert it :) | 12:40 |
| *** tesseract has quit IRC | 12:40 | |
| *** jpena|lunch is now known as jpena|off | 12:40 | |
| *** tesseract has joined #oooq | 12:40 | |
| weshay|ruck | sshnaidm, ya.. it's been a while since I looked at this.. but I think there is a switch.. could be wrong | 12:41 |
| weshay|ruck | sshnaidm, chkumar|rover the latest two rhel8 ovb fs001 jobs in promotion passed | 12:42 |
| weshay|ruck | lovely inconsistent issues | 12:42 |
| weshay|ruck | chkumar|rover, fyi.. just logged https://bugs.launchpad.net/tripleo/+bug/1861803 | 12:42 |
| openstack | Launchpad bug 1861803 in tripleo "ubuntu-bionic | ERROR: Could not find a version that satisfies the requirement oslo.concurrency===4.0.0 (from -c /home/zuul/src/opendev.org/openstack/requirements/upper-constraints.txt " [Critical,Triaged] | 12:42 |
| weshay|ruck | https://github.com/openstack/requirements/commit/0c5ace0ec8ac670b6cbeb64fb8577d11adc90bc5#diff-0bdd949ed8a7fdd4f95240bd951779c8 | 12:43 |
| *** ratailor has quit IRC | 12:45 | |
| owalsh | weshay|ruck: hi, https://review.opendev.org/704919 is green, could you remove the -W? | 12:48 |
| weshay|ruck | owalsh, \0/ | 12:49 |
| weshay|ruck | look at that | 12:49 |
| weshay|ruck | owalsh, was that just rdo-cloud messing w/ our heads?? | 12:50 |
| owalsh | weshay|ruck: yup, it had already passed in the depends-on job | 12:50 |
| weshay|ruck | owalsh, also ran on that real bm box.. worked there too | 12:50 |
| marios | sshnaidm: so https://review.opendev.org/#/c/705446/ is not related to https://bugs.launchpad.net/tripleo/+bug/1861694?comments=all ? you pointed at that bug when i asked last night and i added comment #1 but i think its wrong? | 12:50 |
| openstack | Launchpad bug 1861694 in tripleo "Nonstop restarting ovn_metadata_haproxy container" [Critical,Triaged] | 12:50 |
| weshay|ruck | owalsh++ | 12:50 |
| *** tesseract has quit IRC | 12:50 | |
| marios | sshnaidm: hmm maybe it is related actually not sure (but i noticed you have a different bug on https://review.opendev.org/#/c/705446/ | 12:51 |
| sshnaidm | marios, it is related | 12:51 |
| marios | sshnaidm: k thanks for sanity check | 12:51 |
| sshnaidm | marios, the patch is coming to fix logs but doesn't solve the whole problem, which is described in bug | 12:51 |
| owalsh | weshay|ruck: do we need to poke https://review.opendev.org/704880 to get it in the gate? | 12:52 |
| marios | sshnaidm: ack fine thanks, i was only looking as i was added to the trello card (prod chain) i guess because of my comment and wanted to make sure it wasn't just confusing things | 12:52 |
| marios | sshnaidm: i will add you to the trello card | 12:52 |
| weshay|ruck | owalsh, sec | 12:53 |
| *** ykarel is now known as ykarel|afk | 12:55 | |
| *** rlandy has joined #oooq | 12:58 | |
| owalsh | nevermind, I see the gate is blocked | 13:00 |
| weshay|ruck | heh | 13:00 |
| weshay|ruck | chkumar|rover, does this fix look right to you? https://bugs.launchpad.net/tripleo/+bug/1861803 | 13:01 |
| openstack | Launchpad bug 1861803 in tripleo "ubuntu-bionic | ERROR: Could not find a version that satisfies the requirement oslo.concurrency===4.0.0 (from -c /home/zuul/src/opendev.org/openstack/requirements/upper-constraints.txt " [Critical,Triaged] | 13:01 |
| weshay|ruck | chkumar|rover, need it for tripleo-common as well if that's the right thing | 13:01 |
| *** tesseract has joined #oooq | 13:03 | |
| sshnaidm | weshay|ruck, mtg? | 13:03 |
| *** tesseract has quit IRC | 13:07 | |
| chkumar|rover | sshnaidm, weshay|ruck https://review.opendev.org/#/c/704948/ can we get this merged | 13:08 |
| *** jfrancoa has quit IRC | 13:09 | |
| sshnaidm | chkumar|rover, 020 job better to run on featureset020.yml changes | 13:12 |
| chkumar|rover | sshnaidm, taking a look on that | 13:12 |
| weshay|ruck | sshnaidm, ping me to 1-1 if you want | 13:13 |
| sshnaidm | weshay|ruck, when? later today? | 13:14 |
| weshay|ruck | now, later.. what ever | 13:14 |
| sshnaidm | weshay|ruck, maybe later, the community mtg in 15 mins | 13:14 |
| *** tesseract has joined #oooq | 13:15 | |
| *** amoralej|lunch is now known as amoralej | 13:20 | |
| *** jfrancoa has joined #oooq | 13:25 | |
| chkumar|rover | weshay|ruck, we are merging tempest-23.0.0 and neutron tempest plugin 0.8.0 for train, you might see some little hiccups | 13:25 |
| chkumar|rover | we have both fs020 and full tempest passing | 13:25 |
| chkumar|rover | but still | 13:25 |
| *** dsneddon has joined #oooq | 13:26 | |
| *** jpena|off is now known as jpena | 13:28 | |
| weshay|ruck | chkumar|rover, k.. thanks | 13:28 |
| rfolco | chkumar|rover, weshay|ruck panda zbr|lunch sshnaidm rlandy marios arxcruz: community mtg in 1 min at https://meet.google.com/bqx-xwht-wky?authuser=1 | 13:28 |
| rfolco | https://hackmd.io/IhMCTNMBSF6xtqiEd9Z0Kw?both | 13:29 |
| rfolco | @L56 | 13:29 |
| *** marios is now known as marios|call | 13:30 | |
| *** dsneddon has quit IRC | 13:31 | |
| weshay|ruck | chkumar|rover, hrm.. that py27 patch failed on upper constraints again | 13:31 |
| chkumar|rover | weshay|ruck, which one | 13:32 |
| weshay|ruck | https://review.opendev.org/#/c/705665/ | 13:32 |
| chkumar|rover | ERROR: Could not find a version that satisfies the requirement oslo.concurrency===4.0.0 | 13:34 |
| chkumar|rover | letme find the solution | 13:34 |
| chkumar|rover | weshay|ruck, at all places http://codesearch.openstack.org/?q=oslo.concurrency%3D%3D&i=nope&files=&repos= | 13:36 |
| chkumar|rover | weshay|ruck, oslo.concurrency==3.26.0 | 13:37 |
| *** b3nt_pin has joined #oooq | 13:37 | |
| *** jfrancoa has quit IRC | 13:38 | |
| chkumar|rover | sorry each one is maintaining a lower contrients file | 13:38 |
| chkumar|rover | it is coming from uc, I think we need to disable that | 13:41 |
| *** ykarel|afk is now known as ykarel | 13:41 | |
| weshay|ruck | chkumar|rover, -c{env:UPPER_CONSTRAINTS_FILE:https://opendev.org/openstack/requirements/raw/branch/master/upper-constraints.txt} | 13:45 |
| weshay|ruck | that bit? | 13:45 |
| chkumar|rover | weshay|ruck, yes, it is pulling from uc | 13:47 |
| weshay|ruck | chkumar|rover, k.. put up another patch on tht | 13:47 |
| weshay|ruck | w/o that line | 13:47 |
| chkumar|rover | weshay|ruck, it worked on removing that | 13:50 |
| weshay|ruck | running locally I didn't hit the issue | 13:51 |
| weshay|ruck | tox py27 ? right | 13:51 |
| chkumar|rover | weshay|ruck, yes | 13:52 |
| *** jfrancoa has joined #oooq | 13:52 | |
| chkumar|rover | weshay|ruck, I triggered tox -e py27 locally on removing uc it worked | 13:52 |
| chkumar|rover | on tht | 13:52 |
| weshay|ruck | k | 13:53 |
| weshay|ruck | chkumar|rover, you think we can configure tox.ini to skip uc for just py27? | 13:53 |
| * weshay|ruck is a newb w/ tox | 13:54 | |
| chkumar|rover | weshay|ruck, I think we need a patch here https://github.com/openstack/requirements/blob/master/upper-constraints.txt | 13:55 |
| weshay|ruck | chkumar|rover, ah.. mark 4.0.0 for not py27 | 13:56 |
| chkumar|rover | proposing a patch | 13:57 |
| chkumar|rover | weshay|ruck, https://review.opendev.org/705685 | 14:00 |
| ykarel | chkumar|rover, weshay|ruck more such breakage releases are coming, so need to think off some way to not break on each update | 14:04 |
| ykarel | example oslo.log it's still under review | 14:04 |
| ykarel | good luck swift py2 job is running and using it, so it's not merged yet | 14:04 |
| chkumar|rover | ykarel, https://review.opendev.org/#/c/705685/ | 14:05 |
| ykarel | chkumar|rover, you planning to same review for each upcoming u-c update? | 14:05 |
| chkumar|rover | ykarel, yes, I am seeing multiple entires there | 14:05 |
| chkumar|rover | in uc | 14:05 |
| ykarel | or proactively look all u-c update and fix before they land | 14:06 |
| chkumar|rover | I am not seeing other way around | 14:06 |
| ykarel | what about that wes suggested use u-c only for py27 | 14:07 |
| ykarel | not use u-c for py27 | 14:07 |
| *** marios|call is now known as marios | 14:07 | |
| ykarel | but that also have side-effect if it needs to be get into other tripleo projects | 14:07 |
| chkumar|rover | ok will take a look | 14:08 |
| * chkumar|rover headed home | 14:08 | |
| ykarel | ack | 14:08 |
| *** soniya29 has quit IRC | 14:09 | |
| *** ykarel is now known as ykarel|afk | 14:12 | |
| *** dsneddon has joined #oooq | 14:16 | |
| *** dsneddon has quit IRC | 14:20 | |
| *** dsneddon has joined #oooq | 14:21 | |
| *** udesale has joined #oooq | 14:22 | |
| *** dsneddon has quit IRC | 14:26 | |
| *** dtantsur|afk is now known as dtantsur | 14:27 | |
| *** TrevorV has joined #oooq | 14:32 | |
| weshay|ruck | chkumar|rover, so should we move requirements check off of tht and tripleo-common | 14:35 |
| weshay|ruck | the job and the tox line? | 14:35 |
| rfolco | weshay|ruck, can I takeover https://review.opendev.org/#/c/703125 | 14:35 |
| weshay|ruck | rfolco, k.. thanks | 14:35 |
| *** jaosorior has joined #oooq | 14:40 | |
| mjturek | rfolco: FYI baha and I are seeing this when we try to start locally built containers http://paste.openstack.org/show/789111/ | 14:54 |
| rfolco | mjturek, how did you install kolla ? pip? dnf ? python setup ? | 14:56 |
| rfolco | mjturek, I just clone it and do pip install -e kolla/ | 14:56 |
| mjturek | rfolco: however the tripleo job does it | 14:56 |
| mjturek | rfolco where? on the host or the container? | 14:57 |
| chkumar|rover | weshay|ruck, working on that | 14:57 |
| mjturek | isn't the error referring to a file that should be on the container? | 14:57 |
| rfolco | no this is kolla config | 15:00 |
| weshay|ruck | chkumar|rover, sync or keep it to irc? | 15:01 |
| *** rascasoft has joined #oooq | 15:01 | |
| chkumar|rover | weshay|ruck, give me few mins | 15:02 |
| *** jtomasek has joined #oooq | 15:02 | |
| rfolco | mjturek, I don't have this file either in my manual installation | 15:02 |
| mjturek | rfolco: But to be clear. It's supposed to be in the containers | 15:03 |
| mjturek | are you checking in a container | 15:03 |
| rfolco | does kolla run this inside the container ? | 15:04 |
| rfolco | sudo -E kolla_set_configs | 15:04 |
| *** zbr|lunch is now known as zbr | 15:04 | |
| zbr | i am back | 15:05 |
| mjturek | rfolco: https://github.com/openstack/kolla/blob/master/docker/base/Dockerfile.j2#L498 | 15:05 |
| *** dtantsur is now known as dtantsur|afk | 15:06 | |
| rfolco | mjturek, I am sorry, you are right... let me check my base containre | 15:07 |
| mjturek | rfolco np, thanks | 15:07 |
| *** dsneddon has joined #oooq | 15:07 | |
| chkumar|rover | weshay|ruck, https://review.opendev.org/705665 | 15:09 |
| rlandy | sshnaidm: wrt metalsmith issue ... what's your opinion on the best way to go? I could check release as well or I could try the 'list_instances' command and the check | 15:09 |
| chkumar|rover | weshay|ruck, let'see it will work or not | 15:09 |
| rlandy | chkumar|rover: how are you doing with the podman v1.6 work? | 15:09 |
| chkumar|rover | rlandy, left it in mid due to gate issues | 15:10 |
| chkumar|rover | rlandy, will come back on that soon | 15:10 |
| rlandy | chkumar|rover: ack - will catch up another time | 15:10 |
| chkumar|rover | rlandy, sorry for that | 15:10 |
| rlandy | chkumar|rover: no worries, trying to finish the TLS work now | 15:10 |
| rlandy | I'm getting closer :) | 15:10 |
| *** skramaja has quit IRC | 15:11 | |
| chkumar|rover | rlandy, :-) | 15:11 |
| rfolco | mjturek, re-running my build, it may take a while... will get back to you when I have the base container built. | 15:12 |
| sshnaidm | rlandy, I wonder if we can a better way to understand that nova isn't deployed.. | 15:12 |
| rlandy | sshnaidm: I'm pretty sure nova is deployed | 15:12 |
| sshnaidm | rlandy, the easiest way will be to check releases maybe.. | 15:13 |
| rlandy | but metalsmith might be available | 15:13 |
| rlandy | checking that collect logs gets the right release passed | 15:13 |
| sshnaidm | rlandy, as I understand you need to run list_instances in ussuri only, right? | 15:14 |
| rlandy | and ... in other news ... case of corona virus reported in Flushing - which is about 5 blocks from where I live - no need to travel for that | 15:14 |
| rlandy | sshnaidm: yeah | 15:14 |
| rlandy | only implemented there | 15:14 |
| rlandy | idk why only rocket failed out | 15:14 |
| rlandy | rocky | 15:14 |
| rlandy | queens and stein did not | 15:15 |
| *** dsneddon has quit IRC | 15:17 | |
| mjturek | cool thanks rfolco | 15:17 |
| sshnaidm | rlandy, so either to check if list_instances exists (or wrap with try/except it) or to check release | 15:20 |
| sshnaidm | rlandy, maybe something like that: http://paste.openstack.org/show/789113/ | 15:20 |
| sshnaidm | rlandy, but because we use novaless from ussuri only, I'd add release check as well | 15:21 |
| rlandy | sshnaidm: I think I'll do both | 15:23 |
| rlandy | since there is import available | 15:23 |
| *** Goneri has joined #oooq | 15:23 | |
| sshnaidm | rlandy, ack | 15:26 |
| *** jtomasek has quit IRC | 15:27 | |
| mjturek | rfolco baha: I think we might just be running the container improperly https://docs.openstack.org/kolla/stein/admin/kolla_api.html#passing-the-configuration-file-to-the-container | 15:27 |
| chkumar|rover | rlandy, sshnaidm can you take a look at this https://review.opendev.org/#/c/705631/ I am not sure why it is not working | 15:28 |
| zbr | sshnaidm: weshay|ruck : the standalone move, w/o any other bits: https://review.opendev.org/#/c/705415/ | 15:31 |
| rfolco | mjturek, something might be wrong with your kolla setup that it does not copy the config into the container | 15:32 |
| zbr | sshnaidm: using exit does not exit if used in subshell, tested. | 15:32 |
| *** jtomasek has joined #oooq | 15:32 | |
| sshnaidm | chkumar|rover, The error was: 'ansible_python_interpreter' is undefined : https://65b3e7be3a88d77f93ba-f7b9b971234e50ac40b8e4d23dbcc3e3.ssl.cf5.rackcdn.com/705631/1/check/tripleo-ci-centos-7-scenario007-multinode-oooq-container/65083ef/logs/quickstart_collect_logs.log | 15:32 |
| weshay|ruck | zbr, k.. thanks!! will review it now.. note the gate is blocked due to infra | 15:32 |
| weshay|ruck | and py27 | 15:32 |
| weshay|ruck | FYI.. all https://review.opendev.org/#/c/705665/7 | 15:33 |
| weshay|ruck | hold off on WF for now | 15:33 |
| mjturek | rfolco but it looks like it[s referenced in the docker run cimmand | 15:33 |
| sshnaidm | zbr, ack | 15:33 |
| sshnaidm | chkumar|rover, I'd run both pip list and pip3 list | 15:33 |
| chkumar|rover | then it will pip, pip2 and pip3? | 15:34 |
| zbr | sshnaidm: not that I do like the exit in functions myself, but just wanted to avoid touching anything that was not mandatory. | 15:34 |
| zbr | if can be a mess if you try to source the file using the exits. | 15:34 |
| sshnaidm | zbr, well, return is more logical to use in functions | 15:34 |
| zbr | not arguing against it, but we will do a follow-up to more fixes after we have the script. | 15:35 |
| sshnaidm | zbr, I added exits only because it was an ansible task | 15:35 |
| zbr | shellcheck is great tool for improving shell scripts, much better than bashate. | 15:35 |
| sshnaidm | chkumar|rover, maybe, we can have pip-list, pip3-list, pip2-list :) | 15:35 |
| zbr | that file looked like xmas tree when I opened it on vscode, (due to warnings) | 15:36 |
| chkumar|rover | zbr, use emacs | 15:36 |
| zbr | chkumar|rover: you can add the same linters in vi,emails, i am sure. | 15:37 |
| zbr | i guess nobosy using notepad here :P | 15:37 |
| chkumar|rover | notepad++ may be windows user | 15:37 |
| weshay|ruck | chkumar|rover, friggin queue | 15:38 |
| chkumar|rover | weshay|ruck, yes I am watching that | 15:38 |
| sshnaidm | zbr, don't you need to set an "executable: bash" here? https://review.opendev.org/#/c/705415/4/tasks/collect/container.yml | 15:38 |
| weshay|ruck | chkumar|rover, /me wishes we could have a check job on the upper constraints changes | 15:39 |
| chkumar|rover | zbr, by the way tox sucks for if else need to take a look at nox what it provides | 15:39 |
| zbr | lets not get there, only yesterday i was send a public "f***" on twitter from one of the nox mainterns. | 15:40 |
| chkumar|rover | hmm ok | 15:40 |
| zbr | they were offended by the fact that I argued against their use of calver instead of semver. | 15:40 |
| zbr | chkumar|rover: nox may be good, didn't look deep myself but tox is used everywhere, and does not depend on a very select group of maintainers. | 15:41 |
| zbr | is not only about the tool, is also about the community around it | 15:41 |
| chkumar|rover | zbr, i saw use of nox from httpx project, a upcoming rival of requests | 15:42 |
| chkumar|rover | in httpx, i really liked context manager usage | 15:42 |
| zbr | chkumar|rover: show me first openstack projects adopting it.... | 15:42 |
| chkumar|rover | zbr, tough question | 15:42 |
| zbr | i wonder what infra will say when you will propose openstack-nox-* jobs ;) | 15:43 |
| *** Trevor_V has joined #oooq | 15:43 | |
| *** jaosorior has quit IRC | 15:43 | |
| chkumar|rover | 34 mins and still counting | 15:44 |
| *** TrevorV has quit IRC | 15:44 | |
| *** Trevor_V is now known as TrevorV | 15:44 | |
| chkumar|rover | weshay|ruck, in the mean time, can we get some eyes on this https://bugs.launchpad.net/tripleo/+bug/1861802 ? | 15:46 |
| openstack | Launchpad bug 1861802 in tripleo "overcloud prepare image step time out with moved to provision state "clean failed" from state "clean wait"; on rhel8 fs01 master" [High,Confirmed] | 15:46 |
| chkumar|rover | or want to make cix | 15:46 |
| *** sshnaidm is now known as sshnaidm|afk | 15:46 | |
| sshnaidm|afk | zbr, commented https://review.opendev.org/#/c/705415/ | 15:47 |
| zbr | sshnaidm|afk: it has a shebang that points to bash, no real need. | 15:56 |
| zbr | anyway, we will find and fix issue really soon, as soon arx realise that he can add new ubuntu container to molecule. | 15:57 |
| zbr | i bet the role will choke in many places while doing this, but is not hard to fix. | 15:57 |
| *** ykarel|afk is now known as ykarel | 15:58 | |
| *** bogdando has quit IRC | 16:00 | |
| *** udesale has quit IRC | 16:00 | |
| *** ysandeep is now known as ysandeep|away | 16:07 | |
| weshay|ruck | chkumar|rover, these are hard w/ regards to introspection because often it's the cloud at fault | 16:10 |
| weshay|ruck | give me a few to review the status of rhel8 ovb fs001 because it was passing in promotion | 16:10 |
| chkumar|rover | weshay|ruck, ok | 16:10 |
| weshay|ruck | chkumar|rover, we need to get to vexxhost | 16:10 |
| chkumar|rover | weshay|ruck, can you also take a look at failed to delete stacks, grafana is alerting today | 16:11 |
| weshay|ruck | chkumar|rover, ya.. that's also a signal that the cloud is f.. | 16:12 |
| weshay|ruck | chkumar|rover, this looks normal though http://dashboard-ci.tripleo.org/d/cockpit/cockpit?orgId=1&fullscreen&panelId=231 | 16:13 |
| weshay|ruck | or not extrodinary | 16:13 |
| * weshay|ruck looks at use | 16:13 | |
| mjturek | rfolco: any luck building the containers? | 16:13 |
| weshay|ruck | chkumar|rover, probably has to do w/ use..http://dashboard-ci.tripleo.org/d/cockpit/cockpit?orgId=1&fullscreen&panelId=175 | 16:14 |
| chkumar|rover | weshay|ruck, may be we can get rid of 6 error stacks? | 16:15 |
| rfolco | mjturek, its running fine since... | 16:15 |
| mjturek | rfolco: what command did you run? | 16:15 |
| weshay|ruck | chkumar|rover, I see 3 | 16:16 |
| rfolco | mjturek, manual build, installed kolla with pip install -e kolla/ | 16:16 |
| rfolco | https://hackmd.io/dSagCbocQ4KSVEZR1uf8Tw | 16:16 |
| weshay|ruck | which is some blip in their db | 16:16 |
| weshay|ruck | he can't get rid of them | 16:16 |
| weshay|ruck | where do you see 6? | 16:16 |
| chkumar|rover | weshay|ruck, http://dashboard-ci.tripleo.org/d/cockpit/cockpit?orgId=1&fullscreen&panelId=175 | 16:16 |
| chkumar|rover | rdocloud servers error 6 | 16:16 |
| chkumar|rover | sorry not stack | 16:16 |
| weshay|ruck | ya.. 6 instances | 16:16 |
| mjturek | rfolco: sorry I mean did you run a container | 16:16 |
| weshay|ruck | chkumar|rover, sec | 16:17 |
| chkumar|rover | weshay|ruck, gmeet? | 16:17 |
| rfolco | mjturek, thats the problem, I don't want to screw it up | 16:17 |
| mjturek | rfolco got it so waiting for the build to complete? | 16:17 |
| rfolco | mjturek, yes, sorry, I am also under pressure to complete this task | 16:17 |
| mjturek | okay, let me know when the containers finish building for you | 16:18 |
| mjturek | rfolco ^ | 16:18 |
| rlandy | sshnaidm|afk: on second thoughts, I didn't use release - in case there is a plan for back port | 16:18 |
| weshay|ruck | chkumar|rover, /me reviewing https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001 and https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master and https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-train# | 16:18 |
| rfolco | mjturek, will do. Do you run our workflow to install kolla in your job ? | 16:18 |
| chkumar|rover | weshay|ruck, I looked at check jobs | 16:19 |
| mjturek | I believe so but will double check rfolco | 16:19 |
| weshay|ruck | chkumar|rover, let's hold on rhel8 fs001 until we promote master | 16:20 |
| chkumar|rover | weshay|ruck, ok | 16:20 |
| weshay|ruck | chkumar|rover, one can chase your tail on this shit | 16:20 |
| chkumar|rover | weshay|ruck, ok | 16:20 |
| weshay|ruck | promoter SUCCESS promoting rhel8-master tripleo-ci-testing as current-tripleo ({'timestamp': 1580803917, 'distro_hash': '21e2b632e857b374ad130a5b2aa26d816f6669f5', 'promote_name': 'tripleo-ci-testing', 'user': 'review_rdoproject_org', 'repo_url': 'http://trunk.rdoproject.org/rhel8-master/4d/ea/4dea8060f05fab427cd042934674f08b03927e05_21e2b632', 'full_hash': '4dea8060f05fab427cd042934674f08b03927e05_21e2b632', 'repo_hash': | 16:21 |
| weshay|ruck | '4dea8060f05fab427cd042934674f08b03927e05_21e2b632', 'commit_hash': '4dea8060f05fab427cd042934674f08b03927e05'}) | 16:21 |
| weshay|ruck | 2020-02-04 12:41:25,956 27202 INFO promoter DETAILED SUCCESSFUL STATUS: | 16:21 |
| weshay|ruck | https://trunk.rdoproject.org/api-rhel8-master/api/civotes_detail.html?commit_hash=4dea8060f05fab427cd042934674f08b03927e05&distro_hash=21e2b632e857b374ad130a5b2aa26d816f6669f5 | 16:21 |
| weshay|ruck | chkumar|rover, ^ | 16:21 |
| weshay|ruck | let's see if that improves the success rate | 16:21 |
| chkumar|rover | ok | 16:21 |
| weshay|ruck | chkumar|rover, ya.. the latest timedout jobs were using baseurl=http://trunk.rdoproject.org/rhel8-master/32/93/32930d0440503a746995915cab8b9cc9378a7da2_10e135ca | 16:22 |
| * weshay|ruck checks criteria | 16:22 | |
| chkumar|rover | weshay|ruck, are we still cherry-picking stuff on promoter server? | 16:23 |
| weshay|ruck | nothing remarked out | 16:23 |
| weshay|ruck | no | 16:23 |
| chkumar|rover | ok | 16:23 |
| weshay|ruck | chkumar|rover, we also need the rhel8 job to run instead of centos7 | 16:24 |
| weshay|ruck | imho | 16:24 |
| weshay|ruck | any where we run centos-7 fs001 we could be running rhel8 | 16:24 |
| weshay|ruck | haven't gotten to that though yet | 16:24 |
| chkumar|rover | weshay|ruck, ok, will do that tomorrow | 16:24 |
| weshay|ruck | and needs to be more stable | 16:24 |
| weshay|ruck | so we'll get it.. maybe this week.. we'll watch | 16:24 |
| weshay|ruck | chkumar|rover, let's chat aobut py27 | 16:24 |
| chkumar|rover | weshay|ruck, yes | 16:24 |
| weshay|ruck | https://meet.google.com/rzx-zybz-fof?authuser=1 | 16:24 |
| weshay|ruck | rfolco, check in on ceph for me please | 16:42 |
| weshay|ruck | centos8 | 16:42 |
| rfolco | weshay|ruck, if ceph built | 16:42 |
| rfolco | ? | 16:42 |
| weshay|ruck | rfolco, if you have a repo and if it built | 16:45 |
| weshay|ruck | yes | 16:45 |
| weshay|ruck | need to know today | 16:45 |
| *** jpena is now known as jpena|brb | 16:45 | |
| zbr | many years ago there was a file with metadata about current dir on http/ftp servers, but I do not remember the name of the file. | 16:46 |
| zbr | i think it used to have lst extension or something like this | 16:46 |
| zbr | does anyone remember? | 16:46 |
| zbr | i know that apache used it to display file description from it | 16:46 |
| rfolco | weshay|ruck, dammn did not build... I checked kolla patch and latest ps did not exclude ceph... checking why it got skipped | 16:47 |
| weshay|ruck | k.. thanks | 16:47 |
| rfolco | mjturek, will check base container in a bit | 16:47 |
| mjturek | thanks rfolco let us know | 16:47 |
| chkumar|rover | see ya tomorrow people | 16:48 |
| *** chkumar|rover is now known as raukadah | 16:48 | |
| rfolco | weshay|ruck, oh I forgot to reinstall kolla after commenting out ceph/etcd/collectd excludes, will re-run | 16:52 |
| rfolco | the ceph repo is injected in the base container w/ COPY in dockerfile | 16:53 |
| marios | weshay|ruck: rlandy: unconditional rhel7 enabling https://code.engineering.redhat.com/gerrit/gitweb?p=kolla.git;a=blob;f=docker/base/Dockerfile.j2;h=41d16c7087d3edfae2089ad64298b293f8546bcc;hb=refs/heads/rhos-17.0-trunk-patches#l303 | 16:54 |
| rlandy | marios? | 16:55 |
| *** Trevor_V has joined #oooq | 16:56 | |
| raukadah | weshay|ruck, please a single topic for py27 removal so that we can tackle easily | 16:56 |
| weshay|ruck | raukadah, I did | 16:56 |
| weshay|ruck | it's in the etherpad | 16:56 |
| raukadah | weshay|ruck, thanks :-) | 16:57 |
| marios | rlandy: trying to build just one of those rhos17 kolla containers as per our discussion ysterday | 16:58 |
| marios | rlandy: and came across that in the base dockerfile | 16:58 |
| *** jfrancoa has quit IRC | 16:58 | |
| marios | rlandy: just pointing to one plac that has rhel7 unconditionally | 16:58 |
| rlandy | marios: yep - strange because they have been on rhel8 for a while | 16:59 |
| *** TrevorV has quit IRC | 16:59 | |
| rlandy | marios: been busy trying to finish up the tls work ... will look at the 16 builds this afternoon | 16:59 |
| rlandy | marios: question for jjoyce? | 17:00 |
| marios | rlandy: ack but tomorrow /me going in a bit | 17:02 |
| rlandy | marios: k - have a good night | 17:03 |
| *** tesseract has quit IRC | 17:04 | |
| marios | you too have a good one rlandy | 17:05 |
| raukadah | weshay|ruck, rlandy https://review.opendev.org/#/q/topic:unskipvolume+(status:open+OR+status:merged) are good to go, dependent patches got merged | 17:08 |
| *** dsneddon has joined #oooq | 17:13 | |
| *** ykarel is now known as ykarel|away | 17:13 | |
| rlandy | no logs on fs020 :( | 17:16 |
| raukadah | rlandy, where, try rdo-check | 17:16 |
| raukadah | logserver health was bad it might be healthy now | 17:16 |
| rlandy | https://review.opendev.org/#/c/703936/ | 17:17 |
| rlandy | https://review.opendev.org/#/c/703868/6 fine fs001 tests are passing | 17:17 |
| *** dsneddon has quit IRC | 17:18 | |
| raukadah | rlandy, let me rebase all, we can +w tomorrow | 17:18 |
| rlandy | raukadah: fine by me | 17:19 |
| *** marios is now known as marios|out | 17:23 | |
| *** jpena|brb is now known as jpena | 17:24 | |
| *** marios|out has quit IRC | 17:34 | |
| *** derekh has quit IRC | 18:00 | |
| *** tosky has quit IRC | 18:01 | |
| *** dsneddon has joined #oooq | 18:01 | |
| *** fuzzball81 is now known as jjoyce | 18:06 | |
| jjoyce | rlandy marios: Those lines are not in the generated Dockerfile. Note the if base_distro == 'rhel' I am guessing rhel8 fails that. | 18:07 |
| *** TrevorV has joined #oooq | 18:13 | |
| *** amoralej is now known as amoralej|off | 18:14 | |
| *** Trevor_V has quit IRC | 18:16 | |
| *** rascasoft has quit IRC | 18:19 | |
| *** dsneddon has quit IRC | 18:26 | |
| *** jpena is now known as jpena|off | 18:37 | |
| mjturek | rfolco: any luck? | 18:46 |
| rfolco | mjturek, about to run the command | 18:46 |
| rfolco | mjturek, the command itself also fails for me... looking at dockerfile to see if any previous steps may be required | 18:48 |
| mjturek | same failure?? | 18:49 |
| mjturek | (and x86 right?) | 18:49 |
| rfolco | mjturek, but I suspect something else happens during the build | 18:56 |
| *** dtantsur|afk has quit IRC | 19:00 | |
| weshay|ruck | rfolco, https://review.rdoproject.org/r/#/c/24775/ is no longer triggering the job | 19:01 |
| rfolco | weshay|ruck, yes, I need to add projects.yaml back | 19:01 |
| rfolco | weshay|ruck, but I am trying to fix tripleo-repos first | 19:01 |
| rfolco | without tripleo-repos fixed there is no point in running it again | 19:02 |
| *** dtantsur|afk has joined #oooq | 19:02 | |
| weshay|ruck | rfolco, k.. cool.. I'll look at the stuff past tripleo-repos | 19:02 |
| weshay|ruck | thanks | 19:02 |
| rfolco | weshay|ruck, the problem w/ tripleo-repos is that some repos are centos8 only, some are 7, some are both, which makes logic shitty | 19:03 |
| *** dtantsur|afk has quit IRC | 19:03 | |
| weshay|ruck | content = HIGHAVAILABILITY_REPO_TEMPLATE % args.mirror | 19:04 |
| weshay|ruck | _write_repo(content, args.output_path) | 19:04 |
| weshay|ruck | content = POWERTOOLS_REPO_TEMPLATE % args.mirror | 19:04 |
| weshay|ruck | _write_repo(content, args.output_path) | 19:04 |
| *** jtomasek has quit IRC | 19:04 | |
| weshay|ruck | rfolco, I think that bit is fine.. those two repos are only args.distro = centos8 | 19:04 |
| rfolco | weshay|ruck, what is the euivalent for ha repo in centos7 ? | 19:04 |
| rfolco | http://mirror.centos.org/centos/8/HighAvailability/ | 19:04 |
| weshay|ruck | rfolco, there isn't | 19:04 |
| weshay|ruck | rfolco, let's work this together real quick | 19:04 |
| weshay|ruck | https://meet.google.com/rij-fvvf-xxt?authuser=1 | 19:04 |
| rfolco | weshay|ruck, need to move that up at least to avoid error, ha not found | 19:05 |
| weshay|ruck | rfolco, hrm.. ha not found where do you see that? | 19:05 |
| weshay|ruck | in unit? | 19:05 |
| rfolco | will show you | 19:05 |
| *** jtomasek has joined #oooq | 19:06 | |
| rlandy | weshay|ruck: have a minute? | 19:43 |
| weshay|ruck | rlandy, I do | 19:43 |
| mjturek | rfolco: baha and I did some tests | 19:43 |
| weshay|ruck | rlandy, chat? | 19:43 |
| rlandy | weshay|ruck: sure | 19:43 |
| weshay|ruck | https://meet.google.com/rij-fvvf-xxt?authuser=1 | 19:43 |
| rfolco | mjturek, any conclusions ? | 19:44 |
| *** jmasud has quit IRC | 19:44 | |
| mjturek | rfolco: well, no conclusions but some findings | 19:44 |
| mjturek | sudo docker run -e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS -e KOLLA_CONFIG='{"command":"/bin/bash"}' e3d1dac5aaf1 | 19:44 |
| mjturek | rfolco: This runs without a problem ^ | 19:45 |
| *** jmasud has joined #oooq | 19:45 | |
| mjturek | we then pushed to dockerhub and pulled the image from dockerhub | 19:45 |
| mjturek | the command worked again | 19:45 |
| mjturek | rfolco: so we don't think it's a build problem | 19:45 |
| mjturek | rfolco: possibly a problem with uploading to registry.rdoproject.org? | 19:46 |
| mjturek | thoughts? | 19:46 |
| rfolco | mjturek, ok, would try pushing the base container to dockerhub manually, pull it back to see what happens | 19:46 |
| rlandy | weshay|ruck: http://pastebin.test.redhat.com/833042 | 19:47 |
| mjturek | rfolco see above, we did that | 19:47 |
| mjturek | it worked fine | 19:47 |
| rfolco | aaah | 19:47 |
| rfolco | sorry | 19:47 |
| rlandy | https://review.opendev.org/#/c/700226/28 | 19:48 |
| mjturek | np | 19:48 |
| rlandy | https://review.opendev.org/#/c/695988/11 | 19:48 |
| rlandy | https://review.opendev.org/#/c/704404/2 | 19:48 |
| rfolco | mjturek, did you run the same steps to rdoregistry and it failed ? | 19:50 |
| rfolco | mjturek, push/pull | 19:50 |
| mjturek | rfolco I don't have that kind of access to rdoregistry | 19:50 |
| mjturek | and they probably wouldn't appreciate it | 19:50 |
| rfolco | manually push one container to rdoregistry? I think that would be possible... weshay|ruck ? | 19:51 |
| rfolco | its time to chat w/ infra folks about it, weshay|ruck who you can point mjturek to chat with ? | 19:52 |
| mjturek | agreed | 19:52 |
| weshay|ruck | https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/environments/docker-ha.yaml#L27 | 19:52 |
| weshay|ruck | rlandy, ^ | 19:53 |
| mjturek | weshay|ruck know of anyone who might be able to help us? | 19:59 |
| weshay|ruck | rlandy, https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/ci/environments/scenario004-standalone.yaml#L25-L28 | 19:59 |
| *** jmasud has quit IRC | 20:11 | |
| *** jmasud has joined #oooq | 20:12 | |
| *** dsneddon has joined #oooq | 20:23 | |
| *** dsneddon has quit IRC | 20:28 | |
| *** rfolco is now known as rfolco|bbl | 20:29 | |
| rlandy | weshay|ruck: hmmm ... same error | 20:38 |
| weshay|ruck | whaaaa | 20:38 |
| weshay|ruck | rlandy, try w/ scenario04 | 20:39 |
| rlandy | k | 20:39 |
| *** TrevorV has quit IRC | 20:57 | |
| *** holser has quit IRC | 21:46 | |
| *** holser has joined #oooq | 21:54 | |
| *** holser has quit IRC | 22:14 | |
| rlandy | weshay|ruck: one more ... | 22:50 |
| rlandy | " File \"/var/lib/heat-config/heat-config-script/a8df584f-f3e4-4e6d-932f-b88e265ae585\", line 19, in <module>", | 22:51 |
| rlandy | " from tripleo_common.utils import clouds_yaml", | 22:51 |
| rlandy | "ImportError: cannot import name clouds_yaml", | 22:51 |
| rlandy | ^^ still valid? | 22:51 |
| *** Goneri has quit IRC | 22:57 | |
| *** tosky has joined #oooq | 22:57 | |
| weshay|ruck | rlandy, hrm.. | 23:19 |
| weshay|ruck | that sounds like you almost got to the end | 23:20 |
| rlandy | weshay|ruck: it's still valid - in scenario004 | 23:20 |
| rlandy | it is almost at the end | 23:20 |
| rlandy | should have installed with python- tripleoclient | 23:20 |
| weshay|ruck | rlandy, I think it was switched over to tripleo-ansible | 23:21 |
| weshay|ruck | but I'm not sure | 23:21 |
| weshay|ruck | rlandy, can you paste the larger trace | 23:21 |
| rlandy | weshay|ruck: http://pastebin.test.redhat.com/833102 | 23:22 |
| rlandy | we're so close to done here | 23:22 |
| rlandy | scenario004 has it right | 23:24 |
| rlandy | so I need something else from there | 23:24 |
| rlandy | but not all of it | 23:24 |
| rlandy | that causes ceph issues | 23:24 |
| weshay|ruck | BAH.. | 23:24 |
| weshay|ruck | the logs are broken again | 23:24 |
| weshay|ruck | zbr, https://9221391e161e12b0e4a8-0445deb902ee905c6acad95b9b7016b4.ssl.cf1.rackcdn.com/705757/7/check/tripleo-ci-centos-7-scenario004-standalone/7e1c886/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 23:25 |
| rlandy | https://github.com/openstack/tripleo-common/tree/stable/train/tripleo_common/utils | 23:29 |
| rlandy | in train | 23:29 |
| rlandy | gone from master | 23:29 |
| rlandy | https://github.com/openstack/tripleo-common/commit/6cfb31ffb7d8e165d2dad3d964001fd619b83ae5#diff-177d0684356efc01b6fdd1825342e264 | 23:30 |
| rlandy | weshay|ruck: ^^ | 23:30 |
| weshay|ruck | rlandy, ah | 23:31 |
| weshay|ruck | so are the patches you are working w/ all rebased? | 23:31 |
| rlandy | scenario00 has worked this out | 23:31 |
| rlandy | scenario00 has worked this out | 23:31 |
| rlandy | 4 | 23:32 |
| rlandy | ugh | 23:32 |
| rlandy | no - and I'm not rebasing their patches | 23:32 |
| weshay|ruck | need to rebase | 23:32 |
| weshay|ruck | I'll do it if you want | 23:32 |
| rlandy | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_529/705407/1/gate/tripleo-ci-centos-7-scenario004-standalone/529f67d/logs/undercloud/home/zuul/tripleo-heat-installer-templates/extraconfig/post_deploy/ | 23:32 |
| rlandy | weshay|ruck: ^^ that works | 23:32 |
| rlandy | let me see if I can apply that patch locally | 23:36 |
| rlandy | weshay|ruck: k - idk - where that code went | 23:43 |
| weshay|ruck | rlandy, ping cloudnull | 23:43 |
| weshay|ruck | rlandy, but rebase | 23:43 |
| weshay|ruck | rlandy, should work | 23:43 |
| *** tosky has quit IRC | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!