| *** holser has quit IRC | 00:22 | |
| *** holser has joined #oooq | 00:47 | |
| *** rfolco has joined #oooq | 00:55 | |
| *** holser has quit IRC | 01:06 | |
| *** rfolco has quit IRC | 01:29 | |
| *** ysandeep has quit IRC | 02:03 | |
| *** dsneddon has quit IRC | 02:16 | |
| *** ysandeep has joined #oooq | 02:41 | |
| *** dsneddon has joined #oooq | 02:45 | |
| *** ccamacho has quit IRC | 02:50 | |
| weshay | owalsh, we have results from https://code.engineering.redhat.com/gerrit/#/c/189436/ | 03:09 |
|---|---|---|
| weshay | same job.. but runs baremetal.. so we can log in | 03:09 |
| * weshay forwards | 03:09 | |
| owalsh | weshay: ack, I expect it to fail - we have 2 difference enabled_networks vars in ansible and I'm getting the wrong one :-( | 03:10 |
| weshay | owalsh, k.. just forwarded information on how to access the current live environment | 03:10 |
| weshay | owalsh, if you want to look, try now and I'll help if there are questions | 03:10 |
| *** dsneddon has quit IRC | 03:12 | |
| weshay | owalsh, I have to go to bed.. have to wake up at 4:30am | 03:14 |
| weshay | did you get in? | 03:14 |
| *** dsneddon has joined #oooq | 03:15 | |
| owalsh | weshay: yes, thanks! | 03:17 |
| *** dsneddon has quit IRC | 03:21 | |
| *** dsneddon has joined #oooq | 03:52 | |
| *** dsneddon has quit IRC | 03:57 | |
| *** ykarel|away is now known as ykarel | 04:00 | |
| *** rlandy|bbl is now known as rlandy | 04:08 | |
| *** skramaja has joined #oooq | 04:25 | |
| *** udesale has joined #oooq | 04:40 | |
| *** dsneddon has joined #oooq | 04:48 | |
| *** dsneddon has quit IRC | 04:54 | |
| raukadah | ykarel, https://review.opendev.org/704922 | 04:54 |
| *** ysandeep has quit IRC | 04:54 | |
| *** ysandeep_ has joined #oooq | 04:54 | |
| *** ysandeep_ has quit IRC | 04:56 | |
| ykarel | raukadah, ack | 05:21 |
| *** dsneddon has joined #oooq | 05:25 | |
| *** dsneddon has quit IRC | 05:30 | |
| *** surpatil has joined #oooq | 05:44 | |
| *** raukadah is now known as chkumar|rover | 05:57 | |
| *** soniya29 has joined #oooq | 06:02 | |
| *** ratailor has joined #oooq | 06:34 | |
| *** surpatil has quit IRC | 06:44 | |
| *** marios has joined #oooq | 06:44 | |
| *** udesale_ has joined #oooq | 06:46 | |
| *** udesale has quit IRC | 06:47 | |
| chkumar|rover | zbr, jpena|off when around please have alook at this bug https://bugs.launchpad.net/tripleo/+bug/1861378 | 06:51 |
| openstack | Launchpad bug 1861378 in tripleo "Multiple post_failure on master periodic pipeline" [Critical,Confirmed] | 06:51 |
| chkumar|rover | it is blocking promotion and check jobs | 06:51 |
| chkumar|rover | may be log server got full | 06:51 |
| *** apetrich has joined #oooq | 07:01 | |
| chkumar|rover | sshnaidm|afk, Hello, you have #tripleo channel access? We need to update the topic on not to workflow, gate queue is 10 hrs | 07:04 |
| *** apetrich has quit IRC | 07:22 | |
| *** jtomasek has joined #oooq | 07:22 | |
| *** jfrancoa has joined #oooq | 07:24 | |
| *** ratailor has quit IRC | 07:26 | |
| *** dsneddon has joined #oooq | 07:27 | |
| *** ratailor has joined #oooq | 07:27 | |
| chkumar|rover | heading wework | 07:28 |
| *** dsneddon has quit IRC | 07:32 | |
| *** yolanda has quit IRC | 07:32 | |
| *** ykarel is now known as ykarel|lunch | 07:34 | |
| *** bogdando has joined #oooq | 07:41 | |
| *** chem has quit IRC | 07:46 | |
| *** jtomasek has quit IRC | 07:47 | |
| *** tesseract has joined #oooq | 07:47 | |
| *** chem has joined #oooq | 07:48 | |
| *** udesale_ has quit IRC | 07:58 | |
| *** udesale_ has joined #oooq | 07:58 | |
| *** jpena|off is now known as jpena | 08:10 | |
| jpena | the log server is full again :(. I'm checking | 08:11 |
| *** sshnaidm|afk is now known as sshnaidm | 08:12 | |
| *** dpawlik has quit IRC | 08:14 | |
| *** ccamacho has joined #oooq | 08:19 | |
| sshnaidm | jpena, I see ovb jobs don't gzip files again | 08:19 |
| sshnaidm | zbr, do you know why? ^ | 08:20 |
| jpena | o_O | 08:20 |
| sshnaidm | nothing is gzipped: http://logs.rdoproject.org/17/704917/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/95fc484/logs/undercloud/home/zuul/ | 08:20 |
| *** jtomasek has joined #oooq | 08:25 | |
| *** udesale_ has quit IRC | 08:28 | |
| *** dmsimard5 has joined #oooq | 08:31 | |
| *** soniya29 has quit IRC | 08:32 | |
| *** dmsimard has quit IRC | 08:33 | |
| *** dmsimard5 is now known as dmsimard | 08:33 | |
| zbr | sshnaidm: that file is used upstream | 08:38 |
| sshnaidm | zbr, look at #tripleo | 08:38 |
| *** tosky has joined #oooq | 08:46 | |
| *** apetrich has joined #oooq | 08:48 | |
| *** jpena is now known as jpena|off | 09:00 | |
| *** holser has joined #oooq | 09:16 | |
| chkumar|rover | sshnaidm, zbr I am not sure this one is also related to gzip https://sf.hosted.upshift.rdu2.redhat.com/logs/36/189436/4/check/periodic-tripleo-ci-centos-7-bm_envD-1ctlr_2comp-featureset021-master/ed9ecfa/logs/ | 09:21 |
| chkumar|rover | on downstream bm logs are not getting collected | 09:21 |
| zbr | chkumar|rover: you need to investigate, i do not have the energy to look at ansible debug logs. | 09:24 |
| zbr | even firefox crashes. not only me | 09:24 |
| chkumar|rover | aye | 09:25 |
| zbr | and luckly i escaped rovring last week ;) | 09:25 |
| zbr | chkumar|rover: but few days ago i think i discovered something with wes, that collect_logs.sh script does not not run artcl from the changeset, is probably an older version. | 09:26 |
| *** dsneddon has joined #oooq | 09:27 | |
| zbr | for sure we need to avoid that shell script. zull -> bash -> ansible is not correct. | 09:27 |
| zbr | we can use import_playbook in zuul | 09:28 |
| zbr | even load vars from file | 09:28 |
| zbr | that -vvvv there is just evil | 09:29 |
| *** dtantsur|afk is now known as dtantsur | 09:29 | |
| *** holser has quit IRC | 09:30 | |
| *** dsneddon has quit IRC | 09:32 | |
| *** holser has joined #oooq | 09:32 | |
| *** derekh has joined #oooq | 09:33 | |
| *** ykarel|lunch is now known as ykarel | 09:33 | |
| *** udesale has joined #oooq | 09:35 | |
| matbu | sshnaidm: marios chkumar|rover hey all, do you know if there is a downstream oooq repo for OSP config ? | 09:37 |
| matbu | and rhel deployment | 09:37 |
| sshnaidm | matbu, on internal irc | 09:39 |
| sshnaidm | marios, did you see "Invalid regex: * in provided whitelist file" on http://logs.rdoproject.org/39/24339/9/check/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/45e0a19/job-output.txt | 09:41 |
| marios | sshnaidm: yeah i commmented -1 @ https://review.opendev.org/#/c/701016/15/config/general_config/featureset020.yml | 09:45 |
| marios | sshnaidm: i need to revisit that it needs an update | 09:45 |
| marios | sshnaidm: thanks for ping anout it | 09:46 |
| marios | about | 09:46 |
| sshnaidm | marios, try to remove "|" from "tempest_test_blacklist: |" | 09:48 |
| marios | sshnaidm: i tried originally with >- but wanted it with newlines cos list. it failed same way with > but right i need to edit that syntaxt | 09:49 |
| zbr | sshnaidm: I had to block https://review.opendev.org/#/c/704933/1 because is messing upstream. | 09:59 |
| zbr | already got result | 09:59 |
| sshnaidm | zbr, no, don't block it | 09:59 |
| sshnaidm | zbr, we need logs server up in rdo server, and this is what it does | 09:59 |
| zbr | sshnaidm: we need to rebase https://review.opendev.org/#/c/704938/ on master instead borken on | 10:00 |
| zbr | but that revert is not the solution | 10:00 |
| zbr | solution is https://review.opendev.org/#/c/704938/1 | 10:00 |
| sshnaidm | zbr, please, let me do it, don't block anything right now | 10:00 |
| ykarel | yes please get that merged, no logs are bad then not browsable one | 10:00 |
| zbr | ykarel: why not doing the correct fix? | 10:02 |
| zbr | that revert is replacing one problem with another | 10:02 |
| arxcruz | hey guys, i'll be afk a little bit, solving some german visa issues | 10:02 |
| zbr | we already know and have the right way of fixing it | 10:02 |
| arxcruz | tl;dr there's a typo on my name | 10:02 |
| ykarel | zbr, that patch was wrong, so option 1 is revert to get things back, i am ok with revert + squash fix also | 10:03 |
| ykarel | whichever is quicker | 10:03 |
| chkumar|rover | migi, owalsh another bug https://bugs.launchpad.net/tripleo/+bug/1861393 related to volume tests | 10:12 |
| openstack | Launchpad bug 1861393 in tripleo "Multiple volume related test failed due iscsiadm getting sessions: iscsiadm: and VolumeDeviceNotFound in nova compute" [Critical,Confirmed] | 10:12 |
| zbr | that fixes the gzip problem directly: https://review.opendev.org/#/c/704952/ | 10:14 |
| zbr | also putting a note, to avoid further regression, especially as we have artcl config there | 10:15 |
| migi | chkumar|rover: getting coffee and will look in few min | 10:16 |
| chkumar|rover | migi, fix already up and tested in the same box https://review.opendev.org/#/c/704805/ | 10:17 |
| sshnaidm | zbr, we'll merge firstly patch that allows to log server to come to life, than patch that fixes all gzipping and not gzipping, if you want you can put your patches on top of these two | 10:17 |
| chkumar|rover | migi, I will check the rest of the skip list before that | 10:18 |
| migi | chkumar|rover: ok | 10:18 |
| migi | chkumar|rover: btw why this? https://review.opendev.org/#/c/704805/1/zuul.d/layout.yaml | 10:18 |
| migi | chkumar|rover: just so only one job triggers ? | 10:18 |
| migi | chkumar|rover: cause I don't see 020 featureset there | 10:19 |
| migi | sorry fs21 | 10:19 |
| migi | unless it's depends-on | 10:19 |
| *** ratailor has quit IRC | 10:24 | |
| zbr | sshnaidm: after the fires settle we need to rethink the use of this parameter, i think we need to make the upload server define its value and not us | 10:26 |
| *** ratailor has joined #oooq | 10:26 | |
| zbr | regardless where a job runs, it does upload to a destination, and the destination should decide which kind of storage it wants. | 10:26 |
| *** ratailor has quit IRC | 10:26 | |
| *** ratailor has joined #oooq | 10:27 | |
| zbr | and we can avoid confusions like this, by linking artcl_gzip to the upload server instead of individual environments | 10:27 |
| *** ratailor has quit IRC | 10:29 | |
| *** ratailor has joined #oooq | 10:30 | |
| *** ratailor has quit IRC | 10:31 | |
| *** ratailor has joined #oooq | 10:32 | |
| *** ratailor has quit IRC | 10:40 | |
| *** ratailor has joined #oooq | 10:41 | |
| *** dsneddon has joined #oooq | 10:42 | |
| marios | panda: fyi https://bugs.launchpad.net/tripleo/+bug/1861342/comments/1 | 10:43 |
| openstack | Launchpad bug 1861342 in tripleo "tripleo-ci promotion failing on "pull ppc64le tagged containers"" [Critical,Triaged] | 10:43 |
| marios | panda: not sure if there is something else we need to do to handle the _ppc? so it is reproducible with podman pull, don't think there is something wrong in https://github.com/rdo-infra/ci-config/blob/79bcc9c64b82f1c6806139118e5b9a3663dcdb76/ci-scripts/container-push/roles/containers-promote/tasks/manifest-push.yml#L40-L49 or i can't se it yet | 10:44 |
| *** ratailor has joined #oooq | 10:44 | |
| chkumar|rover | migi, it was just a test review it works, in order to avoid wasting resources, we removed those jobs and tested in downstream review | 10:45 |
| *** apetrich has quit IRC | 10:47 | |
| *** dsneddon has quit IRC | 10:49 | |
| *** dsneddon has joined #oooq | 10:52 | |
| *** zbr has quit IRC | 10:52 | |
| *** udesale has quit IRC | 10:53 | |
| *** ratailor_ has joined #oooq | 10:55 | |
| *** dsneddon has quit IRC | 10:57 | |
| *** ratailor has quit IRC | 10:58 | |
| chkumar|rover | Hey #oooq, please wait for few mins to push or recheck changes on review.rdoproject.org rdo logserver is full, it will take so time to get it cleaned, Sorry for the inconvenience | 11:06 |
| owalsh | chkumar|rover: hey, is that what caused the POST_FAILUREs in https://review.opendev.org/704880 | 11:22 |
| *** ykarel is now known as ykarel|afk | 11:22 | |
| chkumar|rover | owalsh, rdo log server is full we are freeing up space there | 11:23 |
| chkumar|rover | owalsh, https://bugs.launchpad.net/tripleo/+bug/1861378 | 11:23 |
| openstack | Launchpad bug 1861378 in tripleo "Multiple post_failure on master periodic pipeline" [Critical,Confirmed] | 11:23 |
| chkumar|rover | it will take some time | 11:23 |
| *** zbr has joined #oooq | 11:23 | |
| *** dsneddon has joined #oooq | 11:25 | |
| owalsh | ack, thanks | 11:27 |
| owalsh | migi: the healthcheck failure suggests the sudo config has changed on the containers - could you create an LP for it? | 11:33 |
| dtantsur | hi folks! what does quickstart use instead of plain ssh nowadays? I'm getting Could not resolve hostname | 11:33 |
| dtantsur | for a host that can be perfectly used with `ssh hostname` | 11:34 |
| dtantsur | does it require a globally resolvable name? | 11:35 |
| *** zbr has quit IRC | 11:35 | |
| owalsh | dtantsur: IIRC it uses a different ssh config from ~/.quickstart with aliases e.g virthost etc... | 11:35 |
| sshnaidm | ssh.config.ansible maybe | 11:35 |
| *** zbr has joined #oooq | 11:35 | |
| dtantsur | sigh | 11:35 |
| dtantsur | that's why we cannot have nice things :) | 11:36 |
| dtantsur | actually, it does respect ~/.ssh/config in some aspects, just not in aliases | 11:36 |
| sshnaidm | dtantsur, but that's weird | 11:36 |
| dtantsur | yeah. HostName directive doesn't work, User - does | 11:36 |
| *** ykarel|afk is now known as ykarel | 11:36 | |
| sshnaidm | dtantsur, it uses ansible, so if it doesn't work, most likely ansible is to blame | 11:37 |
| migi | owalsh: will do | 11:37 |
| * dtantsur gladly blames ansible | 11:38 | |
| dtantsur | sshnaidm: how naive is my attempt to use RHEL 8 as a virthost oS? | 11:38 |
| * owalsh blames ssh | 11:38 | |
| dtantsur | "No package libvirt-python available.", | 11:38 |
| dtantsur | "No package python-lxml available." | 11:39 |
| dtantsur | I've gotten the answe | 11:39 |
| sshnaidm | dtantsur, this may help: https://review.opendev.org/#/c/702100 | 11:39 |
| sshnaidm | dtantsur, checked it today and it worked for me | 11:40 |
| dtantsur | thanks! | 11:40 |
| *** soniya29 has joined #oooq | 11:42 | |
| owalsh | migi: that suggest something has changed in the container sudo conf... I wonder if it's causing any of the other issues? | 11:43 |
| chkumar|rover | brb | 11:46 |
| *** ratailor_ has quit IRC | 11:46 | |
| weshay | marios, /me looking at /home/centos/ci-config/ci-scripts/dlrnapi_promoter/dlrn-promoter.sh | 11:46 |
| weshay | and other spots.. not clear where that is turned off | 11:46 |
| weshay | Active: active (running) since Wed 2020-01-29 21:40:13 UTC; 14h ago | 11:47 |
| weshay | Main PID: 29711 (dlrn-promoter-s) | 11:47 |
| weshay | Memory: 216.2M | 11:47 |
| weshay | CGroup: /system.slice/dlrn-promoter.service | 11:47 |
| weshay | ├─ 3555 /usr/bin/python2 /usr/bin/ansible-playbook -e manifest_push=true -e target_registries_push=true /home/centos/ci-config-newcode/ci-scripts/container-push/container-push.y. | 11:47 |
| weshay | panda, ^ | 11:47 |
| marios | weshay: the problem is the ppc containers in rdo registry. | 11:48 |
| marios | weshay: we can switch off manifest push but that is not the problem | 11:48 |
| migi | owalsh: is ss to be run in overcloud on in contianer? | 11:48 |
| *** holser has quit IRC | 11:48 | |
| weshay | marios, aye.. agree | 11:48 |
| weshay | just where does one make that switch | 11:48 |
| marios | weshay: https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-7/master.ini | 11:49 |
| marios | manifest_push: true | 11:49 |
| owalsh | migi: https://github.com/openstack/tripleo-common/blob/master/healthcheck/common.sh#L55 | 11:49 |
| weshay | marios, panda found it.. https://review.rdoproject.org/r/#/c/24750/ | 11:49 |
| weshay | thanks | 11:49 |
| marios | weshay: 13:49 < marios> weshay: https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-7/master.ini | 11:50 |
| marios | weshay: yes that 13:49 < weshay> marios, panda found it.. https://review.rdoproject.org/r/#/c/24750/ | 11:50 |
| weshay | thanks | 11:50 |
| marios | weshay: have you even brushed your teeth yet | 11:50 |
| marios | :) | 11:50 |
| weshay | marios, what I ddin't know was that it was in the promotion config | 11:51 |
| weshay | :) | 11:51 |
| weshay | lolz | 11:51 |
| owalsh | migi: so container | 11:51 |
| weshay | marios, what teeth? | 11:51 |
| marios | weshay: well it is in the role as a variable. panda added it into the config to enable it for master/train | 11:51 |
| migi | owalsh: so I did short test and was able to run ss -ntuap as nova user from container | 11:51 |
| migi | owalsh: on same host that complained..., but now the env is gone | 11:52 |
| owalsh | migi: and sudo -u nova ss -ntuap | 11:52 |
| migi | owalsh: that not | 11:52 |
| migi | and yes if sudo is broken then it will be one | 11:52 |
| migi | owalsh: will check again and see | 11:53 |
| marios | weshay: (default false there https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/container-push/roles/containers-promote/defaults/main.yml ) | 11:53 |
| weshay | marios, ya.. but that's a default | 11:54 |
| marios | weshay: right | 11:54 |
| owalsh | migi: it's probably just that command that's failing. I don't think it's white listed but I guess sudo was wide open in the containers until now... | 11:54 |
| owalsh | or it's been failing for ages but nobody noticed :-) | 11:54 |
| weshay | marios, thanks for the assit | 11:55 |
| *** rfolco has joined #oooq | 12:08 | |
| *** ykarel is now known as ykarel|afk | 12:25 | |
| *** soniya29 has quit IRC | 12:40 | |
| chkumar|rover | arxcruz, around, can you take a look at this https://53cfbe20d54fcd346846-7e165b3b1cbfb7e5b567389713168c5c.ssl.cf1.rackcdn.com/703953/4/check/tripleo-ci-centos-7-containers-multinode/94a29e1/logs/undercloud/var/log/tempest/tempest_run.log in free? | 12:41 |
| *** chem has quit IRC | 12:42 | |
| marios | zbr: can you please see my comment (and ideally reply there ) at https://review.rdoproject.org/r/#/c/24705/ - we can discuss on call in a few mins as well | 12:43 |
| marios | panda: wdyt? ^ | 12:43 |
| chkumar|rover | matbu, Hello, Do we have tht tripleo-ansible bump patch up? | 12:46 |
| *** chem has joined #oooq | 12:46 | |
| zbr | marios: done | 12:54 |
| *** rlandy has joined #oooq | 12:59 | |
| *** marios is now known as marios|call | 13:01 | |
| matbu | chkumar|rover: you mean with the new release number ? | 13:01 |
| matbu | chkumar|rover: your patch has been merged ? | 13:01 |
| rfolco | zbr, weshay arxcruz marios|call sshnaidm rlandy panda : scrum time | 13:01 |
| matbu | I didn't check my email after lunch | 13:01 |
| chkumar|rover | matbu, yes | 13:02 |
| matbu | cool, i will update mine | 13:02 |
| chkumar|rover | matbu, you can create the patch, reviewers will take care of that | 13:02 |
| matbu | chkumar|rover: i think only the tripleo-common is enough | 13:02 |
| weshay | panda, you around? | 13:03 |
| chkumar|rover | matbu, we need a tht patch | 13:03 |
| weshay | arxcruz, is pto | 13:03 |
| matbu | chkumar|rover: with the version ? why ? | 13:03 |
| matbu | chkumar|rover: the point of Alex was to put a version in ooo-common so the package can be upgraded with ooo-common | 13:04 |
| chkumar|rover | matbu, ok | 13:04 |
| matbu | iiuc | 13:04 |
| matbu | Alex is not only yet | 13:04 |
| weshay | chkumar|rover, can you join scrum for a hot minute | 13:05 |
| weshay | chkumar|rover, or tell me what the status of the centos-8 ceph repos | 13:05 |
| chkumar|rover | weshay, no updates just asked gfindete on tripleo | 13:06 |
| chkumar|rover | waiting for reply | 13:06 |
| weshay | chkumar|rover, k | 13:07 |
| weshay | thanks | 13:07 |
| *** ykarel|afk is now known as ykarel | 13:10 | |
| weshay | https://review.rdoproject.org/r/#/c/24667/ | 13:12 |
| weshay | https://review.rdoproject.org/r/#/c/24667/ | 13:13 |
| *** ysandeep has joined #oooq | 13:28 | |
| *** apetrich has joined #oooq | 13:29 | |
| chkumar|rover | Why we are running standalone on tripleo-ci-centos-7-scenario004-standalone on stable/stein branch? | 13:32 |
| chkumar|rover | ah sorry, it is running in promotion pipeline | 13:33 |
| *** holser has joined #oooq | 13:33 | |
| rlandy | zbr: https://sf.hosted.upshift.rdu2.redhat.com/logs | 13:37 |
| weshay | https://code.engineering.redhat.com/gerrit/#/q/project:openstack/rrcockpit | 13:41 |
| marios|call | https://code.engineering.redhat.com/gerrit/#/q/topic:17-standup+status:open | 13:46 |
| marios|call | weshay: rlandy: ^ | 13:46 |
| *** chem has quit IRC | 13:47 | |
| chkumar|rover | rlandy, weshay let me know when free, we can start the call early and finish it | 13:48 |
| *** chem has joined #oooq | 13:48 | |
| marios|call | rlandy: podman meeting? (should i join just cos weshay said 'discuss there ' is it about rhel8 or sthing else | 13:52 |
| rlandy | marios|call: you are welcome to join - about gating downstream podman-related changes | 13:53 |
| rlandy | starting on the hour | 13:53 |
| rlandy | will post meeting link | 13:53 |
| marios|call | rlandy: ack ok thanks - its ok i'd only join if it was relevant to the rhel8/containers stuff | 13:53 |
| marios|call | rlandy: no worries thanks | 13:53 |
| rlandy | marios|call: idk think so - but always happy to have your opinion on stuff | 13:54 |
| *** marios|call is now known as marios | 13:56 | |
| zbr | rlandy: the only thing needed is to add artcl_gzip: true -- can you do it? | 13:57 |
| zbr | if you see a artcl_gzip_only, you can remove it, no longer used, correct name "artcl_gzip" | 13:57 |
| rlandy | zbr: where does that get added? | 14:00 |
| chkumar|rover | rlandy, weshay meeting tome | 14:00 |
| chkumar|rover | *time | 14:00 |
| rlandy | marios: if you want ... meet.google.com/vnf-jdnm-tzs | 14:00 |
| zbr | rlandy: you have to add this to the internal specific config | 14:01 |
| marios | rlandy: thanks , would rather progress on other tasks, ping me if i am needed for something otherwise i'll skip | 14:01 |
| marios | rlandy: i was only asking because it was mentioned on scrum during the rhel8/containers discussion | 14:01 |
| rlandy | sure | 14:02 |
| chkumar|rover | weshay, we are waiting for you | 14:02 |
| weshay | I'm coming | 14:03 |
| chkumar|rover | weshay, https://docs.google.com/document/d/1-3ohDnfcj1ptJ_EUb_UAjsz5cAuqLnzURwOqUs40GP0/edit | 14:04 |
| *** sshnaidm is now known as sshnaidm|afk | 14:07 | |
| arxcruz | weshay: rfolco i'm back, turns out it was easier than I though | 14:09 |
| weshay | arxcruz, cool.. thank you | 14:09 |
| *** dtantsur is now known as dtantsur|brb | 14:18 | |
| rfolco | arxcruz, are you in trouble with your name? | 14:19 |
| arxcruz | rfolco: as always | 14:19 |
| arxcruz | rfolco: my name is pereira, and they write in the visa peireira | 14:19 |
| rfolco | f* i | 14:20 |
| rfolco | they put i in everything | 14:20 |
| rfolco | in phone, pad... | 14:20 |
| arxcruz | rfolco: yeah, so, i was "i"legal | 14:20 |
| rfolco | ha | 14:21 |
| rfolco | I see | 14:21 |
| chkumar|rover | weshay, rlandy https://sf.hosted.upshift.rdu2.redhat.com/logs/84/190384/17/check/podman-package-rhel-8/d22a1fe/job-output.txt | 14:21 |
| chkumar|rover | rlandy, https://code.engineering.redhat.com/gerrit/#/c/190384/21/playbooks/podman/pre.yaml | 14:21 |
| arxcruz | marios: chkumar|rover i have a fs020 job with 4:20 minutes ;) | 14:22 |
| arxcruz | with 41 min remain to timeout :D | 14:22 |
| panda | back | 14:22 |
| arxcruz | unfortunately got a post failure... | 14:22 |
| panda | did I miss anything ? | 14:23 |
| rfolco | panda, need 5 min of your time | 14:24 |
| rfolco | panda, to show you something and ask your opinion | 14:24 |
| marios | arxcruz: k, so you mean we should leave all the tempest there? | 14:25 |
| arxcruz | marios: I'm saying it's possible, it was a poc, all i did was increase the cpu and number of concurrency tests | 14:26 |
| marios | arxcruz: ack | 14:28 |
| panda | rfolco: not that thing again. I already told you you would need a different measuring device. | 14:29 |
| rfolco | panda, ? | 14:30 |
| panda | rfolco: jk | 14:31 |
| panda | rfolco: when ? where ? | 14:31 |
| rfolco | panda, ok https://meet.google.com/oiv-geho-mai | 14:32 |
| *** sshnaidm|afk is now known as sshnaidm | 14:32 | |
| marios | zbr: ack replied there again. would you consider removing your -1 then if you don't want to block? https://review.rdoproject.org/r/#/c/24705/ | 14:32 |
| marios | simple one reviews please when you next have time thank you "Refresh start_named_hashes after promotion to prevent false positive" https://review.rdoproject.org/r/#/c/24665/ | 14:33 |
| zbr | marios: am i using -1 wrong? afaik, -1 translates to "i don't like something about it", is far from -2 :D | 14:34 |
| zbr | sadly we do not have a -0.5 | 14:35 |
| zbr | but if others say i should not use it as "i have concerns", i can skip. | 14:35 |
| zbr | bad part is that there is no other way to indicate that i seen a review other than making a decision about +/- | 14:36 |
| marios | zbr: ok fair enough thanks | 14:37 |
| zbr | i mentioned clearly, is "soft" | 14:37 |
| zbr | weshay: bj? | 14:38 |
| weshay | zbr, need a few.. I need to chat.. but hold 10min ok? | 14:40 |
| zbr | sure | 14:40 |
| weshay | zbr, actually ready now.. ping when ur avail | 14:42 |
| weshay | updated mtg invite | 14:43 |
| weshay | I'm in https://meet.google.com/pqf-dxyz-djo?authuser=1 | 14:43 |
| marios | reviews please around removing duplication in promoter molecule rlandy rfolco panda sshnaidm weshay arxcruz chkumar|rover when you next have some time thanks https://review.rdoproject.org/r/#/c/24705 | 14:45 |
| *** ysandeep has quit IRC | 14:46 | |
| *** udesale has joined #oooq | 14:58 | |
| *** ysandeep has joined #oooq | 15:12 | |
| chkumar|rover | rlandy, one downstream question, when we checkout a particular branch like triple-modify image branch, does it actually builds the rpm from that? | 15:15 |
| rlandy | it should | 15:15 |
| rlandy | sec | 15:15 |
| zbr | marios: do you have 10min for me? | 15:16 |
| chkumar|rover | rlandy, can you check? | 15:17 |
| rlandy | chkumar|rover: on meeting - one sec | 15:17 |
| *** holser has quit IRC | 15:18 | |
| *** ysandeep has quit IRC | 15:20 | |
| *** jaosorior has joined #oooq | 15:21 | |
| chkumar|rover | see ya people | 15:23 |
| marios | zbr: whatsup | 15:24 |
| *** chkumar|rover is now known as raukadah | 15:24 | |
| *** jmasud has quit IRC | 15:24 | |
| *** jaosorior has quit IRC | 15:27 | |
| *** ccamacho has quit IRC | 15:27 | |
| zbr | marios: i want to explain what i proposed on https://review.opendev.org/#/c/703586/ -- easier on https://meet.google.com/nah-avgj-ice | 15:31 |
| zbr | others welcome | 15:31 |
| marios | zbr: joining | 15:32 |
| *** marios is now known as marios|call | 15:33 | |
| *** jmasud has joined #oooq | 15:37 | |
| zbr | tx! | 15:42 |
| *** jmasud has quit IRC | 15:43 | |
| *** marios|call is now known as marios | 15:43 | |
| *** TrevorV has joined #oooq | 15:44 | |
| *** Trevor_V has joined #oooq | 15:47 | |
| *** ratailor has joined #oooq | 15:48 | |
| *** ratailor_ has joined #oooq | 15:50 | |
| *** TrevorV has quit IRC | 15:51 | |
| *** ratailor has quit IRC | 15:54 | |
| *** holser has joined #oooq | 15:56 | |
| zbr | weshay: take a look at http://logs.rdoproject.org/86/703586/12/openstack-check/tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset001/398d3d2/logs/ | 15:59 |
| zbr | do you see something interesting? | 15:59 |
| zbr | i think i mentioned to chandan this morning that I seen a very weird "-vvv" when collect logs was called by qs. | 16:01 |
| zbr | i am curious who is brave enough to open the collect logs file in their browser | 16:01 |
| zbr | rlandy: by any change can you help be find out where does the -vvvv comes from on http://logs.rdoproject.org/86/703586/12/openstack-check/tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset001/398d3d2/logs/ovb_collect_logs.sh | 16:06 |
| rlandy | yes - one sec - submitting change | 16:06 |
| *** skramaja has quit IRC | 16:06 | |
| *** dtantsur|brb is now known as dtantsur | 16:06 | |
| *** jmasud has joined #oooq | 16:11 | |
| rlandy | zbr: looking | 16:11 |
| migi | rlandy: do you know any role that is executed in prepare stage of zuul run and it's copying the source code of the single git repo ? if not I will need to work on one, but maybe already used somewhere | 16:11 |
| rlandy | migi: in general or when we build a specific repo? | 16:12 |
| migi | rlandy: in general | 16:13 |
| rlandy | zbr: it's here ... https://github.com/openstack/tripleo-ci/blob/master/roles/run-test/templates/toci_quickstart.sh.j2#L50 | 16:13 |
| *** aakarsh has joined #oooq | 16:14 | |
| rlandy | migi: yeah ... I'd have to find it again - give me a few | 16:14 |
| zbr | rlandy: not sure who put it but that value is insane, have you seen it produced 400MB? it will crash your browser if you click the log | 16:14 |
| zbr | we should never see more than double v on ci. | 16:14 |
| rlandy | zbr: git blames alex - 13 months ago | 16:15 |
| migi | rlandy: something that is kind of this: https://pagure.io/zuul-distro-jobs/blob/master/f/tests/local.yaml.example | 16:15 |
| zbr | nopb, i wasn't looking for a culprit, thanks for pointing. making fix now. | 16:15 |
| rlandy | migi: yeah - it's zuul code itself - what do you need exactly - maybe we can help? | 16:16 |
| zbr | mainly logging of log-collection spamming, bit ironic? | 16:16 |
| *** aakarsh has quit IRC | 16:18 | |
| migi | rlandy: so I have already idea how to do this, but maybe you already use it. I need to run gate job with time trigger, meaning that zuul.executor.src_root is not there and git repo is not copied + the branch is missing, so I need a role which can be run (probably as part of prepare-workspace part) that clones specific git repo on a specific branch and uses it in the later run | 16:18 |
| rlandy | migi: I think so ... | 16:19 |
| *** jmasud has quit IRC | 16:19 | |
| rlandy | like http://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-config.git/tree/zuul.d/projects.yaml#n14 | 16:19 |
| rlandy | migi: ^^ | 16:19 |
| rlandy | that triggers off a change | 16:19 |
| rlandy | but you can run that periodically | 16:20 |
| rlandy | wrt git repo ... | 16:20 |
| migi | rlandy: yes | 16:20 |
| migi | rlandy: so how to make this to run periodicly | 16:20 |
| rlandy | migi or what raukadah and I just added for podman ... | 16:20 |
| *** jmasud has joined #oooq | 16:20 | |
| rlandy | migi: add the job to the periodic pipeline | 16:20 |
| rlandy | migi: let's chat for 5, I'll explain | 16:21 |
| migi | rlandy: yes, but it will be missing the repo, right ? | 16:21 |
| migi | rlandy: later, on the call :) | 16:21 |
| rlandy | migi: k - ping me when ready | 16:21 |
| migi | rlandy: thx | 16:21 |
| rlandy | will post link here | 16:21 |
| rlandy | http://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-config.git/tree/zuul.d/pipelines.yaml#n94 | 16:21 |
| rlandy | migi: ^^ periodic pipeline - runs once a day | 16:22 |
| migi | rlandy: correct, this I know but adding it to the job requires some change of job itself | 16:22 |
| rlandy | migi: job added to it ... http://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-jobs.git/tree/zuul.d/podman-jobs.yaml and http://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-jobs.git/tree/zuul.d/projects.yaml#n39 | 16:22 |
| rlandy | ^^ job added | 16:22 |
| rlandy | with git build dependency | 16:23 |
| migi | rlandy: ah cool, perfect | 16:23 |
| migi | rlandy: anyway will have one question around this later | 16:24 |
| rlandy | migi: k - I'm here | 16:24 |
| migi | rlandy: around job - as I would like to move them to tripleo-ci-internal-jobs rather then keep them in osp-jobs | 16:24 |
| rlandy | fine | 16:24 |
| rlandy | I will need to do some config work to enable that | 16:25 |
| *** jmasud has quit IRC | 16:29 | |
| *** jmasud has joined #oooq | 16:33 | |
| sshnaidm | weshay, raukadah is it known? https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_fb8/704938/5/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/fb8d3f0/logs/undercloud/home/zuul/overcloud_update_run_Controller.log | 16:33 |
| sshnaidm | fails in gates | 16:34 |
| weshay | sshnaidm, yes | 16:34 |
| weshay | sshnaidm, I think this is the revert to fix https://review.opendev.org/#/c/704885/ | 16:35 |
| weshay | sshnaidm, then Cedric just broke scen10 | 16:35 |
| weshay | just another day | 16:35 |
| weshay | sshnaidm, https://review.opendev.org/#/c/705051/ | 16:36 |
| *** jmasud has quit IRC | 16:40 | |
| zbr | weshay: sshnaidm: an 1yr old bug that can hog log servers: https://review.opendev.org/#/c/705059 | 16:42 |
| zbr | discovered one 450MB log file caused by it, and that was on the job that did not fail in POST. | 16:43 |
| *** udesale has quit IRC | 16:43 | |
| sshnaidm | zbr, it's not a bug and done intentionally | 16:44 |
| sshnaidm | zbr, so we can see errors that can't see with regular run | 16:45 |
| zbr | use of -vvvv is unresonable, and you have the proof on https://review.opendev.org/#/c/703586/ | 16:46 |
| zbr | in fact the original toci_quickstart.sh file has the correct value, only the j2 has the debug level. | 16:47 |
| sshnaidm | zbr, unreasonable is not an argument | 16:47 |
| weshay | rfolco, can you get in touch w/ mjturek and try to discover what happened here https://bugs.launchpad.net/tripleo/+bug/1861342 | 16:47 |
| openstack | Launchpad bug 1861342 in tripleo "tripleo-ci promotion failing on "pull ppc64le tagged containers"" [Critical,Triaged] | 16:47 |
| weshay | OMG.. it's only 9:47 | 16:47 |
| weshay | jebus | 16:47 |
| rlandy | lol | 16:47 |
| rlandy | the day is but young | 16:48 |
| sshnaidm | zbr, you need more convincing arguments than "unreasonable" to make our life harder and investigation less possible | 16:48 |
| zbr | sshnaidm: sure, please debug this http://logs.rdoproject.org/86/703586/12/openstack-check/tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset001/398d3d2/logs/quickstart_collect_logs.log | 16:49 |
| sshnaidm | zbr, it's your patch fault, so you need to fix it | 16:50 |
| *** holser__ has joined #oooq | 16:50 | |
| *** ykarel is now known as ykarel|away | 16:51 | |
| weshay | zbr, the collect log console was 450 MB? | 16:52 |
| *** holser has quit IRC | 16:52 | |
| zbr | here is how I see it: putting debug level logging in production code without even putting a cap on it, is the cause of this issue. | 16:52 |
| zbr | use of -vvvv in http://logs.rdoproject.org/86/703586/12/openstack-check/tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset001/398d3d2/logs/ovb_collect_logs.sh | 16:53 |
| weshay | zbr, show me the 450mb file | 16:53 |
| weshay | then continue | 16:53 |
| zbr | weshay: look ^ | 16:54 |
| rfolco | weshay, ack... mjturek lets chat :) | 16:54 |
| sshnaidm | zbr, is this log from this patch? https://review.opendev.org/#/c/703586/1 | 16:54 |
| weshay | k.. 405 http://logs.rdoproject.org/86/703586/12/openstack-check/tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset001/398d3d2/logs/quickstart_collect_logs.log | 16:55 |
| * weshay looking | 16:55 | |
| weshay | zbr, next question.. why is it not compressed? | 16:55 |
| zbr | yes, where on rdo job choked and another one managed to product 450MB file | 16:55 |
| sshnaidm | zbr, so what did you do in your patch that generated so big file? | 16:57 |
| weshay | zbr, go talk to Alex https://opendev.org/openstack/tripleo-ci/commit/8d117bb228c8a79f4c67375a1f990239fa718cd3 | 16:58 |
| *** marios is now known as marios|out | 16:59 | |
| weshay | I can't read the log w/ -vvvv, looks like it was moved to -vvvv to debug a particular issue | 16:59 |
| zbr | weshay: i think alex will say that he forgot to revert it, i am sure. | 16:59 |
| weshay | zbr, probably.. better to ask | 16:59 |
| migi | rlandy: so I confirmed it's not possible to use artefacts. Also pipelines, jobs can be shared between tenants | 16:59 |
| weshay | zbr, and to sshnaidm's point.. let's add more facts to commit msg's and less opinions.. I think that would help | 17:00 |
| rlandy | migi: ok - so can you build the rpm in tripleo-ci-internal? | 17:00 |
| weshay | zbr, for example... | 17:00 |
| weshay | -vvvv is producing a very large log file.. 405 ( + link ); looks like this was moved to -vvvv for a particular bug. Let's move it back to -vv | 17:01 |
| migi | rlandy: there are 2 options, but let's discuss this tomorrow | 17:01 |
| rlandy | migi: ack | 17:01 |
| *** bogdando has quit IRC | 17:04 | |
| *** ratailor_ has quit IRC | 17:05 | |
| *** tesseract has quit IRC | 17:08 | |
| *** marios|out has quit IRC | 17:12 | |
| *** jmasud has joined #oooq | 17:30 | |
| *** jmasud_ has joined #oooq | 17:34 | |
| *** jmasud has quit IRC | 17:34 | |
| zbr | rlandy: a simple one https://review.opendev.org/#/c/669223/ when you can, thanks. | 17:36 |
| *** holser__ has quit IRC | 17:38 | |
| *** holser has joined #oooq | 17:39 | |
| weshay | zbr, help me out w/ https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a83/702754/8/gate/tripleo-ci-centos-7-containers-undercloud-minion/a83eed9/logs/undercloud/home/zuul/undercloud_install.log.txt.gz | 17:40 |
| *** jmasud_ has quit IRC | 17:49 | |
| zbr | what is the problem? i see a success. | 17:50 |
| *** dtantsur is now known as dtantsur|afk | 17:52 | |
| mjturek | rfolco panda: isn't https://bugs.launchpad.net/tripleo/+bug/1861342 happening because the last promoted hash wasn't built by the ppc job (the job was down, it's restored now) | 17:55 |
| openstack | Launchpad bug 1861342 in tripleo "tripleo-ci promotion failing on "pull ppc64le tagged containers"" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 17:55 |
| mjturek | let me know if I'm wrong! | 17:55 |
| weshay | zbr, it's not rendering | 17:57 |
| weshay | it's downloading | 17:57 |
| zbr | yeah, that is correct. gz files are supposed to downloaded. | 17:58 |
| rfolco | mjturek, try to run this | 17:58 |
| rfolco | podman pull trunk.registry.rdoproject.org/tripleomaster/centos-binary-aodh-api:03e7a3a58585eb6d751cb6ea765973241f12f2f9_6e3b098e_ppc64le | 17:58 |
| rfolco | in ppc box | 17:58 |
| weshay | zbr, https://meet.google.com/evr-euer-gad?authuser=1 | 17:59 |
| mjturek | trying | 17:59 |
| *** derekh has quit IRC | 18:00 | |
| *** holser has quit IRC | 18:01 | |
| mjturek | rfolco: unexpected EoF | 18:03 |
| rfolco | hmm | 18:03 |
| rfolco | for that hash... did you notice something in the job run ? any unexpected errors ? | 18:03 |
| rfolco | weird | 18:03 |
| mjturek | do you know when that hash was? | 18:03 |
| rfolco | not sure | 18:04 |
| rfolco | lets try to find out | 18:04 |
| mjturek | I think I found it | 18:05 |
| mjturek | https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/1972/logs/logs/containers-successfully-built.log | 18:05 |
| mjturek | but no errors | 18:05 |
| mjturek | lets try to pull an earlier hash | 18:05 |
| rfolco | yeah, compare sizes for example | 18:05 |
| mjturek | rfolco we built on that hash twice | 18:07 |
| mjturek | is that okay? | 18:07 |
| baha | Potential collision? | 18:08 |
| rfolco | don't see any problems | 18:08 |
| mjturek | trying to pull this now a5f1d5c6e280f12d048ba7ebd2c38189ec9c5070_6e3b098e_ppc64le | 18:09 |
| mjturek | same error | 18:09 |
| mjturek | rfolco seeing it on other containers as well | 18:12 |
| rfolco | mjturek, this is weird | 18:13 |
| mjturek | very | 18:13 |
| rfolco | mjturek, docker pull, did you try? | 18:13 |
| zbr | weshay: sshnaidm: https://review.opendev.org/#/c/704938/ would be the correct fix, but as I said this morning should have being be based directly on master | 18:14 |
| zbr | because its current parrent failed at the gate, and 8h we are back where we started from. | 18:14 |
| weshay | zbr, still have something borked though.. compression looks good upstream.... | 18:16 |
| weshay | https://review.rdoproject.org/zuul/build/7454de5906c34f1ebab2e009b41622a9 | 18:16 |
| weshay | has no undercloud or overcloud logs | 18:16 |
| weshay | but the patch is right.. and not the cause | 18:17 |
| mjturek | rfolco working on it | 18:17 |
| weshay | zbr, jobs are fucked now .. and we can't merge this until some other patches land | 18:17 |
| * weshay steps away | 18:17 | |
| weshay | hungry | 18:18 |
| rfolco | mjturek, same hash for _x86_64 works | 18:19 |
| rfolco | mjturek, see if you can find something, I'm working on something else, if need anything just ping me | 18:20 |
| *** sshnaidm is now known as sshnaidm|afk | 18:20 | |
| mjturek | rfolco: same issue with a docker pull | 18:20 |
| rfolco | :( | 18:22 |
| weshay | rlandy, can you do the honors of another +2 https://review.opendev.org/#/c/705051/ | 18:25 |
| rlandy | weshay: done | 18:26 |
| rlandy | weshay: ok - 29 minutes late ... but this works on baremetal testing ... https://review.opendev.org/#/c/705052/ | 18:29 |
| rlandy | you can check my run on the tmate | 18:29 |
| rlandy | TASK [tripleo-inventory : add overcloud node to ansible] *************************************************************************************************************************************$│··············· | 18:30 |
| rlandy | changed: [localhost] => (item={'value': u'10.9.122.137', 'key': u'overcloud-novacompute-0'}) │··············· | 18:30 |
| rlandy | changed: [localhost] => (item={'value': u'10.9.122.134', 'key': u'overcloud-controller-2'}) │··············· | 18:30 |
| rlandy | changed: [localhost] => (item={'value': u'10.9.122.135', 'key': u'overcloud-controller-1'}) │··············· | 18:30 |
| rlandy | changed: [localhost] => (item={'value': u'10.9.122.146', 'key': u'overcloud-controller-0'}) | 18:30 |
| weshay | rlandy, we should really switch the depends on | 18:31 |
| rlandy | zbr: looking | 18:31 |
| weshay | rlandy, https://review.opendev.org/#/c/680571 depends on https://review.opendev.org/#/c/705052/5 | 18:31 |
| weshay | rlandy, please make that change | 18:31 |
| weshay | we need to merge the inventory patch first | 18:32 |
| rlandy | weshay: I wanted to see the ovb test run | 18:32 |
| weshay | rlandy, you would either way | 18:32 |
| rlandy | true | 18:32 |
| rlandy | weshay: ugh - in the same repo | 18:34 |
| rlandy | rebase | 18:34 |
| *** sshnaidm|afk is now known as sshnaidm|off | 18:53 | |
| migi | FYI upshift is now refusing to spawn instances | 19:00 |
| rlandy | charming | 19:12 |
| rlandy | is the floating ips again> | 19:12 |
| migi | rlandy: yep | 19:16 |
| weshay | rlandy, migi .. you guys haven't heard? | 19:22 |
| weshay | IT'S NOTHING WORKS THURSDAY!!!!! | 19:23 |
| weshay | woot woot | 19:23 |
| mjturek | rfolco: so some debugging it seems to be an issue with the base container | 19:24 |
| mjturek | this one blob of the base container seems to be problematic | 19:25 |
| mjturek | 23bd9eb8fdc0 | 19:25 |
| rfolco | mjturek, is the source container a real centos ppc base one ? | 19:25 |
| mjturek | i am not sure | 19:26 |
| mjturek | hmmm let's see where it comes from I guess? | 19:26 |
| rfolco | mjturek, nah ignore me. its the rhel8 that we build a local base and then the rest on top of it. | 19:31 |
| rfolco | mjturek, for ppc, base should follow kolla dockerfile https://github.com/openstack/kolla/tree/master/docker/base | 19:31 |
| *** jmasud has joined #oooq | 19:31 | |
| rfolco | just like x86 | 19:31 |
| rfolco | mjturek, hmmmmmmmm | 19:33 |
| rfolco | look for arch in that dockerfile... | 19:33 |
| rfolco | hmmmmmm | 19:33 |
| rfolco | hmmmmmm | 19:33 |
| rfolco | I've got a "cow" moment now | 19:34 |
| rfolco | hmmmmm | 19:34 |
| rfolco | maybe its ok, just rabbitmq special case | 19:35 |
| rfolco | mjturek, what did make you think base is wrong/broken? | 19:38 |
| *** jmasud has quit IRC | 19:42 | |
| baha | rfolco: When we attempt to grab containers, the first blob (whose hash matches up with the first blob of the base container) has to re-attempt downloading many times. It ends up being the last blob to download, and as soon as the progress bar completes, we get the unexpected EOF error | 20:11 |
| *** hamzy_ has joined #oooq | 20:11 | |
| *** hamzy has quit IRC | 20:14 | |
| rfolco | baha, mjturek: don't know what to try... maybe build only the base one manually and push to dockerhub, then pull | 20:16 |
| rfolco | I got the steps documented for centos8, you can ignore the specifics and just run the commands | 20:18 |
| rfolco | https://hackmd.io/dSagCbocQ4KSVEZR1uf8Tw | 20:18 |
| rlandy | migi: is it expected for codeng to pass for a password when checking out patches? | 20:19 |
| rlandy | trying to pull these patches https://code.engineering.redhat.com/gerrit/#/q/topic:17-standup+status:open | 20:19 |
| rlandy | weshay: ^^ any idea? | 20:26 |
| weshay | sec | 20:28 |
| mjturek | rfolco baha: we could try building locally and then seeing if the base image works as a container | 20:31 |
| mjturek | thoughts? | 20:31 |
| rfolco | mjturek, yes, and you can even use openstack command by tweaking /usr/share/tripleo-common/container-images/overcloud_containers.yaml.j2 | 20:33 |
| rfolco | from tripleo-common | 20:33 |
| rlandy | ha - git review orked | 20:33 |
| rlandy | worked | 20:33 |
| rlandy | weird | 20:33 |
| rfolco | mjturek, like pick one that does not have much dependencies and remove the rest from that file | 20:37 |
| rfolco | mjturek, this file comes from https://github.com/openstack/tripleo-common/blob/master/container-images/overcloud_containers.yaml.j2 | 20:37 |
| *** jmasud has joined #oooq | 20:37 | |
| mjturek | rfolco we were just gonna run the job and build all the containers | 20:38 |
| mjturek | so we guarantee we're not missing something | 20:38 |
| *** jtomasek has quit IRC | 20:38 | |
| rfolco | mjturek, ok, this was for the manual steps to test quicker | 20:38 |
| mjturek | gotcha gotcha | 20:39 |
| *** jtomasek has joined #oooq | 20:43 | |
| *** jfrancoa has quit IRC | 20:43 | |
| *** jmasud has quit IRC | 20:44 | |
| *** jmasud has joined #oooq | 20:45 | |
| *** jmasud has quit IRC | 20:47 | |
| *** holser has joined #oooq | 20:55 | |
| *** Trevor_V has quit IRC | 20:55 | |
| *** apetrich has quit IRC | 20:55 | |
| *** rfolco has quit IRC | 21:24 | |
| *** rfolco has joined #oooq | 21:24 | |
| weshay | rlandy, didn't we say no patches first pass? | 21:25 |
| weshay | rlandy, baby steps | 21:26 |
| rlandy | no patches ==failure | 21:26 |
| rlandy | tried that baby step | 21:26 |
| rlandy | in big school now | 21:26 |
| rlandy | nvm - I managed to git-review the patches to my box | 21:27 |
| *** zbr_ has joined #oooq | 21:31 | |
| rlandy | ugh | 21:35 |
| rlandy | ok container build experts ... what does this mean? http://pastebin.test.redhat.com/832013 | 21:36 |
| rlandy | rfolco: ^^? | 21:36 |
| *** zbr_ has quit IRC | 22:02 | |
| *** ssbarnea- has joined #oooq | 22:08 | |
| rlandy | weshay: ping re: post failure | 22:19 |
| rlandy | we have no way to qualify inventory change | 22:19 |
| *** apetrich has joined #oooq | 22:55 | |
| weshay | rlandy, oh because of the current state of logging? | 23:06 |
| weshay | rlandy, https://trello.com/c/0pT1zkSe/1316-cixlp1861378tripleociproa-multiple-postfailure-on-master-periodic-pipeline | 23:08 |
| *** jmasud has joined #oooq | 23:12 | |
| rlandy | ok | 23:18 |
| *** jmasud has quit IRC | 23:18 | |
| rlandy | weshay: the patches marked as the fix also show the post failure | 23:23 |
| *** ysandeep has joined #oooq | 23:40 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!