| *** rlandy has quit IRC | 00:50 | |
| *** rfolco has quit IRC | 01:39 | |
| *** apetrich has quit IRC | 02:14 | |
| *** skramaja has joined #oooq | 03:21 | |
| *** skramaja has quit IRC | 03:36 | |
| *** skramaja has joined #oooq | 03:36 | |
| *** jfrancoa has joined #oooq | 04:30 | |
| *** udesale has joined #oooq | 04:33 | |
| *** ykarel|away has joined #oooq | 04:43 | |
| *** ykarel|away is now known as ykarel | 04:46 | |
| *** saneax has joined #oooq | 04:52 | |
| *** raukadah is now known as chkumar|rover | 04:54 | |
| *** ratailor has joined #oooq | 04:56 | |
| *** ratailor has quit IRC | 04:59 | |
| *** ratailor has joined #oooq | 04:59 | |
| *** marios has joined #oooq | 05:08 | |
| *** marios is now known as marios|ruck | 05:08 | |
| chkumar|rover | marios|ruck: Good morning | 05:09 |
|---|---|---|
| chkumar|rover | marios|ruck: running full tempest scenario standlone testproject here https://review.rdoproject.org/r/#/c/28860/ | 05:10 |
| marios|ruck | o/ chkumar|rover | 05:11 |
| *** ykarel has quit IRC | 05:24 | |
| *** ykarel has joined #oooq | 05:29 | |
| *** skramaja_ has joined #oooq | 05:39 | |
| *** skramaja has quit IRC | 05:39 | |
| *** udesale_ has joined #oooq | 05:45 | |
| *** udesale has quit IRC | 05:47 | |
| *** skramaja_ has quit IRC | 06:19 | |
| *** skramaja has joined #oooq | 06:19 | |
| *** udesale_ has quit IRC | 06:24 | |
| chkumar|rover | marios|ruck: full tempest scenario job for c8 train https://review.rdoproject.org/zuul/build/ed6cb7cf707548449725dac971e71ea3 is passing | 06:31 |
| marios|ruck | chkumar|rover: nice | 06:31 |
| chkumar|rover | marios|ruck: Are we seeing introspection related issue on rdo-cloud also? | 06:32 |
| marios|ruck | chkumar|rover: but we are still blocked on the pcs issue https://review.opendev.org/#/c/744192/ | 06:32 |
| marios|ruck | chkumar|rover: well i havn't seen any myself, but someone (maybe jpena|off ? ) mentioned he saw some on that last cix call | 06:32 |
| chkumar|rover | context https://trello.com/c/aH5gJoEl/1571-cixlp1885314tripleociproa-ovb-jobs-running-on-vexxhost-show-some-nodes-failing-introspection-step | 06:33 |
| chkumar|rover | yup I tried to find introspection issue on rdolcloud yesterday but not able to find | 06:34 |
| chkumar|rover | may be sshnaidm|afk knows | 06:34 |
| sshnaidm|afk | chkumar|rover, I didn't see introspection issues on rdo-cloud | 06:37 |
| chkumar|rover | sshnaidm|afk: thanks, let's see what jpena|off say. | 06:37 |
| sshnaidm|afk | at least not to such an extent | 06:39 |
| marios|ruck | brb | 06:40 |
| *** marios|ruck has quit IRC | 06:40 | |
| *** marios has joined #oooq | 06:44 | |
| *** marios is now known as marios|ruck | 06:44 | |
| sshnaidm|afk | chkumar|rover, vexx: https://bit.ly/3kfhIOb rdo-cloud: https://bit.ly/2Xu4jIG | 06:46 |
| sshnaidm|afk | very easy to see the difference | 06:46 |
| akahat | chkumar|rover, o/ | 06:52 |
| *** udesale has joined #oooq | 06:52 | |
| chkumar|rover | sshnaidm|afk: thanks for the links, make sense | 06:52 |
| chkumar|rover | akahat: \o | 06:53 |
| akahat | chkumar|rover, i was checking your comments on: https://review.opendev.org/#/c/731704, you are asking to add other passing tests. | 06:53 |
| chkumar|rover | akahat: yes | 06:53 |
| akahat | chkumar|rover, those tests are not passing.. they are skipping.. still you want me to add it? | 06:53 |
| chkumar|rover | akahat: check the link I have added | 06:53 |
| chkumar|rover | akahat: there are 7 more tests passing, please add those | 06:54 |
| akahat | chkumar|rover, this link: https://00ed44ea7395331a3a0a-0e43b9ad7840c03edacec0d2321d06cd.ssl.cf5.rackcdn.com/744819/1/check/tripleo-ci-centos-8-scenario012-standalone/0ae805d/logs/undercloud/var/log/tempest/stestr_results.html | 06:54 |
| akahat | chkumar|rover, expand it.. those are skipped. | 06:54 |
| chkumar|rover | akahat: ironic_tempest_plugin.tests.api.admin.test_chassis.TestChassis | 06:57 |
| chkumar|rover | akahat: except one all are passing | 06:57 |
| akahat | chkumar|rover, looking | 06:57 |
| akahat | chkumar|rover, okay. I can add it. | 06:58 |
| chkumar|rover | akahat: add one of the test in skip list with a bug | 06:58 |
| chkumar|rover | got it | 06:58 |
| akahat | chkumar|rover, ack | 06:59 |
| *** apetrich has joined #oooq | 07:06 | |
| ykarel | chkumar|rover, marios|ruck filed bug for cliff update https://bugs.launchpad.net/tripleo/+bug/1890389 | 07:08 |
| openstack | Launchpad bug 1890389 in tripleo "[Master] tripleoclient does not play well with cliff-3.4.0" [High,Triaged] | 07:08 |
| chkumar|rover | ykarel: thanks, looking into that | 07:09 |
| marios|ruck | ykarel: ack | 07:12 |
| *** skramaja has quit IRC | 07:13 | |
| *** bogdando has joined #oooq | 07:30 | |
| *** tosky has joined #oooq | 07:38 | |
| *** jpena|off is now known as jpena | 07:54 | |
| *** skramaja has joined #oooq | 07:56 | |
| jpena | chkumar|rover, marios|ruck, sshnaidm|afk two examples of introspection failures from RDO Cloud yesterday: https://logserver.rdoproject.org/62/744762/1/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/7ecd8a2/ and https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-ussuri/081a430/ | 07:59 |
| jpena | that said, it happens much more often on vexxhost, but we need to understand why it happens before we can understand why it happens more often on one cloud than another | 08:00 |
| marios|ruck | jpena: thanks | 08:00 |
| chkumar|rover | jpena: more data here vexx: https://bit.ly/3kfhIOb rdo-cloud: https://bit.ly/2Xu4jIG | 08:00 |
| *** dtantsur|afk is now known as dtantsur | 08:04 | |
| marios|ruck | chkumar|rover: oof https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable2 | 08:10 |
| chkumar|rover | marios|ruck: there was a node failure at particular time | 08:11 |
| marios|ruck | chkumar|rover: yeah everything else was skipped https://review.rdoproject.org/zuul/buildset/63414913629b4829afe5a3ad8818e91c | 08:12 |
| chkumar|rover | it impacted component job also | 08:12 |
| marios|ruck | chkumar|rover: also master | 08:12 |
| chkumar|rover | marios|ruck: checking on #rhos-ops | 08:18 |
| chkumar|rover | regarding node failure | 08:18 |
| chkumar|rover | marios|ruck: regarding introspection issue, do we need a seperate bug for passing it to ironic team? | 08:19 |
| marios|ruck | chkumar|rover: why not https://bugs.launchpad.net/tripleo/+bug/1885314 or you mean some other introspection issue | 08:20 |
| openstack | Launchpad bug 1885314 in tripleo "vexx: OVB master job running on vexxhost show some nodes failing introspection step" [Critical,In progress] - Assigned to Ronelle Landy (rlandy) | 08:20 |
| chkumar|rover | marios|ruck: nope, it will work, let me add the logs from rdo-cloud also there | 08:21 |
| marios|ruck | chkumar|rover: ack let's use that since there is also existing cix card we can use for escalation to hardware provis team (maybe it was already escalated need to check ) | 08:22 |
| chkumar|rover | sure | 08:22 |
| *** tosky has quit IRC | 08:26 | |
| *** tosky has joined #oooq | 08:27 | |
| *** holser has joined #oooq | 08:48 | |
| *** derekh has joined #oooq | 08:51 | |
| *** sshnaidm|afk is now known as sshnaidm | 08:56 | |
| sshnaidm | jpena, I think there was always sporadic introspection failures in jobs, it's not so uncommon since virtual BMs are not so reliable. But percentage of failures is very different, like 25%-30% of vexx jobs fail on that, while it's only 2 jobs yesterday in rdo cloud | 08:58 |
| sshnaidm | jpena, given the latest stack failures because of neutron, I'd blame neutron for this too | 08:59 |
| jpena | sshnaidm: I'm comparing the ironic-inspector and dnsmasq log from a failure on each cloud, and the symptoms are pretty similar: all overcloud nodes start to introspect, some nodes report back but some others don't | 09:02 |
| jpena | In dnsmasq I see all nodes request and receive an IP address via DHCP, so there is connectivity | 09:03 |
| sshnaidm | jpena, the timeouts can be with ipmi requests | 09:03 |
| jpena | sshnaidm: so how can we debug that? | 09:04 |
| sshnaidm | jpena, a one million question | 09:26 |
| sshnaidm | I'd try to involve ironic people maybe, at least for showing how can we find failure reasons | 09:27 |
| sshnaidm | but I don't know what is the scope of our work for vexx | 09:27 |
| ykarel | sshnaidm, is bmc image same on both vexxhost and rdocloud, i remember you pushed different images on both clouds when we were debugging pyghmi and ipmitool issue | 09:28 |
| ykarel | not aware if both were synched later | 09:29 |
| sshnaidm | ykarel, I think so | 09:29 |
| sshnaidm | it just didn't work w/o right image | 09:29 |
| ykarel | sshnaidm, okk if both are same | 09:30 |
| sshnaidm | I'll check for being sure 100% | 09:30 |
| sshnaidm | ykarel, yeah, exactly the same, as ipxe-boot image too | 09:34 |
| ykarel | sshnaidm, Thanks | 09:34 |
| marios|ruck | chkumar|rover: did you update the prog call doc or was that weshay? | 09:46 |
| marios|ruck | chkumar|rover: i just updated it a bit - i think train 8 last promotion 27 not 30th right? | 09:46 |
| chkumar|rover | marios|ruck: nope i have not | 09:46 |
| marios|ruck | chkumar|rover: ack thanks | 09:46 |
| chkumar|rover | marios|ruck: 27th | 09:47 |
| marios|ruck | chkumar|rover: thanks for sanity check | 09:48 |
| holser | marios|ruck good morning man | 09:55 |
| marios|ruck | o/ holser | 09:55 |
| holser | marios|ruck any plans to merge https://review.opendev.org/#/c/742418/7 | 09:55 |
| marios|ruck | holser: yeah ... let me revisit/rebase/recheck those today ... | 09:55 |
| holser | marios|ruck thanks a lot man! | 09:56 |
| holser | marios|ruck++ | 09:56 |
| marios|ruck | holser: this one, the patch ontop, has been playing recheck roullette for many days now | 09:56 |
| marios|ruck | https://review.opendev.org/#/c/742182/4 | 09:56 |
| marios|ruck | holser: ^ | 09:56 |
| marios|ruck | holser: but i can unchain them if standalone is also ready now... | 09:56 |
| holser | standalone - Damina Ciabrini and me are working on it | 09:57 |
| holser | haproxy issues fixed we need to fix rabbitmq | 09:57 |
| holser | but yeah, let' | 09:57 |
| marios|ruck | holser: ack yeah we need ussuri fix https://review.opendev.org/#/c/744676/1 | 09:58 |
| holser | s unchain if no progress with standalone today | 09:58 |
| marios|ruck | chkumar|rover: stepping afk few mins quick errand/food | 10:15 |
| *** marios|ruck is now known as marios|ruck|afk | 10:16 | |
| chkumar|rover | sure | 10:16 |
| *** marios|ruck|afk is now known as marios|ruck | 10:33 | |
| chkumar|rover | marios|ruck: bhagyashris https://review.rdoproject.org/r/#/c/28624/ is now ready to go | 10:50 |
| marios|ruck | holser: https://review.opendev.org/#/c/742418/7 still blocked on https://review.opendev.org/#/c/744676/ merging and we should not merge this until the job is green in test https://review.opendev.org/#/c/739457/ | 10:50 |
| marios|ruck | holser: so won't bother rebasing for now ^ | 10:51 |
| marios|ruck | chkumar|rover: ack | 10:51 |
| holser | marios|ruck rebased | 10:51 |
| marios|ruck | holser: ? | 10:51 |
| holser | marios|ruck I mean - https://review.opendev.org/#/c/739457/ | 10:52 |
| holser | let's wait and merge them all | 10:52 |
| marios|ruck | holser: ack | 10:52 |
| ykarel | chkumar|rover, marios|ruck is the "Too Many Requests for url" issue still there? | 11:26 |
| ykarel | i seen in some of the jobs in a patch | 11:26 |
| chkumar|rover | ykarel: not seen today, can you point me the patch | 11:27 |
| ykarel | https://logserver.rdoproject.org/19/743919/6/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-ussuri-branch/237a2f7/logs/undercloud/home/zuul/undercloud_install.log.txt.gz | 11:27 |
| marios|ruck | ykarel: yeah saw it at least once today in gate fails * http://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d11/743260/3/gate/tripleo-ci-centos-7-standalone-upgrade-train/d11c9ef/logs/undercloud/home/zuul/standalone_deploy.log | 11:28 |
| marios|ruck | chkumar|rover: ykarel: added comment ^ https://bugs.launchpad.net/tripleo/+bug/1889122/comments/16 | 11:30 |
| openstack | Launchpad bug 1889122 in tripleo "mirror timeouts in upstream causing undercloud and standalone failures" [Critical,Fix released] - Assigned to chandan kumar (chkumar246) | 11:30 |
| *** jpena is now known as jpena|lunch | 11:39 | |
| *** rfolco has joined #oooq | 11:51 | |
| *** rlandy has joined #oooq | 12:03 | |
| rlandy | chkumar|rover: hey | 12:08 |
| chkumar|rover | rlandy: Hello | 12:08 |
| rlandy | chkumar|rover: re: comments in https://code.engineering.redhat.com/gerrit/#/c/207802/ | 12:08 |
| rlandy | thanks for pointing out the error | 12:08 |
| rlandy | for the longest time it wouldn't even find the roles | 12:08 |
| rlandy | I think I need some of the depends-on patches you had | 12:09 |
| rlandy | ^^ did you hit that error in train? | 12:09 |
| chkumar|rover | rlandy: nope | 12:09 |
| rlandy | hmmm ... wonder if I need to update tripleo-ci-testing | 12:10 |
| rlandy | hash is out about a week | 12:10 |
| chkumar|rover | rlandy: https://review.opendev.org/#/q/topic:tcib/train+(status:open+OR+status:merged) | 12:10 |
| chkumar|rover | is the series of patches which are yet to land | 12:10 |
| chkumar|rover | but the error is not related in any of the patches | 12:10 |
| chkumar|rover | I need to find out from where that .so file is coming | 12:11 |
| rlandy | chkumar|rover: ok - so those were from errors you got in individual conatiner builds | 12:11 |
| rlandy | I can't depends on those until they hit downstream | 12:11 |
| chkumar|rover | rlandy: https://github.com/containers/podman/issues/3234 | 12:13 |
| chkumar|rover | can you check selinux | 12:13 |
| rlandy | chkumar|rover: eek can't turn selinux off for downstream | 12:14 |
| rlandy | but I can try | 12:14 |
| rlandy | let me update hash and try that | 12:14 |
| chkumar|rover | sure | 12:15 |
| rlandy | chkumar|rover: unrelated ... should any tempest repos be ported downstream? | 12:16 |
| rlandy | tempest skiplist etc. | 12:16 |
| chkumar|rover | rlandy: nope | 12:16 |
| rlandy | ok | 12:16 |
| chkumar|rover | rfolco: rlandy weshay|ruck https://review.rdoproject.org/r/#/c/28624/ is ready now | 12:17 |
| *** derekh has quit IRC | 12:24 | |
| weshay|ruck | chkumar|rover, marios|ruck fyi.. https://review.opendev.org/#/c/744192/ is not looking good in the gate | 12:28 |
| * weshay|ruck looking at it | 12:28 | |
| chkumar|rover | weshay|ruck: checking | 12:29 |
| weshay|ruck | 2020-08-05 11:52:55 | Deploying templates in the directory /tmp/tripleoclient-qsastuai/tripleo-heat-templates | 12:29 |
| weshay|ruck | 2020-08-05 11:52:55 | Initializing overcloud plan deployment | 12:29 |
| weshay|ruck | 2020-08-05 11:52:55 | sys:1: ResourceWarning: unclosed <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.1', 48914), raddr=('192.168.24.1', 8080)> | 12:29 |
| weshay|ruck | 2020-08-05 11:52:55 | sys:1: ResourceWarning: unclosed <socket.socket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.1', 40222)> | 12:29 |
| weshay|ruck | 2020-08-05 11:52:55 | sys:1: ResourceWarning: unclosed <socket.socket fd=7, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.24.1', 58638)> | 12:29 |
| weshay|ruck | 2020-08-05 11:52:55 | sys:1: ResourceWarning: u | 12:29 |
| marios|ruck | weshay|ruck: chkumar|rover: few fails in fact there | 12:29 |
| weshay|ruck | c7 is upload for image", "ERROR:root:Image prepare failed: Modifying image 192.168.24.1:8787/tripleotrain/centos-binary-cron:6f44509dcb4faa4bc0340ae138c86d77ef2e2c84_14a932b4 failed", | 12:30 |
| weshay|ruck | marios|ruck, chkumar|rover this is the second or third attempt in gate.. w/ failures | 12:31 |
| weshay|ruck | standalone hit requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: http://mirror.dfw.rax.opendev.org:8082/v2/tripleotraincentos8/centos-binary-cinder-api/blobs/sha256:ffb7b72532ce039ce5dc10282b883326759365ae37d354647887d10b2ccb80ad | 12:31 |
| weshay|ruck | which should be fixed now | 12:31 |
| rfolco | chkumar|rover, I'll add my notes to https://hackmd.io/M1IjwgcPQLK_ttOLFUSPHQ?both | 12:32 |
| marios|ruck | weshay|ruck: yeah we've seen that in gate couple times today noted in the hackmd | 12:32 |
| marios|ruck | weshay|ruck: under transient/no bug for today | 12:32 |
| chkumar|rover | rfolco: sure | 12:32 |
| rfolco | chkumar|rover, there is an automated way | 12:32 |
| weshay|ruck | marios|ruck, chkumar|rover this was intended to prevent that https://review.opendev.org/#/c/744467/ | 12:33 |
| chkumar|rover | yes, does that patch picked up | 12:34 |
| chkumar|rover | yes, the patch is there | 12:36 |
| *** udesale has quit IRC | 12:37 | |
| *** udesale has joined #oooq | 12:38 | |
| chkumar|rover | rfolco: yup | 12:41 |
| *** jpena|lunch is now known as jpena | 12:46 | |
| weshay|ruck | rlandy, prod-chain mtg is canceled | 12:58 |
| weshay|ruck | rlandy, should we setup a sync w/ jjoyce? | 12:58 |
| rlandy | weshay|ruck: ack | 12:58 |
| rlandy | weshay|ruck: can catch him in the afternoon | 12:58 |
| weshay|ruck | rlandy, may want to give him a heads up.. they are busy getting 16.1 out | 12:59 |
| rlandy | weshay|ruck: just about the missing downstream imports - or something else? | 13:00 |
| weshay|ruck | rlandy, think we also had a question about tripleo-operator-ansible and openstack-ansible-collections | 13:01 |
| rlandy | yeah - missing downstream | 13:01 |
| rlandy | will ping him now | 13:01 |
| weshay|ruck | rlandy, however openstack-ansible-collections is probably only for osp-17 ussuri/master right sagi? | 13:02 |
| weshay|ruck | even though we'll have a path for it to be used w/ any release.. just not via rpm | 13:02 |
| sshnaidm | weshay|ruck, tbh it can be for every release | 13:02 |
| rlandy | there is no train branch atm | 13:02 |
| rlandy | or ussuri | 13:03 |
| rlandy | we're just trying to figure what goes downstream | 13:03 |
| sshnaidm | prior to train it's not needed, but it can be used in any release | 13:03 |
| weshay|ruck | sshnaidm, agree but need to see what rel-del says. re: rpm | 13:03 |
| weshay|ruck | bah.. need a link to the osp-16 dlrn | 13:04 |
| rlandy | sshnaidm: basically, will that repo ever branch? | 13:04 |
| weshay|ruck | the one I have in bm is old | 13:04 |
| rlandy | per release | 13:04 |
| * weshay|ruck looks at release | 13:04 | |
| weshay|ruck | files | 13:04 |
| sshnaidm | rlandy, no | 13:05 |
| sshnaidm | it's completely branchless | 13:06 |
| rlandy | weshay|ruck: ^^ ok so then rel del question whether that branches on a downstream basis | 13:06 |
| sshnaidm | possibly it should be tied in downstream for a specific version.. | 13:06 |
| rlandy | http://git.app.eng.bos.redhat.com/git/?q=openstack-ansible-collections - nothing on collections | 13:06 |
| rlandy | and tripleo-operator-ansible.git/ - only 17. | 13:07 |
| rlandy | weshay|ruck: going to #rhos-delivery | 13:08 |
| *** derekh has joined #oooq | 13:09 | |
| chkumar|rover | rfolco: weshay|ruck https://review.rdoproject.org/r/28624 when free! | 13:09 |
| sshnaidm | tripleo-operators should be from ussuri only | 13:10 |
| sshnaidm | rlandy, weshay|ruck I think we can mess with openstack collections, since it's approved now for pulling from galaxy only, iirc | 13:11 |
| weshay|ruck | k | 13:11 |
| rfolco | chkumar|rover, I finished my notes on https://hackmd.io/M1IjwgcPQLK_ttOLFUSPHQ?both#Promoter-Server---development-environment | 13:11 |
| rfolco | chkumar|rover, weshay|ruck bhagyashris if you do the manual setup, disable the service !!! | 13:12 |
| rfolco | just added a warning at the top of the page | 13:12 |
| chkumar|rover | rfolco: I have added instruction with stage setup | 13:13 |
| chkumar|rover | rfolco: but good point | 13:13 |
| rfolco | nic | 13:13 |
| rfolco | e | 13:13 |
| rfolco | th | 13:13 |
| rfolco | x | 13:13 |
| chkumar|rover | rfolco: I will try your's instruction then we can move it to a review | 13:13 |
| bhagyashris | rfolco, ack | 13:13 |
| weshay|ruck | rfolco, do you have some time to go through the jira board w/ me today? | 13:16 |
| rlandy | chkumar|rover: better w/o selinux ... 108 out of 109 containers built | 13:18 |
| chkumar|rover | rlandy: awesome | 13:21 |
| weshay|ruck | still trying to land https://review.opendev.org/#/c/744192/ | 13:21 |
| *** udesale has quit IRC | 13:21 | |
| marios|ruck | weshay|ruck: Error: Unauthorized for url: http://mirror.gra1.ovh.opendev.org:8082/v2/tripleotrain/centos-binary-swift-proxy-server/blobs/sha256:8c52e4b4073b345c2d2658e336423c14821e875103b07530a839b662afd9efa6\ | 13:22 |
| *** udesale has joined #oooq | 13:22 | |
| marios|ruck | weshay|ruck: https://2eaf2007b9319510891b-f67dc2afe768675df5d36f2071b73ef5.ssl.cf1.rackcdn.com/744192/1/gate/tripleo-ci-centos-7-containers-multinode/37f0795/logs/undercloud/home/zuul/undercloud_install.log | 13:22 |
| marios|ruck | weshay|ruck: looking at /744192/ i just hit recheck | 13:22 |
| weshay|ruck | marios|ruck, aye.. /me poking around http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22429%20Client%20Error%3A%20Too%20Many%20Requests%20for%20url%3A%5C%22%20AND%20voting%3A1 | 13:22 |
| rlandy | chkumar|rover: weird error though ... 2020-08-05 09:08:37 | Stderr: 'error committing container for step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin LANG=en_US.UTF-8 container=oci] Command:user Args:[designate] Flags:[] Attrs:map[] Message:USER designate Original:USER designate}: error copying layers and metadata for container "35ce7e06bce131abc05b2b0c6cdc5136808bf5c207f8bcfff90023882f2295b7": Error | 13:23 |
| rlandy | initializing source containers-storage:rhel-binary-designate-base-working-container-36: error extracting layer "d55d43c90500c11c69f029598b1fe63b0aa0d0733f7ba8ff5f19e0a36d959936": no such file or directory\n': RuntimeError: | 13:23 |
| rlandy | but the container builds | 13:23 |
| marios|ruck | weshay|ruck: and a too many requests https://717a5ffd6a01c6a737e0-3e7434681f5fe092d7172e763d28f5ee.ssl.cf5.rackcdn.com/744192/1/gate/tripleo-ci-centos-8-standalone/17352e9/logs/undercloud/home/zuul/standalone_deploy.log | 13:24 |
| chkumar|rover | weshay|ruck: rlandy rfolco if planning to patch promoter next time | 13:24 |
| weshay|ruck | chkumar|rover, ? | 13:25 |
| chkumar|rover | make sure, we can add build_method as tripleo or kolla in criteria file | 13:25 |
| weshay|ruck | ah k | 13:25 |
| chkumar|rover | weshay|ruck: https://review.rdoproject.org/r/#/c/28624/45/ci-scripts/dlrnapi_promoter/config/CentOS-8/master.ini | 13:25 |
| rlandy | ok | 13:25 |
| weshay|ruck | I think the next time we want to patch or update the promoter we should try a fresh install | 13:25 |
| weshay|ruck | chkumar|rover, btw.. ussuri is trying to promote.. but promoter issues http://promoter.rdoproject.org/logs/centos8_ussuri.log | 13:26 |
| * weshay|ruck looks at it | 13:27 | |
| chkumar|rover | weshay|ruck: https://review.rdoproject.org/r/#/c/28842/ | 13:28 |
| weshay|ruck | ah.. chkumar|rover ok.. shall I update the promoter w/ that and give it a go? | 13:28 |
| chkumar|rover | weshay|ruck: yes | 13:28 |
| chkumar|rover | weshay|ruck: 3 things | 13:29 |
| chkumar|rover | sorry | 13:29 |
| chkumar|rover | please pull recent merge patches on promoter | 13:29 |
| chkumar|rover | weshay|ruck: let me get the list | 13:29 |
| chkumar|rover | https://review.rdoproject.org/r/28624 | 13:30 |
| chkumar|rover | https://review.rdoproject.org/r/28842 | 13:30 |
| chkumar|rover | https://review.rdoproject.org/r/28858 | 13:30 |
| chkumar|rover | https://review.rdoproject.org/r/28854 | 13:30 |
| chkumar|rover | then I think we are good there | 13:30 |
| weshay|ruck | hrm | 13:30 |
| weshay|ruck | chkumar|rover, nak.. want to chat? | 13:31 |
| chkumar|rover | weshay|ruck: yes | 13:32 |
| weshay|ruck | chkumar|rover, https://meet.google.com/yjr-ycid-qyi | 13:33 |
| weshay|ruck | chkumar|rover, jump on the promoter.rdo tmate too | 13:33 |
| rfolco | weshay|ruck, jira board, sure, just ping me | 13:35 |
| weshay|ruck | chkumar|rover, https://registry.rdoproject.org:8443/oapi/v1/namespaces/tripleoussuri/imagestreamtags/ | 13:41 |
| *** ratailor has quit IRC | 13:42 | |
| *** TrevorV has joined #oooq | 13:56 | |
| chkumar|rover | weshay|ruck: https://opendev.org/openstack/tripleo-common/src/branch/stable/ussuri/container-images/tripleo_containers.yaml#L217 | 13:57 |
| chkumar|rover | weshay|ruck: https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push-ussuri/90a5876/logs/containers-successfully-built.log | 13:59 |
| weshay|ruck | chkumar|rover, https://registry.rdoproject.org:8443/oapi/v1/namespaces/tripleoussuri/imagestreamtags/ | 13:59 |
| chkumar|rover | weshay|ruck: cae493e19c28cb42b4b04fdf50434c64 | 14:01 |
| chkumar|rover | weshay|ruck: ece67ea0b644891af4cfa7c9e39157ca | 14:03 |
| chkumar|rover | weshay|ruck: ece67ea0b644891af4cfa7c9e39157ca | 14:07 |
| chkumar|rover | ece67ea0b644891af4cfa7c9e39157ca | 14:07 |
| chkumar|rover | weshay|ruck: UAO-863218. | 14:09 |
| weshay|ruck | marios|ruck, you have a sec to chat w/ chkumar|rover and I | 14:13 |
| *** skramaja has quit IRC | 14:13 | |
| marios|ruck | weshay|ruck: you folks not on the open forum ? | 14:14 |
| marios|ruck | weshay|ruck: sure, where you at? i can drop from this | 14:14 |
| weshay|ruck | marios|ruck, https://meet.google.com/yjr-ycid-qyi | 14:16 |
| marios|ruck | weshay|ruck: joining | 14:17 |
| *** sshnaidm is now known as sshnaidm|afk | 14:24 | |
| weshay|ruck | marios|ruck, chkumar|rover https://review.rdoproject.org/r/28870 | 14:39 |
| marios|ruck | weshay|ruck: ack | 14:42 |
| *** ykarel is now known as ykarel|away | 15:21 | |
| weshay|ruck | chkumar|rover, you still around? | 15:22 |
| weshay|ruck | qq.. re: kibana in rdo if you are | 15:23 |
| chkumar|rover | weshay|ruck: yes tell me | 15:23 |
| chkumar|rover | weshay|ruck: I have not used kibana, will explore | 15:23 |
| weshay|ruck | chkumar|rover, does the kibana in rdo use the same search queries as logstash upstream? | 15:23 |
| weshay|ruck | message: "foo error" | 15:23 |
| weshay|ruck | trying to query something from the ovb overcloud_deploy log | 15:24 |
| chkumar|rover | I think nope but jpena can ^^ confirm | 15:24 |
| weshay|ruck | chkumar|rover, ah.. I got it :) | 15:25 |
| weshay|ruck | it does | 15:25 |
| weshay|ruck | just need to expand the timescope | 15:25 |
| chkumar|rover | cool! | 15:25 |
| chkumar|rover | see ya tomorrow | 15:26 |
| *** chkumar|rover is now known as raukadah | 15:26 | |
| *** tosky has quit IRC | 15:31 | |
| * marios|ruck eod | 15:49 | |
| *** marios|ruck is now known as marios|out | 15:54 | |
| *** ykarel|away has quit IRC | 15:58 | |
| rlandy | weshay|ruck: on a scale on 1-10 how bad is it if we run containers builds downstream with selinux permissive? | 16:01 |
| weshay|ruck | rlandy, so it works in permissive and fails w/ selinux on? | 16:03 |
| rlandy | weshay|ruck: ack ... https://github.com/containers/podman/issues/3234 | 16:04 |
| weshay|ruck | ok.. so let's raise a lp bug.. | 16:05 |
| weshay|ruck | rlandy, add that link | 16:05 |
| weshay|ruck | rlandy, flip the job to permissive | 16:05 |
| weshay|ruck | rlandy, it's permissive in upstream right? | 16:06 |
| weshay|ruck | and we'll take it from there | 16:06 |
| rlandy | weshay|ruck: ack | 16:07 |
| *** bogdando has quit IRC | 16:08 | |
| *** jpena is now known as jpena|off | 16:12 | |
| *** jmasud has quit IRC | 16:14 | |
| *** jmasud has joined #oooq | 16:16 | |
| *** marios|out has quit IRC | 16:26 | |
| *** tosky has joined #oooq | 16:29 | |
| *** udesale has quit IRC | 16:32 | |
| *** dtantsur is now known as dtantsur|afk | 16:39 | |
| *** derekh has quit IRC | 17:00 | |
| weshay|ruck | rlandy, ping do you have a dell box in beaker that I can see the power settings | 17:16 |
| weshay|ruck | oh.. nevermind.. I got it working | 17:16 |
| rlandy | weshay|ruck: ack | 17:17 |
| weshay|ruck | oh no I didnt | 17:17 |
| rlandy | weshay|ruck: I think all our boxes should be in beaker .. try | 17:17 |
| weshay|ruck | ya.. but I can't see them all | 17:18 |
| weshay|ruck | oh I could look at the real bm env | 17:18 |
| weshay|ruck | sec | 17:18 |
| rlandy | rdo-ci-fx2-04-s8.v102.rdoci.lab.eng.rdu2.redhat.com | 17:18 |
| *** sshnaidm|afk is now known as sshnaidm | 17:19 | |
| rlandy | lol -ovb downstream - solid as a rock | 17:19 |
| weshay|ruck | rlandy, really? | 17:19 |
| rlandy | funny, rught? | 17:19 |
| weshay|ruck | rlandy, can you please paste the kickstart meta, kernel and post options to ... | 17:21 |
| rlandy | weshay|ruck: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/daf851d1f17b4e57b42b4932f717ba10 | 17:31 |
| holser | weshay|ruck Good afternoon | 18:17 |
| holser | weshay|ruck Could you look at " | 18:17 |
| holser | zuul check" for https://review.opendev.org/#/c/742182/ | 18:17 |
| holser | I don't understand the purpose of tripleo-ci-centos-8-containers-multinode-ussuri & tripleo-ci-centos-8-containers-multinode-train & tripleo-ci-centos-7-containers-multinode-train | 18:18 |
| holser | as this patch seems to be master branch | 18:18 |
| weshay|ruck | sec.. | 18:19 |
| weshay|ruck | holser, that change is to tripleo-ci | 18:50 |
| weshay|ruck | holser, which is branchless... only master... | 18:50 |
| weshay|ruck | holser, so we run older versions stable/ussuri train etc.. to ensure ci has not busted anything | 18:51 |
| holser | I see | 18:51 |
| weshay|ruck | holser, need help getting something merged? | 18:51 |
| holser | not yet | 18:51 |
| holser | weshay|ruck trying to understand https://365e107531fb80af8f5d-c89a70bd7c387c4ac55b47bafacb63d8.ssl.cf2.rackcdn.com/742182/4/gate/tripleo-ci-centos-8-containers-multinode/21f7bae/logs/undercloud/home/zuul/overcloud_deploy.log | 18:51 |
| holser | to get https://review.opendev.org/#/c/742182/4 merged | 18:52 |
| weshay|ruck | looking | 18:52 |
| weshay|ruck | holser, : \"Too Many Requests. Please see https://docs.docker.com/docker-hub/download-rate-limit/\",", | 18:53 |
| holser | weshay|ruck /0\ | 18:53 |
| weshay|ruck | holser, see my email to opendev-discuss[tripleo] | 18:53 |
| weshay|ruck | holser, we have patches up.. but it's going to be shitty until they merge | 18:53 |
| holser | I've already read it | 18:53 |
| weshay|ruck | holser, aye .. sorry | 18:53 |
| holser | there are a few patches from aschultz to remedy the storm | 18:54 |
| weshay|ruck | holser, exactly | 18:54 |
| holser | weshay|ruck I have rebased https://review.opendev.org/#/c/742182/ Could you WF+1 when it passes check | 18:57 |
| weshay|ruck | holser, aye.. will watch and push it through | 18:58 |
| *** tosky has quit IRC | 19:39 | |
| *** saneax has quit IRC | 21:24 | |
| *** tosky has joined #oooq | 21:25 | |
| *** TrevorV has quit IRC | 21:57 | |
| *** rlandy is now known as rlandy|bbl | 22:17 | |
| *** jbadiapa has joined #oooq | 22:49 | |
| *** jbadiapa has quit IRC | 22:50 | |
| *** tosky has quit IRC | 23:14 | |
| *** irclogbot_2 has quit IRC | 23:34 | |
| *** irclogbot_1 has joined #oooq | 23:37 | |
| *** rlandy|bbl is now known as rlandy | 23:53 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!