| *** jmasud has joined #oooq | 00:21 | |
| *** matbu has quit IRC | 00:21 | |
| *** jmasud has quit IRC | 00:38 | |
| *** jmasud has joined #oooq | 00:54 | |
| *** jmasud has quit IRC | 01:42 | |
| *** yolanda has joined #oooq | 01:52 | |
| *** yolanda has quit IRC | 01:52 | |
| *** yolanda has joined #oooq | 01:53 | |
| *** jmasud has joined #oooq | 03:08 | |
| *** jmasud has quit IRC | 04:03 | |
| *** ratailor has joined #oooq | 04:27 | |
| *** udesale has joined #oooq | 04:46 | |
| *** ykarel|away is now known as ykarel | 04:57 | |
| *** jmasud has joined #oooq | 05:02 | |
| *** jmasud has quit IRC | 05:25 | |
| *** jmasud has joined #oooq | 05:29 | |
| *** skramaja has joined #oooq | 05:32 | |
| *** jmasud has quit IRC | 05:40 | |
| *** ysandeep|away is now known as ysandeep | 05:46 | |
| *** jmasud has joined #oooq | 05:58 | |
| *** dpawlik has joined #oooq | 06:02 | |
| *** matbu has joined #oooq | 06:03 | |
| *** marios has joined #oooq | 06:05 | |
| *** jfrancoa has joined #oooq | 06:13 | |
| *** jmasud has quit IRC | 06:18 | |
| *** jmasud has joined #oooq | 06:19 | |
| *** jbadiapa has quit IRC | 06:23 | |
| *** ccamacho has joined #oooq | 06:32 | |
| *** jmasud has quit IRC | 06:45 | |
| *** jbadiapa has joined #oooq | 06:59 | |
| *** udesale has quit IRC | 07:15 | |
| *** udesale has joined #oooq | 07:15 | |
| *** amoralej|off is now known as amoralej | 07:33 | |
| *** ccamacho has quit IRC | 07:40 | |
| *** tosky has joined #oooq | 07:56 | |
| *** jpena|off is now known as jpena | 07:57 | |
| *** holser has joined #oooq | 08:01 | |
| *** holser_ has joined #oooq | 08:10 | |
| *** holser has quit IRC | 08:10 | |
| *** ysandeep is now known as ysandeep|lunch | 08:21 | |
| *** panda|off is now known as panda | 08:34 | |
| zbr | panda: morning | 08:43 |
|---|---|---|
| panda | zbr: is it ? | 08:44 |
| zbr | i need you help with something on iprom, meet? | 08:45 |
| zbr | https://review.rdoproject.org/r/gitweb?p=rdo-infra/ci-config.git;a=blob;f=zuul.d/jobs.yaml#l58 | 08:45 |
| zbr | i need to move this into a single scenario call | 08:45 |
| arxcruz | panda: chandankumar https://review.opendev.org/#/c/725220/ | 08:46 |
| zbr | now is a sequence of scenarios, does not fit pytest testing logic at all. | 08:46 |
| panda | zbr: zbr take a look at https://review.rdoproject.org/r/#/c/26416 | 08:46 |
| panda | zbr: this entire US has already partially been done by rfolco | 08:47 |
| zbr | haha | 08:47 |
| zbr | i was about to do the same | 08:47 |
| panda | arxcruz: I need an RFE for this change | 08:48 |
| panda | arxcruz: with a blueprint | 08:48 |
| panda | arxcruz: and design discussion | 08:48 |
| panda | arxcruz: with risk assessment | 08:49 |
| arxcruz | panda: I can't say if you're joking or not... | 08:49 |
| panda | arxcruz: and cost/benefit calculation | 08:49 |
| panda | arxcruz: I never joke | 08:49 |
| panda | arxcruz: except for no | 08:49 |
| panda | w | 08:49 |
| arxcruz | panda: that was a joke | 08:49 |
| panda | yo dawg | 08:49 |
| panda | I put a joke on your joke so you can joke while you joke | 08:50 |
| zbr | damn. but i will never be able to implement this, that 5th change in a sequence. | 08:50 |
| panda | zbr: well, you'll be able to implement this if you split things correctly :) | 08:51 |
| *** ykarel is now known as ykarel|lunch | 09:07 | |
| zbr | the new role will be called ensure_promoter, to follow the upstream terminology. sounds ok? | 09:09 |
| zbr | out of curiosity, do we still have jenkins jobs around? | 09:10 |
| *** sshnaidm|afk is now known as sshnaidm | 09:40 | |
| *** ysandeep|lunch is now known as ysandeep | 09:51 | |
| *** ykarel|lunch is now known as ykarel | 09:53 | |
| panda | zbr: only for phase1 | 09:56 |
| panda | zbr: ok with me, the important part is to get it done ... upper constrating in centos7 are starting to strange everything | 09:57 |
| panda | strangle | 09:57 |
| panda | I think in a month we won't be able to test anything anymore. | 09:57 |
| zbr | yep, we could be saved by 7.8 | 09:58 |
| zbr | true | 09:58 |
| zbr | panda: what the heck is https://review.rdoproject.org/zuul/build/9417b8a902704b76ba1707d5a878819b/console ? | 09:59 |
| zbr | joys of CI, trying to fix two jobs and two other unrelated start to fail. | 10:00 |
| sshnaidm | weshay|ruck, I set also promotion-blocker for https://bugs.launchpad.net/tripleo/+bug/1879472 | 10:00 |
| openstack | Launchpad bug 1879472 in tripleo "OVB overcloud deploy on vexxhost fails sporadically with "not enough free physical ports" error" [Critical,Triaged] | 10:00 |
| zbr | i wonder if that issue reproduces upstream, or is specific to rdo centos-8 image. | 10:11 |
| zbr | panda: we are f***, ensure-docker broken on centos-8: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ee1/728640/2/check/zuul-jobs-test-ensure-docker-centos-8/ee118f9/job-output.txt | 10:23 |
| zbr | no job will pass until we fix it. | 10:23 |
| panda | zbr: pin down docker version ? | 10:25 |
| *** derekh has joined #oooq | 10:26 | |
| zbr | we need to investigate, local upgrade works fine, is something else broken, but lets discuss on #zuul channel. | 10:31 |
| panda | zbr: or simply skip it | 10:31 |
| panda | zbr: we ar installing our own docker | 10:32 |
| panda | zbr: well, not in the other tox jobs ... but in this we are installing our own | 10:32 |
| zbr | i am working on it. to fix upstream | 10:46 |
| zbr | the bug is that it does upgrade only the client, and not both. | 10:47 |
| zbr | magic like: package docker-ce-3:19.03.9-3.el7.x86_64 requires containerd.io >= 1.2.2-3, but none of the providers can be installed | 10:48 |
| zbr | f***, on my box that comes from ELRepo.org | 10:49 |
| zbr | those dudes have no clue on how to build an rpm, basically they allow cli to be installed out-of-sync with the server. | 10:50 |
| weshay|ruck | sshnaidm, thank you.. the other thing I saw on the good side of things.. when I dequued a pipeline in zuul I saw ovb-manage correctly tear down the env | 10:53 |
| weshay|ruck | so ++ | 10:53 |
| zbr | panda: obviously that I had a 5 month old change that would have prevented that, but it was abandoned. | 10:57 |
| zbr | https://review.opendev.org/#/c/703053/ -- working on it. | 10:58 |
| weshay|ruck | sshnaidm, have you see this error before? Looks very familiar | 11:01 |
| weshay|ruck | 2020-05-19 10:34:37.610395 | primary | AnsibleUndefinedVariable: 'dict object' has no attribute 'overcloud' | 11:01 |
| weshay|ruck | in 2020-05-19 10:34:36.745173 | primary | TASK [overcloud-deploy : Create config-downlad environment] ******************** | 11:01 |
| weshay|ruck | https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset010-ussuri/ec4d40e/job-output.txt | 11:01 |
| chandankumar | sshnaidm, weshay|ruck can you verify check layout https://review.rdoproject.org/r/#/c/27612/ then I will put config changes | 11:06 |
| sshnaidm | weshay|ruck, not familiar to me | 11:27 |
| *** jpena is now known as jpena|lunch | 11:33 | |
| ykarel | weshay|ruck, sshnaidm i think it's happening due to wrong nodeset in multinode jobs | 11:38 |
| ykarel | https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/integration-jobs-ussuri.yaml#L141 | 11:38 |
| ykarel | nodeset override shouldn't be needed if parent is centos8 and multinode job | 11:38 |
| ykarel | single nodeset is used in these multinode job instead of two-centos-node | 11:39 |
| weshay|ruck | ykarel, oof.. what error and where? | 11:40 |
| ykarel | weshay|ruck, https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset010-ussuri/ec4d40e/job-output.txt | 11:40 |
| ykarel | the log u shared above | 11:40 |
| ykarel | AnsibleUndefinedVariable: 'dict object' has no attribute 'overcloud' | 11:40 |
| *** rfolco|rover|off has joined #oooq | 11:42 | |
| *** rfolco|rover|off is now known as rfolco|rover | 11:44 | |
| weshay|ruck | ykarel, hrm.. k thanks.. /me looks | 11:44 |
| weshay|ruck | ykarel, I see ++ | 11:45 |
| weshay|ruck | thanks | 11:45 |
| sshnaidm | ykarel, weshay|ruck I think there is a wrong job definition | 11:52 |
| weshay|ruck | https://review.rdoproject.org/r/27613 | 11:52 |
| sshnaidm | ykarel, weshay|ruck it should run multinode.yml, instead of that it runs a few playbooks | 11:52 |
| sshnaidm | weshay|ruck, yeah, and this too | 11:53 |
| Tengu | weshay|ruck: hello sir :). I think we might get a tiny-mini-issue in train CI related to the container build on buildah | 11:53 |
| weshay|ruck | Tengu, review? | 11:53 |
| Tengu | weshay|ruck: https://bugs.launchpad.net/tripleo/+bug/1879365 added to the already existing LP | 11:54 |
| openstack | Launchpad bug 1879365 in tripleo "[container build] SystemError: The following jobs were incomplete: state=finished" [High,Incomplete] | 11:54 |
| Tengu | apparently you wanted to see some kind of trend | 11:54 |
| Tengu | https://zuul.openstack.org/builds?job_name=tripleo-build-containers-centos-7-buildah&branch=stable%2Ftrain this is something like a trending thing imho. random failure, probably some race condition. | 11:54 |
| weshay|ruck | Tengu, you are pinging me about 1 fail? | 11:58 |
| zbr | panda: weshay|ruck: https://review.opendev.org/#/c/703053/ -- help me before I start pinging infra about it. | 11:58 |
| Tengu | weshay|ruck: well... not only one | 11:58 |
| weshay|ruck | https://zuul.openstack.org/builds?job_name=tripleo-build-containers-centos-7-buildah&branch=stable%2Ftrain | 11:59 |
| Tengu | weshay|ruck: random things are random ;). | 12:00 |
| *** udesale_ has joined #oooq | 12:00 | |
| weshay|ruck | rfolco|rover, ruck /rover sync | 12:01 |
| rfolco|rover | my headset is not working, one sec | 12:02 |
| *** udesale has quit IRC | 12:02 | |
| *** amoralej is now known as amoralej|lunch | 12:08 | |
| *** ccamacho has joined #oooq | 12:18 | |
| *** rlandy has joined #oooq | 12:28 | |
| rlandy | ysandeep: hello ... I'm fixing the scenario standalone parenting | 12:29 |
| *** TrevorV has joined #oooq | 12:30 | |
| ysandeep | rlandy, o/ good morning.. changing to what? | 12:31 |
| rlandy | it's not picking up the rhel playbooks | 12:32 |
| *** jpena|lunch is now known as jpena | 12:33 | |
| ysandeep | rlandy, okay , I faced this in a testpatch - urllib3.exceptions.LocationParseError: Failed to parse: http://osp-trunk.hosted.upshift.rdu2.redhat.com/api-rhel8-osp17/api/report_result , not sure if its related too.. | 12:33 |
| rlandy | yes | 12:34 |
| rlandy | ysandeep: the tenant dies yesterday :( | 12:34 |
| rlandy | and I couldn't test my changes | 12:34 |
| ysandeep | rlandy, great thanks :) I was currently trying to pin point this only.. | 12:35 |
| rlandy | ysandeep: I"m working on it - will let you know once I'm done. | 12:35 |
| rlandy | ysandeep: also wrt env D | 12:35 |
| rlandy | the undercloud should be accessible | 12:35 |
| ysandeep | rlandy, envD was accessible.. envA was not | 12:36 |
| rlandy | ysandeep: let's meet after community meeting | 12:36 |
| rlandy | and I'll show you how to reprovision | 12:36 |
| ysandeep | rlandy, thanks! I basically need help confirming idrac accesibily to all the machines apart from envD i am struggling with the rest. | 12:38 |
| *** ysandeep is now known as ysandeep|brb | 12:44 | |
| sshnaidm | ysandeep, bhagyashris|ruck chandankumar weshay|ruck zbr rlandy rfolco|rover marios panda arxcruz akahat - please vote +1 on review https://review.rdoproject.org/r/#/c/27605/ if you're agree to have your keys on CI hosts, which allows you to connect during the job. If you don't want your keys be there, tell me and I'll drop it. | 12:49 |
| weshay|ruck | chandankumar, anything you need from me? | 12:53 |
| weshay|ruck | rlandy, can we reserve a bit of time to double check components like we did last time for upstream? | 12:55 |
| *** ratailor has quit IRC | 12:57 | |
| rlandy | weshay|ruck: yes - afternoon? | 12:57 |
| weshay|ruck | rlandy, perfect.. thanks | 12:57 |
| *** derekh has quit IRC | 13:01 | |
| rlandy | ugh - why does scenario001 pull manifests??????? | 13:01 |
| *** derekh has joined #oooq | 13:01 | |
| *** amoralej|lunch is now known as amoralej | 13:03 | |
| rlandy | panda: hi - have you been promoting hashes in downstream? | 13:06 |
| rlandy | panda: 35961d102a25c0ff95ef60e9c8910a80 is not in the containers | 13:07 |
| *** Goneri has joined #oooq | 13:09 | |
| rlandy | marios: ^^ did you notice if we have been pushing containers lately? | 13:10 |
| marios | rlandy: no have not | 13:10 |
| *** ykarel is now known as ykarel|afk | 13:11 | |
| rlandy | periodic-tripleo-containers-rhel-8-rhos-17-build-pushopenstack/tripleo-cimasteropenstack-periodic-rhos-17master10142020-05-19T08:49:20FAILURE | 13:11 |
| rlandy | then how did we promote that hash??? | 13:11 |
| rlandy | hello people??? | 13:11 |
| rlandy | 2020-05-19 05:01:07.819707 | primary | 35961d102a25c0ff95ef60e9c8910a80 | 13:12 |
| *** holser_ has quit IRC | 13:14 | |
| *** holser has joined #oooq | 13:14 | |
| rlandy | panda: ^^? | 13:15 |
| panda | rlandy: the internal promoter is disabled | 13:20 |
| rlandy | panda: are you manually promoting? | 13:20 |
| panda | rlandy: no | 13:20 |
| panda | rlandy: I didn't touch the internal instance ssince Friday | 13:21 |
| rlandy | panda: then how was it updated today? | 13:21 |
| *** ysandeep|brb is now known as ysandeep | 13:22 | |
| rlandy | panda: oh nvm | 13:22 |
| rlandy | dir link | 13:22 |
| ysandeep | sshnaidm, posted +1. my keys are present there | 13:24 |
| weshay|ruck | chandankumar, any handoff items? | 13:27 |
| pojadhav | sshnaidm, I have also added my keys in https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/tenant_vars/common.yml#L56-L57 | 13:28 |
| sshnaidm | pojadhav, ack | 13:28 |
| sshnaidm | soniya29, ^^ | 13:28 |
| weshay|ruck | chandankumar, ok if I kill the jobs in latest-released and retrigger w/ https://review.rdoproject.org/r/#/c/27613/ | 13:28 |
| bhagyashris|ruck | sshnaidm, posted +1 | 13:28 |
| soniya29 | sshnaidm, ahh i forgot, I will do it now, give me a moment | 13:31 |
| rfolco|rover | ping community mtg | 13:31 |
| rfolco|rover | arxcruz, rfolco, zbr, panda, sshnaidm, rlandy, marios, ysandeep, bhagyashris, soniya29, pojadhav, akahat, weshay, chandankumar | 13:31 |
| rfolco|rover | community mtg at https://meet.google.com/bqx-xwht-wky?authuser=0 | 13:32 |
| ysandeep | rlandy, let me know whenever you get some free slot and want to catch up o/ | 13:40 |
| arxcruz | rfolco|rover: sorry, my wife went to the market, and I had to stay with my daughter :/ | 13:45 |
| rfolco|rover | arxcruz, you missed nothing. zero. necas-de-pitibiribas. | 13:46 |
| arxcruz | rfolco|rover: lol, it's been a while i haven't hear that | 13:47 |
| rlandy | zbr: maybe you know, in ansible, can you point to two sets of referenced vars? | 13:50 |
| rlandy | ysandeep: k - just need to check up why containers didn't build | 13:51 |
| *** pojadhav is now known as pojadhav|afk | 13:57 | |
| zbr | rlandy: i need more context | 13:57 |
| rlandy | zbr: you know how we use &myvars and then *myvars? | 13:58 |
| zbr | this has nothing to do with ansible, is pure YAML | 13:58 |
| rlandy | zbr: can you use *myvars *yourvars? | 13:58 |
| zbr | ansible has no idea what anchors we use | 13:58 |
| zbr | yes, AFAIK, you can merge two dictionaries, or even override values. | 13:59 |
| rlandy | zbr: can we use multiple anchors? | 13:59 |
| zbr | i am almost sure it works | 13:59 |
| ysandeep | fyi.. a good read https://confluence.atlassian.com/bitbucket/yaml-anchors-960154027.html | 14:00 |
| rlandy | zbr: will let you know | 14:01 |
| zbr | see https://yaml-online-parser.appspot.com/?yaml=-+job1%3A+%26var1%0A++++foo%3A+bar%0A-+job2%3A+%26var2%0A++++aaa%3A+aaa%0A-+job2%3A%0A++++%3C%3C%3A+*var1%0A++++%3C%3C%3A+*var2%0A++++me%3A+yep%0A%0A%0A&type=json | 14:01 |
| rlandy | Unknown configuration error | 14:02 |
| zbr | when in doubt, use this site to test how it would be loaded | 14:02 |
| rlandy | sec | 14:02 |
| zbr | https://gist.github.com/ssbarnea/49b321c9573946d72174c07e5404be31 | 14:02 |
| zbr | probably the url is too long. | 14:02 |
| zbr | anyway, that works as expected. | 14:03 |
| zbr | yep, atlassian article is one of the best about anchors | 14:03 |
| zbr | panda: what do you think about https://bugzilla.redhat.com/show_bug.cgi?id=1756473#c1 ? | 14:07 |
| openstack | bugzilla.redhat.com bug 1756473 in dnf "dnf: package is excluded without explanation" [Medium,Closed: errata] - Assigned to jmracek | 14:07 |
| zbr | not the bug, the comment. | 14:07 |
| zbr | that works better than previous approach, as we endup with latest docker | 14:08 |
| zbr | but the comment mentions the price to pay for it | 14:08 |
| panda | zbr: using --nobest doesn't work ? | 14:08 |
| panda | zbr: I remember hitting this but too, and after getting crazy, --nobest was really the best option | 14:09 |
| zbr | no longer works because you endup with a mismatch of a newer client that no longer work with the server. | 14:09 |
| panda | zbr: yep, I realized that after writing | 14:09 |
| zbr | since yesterday it no longer works | 14:09 |
| zbr | you endup with a docker server 18.x with a client 19.y that gives the error we seen. | 14:09 |
| panda | zbr: does the client come from docker-ce repo ? | 14:10 |
| zbr | yes | 14:10 |
| panda | zbr: cna't we pin the client too ? | 14:10 |
| zbr | take a look at zuul-roles, i doubt you will be allowed to pin it. | 14:10 |
| zbr | you can propose it, but I have doubts it will pass review | 14:11 |
| zbr | in general pinning is not encouraged, as it creates extra maintenance | 14:11 |
| zbr | the assumption is that when you run ensure-docker, you get latest docker | 14:11 |
| panda | zbr: wgrrr, hen something is deprecated, maybe people should ensure there's a full replacement. | 14:11 |
| zbr | especially when installing it from upstream | 14:12 |
| zbr | the docker-ce distribution is a can of worms, everyone will blame the other side | 14:12 |
| zbr | in the end, is for "7" not for 8 | 14:13 |
| rlandy | not quite | 14:14 |
| rlandy | but I have another solution | 14:14 |
| ysandeep | rlandy, :) | 14:15 |
| panda | ysandeep: It doesn't say an inporntant thing | 14:18 |
| panda | ysandeep: the document you linked | 14:18 |
| panda | ysandeep: tried to find the official documentation but it lacks this information either about anchor, guess I discovered the hard way. | 14:19 |
| rlandy | the double anchor doesn;t work the way we set up our anchors | 14:20 |
| rlandy | ie: at vars | 14:20 |
| rlandy | it duplicates vars | 14:21 |
| rlandy | nvm though - multiple parenting helps | 14:21 |
| ysandeep | panda, yes less good material on internet about anchors and usage | 14:21 |
| rlandy | ysandeep: let's meet at half past the hour, ok? | 14:21 |
| panda | ysandeep:just remember this, an anchor is valid only inside a single document | 14:21 |
| rlandy | rekick'ed containers builds | 14:21 |
| panda | ysandeep: you can't reuse it outside | 14:22 |
| rlandy | panda: ack - you can't reference it elsewhere | 14:22 |
| rlandy | tried that trick | 14:22 |
| ysandeep | rlandy, ack o/ | 14:22 |
| rlandy | ysandeep: https://meet.google.com/hfm-vgpk-bfq | 14:32 |
| *** ykarel|afk is now known as ykarel | 14:43 | |
| marios | chandankumar: when you have time please check pvt re internal image server | 14:44 |
| weshay|ruck | rfolco|rover, bhagyashris|ruck stein promoted.. setting fs020 back to criteria | 15:08 |
| *** hamzy_ is now known as hamzy | 15:09 | |
| rfolco|rover | weshay|ruck, testproject failed for that rabbit bug... containers missing... how do I check which hash/tag is avail ? | 15:11 |
| rfolco|rover | weshay|ruck, https://logserver.rdoproject.org/16/27616/1/check/periodic-tripleo-ci-centos-8-scenario004-standalone-train/bba7f08/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 15:11 |
| weshay|ruck | rfolco|rover, which hash is on the box.. or avail? | 15:12 |
| weshay|ruck | not sure I understand.. | 15:12 |
| rfolco|rover | weshay|ruck, https://review.rdoproject.org/r/#/c/27616/ is a testproject that runs the job that failed.... | 15:13 |
| rfolco|rover | weshay|ruck, should i specify a different hash ? | 15:13 |
| rfolco|rover | weshay|ruck, 404 Client Error: Not Found for url: https://trunk.registry.rdoproject.org/v2/tripleotraincentos8/centos-binary-cron/manifests/9f22f412865b46fe428a1188130ca0f7ace577b0_618fa100 | 15:14 |
| weshay|ruck | https://trunk.registry.rdoproject.org:8443/oapi/v1/namespaces/tripleotraincentos8/imagestreamtags/ | 15:14 |
| weshay|ruck | panda, you still around? | 15:15 |
| weshay|ruck | can we chat for a hot sec.. 3min max | 15:15 |
| *** skramaja has quit IRC | 15:17 | |
| panda | weshay|ruck: link ? | 15:19 |
| weshay|ruck | panda, https://meet.google.com/pfg-tbyb-tks | 15:19 |
| dpawlik | weshay|ruck, em, do I see correct that the ssl cert has been expired on registry host ?! | 15:23 |
| rfolco|rover | marios, do you have a testproject example on how to specify a different hash ? | 15:25 |
| rfolco|rover | marios, so the standalone deploy uses that hash instead of default | 15:25 |
| dpawlik | weshay|ruck, I will focus on the topic tomorrow | 15:25 |
| marios | rfolco|rover: e.g. that? https://docs.openstack.org/tripleo-docs/latest/ci/chasing_promotions.html#specifying-a-particular-hash | 15:26 |
| rfolco|rover | marios, yeah, thanks for the rtfm | 15:27 |
| weshay|ruck | dpawlik, bad timing | 15:27 |
| weshay|ruck | panda, 0b5ec707ab4c049281536e2bfbb1984c | 15:28 |
| panda | weshay|ruck: confirmed https://trunk.rdoproject.org/centos8-master/current-tripleo/delorean.repo.md5 | 15:31 |
| *** ykarel is now known as ykarel|away | 15:38 | |
| ysandeep | rlandy++ | 15:44 |
| *** ysandeep is now known as ysandeep|afk | 15:45 | |
| *** yolanda has quit IRC | 16:07 | |
| weshay|ruck | rlandy, still having stuck job issues in internal? | 16:11 |
| rlandy | weshay|ruck: no but please don't close that card | 16:11 |
| weshay|ruck | just noticed | 16:12 |
| weshay|ruck | 16 hr 12 min | 16:12 |
| weshay|ruck | periodic-tripleo-ci-centos-7-bm_envC-1ctlr_1comp-featureset001-train | 16:12 |
| weshay|ruck | periodic-tripleo-ci-centos-7-bm_envB-3ctlr_1comp-featureset001-stein | 16:12 |
| zbr | rfolco|rover: done with elastic-recheck? | 16:12 |
| rlandy | weshay|ruck: whoever cleaned up the tenant this morning must have left those jobs | 16:12 |
| weshay|ruck | k | 16:12 |
| *** jmasud has joined #oooq | 16:12 | |
| rfolco|rover | zbr, no, let me submit now and you can point me. I ran tox -e queries and it complains it should be string | 16:13 |
| * marios shutdown | 16:13 | |
| chandankumar | rlandy, weshay|ruck https://tree.taiga.io/project/morucci-software-factory/us/3578 | 16:15 |
| rfolco|rover | zbr, https://review.opendev.org/729319 | 16:15 |
| *** marios is now known as marios|out | 16:16 | |
| rfolco|rover | zbr, be easy, this might be obvious for you | 16:16 |
| rfolco|rover | zbr, It deletes some queries after running tox -e queries | 16:17 |
| rfolco|rover | zbr, ignore | 16:17 |
| rlandy | chandankumar: thanks | 16:17 |
| *** udesale_ has quit IRC | 16:24 | |
| *** marios|out has quit IRC | 16:25 | |
| *** jmasud has quit IRC | 16:26 | |
| *** jmasud has joined #oooq | 16:31 | |
| *** dpawlik has quit IRC | 16:37 | |
| *** ysandeep|afk is now known as ysandeep | 16:43 | |
| *** holser has quit IRC | 16:44 | |
| *** holser has joined #oooq | 16:46 | |
| weshay|ruck | zbr, you still around? | 16:48 |
| weshay|ruck | have you seen "2020-05-19 16:16:46.215288 | rdo-centos-8 | Error response from daemon: client version 1.40 is too new. Maximum supported API version is 1.39 | 16:48 |
| weshay|ruck | " | 16:48 |
| weshay|ruck | in ensure-docker | 16:48 |
| *** jmasud has quit IRC | 16:51 | |
| rlandy | ysandeep: fyi - registry-proxy is down - reldel confirmed the outage | 16:54 |
| rlandy | will try again later | 16:54 |
| ysandeep | rlandy, thanks! | 16:54 |
| *** udesale has joined #oooq | 16:55 | |
| *** derekh has quit IRC | 16:59 | |
| zbr | yes, i am still around but busy with infra, about elastic-recheck | 17:00 |
| *** udesale has quit IRC | 17:01 | |
| rlandy | weshay|ruck: think I found what was broken in container updates | 17:01 |
| rlandy | testing now | 17:01 |
| rlandy | containers may even be building now | 17:02 |
| rlandy | lunch - biab | 17:02 |
| *** pojadhav|afk is now known as pojadhav | 17:03 | |
| *** amoralej is now known as amoralej|off | 17:03 | |
| chandankumar | weshay|ruck, I am putting up config changes for job | 17:03 |
| *** jpena is now known as jpena|off | 17:04 | |
| weshay|ruck | k | 17:04 |
| zbr | rfolco|rover: basically, I seen the error, you will need to wait for me to ping you back. | 17:07 |
| rfolco|rover | zbr, ok np | 17:07 |
| weshay|ruck | rfolco|rover, something busted w/ elastic recheck? | 17:14 |
| rfolco|rover | weshay|ruck, this query does not work locally https://review.opendev.org/#/c/729319/1/queries/1879267.yaml | 17:15 |
| weshay|ruck | zbr, pretty sure I saw you had a patch on docker installs | 17:15 |
| rfolco|rover | weshay|ruck, must be my env, coz it works on zuul | 17:15 |
| zbr | yep, wip. took me most of the day | 17:15 |
| weshay|ruck | https://review.opendev.org/#/c/703053/ | 17:15 |
| weshay|ruck | zbr, ++ | 17:16 |
| weshay|ruck | zbr, does this describe the bug? https://bugs.launchpad.net/tripleo/+bug/1879534 | 17:16 |
| openstack | Launchpad bug 1879534 in tripleo "rdo / docker Error response from daemon: client version 1.40 is too new. Maximum supported API version is 1.39" [Critical,Triaged] - Assigned to Sorin Sbarnea (ssbarnea) | 17:16 |
| weshay|ruck | rfolco|rover, really? | 17:17 |
| rfolco|rover | weshay|ruck, I am unable to run local queries | 17:19 |
| weshay|ruck | rfolco|rover, try py27 | 17:21 |
| weshay|ruck | WFM | 17:22 |
| rfolco|rover | weshay|ruck, fixed, thx | 17:26 |
| rfolco|rover | zbr, fixed w/ py27 | 17:26 |
| weshay|ruck | chandankumar, still have something funky w/ introspection | 17:27 |
| *** jmasud has joined #oooq | 17:29 | |
| *** jmasud has quit IRC | 17:31 | |
| *** jmasud has joined #oooq | 17:32 | |
| rlandy | ok we have containers now again downstream | 17:42 |
| rlandy | weird ... | 17:45 |
| rlandy | 2020-05-19T17:28:04.527757975+00:00 stderr F ++ chmod 755 /var/log/kolla/collectd | 17:45 |
| rlandy | 2020-05-19T17:28:04.531815613+00:00 stderr F + echo 'Running command: '\''/usr/sbin/collectd -f'\''' | 17:45 |
| rlandy | 2020-05-19T17:28:04.531841809+00:00 stdout F Running command: '/usr/sbin/collectd -f' | 17:45 |
| rlandy | 2020-05-19T17:28:04.531847761+00:00 stderr F + exec /usr/sbin/collectd -f | 17:45 |
| rlandy | 2020-05-19T17:28:04.552468794+00:00 stderr F [2020-05-19 17:28:05] plugin_load: plugin "logfile" successfully loaded. | 17:45 |
| rlandy | 2020-05-19T17:28:04.698224866+00:00 stderr F Error: Parsing the config file failed! | 17:45 |
| rlandy | periodic-tripleo-ci-centos-7-bm_envB-3ctlr_1comp-featureset001-steinopenstack/tripleo-ci-internal-jobsmasterperiodicmaster172432020-05-19T10:47:27SUCCESS | 17:46 |
| rlandy | passed eventually | 17:46 |
| ysandeep | rlandy, o/ was brainstorming the rework http://paste.openstack.org/show/793782/ | 17:52 |
| chandankumar | weshay|ruck, did we fixed the nodeset? | 17:54 |
| weshay|ruck | chandankumar, yup | 17:56 |
| weshay|ruck | chandankumar, it's in tempest.. | 17:56 |
| weshay|ruck | chandankumar, so .. most everything is working.. ovb still busted on introspection | 17:56 |
| weshay|ruck | and ovn jobs still not right | 17:56 |
| weshay|ruck | but.. they may not be criteria | 17:56 |
| weshay|ruck | rlandy, ysandeep chandankumar we should fire up a bm on ussuri :) | 17:57 |
| weshay|ruck | since we're hitting introspection errors | 17:57 |
| weshay|ruck | still | 17:57 |
| rlandy | weshay|ruck: funny, not funny | 17:58 |
| chandankumar | weshay|ruck, ok | 17:58 |
| rlandy | ysandeep: latest run of scenario001 did better | 18:00 |
| rlandy | checking the container start failure | 18:00 |
| ysandeep | gnoochi | 18:02 |
| weshay|ruck | rlandy, when ever ur ready | 18:02 |
| weshay|ruck | oh wait | 18:02 |
| weshay|ruck | I have a mtg | 18:03 |
| rlandy | ok | 18:03 |
| ysandeep | rlandy, hmmm seems like some gnocchi failed to create connection with ceph(rados) | 18:06 |
| ysandeep | https://sf.hosted.upshift.rdu2.redhat.com/logs/82/199982/24/check/periodic-tripleo-ci-rhel-8-scenario001-standalone-cloudops-rhos-17/c5b4e03/logs/undercloud/var/log/containers/stdouts/gnocchi_db_sync.log | 18:06 |
| rlandy | may be a real problem ... no such container here ... https://sf.hosted.upshift.rdu2.redhat.com/logs/40/189440/40/check/periodic-tripleo-ci-rhel-8-standalone-network-rhos-17/d3e2b19/logs/undercloud/var/log/containers/ | 18:08 |
| ysandeep | rlandy, i think you are right.. no ceph container running on podman output - https://sf.hosted.upshift.rdu2.redhat.com/logs/40/189440/40/check/periodic-tripleo-ci-rhel-8-standalone-network-rhos-17/d3e2b19/logs/undercloud/var/log/extra/podman/podman_allinfo.log | 18:10 |
| rlandy | Upgrading: | 18:10 |
| rlandy | openstack-neutron noarch 1:16.1.0-0.20200518112003.ba05201.el8ost network 99 k | 18:10 |
| rlandy | openstack-neutron-common noarch 1:16.1.0-0.20200518112003.ba05201.el8ost network 260 k | 18:10 |
| rlandy | openstack-neutron-ml2 noarch 1:16.1.0-0.20200518112003.ba05201.el8ost network 89 k | 18:10 |
| rlandy | well at least container upgrades are fixed | 18:10 |
| rlandy | and containers are pushing | 18:10 |
| rlandy | ysandeep: let me rerun with ^^ | 18:11 |
| rlandy | and I can test the integration line test as well now that we have tripleo-ci-testing containers back | 18:11 |
| weshay|ruck | rlandy, is now a decent time or you slammed | 18:12 |
| rlandy | weshay|ruck: give 10 mins to restart this | 18:12 |
| ysandeep | rlandy, ack o/ I am leaving now.. too sleepy zzzz.. we can discuss rework tomorrow http://paste.openstack.org/show/793782/ | 18:12 |
| rlandy | then we can meet | 18:12 |
| weshay|ruck | k | 18:12 |
| rlandy | ysandeep: ack | 18:12 |
| rlandy | good night | 18:12 |
| ysandeep | rlandy, thank you for great help today! you are awesome! | 18:13 |
| rlandy | sure | 18:13 |
| *** ysandeep is now known as ysandeep|away | 18:13 | |
| zbr | weshay|ruck: managed to bring the ensure-docker fix to ready for upstream review, we should have results in~30min, and after this poking infra to merge it. | 18:20 |
| weshay|ruck | heh.. | 18:20 |
| weshay|ruck | zbr, k.. dang | 18:20 |
| zbr | i checked with clarkb and mordred, it should not be hard | 18:20 |
| weshay|ruck | ok | 18:20 |
| weshay|ruck | thanks man | 18:20 |
| rlandy | weshay|ruck: k - when you are ready | 18:22 |
| weshay|ruck | k.. now I need 5 lolz | 18:23 |
| weshay|ruck | rlandy, take a quick nap | 18:23 |
| chandankumar | weshay|ruck, rlandy https://review.rdoproject.org/r/#/q/topic:ussuriovb+(status:open+OR+status:merged) remaining patches for ussuri check | 18:30 |
| rlandy | ok | 18:31 |
| weshay|ruck | rlandy, ok.. when ever | 18:43 |
| rlandy | weshay|ruck: no time like the present | 18:43 |
| chandankumar | weshay|ruck, anything else left on ussuri side? | 18:43 |
| chandankumar | on jobs | 18:43 |
| weshay|ruck | chandankumar, introspection issues.. that's it | 18:44 |
| weshay|ruck | chandankumar, I think we have to wait for #rdo to tell us they are ready too | 18:44 |
| weshay|ruck | rlandy, https://meet.google.com/nuj-hhno-wis | 18:44 |
| chandankumar | weshay|ruck, ack | 18:44 |
| rlandy | chandankumar: looking at those reviews | 18:45 |
| chandankumar | rlandy, aye thanks! | 18:46 |
| *** chandankumar is now known as raukadah | 18:48 | |
| sshnaidm | why do we need docker to run linters | 18:52 |
| weshay|ruck | rlandy, https://sf.hosted.upshift.rdu2.redhat.com/zuul/api/tenant/tripleo-ci-internal/builds | 18:58 |
| *** jmasud has quit IRC | 19:03 | |
| *** jmasud has joined #oooq | 19:04 | |
| weshay|ruck | rlandy, https://sf.hosted.upshift.rdu2.redhat.com/zuul/api/tenant/tripleo-ci-internal/builds | 19:07 |
| weshay|ruck | rlandy, https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-ussuri/a06153b/logs/ | 19:30 |
| *** TrevorV has quit IRC | 20:01 | |
| rfolco|rover | weshay|ruck, train failed last round, https://logserver.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-train-containers-build-push/4ebd104/logs/buildah-builds/kolla-builds/c7996e7a-0771-4ef5-a1e7-b5f23e8cc2c0/docker/base/base-build.log | 20:05 |
| rfolco|rover | weshay|ruck, is this known issue ? Error: No matching repo to modify: epel. | 20:06 |
| weshay|ruck | rfolco|rover, you know that error :) | 20:06 |
| rfolco|rover | weshay|ruck, I don't | 20:07 |
| weshay|ruck | remember you hit that when working w/ containers? | 20:07 |
| weshay|ruck | look at kolla or tripleo common re: epel.. | 20:08 |
| rlandy | weshay|ruck: ok - let's see what damage we do here ... https://code.engineering.redhat.com/gerrit/200870 DNM: Testing baremetal ussuri job | 20:08 |
| weshay|ruck | :) | 20:08 |
| weshay|ruck | rfolco|rover, can't help atm.. I'm unfucking queens atm | 20:11 |
| weshay|ruck | give me about 20 min | 20:11 |
| rfolco|rover | weshay|ruck, k will do some research | 20:11 |
| rlandy | weshay|ruck: where are the equivalent rhel baseos images? | 20:47 |
| weshay|ruck | rlandy, had one on the rdo image server.. but it's not visible | 20:48 |
| weshay|ruck | rlandy, they are on the rhn portal if you need to download | 20:48 |
| rlandy | weshay|ruck: it's fine - just setting up for internal baremetal while ussuri runs | 20:49 |
| weshay|ruck | rlandy, https://access.redhat.com/downloads/content/479/ver=/rhel---8/8.2/x86_64/product-software | 20:49 |
| weshay|ruck | rlandy, download, upload to internal server.. | 20:49 |
| rlandy | ok | 20:50 |
| rlandy | weshay|ruck: the guest image, right? | 20:51 |
| weshay|ruck | aye | 20:52 |
| *** jmasud has quit IRC | 20:53 | |
| weshay|ruck | zbr++ | 20:55 |
| *** jmasud has joined #oooq | 20:59 | |
| *** Goneri has quit IRC | 21:03 | |
| *** ccamacho has quit IRC | 21:09 | |
| *** jmasud has quit IRC | 21:12 | |
| *** Goneri has joined #oooq | 21:20 | |
| weshay|ruck | rlandy, where is the image build log now for to_build: true | 21:30 |
| weshay|ruck | ? | 21:30 |
| rlandy | in ovb? | 21:30 |
| weshay|ruck | ya | 21:31 |
| rlandy | you should see a build log in undercloud/zuul/home | 21:31 |
| rlandy | you should see a build log in undercloud/home/zuul | 21:31 |
| rlandy | otherwise to_build is false | 21:31 |
| weshay|ruck | 2020-05-19 15:26:45.804693 | primary | TASK [check-to-build-or-not-to-build : Always build images in the periodic jobs] *** | 21:34 |
| weshay|ruck | 2020-05-19 15:26:45.804809 | primary | Tuesday 19 May 2020 15:26:45 +0000 (0:00:00.033) 0:04:01.379 *********** | 21:34 |
| weshay|ruck | 2020-05-19 15:26:45.832418 | primary | ok: [undercloud] | 21:34 |
| weshay|ruck | 2020-05-19 15:26:45.836739 | primary | | 21:34 |
| weshay|ruck | I see that | 21:34 |
| weshay|ruck | looks like we're downloading | 21:35 |
| weshay|ruck | rlandy, https://review.opendev.org/729404 | 21:41 |
| rlandy | weshay|ruck: we hit the nmcli network-scripts fun on rhel machine | 21:42 |
| rlandy | working through that again | 21:42 |
| weshay|ruck | rlandy, in the job you kicked or manually? | 21:42 |
| rlandy | manually to try get through it | 21:42 |
| weshay|ruck | rlandy, ah dang it.. error right in front of us | 21:44 |
| weshay|ruck | Error: Failed to set boot device to PXE: Failed to set boot device for node | 21:44 |
| weshay|ruck | https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-ussuri/d019289/logs/undercloud/var/log/extra/errors.txt.txt.gz | 21:44 |
| *** Goneri has quit IRC | 21:48 | |
| *** jfrancoa has quit IRC | 22:12 | |
| *** jmasud has joined #oooq | 22:20 | |
| *** jmasud has quit IRC | 22:24 | |
| *** holser has quit IRC | 22:31 | |
| *** holser has joined #oooq | 22:33 | |
| weshay|ruck | rlandy, starting to look like it's the bmc | 22:34 |
| weshay|ruck | rlandy, we may not need a bm repro | 22:35 |
| rlandy | weshay|ruck: we need to sort this out anyways | 22:35 |
| weshay|ruck | rlandy, hrm... bmc changed a lot | 22:43 |
| weshay|ruck | master = https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/e21345c/logs/bmc-console.log | 22:43 |
| rlandy | weshay|ruck: just recently? | 22:43 |
| weshay|ruck | https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-ussuri/78dfd0e/logs/bmc-console.log | 22:44 |
| weshay|ruck | ussuri ^ | 22:44 |
| weshay|ruck | no setup on the bmc | 22:44 |
| weshay|ruck | just straight into reporting power state? | 22:44 |
| *** jmasud has joined #oooq | 23:00 | |
| *** holser has quit IRC | 23:30 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!