| *** tosky has quit IRC | 00:09 | |
| *** sshnaidm is now known as sshnaidm|off | 00:12 | |
| *** jmasud has joined #oooq | 00:17 | |
| *** jmasud has quit IRC | 01:23 | |
| *** jmasud has joined #oooq | 01:26 | |
| *** jmasud has quit IRC | 01:38 | |
| *** jmasud has joined #oooq | 01:54 | |
| *** jmasud has quit IRC | 02:05 | |
| *** jmasud has joined #oooq | 03:02 | |
| *** apetrich has quit IRC | 03:09 | |
| *** jmasud has quit IRC | 03:17 | |
| *** skramaja has joined #oooq | 03:58 | |
| *** ykarel has joined #oooq | 04:28 | |
| *** jmasud has joined #oooq | 04:33 | |
| *** jmasud has quit IRC | 05:05 | |
| *** jmasud has joined #oooq | 05:20 | |
| *** ykarel has quit IRC | 05:23 | |
| *** ykarel has joined #oooq | 05:25 | |
| *** udesale has joined #oooq | 05:26 | |
| *** ysandeep|away is now known as ysandeep | 05:52 | |
| chkumar|ruck | ykarel: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/782362 this one | 05:59 |
|---|---|---|
| chkumar|ruck | to fix quay dependency | 05:59 |
| ykarel | chkumar|ruck, yes was looking that only, was trying to understand why not using images list | 06:00 |
| ykarel | and why it's failing with trunk.rdo registry | 06:00 |
| chkumar|ruck | ykarel: actually it contains the address of quay one from tripleo common | 06:01 |
| chkumar|ruck | which does not exists locally that's why it is failing | 06:01 |
| chkumar|ruck | we are re-constrcuting the non_tripleo_containers from failover | 06:02 |
| chkumar|ruck | pull | 06:02 |
| ykarel | why we not fixing that than | 06:02 |
| chkumar|ruck | each time then we need to do set facts | 06:04 |
| chkumar|ruck | ykarel: we can simly retrive the image list | 06:04 |
| chkumar|ruck | we have non_tripleo containers list , we get the container name and get their full url from image list and push it | 06:04 |
| ykarel | may be i am looking at different issue, re checking | 06:05 |
| chkumar|ruck | ykarel: https://ef9619c1056c463d9738-f79c70347d10b2f62e687e2ad6cae828.ssl.cf2.rackcdn.com/781815/1/check/tripleo-ci-centos-8-content-provider/9b5d9b2/job-output.txt | 06:05 |
| chkumar|ruck | ykarel: [container-build : Pull non-tripleo containers (ceph, alertmanager, prometheus) to the content provider registry] *** | 06:05 |
| chkumar|ruck | and above tasks | 06:05 |
| chkumar|ruck | better let me open a seperate bug and link it | 06:07 |
| ykarel | chkumar|ruck, i was looking at https://447f476af5555fa473a2-ba0bbef8fa5bd9d33ddbd8694210833c.ssl.cf5.rackcdn.com/781622/2/check/tripleo-ci-centos-8-content-provider/e902500/job-output.txt | 06:08 |
| ykarel | which is different | 06:08 |
| ykarel | but related to fallback only | 06:08 |
| ykarel | okk that image don't exist in trunk registry, is that handled manually and periodicallY? | 06:10 |
| chkumar|ruck | ykarel: image exists in rdo registry i think | 06:10 |
| ykarel | i mean trunk.registry.rdoproject.org/ceph/grafana:6.7.4 | 06:10 |
| chkumar|ruck | we pushed it there | 06:11 |
| ykarel | in master different tag is used | 06:11 |
| ykarel | so likely new images are not pushed yet | 06:11 |
| chkumar|ruck | on ho | 06:12 |
| ykarel | and will this be handling issues in ovb jobs too? | 06:12 |
| ykarel | likely no as the fix is for provider jobs and ovb jobs don't enabled provider jobs | 06:12 |
| chkumar|ruck | ykarel: for ovb I need to check whether is pulling from rdo registry or not | 06:13 |
| ykarel | it failed fetching from quay.io | 06:13 |
| ykarel | https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001 | 06:13 |
| *** jpodivin has quit IRC | 06:16 | |
| *** jpodivin has joined #oooq | 06:19 | |
| chkumar|ruck | ykarel: as a sort term solution, I am going to mirror grafana image on our quay namespace | 06:29 |
| ykarel | chkumar|ruck, the issue in quay.io is namespace related? | 06:29 |
| ykarel | i thought issue is across quay.io | 06:30 |
| chkumar|ruck | ykarel: they removed this image quay.io/app-sre/grafana:6.7.4 | 06:31 |
| ykarel | chkumar|ruck, ack and what's the reason for removal? | 06:33 |
| chkumar|ruck | ykarel: need to find out the app-sre team with in Red Hat | 06:35 |
| ykarel | chkumar|ruck, ack that would be good to know and handle such cases in future | 06:36 |
| chkumar|ruck | ykarel: https://hub.docker.com/r/grafana/grafana there is no 6.7.4 tag there | 06:38 |
| *** ratailor has joined #oooq | 06:47 | |
| ysandeep | folks o/ need reviews on https://review.rdoproject.org/r/c/rdo-jobs/+/32629 | 06:49 |
| *** amoralej|off is now known as amoralej | 07:01 | |
| chkumar|ruck | arxcruz: hello when around, ping me back | 07:05 |
| *** slaweq_ has joined #oooq | 07:15 | |
| *** marios has joined #oooq | 07:25 | |
| chkumar|ruck | marios: Good morning, content provider for all releases is busted as grafana image is removed from quay: https://launchpad.net/bugs/1920873 series of patches: https://review.opendev.org/q/e0b0d859efb179c6868020b3fc10c2155ed96820 | 07:32 |
| openstack | Launchpad bug 1920873 in tripleo "quay.io/app-sre/grafana:6.7.4 went missing from quay leading to fail all content provider jobs" [Critical,Triaged] | 07:32 |
| *** jmasud has quit IRC | 07:32 | |
| marios | ack chkumar|ruck | 07:34 |
| marios | chkumar|ruck: tests need update? | 07:36 |
| chkumar|ruck | marios: please have a look at patchset 2 | 07:38 |
| *** jmasud has joined #oooq | 07:41 | |
| marios | chkumar|ruck: oh i see you already updated it and we are waiting for zuul to report for v2 thanks https://zuul.openstack.org/status#782366 | 07:43 |
| *** ysandeep is now known as ysandeep|lunch | 07:46 | |
| *** slaweq_ is now known as slaweq | 07:53 | |
| *** apetrich has joined #oooq | 08:14 | |
| *** jmasud has quit IRC | 08:25 | |
| chkumar|ruck | marios: against ussuri and victoria tripleo common patches, we are still run train and ussuri content provider due to upgrade jobs, Do we want to tighten that up? or remove the content old release provider job there | 08:33 |
| chkumar|ruck | grafana patch for stable release is not going land | 08:33 |
| *** sshnaidm|off has quit IRC | 08:34 | |
| marios | chkumar|ruck: we shouldn't be let will check what's going on there thanks | 08:36 |
| marios | chkumar|ruck: why 10:33 < chkumar|ruck> grafana patch for stable release is not going land | 08:36 |
| marios | chkumar|ruck: in victoria patch i only see ussuri content provider? https://zuul.openstack.org/status#782367 | 08:37 |
| marios | chkumar|ruck: /me wipes eyes | 08:37 |
| marios | am i missing it? | 08:37 |
| marios | chkumar|ruck: and in ussuri we have train content provider https://zuul.openstack.org/status#782127 | 08:38 |
| * marios widens eyes stares at the screen | 08:38 | |
| zbr|rover | chkumar|ruck: marios: low hanging https://review.opendev.org/c/openstack/tripleo-heat-templates/+/780854 | 08:40 |
| marios | zbr|rover: what kind of fruit is it i prefer passion fruit in the morning | 08:40 |
| zbr|rover | durian | 08:40 |
| marios | i am guessing without the google search that's the stinky one right? | 08:41 |
| zbr|rover | yeah | 08:41 |
| marios | thanks | 08:41 |
| marios | :( | 08:41 |
| zbr|rover | and believe me, it stinks. In Asia you can see signs in hotels and other areas with "Durian not allowed", near the no-smoke ones. | 08:42 |
| zbr|rover | but it is also tasty, if you can pass the stinkness... | 08:42 |
| zbr|rover | what really happened over the night with quay, i seen emails, how can an image vanish? | 08:43 |
| chkumar|ruck | marios: yes, in victoria patch there is victoria + ussuri content provider and in ussuri patch it has ussuri + train content provider | 08:48 |
| marios | chkumar|ruck: so what is the problem | 08:48 |
| marios | chkumar|ruck: this is fine. before we had all the content providers in each patch | 08:48 |
| chkumar|ruck | in victoria patch, ussuri content provider is going to fails due to grafana | 08:48 |
| marios | chkumar|ruck: these ones are the required ones. we deploy n-1 and upgrde to n | 08:48 |
| marios | chkumar|ruck: 10:33 < chkumar|ruck> marios: against ussuri and victoria tripleo common patches, we are still run train and ussuri content provider due to upgrade jobs, Do we want to tighten that up? or remove the content old release provider job there | 08:49 |
| marios | chkumar|ruck: referring to that? ^ what can we tighten up it is ok | 08:49 |
| chkumar|ruck | marios: ok, but my question is how to land it https://review.opendev.org/c/openstack/tripleo-common/+/782367/ when ussuri content will fail | 08:50 |
| marios | chkumar|ruck: so we need that back to train | 08:51 |
| marios | chkumar|ruck: i think we can merge the train one because there is no upgrade job there only update | 08:51 |
| marios | chkumar|ruck: then use that to merge ussuri | 08:51 |
| marios | chkumar|ruck: then victoria | 08:51 |
| marios | chkumar|ruck: maybe ... thinking | 08:51 |
| chkumar|ruck | marios: ok that strategy will work | 08:51 |
| chkumar|ruck | marios: let me propose train one | 08:51 |
| marios | chkumar|ruck: ack | 08:51 |
| zbr|rover | marios: regarding the lack of details in reviews, i already got some remarks from infra regarding use of internal issue tracking... so I stopped putting links. | 08:53 |
| zbr|rover | weshay|ruck: add the goal to prepare tripleo repos for py39, this means do whatever it takes to run at least one tox job using py39 on each of those that have python code. | 08:54 |
| marios | zbr|rover: "please add more than 7 words " != "please add link to issue tracker" :) | 08:54 |
| zbr|rover | yeah, i am cheap on words :D | 08:55 |
| marios | zbr|rover: wrt to issue trackers yes i always call that out we cna't do that until it becomes available publicly | 08:55 |
| zbr|rover | there is one aspect we need to discuss, sometimes we need to combine unrelated changes, like https://review.opendev.org/c/openstack/tripleo-heat-templates/+/780854/1/setup.cfg | 08:56 |
| zbr|rover | because our testing pipelines are so slow, it would be impossible to update all repos in purely "atomic" changes (not enough people and hw resources). | 08:57 |
| *** ysandeep|lunch is now known as ysandeep | 08:57 | |
| *** jpenag is now known as jpena | 08:57 | |
| *** ykarel is now known as ykarel|lunch | 08:57 | |
| zbr|rover | so I hope that the team will be ok to bundle *few* low-risk changes, like fixing the warning about using underline in setup.cfg | 08:58 |
| zbr|rover | setuptools deprecated dashes and announced they will be removed in the future, now iw warning, but at any moment it may become an error. | 08:58 |
| marios | zbr|rover: if only there were some space provided in the review where you could add such context ! | 08:59 |
| marios | zbr|rover: we should put in a feature request | 08:59 |
| zbr|rover | point taken, i will update the comment | 08:59 |
| chkumar|ruck | marios: https://review.opendev.org/c/openstack/tripleo-common/+/782379 train one | 09:00 |
| zbr|rover | it would be so cool to be able to edit message without having to rebuild | 09:00 |
| marios | chkumar|ruck: ack thanks | 09:02 |
| marios | zbr|rover: thanks :) ... well it would have to detect that there is no depends-on i guess to allow no rebuild | 09:02 |
| *** derekh has joined #oooq | 09:02 | |
| *** tosky has joined #oooq | 09:04 | |
| *** saneax has quit IRC | 09:04 | |
| zbr|rover | marios: regarding openstack-python3-xena-jobs -- I do not really know. Bad part is that xena has py39 only in check as non-voting, clearly less that what we want. My impression was that we want voting. | 09:08 |
| zbr|rover | on the other hand I know for sure that we do *NOT* want py35 | 09:09 |
| zbr|rover | It would make sense to replace openstack-python3-victoria-jobs with openstack-python3-xena-jobs | 09:11 |
| marios | zbr|rover: ack ok i wonder if they will update the -wallaby one to include py39 anyway | 09:11 |
| zbr|rover | i will try to switch to xena as you asked, it seems as a resonable approach, with potential of saving on maintenance in longer term | 09:13 |
| *** holser has joined #oooq | 09:14 | |
| chkumar|ruck | kopecmartin: currently content provider jobs are busted | 09:16 |
| zbr|rover | marios: does the new version of patch looks ok? i see it did start the expected list of jobs. | 09:17 |
| marios | zbr|rover: ack will check but not immediately thanks for update | 09:19 |
| *** dtantsur has joined #oooq | 09:42 | |
| *** jmasud has joined #oooq | 09:49 | |
| ysandeep | chkumar|ruck, ykarel hey o/ how to create https://images.rdoproject.org/centos8/wallaby | 09:50 |
| *** jmasud has quit IRC | 09:51 | |
| *** saneax has joined #oooq | 09:57 | |
| *** ykarel|lunch is now known as ykarel | 10:01 | |
| *** sanjayu_ has joined #oooq | 10:03 | |
| *** saneax has quit IRC | 10:05 | |
| ykarel | ysandeep, iirc that used to be created manually, jpena ^? | 10:10 |
| jpena | hm I'm not sure. I can create it right away | 10:10 |
| jpena | done | 10:12 |
| *** sshnaidm has joined #oooq | 10:13 | |
| ysandeep | ykarel++ jpena++ thanks | 10:13 |
| *** sshnaidm is now known as sshnaidm|off | 10:13 | |
| *** tosky has quit IRC | 10:13 | |
| *** tosky has joined #oooq | 10:14 | |
| ykarel | i checked the upload script and it seems it expects directory to be present | 10:15 |
| ykarel | config/ci-scripts/tripleo-upstream/upload-cloud-images.sh one | 10:15 |
| ysandeep | ykarel, after directory creation, what is calling ^^ above script to upload the images.. is it in image build job definitions? | 10:20 |
| ykarel | yes | 10:20 |
| ysandeep | ykarel, ack.. i had a green run on image build jobs.. i will trigger them | 10:21 |
| ysandeep | periodic-tripleo-centos-8-buildimage-ironic-python-agent-wallaby https://review.rdoproject.org/zuul/build/e4852e0bd4b24128b258a7816fab6845 : SUCCESS in 23m 42s | 10:21 |
| ykarel | okk rerun, rsync: mkdir "/var/www/html/images/centos8/wallaby/rdo_trunk" failed: No such file or directory (2) | 10:22 |
| ysandeep | ack o/ | 10:27 |
| ysandeep | chkumar|ruck, I am hitting this failure in wallaby, Have you noticed similiar issue in master/any other branch recently? | 11:21 |
| ysandeep | https://logserver.rdoproject.org/58/28458/56/check/periodic-tripleo-ci-centos-8-standalone-wallaby/187f01b/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 11:21 |
| ysandeep | error={"ansible_loop_var": "item", "changed": true, "cmd": "podman exec -u root swift_object_expirer /usr/local/bin/kolla_set_configs", "delta": "0:00:00.082147", "end": "2021-03-23 09:43:21.439881", "failed_when_result": true, "item": "swift_object_expirer", "msg": "non-zero return code", "rc": 126, "start": "2021-03-23 09:43:21.357734", "stderr": "Error: cannot exec into container that is not running: container state improper", | 11:22 |
| ysandeep | "stderr_lines": ["Error: cannot exec into container that is not running: container state improper"], "stdout": "", "stdout_lines": []} | 11:22 |
| ysandeep | hmm, /usr/local/bin/kolla_start: line 18: /usr/bin/swift-object-expirer: No such file or directory | 11:29 |
| ysandeep | https://logserver.rdoproject.org/58/28458/57/check/periodic-tripleo-ci-centos-8-standalone-wallaby/986ecac/logs/undercloud/var/log/extra/podman/containers/swift_object_expirer/stdout.log.txt.gz ^^ | 11:29 |
| chkumar|ruck | ysandeep: check the dist git of swift | 11:29 |
| * ysandeep looking | 11:30 | |
| chkumar|ruck | ysandeep: https://github.com/rdo-packages/swift-distgit/blob/rpm-master/openstack-swift.spec | 11:31 |
| chkumar|ruck | man dir exist but binary is missing there | 11:31 |
| chkumar|ruck | ysandeep: https://github.com/rdo-packages/swift-distgit/commit/0014419a4958e39c652a74d6cff8f46464bce8ad | 11:32 |
| chkumar|ruck | ysandeep: can you check swift componnt promoted or not | 11:33 |
| ysandeep | chkumar|ruck, https://trunk.rdoproject.org/centos8-wallaby/component/swift/ tripleo-ci-testing and current/consistent is same | 11:34 |
| *** rlandy has joined #oooq | 11:35 | |
| chkumar|ruck | ysandeep: binary is also there https://github.com/rdo-packages/swift-distgit/blob/rpm-master/openstack-swift.spec#L554 | 11:40 |
| chkumar|ruck | can you pull the package version which is getting installed or check the swift component repo used there | 11:40 |
| chkumar|ruck | ? | 11:40 |
| ysandeep | chkumar|ruck, ack | 11:41 |
| rlandy | chkumar|ruck: took a shot at the internal providers - kind of need a multiple inheritance situation | 12:11 |
| rlandy | need to rework the parenting to avoid copying the playbooks | 12:11 |
| chkumar|ruck | marios: rlandy: weshay|ruck please donot approve any upstream patches till it lands https://review.opendev.org/c/openstack/tripleo-common/+/782366 | 12:12 |
| chkumar|ruck | rlandy: sure, I didnot get a chance to go over that patch quay ruined my day | 12:13 |
| rlandy | chkumar|ruck: no worries- I'll ping you when I have another solution | 12:18 |
| chkumar|ruck | rlandy: marios arxcruz I have moved tripleo-repos to tomorrow as sshnaidm|off is having public holiday there | 12:21 |
| rlandy | chkumar|ruck: ack - elections there | 12:21 |
| chkumar|ruck | akahat: rlandy arxcruz regarding promoter, we can meet early just after community call if free. | 12:21 |
| rlandy | chkumar|ruck: ok by me | 12:22 |
| arxcruz | ok | 12:22 |
| akahat | chkumar|ruck, np. | 12:22 |
| rlandy | ysandeep: need reviews on any wallaby patches? | 12:26 |
| ysandeep | rlandy, https://review.rdoproject.org/r/c/rdo-jobs/+/32629 | 12:26 |
| rlandy | ysandeep: ok to merge ^^? | 12:28 |
| ysandeep | rlandy, yes | 12:28 |
| marios | ack chkumar|ruck | 12:29 |
| ysandeep | bhagyashris, rlandy, marios wallaby branching mtg | 12:30 |
| marios | ysandeep: joining | 12:31 |
| pojadhav | ysandeep, bhagyashris is on sick leave | 12:31 |
| *** jpena is now known as jpena|lunch | 12:31 | |
| *** amoralej is now known as amoralej|lunch | 12:32 | |
| weshay|ruck | in mtgs | 12:33 |
| *** ykarel has quit IRC | 12:38 | |
| *** ykarel has joined #oooq | 12:38 | |
| chkumar|ruck | zbr|rover: please have a look at this review https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/782362 | 12:41 |
| marios | sshnaidm|off: weshay|ruck: rlandy: *** when you next have time please grateful for eyes/opinions @ https://review.opendev.org/c/openstack/tripleo-ci/+/779992 ( context https://review.opendev.org/c/openstack/tripleo-ci/+/779992/1/zuul.d/multinode-jobs.yaml#125 ) thank you | 12:43 |
| *** ykarel_ has joined #oooq | 12:44 | |
| *** ykarel has quit IRC | 12:47 | |
| rlandy | marios: ysandeep: frenzy_friday: pojadhav: akahat: chkumar|ruck: weshay|ruck: arxcruz: zbr|rover: community call in ~30 mins ... pls add agenda items .. https://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg#2021-03-23-Community-call | 12:52 |
| marios | thanks rlandy | 12:53 |
| ysandeep | marios frenzy_friday fyi.., rlandy -your guess was right.. that error was hidden in master component line(swift) | 13:06 |
| ysandeep | chkumar|ruck, its hitting in master as well.. i think worth to write a bug for master https://logserver.rdoproject.org/openstack-component-swift/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-swift-master/08abce7/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 13:07 |
| ysandeep | ^^ swift component line | 13:07 |
| chkumar|ruck | ysandeep: on it | 13:08 |
| rlandy | ysandeep: at least there is an explanation :) | 13:09 |
| rlandy | ysandeep: seen this error before in downstream: http://pastebin.test.redhat.com/949709 | 13:15 |
| ysandeep | rlandy, nope | 13:16 |
| marios | ack ysandeep | 13:16 |
| rlandy | ysandeep: no worries - seems to be working now .. latest run ok | 13:17 |
| *** ykarel_ is now known as ykarel | 13:17 | |
| ysandeep | rlandy, fyi.. patch to switch.. periodic verion of check jobs to non-periodic version is almost complete.. its blocked because of 17 deps issue.. container build failing there | 13:18 |
| chkumar|ruck | ysandeep: https://bugs.launchpad.net/tripleo/+bug/1920924 | 13:19 |
| openstack | Launchpad bug 1920924 in tripleo "/usr/bin/swift-object-expirer: No such file or directory while deploying standalone " [High,Triaged] | 13:19 |
| rlandy | ysandeep: k - have something of a duplicate patch to get provider jobs in | 13:19 |
| ysandeep | chkumar|ruck, ack o/ | 13:19 |
| rlandy | https://code.engineering.redhat.com/gerrit/#/c/232088/2/zuul.d/standalone-jobs.yaml | 13:19 |
| rlandy | we can sync up | 13:19 |
| weshay|ruck | chkumar|ruck, zbr|rover do you folks have a handle on the c8-train failures? Looks like 1 or 2 tempest tests are holding us back | 13:20 |
| *** ykarel_ has joined #oooq | 13:22 | |
| ysandeep | rlandy, ack.. mine was .. https://code.engineering.redhat.com/gerrit/#/c/231711/ | 13:22 |
| rlandy | ysandeep: k - np - can rebase | 13:22 |
| ysandeep | rlandy, you go ahead with your patch.. i will just create check version of job for image build.. i will remove container build job | 13:23 |
| ysandeep | from my patch | 13:23 |
| chkumar|ruck | weshay|ruck: links, please add the tasks, will look into that | 13:24 |
| weshay|ruck | k | 13:24 |
| *** ykarel has quit IRC | 13:24 | |
| *** ykarel_ has quit IRC | 13:26 | |
| chkumar|ruck | ysandeep: where we look the swift rpm version in the standalone job? | 13:26 |
| ysandeep | chkumar|ruck, for wallaby, I looked in container build job not standalone | 13:28 |
| chkumar|ruck | ysandeep: ok | 13:28 |
| ysandeep | in component line i think we are updating containers.. but not capturing rpm package info from inside containers | 13:29 |
| ysandeep | weshay|ruck, our 1:1 conflicting with tripleo-ci community call | 13:30 |
| zbr|rover | what was the podman replacement for docker-compose? | 13:30 |
| *** amoralej|lunch is now known as amoralej | 13:33 | |
| *** jpena|lunch is now known as jpena | 13:34 | |
| *** ratailor has quit IRC | 13:35 | |
| zbr|rover | neverming, found, and sorted: https://github.com/containers/podman-compose/issues/151 | 13:39 |
| zbr|rover | as long the project does not have a release on pypi, i would not bother. | 13:39 |
| zbr|rover | i tried the devel version and discovered that it does not support remote connections. | 13:40 |
| weshay|ruck | soniya29, are you hitting a particular error? | 13:40 |
| soniya29 | weshay|ruck, yeah | 13:40 |
| weshay|ruck | soniya29, can you please pastebin | 13:41 |
| soniya29 | weshay|ruck, i want to spwan a local vm but each time i try installation fails | 13:41 |
| soniya29 | weshay|ruck, i dont think we can copy within a vm | 13:41 |
| weshay|ruck | soniya29, k.. that's odd.. you actually don't need to install the os | 13:42 |
| weshay|ruck | if you import a qcow2 file into your kvm images path | 13:42 |
| soniya29 | weshay|ruck, i tried with qcow2 as well | 13:42 |
| soniya29 | weshay|ruck, no luck | 13:43 |
| chkumar|ruck | weshay|ruck: do you know where collects the logs of package update with in containers in standalone done via container prep parameter? | 13:44 |
| weshay|ruck | soniya29, on your laptop ya? | 13:44 |
| chkumar|ruck | for example this one https://logserver.rdoproject.org/openstack-component-swift/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-swift-master/08abce7/logs/undercloud/home/zuul/ | 13:44 |
| soniya29 | weshay|ruck, yeah | 13:45 |
| weshay|ruck | soniya29, same issue on your test box? rdo-ci-fx2-03-s2-drac.mgmt.rdoci.lab.eng.rdu2.redhat.com ? | 13:45 |
| weshay|ruck | chkumar|ruck, so the update is no longer done in child jobs | 13:46 |
| weshay|ruck | chkumar|ruck, we build the containers initially w/ the change | 13:46 |
| weshay|ruck | chkumar|ruck, so the containers are NOT updated if they are using a content provider | 13:46 |
| chkumar|ruck | weshay|ruck: the job is running in component pipelone | 13:46 |
| weshay|ruck | ah.. | 13:47 |
| weshay|ruck | k | 13:47 |
| weshay|ruck | so.. it comes from | 13:47 |
| weshay|ruck | modify_role: tripleo-modify-image | 13:47 |
| weshay|ruck | modify_vars: | 13:47 |
| weshay|ruck | tasks_from: yum_update.yml | 13:47 |
| weshay|ruck | update_repo: 'delorean-current,swift ' | 13:47 |
| weshay|ruck | yum_repos_dir_path: /etc/yum.repos.d | 13:47 |
| chkumar|ruck | weshay|ruck: https://logserver.rdoproject.org/openstack-component-swift/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-swift-master/08abce7/logs/undercloud/home/zuul/containers-prepare-parameters.yaml.txt.gz | 13:47 |
| * weshay|ruck looks for the log | 13:47 | |
| weshay|ruck | chkumar|ruck, https://logserver.rdoproject.org/openstack-component-swift/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-swift-master/08abce7/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz | 13:49 |
| weshay|ruck | look for yum_update.yml | 13:49 |
| weshay|ruck | chkumar|ruck, you should find one yum transaction in that log | 13:50 |
| weshay|ruck | chkumar|ruck, this is a concern | 13:51 |
| weshay|ruck | update_repo: 'delorean-current,swift ' | 13:51 |
| weshay|ruck | it should have one more repo | 13:51 |
| weshay|ruck | chkumar|ruck, right.. we should see https://logserver.rdoproject.org/openstack-component-swift/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-swift-master/08abce7/logs/undercloud/etc/yum.repos.d/swift-component.repo.txt.gz | 13:52 |
| chkumar|ruck | weshay|ruck: I went through that log | 13:52 |
| weshay|ruck | no transactions | 13:52 |
| chkumar|ruck | weshay|ruck: I am looking for specific container logs | 13:52 |
| weshay|ruck | the component update repo is not listed in the yaml | 13:53 |
| weshay|ruck | rlandy, that's how it should work right? | 13:53 |
| * rlandy reads back | 13:53 | |
| rlandy | it should be added in all component jobs | 13:54 |
| weshay|ruck | yes | 13:54 |
| weshay|ruck | but I don't see it | 13:54 |
| rlandy | weshay|ruck: ^^ the component repo should be in the repo update | 13:54 |
| * rlandy looks | 13:54 | |
| weshay|ruck | rlandy, sorry | 13:55 |
| weshay|ruck | it's called swift | 13:55 |
| rlandy | definitely a problem | 13:55 |
| weshay|ruck | it's there | 13:55 |
| weshay|ruck | false alarm | 13:55 |
| weshay|ruck | I can't read | 13:55 |
| rlandy | breathe, breathe | 13:56 |
| weshay|ruck | rlandy, :) | 13:56 |
| weshay|ruck | sorry sorry | 13:56 |
| rlandy | weshay|ruck: np :) | 13:56 |
| weshay|ruck | chkumar|ruck, https://logserver.rdoproject.org/openstack-component-swift/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-swift-train/0138736/logs/undercloud/etc/yum.repos.d/swift-component.repo.txt.gz | 13:56 |
| weshay|ruck | [swift] | 13:56 |
| weshay|ruck | chkumar|ruck, so now we have to check what's in there | 13:56 |
| *** sanjayu_ has quit IRC | 13:58 | |
| weshay|ruck | chkumar|ruck, afaict.. | 13:59 |
| weshay|ruck | https://trunk.rdoproject.org/centos8-master/component/swift/component-ci-testing/ | 13:59 |
| weshay|ruck | https://trunk.rdoproject.org/centos8-master/component/swift/current-tripleo/ | 13:59 |
| weshay|ruck | we should see an update | 13:59 |
| weshay|ruck | chkumar|ruck, looking at the container logs | 14:00 |
| weshay|ruck | https://logserver.rdoproject.org/openstack-component-swift/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-swift-master/08abce7/logs/undercloud/var/log/extra/podman/containers/swift_container_server/podman_info.log.txt.gz | 14:00 |
| weshay|ruck | openstack-swift-container.noarch 2.27.1-0.20210322142351.7e27829.el8 @swift | 14:00 |
| weshay|ruck | chkumar|ruck, the swift containers were updated | 14:00 |
| weshay|ruck | chkumar|ruck, so perhaps the containers-update playbook in tripleo-common just does a shitty job logging | 14:01 |
| weshay|ruck | chkumar|ruck, see that? | 14:01 |
| chkumar|ruck | yes | 14:02 |
| chkumar|ruck | arxcruz: akahat rlandy want to start promoter meeting now? | 14:03 |
| chkumar|ruck | weshay|ruck: thanks! | 14:04 |
| akahat | chkumar|ruck, i'm okay | 14:04 |
| rlandy | ok | 14:04 |
| chkumar|ruck | akahat: rlandy arxcruz weshay|ruck https://meet.google.com/pqq-cxwo-wym?authuser=0 promoter meeting | 14:05 |
| chkumar|ruck | akahat: arxcruz joining? | 14:06 |
| chkumar|ruck | arxcruz: we are waiting for you | 14:08 |
| *** ysandeep is now known as ysandeep|dinner | 14:20 | |
| *** gchamoul has quit IRC | 14:32 | |
| *** gchamoul has joined #oooq | 14:44 | |
| *** jlarriba has joined #oooq | 15:02 | |
| jlarriba | weshay|ruck: tripleo-quickstart is failing on CI due to an authentication error on quay.io, have you seen it? | 15:04 |
| jlarriba | akahat: ^ | 15:06 |
| akahat | jlarriba, o/ can you please point to some logs? | 15:07 |
| akahat | chkumar|ruck, zbr|rover ^^ | 15:07 |
| weshay|ruck | jlarriba, we've had a number of issues w/ quay.. I emailed the list and there are some launchpad bugs on it | 15:08 |
| weshay|ruck | jlarriba, can you paste the log? | 15:09 |
| jlarriba | akahat, the error is Container image prepare | undercloud | error={"changed": false, "error": "401 Client Error: UNAUTHORIZED for url: https://quay.io/v2/app-sre/grafana/manifests/6.7.4", "msg": "Error running container image prepare: 401 Client Error: UNAUTHORIZED for url: https://quay.io/v2/app-sre/grafana/manifests/6.7.4", "params": {}, "success": false} | 15:09 |
| jlarriba | weshay: ^ | 15:09 |
| weshay|ruck | jlarriba, paste the log url please | 15:09 |
| jlarriba | yeah, one sec | 15:10 |
| chkumar|ruck | jlarriba: https://bugs.launchpad.net/tripleo/+bug/1920873 | 15:15 |
| openstack | Launchpad bug 1920873 in tripleo "quay.io/app-sre/grafana:6.7.4 went missing from quay leading to fail all content provider jobs" [Critical,Triaged] | 15:15 |
| jlarriba | chkumar|ruck, thanks, that is the problem | 15:16 |
| jlarriba | weshay|ruck: https://zuul.opendev.org/t/openstack/build/2cca4dbc36c94da0a16fb98b12d8fbfd | 15:16 |
| jlarriba | akahat: ^ | 15:17 |
| jlarriba | and the more detailed errors are in https://17c296bb94b1fb2ecf41-ea9f828f8eee3ae0801a665a4594e601.ssl.cf1.rackcdn.com/772893/7/check/tripleo-ci-centos-8-content-provider/2cca4db/job-output.txt | 15:17 |
| weshay|ruck | jlarriba, thanks | 15:18 |
| *** ysandeep|dinner is now known as ysandeep|away | 15:21 | |
| zbr|rover | i am back, was in e-r meeting with frenzy_friday | 15:27 |
| *** jmasud has joined #oooq | 15:35 | |
| *** owalsh has quit IRC | 15:35 | |
| *** owalsh has joined #oooq | 15:36 | |
| *** apetrich has quit IRC | 15:36 | |
| *** marios has quit IRC | 15:37 | |
| *** irclogbot_0 has quit IRC | 15:37 | |
| *** marios has joined #oooq | 15:37 | |
| *** irclogbot_1 has joined #oooq | 15:38 | |
| *** irclogbot_1 has quit IRC | 15:49 | |
| *** irclogbot_0 has joined #oooq | 15:52 | |
| *** amoralej is now known as amoralej|off | 16:04 | |
| *** skramaja has quit IRC | 16:08 | |
| *** jmasud has quit IRC | 16:49 | |
| rlandy | weshay|ruck: chkumar|ruck: ok if I try reboot the dns server in vexxhost? | 16:50 |
| rlandy | no response on 53 according to ade | 16:50 |
| weshay|ruck | rlandy, sure | 16:50 |
| weshay|ruck | rlandy, try and get on the console first | 16:50 |
| weshay|ruck | then reboot | 16:50 |
| rlandy | weshay|ruck: good idea - seeing if I can | 16:50 |
| weshay|ruck | or just try to restart the service as well | 16:50 |
| *** marios is now known as marios|out | 16:51 | |
| chkumar|ruck | rlandy: ack | 16:52 |
| rlandy | ha - on the node | 16:53 |
| *** marios|out has quit IRC | 16:56 | |
| rlandy | Active: active (running) since Thu 2021-02-11 20:21:59 UTC; 1 months 9 days ago | 16:58 |
| rlandy | service is running | 16:58 |
| rlandy | will try restart | 16:58 |
| chkumar|ruck | weshay|ruck: please quay.io bug as a invalid | 17:00 |
| *** jmasud has joined #oooq | 17:01 | |
| rlandy | ; <<>> DiG 9.11.20-RedHat-9.11.20-5.el8_3.1 <<>> @38.102.83.187 docker.io | 17:01 |
| rlandy | ; (1 server found) | 17:01 |
| rlandy | ;; global options: +cmd | 17:01 |
| rlandy | ;; connection timed out; no servers could be reached | 17:01 |
| rlandy | weshay|ruck: chkumar|ruck: ^^ no luck on restarting service - trying node reboot now | 17:02 |
| rlandy | hmmm ... need to edit the /etc/resolve.conf on that box - pinging infra | 17:09 |
| *** jmasud has quit IRC | 17:20 | |
| *** jpodivin has quit IRC | 17:26 | |
| *** udesale has quit IRC | 17:30 | |
| *** derekh has quit IRC | 18:03 | |
| *** jpena is now known as jpena|off | 18:05 | |
| rlandy | weshay|ruck: can I tear two the two candudate instances on vexx | 18:10 |
| *** frenzy_friday is now known as frenzyfriday|bbl | 18:15 | |
| weshay|ruck | rlandy, ya | 18:39 |
| *** dtantsur is now known as dtantsur|afk | 18:55 | |
| *** dtantsur|afk is now known as dtantsur|afk|afk | 18:55 | |
| *** dtantsur|afk|afk is now known as dtantsur|afk | 18:55 | |
| rlandy | weshay|ruck: hey - ovb test with newer centos 8 stream image ... | 19:09 |
| rlandy | https://logserver.rdoproject.org/53/18953/43/check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/079e4c0/logs/undercloud/home/zuul/build.log.txt.gz | 19:09 |
| rlandy | image built | 19:09 |
| rlandy | deploy failure | 19:09 |
| rlandy | but that may be a master issue | 19:09 |
| * weshay|ruck looks | 19:09 | |
| rlandy | FATAL | Container image prepare | undercloud | error={"changed": false, "error": "401 Client Error: UNAUTHORIZED for url: https://quay.io/v2/app-sre/grafana/manifests/6.7.4", "msg": "Error running container image prepare: 401 Client Error: UNAUTHORIZED for url: https://quay.io/v2/app-sre/grafana/manifests/6.7.4", "params": {}, "su | 19:10 |
| rlandy | https://logserver.rdoproject.org/53/18953/43/check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/079e4c0/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 19:10 |
| weshay|ruck | rlandy, fs001 is back to green https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset001&job_name=tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001 | 19:10 |
| weshay|ruck | oh.. | 19:11 |
| weshay|ruck | 2021-03-23 00:03:23 | 2021-03-23 00:03:23.603981 | fa163eef-5567-e0e9-27e9-000000003242 | FATAL | Container image prepare | undercloud | error={"changed": false, "error": "401 Client Error: UNAUTHORIZED for url: https://quay.io/v2/app-sre/grafana/manifests/6.7.4", "msg": "Error running container image prepare: 401 Client Error: UNAUTHORIZED for url: https://quay.io/v2/app-sre/grafana/manifests/6.7.4", "params": {}, "success": false} | 19:11 |
| weshay|ruck | ya.. | 19:11 |
| weshay|ruck | sorry.. looked at the wrong log | 19:11 |
| weshay|ruck | rlandy, kick it again.. should work | 19:11 |
| weshay|ruck | or at least get past that | 19:11 |
| rlandy | weshay|ruck: ack - ack rekicked | 19:14 |
| *** frenzyfriday|bbl is now known as frenzy_friday | 19:46 | |
| *** jmasud has joined #oooq | 20:17 | |
| rlandy | weshay|ruck: do you remember some talk about multiple inheritance in zuul? | 20:56 |
| rlandy | maybe I'm dreaming | 20:56 |
| rlandy | provider jobs downstream could benefit from it | 20:57 |
| weshay|ruck | not really | 20:57 |
| rlandy | k - qualifying 17 on 8.4 | 21:00 |
| *** jmasud has quit IRC | 21:02 | |
| *** jfrancoa has joined #oooq | 21:17 | |
| *** jmasud has joined #oooq | 21:38 | |
| *** jbadiapa has quit IRC | 21:45 | |
| *** slaweq has quit IRC | 21:52 | |
| *** slaweq has joined #oooq | 21:54 | |
| *** jfrancoa has quit IRC | 22:09 | |
| *** jfrancoa has joined #oooq | 22:11 | |
| *** jfrancoa has quit IRC | 22:28 | |
| *** jmasud has quit IRC | 22:55 | |
| *** slaweq has quit IRC | 23:05 | |
| *** slaweq has joined #oooq | 23:06 | |
| rlandy | weshay|ruck: provider-jobs never report to dlrn, right? | 23:08 |
| rlandy | weshay|ruck: ysandeep|away: chkumar|ruck: fyi ... https://code.engineering.redhat.com/gerrit/232278 Add base provider internal job | 23:16 |
| rlandy | config job | 23:16 |
| rlandy | will need to merge to test out | 23:16 |
| rlandy | have to parent from upstream to avoid dup'ing the playbooks called | 23:16 |
| *** jmasud has joined #oooq | 23:21 | |
| *** rlandy has quit IRC | 23:53 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!