| *** jmasud has quit IRC | 00:01 | |
| *** jmasud has joined #oooq | 00:22 | |
| *** tosky has quit IRC | 00:38 | |
| rlandy | weshay|ruck: frenzy_friday: stream looking good | 01:10 |
|---|---|---|
| *** ysandeep|away is now known as ysandeep|ruck | 01:14 | |
| ysandeep|ruck | rlandy, weshay|ruck hey o/ anything you want me to pick up? | 01:14 |
| ysandeep|ruck | rlandy, thanks for https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/776692 , looks good | 01:14 |
| rlandy | ysandeep|ruck: weshay|ruck: 17 is looking good minus ipa - I have a fix for that | 01:14 |
| rlandy | ysandeep|ruck: yep - that can go | 01:15 |
| rlandy | ysandeep|ruck: ovb is failing on deploy - no available host | 01:16 |
| rlandy | possible that the 8.4 image is bigger | 01:16 |
| rlandy | will have to look into that | 01:16 |
| ysandeep|ruck | 17 is missing some component jobs, i will add them today.. | 01:16 |
| rlandy | last 16.2 failure is multinode ... | 01:16 |
| rlandy | it's failing on update - that should not be running | 01:17 |
| rlandy | 2021-02-21 03:08:43.770745 | primary | TASK [build-test-packages : gather facts used by role] ************************* | 01:17 |
| rlandy | 2021-02-21 03:08:43.770777 | primary | Sunday 21 February 2021 03:08:43 +0000 (0:00:00.079) 1:33:24.476 ******* | 01:17 |
| rlandy | 2021-02-21 03:09:27.893974 | primary | fatal: [subnode-1]: UNREACHABLE! => { | 01:17 |
| rlandy | 2021-02-21 03:09:27.894065 | primary | "changed": false, | 01:17 |
| rlandy | 2021-02-21 03:09:27.894075 | primary | "unreachable": true | 01:17 |
| rlandy | 2021-02-21 03:09:27.894083 | primary | } | 01:17 |
| rlandy | although it was running and passing before | 01:17 |
| rlandy | and is passing in rdo | 01:17 |
| rlandy | but never the less it shouldn't run | 01:18 |
| * ysandeep|ruck looking into multinode | 01:18 | |
| rlandy | I tried remove-tags but it didn't help | 01:18 |
| rlandy | ysandeep|ruck: the issue started I think when we used one playbook for multinode | 01:18 |
| rlandy | https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/build-test-packages/tasks/main.yml#L4 | 01:18 |
| rlandy | ^^ failure is there | 01:19 |
| rlandy | but build-test packages should not be running | 01:19 |
| rlandy | ysandeep|ruck: stream 8 is also looking promising | 01:21 |
| rlandy | will discuss more about that tomorrow | 01:21 |
| ysandeep|ruck | with 8.3 - build-test package was running okay earlier on 8.3 | 01:23 |
| rlandy | ysandeep|ruck: ack - it was | 01:24 |
| *** jmasud has quit IRC | 01:24 | |
| rlandy | ysandeep|ruck: but this is a diff problem .. | 01:26 |
| rlandy | we should not run this playbook at all | 01:26 |
| rlandy | https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/playbooks/multinode-undercloud-upgrade.yml#L22 | 01:26 |
| rlandy | ^^ not an upgrade job | 01:26 |
| rlandy | we're running tags all | 01:26 |
| rlandy | ysandeep|ruck: anyways - need to step away ... will be back in my morning ... | 01:27 |
| ysandeep|ruck | rlandy, i will check this o/ have a great evening | 01:27 |
| rlandy | tried to get stream going today so didn't have a ton of time to look at 16.2 | 01:27 |
| rlandy | but we're basically at deploy with ovb and this problem with multinode | 01:28 |
| *** rlandy has quit IRC | 01:28 | |
| *** jmasud has joined #oooq | 01:32 | |
| weshay|ruck | ysandeep|ruck, any bm jobs yet on 17? | 01:41 |
| ysandeep|ruck | weshay|ruck, On friday, they are failing on image build.. i will relook today | 01:44 |
| ysandeep|ruck | were* | 01:44 |
| weshay|ruck | k.. ysandeep|ruck both image build jobs passed in the line | 01:44 |
| ysandeep|ruck | weshay|ruck, aye i will try getting a pass in integration line first , i was checking status of component line bm jobs, https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-bm_envE-3ctlr_1comp-featureset001-tripleo-rhos-17/17b0839/job-output.txt | 01:46 |
| weshay|ruck | ysandeep|ruck, k.. will look ... I'm going to promote-17 but note it's w/o a full deployment | 01:47 |
| weshay|ruck | one step closer :) | 01:47 |
| ysandeep|ruck | weshay|ruck, patch is up to add 17 https://code.engineering.redhat.com/gerrit/#/c/228720/ .. we can merge this today if we get a green run. | 02:01 |
| ysandeep|ruck | weshay|ruck, fyi.. we are reusing envA - we might need to adjust the timing of baremetal line trigger so that.. integration 17 line run vs baretal line don't conflicts.. i will work on that | 02:03 |
| ysandeep|ruck | weshay|ruck, could you please review https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/776692 | 02:08 |
| weshay|ruck | looking | 02:09 |
| weshay|ruck | ysandeep|ruck, wf | 02:15 |
| *** ysandeep|ruck has quit IRC | 02:18 | |
| *** ysandeep has joined #oooq | 02:19 | |
| weshay|ruck | ysandeep, need anything else? | 02:19 |
| *** jmasud has quit IRC | 02:19 | |
| ysandeep | weshay|ruck, stein is still not fixed.. i will check with arxcruz | 02:20 |
| ysandeep | After your repush - https://quay.io/repository/tripleostein/centos-binary-aodh-api?tab=tags - Containers with suffix x86_64 is available but we are still missing containers without x86_64 suffix. | 02:20 |
| ysandeep | ~~~ | 02:20 |
| ysandeep | db93471c423322fc24743db3545da0a11b073599_a1cf5a25_x86_64 | 02:20 |
| ysandeep | ~~~ | 02:20 |
| ysandeep | arxcruz, ^^ | 02:20 |
| weshay|ruck | k.. | 02:21 |
| weshay|ruck | :( | 02:21 |
| weshay|ruck | ysandeep, can you ask chandankumar about the retry across registries | 02:21 |
| ysandeep | weshay|ruck, yes, i will chat with him today. | 02:21 |
| weshay|ruck | rock on.. | 02:23 |
| weshay|ruck | ysandeep++ take care | 02:23 |
| weshay|ruck | 0/ | 02:23 |
| ysandeep | o/ have a great evening | 02:23 |
| *** jmasud has joined #oooq | 02:38 | |
| *** ysandeep is now known as ysandeep|away | 02:48 | |
| *** jmasud has quit IRC | 03:24 | |
| *** udesale has joined #oooq | 04:08 | |
| *** ysandeep|away is now known as ysandeep|ruck | 04:27 | |
| *** jmasud has joined #oooq | 04:33 | |
| *** ykarel has joined #oooq | 04:36 | |
| *** ratailor has joined #oooq | 04:46 | |
| *** saneax has joined #oooq | 04:51 | |
| *** saneax has quit IRC | 04:52 | |
| *** jmasud has quit IRC | 05:08 | |
| *** saneax has joined #oooq | 05:10 | |
| *** jmasud has joined #oooq | 05:29 | |
| *** marios has joined #oooq | 06:00 | |
| *** sanjayu_ has joined #oooq | 06:00 | |
| *** saneax has quit IRC | 06:03 | |
| *** udesale_ has joined #oooq | 06:08 | |
| *** udesale has quit IRC | 06:11 | |
| *** slaweq_ has joined #oooq | 06:50 | |
| *** udesale__ has joined #oooq | 07:26 | |
| *** jpodivin has quit IRC | 07:29 | |
| *** udesale_ has quit IRC | 07:30 | |
| *** jpodivin has joined #oooq | 07:30 | |
| *** ysandeep|ruck is now known as ysandeep|lunch | 07:35 | |
| *** udesale__ has quit IRC | 08:00 | |
| *** jmasud has quit IRC | 08:01 | |
| *** ykarel_ has joined #oooq | 08:31 | |
| *** ykarel has quit IRC | 08:33 | |
| *** jlarriba has joined #oooq | 08:34 | |
| *** ysandeep|lunch is now known as ysandeep|ruck | 08:39 | |
| *** jlarriba has quit IRC | 08:48 | |
| *** tosky has joined #oooq | 08:50 | |
| *** jlarriba has joined #oooq | 08:51 | |
| arxcruz | ysandeep|ruck: the containers on centos 7 have only the x86_64 sufix | 08:53 |
| arxcruz | there's no one without, I can do a workaround though | 08:53 |
| ysandeep|ruck | arxcruz, checking container build logs | 08:55 |
| *** jpena|off is now known as jpena | 08:58 | |
| ysandeep|ruck | hmm.. we don't have report.html for older releases | 08:58 |
| ysandeep|ruck | arxcruz, rdo registry have both the containers with/without x86_64 sufix | 09:01 |
| ysandeep|ruck | https://trunk.registry.rdoproject.org:8443/oapi/v1/namespaces/tripleostein/imagestreamtags/ | 09:01 |
| ysandeep|ruck | name": "centos-binary-aodh-api:db93471c423322fc24743db3545da0a11b073599_a1cf5a25", | 09:01 |
| ysandeep|ruck | "name": "centos-binary-aodh-api:db93471c423322fc24743db3545da0a11b073599_a1cf5a25_x86_64" | 09:01 |
| *** udesale has joined #oooq | 09:06 | |
| arxcruz | ysandeep|ruck: yeah, but, in the job, it only shows up the x864 | 09:14 |
| arxcruz | ysandeep|ruck: https://logserver.rdoproject.org/openstack-periodic-integration-stable3-centos7/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-train-containers-build-push/076c28e/job-output.txt | 09:15 |
| arxcruz | 2021-02-20 11:06:44.465697 | primary | 09d228d73c4afb12fb9f85e0367086fdf8c0ddfb_effdb661_x86_64: digest: sha256:6dfbd1a2cb32f8511c23363d3b26cf667de66eb914ab9f70419154a7f250408e size: 6574 | 09:15 |
| arxcruz | 2021-02-20 11:06:44.513576 | | 09:15 |
| arxcruz | 2021-02-20 11:06:44.513789 | LOOP [build-containers : Tag w/ arch suffix and push image: trunk.registry.rdoproject.org/tripleotrain/centos-binary-nova-compute] | 09:15 |
| ysandeep|ruck | ^^ train | 09:16 |
| *** udesale has quit IRC | 09:17 | |
| ysandeep|ruck | arxcruz, stein logs: https://logserver.rdoproject.org/openstack-periodic-integration-stable4-5/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-stein/9aa1c86/job-output.txt | 09:18 |
| arxcruz | ysandeep|ruck: ? | 09:18 |
| arxcruz | ysandeep|ruck: doesn't matter, here's stein logs https://logserver.rdoproject.org/openstack-periodic-integration-stable4-5/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-stein-containers-build-push/6f2085c/job-output.txt | 09:19 |
| arxcruz | same thing | 09:19 |
| ysandeep|ruck | arxcruz, i found some difference between last execution vs older execution - https://logserver.rdoproject.org/openstack-periodic-integration-stable4-5/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-stein-containers-build-push/d3c219a/job-output.txt - search for TASK [build-containers : Get actual built containers] | 09:24 |
| ysandeep|ruck | ^^ this task was skipped in last execution. | 09:25 |
| arxcruz | ysandeep|ruck: aha! | 09:25 |
| arxcruz | you're right | 09:25 |
| arxcruz | and only output is from the x86)64 | 09:26 |
| arxcruz | wondering now why it was skipped | 09:26 |
| arxcruz | ysandeep|ruck: the task doesn't have any condition to be skipped | 09:30 |
| arxcruz | https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/build-containers/tasks/build-report.yaml#L57 | 09:30 |
| *** skramaja has joined #oooq | 09:34 | |
| *** slaweq_ is now known as slaweq | 09:40 | |
| ysandeep|ruck | arxcruz, correct.. wondering if https://github.com/openstack/tripleo-ci/commit/a3c9dc40e48637fcc1b0f847cc7a506b23e8dd44 is somehow related .. that's the only change in the file.. | 09:41 |
| * ysandeep|ruck thinks container without x86 suffix is also creating properly .. because we can see them in rdo | 09:47 | |
| arxcruz | ysandeep|ruck: aha, we can blame ykarel_ https://opendev.org/openstack/tripleo-ci/commit/3b387897994ee9ba02d467b1ff73251e188cdd29 | 09:47 |
| ysandeep|ruck | arxcruz++ yeah looks like this broke the reporting for c7 container build | 09:50 |
| *** derekh has joined #oooq | 09:51 | |
| *** udesale has joined #oooq | 09:53 | |
| arxcruz | ysandeep|ruck: i think the logic is correct, that report should never be executed because it's kolla | 09:53 |
| arxcruz | but we can add on this post the same task just to show's up the containers | 09:53 |
| arxcruz | because it's all we need | 09:53 |
| arxcruz | ysandeep|ruck: https://review.opendev.org/c/openstack/tripleo-ci/+/777084 | 09:58 |
| arxcruz | ysandeep|ruck: is it safe to run the periodic-tripleo-centos-7-stein-containers-build-push with a depends-on ? | 09:59 |
| ysandeep|ruck | arxcruz, yes.. tripleo-ci-testing is same | 09:59 |
| arxcruz | ysandeep|ruck: ack | 09:59 |
| ysandeep|ruck | arxcruz, even docker have the same hash container without x86 suffix.. wondering how it worked for docker? | 10:01 |
| ysandeep|ruck | ~~~[root@undercloud ~]# podman pull docker.io/tripleostein/centos-binary-nova-api:db93471c423322fc24743db3545da0a11b073599_a1cf5a25 | 10:01 |
| ysandeep|ruck | Trying to pull docker.io/tripleostein/centos-binary-nova-api:db93471c423322fc24743db3545da0a11b073599_a1cf5a25... | 10:01 |
| ysandeep|ruck | Getting image source signatures | 10:01 |
| ysandeep|ruck | ... | 10:01 |
| ysandeep|ruck | Storing signatures | 10:01 |
| ysandeep|ruck | 80039942fd0f9acbe104b487c8313c4ca92e513e28c1077485f37ee72bd8a7a2 | 10:01 |
| arxcruz | ysandeep|ruck: centos-8 have a very nice report | 10:02 |
| arxcruz | with both hashes | 10:02 |
| arxcruz | ysandeep|ruck: http://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push-victoria/d860331/logs/containers-built.log for example | 10:03 |
| ysandeep|ruck | yup, earlier same thing was getting captured for c7 stein too - https://logserver.rdoproject.org/openstack-periodic-integration-stable4-5/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-stein-containers-build-push/d3c219a/logs/containers-built.log | 10:05 |
| *** udesale has quit IRC | 10:05 | |
| ysandeep|ruck | arxcruz, apart from this weshay|ruck want us to discuss with chandankumar, if we can configure failover of registries for these older branches.. | 10:05 |
| arxcruz | ok... | 10:05 |
| *** jmasud has joined #oooq | 10:16 | |
| *** ykarel_ is now known as ykarel | 10:20 | |
| chandankumar | ysandeep|ruck: you mean fallback for c7 containers registeries? | 10:23 |
| ysandeep|ruck | chandankumar, yes | 10:24 |
| ykarel | ysandeep|ruck, arxcruz what's this issue with build-report disable patch? | 10:51 |
| ykarel | is it still not getting disabled? | 10:52 |
| arxcruz | ykarel: it is | 10:52 |
| arxcruz | ykarel: the problem is that our tool to copy containers to quay, was using that report to parse the list of containers | 10:52 |
| ykarel | ahh | 10:53 |
| ykarel | so u collect that report from logs and then push | 10:54 |
| arxcruz | ykarel: yup | 10:55 |
| ykarel | arxcruz, is this same happens while pushing to docker.io? | 10:55 |
| arxcruz | ykarel: nope, just quay | 10:55 |
| ykarel | and why not using same methods(to collect container images list) as push to docker.io | 10:55 |
| ykarel | i think it's better to do the same while doing promotions itself | 10:56 |
| ykarel | along with push to docker.io | 10:56 |
| *** dtantsur|afk is now known as dtantsur | 11:03 | |
| ysandeep|ruck | ykarel, +1 to idea of using same method for docker and quay. arxcruz chandankumar ^^ | 11:06 |
| arxcruz | ysandeep|ruck: can you show me where it is? | 11:06 |
| ysandeep|ruck | arxcruz, from logs http://38.102.83.109/promoter_logs/centos7_stein.log-20200721 looks like pushing containers to docker is part of promotion process. | 11:11 |
| arxcruz | ysandeep|ruck: hmmmm i need to check, the tool is already done to get the latest success job from zuul | 11:12 |
| arxcruz | not of checks | 11:12 |
| ysandeep|ruck | where that tool resides - tox-box vm? | 11:14 |
| ykarel | ysandeep|ruck, arxcruz it's in https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/repo_client.py#L92 | 11:17 |
| arxcruz | ysandeep|ruck: yes, in toolbox vm | 11:18 |
| arxcruz | ykarel: ysandeep|ruck it's written in go ;) | 11:18 |
| arxcruz | i can work on consume dlrnapi instead of zuul log | 11:19 |
| ykarel | promoter uses tripleo-common source to fetch containers list | 11:20 |
| *** apetrich has quit IRC | 11:47 | |
| *** apetrich has joined #oooq | 11:48 | |
| *** jlarriba has quit IRC | 11:50 | |
| *** jlarriba has joined #oooq | 11:52 | |
| *** skramaja has quit IRC | 11:59 | |
| *** ratailor has quit IRC | 12:04 | |
| *** chem has joined #oooq | 12:15 | |
| *** apetrich has quit IRC | 12:24 | |
| *** jmasud has quit IRC | 12:26 | |
| *** rlandy has joined #oooq | 12:29 | |
| chandankumar | marios: o/ need some help on fixing tox -e docs issue https://review.opendev.org/c/openstack/tripleo-specs/+/772442 - https://zuul.opendev.org/t/openstack/build/925f0081e4d745a08a14cd8365ad1c1e please have a look when free, thanks ! | 12:30 |
| *** jpena is now known as jpena|lunch | 12:30 | |
| rlandy | ysandeep|ruck: hey - any progress with ovb or multinode on 16.2? if not, no worries, I'll look at it again after training class | 12:31 |
| marios | chandankumar: looking | 12:32 |
| rlandy | marios: hi - is there a reason that fs010 multinode runs the update playbooks? | 12:32 |
| marios | rlandy: o/ what is that job even? have to dig there i am not familiar with it | 12:33 |
| *** apetrich has joined #oooq | 12:33 | |
| rlandy | marios: posting links ... | 12:33 |
| ysandeep|ruck | rlandy, with multinode i agree with that task should not run, that's because we moved everything to multinode.yml and this task run on build tag | 12:34 |
| ysandeep|ruck | https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/playbooks/multinode-overcloud-upgrade.yml#L26 | 12:34 |
| marios | rlandy: are you referring to tripleo-ci-centos-8-containers-multinode | 12:34 |
| rlandy | ysandeep|ruck: ^^ ack - talking with marios about that now | 12:34 |
| rlandy | just making sure before I get rid of it | 12:34 |
| rlandy | marios: https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-multinode-1ctlr-featureset010-rhos-16.2/9cbcbba/job-output.txt | 12:34 |
| marios | rlandy: k but gimme sec looking at sthing else first for chandankumar | 12:34 |
| rlandy | marios: no rush - just posting evidence | 12:35 |
| rlandy | 2021-02-23 02:23:39.699794 | primary | PLAY [Build the gerrit changes on the relevant release for the upgrade] ******** | 12:35 |
| rlandy | 2021-02-23 02:23:39.722778 | primary | | 12:35 |
| rlandy | 2021-02-23 02:23:39.722814 | primary | TASK [build-test-packages : gather facts used by role] ************************* | 12:35 |
| rlandy | 2021-02-23 02:23:39.722821 | primary | Tuesday 23 February 2021 02:23:39 +0000 (0:00:00.060) 1:13:36.711 ****** | 12:35 |
| rlandy | ^^ fails on an update task | 12:35 |
| rlandy | it should not be running iiuc, | 12:35 |
| marios | rlandy: well it should not be so i'm guessing we have a conditional wrong somewhere but will dig in a bit | 12:35 |
| rlandy | I think this issue started with the multinode.yaml (one shot playbook) | 12:35 |
| marios | rlandy: yes | 12:35 |
| rlandy | we run tags all | 12:36 |
| ysandeep|ruck | https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/playbooks/multinode-overcloud-upgrade.yml#L26 - build tag | 12:36 |
| marios | rlandy: i saw that in the zuul.d/multinode definition of it with 401 playbooks:$ | 12:36 |
| marios | 402 - multinode.yml$ | 12:36 |
| marios | i.e. all the playbooks in multinode.yml | 12:36 |
| rlandy | ysandeep|ruck: ack - we can't get rid of the build tag in total | 12:36 |
| rlandy | we need to switch case that playbook better | 12:36 |
| rlandy | all of them | 12:36 |
| marios | rlandy: well they rely on things set in the featureset sec lemme fetch pointer | 12:37 |
| marios | rlandy: e.g. 401 playbooks:$ | 12:37 |
| rlandy | marios: ^^ thanks - that will help set the right conditional for skipping those playbooks | 12:37 |
| marios | 402 - multinode.yml$ | 12:37 |
| marios | rlandy: sorry e.g. https://opendev.org/openstack/tripleo-quickstart-extras/src/commit/0dbe78b749bcabfe14ed75c49391c84c36c2f80a/playbooks/multinode-overcloud-update.yml#L9 | 12:37 |
| marios | rlandy: i.e. that 'overcloud_update' is set on the featureset, when it *is* an update | 12:38 |
| rlandy | somehow this thing is kicking | 12:38 |
| marios | rlandy: definitely not set in fs10 https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset010.yml | 12:39 |
| rlandy | marios: I tried ... | 12:40 |
| rlandy | remove-tags: - overcloud-update | 12:40 |
| rlandy | didn't help | 12:40 |
| rlandy | the role is run on when: overcloud_update|default(false)|bool | 12:40 |
| rlandy | but the failure is in get_facts | 12:40 |
| * ysandeep|ruck trying to recall a patch where marios and sshnaidm was working.. around multinode.yml | 12:40 | |
| rlandy | before the role decision is made | 12:40 |
| rlandy | marios: so iiuc, gather_facts will still run and the swit conly happens on the role | 12:41 |
| marios | rlandy: ah i se | 12:41 |
| marios | rlandy: yeah so it isn't running the role | 12:41 |
| marios | rlandy: it is just evaluating the playbook | 12:42 |
| rlandy | marios: we need to when the enire playbook | 12:42 |
| rlandy | yep | 12:42 |
| marios | rlandy: could split the gather_facts into a conditional task, instead of on the play? | 12:42 |
| rlandy | marios: or move the when up | 12:44 |
| rlandy | if possible | 12:44 |
| marios | rlandy: yeah but can't cos there is no up i mean it doesn't even have tasks it is all at the play level there | 12:45 |
| marios | rlandy: so i *think* adding two tasks there, in a block with the when condition | 12:45 |
| marios | rlandy: one with 'setup' module and the other iwth include_role | 12:46 |
| rlandy | marios: ok - putting in patch for your review | 12:46 |
| rlandy | sec | 12:46 |
| marios | rlandy: but i am not 100% sure if the setup will then be OK, i.e. because it is a task instead of on the play | 12:46 |
| marios | rlandy: but worth a shot imo | 12:46 |
| marios | rlandy: otherwise you will have to make the inclusion of the entire playbook conditional i.e. something in multinode.yml | 12:47 |
| marios | rlandy: but hold no its failing because it can't reach undercloud? | 12:48 |
| marios | rlandy: 2021-02-23 02:23:39.722814 | primary | TASK [build-test-packages : gather facts used by role] ************************* | 12:48 |
| marios | 2021-02-23 02:24:23.833610 | primary | fatal: [subnode-1]: UNREACHABLE! => { | 12:48 |
| rlandy | marios: true | 12:49 |
| rlandy | but really it's a different issue | 12:49 |
| rlandy | it will probably fail down the line in tempest | 12:49 |
| rlandy | but the playbook still should not be running | 12:49 |
| marios | rlandy: so it didn't run the update playbook yet? | 12:49 |
| rlandy | marios: no - but it started the gather_fact from there | 12:49 |
| marios | rlandy: 2021-02-23 02:24:24.379973 | primary | +(./toci_quickstart.sh:166): main(): echo 'Playbook run of multinode.yml failed' | 12:49 |
| ysandeep|ruck | unable to connect secondary node | 12:50 |
| marios | rlandy: k i think i follow now... was confused by it failing on build-test-packages | 12:50 |
| rlandy | Build the gerrit changes on the relevant release for the upgrade | 12:50 |
| rlandy | ^^ not needed at all | 12:50 |
| rlandy | marios; you are right - the real problem is that it can't reach the undercloud | 12:51 |
| marios | rlandy: https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-multinode-1ctlr-featureset010-rhos-16.2/9cbcbba/logs/quickstart_files/playbook_executions.log | 12:51 |
| marios | rlandy: it hasn't run the update play yet i think | 12:51 |
| marios | rlandy: see the playbook executions? | 12:51 |
| marios | rlandy: damnit i keep forgetting they are all in one file | 12:51 |
| ysandeep|ruck | rlandy, undercloud or secondary node.. ? | 12:51 |
| marios | rlandy: --skip-tags tripleo-validations,teardown-all /home/zuul/workspace/.quickstart/playbooks/multinode.yml | 12:51 |
| rlandy | secondary node failure | 12:52 |
| * rlandy runs again gets on node | 12:52 | |
| rlandy | frenzy_friday: hey - you got a full pass here ... https://review.rdoproject.org/r/#/c/32041/ | 12:53 |
| rlandy | https://logserver.rdoproject.org/41/32041/5/check/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/fabdcc6/logs/undercloud/etc/yum.repos.d/quickstart-centos-base.repo.txt.gz | 12:54 |
| rlandy | stream - correct | 12:54 |
| rlandy | also here https://logserver.rdoproject.org/41/32041/5/check/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/fabdcc6/logs/overcloud-novacompute-1/etc/yum.repos.d/quickstart-centos-base.repo.txt.gz | 12:54 |
| rlandy | 2021-02-22 22:26:49.856185 | TASK [get_hash : read md5] | 12:55 |
| rlandy | 2021-02-22 22:26:50.879806 | primary | 859a30b5d9a2d95eff1feef91a99a37b | 12:55 |
| frenzy_friday | rlandy, yep! What next? | 12:55 |
| rlandy | ContainerZaqarImage: 192.168.24.1:8787/tripleomaster/openstack-zaqar-wsgi:859a30b5d9a2d95eff1feef91a99a37b-updated-20210222222913 | 12:55 |
| rlandy | right containers | 12:55 |
| rlandy | frenzy_friday: we switch the main line ... | 12:55 |
| rlandy | will discuss at community call | 12:56 |
| rlandy | weshay|ruck: arxcruz: ysandeep|ruck: ^^ | 12:56 |
| rlandy | fyi | 12:56 |
| rlandy | looks like we are ready to roll stream | 12:56 |
| frenzy_friday | Cool. I fixed https://review.opendev.org/c/openstack/tripleo-ci/+/775429 this one as well, it is passing now | 12:56 |
| rlandy | nice | 12:56 |
| *** amoralej is now known as amoralej|lunch | 13:00 | |
| arxcruz | ysandeep|ruck: https://logserver.rdoproject.org/71/31971/5/check/periodic-tripleo-centos-7-stein-containers-build-push/c490464/job-output.txt it passes and shows up the containers, let's do it for now, then i'll work to collect it from dlrn | 13:03 |
| ysandeep|ruck | arxcruz, ack o/ | 13:03 |
| marios | chandankumar: posting a fix sec | 13:04 |
| chandankumar | marios: thanks! | 13:04 |
| chandankumar | marios: zbr rlandy https://review.rdoproject.org/r/#/c/32059/ when free, thansk! | 13:14 |
| weshay|ruck | bhagyashris, akahat chandankumar we should remove the docker prune temporarily so we can interate more quickly on the promoter | 13:17 |
| chandankumar | weshay|ruck: ok | 13:17 |
| akahat | weshay|ruck, we might run out of space . | 13:17 |
| bhagyashris | weshay|ruck, ack | 13:17 |
| akahat | if we are targeting 3-4 releases | 13:17 |
| weshay|ruck | akahat, I think we'll be ok.. we can watch it | 13:18 |
| akahat | weshay|ruck, okay. on it. | 13:18 |
| weshay|ruck | you folks figure out the overcloud image settings? | 13:18 |
| weshay|ruck | I'd like to see a review w/ the settings in source control | 13:19 |
| chandankumar | weshay|ruck: https://review.rdoproject.org/r/32069 | 13:19 |
| bhagyashris | weshay|ruck, yes | 13:19 |
| weshay|ruck | chandankumar, k.. I think the key is just ~/.ssh/id_rsa | 13:19 |
| weshay|ruck | will check | 13:19 |
| weshay|ruck | chandankumar, bhagyashris akahat the keys have to be manually added in the setup | 13:20 |
| weshay|ruck | here's what we have atm | 13:20 |
| weshay|ruck | authorized_keys bak id_rsa id_rsa-blah id_rsa-old id_rsa.pub id_rsa_uploader known_hosts | 13:20 |
| weshay|ruck | :) | 13:20 |
| chandankumar | weshay|ruck: in old c8 promoter, we have a seperate key for uploader | 13:20 |
| chandankumar | I copied it from there to new one | 13:20 |
| weshay|ruck | chandankumar, don't see it on the promoter user | 13:21 |
| weshay|ruck | oh.. | 13:21 |
| weshay|ruck | now I do | 13:21 |
| weshay|ruck | chandankumar, what's the public key for id_rsa_uploader? | 13:21 |
| * weshay|ruck creates one | 13:21 | |
| bhagyashris | arxcruz, zbr, sshnaidm, rlandy, marios, ysandeep|ruck , bhagyashris, svyas, soniya29, pojadhav, akahat, weshay|ruck , chandankumar, frenzy_friday | 13:21 |
| chandankumar | weshay|ruck: no idea, but it matches which the key present on infra doc | 13:21 |
| weshay|ruck | chandankumar, or I guess we can just test the sftp connection | 13:21 |
| bhagyashris | Community call in 9 min | 13:22 |
| bhagyashris | https://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg?both#2021-02-23-Community-call | 13:22 |
| weshay|ruck | chandankumar++ | 13:22 |
| bhagyashris | https://meet.google.com/bqx-xwht-wky?authuser=0 | 13:22 |
| weshay|ruck | that key works fine | 13:22 |
| chandankumar | akahat++ bhagyashris++ for promoter work | 13:23 |
| weshay|ruck | I can't get a tmux session attached | 13:24 |
| weshay|ruck | it just hangs | 13:24 |
| weshay|ruck | perhaps we can kill all the sessions? | 13:24 |
| weshay|ruck | and start over? | 13:24 |
| bhagyashris | weshay|ruck, yes, same i am facing | 13:24 |
| weshay|ruck | akahat, chandankumar can I kill tmux? | 13:25 |
| chandankumar | weshay|ruck: yes go ahead | 13:25 |
| weshay|ruck | k.. it's working now | 13:26 |
| akahat | chandankumar, yes | 13:27 |
| chandankumar | weshay|ruck: rlandy: https://review.opendev.org/c/openstack/tripleo-specs/+/777101/1 | 13:28 |
| chandankumar | marios: rlandy weshay|ruck sshnaidm moved cli design here https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ea5/772442/18/check/openstack-tox-docs/ea503d9/docs/specs/wallaby/tripleo-repos-single-source.html#cli-design | 13:29 |
| * akahat thinking container images can get removed by container-push role and again get pulled, there is no effect of "docker system prune" | 13:33 | |
| *** jpena|lunch is now known as jpena | 13:33 | |
| chandankumar | akahat: you are correct | 13:33 |
| weshay|ruck | chandankumar, added Alex's notes to line 74 on https://hackmd.io/v2jCX9RwSeuP8EEFDHRa8g | 13:57 |
| *** rlandy is now known as rlandy|training | 13:57 | |
| *** jlarriba has quit IRC | 14:02 | |
| *** jlarriba has joined #oooq | 14:03 | |
| *** amoralej|lunch is now known as amoralej | 14:05 | |
| *** udesale has joined #oooq | 14:07 | |
| arxcruz | brb 30 min | 14:12 |
| chandankumar | ysandeep|ruck: I need to move https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/container-build/tasks/non_tripleo_containers.yml for this for c7 roles some where here https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/build-containers/tasks/provider_push.yaml | 14:21 |
| ysandeep|ruck | chandankumar, ack | 14:22 |
| bhagyashris | weshay|ruck, here we are using stage_root config https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/qcow_client.py#L79 | 14:25 |
| bhagyashris | thats why i changed the value here https://review.rdoproject.org/r/#/c/32069/5/ci-scripts/dlrnapi_promoter/config_environments/rdo/defaults.yaml@16 | 14:26 |
| weshay|ruck | bhagyashris, akahat I have a quick 1-1 . .then we should sync | 14:27 |
| bhagyashris | weshay|ruck, ack | 14:27 |
| akahat | weshay|ruck, okay | 14:27 |
| weshay|ruck | bhagyashris, akahat https://meet.google.com/dkj-vuij-yup | 14:41 |
| weshay|ruck | chandankumar, talking about the promoter ^ | 14:41 |
| weshay|ruck | if you want to join.. no requirement | 14:41 |
| akahat | weshay|ruck, http://10.0.148.74/promoter_logs/centos8_victoria.log search for "toomanyrequests" | 14:46 |
| *** ysandeep|ruck is now known as ysandeep|dinner | 14:51 | |
| akahat | weshay|ruck, https://review.rdoproject.org/r/32069 | 14:52 |
| *** ykarel has quit IRC | 14:59 | |
| *** udesale has quit IRC | 14:59 | |
| *** udesale has joined #oooq | 15:00 | |
| rlandy|training | frenzy_friday: how long are you still around? | 15:08 |
| rlandy|training | frenzy_friday: ^^ can you put up a change to out the same that is in the testproject into the man line? | 15:10 |
| rlandy|training | main | 15:10 |
| rlandy|training | weshay|ruck: ^^ fyi | 15:11 |
| frenzy_friday | rlandy|training, I'll be on till late today. | 15:11 |
| rlandy|training | also, can you annotate the hackmd a bit more | 15:11 |
| rlandy|training | so we can explain what review does what | 15:11 |
| frenzy_friday | But we need another testproject with all the jobs running on c8 stream nodes instead of s8 too right? | 15:11 |
| frenzy_friday | using the nodeset names from https://review.rdoproject.org/r/#/c/32061/ | 15:13 |
| *** ysandeep|dinner is now known as ysandeep|ruck | 15:25 | |
| bhagyashris | jpena, hi, we are getting unauthorized while doing promotion at the end http://paste.openstack.org/show/802934/ | 15:26 |
| bhagyashris | jpena, we are working with new promoter code | 15:26 |
| jpena | bhagyashris: can you check how you passed the user and password? You have the DLRNAPI_USERNAME and DLRNAPI_PASSWORD environment variables | 15:28 |
| rlandy|training | frenzy_friday: yes - we need both | 15:29 |
| frenzy_friday | rlandy|training, ok, adding the other one | 15:32 |
| bhagyashris | weshay|ruck, https://trunk.rdoproject.org/centos8-victoria/current-tripleo/delorean.repo.md5 | 15:37 |
| akahat | jpena, there was issue in the code.. now it's fixed. we've promoted successfully. | 15:38 |
| bhagyashris | jpena, thank you :) there was issue in the code thanks :) | 15:38 |
| akahat | you can see the promoted hash above ^^ | 15:38 |
| jpena | nice! | 15:38 |
| weshay|ruck | nice.. that looks right | 15:38 |
| akahat | jpena, thank you :) | 15:38 |
| weshay|ruck | bhagyashris, akahat can you please capture the change in a review before taking off? | 15:39 |
| akahat | weshay|ruck, promoting ussuri now. | 15:39 |
| weshay|ruck | arxcruz, ysandeep|ruck ^ | 15:39 |
| akahat | weshay|ruck, yup. | 15:39 |
| bhagyashris | weshay|ruck, yes | 15:39 |
| weshay|ruck | thanks for helping jpena | 15:40 |
| weshay|ruck | oh man.. | 15:40 |
| weshay|ruck | it was just the registry info? | 15:40 |
| weshay|ruck | bah | 15:40 |
| weshay|ruck | I should have caught that | 15:40 |
| weshay|ruck | akahat, bhagyashris use the vexhost promoter as a reference please :) | 15:40 |
| chandankumar | weshay|ruck: nope, it was taking wrong password something wrong in the code side | 15:40 |
| weshay|ruck | OH | 15:40 |
| weshay|ruck | k | 15:40 |
| weshay|ruck | bhagyashris, akahat btw.. one great improvement to logs.. would be to include the agg, commit, distro hash in this message 2021-02-23 15:36:38,832 2681321 WARNING promoter Candidate label 'current-tripleo': NO candidate hash promoted to current-tripleo-rdo | 15:42 |
| weshay|ruck | bhagyashris, akahat perhaps a full summary of the promotion w/ links | 15:43 |
| weshay|ruck | I think you have all that info available | 15:43 |
| akahat | weshay|ruck, okay. I'll note it down. | 15:43 |
| bhagyashris | weshay|ruck, sure | 15:43 |
| weshay|ruck | ++ | 15:43 |
| weshay|ruck | chandankumar, akahat bhagyashris fyi.. [08:44:56] <apevec> weshay, sshnaidm - say goodbye to 38.145.34.55 (old promoter-server in rdocloud), shutting it down now! | 15:45 |
| bhagyashris | weshay|ruck, ussuri promoted https://trunk.rdoproject.org/centos8-ussuri/current-tripleo/delorean.repo.md5 | 15:51 |
| weshay|ruck | bhagyashris, k.. thanks /me checking it | 15:51 |
| weshay|ruck | containers look good | 15:52 |
| weshay|ruck | 12591acf172d2cc621866bca77b2b0f8 | 15:52 |
| weshay|ruck | docker pull tripleou/centos-binary-base:12591acf172d2cc621866bca77b2b0f8 | 15:52 |
| weshay|ruck | Last pushed7 hours agobyrdotripleomirror | 15:52 |
| weshay|ruck | DIGEST | 15:52 |
| weshay|ruck | OS/ARCH | 15:52 |
| weshay|ruck | COMPRESSED SIZE | 15:52 |
| weshay|ruck | 421733af5f25 | 15:52 |
| weshay|ruck | current-tripleo | 15:52 |
| weshay|ruck | docker pull tripleou/centos-binary-base:current-tripleo | 15:52 |
| weshay|ruck | Last pushed7 hours agobyrdotripleomirror | 15:52 |
| weshay|ruck | DIGEST | 15:52 |
| weshay|ruck | OS/ARCH | 15:52 |
| weshay|ruck | COMPRESSED SIZE | 15:52 |
| weshay|ruck | 421733af5f25 | 15:52 |
| weshay|ruck | images look good | 15:53 |
| weshay|ruck | current-tripleo/2021-02-22 04:53- | 15:53 |
| weshay|ruck | [DIR]12591acf172d2cc62186..>2021-02-22 04:53- | 15:53 |
| weshay|ruck | nice work | 15:53 |
| chandankumar | weshay|ruck: ysandeep|ruck arxcruz tomorrow we will be testing promoter to push containers to quay for victoria or ussuri FYI | 15:59 |
| ysandeep|ruck | chandankumar, ack | 15:59 |
| arxcruz | chandankumar: let me know prior it, so i can remove it from the list of containers | 16:01 |
| chandankumar | arxcruz: sure | 16:01 |
| *** sshnaidm is now known as sshnaidm|afk | 16:04 | |
| *** amoralej is now known as amoralej|off | 16:13 | |
| ysandeep|ruck | weshay|ruck, is it okay to use run_test_role_vars in job definations(or its just a test project thing) - we basically only need it for c7 sc004 train , Similiar to what I have tried here - https://review.rdoproject.org/r/#/c/32054/5/zuul.yaml , other option is to add standalone_control_virtual_ip in c7 release file? | 16:17 |
| ysandeep|ruck | context: https://trello.com/c/3Gx64Z58/1840-cixlp1915519tripleociproatraincentos7scenario004-failing-with-error-ip-192168243-already-exists-too-many-tries | 16:17 |
| * weshay|ruck looking | 16:17 | |
| weshay|ruck | ysandeep|ruck, https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/role-vars.j2 | 16:19 |
| weshay|ruck | yes :) | 16:19 |
| ysandeep|ruck | weshay|ruck, ack | 16:20 |
| weshay|ruck | ysandeep|ruck, fyi.. victoria and ussuri promoted | 16:20 |
| ysandeep|ruck | weshay|ruck, ++ awesome news, what's the situation with c8 train? | 16:21 |
| weshay|ruck | ysandeep|ruck, I think akahat and bhagyashris are live patching the server now | 16:22 |
| weshay|ruck | so.. perhaps tomorrow | 16:22 |
| ysandeep|ruck | ack o/ | 16:22 |
| ysandeep|ruck | we are completely green for c8 train https://review.rdoproject.org/zuul/buildset/0c074bf21d4e463d82d51e42baf7400b | 16:22 |
| *** jpodivin has quit IRC | 16:23 | |
| bhagyashris | weshay|ruck, here is the patch https://review.rdoproject.org/r/#/c/32069/ which help us to promote | 16:24 |
| bhagyashris | and the last change we did to resolve the authentication was this https://review.rdoproject.org/r/#/c/32069/6/ci-scripts/dlrnapi_promoter/logic.py | 16:24 |
| bhagyashris | fyi ^ | 16:25 |
| weshay|ruck | that looks better :) | 16:25 |
| bhagyashris | weshay|ruck, sorry this one | 16:26 |
| bhagyashris | https://review.rdoproject.org/r/#/c/32069/6/ci-scripts/dlrnapi_promoter/dlrn_client.py | 16:26 |
| weshay|ruck | aye | 16:26 |
| bhagyashris | weshay|ruck, anything you want me to share | 16:27 |
| bhagyashris | ? | 16:27 |
| weshay|ruck | I'm not in context of your question.. your patch looks good... what's the question? | 16:28 |
| bhagyashris | ok | 16:28 |
| bhagyashris | as i am leaving for the day | 16:28 |
| bhagyashris | so before that you want me to share in case if i miss anything ? | 16:29 |
| weshay|ruck | nope.. have a good night, we'll check in tomorrow | 16:29 |
| bhagyashris | ok | 16:29 |
| bhagyashris | Have a great day :) | 16:30 |
| * bhagyashris out | 16:30 | |
| frenzy_friday | Hey weshay|ruck, which job in integration line goes with which c8 stream nodeset https://review.rdoproject.org/r/#/c/32080/ ? | 16:38 |
| *** jmasud has joined #oooq | 16:42 | |
| weshay|ruck | frenzy_friday, it's integration-main | 16:59 |
| weshay|ruck | but oddly I don't see it atm | 16:59 |
| weshay|ruck | https://review.rdoproject.org/zuul/status | 16:59 |
| weshay|ruck | what gives? | 16:59 |
| weshay|ruck | this is weird | 16:59 |
| weshay|ruck | frenzy_friday, https://review.rdoproject.org/r/gitweb?p=rdo-jobs.git;a=blob;f=README.rst#l14 | 17:00 |
| weshay|ruck | integration-pipeline-main.yaml | 17:01 |
| frenzy_friday | I have duplicated the integration main https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/integration-pipeline-main.yaml in testproject. Now I want to make the jobs in integration line (testproject) use c8 stream nodes defined in patch https://review.rdoproject.org/r/#/c/32061/ | 17:02 |
| *** udesale has quit IRC | 17:03 | |
| frenzy_friday | something like this https://review.rdoproject.org/r/#/c/32080/ ? | 17:04 |
| weshay|ruck | https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-main | 17:04 |
| * weshay|ruck looks | 17:04 | |
| weshay|ruck | frenzy_friday, that's correct.. | 17:06 |
| weshay|ruck | is this for the first step, centos-8-stream on top of centos-8 nodes... or the second step where nodes + content are centos-8-stream | 17:07 |
| frenzy_friday | https://review.rdoproject.org/r/#/c/32041/ - stream repos on 8 node ; https://review.rdoproject.org/r/#/c/32080/ - stream repos on stream node | 17:08 |
| weshay|ruck | frenzy_friday, k.. thanks, updated https://hackmd.io/9Xve-rYpRaKbk5NMe7kukw?view | 17:10 |
| weshay|ruck | frenzy_friday, https://review.rdoproject.org/r/#/c/32080/1/.zuul.yaml | 17:10 |
| weshay|ruck | so line 62 | 17:10 |
| weshay|ruck | what is changing that node to a centos-8-stream? | 17:10 |
| weshay|ruck | frenzy_friday, I guess https://review.rdoproject.org/r/#/c/32061/2/zuul.d/nodesets.yaml | 17:12 |
| weshay|ruck | lines 394 - 410 | 17:12 |
| weshay|ruck | frenzy_friday, k.. give it a try :) | 17:13 |
| frenzy_friday | that is my question. Line 18 - I changed it to use stream node. But what about the other jobs? Do they use single-stream or ovb-stream or two-centos-stream ? | 17:13 |
| * weshay|ruck checks | 17:13 | |
| weshay|ruck | frenzy_friday, follow my links | 17:14 |
| weshay|ruck | https://review.rdoproject.org/codesearch/?q=periodic-tripleo-ci-centos-8-undercloud-containers-master&i=nope&files=&repos= | 17:14 |
| weshay|ruck | undercloud-jobs.yaml | 17:14 |
| weshay|ruck | line 8 | 17:15 |
| ysandeep|ruck | weshay|ruck, fyi.. 17 line seems hitting a legit issue https://sf.hosted.upshift.rdu2.redhat.com/logs/20/228720/4/check/periodic-tripleo-ci-rhel-8-bm_envA-3ctlr_1comp-featureset035-rhos-17/ba1273f/logs/undercloud/home/zuul/overcloud_deploy.log .. i will debug that further tomorrow incase fix is behind any component. | 17:16 |
| * weshay|ruck looking for the def of the parent | 17:16 | |
| weshay|ruck | ysandeep|ruck, k.. I'll poke in a minute.. thank you :) | 17:16 |
| weshay|ruck | have a good night | 17:16 |
| ysandeep|ruck | but i think we can merge the job defination and trigger https://code.engineering.redhat.com/gerrit/#/c/228720/ and https://code.engineering.redhat.com/gerrit/#/c/228826/ | 17:17 |
| weshay|ruck | frenzy_friday, https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/tripleo-rdo-base.yaml#L412 | 17:17 |
| weshay|ruck | https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/tripleo-rdo-base.yaml#L180 | 17:18 |
| frenzy_friday | oh! got it now, single node | 17:19 |
| weshay|ruck | frenzy_friday, https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/base.yaml#L285 | 17:19 |
| ysandeep|ruck | weshay|ruck, please add these two in your review list as well https://code.engineering.redhat.com/gerrit/#/c/226758/ and https://code.engineering.redhat.com/gerrit/#/c/226754/ | 17:20 |
| weshay|ruck | frenzy_friday, so override nodeset | 17:20 |
| *** ysandeep|ruck is now known as ysandeep|away | 17:21 | |
| weshay|ruck | what is the nodeset for centos-8-stream? | 17:21 |
| weshay|ruck | I don't see it | 17:21 |
| * ysandeep|away leaves for the day, see you tomorrow | 17:21 | |
| weshay|ruck | - name: centos-8-stream | 17:22 |
| weshay|ruck | 46 label: centos-8-stream | 17:22 |
| weshay|ruck | ysandeep|away, 0/ | 17:22 |
| weshay|ruck | name: puppet-openstack-integration-6-scenario001-tempest-centos-8-stream | 17:22 |
| weshay|ruck | 30 parent: puppet-openstack-integration-6-scenario001 | 17:22 |
| weshay|ruck | 31 nodeset: centos-8-stream | 17:22 |
| weshay|ruck | 32 voting: false | 17:22 |
| weshay|ruck | ya.. so that should work | 17:22 |
| weshay|ruck | frenzy_friday, so override the nodeset in each of those singlenode jobs | 17:22 |
| weshay|ruck | not 100% sure that's the right name.. for rdo.. but upstream and rdo should be in sync | 17:23 |
| weshay|ruck | https://review.rdoproject.org/codesearch/?q=nodeset%3A%20centos-8-stream&i=nope&files=&repos= | 17:23 |
| weshay|ruck | but that tells me we don't have anything merged w/ it | 17:24 |
| *** marios is now known as marios|out | 17:24 | |
| weshay|ruck | frenzy_friday, am I making sense? | 17:24 |
| frenzy_friday | yep. i'll search for the other jobs on codesearch, get the parents and check which nodeset they use. Then switch accordingly | 17:25 |
| frenzy_friday | No, the nodeset.yaml changes are not merged. It it workflow -1ed. But I added it as a depends on patch | 17:26 |
| weshay|ruck | frenzy_friday, ya.. note that not everything is indexed in codesearch for rdo | 17:26 |
| weshay|ruck | so.. I have all the repos checked out... in a dir | 17:26 |
| weshay|ruck | and use "egrep -rn foo *" | 17:26 |
| weshay|ruck | al ot | 17:26 |
| weshay|ruck | a lot | 17:26 |
| frenzy_friday | cool, thanks | 17:27 |
| weshay|ruck | frenzy_friday, k.. I emailed the upstream list w/ our plans | 17:32 |
| weshay|ruck | thanks for working on this for us | 17:32 |
| frenzy_friday | cool! It was fun. Also, updated the hackmd, do the patch descriptions make sense? | 17:33 |
| weshay|ruck | yes.. it's more clear now | 17:34 |
| weshay|ruck | which is why I felt comfortable sharing it | 17:34 |
| weshay|ruck | thank you | 17:34 |
| *** jmasud has quit IRC | 17:39 | |
| *** marios|out has quit IRC | 17:43 | |
| *** derekh has quit IRC | 18:00 | |
| *** dtantsur is now known as dtantsur|afk | 18:00 | |
| *** jpena is now known as jpena|off | 18:02 | |
| *** rlandy|training is now known as rlandy | 18:02 | |
| rlandy | frenzy_friday: sorry - out of training | 18:03 |
| rlandy | are you all set? | 18:03 |
| frenzy_friday | rlandy, finishing the stream repo on stream node patch | 18:08 |
| rlandy | ok | 18:09 |
| frenzy_friday | rlandy, stream repo on stream node jobs are starting https://review.rdoproject.org/r/#/c/32080/ | 18:21 |
| rlandy | frenzy_friday: weshay|ruck: reading back | 18:22 |
| rlandy | that's fin in testproject | 18:22 |
| rlandy | I think we want to look at the review that moves the line itself | 18:23 |
| rlandy | so we can restart promotions | 18:23 |
| rlandy | OMG ... periodic-tripleo-ci-rhel-8-ovb-1ctlr_2comp-featureset020-internal-rhos-16.2openstack/tripleo-cimasteropenstack-periodic-integration-rhos-16.2master4 hrs 28 mins 55 secs2021-02-23 13:04:47SUCCESS | 18:32 |
| rlandy | 2021-02-23 13:05:02.346101 | localhost | Label: upstream-rhel-8-4 | 18:33 |
| rlandy | on 8.4 | 18:33 |
| rlandy | weshay|ruck: ysandeep|away: ^^ my life is complete | 18:33 |
| rlandy | one less thing to debug | 18:33 |
| rlandy | a pass on OVB | 18:33 |
| rlandy | so excited | 18:33 |
| rlandy | periodic-tripleo-ci-rhel-8-bm_envD-3ctlr_1comp-featureset035-rhos-16.2openstack/tripleo-cimasteropenstack-periodic-integration-rhos-16.2master4 hrs 55 mins 2 secs2021-02-23 13:05:11SUCCESS | 18:33 |
| rlandy | and that | 18:33 |
| rlandy | Red Hat Enterprise Linux release 8.4 Beta (Ootpa) | 18:34 |
| rlandy | so exciting | 18:34 |
| weshay|ruck | rlandy, nice | 18:34 |
| weshay|ruck | congrats :) | 18:34 |
| rlandy | weshay|ruck: oh joy!! | 18:35 |
| rlandy | one passing bm one passing ovb | 18:35 |
| rlandy | now for the failing multinode | 18:35 |
| rlandy | weshay|ruck: legit issue there | 18:35 |
| rlandy | the undercloud/subnode-1 not ssh accessibel after deployment | 18:35 |
| rlandy | debugging | 18:35 |
| rlandy | so on BM we need to update the undercloud | 18:36 |
| rlandy | ok - task to do | 18:36 |
| rlandy | "tripleo_common.image.exception.ImageNotFoundException: Not found image: https://docker-registry.upshift.redhat.com/v2/tripleorhos-16-2/openstack-nova-libvirt/manifests/35f2fbf930506e3b5d9b8a1365b32c9b"], "stdout": "", "stdout_lines": []} | 18:38 |
| rlandy | fs001 and 035 seem like registry access issues | 18:38 |
| rlandy | Packages download failure. Overcloud stack: FAILED. Overcloud deploy failed. | 18:39 |
| rlandy | Reason: infra | 18:39 |
| rlandy | almost ... neutron_tempest_plugin.scenario.test_floatingip.FloatingIpMultipleRoutersTest 1 0 1 0 0 Detail | 18:40 |
| rlandy | test_reuse_ip_address_with_other_fip_on_other_router[id-b0382ab3-3c86-4415-84e3-649a8b040dab] | 18:40 |
| rlandy | ok - multinode | 18:40 |
| rlandy | 1 failure left on the debug list | 18:40 |
| rlandy | fatal: [subnode-1]: UNREACHABLE! => { | 18:44 |
| rlandy | "changed": false, | 18:44 |
| rlandy | "unreachable": true | 18:44 |
| rlandy | } | 18:44 |
| *** jmasud has joined #oooq | 19:05 | |
| rlandy | weshay|ruck: ^^ any thoughts off the top of your head? | 19:07 |
| rlandy | ssh unreachable after deploy | 19:08 |
| rlandy | muct have been reachable before | 19:08 |
| rlandy | must | 19:08 |
| rlandy | deploy passed | 19:08 |
| rlandy | collect logs failed | 19:08 |
| rlandy | on subnode-1 | 19:08 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-multinode-1ctlr-featureset010-rhos-16.2/b7672c3/logs/undercloud/etc/hosts looks right | 19:09 |
| *** jmasud has quit IRC | 19:10 | |
| rlandy | openvswitch-selinux-extra-policy.noarch 1.0-28.el8fdp @rhosp-rhel-8.4-fdp | 19:12 |
| rlandy | openvswitch2.13.x86_64 2.13.0-79.5.el8fdp @rhosp-rhel-8.4-fdp | 19:12 |
| rlandy | maybe openvswitch | 19:12 |
| rlandy | openvswitch-selinux-extra-policy.noarch 1.0-28.el8fdp @rhosp-rhel-8.3-fdp | 19:15 |
| rlandy | openvswitch2.13.x86_64 2.13.0-79.5.el8fdp @rhosp-rhel-8.3-fdp | 19:15 |
| rlandy | nah - same | 19:15 |
| rlandy | ha no netstat | 19:24 |
| weshay|ruck | rlandy, at somepoint I'd like to better understand why downstream can't use the same process to build images | 19:50 |
| weshay|ruck | as upstream | 19:50 |
| weshay|ruck | pita | 19:50 |
| rlandy | weshay|ruck: you mean DIB as opposed to virt-customize | 19:55 |
| rlandy | nodepool images? | 19:56 |
| rlandy | uses the same method to build overcloud images | 19:56 |
| rlandy | hmmm ... I can definitely ssh around the node before deploy | 19:59 |
| frenzy_friday | rlandy, Which repo has this ipaserver-undercloud-setup.yml ? | 20:13 |
| rlandy | tripelo-quickstart-extras | 20:13 |
| frenzy_friday | thanks | 20:13 |
| weshay|ruck | rlandy, oh.. this was an overcloud image build issue? | 20:14 |
| rlandy | actually tasks_from | 20:14 |
| rlandy | frenzy_friday:^^ sec | 20:16 |
| *** sanjayu_ has quit IRC | 20:16 | |
| rlandy | I'm wrong | 20:16 |
| rlandy | it calls that | 20:16 |
| rlandy | getting | 20:16 |
| frenzy_friday | no, I got the yml in quickstart extras | 20:16 |
| frenzy_friday | testproject threw an error : The task includes an option with an undefined variable. The error was: "hostvars['subnode-1']" is undefined | 20:17 |
| rlandy | oh ok | 20:17 |
| rlandy | weshay|ruck: ? | 20:17 |
| rlandy | mutinode deployment | 20:18 |
| weshay|ruck | you mentioned netstat was missing from an image? | 20:18 |
| rlandy | it's missing in the logs | 20:18 |
| rlandy | I have a run going now where I installed netstat on both nodes manually | 20:19 |
| rlandy | want to see if that helps | 20:19 |
| weshay|ruck | right.. so I'll reask the question | 20:19 |
| rlandy | idk what caused it yet exactly | 20:19 |
| weshay|ruck | why are the rhel nodes in internal sf built w/ a different process than upstream | 20:19 |
| weshay|ruck | why are we finding missing packages | 20:20 |
| rlandy | weshay|ruck: idk | 20:20 |
| weshay|ruck | what is the root cause | 20:20 |
| weshay|ruck | seems like if we can align in the how they are built | 20:20 |
| weshay|ruck | we can avoid chasing this kind of a thing down | 20:20 |
| rlandy | one uses DIB and the other virt-customize | 20:20 |
| weshay|ruck | ya | 20:20 |
| weshay|ruck | why.. we should poke at it | 20:20 |
| rlandy | idk why there is that diff tbh | 20:21 |
| rlandy | it's not always the nodepool build process | 20:21 |
| weshay|ruck | but you are spending time working around it | 20:21 |
| rlandy | it can be the base guest image | 20:21 |
| rlandy | between rhel 8.3 and 8.4 there is not much diff in the nodepool virt0customize file | 20:22 |
| rlandy | weshay|ruck: however if this works, I will add netstat to the nodepool image | 20:29 |
| *** frenzy_friday is now known as frenzyfriday | 20:37 | |
| *** frenzyfriday is now known as frenzyfriday|123 | 20:42 | |
| *** frenzyfriday|123 is now known as frenzyfriday|brb | 20:42 | |
| *** slaweq has quit IRC | 20:50 | |
| rlandy | [zuul@upstream-rhel-8-4-tripleo-ci-0000204323 ~]$ ssh zuul@192.168.100.171 -i /etc/nodepool/id_rsa | 20:55 |
| rlandy | it can - why does it say it can't | 20:55 |
| rlandy | weshay|ruck: ugh - same story | 21:08 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/95/200295/56/check/periodic-tripleo-ci-rhel-8-multinode-1ctlr-featureset010-rhos-16.2/247ac75/logs/ | 21:08 |
| rlandy | unreachable after deploy | 21:08 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/95/200295/56/check/periodic-tripleo-ci-rhel-8-multinode-1ctlr-featureset010-rhos-16.2/247ac75/logs/undercloud/var/log/extra/netstat.txt exists now | 21:11 |
| weshay|ruck | rlandy, right.. so openvswitch is probably the issue | 21:14 |
| rlandy | same version of openvswitch | 21:14 |
| *** frenzyfriday|brb is now known as frenzy_friday | 21:41 | |
| rlandy | weshay|ruck: k - I am trying another run ... but decision to promote what we have? | 22:03 |
| rlandy | I'm inclined to say no | 22:03 |
| rlandy | because then we will switch the component lines | 22:03 |
| rlandy | and that will alos fail multinode if we have not worke dit out yet | 22:03 |
| weshay|ruck | rlandy, the only thing that is a concern is 035 | 22:04 |
| rlandy | fs035? | 22:04 |
| weshay|ruck | ya | 22:04 |
| weshay|ruck | oh wait | 22:04 |
| weshay|ruck | the bm is fs035 \0/ | 22:05 |
| rlandy | fs010 | 22:05 |
| weshay|ruck | rlandy, let's chat | 22:05 |
| weshay|ruck | rlandy, no.. don't car | 22:05 |
| weshay|ruck | e | 22:05 |
| weshay|ruck | I'll tell you why | 22:05 |
| rlandy | ok | 22:05 |
| weshay|ruck | https://meet.google.com/fer-yxhm-xyo?authuser=1 | 22:05 |
| *** jmasud has joined #oooq | 22:06 | |
| *** jmasud has quit IRC | 22:37 | |
| rlandy | weshay|ruck: https://code.engineering.redhat.com/gerrit/228873 Move release files for 16.2 to rhel 8.4 | 22:52 |
| rlandy | sec - moving nodesets to 8.4 for component jobs | 22:52 |
| weshay|ruck | shall I merge? | 22:55 |
| rlandy | weshay|ruck: not yet | 22:56 |
| rlandy | weshay|ruck: I can't move the node label as 17 is still 8.3 | 22:56 |
| rlandy | doing a nodeset patch job | 22:56 |
| rlandy | couple minutes | 22:57 |
| weshay|ruck | rlandy, k.. I'm back and forth w/ kids.. | 22:57 |
| weshay|ruck | I'll keep checking back | 22:57 |
| rlandy | weshay|ruck: no worries - I'm here for a while | 22:57 |
| rlandy | weshay|ruck: https://code.engineering.redhat.com/gerrit/228875 Move component 16.2 jobs to rhel 8.4 | 23:16 |
| *** jmasud has joined #oooq | 23:17 | |
| *** jmasud_ has joined #oooq | 23:52 | |
| *** jmasud has quit IRC | 23:54 | |
| rlandy | weshay|ruck: think it's the virt: av module | 23:58 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!