| *** rf0lc0 has quit IRC | 00:22 | |
| *** Goneri has quit IRC | 01:06 | |
| *** ysandeep|away is now known as ysandeep | 03:16 | |
| *** udesale has joined #oooq | 03:31 | |
| *** skramaja has joined #oooq | 03:35 | |
| *** udesale has quit IRC | 04:08 | |
| *** jjoyce has quit IRC | 04:33 | |
| *** ratailor has joined #oooq | 04:36 | |
| *** fuzzball81 has joined #oooq | 04:52 | |
| bhagyashris | zbr, Good morning ! containers build successfully https://review.rdoproject.org/r/#/c/29917/ | 04:54 |
|---|---|---|
| bhagyashris | \0/ | 04:54 |
| *** ykarel has joined #oooq | 04:55 | |
| *** marios has joined #oooq | 05:56 | |
| *** amoralej|off is now known as amoralej | 06:02 | |
| *** apetrich has joined #oooq | 06:34 | |
| *** holser has joined #oooq | 06:52 | |
| *** skramaja has quit IRC | 06:53 | |
| *** skramaja has joined #oooq | 06:54 | |
| sshnaidm | chandankumar, do we need to pass ceph repos to kolla? we don't build ceph containers | 07:13 |
| *** jpena|off is now known as jpena | 07:32 | |
| *** jtomasek_ has joined #oooq | 07:44 | |
| *** jtomasek has quit IRC | 07:45 | |
| *** tosky has joined #oooq | 07:54 | |
| *** jtomasek_ has quit IRC | 07:56 | |
| chandankumar | sshnaidm: I am not sure, But I checked it the tripleo-repos tools there it populates that's why added there | 07:57 |
| sshnaidm | chandankumar, it failed last time, I'm trying now w/o ceph repos | 07:57 |
| chandankumar | sshnaidm: ok | 07:58 |
| *** holser has quit IRC | 07:58 | |
| ykarel | sshnaidm, ceph repos are needed for some openstack containers | 07:59 |
| ykarel | like nova-compute,cinder-volume,glance-api etc which work with ceph backend | 07:59 |
| sshnaidm | ykarel, ack. | 07:59 |
| sshnaidm | chandankumar, so need to understand why ceph repo is not downloaded, it has html error inside instead of repo file | 08:00 |
| ykarel | mainly for ceph client packages like ceph-common,rbd,rados etc | 08:00 |
| *** jtomasek has joined #oooq | 08:00 | |
| *** holser has joined #oooq | 08:00 | |
| *** jtomasek has quit IRC | 08:00 | |
| zbr | ykarel: thanks for this fix | 08:01 |
| chandankumar | sshnaidm: is it coming for both train and queens? | 08:01 |
| sshnaidm | chandankumar, yep | 08:12 |
| sshnaidm | chandankumar, maybe the repo renamed after? idk | 08:12 |
| chandankumar | ykarel: for train, I picked it from buildah container build | 08:12 |
| chandankumar | but need to check one more time for actual jobs | 08:13 |
| chandankumar | that's why i cleaned up most of the repos | 08:13 |
| chandankumar | ykarel: it was for sshnaidm | 08:13 |
| ykarel | ack | 08:13 |
| sshnaidm | chandankumar, I see currently it fails because kolla can't download ceph repo | 08:14 |
| sshnaidm | chandankumar, and no idea why.. | 08:14 |
| chandankumar | time to reproduce locally then | 08:14 |
| chandankumar | sshnaidm: i think i got the issue | 08:21 |
| chandankumar | sshnaidm: https://logserver.rdoproject.org/openstack-periodic-integration-stable2-centos7/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-train-containers-build-push/bb3be86/etc/yum.repos.d/ | 08:21 |
| chandankumar | and https://logserver.rdoproject.org/openstack-periodic-integration-stable4-5/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-queens-containers-build/d39fdf0/logs/undercloud/etc/yum.repos.d/ | 08:21 |
| chandankumar | tripleo-centos and quickstart-centos | 08:21 |
| chandankumar | we need to figure out the correct repo names and we are done there I think | 08:22 |
| sshnaidm | chandankumar, oh, I see | 08:22 |
| chandankumar | that's why it is giiving above issue | 08:22 |
| chandankumar | i need to take a look at look at release for queens and train and update the review after meeting | 08:23 |
| *** derekh has joined #oooq | 08:35 | |
| chandankumar | zbr: morning, https://projects.engineering.redhat.com/browse/RHOSZUUL-170 now fixed by jpena , please have a look thanks jpena :-) | 08:42 |
| *** ysandeep is now known as ysandeep|lunch | 08:55 | |
| *** dtantsur|afk is now known as dtantsur | 09:05 | |
| *** johnsom has quit IRC | 09:25 | |
| *** johnsom has joined #oooq | 09:25 | |
| *** ysandeep|lunch is now known as ysandeep | 09:27 | |
| zbr | bhagyashris: ok to abandon https://review.opendev.org/#/c/756646/ ? | 09:45 |
| bhagyashris | zbr, on ^ patch the standalone job is passing now https://review.rdoproject.org/r/#/c/29917/ but this is not the right way to do chandankumar has given me some pointers to fix this so now working on it | 09:46 |
| bhagyashris | zbr, but yeah will need to abandon that patch once we have the correct fix up | 09:47 |
| bhagyashris | currently working on it | 09:47 |
| bhagyashris | give me some time | 09:47 |
| sshnaidm | marios, dlrn tag passing doesn't work now, fixing it.. | 09:48 |
| marios | sshnaidm: ack, had some progress on the upgrade jobs, still few nits working today and we can sync later | 09:49 |
| marios | sshnaidm: chandankumar: something to think about for later please see comments there https://review.opendev.org/#/c/755472/3/config/release/tripleo-ci/CentOS-8/master.yml@23 https://review.opendev.org/#/c/753708/12/roles/standalone/templates/standalone_config.yaml.j2@18 | 09:49 |
| marios | sshnaidm: yeah i see what you mean provider_dlrn_hash_tag_{{ ci_branch }}: '' | 09:50 |
| marios | sshnaidm: so actually i have the same issue with registry_ip_address_{{ ci_branch }}: 104.130.239.154 | 09:51 |
| marios | sshnaidm: i think my comment i pointed at ^^^ is related i.e. about jinja templating | 09:51 |
| sshnaidm | marios, commented there, it should be specific variable for each release file | 09:52 |
| sshnaidm | marios, master will have registry_ip_address_master | 09:52 |
| sshnaidm | etc | 09:52 |
| marios | sshnaidm: ack thinking about it /how it works during the upgrade workflow... ok we can discuss more on the phone | 09:53 |
| marios | sshnaidm: maybe its ok | 09:53 |
| sshnaidm | marios, ok, I think we'd better to change variable structure there as: http://paste.openstack.org/show/798832/ | 10:04 |
| sshnaidm | marios, then you'll need to check like: if 'ussuri' in registry_ip_address_branch ..... use registry_ip_address_branch[release] | 10:05 |
| marios | sshnaidm: yeah i see... so we can always check the constant key | 10:05 |
| sshnaidm | marios, yeah, and instead of dynamic variable just need to take dictionary value | 10:06 |
| marios | sshnaidm: ack, i will copy paste into my next run when i update https://review.opendev.org/#/c/753403/16/playbooks/tripleo-ci/run-provider.yml | 10:06 |
| marios | sshnaidm: i'll check your review | 10:06 |
| marios | sshnaidm: (when you update i mean) | 10:07 |
| *** dsneddon has quit IRC | 10:09 | |
| *** dsneddon has joined #oooq | 10:10 | |
| zbr | bhagyashris: are we ok to merge https://review.rdoproject.org/r/#/c/29915/ ? | 10:28 |
| sshnaidm | chandankumar, seems like quickstart-centos-opstools.repo has the same problem, should it be tripleo-centos-opstools.repo? | 10:32 |
| chandankumar | sshnaidm: I can switch to that for both and see | 10:32 |
| chandankumar | sshnaidm: for queens this time it is file:///etc/yum.repos.d/quickstart-centos-ceph-luminous.repo, line: 1 | 10:34 |
| chandankumar | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a99/752421/51/check/tripleo-ci-centos-7-content-provider-queens/a991dfe/logs/undercloud/home/zuul/workspace/build.log | 10:34 |
| chandankumar | repo is here https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a99/752421/51/check/tripleo-ci-centos-7-content-provider-queens/a991dfe/logs/undercloud/etc/yum.repos.d/ | 10:37 |
| chandankumar | let me check one of the queens build container job where it was successful | 10:38 |
| ykarel | chandankumar, i didn't checked the patches, are you injecting the repos explicitly? | 10:39 |
| ykarel | so kolla should be setting up ceph repos by it's own from extras repos | 10:39 |
| chandankumar | ykarel: https://review.opendev.org/752419 and https://review.opendev.org/752421 | 10:39 |
| ykarel | similar should be true for opstools | 10:39 |
| chandankumar | ykarel: so we donot need those ? | 10:40 |
| chandankumar | ykarel: for train | 10:40 |
| ykarel | you can inject additional repos but i think for that name should be start from delorean as that's what container build role expect to server from http | 10:40 |
| chandankumar | https://review.opendev.org/#/c/752419/29/playbooks/multinode-standalone-pre.yml@84 | 10:40 |
| ykarel | chandankumar, yes try without atleast ceph and opstools | 10:41 |
| ykarel | if u see logs u will see yum -y install centos-release-ceph-luminous centos-release-openstack-queens centos-release-opstool | 10:41 |
| ykarel | ^ pulled from extras | 10:41 |
| ykarel | btw why you added those, was it requrired earlier? | 10:42 |
| chandankumar | ykarel: we were seeing missing packages so added there | 10:42 |
| chandankumar | ykarel: in case of train, we need those repos? | 10:42 |
| ykarel | chandankumar, shouldn't be needed i think for train too, but good to check container build logs for train | 10:43 |
| ykarel | i mean periodic | 10:43 |
| ykarel | https://logserver.rdoproject.org/openstack-periodic-integration-stable2-centos7/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-train-containers-build-push/bb3be86/conf/kolla-build.conf | 10:44 |
| ykarel | not passed | 10:44 |
| ykarel | https://logserver.rdoproject.org/openstack-periodic-integration-stable2-centos7/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-train-containers-build-push/bb3be86/logs/httpd_data/ | 10:45 |
| ykarel | train too have those tasks:- yum -y install centos-release-nfs-ganesha28 centos-release-openstack-train centos-release-opstools centos-release-ceph-nautilus centos-release-qemu-ev yum-plugin-priorities centos-release-opstools | 10:46 |
| ykarel | in base conatiner | 10:46 |
| ykarel | so if you have logs where you found missing packages, i can check what missed there | 10:47 |
| *** ratailor has quit IRC | 10:58 | |
| *** ratailor has joined #oooq | 11:02 | |
| bhagyashris | zbr, yes we are ok to merge this https://review.rdoproject.org/r/#/c/29915/ and test patch is also green https://review.rdoproject.org/r/#/c/29917/ | 11:12 |
| *** ratailor has quit IRC | 11:12 | |
| bhagyashris | zbr, i have created separate patch for standalone, scenarios and fs001 jobs https://review.rdoproject.org/r/#/c/29966/ | 11:12 |
| *** jpena is now known as jpena|lunch | 11:37 | |
| *** Goneri has joined #oooq | 11:37 | |
| marios | sshnaidm: chandankumar: is one of you already working on this file? https://opendev.org/openstack/tripleo-quickstart-extras/src/commit/5728258f832dfd6d4eceffd6473dfd555d857467/roles/overcloud-prep-containers/templates/overcloud-prep-containers.sh.j2#L34 i am looking for overcloud upgrade job so we can set insecure registry forcontent provider registry | 11:41 |
| marios | sshnaidm: chandankumar: nm.. i see it already has something there --namespace "{{ job.registry_ip_address }}:{{ local_registry_port }}/{{ docker_registry_namespace_used }}" \ | 11:42 |
| chandankumar | marios: I fixed that one | 11:43 |
| marios | chandankumar: ack cool | 11:43 |
| chandankumar | marios: https://review.opendev.org/#/c/753406/ | 11:43 |
| marios | thanks chandankumar | 11:44 |
| marios | chandankumar: i even voted on it ;) | 11:44 |
| chandankumar | :-) | 11:44 |
| *** amoralej is now known as amoralej|lunch | 11:57 | |
| *** jtomasek has joined #oooq | 11:58 | |
| *** rlandy has joined #oooq | 12:01 | |
| *** rlandy is now known as rlandy|rover | 12:01 | |
| *** jtomasek_ has joined #oooq | 12:06 | |
| *** jtomasek has quit IRC | 12:07 | |
| rlandy|rover | soniya29|ruck: hi ... | 12:08 |
| rlandy|rover | how's it going today? | 12:08 |
| weshay | rlandy|rover, soniya29|ruck 0/ | 12:08 |
| rlandy|rover | weshay: hey | 12:08 |
| rlandy|rover | weshay: looking at master failures | 12:09 |
| weshay | aye | 12:09 |
| rlandy|rover | weshay: you were right about mistral | 12:09 |
| rlandy|rover | test scen003 w/o it in THT - worked | 12:09 |
| rlandy|rover | idk if we can remove it yet - will that fail check? | 12:09 |
| rlandy|rover | make sc003 non-voting in check, remove mistral and get promotion to pass | 12:10 |
| rlandy|rover | weshay: ^^?? | 12:10 |
| weshay | rlandy|rover, we need to verify this w/ df folks | 12:10 |
| rlandy|rover | ack | 12:10 |
| rlandy|rover | also stein still failing - should have been fixed - will check that next | 12:11 |
| rlandy|rover | ussuri also read | 12:11 |
| rlandy|rover | red | 12:11 |
| rlandy|rover | image server | 12:12 |
| weshay | soniya29|ruck, ping | 12:13 |
| rlandy|rover | weshay: soniya29|ruck: we have image download and image build failures | 12:14 |
| rlandy|rover | talking with rhos-ops | 12:14 |
| weshay | rlandy|rover, I think you can remove the DNM | 12:18 |
| rlandy|rover | sec - checking images | 12:20 |
| rlandy|rover | still have image build problem | 12:21 |
| *** rf0lc0 has joined #oooq | 12:22 | |
| chandankumar | zbr: weshay: rlandy|rover https://pypi.org/project/zuul-client/ | 12:23 |
| rlandy|rover | nice :) | 12:24 |
| rlandy|rover | would have helped a lot earlier | 12:24 |
| chandankumar | weshay: https://review.opendev.org/#/c/756128/ it need some changes | 12:25 |
| weshay | chandankumar, that seems like it's geared towards admins | 12:25 |
| rlandy|rover | 2020-10-08 10:54:40.493 | CentOS-8 - AppStream 0.0 B/s | 0 B 00:30 | 12:25 |
| rlandy|rover | 2020-10-08 10:54:40.493 | Failed to download metadata for repo 'AppStream' | 12:25 |
| rlandy|rover | 2020-10-08 10:54:40.496 | Error: Failed to download metadata for repo 'AppStream' | 12:25 |
| weshay | chandankumar, should I remove the wf? the test patch in rdo from rf0lc0 worked fined | 12:26 |
| weshay | fine | 12:26 |
| chandankumar | weshay: yes please | 12:26 |
| chandankumar | rf0lc0: please have a look https://review.opendev.org/#/c/756128/ when free, thanks :-) | 12:26 |
| soniya29|ruck | rlandy|rover, weshay hello | 12:27 |
| weshay | soniya29|ruck, can you please sync w/ rlandy|rover and split up the debug work on the periodic jobs? | 12:28 |
| soniya29|ruck | weshay ssure | 12:29 |
| soniya29|ruck | s/ssure/sure | 12:29 |
| weshay | soniya29|ruck, thanks | 12:29 |
| rlandy|rover | sec - updated scneario003 patch | 12:31 |
| marios | chandankumar: joining us? | 12:32 |
| chandankumar | marios: yes | 12:32 |
| marios | sshnaidm: https://review.opendev.org/753403 & https://review.opendev.org/753708 & https://review.opendev.org/755472 | 12:32 |
| *** jpena|lunch is now known as jpena | 12:32 | |
| sshnaidm | chandankumar, weshay mtg | 12:32 |
| marios | https://review.opendev.org/#/c/756517/3 | 12:33 |
| rlandy|rover | fatal: unable to access 'https://github.com/redhat-openstack/rdoinfo/': Failed to connect to github.com port 443: Connection timed out | 12:36 |
| rlandy|rover | soniya29|ruck: maybe we had some sort of connection outage on rdocloud ... ^^ a lot of connection errors | 12:38 |
| rlandy|rover | downloading images | 12:38 |
| rlandy|rover | soniya29|ruck: we do have a real image build failure | 12:38 |
| rlandy|rover | and a real scenario003 failure https://bugs.launchpad.net/tripleo/+bug/1898931 | 12:39 |
| openstack | Launchpad bug 1898931 in tripleo "periodic-tripleo-ci-centos-8-scenario003-standalone-master is failing standalone deploy "Service mistral_executor has not started yet"" [Critical,In progress] - Assigned to Ronelle Landy (rlandy) | 12:39 |
| rlandy|rover | fix in progress there | 12:39 |
| rlandy|rover | I think if we reran the ussuri failing jobs - we may get a pass | 12:40 |
| rlandy|rover | soniya29|ruck: have you ever used testproject? | 12:40 |
| soniya29|ruck | rlandy|rover nope | 12:41 |
| rlandy|rover | soniya29|ruck: k - will send you a meeting invite in a few to explain | 12:41 |
| soniya29|ruck | rlandy|rover, okay | 12:42 |
| ysandeep | rlandy|rover weshay hey o/ when you have time could you please merge https://code.engineering.redhat.com/gerrit/#/c/214531/ , We finally got green for multinode fs010 in integration line after new podman. | 12:43 |
| marios | https://docs.ansible.com/ansible/latest/reference_appendices/faq.html#when-should-i-use-also-how-to-interpolate-variables-or-dynamic-variable-names | 12:44 |
| ysandeep | weshay: thanks | 12:45 |
| marios | * https://review.opendev.org/#/c/756128/6 | 12:46 |
| rlandy|rover | ysandeep: sure - after we deal with all these broken upstream jobs :) | 12:50 |
| ysandeep | rlandy|rover: ignore, wes merged it , thanks! | 12:51 |
| rlandy|rover | great | 12:51 |
| marios | https://review.opendev.org/#/c/756128/6/zuul.d/base.yaml | 12:53 |
| *** amoralej|lunch is now known as amoralej | 12:55 | |
| rlandy|rover | soniya29|ruck: sorry - got distracted ... https://meet.google.com/ozd-mtfs-cfa | 12:57 |
| chandankumar | sshnaidm: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6b3/752421/51/check/tripleo-ci-centos-7-content-provider-queens/6b35c49/logs/undercloud/home/zuul/workspace/build.log | 12:59 |
| chandankumar | and https://api.us-east.open-edge.io:8080/swift/v1/AUTH_e02c11e4e2c24efc98022353c88ab506/zuul_opendev_logs_e0f/752421/51/check/tripleo-ci-centos-7-content-provider-train/e0fcf2a/logs/undercloud/home/zuul/workspace/build.log | 12:59 |
| chandankumar | for train and queens now we are stuck at repos | 12:59 |
| chandankumar | https://api.us-east.open-edge.io:8080/swift/v1/AUTH_e02c11e4e2c24efc98022353c88ab506/zuul_opendev_logs_e0f/752421/51/check/tripleo-ci-centos-7-content-provider-train/e0fcf2a/logs/undercloud/home/zuul/workspace/containers-consolidated-builds.log | 13:00 |
| chandankumar | ceph repos in both cases | 13:00 |
| chandankumar | sshnaidm: let's remove those repos and see ? | 13:00 |
| chandankumar | no ceph repos | 13:00 |
| chandankumar | let kolla handle it | 13:00 |
| weshay | mtg arxcruz, rfolco, zbr, sshnaidm, rlandy, marios, ysandeep, bhagyashris, svyas, pojadhav, akahat, weshay, chandankumar | 13:02 |
| weshay | rlandy|rover, soniya29|ruck ignore | 13:02 |
| chandankumar | ok joining | 13:02 |
| sshnaidm | chandankumar, ok | 13:03 |
| weshay | rf0lc0, you coming? | 13:04 |
| rf0lc0 | weshay, yes sec | 13:04 |
| weshay | zbr, get ur azz in here | 13:04 |
| *** TrevorV has joined #oooq | 13:05 | |
| sshnaidm | chandankumar, I have a node with failed train job | 13:06 |
| sshnaidm | chandankumar, send me your key | 13:06 |
| chandankumar | sshnaidm: https://github.com/raukadah.keys | 13:06 |
| chandankumar | sshnaidm: i am in | 13:19 |
| chandankumar | sshnaidm: cd cd /var/www/html/ | 13:19 |
| chandankumar | sshnaidm: delorean-current.repo delorean.repo delorean-train-build-deps.repo delorean-train-deps.repo | 13:19 |
| chandankumar | only these repos exists | 13:19 |
| chandankumar | rest is not there | 13:20 |
| chandankumar | so it is not there | 13:20 |
| sshnaidm | chandankumar, how repos are got there? | 13:22 |
| sshnaidm | who does put them in this dir | 13:22 |
| chandankumar | figuring out | 13:23 |
| chandankumar | sshnaidm: https://opendev.org/openstack/tripleo-ci/src/commit/331bd37b25414826409d358b4fde79833dfbad4c/roles/build-containers/tasks/pre.yaml#L48 | 13:25 |
| chandankumar | https://opendev.org/openstack/tripleo-ci/src/commit/331bd37b25414826409d358b4fde79833dfbad4c/roles/build-containers/tasks/pre.yaml#L65 | 13:25 |
| chandankumar | find /etc/yum.repos.d/ -name delorean\* | xargs -I {} cp {} /var/www/html/ | 13:25 |
| chandankumar | it is hardcoded | 13:25 |
| chandankumar | that's why repos are not there | 13:25 |
| sshnaidm | chandankumar, I see.. | 13:25 |
| chandankumar | let me fix that | 13:27 |
| soniya29|ruck | weshay rlandy|rover re-runed ussuri failed periodic jobs.. | 13:31 |
| rlandy|rover | soniya29|ruck: thanks - let's watch those for the connect/image download issues | 13:32 |
| weshay | https://hackmd.io/vRMVeZXZRgK5Vxqi6ENUDg?both#Documentation | 13:33 |
| rlandy|rover | soniya29|ruck: ok- we still don't have the fix in stein tripleo-ci-testing yet | 13:34 |
| rlandy|rover | see https://github.com/openstack/neutron/commits/stable/stein | 13:34 |
| rlandy|rover | https://trunk.rdoproject.org/centos7-stein/tripleo-ci-testing/ has [ ]openstack-neutron-14.3.2-0.20200922014701.db62fc5.el7.noarch.rpm2020-09-22 01:49 21K | 13:34 |
| rlandy|rover | https://trunk.rdoproject.org/centos7-stein/consistent/ has openstack-neutron-14.4.0-0.20201007174629.b3582e1.el7.noarch.rpm2020-10-07 17:48 21K | 13:35 |
| rlandy|rover | which is the right version | 13:35 |
| rlandy|rover | the next run should run ... | 13:35 |
| soniya29|ruck | rlandy|rover: then we need to wait for next run? | 13:36 |
| rlandy|rover | yep | 13:36 |
| rlandy|rover | unless weshay will kindly retrigger the line for us ... | 13:36 |
| weshay | ya.. sec.. which line? | 13:36 |
| rlandy|rover | weshay: pretty pls - when you get off the call ... openstack-periodic-integration-stable3 | 13:36 |
| rlandy|rover | testprojects are looking better | 13:38 |
| rlandy|rover | soniya29|ruck: ^^ | 13:38 |
| rlandy|rover | so far :) | 13:38 |
| soniya29|ruck | rlandy|rover yes | 13:38 |
| rf0lc0 | chandankumar, still presenting | 13:44 |
| ysandeep | rlandy|rover, weshay we are all set to flip selinux to enforcing mode for standalone jobs - Please review https://code.engineering.redhat.com/gerrit/#/c/211492/ , test runs are all green for both 16.2 and 17(tested with component line jobs as integration line is busted for 17) | 13:46 |
| weshay | ysandeep, have you launched any test jobs against that? | 13:46 |
| weshay | yup https://code.engineering.redhat.com/gerrit/#/c/211494/ | 13:46 |
| weshay | ysandeep++ | 13:47 |
| weshay | ysandeep, fs001? | 13:47 |
| ysandeep | weshay, that's already have selinux enabled | 13:47 |
| weshay | ah ya.. you told me that | 13:47 |
| weshay | k | 13:47 |
| weshay | rlandy|rover, soniya29|ruck https://code.engineering.redhat.com/gerrit/#/c/211492/ | 13:48 |
| rlandy|rover | marios: weshay: could we get a second vote/w+ on https://review.opendev.org/#/c/756590/ to clear up scenario003 in master promotions .. commit meassage has passed testproject link - check scenario003 test passed in first patch run | 13:48 |
| weshay | ysandeep, /me notes.. debugging deployment failures caused by selinux is kind of a pita.. | 13:48 |
| ysandeep | weshay, yeah but hopefully now we make customer and field engineers life little easier | 13:50 |
| rlandy|rover | weshay: ysandeep: we also need to remove permissive from HW envs .. http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/hardware_environments/dell_fc430_envB/network_configs/single_nic_vlans/env_settings.yml#n79 | 13:50 |
| weshay | arxcruz++ | 13:50 |
| rlandy|rover | ^^ example | 13:50 |
| ysandeep | rlandy|rover, weshay i will but its already overridden , bm jobs already have selinux in enforcing - https://sf.hosted.upshift.rdu2.redhat.com/logs/70/210870/5/check/periodic-tripleo-ci-rhel-8-bm_envB-3ctlr_1comp-featureset001-baremetal-rhos-16.2/5f66c09/logs/overcloud-controller-0/var/log/extra/selinux.txt | 13:51 |
| marios | rlandy|rover: ack (wes already voted there) | 13:51 |
| rlandy|rover | marios; thanks | 13:51 |
| weshay | sshnaidm, hey.. need to chat w/ you for 3min | 13:53 |
| weshay | sshnaidm, https://meet.google.com/htp-pqxj-vei?authuser=1 | 13:53 |
| arxcruz | weshay: ? | 13:54 |
| weshay | arxcruz, for the rewards | 13:54 |
| arxcruz | weshay: ahh ;) | 13:54 |
| arxcruz | weshay: is it possible to send rewards for marios or just a e-thankyou ? | 13:55 |
| chandankumar | sshnaidm: we can remove the hold from that vm | 14:03 |
| weshay | arxcruz, just e-thankyou | 14:07 |
| sshnaidm | chandankumar, if this works, maybe better to handle quickstart-centos\* in our provider/consumer part, I'm not sure if it doesn't break other stuff in build containers jobs | 14:07 |
| sshnaidm | chandankumar, or maybe to make it as parameter to pass, actually it's just copying files | 14:08 |
| sshnaidm | chandankumar, let's see though | 14:08 |
| chandankumar | yup | 14:08 |
| marios | arxcruz: :) thanks why? reading back | 14:13 |
| *** Goneri has quit IRC | 14:15 | |
| rlandy|rover | k - periodics are back in business | 14:15 |
| *** Goneri has joined #oooq | 14:24 | |
| soniya29|ruck | rlandy|rover, yeah :) | 14:28 |
| ysandeep | rlandy|rover, weshay https://code.engineering.redhat.com/gerrit/#/c/211492/ - updated env_settings.yml for sanity too.. but bm jobs already had selinux in enforcing. | 14:32 |
| ysandeep | thank you for reviews :) | 14:35 |
| ysandeep | weshay: we promoted 16.2 yesterday but graphana dashboard missed that, not sure what went wrong http://tripleo-cockpit.usersys.redhat.com/d/KyHCwLHMk/rhos-16-2-full-component-pipeline | 14:40 |
| ysandeep | we can promote again today.. all jobs passed including ovb | 14:41 |
| rlandy|rover | soniya29|ruck: weshay: we have a rise in gate failures - timeouts on vexxhost | 14:42 |
| rlandy|rover | ysandeep: yes - we should promote today ... unfortunately the hash that passed OVB is no longer tripleo-ci-testing | 14:43 |
| rlandy|rover | OVB is still running now | 14:43 |
| rlandy|rover | if it passes, we should promote this one | 14:43 |
| rlandy|rover | weshay: ^^ we need a way downstream to promote a particular hash | 14:44 |
| rlandy|rover | not just tripleo-ci-testing | 14:44 |
| * rlandy|rover looks at script | 14:44 | |
| ysandeep | ahh.. ovb passed.. multinode is running (which we added today) | 14:44 |
| rlandy|rover | periodic-tripleo-ci-rhel-8-multinode-1ctlr-featureset010-rhos-16.2 | 14:44 |
| rlandy|rover | sorry - 10 is still running | 14:44 |
| rlandy|rover | ysandeep: yeah - so we just added that today | 14:45 |
| rlandy|rover | so it's our call if that fails | 14:45 |
| rlandy|rover | what we do with the hash | 14:45 |
| rlandy|rover | let's see | 14:45 |
| chandankumar | One query, Does RHOS-16.2 integration pipeline jobs are triggered manaully? or we are running failed jobs using test project? | 14:46 |
| soniya29|ruck | rlandy|rover yeah..gate failures have increased | 14:46 |
| ysandeep | i already had a green run from today for that job.. so i am hopefull that will pass | 14:46 |
| ysandeep | chandankumar, integration pipeline autotriggers.. we sometime run testproject on failed jobs | 14:47 |
| chandankumar | ysandeep: ok, Does internal promoter working fine also? | 14:48 |
| rlandy|rover | primary | "success": true, | 14:48 |
| ysandeep | chandankumar, no.. rlandy|rover developed a script to do the promotions | 14:49 |
| rlandy|rover | ysandeep: ^^ let's do this | 14:49 |
| chandankumar | ysandeep: ah, ok, thanks :-) | 14:49 |
| rlandy|rover | actually it's weshay's script | 14:49 |
| rlandy|rover | ysandeep: pls join me on internal promoter | 14:50 |
| rlandy|rover | tmux | 14:50 |
| *** skramaja has quit IRC | 14:50 | |
| ysandeep | rlandy|rover, coming | 14:50 |
| weshay | rlandy|rover, ysandeep nice | 14:50 |
| rlandy|rover | weshay: we're legit promoting today | 14:51 |
| rlandy|rover | I didn't cheat | 14:51 |
| weshay | rlandy|rover, I see :) | 14:51 |
| weshay | rlandy|rover, pretty big swing in the job time 4 hrs 10 mins 7 secs | 14:52 |
| weshay | https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-rhos-16.2 | 14:52 |
| rlandy|rover | weshay: right - just when we massively increase the time | 14:53 |
| weshay | heh.. still a good time to increase it | 14:53 |
| rlandy|rover | soniya29|ruck: can you check into the failures in the gate on openstack/ansible-role-collect-logs | 14:57 |
| rlandy|rover | we have two jobs there failing tox | 14:57 |
| soniya29|ruck | rlandy|rover okay | 14:57 |
| *** ykarel is now known as ykarel|away | 14:57 | |
| ysandeep | I am out, see you tomorrow guys o/ | 15:04 |
| *** ysandeep is now known as ysandeep|away | 15:04 | |
| rf0lc0 | akahat, suggestion to reduce vars block in https://review.opendev.org/#/c/756568 | 15:07 |
| rf0lc0 | akahat, pls see my comment | 15:07 |
| rf0lc0 | akahat, this applies to any config block that you want to reuse | 15:08 |
| marios | sshnaidm: rlandy|rover: any idea why it doesn't kick the jobs? does depends-on work here? https://code.engineering.redhat.com/gerrit/#/c/214569/ can't see it in https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status does it take a while to kick ? | 15:12 |
| rlandy|rover | marios: idk if it will pick up templates | 15:13 |
| sshnaidm | marios, why not in rdo testproject? | 15:13 |
| sshnaidm | chandankumar, queens succeeded! train still fails https://review.rdoproject.org/r/#/c/29921/ | 15:14 |
| chandankumar | sshnaidm: same issue? | 15:15 |
| weshay | sshnaidm, chandankumar nice.. on queens | 15:15 |
| sshnaidm | chandankumar, hmm.. it's still configured with tripleo-centos-ceph-nautilus.repo and need to change to quickstart-centos-ceph-nautilus.repo | 15:16 |
| chandankumar | ok updating the patch | 15:17 |
| marios | rlandy|rover: sshnaidm: ack thanks ... i guess perhaps the irrelevant files on the jobs is stopping them from running here possibly? | 15:17 |
| marios | sshnaidm: i thought this one would be quieter queue | 15:17 |
| marios | sshnaidm: i can also try on rdo testproject ack thanks | 15:17 |
| rlandy|rover | weshay: actually I think we should merge https://review.opendev.org/#/c/755867/ | 15:18 |
| rlandy|rover | until we have a better solution | 15:18 |
| rlandy|rover | at least it will pass jobs | 15:18 |
| rlandy|rover | thanks | 15:18 |
| weshay | rlandy|rover, k.. we can fold back to it.. when there is more time | 15:18 |
| rlandy|rover | weshay: yeah - no time to fix that atm | 15:19 |
| akahat | rf0lc0, thanks.. i'll take care of it. | 15:20 |
| chandankumar | sshnaidm: updated and rechecked | 15:21 |
| rlandy|rover | marios: try one check job first | 15:22 |
| weshay | arxcruz, chandankumar the tempest.yml file in tqe/playbooks is really a task file | 15:23 |
| marios | rlandy|rover: yeah am copy/pasting in the layout | 15:23 |
| *** amoralej is now known as amoralej|off | 15:23 | |
| weshay | arxcruz, chandankumar not for nuthin... that is a little confusing | 15:23 |
| arxcruz | weshay: yeah... :( | 15:23 |
| *** ykarel|away has quit IRC | 15:24 | |
| chandankumar | weshay: we will fix it | 15:26 |
| weshay | chandankumar, you folks think you'll move it.. or make it a playbook? | 15:27 |
| chandankumar | weshay: yes, I can make it playbook | 15:28 |
| weshay | chandankumar, k.. and you know what changes to make in tripleo-ci re: playbooks called right? | 15:29 |
| weshay | and tags | 15:30 |
| chandankumar | weshay: you want to call it via tags | 15:30 |
| chandankumar | from tripleo-ci, I can do that | 15:31 |
| soniya29|ruck | rlandy|rover: weshay,https://bugs.launchpad.net/tripleo/+bug/1899054 | 15:32 |
| openstack | Launchpad bug 1899054 in tripleo "'module' object has no attribute 'which'" [High,New] | 15:32 |
| weshay | soniya29|ruck, ugh.. that's the infrared check | 15:35 |
| weshay | kopecmartin, ^ | 15:35 |
| weshay | can you check this out? https://bugs.launchpad.net/tripleo/+bug/1899054 | 15:36 |
| openstack | Launchpad bug 1899054 in tripleo "'module' object has no attribute 'which'" [High,New] | 15:36 |
| weshay | arxcruz, chandankumar before you move anything https://review.opendev.org/#/c/756868/1 | 15:37 |
| rlandy|rover | soniya29|ruck: thanks | 15:37 |
| soniya29|ruck | rlandy|rover :) | 15:38 |
| rlandy|rover | marking promotion-blocker | 15:38 |
| rlandy|rover | blocking the gate | 15:38 |
| kopecmartin | weshay: sure, i hope it's not what i think it is | 15:39 |
| rlandy|rover | updated heading | 15:40 |
| weshay | kopecmartin, could just be infra... messing up | 15:42 |
| chandankumar | arxcruz: weshay https://review.opendev.org/#/c/756873/ | 15:43 |
| rlandy|rover | weshay: kopecmartin: failed twice on the gate this morning | 15:44 |
| rlandy|rover | weshay: could we get that rerun on openstack-periodic-integration-stable3? | 15:44 |
| weshay | ah ya | 15:46 |
| weshay | rlandy|rover, coming up | 15:46 |
| rlandy|rover | weshay: thank you! | 15:46 |
| rlandy|rover | let's try clear one line | 15:46 |
| weshay | rlandy|rover, kicked | 15:48 |
| rlandy|rover | soniya29|ruck: updated hackmd - feel free to add whatever else to track | 15:49 |
| *** marios is now known as marios|out | 15:55 | |
| weshay | chandankumar, arxcruz is this not deprecated? https://review.opendev.org/#/c/756680/2 | 16:02 |
| chandankumar | weshay: https://opendev.org/openstack/openstack-tempest-skiplist/src/branch/master/roles/install_skiplist/defaults/main.yaml#L7 | 16:03 |
| weshay | chandankumar, oh.. sorry | 16:05 |
| weshay | my mistake | 16:05 |
| weshay | read that incorrectly | 16:05 |
| *** marios|out has quit IRC | 16:06 | |
| weshay | bhagyashris++ | 16:06 |
| bhagyashris | :) | 16:08 |
| rf0lc0 | weshay, multinode-experimental template is not used anywhere, ignore, remove or convert to content provider ? https://github.com/openstack/tripleo-ci/blob/master/zuul.d/multinode-jobs.yaml#L443-L449 | 16:26 |
| chandankumar | soniya29|ruck: around? | 16:27 |
| chandankumar | sorry | 16:27 |
| chandankumar | sshnaidm: around? | 16:28 |
| chandankumar | sshnaidm: https://logserver.rdoproject.org/21/29921/3/check/tripleo-ci-centos-7-content-provider-train/55e791b/logs/undercloud/home/zuul/workspace/containers-consolidated-builds.log.txt.gz | 16:28 |
| chandankumar | it is very hard to find out what actually failed | 16:28 |
| rf0lc0 | chandankumar, I am working on it at https://review.opendev.org/#/c/755124/, but weshay did not let me work on it this sprint | 16:29 |
| rf0lc0 | chandankumar, improving reports and logs for container builds. The consolidated is not quite useful... I'm going to upload separate logs, and more clear indication of errors | 16:30 |
| rf0lc0 | chandankumar, wip | 16:30 |
| rf0lc0 | just fyi | 16:30 |
| chandankumar | rf0lc0: thank you for working on that, that is much needed | 16:30 |
| chandankumar | rf0lc0: I will keep an eye on that | 16:31 |
| rf0lc0 | cool | 16:31 |
| weshay | tru story! | 16:32 |
| soniya29|ruck | chandankumar yeah | 16:38 |
| chandankumar | soniya29|ruck: sorry it was for sshnaidm | 16:38 |
| soniya29|ruck | chandankumar okay | 16:38 |
| soniya29|ruck | rlandy|rover sure | 16:38 |
| rf0lc0 | weshay, multinode-experimental template is not used anywhere, ignore, remove or convert to content provider ? https://github.com/openstack/tripleo-ci/blob/master/zuul.d/multinode-jobs.yaml#L443-L449 | 16:43 |
| rf0lc0 | to ignore, press 1, to remove press 2, to convert press 3 | 16:44 |
| rf0lc0 | press 4 to talk to one of our representatives | 16:44 |
| weshay | rf0lc0, 2 KILL THAT SHIT :) | 16:44 |
| weshay | rf0lc0, you may run into a few of these, thanks for raising | 16:45 |
| rf0lc0 | weshay, ok thx | 16:45 |
| rf0lc0 | :) | 16:45 |
| *** dtantsur is now known as dtantsur|afk | 16:46 | |
| *** jpena is now known as jpena|off | 16:51 | |
| rlandy|rover | lunch - brb | 17:08 |
| sshnaidm | chandankumar, yeah, not clear what happened.. | 17:08 |
| sshnaidm | chandankumar, will hold another node | 17:08 |
| akahat | weshay, rf0lc0 can someone help me to identify the error: https://review.opendev.org/#/c/756573/ multinode job. | 17:14 |
| rf0lc0 | akahat, looking | 17:15 |
| *** derekh has quit IRC | 17:16 | |
| weshay | AnsibleUndefinedVariable: 'dict object' has no attribute 'overcloud' | 17:17 |
| akahat | yeah.. what is causing this? | 17:17 |
| rf0lc0 | missing tag | 17:18 |
| rf0lc0 | you have 5 | 17:18 |
| rf0lc0 | - overcloud-scripts | 17:18 |
| rf0lc0 | - overcloud-prep-config | 17:18 |
| rf0lc0 | - overcloud-prep-containers | 17:18 |
| rf0lc0 | - overcloud-post-deploy | 17:18 |
| rf0lc0 | - overcloud-validate | 17:18 |
| rf0lc0 | should be 6 | 17:18 |
| rf0lc0 | - overcloud-scripts | 17:18 |
| rf0lc0 | - overcloud-prep-config | 17:18 |
| rf0lc0 | - overcloud-prep-containers | 17:18 |
| rf0lc0 | - overcloud-deploy | 17:18 |
| rf0lc0 | - overcloud-post-deploy | 17:18 |
| rf0lc0 | - overcloud-validate | 17:18 |
| rf0lc0 | akahat, ^ | 17:18 |
| akahat | oh.. overcloud-deploy is missing. :) | 17:19 |
| rf0lc0 | :) | 17:19 |
| akahat | rf0lc0, let me try it. | 17:19 |
| weshay | https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/overcloud-deploy/templates/config-download.yaml.j2#L7 | 17:19 |
| weshay | ya.. was thinking the second node didn't come up | 17:20 |
| weshay | but it's there https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_955/756573/1/check/tripleo-ci-centos-8-containers-multinode/955ef97/zuul-info/inventory.yaml | 17:20 |
| rf0lc0 | weshay, missing tag | 17:21 |
| akahat | weshay, good news is we are not in big trouble :P | 17:21 |
| weshay | akahat, meh.. we'd know if that job didn't work | 17:22 |
| weshay | it's everywhere | 17:22 |
| weshay | rlandy|rover, need help w/ anything? | 17:24 |
| akahat | sshnaidm, since you are taking care tripleo-ci repo, so this template is modified https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/build-containers.yaml#L6-L28 ? | 17:40 |
| sshnaidm | akahat, we don't need to change this template | 17:44 |
| sshnaidm | akahat, it's building containers job | 17:44 |
| sshnaidm | akahat, what we need to change it's pulling containers job - deploying tripleo | 17:44 |
| akahat | sshnaidm, so this job only build containers.. and do not pull any. | 17:46 |
| rlandy|rover | weshay: we have an uptick in gate failures | 17:47 |
| rlandy|rover | but nothing we can pin it on yet | 17:47 |
| weshay | rlandy|rover, ya.. taken care of | 17:47 |
| rlandy|rover | other thanopenstack/ansible-role-collect-logs | 17:47 |
| rlandy|rover | weshay: so not really ... we had a pass on one of the centos08 stream jobs | 17:49 |
| weshay | rlandy|rover, no idea why this change is attempting to build something https://review.opendev.org/#/c/756504/1 | 17:49 |
| weshay | build a rpm | 17:49 |
| rlandy|rover | looking at the container build failure | 17:49 |
| rlandy|rover | looking | 17:50 |
| weshay | nice | 17:50 |
| rlandy|rover | weshay: no logs? https://e83c43581d4898374410-212ab268b5e4bbb4b3348f98a2a831ee.ssl.cf2.rackcdn.com/756504/1/check/zuul-ansible-role-collect-logs/914415d/ | 17:51 |
| rlandy|rover | love those mystery debug challenges | 17:52 |
| weshay | rlandy|rover, meh.. that patch :( | 17:53 |
| weshay | rlandy|rover, reset that patch | 17:53 |
| rlandy|rover | k - let's see if we get a repeat performance | 17:55 |
| weshay | pojadhav, requried chagnes to some telegraf files | 18:01 |
| weshay | incoming | 18:01 |
| weshay | https://review.rdoproject.org/r/#/c/29983/ | 18:02 |
| pojadhav | weshay, can we have short call..? | 18:07 |
| weshay | pojadhav, soure | 18:07 |
| weshay | sure even | 18:08 |
| weshay | https://meet.google.com/kbp-fhih-icz?authuser=1 | 18:08 |
| weshay | pojadhav, https://review.rdoproject.org/r/#/c/29983/1/ci-scripts/infra-setup/roles/rrcockpit/files/telegraf/promoter_utils.py | 18:12 |
| weshay | https://review.rdoproject.org/r/#/c/29983/ | 18:12 |
| akahat | rf0lc0, hey.. i've taken care the rest of the projects. | 18:29 |
| rf0lc0 | akahat, which ones ? | 18:30 |
| rf0lc0 | akahat, I am working on the ones that were empty in that table | 18:30 |
| akahat | rf0lc0, ansible-role-tripleo-modify-image, os-apply-config, os-net-config, os-refresh-config. | 18:30 |
| akahat | rf0lc0, sorry.. i thought i'll update all in 1 go. | 18:31 |
| akahat | i'm really sorrry. | 18:31 |
| rf0lc0 | np | 18:31 |
| akahat | rf0lc0, give me few mins. i'll update the table. | 18:31 |
| rf0lc0 | yes put your name there | 18:31 |
| rf0lc0 | I'll abandon my patches | 18:31 |
| rf0lc0 | please use the jira task | 18:32 |
| rf0lc0 | or hackmd | 18:32 |
| rf0lc0 | or irc | 18:32 |
| rf0lc0 | overcommunication is never waste of time | 18:32 |
| rf0lc0 | miscommunication is | 18:32 |
| akahat | rf0lc0, i've update the table please take a look. | 18:32 |
| akahat | agreed. : | 18:32 |
| akahat | :) | 18:33 |
| rf0lc0 | thanks akahat :) | 18:33 |
| *** jtomasek_ has quit IRC | 19:03 | |
| *** rlandy|rover is now known as rlandy|rover|afk | 19:18 | |
| *** TrevorV has quit IRC | 20:30 | |
| *** rf0lc0 has quit IRC | 21:30 | |
| *** jmasud has quit IRC | 21:48 | |
| *** apetrich has quit IRC | 22:14 | |
| *** rlandy|rover|afk is now known as rlandy|rover | 22:37 | |
| *** rlandy|rover has quit IRC | 22:46 | |
| *** tosky has quit IRC | 22:59 | |
| *** pmannidi has joined #oooq | 23:23 | |
| *** jmasud has joined #oooq | 23:35 | |
| *** rfolco has joined #oooq | 23:38 | |
| *** rfolco has quit IRC | 23:40 | |
| *** rfolco has joined #oooq | 23:41 | |
| *** rfolco has quit IRC | 23:45 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!