| *** sanjayu_ has joined #oooq | 00:52 | |
| *** sanjayu_ has quit IRC | 01:37 | |
| *** sanjayu_ has joined #oooq | 01:49 | |
| *** sanjayu_ has quit IRC | 02:06 | |
| weshay|ruck | ysandeep|away, arxcruz|rover anyone around? | 02:54 |
|---|---|---|
| *** ykarel has joined #oooq | 04:01 | |
| *** ysandeep|away is now known as ysandeep|ruck | 04:04 | |
| ysandeep|ruck | weshay|ruck, o/ | 04:04 |
| sshnaidm | ysandeep|ruck, hi, any idea why branched multinode failed in tempest? actually all before wallaby, because wallaby is passing | 04:32 |
| ykarel | sshnaidm, https://bugs.launchpad.net/tripleo/+bug/1929634 | 04:33 |
| openstack | Launchpad bug 1929634 in tripleo "containers-multinode victoria, ussuri, train failing in tempest w/ libvirt.libvirtError: internal error: unknown feature amd-sev-es" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 04:33 |
| ysandeep|ruck | https://bugs.launchpad.net/tripleo/+bug/1929634 | 04:33 |
| ysandeep|ruck | sshnaidm, issue affecting victoria, ussuri, train | 04:33 |
| ysandeep|ruck | sshnaidm, patches are up to make that job non-voting till we debug the issue:- | 04:34 |
| ysandeep|ruck | branched job non-voting: https://review.opendev.org/c/openstack/tripleo-ci/+/793089 | 04:34 |
| ysandeep|ruck | non-branch job non-voting: https://review.opendev.org/c/openstack/tripleo-ci/+/793091 | 04:34 |
| ykarel | ysandeep|ruck, is it known why only multinodes affected? | 04:34 |
| ykarel | i seen in standalone too in rdo jobs yesterday | 04:35 |
| sshnaidm | ysandeep|ruck, there was big kernel update for centos8 stream before yesterday | 04:36 |
| ykarel | https://logserver.rdoproject.org/28/33828/1/check/rdoinfo-tripleo-victoria-testing-centos-8-scenario001-standalone/1b845cb/logs/undercloud/var/log/containers/nova/nova-compute.log.txt.gz | 04:37 |
| ykarel | so don't know if making just multinode makes sense if standalone impacted too | 04:38 |
| sshnaidm | ykarel, we can't fix CI if it votes | 04:39 |
| sshnaidm | we don't run branched SA afaik | 04:39 |
| ykarel | ohkk so fix is already proposed? | 04:39 |
| ykarel | i didn't seen yet | 04:39 |
| ykarel | and do we already know why wallaby+ not impacted | 04:40 |
| ykarel | ok let me re read the bug details to find out | 04:40 |
| sshnaidm | probably related https://bugzilla.redhat.com/show_bug.cgi?id=1961562 | 04:40 |
| openstack | bugzilla.redhat.com bug 1961562 in libvirt "vm can not start with error as "internal error: unknown feature amd-sev-es"" [Urgent,Post] - Assigned to phrdina | 04:40 |
| * ysandeep|ruck just starting my day, gathering more information | 04:41 | |
| *** ykarel has quit IRC | 04:41 | |
| *** ykarel has joined #oooq | 04:42 | |
| *** ratailor has joined #oooq | 04:44 | |
| ysandeep|ruck | sshnaidm, yes kernel is among one of the bumped packages:- | 04:47 |
| ysandeep|ruck | ~~~ | 04:47 |
| ysandeep|ruck | older: kernel.x86_64 4.18.0-301.1.el8 @baseos | 04:47 |
| ysandeep|ruck | affected: kernel.x86_64 4.18.0-305.el8 @baseos | 04:47 |
| ysandeep|ruck | ~~~ | 04:47 |
| *** ykarel is now known as ykarel|afk | 04:48 | |
| sshnaidm | this looks like simple wrkrnd https://bugzilla.redhat.com/show_bug.cgi?id=1961562#c16 but it should be done on overcloud nova..? probably some feature in nova in old branches doesn't cover it ,worth to ask compute folks | 04:48 |
| openstack | bugzilla.redhat.com bug 1961562 in libvirt "vm can not start with error as "internal error: unknown feature amd-sev-es"" [Urgent,Post] - Assigned to phrdina | 04:48 |
| *** jpodivin has joined #oooq | 04:49 | |
| ysandeep|ruck | sshnaidm: i will try to reach compute folks, wes have emailed brian also if someone from compute can look at this on priority. | 04:51 |
| sshnaidm | ysandeep|ruck++ | 04:51 |
| *** ykarel|afk is now known as ykarel | 04:57 | |
| ykarel | ysandeep|ruck, sshnaidm what about excluding buggy rpm? that should avoid making jobs non-voting? | 04:57 |
| ykarel | edk2-ovmf | 04:57 |
| sshnaidm | I don't see it installed tbh | 04:59 |
| ysandeep|ruck | ykarel: i like the idea.. excluding rpm or pinning older version of edk2-ovmf? | 04:59 |
| ykarel | sshnaidm, it's in containers nova-compute | 04:59 |
| sshnaidm | ykarel, hmm.. and how do we exclude it from there? | 04:59 |
| ykarel | sshnaidm, exclude=edk2-ovmf-20200602gitca407c7246bf-5* | 05:00 |
| ykarel | as we build containers from repo setup, adding exclude in Appstream should do the trick | 05:00 |
| sshnaidm | in main dnf.conf? | 05:00 |
| ykarel | sshnaidm, quickstart release files for appstream repo | 05:00 |
| *** udesale has joined #oooq | 05:00 | |
| sshnaidm | yeah, worth try | 05:01 |
| ykarel | sshnaidm, if we managing dnf.conf already then can add exclude there too | 05:01 |
| ykarel | outside of release files, as multiple releases are impacted | 05:01 |
| ykarel | and this is a temporary fix | 05:01 |
| sshnaidm | maybe even "exclude=edk2-ovmf*" | 05:02 |
| ykarel | no with that we may have other issue | 05:02 |
| ykarel | with excluding all version it might fail if that's tried to being install | 05:02 |
| ykarel | buggy one is edk2-ovmf-20200602gitca407c7246bf-5*, so we exclude it, and c8-stream contains two versions in parallel | 05:03 |
| ykarel | so we use the lower version -4 | 05:03 |
| ykarel | and when there is fix and newer version is available that will get installed | 05:03 |
| ykarel | and later we remove the workaround | 05:04 |
| *** marios has joined #oooq | 05:04 | |
| ykarel | ysandeep|ruck, can u preapare patch to try ^ out? | 05:05 |
| ykarel | i have to be afk for some time | 05:05 |
| ysandeep|ruck | ykarel: do we manage dnf.conf? | 05:05 |
| ykarel | i doubt | 05:06 |
| ysandeep|ruck | ack o/ release files changes then.. sshnaidm have proposed a change.. let's see how it goes | 05:07 |
| ysandeep|ruck | ykarel++ sshnaidm++ | 05:07 |
| *** ykarel is now known as ykarel|afk | 05:17 | |
| *** ykarel|afk is now known as ykarel | 05:34 | |
| *** jfrancoa has joined #oooq | 05:36 | |
| *** slaweq has joined #oooq | 05:42 | |
| *** jpodivin has quit IRC | 06:06 | |
| *** sanjayu_ has joined #oooq | 06:10 | |
| *** jpodivin has joined #oooq | 06:11 | |
| *** ysandeep|ruck is now known as ysandeep|brb | 06:28 | |
| *** ysandeep|brb is now known as ysandeep|ruck | 06:45 | |
| *** sanjayu_ has quit IRC | 06:50 | |
| *** jfrancoa has quit IRC | 07:00 | |
| *** amoralej|off is now known as amoralej | 07:14 | |
| *** jmasud has quit IRC | 07:15 | |
| *** jfrancoa has joined #oooq | 07:18 | |
| *** sanjayu_ has joined #oooq | 07:22 | |
| *** tosky has joined #oooq | 07:35 | |
| arxcruz|rover | ysandeep|ruck: namaste | 07:47 |
| ysandeep|ruck | arxcruz|rover: Namaste and bom Dia | 07:49 |
| arxcruz|rover | ysandeep|ruck: oh, that was unexpected :) | 07:50 |
| ysandeep|ruck | :) | 07:51 |
| ysandeep|ruck | arxcruz|rover: fyi.. regarding bug: https://launchpad.net/bugs/1929634 we are trying to exclude the buggy rpm (edk2-ovmf-20200602gitca407c7246bf-5*) with https://review.opendev.org/c/openstack/tripleo-quickstart/+/793098 . If we exclude it, c8-stream contains two versions in parallel , so we use the lower version -4. | 07:54 |
| openstack | Launchpad bug 1929634 in tripleo "containers-multinode victoria, ussuri, train failing in tempest w/ libvirt.libvirtError: internal error: unknown feature amd-sev-es" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 07:54 |
| ysandeep|ruck | 793098 is currently in check.. lets see how it goes.. otherwise wes have some patches to make the container-multinode job non-voting.. | 07:55 |
| ysandeep|ruck | We had issue with content-providers too.. for the time being we reverted https://review.rdoproject.org/r/c/openstack/tripleo-ansible-distgit/+/33840, https://review.rdoproject.org/r/c/openstack/tripleo-ansible-distgit/+/33839, https://review.rdoproject.org/r/c/openstack/tripleo-ansible-distgit/+/33838 | 07:56 |
| ysandeep|ruck | I think further plan is to merge: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/792904 first then we revert back these ^^ again. | 07:57 |
| * ysandeep|ruck starting to look at periodic failures.. | 07:58 | |
| *** sshnaidm is now known as sshnaidm|afk | 07:58 | |
| ysandeep|ruck | arxcruz|rover, periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master was failing i have reopened https://bugs.launchpad.net/tripleo/+bug/1928926 | 07:58 |
| openstack | Launchpad bug 1928926 in tripleo "tripleo-ansible-inventory error's with - Error generating inventory for overcloud: 'ctlplane_cidr'" [High,Triaged] - Assigned to Harald Jensås (harald-jensas) | 07:58 |
| *** jpena|off is now known as jpena | 08:01 | |
| *** jmasud has joined #oooq | 08:06 | |
| arxcruz|rover | ysandeep|ruck: ack | 08:09 |
| *** sanjayu_ has quit IRC | 08:13 | |
| arxcruz|rover | ysandeep|ruck: sorry the delay, just finish to read the tripleo-ci chat, still a little bit confusing for me to follow it, so the multinode issue sees to be handled right ? | 08:33 |
| ysandeep|ruck | arxcruz|rover, yes hoping https://review.opendev.org/c/openstack/tripleo-quickstart/+/793098 (excluding the latest edk2-ovmf will make the jobs green again) | 08:35 |
| arxcruz|rover | ysandeep|ruck: ack, so will you be on prog. call today ? | 08:36 |
| arxcruz|rover | you make my life easy <3 | 08:36 |
| ysandeep|ruck | arxcruz|rover, i will cover pgrm call o/ | 08:36 |
| bhagyashris | ysandeep|ruck, arxcruz|rover hey i am taking sick leave today | 08:38 |
| ysandeep|ruck | bhagyashris, take rest, feel better | 08:39 |
| bhagyashris | regarding promoter all the c8 branches are running on old promoter and new server is off | 08:39 |
| bhagyashris | ysandeep|ruck, sandeep in case master promotion fails at manifest then trun off the manifest_push: false in master.yaml config file and restart the service | 08:40 |
| bhagyashris | 10.0.148.74 | 08:40 |
| bhagyashris | or you can ping me i will try to check in between | 08:40 |
| ysandeep|ruck | bhagyashris, I don't think my keys will be on promoter.. but I can ask weshay|ruck .. Anway ci is in bad shape.. i am not expecting promotions today.. take rest | 08:41 |
| arxcruz|rover | bhagyashris: take care, get well | 08:41 |
| bhagyashris | ysandeep|ruck, ack | 08:41 |
| * ysandeep|ruck stepping out for lunch | 08:47 | |
| *** ykarel is now known as ykarel|lunch | 08:53 | |
| *** jmasud has quit IRC | 09:02 | |
| *** ykarel|lunch is now known as ykarel | 09:33 | |
| *** ratailor has quit IRC | 09:43 | |
| *** ratailor has joined #oooq | 09:45 | |
| *** ratailor has quit IRC | 10:03 | |
| *** sshnaidm|afk is now known as sshnaidm | 10:25 | |
| sshnaidm | ysandeep|ruck, ykarel seems like this worked: https://review.opendev.org/c/openstack/tripleo-quickstart/+/793098 | 10:25 |
| ysandeep|ruck | sshnaidm, | 10:25 |
| ysandeep|ruck | yes | 10:25 |
| sshnaidm | marios, can we get your +w please ^ | 10:25 |
| ykarel | yeap let's get it | 10:26 |
| sshnaidm | ysandeep|ruck, ovb is failing again? | 10:26 |
| sshnaidm | "Failed to attach network adapter device to " (HTTP 500) | 10:27 |
| sshnaidm | seems like vexxhost problem | 10:27 |
| ysandeep|ruck | sshnaidm, worked for a testproject i posted 1 hour back https://review.rdoproject.org/zuul/stream/bd84310ef36543ccb59dfe95f3226fbd?logfile=console.log | 10:27 |
| ysandeep|ruck | may be issue is intermittent. | 10:27 |
| sshnaidm | ack | 10:28 |
| marios | sshnaidm: ack commented there though https://review.opendev.org/c/openstack/tripleo-quickstart/+/793098/3/config/release/tripleo-ci/CentOS-8/ussuri.yml#159 +2 to unblock but not clear what the plan/issue actually is with that | 10:30 |
| sshnaidm | marios, the details are in linked BZ | 10:31 |
| sshnaidm | https://bugzilla.redhat.com/show_bug.cgi?id=1961558 | 10:31 |
| openstack | bugzilla.redhat.com bug 1961558 in libvirt "virsh domcapabilities fails with the error: internal error: unknown feature amd-sev-es" [Unspecified,Post] - Assigned to phrdina | 10:31 |
| marios | sshnaidm: ok i didn't check that one | 10:31 |
| marios | sshnaidm: do we need that for the wallaby config too? | 10:32 |
| sshnaidm | marios, no, it works in wallaby | 10:38 |
| marios | sshnaidm ack ykarel replied ont he patch thanks | 10:38 |
| sshnaidm | broken in t/u/v | 10:38 |
| *** ratailor has joined #oooq | 10:40 | |
| *** ratailor_ has joined #oooq | 10:42 | |
| *** ratailor has quit IRC | 10:45 | |
| *** ratailor has joined #oooq | 11:00 | |
| *** ratailor_ has quit IRC | 11:02 | |
| soniya29 | arxcruz|rover, kopecmartin, ysandeep|ruck, do you have anything to discuss in tempest meeting today? | 11:06 |
| ysandeep|ruck | soniya29: i don't have any agenda | 11:10 |
| soniya29 | ysandeep|ruck, ack | 11:12 |
| soniya29 | kopecmartin, arxcruz|rover, ^^ | 11:22 |
| arxcruz|rover | soniya29: besides we need to define the skiplist on downstream i have nothing | 11:22 |
| *** holser has joined #oooq | 11:25 | |
| soniya29 | arxcruz|rover, ysandeep|ruck, Since you both are ruck/rovering and I am busy with some other stuffs, Will it be okay with you if we cancel the today's meeting? | 11:27 |
| ysandeep|ruck | soniya29, yes works for me. | 11:27 |
| kopecmartin | soniya29: i have nothing, sure | 11:30 |
| kopecmartin | arxcruz|rover: something i can help with? | 11:30 |
| kopecmartin | arxcruz|rover: we can talk / chat regardless the meeting | 11:31 |
| *** holser has quit IRC | 11:35 | |
| *** jpena is now known as jpena|lunch | 11:35 | |
| *** holser has joined #oooq | 11:36 | |
| arxcruz|rover | kopecmartin: actually, yeah, check the patch https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/789625 | 11:40 |
| kopecmartin | arxcruz|rover: ack | 11:40 |
| rlandy | ysandeep|ruck: arxcruz|rover: need help with anything? | 11:43 |
| arxcruz|rover | rlandy: ysandeep|ruck https://i.imgur.com/etFbE29.jpg | 11:44 |
| rlandy | yep | 11:45 |
| ysandeep|ruck | rlandy: we are getting in better shape, thanks! things are under control now | 11:45 |
| rlandy | ok | 11:45 |
| arxcruz|rover | rlandy: we are not expecting promotion today, but multinode patches are fixed | 11:45 |
| arxcruz|rover | i mean, multinode jobs ahve a patch to get it fixed | 11:45 |
| rlandy | ysandeep|ruck: fyi ... https://review.rdoproject.org/r/c/rdo-jobs/+/33861 | 11:46 |
| rlandy | works to get image build back in ovb on check | 11:47 |
| ysandeep|ruck | arxcruz|rover: this will fix periodic t/u/v jobs: https://review.opendev.org/c/openstack/tripleo-quickstart/+/793145 | 11:47 |
| rlandy | going to un -DNM and merge that | 11:47 |
| ysandeep|ruck | rlandy, i hope this will not build image against all repos and or just for projects we need. | 11:48 |
| ysandeep|ruck | and just* | 11:48 |
| rlandy | ysandeep|ruck: ack - updated the commit message ... https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/check-to-build-or-not-to-build/defaults/main.yml | 11:49 |
| ysandeep|ruck | rlandy: +1 thanks for fixing this! | 11:50 |
| rlandy | marios: hi ... https://review.opendev.org/c/openstack/tripleo-specs/+/772442 spec updated and blueprint added | 11:50 |
| marios | rlandy: thanks i saw | 11:51 |
| marios | rlandy: didnt review the udpated spec yet though in my list for next reviews | 11:51 |
| rlandy | marios: let me know if we are hand-waving too much now | 11:51 |
| ysandeep|ruck | rlandy, fyi.. i had open a bug for this earlier.. https://bugs.launchpad.net/tripleo/+bug/1928784 | 11:51 |
| openstack | Launchpad bug 1928784 in tripleo "Image build not executing in ovb check jobs for projects which affect the image building itself. " [Critical,Triaged] - Assigned to Amol Kahat (amolkahat) | 11:51 |
| marios | rlandy: ;) | 11:51 |
| rlandy | ysandeep|ruck: ah - thanks - will add to the review | 11:51 |
| gchamoul | rlandy: Hi Ronelle, we are hitting deps issue https://bugs.launchpad.net/tripleo/+bug/1929695 | 11:53 |
| openstack | Launchpad bug 1929695 in tripleo "tripleo-ci-centos-8-standalone-validation-libs fails due to unmet dependencies" [Undecided,New] | 11:53 |
| gchamoul | rlandy: which are caused by https://review.opendev.org/c/openstack/tripleo-quickstart/+/792647 | 11:53 |
| rlandy | gchamoul: in the component line? | 11:54 |
| gchamoul | rlandy: it seems that a DNM patch has been merged, doesn't it? or it is more complex | 11:54 |
| gchamoul | rlandy: https://zuul.opendev.org/t/openstack/build/0971a386e2184ca99f61a7c879388cd5/log/logs/undercloud/home/zuul/repo_setup.log#253 | 11:54 |
| rlandy | gchamoul: yep - that was a DNM patch which was merged | 11:54 |
| gchamoul | rlandy: delorean-component-validation is disabled and python3-tripleoclient cannot be installed because it misses validations-common and python3-validations-libs | 11:55 |
| *** ysandeep|ruck is now known as ysandeep|mtg | 11:57 | |
| *** holser has quit IRC | 11:57 | |
| rlandy | gchamoul: hmmm ... you are right ... your jobs defined component without running in the component line ... https://opendev.org/openstack/validations-libs/src/branch/master/.zuul.yaml#L25 | 11:58 |
| rlandy | that was not expected | 11:58 |
| gchamoul | rlandy: the same for all our repos, tripleo-validations and validations-common | 11:59 |
| gchamoul | git repos | 11:59 |
| rlandy | gchamoul: k - going to have to add another condition in that check | 12:00 |
| rlandy | gchamoul: I'll post a revert in the mean time | 12:00 |
| rlandy | but we need that change | 12:00 |
| rlandy | will have to think of an addition condition | 12:00 |
| gchamoul | rlandy: Rabi already reverted it but abandoned it too | 12:01 |
| rlandy | gchamoul: yep - but that was another unrelated issue | 12:01 |
| rlandy | had nothing to do with the change | 12:02 |
| rlandy | component: usually means 'in the component pipeline' | 12:02 |
| gchamoul | ack | 12:02 |
| rlandy | https://review.opendev.org/c/openstack/tripleo-quickstart/+/793050 posted | 12:03 |
| rlandy | there is another review | 12:03 |
| rlandy | to revert | 12:03 |
| rlandy | sec | 12:03 |
| rlandy | gchamoul: ok - only one patch ... https://review.opendev.org/c/openstack/tripleo-quickstart/+/792818 didn't merge yet | 12:05 |
| gchamoul | rlandy: thanks Ronelle! | 12:06 |
| gchamoul | rlandy: will we hit the same issue for W,V,U and Train? :/ | 12:10 |
| gchamoul | rlandy: If I read and understand well the second patch #793050 | 12:11 |
| gchamoul | sorry #792818 instead | 12:11 |
| rlandy | gchamoul: no - that patch did not merge yet | 12:12 |
| rlandy | gchamoul: if we checked that the job name also had component in it, would that work for you? | 12:12 |
| rlandy | should avoid your jobs | 12:13 |
| gchamoul | rlandy: honestly, I don't know ... I trust you I would say :D | 12:14 |
| rlandy | gchamoul: trusting me is how we got into this mess in the first place :) | 12:14 |
| rlandy | well - and the fact that I marked the patch DNM and it got merged anyways | 12:14 |
| weshay|ruck | rlandy, let's just rework it w/ the concept but use priorities rlandy | 12:15 |
| weshay|ruck | yum priorities | 12:15 |
| rlandy | weshay|ruck: we used yum priorities to begin with | 12:16 |
| rlandy | we marked the component repo 1 and the delorean repo 20 | 12:16 |
| rlandy | ysandeep|mtg, pointed out that it is a dnf behaviour to go through all the repos | 12:16 |
| rlandy | if the highest priority does not supply the rpm | 12:17 |
| gchamoul | rlandy: so like we have component name defined in our job, the dnf repo will still be disabled | 12:17 |
| rlandy | or there is a failure | 12:17 |
| rlandy | weshay|ruck: so the only way to avoid that behaviour altogether is to disable the repo | 12:17 |
| weshay|ruck | rlandy, k.. | 12:18 |
| rlandy | weshay|ruck: gchamoul: I can try disable the repo if and only if the component repo exists | 12:18 |
| weshay|ruck | gchamoul, background is that we were trying to avoid repoclosure issues | 12:18 |
| rlandy | not rely on job vars | 12:18 |
| weshay|ruck | rlandy, w/ a stat? sounds fine | 12:18 |
| weshay|ruck | gchamoul, so ur watching those jobs eh? | 12:19 |
| rlandy | weshay|ruck: well it's in the release file so something like that | 12:19 |
| rlandy | weshay|ruck: nah - we broke their jobs | 12:19 |
| gchamoul | weshay|ruck: WE are now, yes! ;-) | 12:19 |
| rlandy | https://bugs.launchpad.net/tripleo/+bug/1929695 | 12:19 |
| openstack | Launchpad bug 1929695 in tripleo "tripleo-ci-centos-8-standalone-validation-libs fails due to unmet dependencies" [Undecided,New] | 12:19 |
| weshay|ruck | rlandy, ya.. but just component :) | 12:19 |
| rlandy | ^^ that's a check job | 12:19 |
| rlandy | weshay|ruck: I posted a revert of that review in the mean time | 12:20 |
| rlandy | while I work on the fix | 12:20 |
| gchamoul | weshay|ruck: jpodivin detected and reported the issue first though! | 12:20 |
| rlandy | https://review.opendev.org/c/openstack/tripleo-quickstart/+/792818 didn't merge yet | 12:20 |
| rlandy | so _ w-1'ed that | 12:20 |
| weshay|ruck | gchamoul, k.. thanks.. can you please show him how to triage a tripleo bug | 12:20 |
| gchamoul | ack :D | 12:21 |
| weshay|ruck | thanks | 12:21 |
| *** jpena|lunch is now known as jpena | 12:24 | |
| weshay|ruck | rlandy, I think the job name used in validations may be the culprit | 12:29 |
| rlandy | weshay|ruck: it's the fact that they define a var component: | 12:29 |
| rlandy | in their check jobs | 12:29 |
| weshay|ruck | rlandy, and we can condition the removal of the main repo based on if it's not upstream | 12:29 |
| rlandy | ie: there is no add repo | 12:29 |
| weshay|ruck | ya | 12:29 |
| weshay|ruck | ya | 12:29 |
| rlandy | weshay|ruck: I am conditioning it based on whether the componet-repo exists | 12:30 |
| weshay|ruck | rlandy, k.. that works | 12:30 |
| weshay|ruck | thanks | 12:30 |
| rlandy | that way it is not reliant on any var | 12:30 |
| rlandy | either it exists or it doesn't | 12:30 |
| rlandy | if it exists, this should pass in any job | 12:30 |
| rlandy | will testproject the change with gchamoul's job | 12:30 |
| *** ysandeep|mtg is now known as ysandeep|ruck | 12:33 | |
| weshay|ruck | rlandy, that's my bad.. I should have briefed the team on these validation check jobs | 12:37 |
| rlandy | weshay|ruck: no worries - we're briefed now :) | 12:37 |
| rlandy | I just thought we had the monopoly on 'component' | 12:37 |
| weshay|ruck | rlandy, I'm discussing that w/ them now | 12:40 |
| weshay|ruck | to use another variable so we avoid this in the future.. | 12:40 |
| weshay|ruck | in #validation-framework | 12:40 |
| weshay|ruck | internal | 12:40 |
| weshay|ruck | <matbu> weshay|ruck: yep ack, I will change that | 12:41 |
| weshay|ruck | [06:40:53] <matbu> weshay|ruck: all standalone check jobs is using that | 12:41 |
| weshay|ruck | rlandy, ^ | 12:41 |
| weshay|ruck | arxcruz|rover, ysandeep|ruck let's sync | 12:42 |
| ysandeep|ruck | weshay|ruck, ack | 12:42 |
| arxcruz|rover | weshay|ruck: ack | 12:42 |
| arxcruz|rover | weshay|ruck: meet? | 12:42 |
| rlandy | weshay|ruck: gchamoul: https://review.opendev.org/c/openstack/tripleo-quickstart/+/793149 | 12:43 |
| rlandy | going to test that out | 12:43 |
| rlandy | it's w-1 | 12:43 |
| rlandy | until I testproject it | 12:43 |
| ysandeep|ruck | weshay|ruck, arxcruz|rover meet.google.com/rko-tjxv-hnv | 12:44 |
| *** arxcruz|rover has quit IRC | 12:50 | |
| ysandeep|ruck | weshay|ruck, https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-build-containers-ubi-8-push-train&project=openstack/tripleo-ci | 12:53 |
| weshay|ruck | rlandy, https://review.opendev.org/q/topic:%22ci%252Frole%252Fvalidation%22+(status:open%20OR%20status:merged) | 13:14 |
| rlandy | weshay|ruck: ack - thanks | 13:14 |
| rlandy | I still think https://review.opendev.org/c/openstack/tripleo-quickstart/+/793149 is a better way to go | 13:15 |
| *** ratailor has quit IRC | 13:25 | |
| marios | weshay|ruck: * http://lists.openstack.org/pipermail/openstack-discuss/2021-May/022703.html | 13:38 |
| marios | weshay|ruck: 1430 utc | 13:39 |
| weshay|ruck | ysandeep|ruck, https://review.opendev.org/c/openstack/tripleo-quickstart/+/793145/ | 13:39 |
| weshay|ruck | is failing | 13:39 |
| ysandeep|ruck | weshay|ruck, will check | 13:40 |
| ysandeep|ruck | weshay|ruck: please merge this https://review.rdoproject.org/r/c/config/+/33836 rdo-tox-molecule is fixed. | 13:40 |
| weshay|ruck | 2021-05-26 12:06:01 | centos-8-fix 0.0 B/s | 0 B 02:00 | 13:40 |
| weshay|ruck | 2021-05-26 12:06:01 | Errors during downloading metadata for repository 'centos-8-fix': | 13:40 |
| weshay|ruck | 2021-05-26 12:06:01 | - Curl error (28): Timeout was reached for http://mirror.regionone.limestone.opendev.org:8080/rdo/centos8-master/deps/c8-fix/repodata/repomd.xml [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds] | 13:40 |
| ysandeep|ruck | mirror issues again? | 13:41 |
| weshay|ruck | ykarel, ^ | 13:41 |
| weshay|ruck | was that just added to deps? | 13:41 |
| ysandeep|ruck | sshnaidm, weshay|ruck need reviews https://review.rdoproject.org/r/c/config/+/33867 | 13:51 |
| ysandeep|ruck | ykarel, ^^ | 13:52 |
| weshay|ruck | ysandeep|ruck, ready to merge or no? | 13:52 |
| ykarel | seems region specific? | 13:52 |
| ykarel | limestone | 13:52 |
| ykarel | or it's happening to more providers? | 13:52 |
| ysandeep|ruck | weshay|ruck, lgtm to merge. | 13:53 |
| ykarel | ok i now read the question | 13:55 |
| ykarel | deps/c8-fix exists from long | 13:55 |
| ykarel | it's not repo specific, infra side have issues | 13:56 |
| ysandeep|ruck | https://99e02e92a26472cf4bfd-76db863cb86d059bc445c2f80e5d4947.ssl.cf5.rackcdn.com/772571/26/check/tripleo-ci-centos-8-containers-multinode-ussuri/4503771/job-output.txt | 13:56 |
| ysandeep|ruck | pip._vendor.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='opendev.org', port=443): Max retries exceeded with url: /openstack/requirements/raw/branch/stable/ussuri/upper-constraints.txt (Caused by NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f9537ae4470>: Failed to establish a new connection: [Errno 101] Network is unreachable',)) | 13:56 |
| ysandeep|ruck | many jobs in upstream failing on post_failure.. | 13:56 |
| * ysandeep|ruck pinging on #opendev | 13:58 | |
| ysandeep|ruck | ykarel, looks like multiple mirrors are affected.. Curl error (6): Couldn't resolve host name for http://mirror.gra1.ovh.opendev.org/epel/7/x86_64/repodata/repomd.xml [Could not resolve host: mirror.gra1.ovh.opendev.org] | 14:06 |
| ysandeep|ruck | multiple providers* | 14:07 |
| ykarel | ohkk good to reach upstream infra | 14:07 |
| weshay|ruck | rlandy, fyi.. we got facter3 working on puppet7 so.. I think we're closing in on the puppet version to use now.. which would be 7. | 14:23 |
| ysandeep|ruck | weshay|ruck, fyi.. https://bugs.launchpad.net/tripleo/+bug/1929715 chatting with infra | 14:24 |
| openstack | Launchpad bug 1929715 in tripleo "Upstream infra hitting network related issue: [Errno 101] Network is unreachable' / Curl error (28): Timeout was reached [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds]" [Critical,Triaged] | 14:24 |
| rlandy | weshay|ruck: np - both are there now - 6 and 7 | 14:24 |
| *** ykarel has quit IRC | 14:32 | |
| *** ykarel has joined #oooq | 14:32 | |
| *** tristanC has left #oooq | 14:43 | |
| *** gchamoul is now known as gchamoul_ | 15:07 | |
| ysandeep|ruck | weshay|ruck, upstream infra seems to had an intermittent issue, when i reached to infra folks we can reach the mirror and things looks fine. If issue reappers/persists in job, they will dig deeper. | 15:08 |
| ysandeep|ruck | weshay|ruck: i also got a recommendation about using constraints from the local openstack/requirements repo instead of pulling over network. So that we don;t hit issue like below:- | 15:08 |
| ysandeep|ruck | https://99e02e92a26472cf4bfd-76db863cb86d059bc445c2f80e5d4947.ssl.cf5.rackcdn.com/772571/26/check/tripleo-ci-centos-8-containers-multinode-ussuri/4503771/job-output.txt | 15:08 |
| weshay|ruck | ysandeep|ruck, ya.. just another patch to follow and recheck | 15:10 |
| weshay|ruck | thanks | 15:10 |
| weshay|ruck | rlandy, ysandeep|ruck metadata service discussed in email "PSI Openstack Cloud D Incident | 2021-04-27 20:29 UTC" TLDR... is do not move back to the metadata service | 15:14 |
| *** dmellado has quit IRC | 15:14 | |
| *** dmellado has joined #oooq | 15:15 | |
| *** jpodivin has quit IRC | 15:16 | |
| ysandeep|ruck | weshay|ruck, ack, to be clear.. we moved to config-drive for vexx cloud only.. Will need some work in downstream to move to config-drive | 15:16 |
| weshay|ruck | ysandeep|ruck, ya.. def.. required afaict | 15:16 |
| ysandeep|ruck | weshay|ruck, could you please review/merge https://review.rdoproject.org/r/c/config/+/33836 rdo-tox-molecule is back to green | 15:19 |
| rlandy | ok | 15:20 |
| rlandy | https://review.opendev.org/c/openstack/tripleo-quickstart/+/793149 running now | 15:20 |
| rlandy | weshay|ruck: ^^ w-1'ed the revert https://review.opendev.org/c/openstack/tripleo-quickstart/+/793050 | 15:21 |
| rlandy | would rather go with the check solution if it works | 15:21 |
| ysandeep|ruck | ykarel, sshnaidm fyi.. weshay|ruck pointed integration line container pull don't use tripleo-quickstart release files, arx have put up a patch in tripleo-repos: https://review.opendev.org/c/openstack/tripleo-repos/+/793157 | 15:25 |
| *** ykarel is now known as ykarel|away | 15:26 | |
| ykarel|away | ack | 15:26 |
| *** jmasud has joined #oooq | 15:26 | |
| * ysandeep|ruck out for the day, see you tomorrow folks o/ | 15:28 | |
| *** ysandeep|ruck is now known as ysandeep|away | 15:28 | |
| ysandeep|away | s/container pull/container build | 15:29 |
| *** ykarel|away has quit IRC | 15:32 | |
| marios | need votes there please add to your reviews tripleo-ci o/ https://review.opendev.org/c/openstack/tripleo-ci/+/793144 | 15:36 |
| marios | thank you | 15:36 |
| *** dsneddon has quit IRC | 15:39 | |
| zbr | @oooq: if we are to pick only one version of ansible for running molecule tests it should be the newest or the oldest we do support? | 15:40 |
| weshay|ruck | ysandeep|away, can you share the git url to your new project? | 15:40 |
| weshay|ruck | zbr, imho.. can we stick w/ the same requirements we have in tq? | 15:41 |
| weshay|ruck | 2.9 > 2.10 | 15:41 |
| zbr | https://github.com/openstack/tripleo-quickstart/blob/master/requirements.txt#L2 | 15:42 |
| zbr | so the answer is oldest | 15:42 |
| zbr | i was inclined to believe that would be the case, still I am inlined to avoid using this file as a constraint | 15:42 |
| weshay|ruck | zbr, ya.. for now I think that's safe | 15:42 |
| weshay|ruck | we'll have to discuss collections etc.. just not yet | 15:43 |
| *** marios has quit IRC | 15:45 | |
| weshay|ruck | frenzy_friday, appologies.. it's been a week of ruck/rovering in just a few short days | 15:53 |
| weshay|ruck | haven't been able to get the bandwidth for tripleo-health and our new rucks | 15:53 |
| weshay|ruck | keep rocking it though.. WE NEED IT | 15:53 |
| frenzy_friday | np, if you have current bugs/error strings that you want to track I can add them | 15:54 |
| weshay|ruck | in fact.. I need to add this mirror issue | 15:54 |
| rlandy | weshay|ruck: frenzy_friday: maybe we can add the mirror failure in upcoming meeting^^? | 15:57 |
| frenzy_friday | yep, are you joining the 1-1? we can add it there | 15:58 |
| weshay|ruck | if you want me to | 15:58 |
| weshay|ruck | we can chat there | 15:58 |
| weshay|ruck | frenzy_friday, rlandy https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/793200 | 16:06 |
| rlandy | looking thanks | 16:07 |
| rlandy | missing file | 16:08 |
| *** ykarel|away has joined #oooq | 16:10 | |
| weshay|ruck | ah sova | 16:16 |
| *** ykarel|away has quit IRC | 16:22 | |
| weshay|ruck | frenzy_friday, rlandy https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/793204 | 16:27 |
| rlandy | weshay|ruck: was an error in the id | 16:28 |
| rlandy | needed underscores | 16:28 |
| weshay|ruck | or quotes maybe? | 16:29 |
| weshay|ruck | nope.. Underscores.. got it | 16:30 |
| * frenzy_friday checks | 16:41 | |
| rlandy | weshay|ruck: frenzy_friday is working on it | 16:41 |
| rlandy | weshay|ruck: want to keep 1-on-1 or too busy? | 16:42 |
| weshay|ruck | ya.. we can chat for a sec about stuff | 16:43 |
| rlandy | ok | 16:44 |
| weshay|ruck | frenzy_friday, so my query doesn't match I guess | 16:58 |
| *** jbadiapa is now known as jbadiapa|away | 16:58 | |
| weshay|ruck | looks like it should though | 16:58 |
| rlandy | we were having tox issues | 16:59 |
| weshay|ruck | k k | 16:59 |
| frenzy_friday | weshay|ruck, updated. Lets see if it passes | 17:00 |
| rlandy | frenzy_friday: fixed tox? | 17:00 |
| weshay|ruck | frenzy_friday, bah.. sorry for typos | 17:02 |
| weshay|ruck | tox passes for me | 17:02 |
| rlandy | lucky you :) | 17:02 |
| weshay|ruck | frenzy_friday, what is the process for that getting included in the dash? | 17:02 |
| weshay|ruck | any manual intervention required? | 17:02 |
| frenzy_friday | weshay|ruck, output/elastic-recheck/1929461.yaml this file was missing. After running tox 2 files are modifies/generated - sova-generated json and <id>.yml Need to commit both . https://opendev.org/openstack/tripleo-ci-health-queries/src/branch/master/README.md#adding-a-new-item-in-queries-yml | 17:02 |
| frenzy_friday | rlandy, tox was a local issue on your system. Checking what happened (asking google) | 17:03 |
| frenzy_friday | I'll fix the underscore stuff. | 17:03 |
| *** jpena is now known as jpena|off | 17:05 | |
| frenzy_friday | weshay|ruck, rlandy https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/793200/ passed | 17:12 |
| rlandy | great | 17:13 |
| weshay|ruck | frenzy_friday, any manual steps to get it in the dash? | 17:15 |
| *** udesale has quit IRC | 17:15 | |
| frenzy_friday | no, but we need to commit both sova-gerenated..json and <id>.yml This <id>.yml will be a new file and we will have to git add it | 17:16 |
| *** jlarriba has quit IRC | 17:24 | |
| weshay|ruck | frenzy_friday, feel free to change my README_Quick | 17:25 |
| zbr | before i leave, two quick reviews: https://review.rdoproject.org/r/c/config/+/33870 | 17:40 |
| *** cgoncalves has quit IRC | 17:42 | |
| *** irclogbot_0 has quit IRC | 17:43 | |
| *** cgoncalves has joined #oooq | 17:43 | |
| *** irclogbot_2 has joined #oooq | 17:47 | |
| weshay|ruck | rlandy, https://review.opendev.org/c/openstack/ansible-role-tripleo-modify-image/+/793028/2/tasks/yum_update_buildah.yml#150 | 17:48 |
| weshay|ruck | rlandy, frenzy_friday WOOOOO HOOOOOO! http://health.sbarnea.com/#1929461 | 17:55 |
| weshay|ruck | 33 fails in 10 days.. and links to elastic search :)) | 17:55 |
| frenzy_friday | weshay|ruck, wow already! | 17:55 |
| rlandy | nice! | 17:56 |
| weshay|ruck | awesome.. added a link to the card https://trello.com/c/2KRxJzov/1960-cixlp1929461tripleociproa-yum-repo-mirrors-are-down-5-24-2021 | 17:56 |
| weshay|ruck | so we can really see when issues are resolved | 17:56 |
| * weshay|ruck does a happy dance | 17:56 | |
| weshay|ruck | going to show this to pweeks | 17:56 |
| *** dsneddon has joined #oooq | 18:00 | |
| weshay|ruck | frenzy_friday, so is the gerrit updates on? | 18:01 |
| weshay|ruck | like if a patch hits this? | 18:01 |
| weshay|ruck | or not yet | 18:01 |
| frenzy_friday | the bot? No. Working on it | 18:01 |
| weshay|ruck | ++ | 18:03 |
| frenzy_friday | *working on it next week | 18:03 |
| *** openstack has quit IRC | 18:09 | |
| *** openstack has joined #oooq | 18:10 | |
| *** ChanServ sets mode: +o openstack | 18:10 | |
| *** jmasud has quit IRC | 18:13 | |
| frenzy_friday | weshay|ruck, should we add the quick readme to the main readme? | 18:16 |
| weshay|ruck | frenzy_friday, that's fine.. but put it at the top... just enough info to GO GO GO | 18:25 |
| weshay|ruck | then get into the details after that | 18:25 |
| *** dmellado has quit IRC | 18:39 | |
| *** dmellado has joined #oooq | 18:39 | |
| frenzy_friday | weshay|ruck, rlandy when you get time https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/793217 - for the underscore | 18:50 |
| rlandy | thanks | 18:51 |
| *** jmasud has joined #oooq | 18:56 | |
| *** jmasud has quit IRC | 18:57 | |
| *** jfrancoa has quit IRC | 19:00 | |
| *** jfrancoa has joined #oooq | 19:01 | |
| rlandy | weshay|ruck: so you were right - it is in collect logs ... https://opendev.org/openstack/ansible-role-collect-logs/src/branch/master/roles/collect_logs/files/collect-container-logs.sh#L71 | 19:03 |
| rlandy | ${engine} exec -u root "$cont" bash -c "\$(command -v dnf || command -v yum) list installed"; | 19:03 |
| rlandy | ) &>> "$INFO_DIR/${engine}_info.log"; | 19:03 |
| *** jfrancoa has quit IRC | 19:04 | |
| *** gchamoul_ has quit IRC | 19:07 | |
| *** gchamoul has joined #oooq | 19:07 | |
| *** gchamoul has quit IRC | 19:14 | |
| *** gchamoul has joined #oooq | 19:14 | |
| frenzy_friday | see you guys on monday, thanks | 19:25 |
| *** frenzy_friday is now known as frenzyfriday|off | 19:25 | |
| *** jmasud has joined #oooq | 19:26 | |
| *** openstackstatus has quit IRC | 19:50 | |
| *** openstackstatus has joined #oooq | 19:50 | |
| *** ChanServ sets mode: +v openstackstatus | 19:50 | |
| *** jmasud has quit IRC | 19:56 | |
| rlandy | weshay|ruck: taking a cut at collecting the update repo info | 20:02 |
| weshay|ruck | k | 20:03 |
| weshay|ruck | rlandy, I think ur on to something there | 20:31 |
| weshay|ruck | but let's not use podman.. use buildah.. | 20:31 |
| weshay|ruck | this works | 20:31 |
| weshay|ruck | buildah run openstack-base-working-container dnf list installed | 20:31 |
| rlandy | maybe | 20:32 |
| rlandy | weshay|ruck: needs some tweaking | 20:32 |
| rlandy | but I kind if wanted to parse that log | 20:32 |
| rlandy | and if it's empty, then nothing has updated from the update_repos | 20:32 |
| rlandy | https://opendev.org/openstack/ansible-role-collect-logs/src/branch/master/roles/collect_logs/files/collect-container-logs.sh#L71 | 20:33 |
| rlandy | copied that | 20:33 |
| rlandy | https://review.rdoproject.org/r/c/testproject/+/18953 | 20:33 |
| weshay|ruck | ya.. I think my comment is mostly around s/podman/buildah | 20:33 |
| rlandy | testprojecting ther | 20:33 |
| weshay|ruck | buildah is used there for a reason | 20:34 |
| *** slaweq has quit IRC | 20:34 | |
| rlandy | ok - easy enough change | 20:34 |
| weshay|ruck | ++ | 20:34 |
| rlandy | podman exec -u root nova_metadata bash -c '$(command -v dnf || command -v yum) list installed' | 20:36 |
| rlandy | weshay|ruck: https://logserver.rdoproject.org/53/18953/62/check/periodic-tripleo-ci-centos-8-standalone-network-master/8e7a40a/logs/undercloud/var/log/extra/podman/containers/nova_metadata/podman_info.log.txt.gz | 20:36 |
| rlandy | either should work, no? | 20:36 |
| *** slaweq has joined #oooq | 20:37 | |
| *** dmellado has quit IRC | 20:37 | |
| rlandy | can try both | 20:37 |
| *** slaweq has quit IRC | 20:37 | |
| *** dmellado has joined #oooq | 20:38 | |
| *** slaweq has joined #oooq | 20:38 | |
| *** slaweq has quit IRC | 20:40 | |
| *** slaweq has joined #oooq | 20:40 | |
| *** slaweq has quit IRC | 20:41 | |
| *** slaweq has joined #oooq | 20:42 | |
| *** slaweq has quit IRC | 20:49 | |
| *** slaweq has joined #oooq | 20:50 | |
| *** openstack has joined #oooq | 21:25 | |
| *** ChanServ sets mode: +o openstack | 21:25 | |
| *** jmasud has quit IRC | 21:26 | |
| *** jmasud has joined #oooq | 21:28 | |
| *** slaweq_ has joined #oooq | 21:29 | |
| *** slaweq has quit IRC | 21:31 | |
| *** slaweq_ has quit IRC | 21:33 | |
| *** slaweq has joined #oooq | 21:34 | |
| *** slaweq has quit IRC | 21:35 | |
| *** jmasud has quit IRC | 21:36 | |
| *** jmasud has joined #oooq | 21:37 | |
| *** slaweq has joined #oooq | 21:39 | |
| *** jmasud has quit IRC | 21:41 | |
| *** slaweq has quit IRC | 21:41 | |
| *** slaweq has joined #oooq | 21:42 | |
| *** jmasud has joined #oooq | 21:43 | |
| *** jmasud has quit IRC | 22:01 | |
| *** rlandy has quit IRC | 22:04 | |
| *** jmasud has joined #oooq | 22:05 | |
| *** jmasud has quit IRC | 22:07 | |
| *** jmasud has joined #oooq | 22:09 | |
| *** jmasud has quit IRC | 22:11 | |
| *** jmasud has joined #oooq | 22:13 | |
| *** jmasud has quit IRC | 22:15 | |
| *** fuzzball81 has quit IRC | 22:19 | |
| *** jmasud has joined #oooq | 22:23 | |
| *** tosky has quit IRC | 22:58 | |
| *** jmasud has quit IRC | 22:59 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
