| *** frenzy_friday has quit IRC | 00:04 | |
| *** frenzy_friday has joined #oooq | 00:05 | |
| *** jlarriba has quit IRC | 00:11 | |
| *** pojadhav has quit IRC | 00:16 | |
| *** pojadhav has joined #oooq | 00:18 | |
| *** rlandy_ has joined #oooq | 00:18 | |
| *** rlandy has quit IRC | 00:23 | |
| *** ysandeep has joined #oooq | 01:24 | |
| ysandeep | weshay|ruck, hey aroung o/ | 01:28 |
|---|---|---|
| ysandeep | around* | 01:28 |
| ysandeep | rlandy_, weshay|ruck: what was decided for new ovn thing in downsteam? | 01:29 |
| ysandeep | i see a comment fro jason on https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/244293 | 01:30 |
| ysandeep | that repo which provide new ovn is also present in rhos-release, we just need to enable it. | 01:30 |
| * ysandeep need to increase buffer for his bouncer.. seems i may have missed some msgs and just see we are promoting 16.2 and components | 01:35 | |
| weshay|ruck | ysandeep, oh boy | 01:35 |
| weshay|ruck | how was your sleep | 01:35 |
| weshay|ruck | ysandeep, ya.. we can't patch around that.. Jason is fixing it afaik | 01:36 |
| ysandeep | weshay|ruck: thanks, i just had 6 hour of okish sleep :) | 01:36 |
| weshay|ruck | ysandeep, well.. restate... afaik.. Jason fixed it | 01:36 |
| weshay|ruck | and we promoted w/o full successful results.. based on baremetal and waived a few others | 01:36 |
| weshay|ruck | ya.. I cant force myself to sleep much more | 01:37 |
| weshay|ruck | ysandeep, it's weird though to catch you on both sides.. | 01:37 |
| weshay|ruck | lolz | 01:37 |
| weshay|ruck | ysandeep, /me watching | 01:37 |
| weshay|ruck | https://zuul.openstack.org/status#793815 https://zuul.openstack.org/status#792904 https://zuul.openstack.org/status#793867 | 01:37 |
| ysandeep | weshay|ruck: https://code.engineering.redhat.com/gerrit/gitweb?p=rhos-release.git;a=blob;f=repos/rhos-release-rhel-8.4.repo;h=2b02247e915dfb554e616ec9cad4946dafed1d6b;hb=HEAD#l64 i see its still disabled. | 01:37 |
| weshay|ruck | we're merging a lot of patches upstream.. so that's good | 01:37 |
| ysandeep | rhelosp-fdp-pending | 01:38 |
| weshay|ruck | hrm | 01:38 |
| weshay|ruck | that's unfortunate | 01:38 |
| ysandeep | we are currently using rhosp-rhel-8.4-fdp that gives a old ovn | 01:39 |
| weshay|ruck | ysandeep, wf'd scenario003 https://review.opendev.org/c/openstack/tripleo-heat-templates/+/793968 | 01:43 |
| weshay|ruck | please watch it through | 01:43 |
| ysandeep | weshay++ thanks | 01:43 |
| ysandeep | weshay|ruck: you will cover pgrm call? i see you have added your name there.. i can if its early for you. | 01:43 |
| weshay|ruck | ya.. it's fine | 01:44 |
| ysandeep | ++ | 01:44 |
| weshay|ruck | ysandeep, OH | 01:44 |
| weshay|ruck | I know what I needed to ask.. who was asking you re: overcloud-full w/ hardcoded pass? | 01:45 |
| ysandeep | weshay|ruck, actually it was sshnaidm idea that we can set password and check what's going on http://eavesdrop.openstack.org/irclogs/%23tripleo/%23tripleo.2021-05-27.log.html#t2021-05-27T16:32:07 | 01:47 |
| ysandeep | alex suggested to set the password on overcloud-full image itself | 01:48 |
| weshay|ruck | ysandeep, ok... so here's the update | 01:48 |
| weshay|ruck | we tried a bunch of things.. but we were never able to successfully set the root passwd on the overcloud-full.qcow2 image.. We were able to set the root passwd on the centos-stream-8 image overcloud-full is based on | 01:49 |
| weshay|ruck | and boot it and login | 01:49 |
| weshay|ruck | ysandeep, arxcruz|rover I suppose late in our day.. but as we're trying to debug this thing | 01:49 |
| weshay|ruck | hrm..https://zuul.openstack.org/status#792904 | 01:53 |
| weshay|ruck | failed again | 01:53 |
| ysandeep | weshay|ruck: limestone issues are ack :) | 01:54 |
| ysandeep | back* | 01:54 |
| weshay|ruck | 2021-06-02 01:35:40 | [FAILED] os-apply-config-13.0.1-0.20210226003419.891e996.el8.noarch.rpm: No more mirrors to try - All mirrors were already tried without success | 01:54 |
| weshay|ruck | ysandeep, yup 2021-06-02 01:23:31.556571 | localhost | Provider: limestone-regionone | 01:55 |
| weshay|ruck | ysandeep, updated https://trello.com/c/2KRxJzov/1960-cixlp1929461tripleociproa-yum-repo-mirrors-are-down-5-24-2021 | 01:56 |
| weshay|ruck | ysandeep, abandon / restart | 01:56 |
| ysandeep | weshay|ruck, ^^ only commiter/core can abandon / restore | 02:07 |
| weshay|ruck | ysandeep, ah.. we'll need to get you core... | 02:08 |
| * weshay|ruck fixes | 02:08 | |
| weshay|ruck | ysandeep, ok.. reset | 02:09 |
| ysandeep | weshay|ruck: thanks! | 02:09 |
| ysandeep | weshay|ruck: good night zzz..., i am afk for sometime. | 02:11 |
| *** ysandeep is now known as ysandeep|afk | 02:11 | |
| *** frenzy_friday has quit IRC | 02:53 | |
| *** frenzy_friday has joined #oooq | 02:54 | |
| *** ykarel|away has joined #oooq | 03:51 | |
| *** ykarel|away is now known as ykarel | 04:04 | |
| *** ysandeep|afk is now known as ysandeep|ruck | 04:57 | |
| *** bhagyashris_ has joined #oooq | 05:03 | |
| *** marios has joined #oooq | 05:04 | |
| *** bhagyashris has quit IRC | 05:10 | |
| *** rlandy_ has quit IRC | 05:14 | |
| *** bhagyashris_ has quit IRC | 05:47 | |
| *** jbadiapa has quit IRC | 05:50 | |
| ysandeep|ruck | frenzy_friday, hey good morning, do you know where I can check which files we are including in logstash.txt? | 06:07 |
| ysandeep|ruck | I want a add a new tripleo-ci-health rule that capture the current failure from nhttps://e76b8a042504872b0b7d-f18cd0734c7002742f0baf633d0573f2.ssl.cf1.rackcdn.com/792904/5/check/tripleo-ci-centos-8-containers-multinode/54821d1/logs/undercloud/var/log/tempest/tempest_run.log | 06:08 |
| ysandeep|ruck | looks like we are not including tempest_run.log in https://e76b8a042504872b0b7d-f18cd0734c7002742f0baf633d0573f2.ssl.cf1.rackcdn.com/792904/5/check/tripleo-ci-centos-8-containers-multinode/54821d1/logs/undercloud/var/log/extra/logstash.txt | 06:09 |
| *** bhagyashris has joined #oooq | 06:15 | |
| *** jbadiapa has joined #oooq | 06:34 | |
| *** bhagyashris has quit IRC | 07:23 | |
| *** bhagyashris has joined #oooq | 07:23 | |
| *** amoralej|off is now known as amoralej | 07:24 | |
| *** bhagyashris_ has joined #oooq | 07:38 | |
| ysandeep|ruck | sshnaidm, https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/792904/ failed in check for molecule job, may be abandon/restore to save time. | 07:38 |
| *** ysandeep|ruck is now known as ysandeep|lunch | 07:41 | |
| *** bhagyashris has quit IRC | 07:44 | |
| arxcruz|rover | ysandeep|lunch: namaste | 07:55 |
| ysandeep|lunch | arxcruz|rover, o/ hey namaste and bom Dia | 07:56 |
| *** jpena|off is now known as jpena | 07:58 | |
| * ysandeep|lunch stepping out for lunch | 08:00 | |
| arxcruz|rover | ysandeep|lunch: i don't know about our health, but tempest have a subunit file that upstream health uses to parse the tests | 08:04 |
| *** ykarel_ has joined #oooq | 08:07 | |
| sshnaidm | ysandeep|lunch, ack.. even molecule jobs fail :/ | 08:07 |
| *** ykarel has quit IRC | 08:13 | |
| *** ykarel_ is now known as ykarel | 08:16 | |
| *** derekh has joined #oooq | 08:21 | |
| frenzy_friday | Hey ysandeep|lunch , lemme check | 08:27 |
| frenzy_friday | Is it https://opendev.org/openstack/ansible-role-collect-logs/src/branch/master/roles/collect_logs/defaults/main.yml ? But I think this is where sova looks | 08:34 |
| frenzy_friday | Hey zbr, is https://opendev.org/openstack/ansible-role-collect-logs/src/branch/master/roles/collect_logs/defaults/main.yml the place where we define fines to be included in logstash? | 08:35 |
| zbr | yep, there is also an exclude list there | 08:36 |
| zbr | look for artcl_logstash_files | 08:36 |
| *** ykarel is now known as ykarel|lunch | 08:38 | |
| *** ykarel_ has joined #oooq | 08:47 | |
| *** ykarel|lunch has quit IRC | 08:47 | |
| *** ysandeep|lunch is now known as ysandeep | 09:04 | |
| *** ysandeep is now known as ysandeep|ruck | 09:04 | |
| ysandeep|ruck | frenzy_friday, zbr thanks! iiuc.. I need to first add /var/log/tempest/tempest_run.log under artcl_logstash_files so that logstash pick them up, before adding new tripleo-ci-health rule. | 09:14 |
| *** ykarel_ has quit IRC | 09:15 | |
| ysandeep|ruck | may be better to include var/log/extra/errors.txt instead | 09:16 |
| * ysandeep|ruck will post a patch | 09:24 | |
| zbr | yamamoto: 2nd option better, tempest log can be huge. | 09:41 |
| ysandeep|ruck | yamamoto: ignore i think that was for me :) | 09:42 |
| *** ykarel_ has joined #oooq | 10:20 | |
| *** ykarel_ is now known as ykarel | 10:21 | |
| zbr | warning to others, I am ISO 8601 addict, https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/33855 ;) | 10:21 |
| zbr | probably ptsd after working in i18n for several years | 10:22 |
| ysandeep|ruck | we have iso for date format.. interesting :) | 10:23 |
| *** yamamoto has quit IRC | 10:23 | |
| *** jlarriba has joined #oooq | 10:44 | |
| arxcruz|rover | ykarel: around? | 10:45 |
| arxcruz|rover | ykarel: how can we downgrade the cloud-init on overcloud image to 20.3-10.el8 instead of 20.3-10.el8_4.2 ? | 10:45 |
| ykarel | arxcruz|rover, you mean downgrade just for testing? | 10:55 |
| ykarel | is there come issue with el8_4.2 | 10:55 |
| ykarel | in stream repo i see both versions are there http://mirror.centos.org/centos/8-stream/AppStream/x86_64/os/Packages/ | 10:56 |
| ykarel | so downgrade should work until a new release is published, as c-stream keeps only last two versions | 10:56 |
| ykarel | if there is an issue, is bz open for it, so it's getting tracked and closed | 10:57 |
| *** yamamoto has joined #oooq | 11:00 | |
| arxcruz|rover | ykarel: another quest, i've been in contact with the cloud-init guy and he said there's no 8_4.2 release on their side | 11:01 |
| arxcruz|rover | there is 8_4.1 and 8_4.3 | 11:02 |
| arxcruz|rover | ykarel: https://gitlab.com/redhat/rhel/src/cloud-init/-/commits/rhel-8.4.0/ | 11:02 |
| arxcruz|rover | ykarel: and there are two big patches related to open vswitch | 11:02 |
| *** akahat has joined #oooq | 11:04 | |
| ykarel | ^ returns 404 | 11:05 |
| ykarel | anyway it's c8-stream so https://git.centos.org/rpms/cloud-init/c/8855a08dfb6587b0cb78a1420b82d4f15b4d678e?branch=c8s | 11:07 |
| ykarel | so changelog have both 20.3-10.el8_4.2 and 20.3-10.el8_4.3 | 11:07 |
| ykarel | but builds have _4.2 https://koji.mbox.centos.org/koji/packageinfo?packageID=235, need to check with centos-devel why _4.2 got build but not _4.3 | 11:08 |
| ykarel | arxcruz|rover, ^, _4.3 don't have issue and _4.2 have the issue? | 11:10 |
| arxcruz|rover | ykarel: you need to link your redhat account to gitlab | 11:10 |
| arxcruz|rover | ykarel: https://red.ht/GitLabSSO | 11:10 |
| arxcruz|rover | ykarel: the latest run passing we have , had the 20.3-10.el8 | 11:10 |
| arxcruz|rover | now we have 20.3-10.el8_4.2 | 11:11 |
| arxcruz|rover | but according the guy from cloud-init they did not release 20.3-10.el8_4.2 only 20.3-10.el8_4.1 and 20.3-10.el8_4.3 | 11:11 |
| arxcruz|rover | ykarel: right now i can't tell you if the issue is cloud-init or not | 11:11 |
| arxcruz|rover | but having changes on cloud-init related to openvswitch might worth investigate | 11:12 |
| ykarel | okk after https://red.ht/GitLabSSO i can open that gitlab link | 11:12 |
| *** soniya29 has joined #oooq | 11:13 | |
| ykarel | for reasons on why _4.3 is not built, can ask on #centos-devel | 11:13 |
| arxcruz|rover | ykarel coincidentally the date the 4.2 was released was when we started to see failures | 11:14 |
| arxcruz|rover | cloud-init-20.3-10.el8_4.2 hughesjr 2021-05-27 15:46:10 | 11:14 |
| *** yamamoto has quit IRC | 11:15 | |
| ykarel | ok so u can try with downgrade and see if this helps and in parallel reasons for why _4.3 is not built can be find out | 11:16 |
| arxcruz|rover | ykarel: is there a easy way to do it ? | 11:17 |
| ykarel | arxcruz|rover, u checked logs for build image to see if it's getting installed/updated there | 11:18 |
| ykarel | i think can modify image elements to do downgrade | 11:18 |
| ykarel | and there ovb should trigger and test the modified image | 11:19 |
| *** soniya29 has quit IRC | 11:21 | |
| arxcruz|rover | ykarel: it is https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-full-master/21cedcd/build.log | 11:21 |
| *** soniya29 has joined #oooq | 11:21 | |
| soniya29 | arxcruz|rover, ysandeep|ruck, you both are ruck/rovering and chandan is also on leave so shall we skip the meeting for today or do you want to discuss anything? | 11:24 |
| soniya29 | s/meeting/tempest meeting | 11:25 |
| arxcruz|rover | soniya29: yeah, please | 11:25 |
| soniya29 | kopecmartin, akahat ^^ | 11:25 |
| akahat | soniya29: okay with me | 11:26 |
| kopecmartin | soniya29: sure, fine by me | 11:27 |
| ykarel | arxcruz|rover, yes cloud-init noarch 20.3-10.el8_4.2 appstream 1.0 M | 11:27 |
| *** yamamoto has joined #oooq | 11:27 | |
| ykarel | so it's getting upgraded, so with exclude you could skip the upgrade and test | 11:27 |
| ykarel | exclude in repo | 11:28 |
| soniya29 | arxcruz|rover, kopecmartin, ysandeep|ruck, akahat, weshay|ruck, Okay | 11:29 |
| ykarel | can try in rdo-jobs playbooks after configure-mirrors | 11:29 |
| arxcruz|rover | ykarel: like we did with the e2dk right ? | 11:29 |
| ykarel | arxcruz|rover, yes right | 11:30 |
| *** yamamoto has quit IRC | 11:30 | |
| arxcruz|rover | weshay|ruck: 1-1 ? | 11:32 |
| arxcruz|rover | i mean ruck rover sync | 11:32 |
| *** jpena is now known as jpena|lunch | 11:33 | |
| *** yamamoto has joined #oooq | 11:37 | |
| arxcruz|rover | ysandeep|ruck: so, i think i should add here https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/tripleo-repos/tasks/main.yml#L77 | 11:45 |
| arxcruz|rover | right ? | 11:45 |
| *** yamamoto has quit IRC | 11:45 | |
| ysandeep|ruck | arxcruz|rover, checking cloud-int comes from which repo | 11:46 |
| *** yamamoto has joined #oooq | 11:46 | |
| arxcruz|rover | ysandeep|ruck: it might not be this one, let me check | 11:47 |
| *** rlandy has joined #oooq | 11:48 | |
| arxcruz|rover | ysandeep|ruck: 2021-06-02 01:57:50.151 | cloud-init noarch 20.3-10.el8_4.2 appstream 1.0 M | 11:50 |
| ysandeep|ruck | arxcruz|rover, its comes from appstream http://paste.openstack.org/show/806268/ | 11:50 |
| *** soniya29 has quit IRC | 11:50 | |
| ysandeep|ruck | checking image build use quickstart-centos-appstreams or appstreams which comes from configure-mirror | 11:51 |
| ysandeep|ruck | image build use appstreams which comes from configure-mirror : https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-full-master/627c7f4/yum.repos.d/ | 11:54 |
| rlandy | ysandeep|ruck: hey - we promoted 16.2 yesterday at Jason's request | 11:55 |
| ysandeep|ruck | rlandy, yeah i saw that thanks!.. i am not sure what should we do about below:- | 11:56 |
| ysandeep|ruck | <jjoyce> weshay|ruck ysandeep shreshthaway: We don't want fdp-pending enabled by default in rhos-release. It would have to be enabled after setup so that it's clear we are testing pre-release content. Typically we only want cdn content tested. | 11:56 |
| rlandy | ysandeep|ruck: merge your change then | 11:56 |
| rlandy | ysandeep|ruck: also add it to 17 | 11:57 |
| rlandy | we have no other choice | 11:57 |
| ysandeep|ruck | rlandy, yeah but weshay|ruck was against that.. lets chat with him when he comes online | 11:57 |
| rlandy | ysandeep|ruck: the question is what will QE do. | 11:57 |
| rlandy | ysandeep|ruck: I was against it as well | 11:57 |
| rlandy | but that is exactly what jason is requesting | 11:57 |
| rlandy | ysandeep|ruck: we could raise it at the platform call | 11:58 |
| ysandeep|ruck | true.. | 11:58 |
| akahat | Reviews requested: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/33913, https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/33855 | 11:59 |
| ysandeep|ruck | rlandy, weshay|ruck was suppose to cover pgrm call.. i think he is not online yet.. i will join | 11:59 |
| rlandy | ysandeep|ruck: yep - I'll join as well | 12:00 |
| weshay|ruck | ysandeep|ruck, rlandy I got it | 12:00 |
| rlandy | weshay|ruck: hey - morning | 12:00 |
| weshay|ruck | ysandeep|ruck, arxcruz|rover ha... sorry I forgot I called a sync a 1/2 hour ago | 12:00 |
| ysandeep|ruck | weshay|ruck, morning no worries, we can chat later in the day | 12:01 |
| arxcruz|rover | ysandeep|ruck: weshay|ruck want to sync now ? | 12:01 |
| ysandeep|ruck | arxcruz|rover, weshay|ruck in prgm call, we can do later | 12:01 |
| arxcruz|rover | ysandeep|ruck: outch | 12:02 |
| arxcruz|rover | ysandeep|ruck: want me to report? | 12:02 |
| weshay|ruck | arxcruz|rover, as soon as I give our status we can sync | 12:02 |
| arxcruz|rover | weshay|ruck: ok | 12:02 |
| weshay|ruck | probably 28 min | 12:02 |
| weshay|ruck | ysandeep|ruck, what is the review # for the limestone fix.. https://review.opendev.org/c/openstack/project-config/+/79414 is not valid | 12:06 |
| ysandeep|ruck | weshay|ruck, https://review.opendev.org/c/openstack/project-config/+/794142 | 12:06 |
| weshay|ruck | ysandeep|ruck++ | 12:07 |
| *** bhagyashris_ is now known as bhagyashris | 12:09 | |
| weshay|ruck | ysandeep|ruck, arxcruz|rover any test projects on scen003 master? | 12:10 |
| weshay|ruck | https://review.opendev.org/c/openstack/tripleo-heat-templates/+/793968 merged | 12:10 |
| *** akahat1 has joined #oooq | 12:10 | |
| ysandeep|ruck | weshay|ruck, already tested via testproject yesterday.. that fixes the issue.. will recheck for promotion. | 12:11 |
| arxcruz|rover | marios: let's discuss here :) | 12:11 |
| *** akahat has quit IRC | 12:11 | |
| marios | arxcruz|rover: ;D | 12:11 |
| marios | the freenode chans are first | 12:11 |
| marios | sory about that i didn't even notice | 12:11 |
| weshay|ruck | ysandeep|ruck, that's good :) we may need to promote tripleo component in master | 12:11 |
| marios | 15:09 < marios> ysandeep: arxcruz|rover: any point recheck for the ovb-controller container build issue or are we waitng for something first | 12:11 |
| marios | 15:10 < marios> ysandeep: arxcruz|rover: eg there | 12:11 |
| marios | https://9b6014e80e764b848f3d-c29773bdeee4530a738751d9e026e2a7.ssl.cf1.rackcdn.com/793144/1/check/tripleo-ci-centos-8-content-provider/ddcf088/logs/report.html | 12:11 |
| marios | i mean ovn-controller not ovb | 12:12 |
| jlarriba | hi guys, is there an issue with tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001 in RDO gate? | 12:12 |
| arxcruz|rover | jlarriba: yes, there is :) | 12:12 |
| marios | arxcruz|rover: ysandeep|ruck: do you have a hackmd/somewhere that lists your bugs? | 12:12 |
| ysandeep|ruck | marios, that was fixed yesterday | 12:12 |
| *** amoralej is now known as amoralej|lunch | 12:12 | |
| rlandy | weshay|ruck: ysandeep|ruck: periodic-tripleo-ci-rhel-8-standalone-on-multinode-ipa-security-rhos-17 (non-voting)openstack/tripleo-cimasteropenstack-component-security1 hr 30 mins 34 secs2021-05-31 16:24:32SUCCESS | 12:12 |
| marios | ysandeep|ruck: thanks so i can recheck | 12:12 |
| jlarriba | arxcruz|rover: ok, that is all the conversation i have been looking at this morning, right? | 12:12 |
| rlandy | ipa jobs should be working now in 17 | 12:12 |
| ysandeep|ruck | marios, fyi.. https://bugs.launchpad.net/tripleo/+bug/1930327 | 12:13 |
| opendevmeet | Launchpad bug 1930327 in tripleo "Container build is failing with error: /usr/share/ovn/scripts/ovndb-servers.ocf from install of ovn-2021-21.03.0-21.el8s.x86_64 conflicts with file from package ovn2.13-20.12.0-85.el8s.x86_64" [Critical,Triaged] | 12:13 |
| marios | thanks ysandeep|ruck | 12:13 |
| arxcruz|rover | jlarriba: sorry, i didn't get it... we are having issues on that job, we are investigating | 12:13 |
| arxcruz|rover | jlarriba: did meant to be rude, sorry if you understand that way | 12:13 |
| arxcruz|rover | jlarriba: we don't know yet exactly the root cause that's why i can't tell you for sure, but be sure we are working on that | 12:14 |
| jlarriba | arxcruz|rover: ack, you definitively wasn't rude. I have been reading this chan all monring and I thought the conversation was around trying to identify exaclty this error | 12:14 |
| jlarriba | arxcruz|rover: thanks for confirming | 12:14 |
| arxcruz|rover | jlarriba: yeah, we are trying to identify the root cause :) | 12:14 |
| arxcruz|rover | ysandeep|ruck: weshay|ruck https://review.opendev.org/c/openstack/tripleo-ci/+/794208 | 12:19 |
| weshay|ruck | arxcruz|rover, https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-full-master/627c7f4/yum.repos.d/ | 12:21 |
| arxcruz|rover | ysandeep|ruck: ? | 12:22 |
| weshay|ruck | arxcruz|rover, comment on ur patch | 12:23 |
| arxcruz|rover | i mean weshay|ruck ^ | 12:23 |
| weshay|ruck | ping me w/ questions | 12:23 |
| arxcruz|rover | weshay|ruck: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-full-master/627c7f4/job-output.txt | 12:24 |
| arxcruz|rover | it does use tripleo-repos | 12:24 |
| weshay|ruck | arxcruz|rover, ysandeep|ruck ready | 12:25 |
| weshay|ruck | arxcruz|rover, look at the repos | 12:26 |
| ysandeep|ruck | yes | 12:26 |
| ysandeep|ruck | weshay|ruck, arxcruz|rover meet.google.com/qif-xuzg-fdc | 12:26 |
| *** jpena|lunch is now known as jpena | 12:32 | |
| *** akahat1 is now known as akahat | 12:33 | |
| ysandeep|ruck | weshay|ruck, https://review.rdoproject.org/r/c/testproject/+/32054 | 12:56 |
| *** soniya29 has joined #oooq | 13:04 | |
| arxcruz|rover | ysandeep|ruck: let me know when you have the patch up | 13:08 |
| ysandeep|ruck | arxcruz|rover, on it | 13:08 |
| *** amoralej|lunch is now known as amoralej | 13:11 | |
| weshay|ruck | akahat, will need a master promotion today w/ a few jobs removed from criteria | 13:20 |
| weshay|ruck | akahat, details coming | 13:21 |
| weshay|ruck | akahat, master... skip periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master periodic-tripleo-ci-centos-8-scenario003-standalone-master periodic-tripleo-ci-centos-8-scenario010-kvm-standalone-master periodic-tripleo-ci-centos-8-standalone-full-tempest-api-master periodic-tripleo-ci-centos-8-standalone-upgrade-master | 13:22 |
| weshay|ruck | akahat, please promote, after promotion please set it back to how it was originally | 13:23 |
| weshay|ruck | frenzy_friday, any idea why http://health.sbarnea.com/#1926764 hasn't been deleted? | 13:25 |
| frenzy_friday | weshay|ruck, yeah.. right now to delete a query we have to manually detele it from queries.yml+ delete the <id>.yml I am checking what I can fix so that only a change in queries.yml is enough | 13:26 |
| bhagyashris | weshay|ruck, hi, you want me to do the promotion stuff if it's urgent | 13:28 |
| bhagyashris | i saw your messages | 13:28 |
| weshay|ruck | bhagyashris, yes please | 13:29 |
| *** ykarel has quit IRC | 13:30 | |
| *** ykarel has joined #oooq | 13:30 | |
| ysandeep|ruck | arxcruz|rover, rpm changelog.. http://paste.openstack.org/show/806273/ does it contain the ovn change you were talking about.. | 13:31 |
| arxcruz|rover | ysandeep|ruck: yeah... | 13:32 |
| arxcruz|rover | ysandeep|ruck well, where we are right now doesn't hurt to check anyway | 13:32 |
| arxcruz|rover | ysandeep|ruck: we started to see the failure when we got this new release | 13:32 |
| ysandeep|ruck | true.. patching incoming | 13:32 |
| arxcruz|rover | at least is too much coincidence to let it pass | 13:32 |
| bhagyashris | akahat, weshay|ruck done | 13:34 |
| sshnaidm | ysandeep|ruck, weshay|ruck https://review.rdoproject.org/r/c/testproject/+/28004 | 13:40 |
| ysandeep|ruck | arxcruz|rover: https://review.opendev.org/c/openstack/tripleo-quickstart/+/794225 | 13:40 |
| weshay|ruck | bhagyashris, thanks see it | 13:41 |
| bhagyashris | weshay|ruck, http://10.0.148.74/promoter_logs/container-push/20210602-133305.log | 13:42 |
| weshay|ruck | ya | 13:42 |
| rlandy | weshay|ruck: periodic-tripleo-ci-rhel-8-bm_envC-3ctlr_1comp-featureset001-tripleo-rhos-16.2openstack/tripleo-cimasteropenstack-component-tripleo4 hrs 4 mins 28 secs2021-05-31 20:53:52SUCCESS\ | 13:42 |
| rlandy | passed yesterday | 13:42 |
| rlandy | so fs010 is the consistent failure | 13:42 |
| *** soniya29 has quit IRC | 13:43 | |
| weshay|ruck | frenzy_friday, ah.. that makes sense | 13:50 |
| weshay|ruck | frenzy_friday, /me wonders if tox could do that for us... | 13:50 |
| frenzy_friday | weshay|ruck, yes, dont change your patch. I am putting up a fix | 13:51 |
| weshay|ruck | rock on | 13:52 |
| *** tosky has joined #oooq | 13:54 | |
| ysandeep|ruck | weshay|ruck, arxcruz|rover testing ovb after cloud-int pinning here https://review.rdoproject.org/r/c/testproject/+/28458 with Depends-On: https://review.opendev.org/c/openstack/tripleo-quickstart/+/794225 | 13:55 |
| *** zbr has quit IRC | 13:55 | |
| *** zbr has joined #oooq | 13:55 | |
| arxcruz|rover | ysandeep|ruck: ack | 13:57 |
| rlandy | weshay|ruck: want to hang on after the call - touch base re: tripleo component? | 14:00 |
| marios | zbr: when you next have some time can you please checkout https://bugs.launchpad.net/tripleo/+bug/1930416 (possibly related to https://bugs.launchpad.net/tripleo/+bug/1921679 but see my latest comment at 1930416) thank you | 14:14 |
| opendevmeet | Launchpad bug 1930416 in tripleo "master undercloud upgrade fails "not an openstack command"" [Undecided,Triaged] - Assigned to Marios Andreou (marios-b) | 14:14 |
| opendevmeet | Launchpad bug 1921679 in tripleo "openstack: 'undercloud install' is not an openstack command. See 'openstack --help'." [Critical,Triaged] | 14:14 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/buildset/deb2035c4d5649eda771df9b511a522e bets last run 05/20 | 14:15 |
| ysandeep|ruck | weshay|ruck, rlandy: please revisit https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/244293 to fix downstream ovn situation | 14:15 |
| ysandeep|ruck | rlandy, for 16.2 i recall tripleo-component was blocked on infrared job | 14:18 |
| bhagyashris | weshay|ruck, ysandeep|ruck arxcruz|rover master promoted https://trunk.rdoproject.org/centos8-master/current-tripleo/delorean.repo.md5 | 14:18 |
| ysandeep|ruck | bhagyashris++ | 14:18 |
| rlandy | ysandeep|ruck: yes - but we have not promoted tripleo component since 05/07 | 14:18 |
| arxcruz|rover | bhagyashris: \o/ | 14:18 |
| akahat | weshay|ruck: hey.. sorry i missed your msg.. | 14:18 |
| bhagyashris | reverting back changes | 14:18 |
| akahat | bhagyashris: thanks | 14:19 |
| rlandy | and we have failing multinode job | 14:19 |
| rlandy | ysandeep|ruck: last pass on that was 05/20 | 14:19 |
| rlandy | looking into that | 14:19 |
| ysandeep|ruck | rlandy, where? | 14:19 |
| ysandeep|ruck | rlandy, multinode passes: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/cbe1462b24c3464fa3dcc9f639bfea12 | 14:19 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-8-multinode-1ctlr-featureset010-tripleo-rhos-16.2 | 14:20 |
| rlandy | ysandeep|ruck: you pointed to the last pass I referenced on 05/20 | 14:20 |
| rlandy | which was the last time multinode passed | 14:20 |
| rlandy | on that occasion bm failed | 14:21 |
| rlandy | so we could ignore QE | 14:21 |
| rlandy | (pinged attila) | 14:21 |
| rlandy | but we have our own problems | 14:21 |
| rlandy | working on it | 14:21 |
| ysandeep|ruck | ack o/ | 14:22 |
| ysandeep|ruck | With upstream on fires.. i hardly got time to look at 16.2 | 14:22 |
| rlandy | 2021-05-30 18:01:26 | Timed out waiting for port 22 from 192.168.24.3 | 14:22 |
| rlandy | ysandeep|ruck: np - on it | 14:22 |
| rlandy | 021-05-30 18:01:26 | Overcloud Deployed with error | 14:23 |
| rlandy | 2021-06-01 18:31:28 | Enabling ssh admin (tripleo-admin) for hosts:Host 192.168.24.13 not found in /home/zuul/.ssh/known_hosts | 14:24 |
| rlandy | 2021-06-01 18:31:28 | Timed out waiting for port 22 from 192.168.24.3 | 14:24 |
| ysandeep|ruck | train multinode working? checking | 14:26 |
| rlandy | definitely dying there | 14:28 |
| rlandy | ysandeep|ruck: rerun in progress with latest promoted | 14:34 |
| ysandeep|ruck | rlandy, ack thanks! | 14:34 |
| rlandy | ysandeep|ruck: weshay|ruck: the only two components in 16.2 that are behind are network and tripleo | 14:35 |
| rlandy | network is running now | 14:35 |
| rlandy | so not so bad | 14:35 |
| ysandeep|ruck | ack o/ rlandy++ | 14:35 |
| weshay|ruck | thanks | 14:43 |
| rlandy | ysandeep|ruck: weshay|ruck: so the network component will continue to fail tempest ... waiting on ovn | 14:43 |
| rlandy | if we don;t see jason's new build soon, we can reconsider running with ysandeep|ruck patch to promote that | 14:44 |
| *** ykarel is now known as ykarel|away | 14:44 | |
| ysandeep|ruck | rlandy: i think we can merge https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/244293 and revert later.. once pcd fixes it | 14:44 |
| rlandy | ysandeep|ruck: I'm ok with it - weshay|ruck still has a -1 | 14:45 |
| rlandy | it's an easy enough revert | 14:45 |
| rlandy | periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset010-tripleo-trainopenstack/tripleo-cimasteropenstack-component-tripleo1 hr 41 mins 14 secs2021-06-01 21:53:01SUCCESS | 14:48 |
| rlandy | fine in train | 14:48 |
| ysandeep|ruck | weshay|ruck: you want to revisit https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/244293 , we revert when pcd fixes | 14:51 |
| weshay|ruck | bhagyashris, akahat master promoted.. please set the criteria jobs back to how it was | 15:08 |
| akahat | weshay|ruck: https://review.rdoproject.org/r/c/rdo-jobs/+/33574 | 15:09 |
| *** ykarel|away has quit IRC | 15:11 | |
| weshay|ruck | akahat, did I loose u? | 15:19 |
| weshay|ruck | akahat, catch up w/ you later | 15:20 |
| weshay|ruck | ALL.. quick errand bbiab | 15:20 |
| *** akahat1 has joined #oooq | 15:20 | |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-multinode-1ctlr-featureset010-rhos-16.2/9be07be/logs/undercloud/home/zuul/overcloud_deploy.log | 15:23 |
| rlandy | ysandeep|ruck: ^^ we have the same failure in the integration line now | 15:24 |
| rlandy | multinode is hosed | 15:25 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-8-multinode-1ctlr-featureset010-rhos-16.2 | 15:25 |
| rlandy | last siccess 05/27 | 15:25 |
| ysandeep|ruck | may be some network issue.. | 15:26 |
| rlandy | wonder if we have multinode passing anywhere downstream | 15:27 |
| *** akahat has quit IRC | 15:27 | |
| weshay|ruck | back | 15:33 |
| ysandeep|ruck | weshay|ruck, please revisit https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/244293 to fix downstream ovn situation , we revert when pcd fixes it. | 15:34 |
| ysandeep|ruck | unless we discuss anything else in prod chain mtg | 15:34 |
| *** akahat1 has quit IRC | 15:36 | |
| ysandeep|ruck | rlandy: if integration line is also affected, we can neglect that tripleo component is a cause.. we haven't promoted that in ~ a month. | 15:36 |
| rlandy | ysandeep|ruck: I gather it's from when the cloud was down and came back | 15:37 |
| rlandy | looking at the dates | 15:37 |
| rlandy | periodic-tripleo-ci-rhel-8-multinode-1ctlr-featureset010-rhos-16.2openstack/tripleo-cimasteropenstack-periodic-integration-rhos-16.20000000 | 15:37 |
| rlandy | 1 hr 40 mins 20 secs2021-05-27 22:17:04SUCCESS | 15:37 |
| rlandy | suspicious though | 15:38 |
| ysandeep|ruck | ssh from undercloud to subnode 1 is working | 15:38 |
| ysandeep|ruck | but i tried public IP | 15:38 |
| ysandeep|ruck | deployment is trying.. ip which set on br-ex Timed out waiting for port 22 from 192.168.24.3 | 15:39 |
| weshay|ruck | ysandeep|ruck, arxcruz|rover master promoted | 15:39 |
| ysandeep|ruck | weshay|ruck, woot!, sc03 and api master also passed in my testproject so i think we are good | 15:40 |
| rlandy | https://review.opendev.org/c/openstack/python-tripleoclient/+/701544/ | 15:40 |
| rlandy | we should we able to tune that value | 15:41 |
| rlandy | maybe we just give it some wiggle room | 15:41 |
| weshay|ruck | rlandy, https://review.opendev.org/c/openstack/tripleo-ci/+/794208 | 15:41 |
| ysandeep|ruck | rlandy, the testproject you triggered is progressing good | 15:43 |
| rlandy | weshay|ruck: https://review.opendev.org/c/openstack/tripleo-ci/+/794208 - why add in both places? | 15:43 |
| weshay|ruck | bhagyashris, you still around? we can promote victoria w/ https://review.rdoproject.org/zuul/buildset/5ddb9b01e6864c469df4adb714fa3072 | 15:44 |
| ysandeep|ruck | rlandy: testpatch already passed the failing step | 15:44 |
| weshay|ruck | ovb issues are known.. arxcruz|rover and ysandeep|ruck are working it out | 15:44 |
| rlandy | let's see | 15:45 |
| rlandy | no celebration yet | 15:45 |
| rlandy | ysandeep|ruck: may be sporadic | 15:45 |
| rlandy | if we up the two timeouts | 15:45 |
| rlandy | we may get better performance | 15:45 |
| rlandy | trying that | 15:45 |
| ysandeep|ruck | rlandy, wes merged my ovn patch in release.. | 15:45 |
| rlandy | yay! | 15:46 |
| rlandy | happiness is | 15:46 |
| rlandy | will need to rerun network line | 15:46 |
| rlandy | weshay|ruck: ^^ you have that power?? | 15:46 |
| weshay|ruck | yup.. doing so now | 15:46 |
| rlandy | rekick the network component line for 16.2 | 15:46 |
| rlandy | thank you, sir | 15:46 |
| rlandy | save me a testproject | 15:46 |
| ysandeep|ruck | rlandy: but lets not wave QE job for tripleo component.. that's a legit issue | 15:47 |
| ysandeep|ruck | weshay|ruck, ^^ fyi.. | 15:47 |
| weshay|ruck | rlandy, internal right? | 15:47 |
| rlandy | weshay|ruck: ack | 15:47 |
| rlandy | ysandeep|ruck: if we can get tripleo component clean on our side, I think we will need to | 15:47 |
| rlandy | promote | 15:47 |
| rlandy | alex needs the update | 15:47 |
| rlandy | maybe we can offer QE the same solution | 15:47 |
| weshay|ruck | rlandy, it's running now.. should I kill and restart? | 15:48 |
| ysandeep|ruck | i don't get it that's not a solution... integration line will break if tripleo promotes | 15:48 |
| weshay|ruck | oh.. wrong zuul | 15:48 |
| weshay|ruck | ah.. both are running | 15:48 |
| weshay|ruck | rlandy, kill && restart? | 15:49 |
| rlandy | yep | 15:49 |
| rlandy | we need the env file updates | 15:49 |
| rlandy | to get component network through tempest failures | 15:49 |
| rlandy | so ofcourse the test project is passing | 15:50 |
| rlandy | the one we kicked to debug failure :) | 15:50 |
| rlandy | lol | 15:50 |
| ysandeep|ruck | rlandy, https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/792904 is in gate .. once it merges.. | 15:50 |
| ysandeep|ruck | sshnaidm ^^ will revert his distgit patches | 15:51 |
| ysandeep|ruck | and we can promote tripleo component | 15:51 |
| rlandy | k | 15:51 |
| * sshnaidm is praying | 15:51 | |
| rlandy | amen | 15:52 |
| ysandeep|ruck | :) | 15:52 |
| rlandy | ysandeep|ruck: to check ... promote tripleo component in 16.2 | 15:52 |
| rlandy | to fix QE's issue? | 15:52 |
| rlandy | or am I mixed up here | 15:52 |
| * rlandy losing it today | 15:52 | |
| ysandeep|ruck | rlandy, yes | 15:52 |
| rlandy | ok | 15:53 |
| rlandy | ysandeep|ruck: lol - this multinode test has not passed in a month | 15:54 |
| rlandy | now it does | 15:54 |
| rlandy | when we are watching it | 15:54 |
| ysandeep|ruck | https://review.rdoproject.org/r/c/openstack/tripleo-ansible-distgit/+/33840 tripleo-ansible related changes which need revert. | 15:54 |
| sshnaidm | rlandy, weshay|ruck please vote https://review.opendev.org/c/openstack/tripleo-ci/+/794194 | 15:54 |
| ysandeep|ruck | rlandy, i have tried ssh from undercloud to subnode1 before overcloud deployment started.. hoping that don't changed anything... | 15:55 |
| rlandy | hmmm | 15:55 |
| rlandy | maybe | 15:55 |
| rlandy | still going to test the two timeout vars | 15:56 |
| ysandeep|ruck | because in failing job.. it complain Host 192.168.24.19 not found in /home/zuul/.ssh/known_hosts ... before imed out waiting for port 22 from 192.168.24.3 | 15:56 |
| *** ysandeep|ruck is now known as ysandeep|dinner | 15:57 | |
| rlandy | we will have to run again | 15:58 |
| weshay|ruck | ugh... not working | 16:00 |
| *** marios has quit IRC | 16:00 | |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-multinode-1ctlr-featureset010-tripleo-rhos-16.2/5aa1a79/logs/undercloud/etc/hosts | 16:03 |
| rlandy | 192.168.101.41 secondary | 16:03 |
| rlandy | 192.168.101.41 node-0000284817 | 16:03 |
| rlandy | 192.168.24.19 no where | 16:03 |
| bhagyashris | weshay|ruck, hey i set back the changes | 16:04 |
| rlandy | https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset010-tripleo-train/414564c/logs/undercloud/etc/hosts.txt.gz | 16:04 |
| rlandy | hosts files | 16:05 |
| bhagyashris | for master | 16:06 |
| bhagyashris | weshay|ruck, for victoria should i comment failing jobs from [1] in the criteria [1]: https://review.rdoproject.org/zuul/buildset/5ddb9b01e6864c469df4adb714fa3072 | 16:09 |
| weshay|ruck | bhagyashris, k.. can you please promote victoria w/ https://review.rdoproject.org/zuul/buildset/5ddb9b01e6864c469df4adb714fa3072 | 16:09 |
| weshay|ruck | yes | 16:10 |
| bhagyashris | ok | 16:10 |
| *** ykarel|away has joined #oooq | 16:11 | |
| weshay|ruck | bhagyashris, thanks | 16:14 |
| bhagyashris | http://10.0.148.74/promoter_logs/container-push/20210602-160717.log | 16:17 |
| rlandy | ysandeep|dinner: weshay|ruck: going to rekick multinode to check again | 16:18 |
| weshay|ruck | bhagyashris, thanks | 16:18 |
| weshay|ruck | rlandy, mhunn is going to work on the internal kicking and get back to me tomorrow.. sorry | 16:19 |
| rlandy | weshay|ruck: np - I'll tsetproject | 16:20 |
| rlandy | https://review.opendev.org/c/openstack/python-tripleoclient/+/701544/ | 16:20 |
| * rlandy wants to test that | 16:20 | |
| rlandy | need to see where that optoion goes | 16:20 |
| rlandy | https://opendev.org/openstack/tripleo-operator-ansible/src/branch/master/roles/tripleo_overcloud_deploy/molecule/default/converge.yml#L159 | 16:22 |
| weshay|ruck | frenzy_friday, ++ https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/794232 | 16:25 |
| frenzy_friday | weshay|ruck, thanks. It should delete the graph in around 30-40 mins | 16:26 |
| weshay|ruck | frenzy_friday, nice | 16:26 |
| weshay|ruck | frenzy_friday, any luck w/ rdo elk? | 16:26 |
| *** bhagyashris_ has joined #oooq | 16:26 | |
| frenzy_friday | this is what we found out https://hackmd.io/CqKnNTOVRhC6K83sBZrbrQ We asked on thr rdo chanel if someone knows the correct api for downstream | 16:28 |
| frenzy_friday | maybe people will get back by tomorrow | 16:28 |
| *** bhagyashris has quit IRC | 16:34 | |
| weshay|ruck | frenzy_friday, are you asking for https://review.rdoproject.org/analytics/app/dev_tools#/console ? | 16:34 |
| *** ykarel|away has quit IRC | 16:36 | |
| weshay|ruck | frenzy_friday, you need to authenticate | 16:37 |
| weshay|ruck | frenzy_friday, https://hackmd.io/CqKnNTOVRhC6K83sBZrbrQ updated | 16:38 |
| weshay|ruck | zbr, ^ | 16:38 |
| *** tosky has quit IRC | 16:48 | |
| frenzy_friday | weshay|ruck, no, all this ^ happened after authentication | 16:49 |
| bhagyashris_ | weshay|ruck, plz add in your review list https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/33830 | 16:49 |
| *** bhagyashris_ is now known as bhagyashris | 16:50 | |
| *** ysandeep|dinner is now known as ysandeep | 16:50 | |
| *** ysandeep is now known as ysandeep|ruck | 16:50 | |
| frenzy_friday | the urls starting with https://review.rdoproject.org/analytics/app/... are browser endpoints, not api endpoints. With/without auth they return html - we need the api endpoints. The only one I found is elk.review.rdoproject.org . But it times out | 16:52 |
| weshay|ruck | frenzy_friday, k.. note that you were using auth.. | 16:52 |
| weshay|ruck | k.. add detail to that doc | 16:52 |
| frenzy_friday | updated | 16:54 |
| weshay|ruck | arxcruz|rover, re: console log path https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/ovb-manage/tasks/ovb-delete-stack.yml#L50-L55 | 16:56 |
| rlandy | weshay|ruck: ysandeep|ruck: compare these two ... | 16:56 |
| rlandy | https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset010-tripleo-train/414564c/logs/undercloud/etc/hosts.txt.gz | 16:56 |
| rlandy | ^^ train version | 16:56 |
| bhagyashris | weshay|ruck, victoria promoted https://trunk.rdoproject.org/centos8-victoria/current-tripleo/delorean.repo.md5 | 16:56 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/95/200295/85/check/periodic-tripleo-ci-rhel-8-multinode-1ctlr-featureset010-tripleo-rhos-16.2/0f5356b/logs/undercloud/etc/hosts | 16:56 |
| rlandy | 16.2 version | 16:56 |
| rlandy | NOTE ... | 16:56 |
| weshay|ruck | bhagyashris, thanks :) | 16:57 |
| bhagyashris | weshay|ruck, i am reverting back the criteria file | 16:57 |
| weshay|ruck | ++ | 16:57 |
| rlandy | # START_HOST_ENTRIES_FOR_STACK: overcloud | 16:57 |
| ysandeep|ruck | bhagyashris++ thanks! | 16:57 |
| rlandy | # END_HOST_ENTRIES_FOR_STACK: overcloud | 16:57 |
| rlandy | # START_HOST_ENTRIES_FOR_STACK: undercloud | 16:57 |
| rlandy | etc | 16:57 |
| rlandy | http://git.app.eng.bos.redhat.com/git/tripleo-ansible.git/tree/tripleo_ansible/roles/tripleo-hosts-entries/tasks/main.yml?h=rhos-16.2-trunk-patches | 16:58 |
| weshay|ruck | bhagyashris, lgtm :) | 16:58 |
| bhagyashris | ysandeep|ruck, :) | 16:59 |
| ysandeep|ruck | rlandy, checking if something changed recently there in tripleo-hosts-entries/tasks/main.yml ? | 16:59 |
| rlandy | tripleo-ansible.noarch 0.7.1-2.20210504014759.76a537a.el8ost @delorean-component-tripleo | 16:59 |
| rlandy | http://git.app.eng.bos.redhat.com/git/tripleo-ansible.git/commit/?h=rhos-16.2-trunk-patches&id=8c3d5a497097938d2554e2796a5e13147cc1d93b | 17:01 |
| *** jpena is now known as jpena|off | 17:03 | |
| arxcruz|rover | ysandeep|ruck: weshay|ruck didn't work https://logserver.rdoproject.org/58/28458/75/check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/df4603c/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 17:04 |
| *** ykarel|away has joined #oooq | 17:04 | |
| arxcruz|rover | but it's with the new cloudinit | 17:04 |
| arxcruz|rover | the appstream.repo doesn't have the exclude | 17:04 |
| arxcruz|rover | well, it is on https://logserver.rdoproject.org/58/28458/75/check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/df4603c/logs/undercloud/etc/yum.repos.d/quickstart-centos-appstreams.repo.txt.gz | 17:05 |
| arxcruz|rover | but not in the AppStream | 17:05 |
| ysandeep|ruck | arxcruz|rover, ^^ expected | 17:05 |
| rlandy | http://git.app.eng.bos.redhat.com/git/tripleo-ansible.git/commit/?h=rhos-16.2-trunk-patches&id=8c3d5a497097938d2554e2796a5e13147cc1d93b | 17:06 |
| rlandy | should have that patch | 17:06 |
| bhagyashris | weshay|ruck, ysandeep|ruck leaving for the day. If you guys want promotion for any other branches, let me know over an email - i will do tomorrow early morning my time :) | 17:06 |
| *** derekh has quit IRC | 17:06 | |
| ysandeep|ruck | bhagyashris, thanks! | 17:07 |
| *** akahat has joined #oooq | 17:07 | |
| bhagyashris | Good night zzzzz..... | 17:07 |
| ysandeep|ruck | arxcruz|rover, failure is with different error | 17:08 |
| arxcruz|rover | ysandeep|ruck: i'll check tomorrow | 17:08 |
| ysandeep|ruck | me too leaving for the day.. too tired now | 17:08 |
| rlandy | kicking 16.2 multinode job again | 17:09 |
| weshay|ruck | arxcruz|rover, ysandeep|ruck you guys have a patch to follow? | 17:09 |
| ysandeep|ruck | https://logserver.rdoproject.org/58/28458/75/check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/df4603c/logs/overcloud-controller-0/var/log/extra/package-list-installed.txt.gz cloud 20.3-10.el8_4.2.. still.. something didn't work.. | 17:09 |
| rlandy | weshay|ruck: ah - network kicked again | 17:09 |
| weshay|ruck | rlandy, rock | 17:10 |
| rlandy | weshay|ruck, I did nothing | 17:10 |
| arxcruz|rover | weshay|ruck: we couldn't test the cloud-init, the package installed still the 4.2 | 17:10 |
| ysandeep|ruck | we need to disable https://logserver.rdoproject.org/58/28458/75/check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/df4603c/logs/undercloud/etc/yum.repos.d/CentOS-Stream-AppStream.repo.txt.gz ... cloud-init coming from here. and not from quickstart-centos-appstreams | 17:12 |
| ysandeep|ruck | weshay|ruck, arxcruz|rover ^^ fyi.. | 17:13 |
| weshay|ruck | 2021-06-02 15:18:32.521 | > Excludes in repo quickstart-centos-appstreams: cloud-init-20.3-10.el8_4.2* | 17:13 |
| weshay|ruck | https://logserver.rdoproject.org/58/28458/75/check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/df4603c/logs/undercloud/home/zuul/build.log.txt.gz | 17:14 |
| weshay|ruck | 2021-06-02 15:18:40.767 | cloud-init noarch 20.3-10.el8_4.2 appstream 1.0 M | 17:14 |
| ysandeep|ruck | wait.. i have an idea.. we run ovb only on rdo.. we can fix this in configure-mirror role | 17:15 |
| ysandeep|ruck | something like in pre after this task https://github.com/rdo-infra/rdo-jobs/blob/master/playbooks/base/pre.yaml#L23 | 17:19 |
| ysandeep|ruck | we exclude cloud-init-20.3-10.el8_4.2 | 17:20 |
| weshay|ruck | ysandeep|ruck, export DIB_YUM_REPO_CONF='/etc/yum.repos.d/delorean* /etc/yum.repos.d/quickstart-*' | 17:20 |
| weshay|ruck | https://logserver.rdoproject.org/58/28458/75/check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/df4603c/logs/undercloud/home/zuul/build_images.sh.txt.gz | 17:20 |
| *** ykarel|away has quit IRC | 17:21 | |
| ysandeep|ruck | weshay|ruck, ^^ this looks okay, where its picking CentOS-Stream-AppStream.repo.txt | 17:22 |
| rlandy | 'update_repo': '' | 17:22 |
| ysandeep|ruck | or may be the centos qcow itself contain that already | 17:22 |
| rlandy | not good | 17:22 |
| weshay|ruck | 2021-06-02 15:33:07.223 | dib-run-parts Running /tmp/in_target.d/install.d/00-up-to-date | 17:22 |
| weshay|ruck | shouldn't be | 17:22 |
| rlandy | 'update_repo': 'delorean-current,tripleo ' | 17:23 |
| rlandy | better | 17:23 |
| weshay|ruck | 2021-06-02 15:36:14.397 | Verifying : cloud-init-20.3-10.el8_4.2.noarch 48/411 | 17:26 |
| weshay|ruck | 2021-06-02 15:36:14.397 | Verifying : cloud-init-20.3-5.el8.noarch 49/411 | 17:26 |
| *** amoralej is now known as amoralej|off | 17:29 | |
| weshay|ruck | ysandeep|ruck, may I take a crack at it? | 17:32 |
| ysandeep|ruck | weshay|ruck, go ahead, feel free to make any changes to patch | 17:33 |
| * ysandeep|ruck leaving for the day.. will pick up tomorrow | 17:34 | |
| *** ysandeep|ruck is now known as ysandeep | 17:36 | |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/95/200295/85/check/periodic-tripleo-ci-rhel-8-multinode-1ctlr-featureset010-tripleo-rhos-16.2/0f5356b/logs/undercloud/home/zuul/containers-prepare-parameter.yaml | 17:36 |
| rlandy | hmm - we still have an issue with update repo | 17:36 |
| rlandy | standalone works fine | 17:37 |
| *** ysandeep is now known as ysandeep|away | 17:39 | |
| *** ysandeep|away has quit IRC | 17:39 | |
| weshay|ruck | rlandy, I see a test project job on 16.2 tripleo component.. that's whats blocking I assume? | 17:39 |
| rlandy | yeah | 17:40 |
| rlandy | that to run what failed before | 17:40 |
| rlandy | weshay|ruck: but I need to change the release file | 17:40 |
| rlandy | for conatiner update | 17:40 |
| rlandy | standalone is fine | 17:40 |
| rlandy | but multinode is not | 17:40 |
| rlandy | it's overriding the set_fact | 17:40 |
| * rlandy will update | 17:40 | |
| weshay|ruck | rlandy, that hash passed here: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/8ea5be983feb4bda850239c64b65c840 | 17:41 |
| weshay|ruck | rlandy, 583c625c52d3a386d4cae90db02a1ddfe62b3a70 | 17:41 |
| weshay|ruck | && periodic-tripleo-ci-rhel-8-multinode-1ctlr-featureset010-tripleo-rhos-16.2 passed in your test project | 17:42 |
| weshay|ruck | rlandy, let's promote the component | 17:42 |
| weshay|ruck | rlandy, let's promote tripleo from component-ci-testing --> current-tripleo | 17:43 |
| weshay|ruck | I'll let jjoyce know we're updating tripleo all the way | 17:43 |
| weshay|ruck | rlandy, ? | 17:47 |
| * weshay|ruck doing it | 17:49 | |
| weshay|ruck | done.. asked jjoyce to import | 17:57 |
| rlandy | weshay|ruck: sorry meeting | 18:00 |
| weshay|ruck | I promoted tripleo 16.2 component all the way through | 18:01 |
| rlandy | weshay|ruck: review coming up for env files | 18:21 |
| rlandy | https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/undercloud-deploy/tasks/create-scripts.yml#L136 | 18:22 |
| rlandy | was getting overwritten | 18:22 |
| rlandy | the default needs to change there | 18:23 |
| *** akahat has quit IRC | 18:30 | |
| *** tosky has joined #oooq | 19:23 | |
| rlandy | 2021-06-02 15:34:19.141535 | primary | TASK [undercloud-setup : Run the package installation script] ****************** | 19:48 |
| rlandy | real failure here | 19:48 |
| rlandy | weshay|ruck: ugh ... | 19:49 |
| rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-tempest/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-tempest-rhos-16.2/4b94358/logs/undercloud/home/zuul/install_packages.sh.log | 19:49 |
| rlandy | python3-tripleoclient-12.5.1-2.20210601024819.b0fa425.el8osttrunk.noarch requires python3-tripleoclient-heat-installer | 19:49 |
| rlandy | is that sagi | 19:49 |
| rlandy | s review we need to revert? | 19:49 |
| sshnaidm | not me! | 19:50 |
| sshnaidm | ephemeral heat probably..? | 19:50 |
| rlandy | sshnaidm: you had a review sandeep wanted to revert | 19:51 |
| rlandy | <ysandeep|ruck> rlandy, https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/792904 is in gate .. once it merges.. | 19:51 |
| rlandy | <ysandeep|ruck> sshnaidm ^^ will revert his distgit patches | 19:51 |
| sshnaidm | rlandy, yeah, it's about ansible collections | 19:51 |
| sshnaidm | rlandy, correct | 19:52 |
| rlandy | not merged yet | 19:52 |
| sshnaidm | rlandy, python3-tripleoclient-heat-installer is a different story, not related to this | 19:52 |
| rlandy | k - remove blame | 19:52 |
| sshnaidm | I "suspect" it's related to latest ephemeral heat work, but not sure | 19:52 |
| rlandy | probably | 19:52 |
| rlandy | 16.2 components are blod red | 20:01 |
| rlandy | blood | 20:01 |
| rlandy | 2021-06-02 15:51:17 | - nothing provides ansible-collection-ansible-netcommon >= 1.5.0 needed by tripleo-ansible-0.7.1-2.20210601021641.2979a00.el8osttrunk.noarch | 20:02 |
| rlandy | 2021-06-02 15:51:17 | - nothing provides ansible-collection-ansible-posix >= 1.2.0 needed by tripleo-ansible-0.7.1-2.20210601021641.2979a00.el8osttrunk.noarch | 20:02 |
| rlandy | 2021-06-02 15:51:17 | - nothing provides ansible-collection-community-general >= 2.5.1 needed by tripleo-ansible-0.7.1-2.20210601021641.2979a00.el8osttrunk.noarch | 20:02 |
| rlandy | 2021-06-02 15:51:17 | - nothing provides ansible-collection-containers-podman >= 1.4.1 needed by tripleo-ansible-0.7.1-2.20210601021641.2979a00.el8osttrunk.noarch | 20:02 |
| rlandy | need to promote tripleo component to current-tripleo | 20:04 |
| rlandy | weshay|ruck: tripleo-ci-testing is behind current-tripleo | 20:06 |
| *** pojadhav has quit IRC | 20:11 | |
| *** dviroel is now known as dviroel|afk | 20:11 | |
| rlandy | toplink/vol/rhel-8/packages/ansible-collections-openstack/1.1.1/0.20200820111406.a67272d.el8ost/noarch/ansible-collections-openstack-1.1.1-0.20200820111406.a67272d.el8ost.noarch.rpm | 20:14 |
| rlandy | only one I see | 20:14 |
| rlandy | sshnaidm: help me out here ... where would the collections be installed from? | 20:15 |
| rlandy | ansible-collection-ansible-netcommon | 20:15 |
| rlandy | ansible-collection-ansible-posix | 20:15 |
| sshnaidm | rlandy, for which osp - 16 ? | 20:15 |
| rlandy | yep | 20:15 |
| rlandy | component jobs are dead | 20:16 |
| sshnaidm | rlandy, we don't need to install them, I'll revert distgit change, it's what Sandeep was talking about | 20:16 |
| sshnaidm | just waiting for that patch to get in.. | 20:16 |
| sshnaidm | when it's merged, I revert distgit for stable branches, and jobs will come back | 20:17 |
| rlandy | sshnaidm: cool - thanks | 20:17 |
| rlandy | will wait | 20:17 |
| sshnaidm | it passed gate jobs, just waiting for zuul to merge | 20:17 |
| sshnaidm | after a week... | 20:17 |
| rlandy | understand the issue | 20:18 |
| weshay|ruck | rlandy, on 16.2? | 20:31 |
| *** slaweq has quit IRC | 20:34 | |
| rlandy | weshay|ruck: yep | 20:53 |
| rlandy | waiting for revert | 20:53 |
| frenzy_friday | weshay|ruck, Graph 1926764 is removed now | 20:53 |
| rlandy | did rdo zuul die? | 20:55 |
| weshay|ruck | frenzy_friday, ya.. saw that :) | 20:57 |
| weshay|ruck | thanks | 20:57 |
| weshay|ruck | rlandy, I think rdo did | 20:58 |
| weshay|ruck | I'll be back later.. 7pm my time | 20:59 |
| rlandy | np | 21:01 |
| *** dviroel|afk is now known as dviroel | 21:03 | |
| rlandy | https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/792904 merged | 21:09 |
| rlandy | https://review.rdoproject.org/r/c/openstack/tripleo-ansible-distgit/+/33840 need revert but rdo is down | 21:10 |
| *** amoralej|off has quit IRC | 21:22 | |
| *** amoralej|off has joined #oooq | 21:38 | |
| *** pojadhav has joined #oooq | 22:01 | |
| *** amoralej|off has quit IRC | 22:12 | |
| *** dviroel is now known as dviroel|oof | 22:16 | |
| *** dviroel|oof is now known as dviroel|out | 22:16 | |
| *** tosky has quit IRC | 23:25 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!