Tuesday, 2022-02-01

rlandy|ruckand the revert the change to remove fs039 wallaby from criteria00:00
rlandy|ruckafter that promotes00:00
rlandy|ruckwe were missing images00:00
rlandy|ruckysandeep: main patch https://review.opendev.org/c/openstack/tripleo-quickstart/+/827111/00:00
rlandy|ruckthen I think we have an issue with ceph00:00
* ysandeep looking 00:00
rlandy|ruckwill have to test that afterwards00:00
rlandy|ruckhttps://github.com/openstack/tripleo-repos/blob/master/plugins/module_utils/tripleo_repos/main.py#L41300:01
rlandy|ruckysandeep: ^^00:01
rlandy|ruckgone from 800:01
rlandy|ruckmay need to change that to 700:01
rlandy|ruckysandeep: hands down the craziest day on ruck rover ever00:02
*** dviroel|brb is now known as dviroel00:03
ysandeeprlandy|ruck, https://github.com/openstack/tripleo-repos/blob/master/plugins/module_utils/tripleo_repos/main.py#L413 - I wonder if rdo team is also creating a temp repo for this one, just like alfredo did yesterday in that zuul-jobs patch?00:04
ysandeephttps://review.opendev.org/c/zuul/zuul-jobs/+/82703200:05
rlandy|ruckysandeep: opstools is already 7 there00:05
rlandy|ruckysandeep: ack that already merged00:06
ysandeeprlandy|ruck, yeah i mean similiar approach for ceph one, create a temp repo for what we removed in c800:06
rlandy|ruckit exists in 700:06
rlandy|ruckwe can just refer to that00:06
ysandeepack00:07
rlandy|ruckhttps://logserver.rdoproject.org/openstack-regular/opendev.org/openstack/tripleo-ci/master/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby-branch/055b63d/job-output.txt00:07
rlandy|ruck^^ will pass again after promotion of wallaby00:07
rlandy|ruckgoing on now00:07
ysandeepgreat00:08
rlandy|ruckysandeep: so yeah - just getting marios patch merged00:09
rlandy|ruckthen rcastillo's tox one00:09
rlandy|ruckthen we can go from there00:09
rlandy|ruckwhere is the tripleo-ci you are waiting for?00:09
ysandeeprlandy|ruck, do we have a patch for rcastillo's tox or  wallaby promotion will solve that?00:10
rlandy|ruckysandeep: https://review.opendev.org/c/openstack/tripleo-quickstart/+/82716600:10
rlandy|ruckneeds to go after marios patch00:10
ysandeepunderstood now, thanks!00:11
ysandeepyes my patch failed on tox one.00:11
rlandy|ruckysandeep: rechecked https://review.opendev.org/c/openstack/tripleo-ci/+/82652100:11
rlandy|ruckit may still fail00:11
rlandy|rucklet's see00:11
ysandeepi will keep an eye on these patches.00:13
* ysandeep looking at ceph, checking what is failing now00:14
rlandy|ruckack - something failing now00:16
*** dviroel is now known as dviroel_00:19
*** dviroel_ is now known as dviroel00:19
dviroelrlandy|ruck: o/ 00:20
rlandy|ruckdviroel: hey00:20
dviroelrlandy|ruck: need help with something?00:20
rlandy|ruckdviroel: so many failures ... atm we're just trying to get marios patch through00:21
rlandy|ruckbut it failed 00:21
rlandy|ruckchecking on failure https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b44/827111/3/check/tripleo-ci-centos-8-containers-multinode/b44e1eb/logs/undercloud/home/zuul/undercloud_install.log00:22
rlandy|ruckError: Error downloading packages:\n<13>Feb  1 00:07:42 puppet-user:   mariadb-connector-c-3.1.11-2.el8_3.x86_64: Cannot download, all mirrors were already tried without success\n<13>Feb  1 00:07:42 puppet-user: Error: /Stage[main]/Snmp/Package[snmpd]/ensure: change from 'purged' to 'present' failed: Execution of '/bin/dnf -d 0 -e 1 -y install net-snmp' returned 1: Error: Error downloading packages:\n<13>Feb  1 00:07:42 00:22
rlandy|ruckpuppet-user:   mariadb-connector-c-3.1.11-2.el8_3.x86_64: Cannot download, all mirrors were already tried without success\n<13>Feb  1 00:07:42 puppet-user: Notice: /Stage[main]/Snmp/File[var-net-snmp]: Dependency Package[snmpd] has failures: true\n<13>Feb  1 00:07:42 puppet-user: Warning: /Stage[main]/Snmp/File[var-net-snmp]: Skipping because 00:22
dviroellooking00:23
rlandy|ruckrcastillo: https://images.rdoproject.org/centos8/wallaby/rdo_trunk/current-tripleo/00:24
rlandy|ruckyou can recheck your patch now00:24
rlandy|ruckor rebase on marios one00:25
rlandy|ruckimages are back00:25
* rlandy|ruck reverts criteria patch00:25
rlandy|ruckhttps://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3879500:25
rlandy|ruckysandeep: pls ensure this merges: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3879500:26
rlandy|ruckw+'ed it00:26
ysandeeprlandy|ruck, sure00:26
rlandy|ruckFailed to download packages: dmidecode-1:3.3-1.el8.x86_64: Cannot download, all mirrors were already tried without success", "results": []}00:27
rcastillorlandy|ruck: thanks00:27
rlandy|ruckdviroel: ysandeep: ^^ both look like package download issues00:27
rlandy|ruckdviroel: ysandeep: going to abandon restore to restart testing00:28
rlandy|ruckysandeep: I'll put in a patch to move ceph to 700:29
rlandy|ruckfor t/u/v00:29
dviroelrlandy|ruck: ok00:29
ysandeepack, i see http://mirror.centos.org/centos/8/storage/x86_64/ceph-nautilus/ does not exists anymore.. but I am not sure using el7 will work (I guess we have to try) - because trying to install el7 rpms on el8 sometime give issues because of deps.00:31
dviroelhttps://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b44/827111/3/check/tripleo-ci-centos-8-containers-multinode/b44e1eb/logs/undercloud/etc/yum.repos.d/quickstart-centos-base.repo00:32
dviroel^ mirror very slow for me00:33
dviroelhttp://mirror.kna1.airship-citycloud.opendev.org/centos/8-stream/BaseOS/x86_64/os/00:33
rlandy|ruckdviroel: late for you00:36
dviroelrlandy|ruck: i'm fine00:36
dviroelrlandy|ruck: what if https://vault.centos.org/8.5.2111/storage/x86_64/ceph-nautilus/00:37
rlandy|ruckdviroel; anything stable atm 00:40
dviroelyep00:41
rlandy|ruckor we can ask fmount where to get this from now00:41
rlandy|ruckin his morning00:41
rlandy|ruckdviroel: ysandeep: asking on #tripleo00:41
rlandy|ruckwe can update tomorrow00:42
rlandy|ruckok - I'm out00:43
rlandy|ruckit's been 13.5 hours00:44
rlandy|ruckrcastillo: ysandeep: dviroel: see you guys tomorrow00:44
rlandy|ruckhopefully for less craziness00:44
ysandeeprlandy|ruck, good night o/ have a good rest00:44
*** rlandy|ruck is now known as rlandy|out00:44
ysandeepdviroel, see comment on https://review.rdoproject.org/r/c/rdo-infra/ansible-role-dlrn/+/38673/8/files/centos8-master-dlrn-deps.repo00:44
rcastillorlandy|ruck: good night, thanks for the help00:44
ysandeep"vault.c.o is not scalable environment, it is meant to be digital archive not for everyday use esp. not by high-load like CI"00:44
dviroelysandeep: yeah, it was more like a emergency solution00:47
ysandeep+1 but not sure it will be able to take the load, We need to take infra permission (#rdo) before using that in ci, We can check with amoralej|off once he is in tomorrow - where we can pull this content from?00:49
ysandeepamoralej|off: hey o/ Once you are in, https://github.com/openstack/tripleo-repos/blob/master/plugins/module_utils/tripleo_repos/main.py#L41300:50
ysandeepno longer exists http://mirror.centos.org/centos/8/storage/x86_64/ , where should we pull that from now - needed for t/u/v?00:50
dviroelysandeep: ack, make available on 8-stream or maintain our own nautilus repos00:52
dviroelnow that it is not available in 8 anymore, would make sense to have it in 8-stream00:53
ysandeepdviroel: agree, let's see what alfredo decides00:54
dviroelhttps://trunk.rdoproject.org/centos8-master/deps/storage/storage8-ceph-nautilus/x86_64/ - from 2020 :P01:14
ysandeepdviroel, we can compare ceph rpm versions from old jobs vs in this repo ^^, if its same I think its safe to try this01:20
dviroelysandeep: i was comparing with the vault one, there are lot of new pkg version 01:21
ysandeeperrr.., it means https://trunk.rdoproject.org/centos8-master/deps/storage/storage8-ceph-nautilus/x86_64 need to be refreshed then by #rdo team.01:22
dviroelyeah, looks like the fastest way to solve our problem01:24
dviroelthen update tripleo-repos01:24
ysandeepdviroel: thanks, I will discuss with marios and amoralej|off once they are in.01:24
ysandeepdviroel++01:25
dviroelyeah, looking at https://review.opendev.org/c/openstack/tripleo-quickstart/+/827166/ - not failing anymore for opstool 01:32
dviroeleven without marios patch01:32
dviroelhttps://review.rdoproject.org/r/c/rdo-infra/ansible-role-dlrn/+/38673 seems to solve 01:35
dviroelyeah, but in a near future, mirrors will also remove that content01:38
ysandeepyeah that will eventually happen01:43
* ysandeep away for few hours.01:44
dviroelo/01:44
*** ysandeep is now known as ysandeep|away01:44
ysandeep|awaydviroel, rcastillo have a great rest of the evening. o/ see you tomorrow01:44
dviroelysandeep|away: see you o/01:44
* dviroel out01:46
*** dviroel is now known as dviroel|out01:47
*** marios is now known as marios|ruck06:04
marios|ruck\o/06:04
Tenguhello there06:17
Tengucan anyone have a new review on https://review.opendev.org/c/openstack/tripleo-ci/+/825728 ?06:17
Tenguit should speed up a little bit DIB on CI, hopefully, since it will install less things.06:18
marios|ruckTengu: will have a look in a bit06:36
marios|ruckneed votes here please arxcruz sshnaidm frenzyfriday https://review.opendev.org/c/openstack/tripleo-quickstart/+/827111 06:37
marios|ruckTengu: minor nit please https://review.opendev.org/c/openstack/tripleo-ci/+/825728/10#message-4c67e3269995fbca5d6334d5bed0b8feea4558bc 06:42
marios|ruckakahat: please check when you have time https://review.opendev.org/c/openstack/tripleo-ci/+/823688/15#message-7d0af20acfc0f228c03469561373cf6ac3b197c9 06:50
marios|ruckakahat: and please check the 8 build jobs needs some attention issue with pip 06:51
Tengumarios|ruck: updated the comment. the whole file has to be dropped once LP is fixed anyway.07:00
Tengumarios|ruck: that same comment/TODO is also in the main.yaml07:00
marios|ruckTengu: thanks for update but why cant you move it to the task it belongs to https://review.opendev.org/c/openstack/tripleo-ci/+/825728/11/roles/oooci-build-images/tasks/remove_redfish.yaml#7 07:03
Tengumarios|ruck: because useless? whole file has to be removed; and as said, the same TODO is in the main.yaml, on the include_tasks directly.07:03
Tengutake that comment as a file header?07:04
*** ysandeep|away is now known as ysandeep07:04
Tengua working brain can more than probably understand the include_tasks: guestfs.yaml is a dependency for the following part..07:04
marios|ruckTengu: don't agree/follow what you mean... there is TODO above a task and it applies to the task. why do you want to move it? 07:05
Tengubecause the TODO is now "remove *file* once ..."07:06
Tengunot "remove task"07:06
Tengui.e. it should ensure the file is removed, not just the redfish related task - that would keep the guestfs install, which is useless.07:07
Tenguor, maybe, keep a useless task file because the include_tasks would be removed.07:07
marios|ruckTengu: still disagree it takes away some context. someone would have to go dig to realise that the launchpad bug there applies specifically to the redfish task. i am not going to -1 but i disagree with your reasoning and i can't understand why this even needs a discussion07:08
Tenguwhat about now?07:09
Tengukeeping the "delete file" at the top, and pointing the task for the LP once more.07:10
marios|ruckthanks will check in a bit the world is on fire :/07:10
TenguSounds like Swiss logic vs The World :D07:10
marios|ruck:)07:10
Tengulemme know if I can help for the fire07:10
Tengueither with water or some more fuel07:11
marios|ruckTengu: send whisky07:11
Tengufuel that is!07:11
Tenguwould an Oban be ok? :)07:11
marios|ruckit would suffice i suppose ;)07:14
Tengumaybe a Smokehead then? stronger :]07:14
*** amoralej|off is now known as amoralej07:52
marios|ruckamoralej: :/ so now we are back to green on centos8 master so... confirmed mirror issue? https://bugs.launchpad.net/tripleo/+bug/1959524/comments/7 08:27
amoralejmarios|ruck, tbh i'm still not sure what happened08:29
amoralejlet me find out what new packages have been synced in mirror since yesterday08:30
amoralejwould you know what was the first job passing?08:30
*** jpena|off is now known as jpena08:36
marios|ruckamoralej: yeah i mean, it would be pretty bad if it was just 'mirror sync' and quite unbelievable cos it was like 3 days... don't think weekend should matter for that 08:38
marios|ruckamoralej: k not quite 3 days... looks like 30th (2 days?) looking at https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario001-standalone-master as an example08:40
marios|ruckamoralej: well.. 3 days... so first one is there https://logserver.rdoproject.org/openstack-periodic-integration-main-centos8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-master/d462e97/job-output.txt 29th 08:40
amoralejit's weird i see no mirrors sync for 8-stream since 31st-Jan 06:47 https://static.opendev.org/mirror/logs/rsync-mirrors/08:40
amoraleji don't understand08:41
amoralejmmm08:41
amoralejmaybe some change related to images...08:41
amoralejimages content08:41
akahatmarios|ruck, o/ thanks for the comments. i'll fix it. :)08:53
amoralejmarios|ruck, we were having the issue only with jobs running in sf zuul or also in upstream?08:53
marios|ruckakahat: thanks ... wrt the pip dasm|off was poking with https://review.opendev.org/c/openstack/tripleo-ci/+/827157 so check that too 08:55
marios|ruckakahat: maybe rebase onto it or something 08:55
marios|ruckamoralej: i don't think we saw that one in gates it was only in the rdo SF/periodic jobs08:56
amoralejso, probably something related to image08:56
ysandeepmarios|ruck, amoralej http://mirror.centos.org/centos/8/storage/x86_64/ceph-nautilus/ doesn't exists anymore09:00
ysandeepmarios|ruck, https://github.com/openstack/tripleo-repos/blob/master/plugins/module_utils/tripleo_repos/main.py#L41309:00
marios|ruckysandeep: yeah i think the ceph folks are looking there already 09:00
amoralejysandeep, yes, storage sig didn't build it for CS809:00
amoralejthey only built from  pacific for stream 809:00
amoraleji'm not sure if they plan to build nautilus or we should move to pacific at this point09:00
ysandeepfor the timebeing i was thinking if we can use https://trunk.rdoproject.org/centos8-master/deps/storage/storage8-ceph-nautilus/x86_64 - but this is not updated when comparing to https://vault.centos.org/8.5.2111/storage/x86_64/ceph-nautilus/09:02
dpawlikhey, since yesterday Centos 8 is EOL so we will remove the centos 8 images and nodesets (due repositories are down/changed so they will not work). Please move them to centos 8 stream if possible09:15
marios|ruckgthiemonge: o/ hey can we try get that merged https://review.opendev.org/c/openstack/octavia/+/816370 https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/816369 - we had good results in test see https://bugs.launchpad.net/tripleo/+bug/1958531/comments/709:41
gthiemongemarios|ruck: yes, I'll ping our core reviewers, but I think there are some issues in the gates that will block the octavia-tempest-plugin patch :/09:47
marios|ruckthank you gthiemonge 09:48
marios|ruckfooood10:01
sshnaidmmarios|ruck, ysandeep, rlandy please review https://review.opendev.org/c/openstack/tripleo-ci/+/827047 https://review.opendev.org/c/openstack/tripleo-ci/+/82706410:13
ysandeepsshnaidm: ack10:13
marios|rucksshnaidm: adding to list10:30
*** marios|ruck is now known as marios10:41
*** marios is now known as marios|ruck10:41
*** rlandy is now known as rlandy|ruck11:14
bhagyashrismarios|ruck, c7 train promoted on old promoter server http://10.0.148.74/promoter_logs/container-push/20220201-101654.log11:18
bhagyashrisnot sure why it failed on new one/production server11:19
bhagyashrishttps://trunk.rdoproject.org/centos7-train/current-tripleo/delorean.repo11:19
*** dviroel|out is now known as dviroel11:27
dviroelo/11:29
sshnaidmarxcruz, jm1, frenzyfriday, ysandeep, rcastillo, rlandy|ruck I've prepared list of collections with priorities: https://hackmd.io/7NtovjRkRn-tKraBXfz9jw11:32
sshnaidmwill add it to jira ticket as well11:32
frenzyfridaysshnaidm, thanks!11:33
rlandy|ruckhttps://trunk.rdoproject.org/centos8-master/deps/storage/11:35
sshnaidmif you use devstack for modules test, you can try my local.conf https://paste.opendev.org/show/812460/11:43
sshnaidmjust change 192.168.2.139 to your IOP11:43
sshnaidms/IOP/IP11:43
amoralejmarios|ruck, you can use  https://trunk.rdoproject.org/centos8-master/deps/storage/nautilus/11:46
dviroelmarios|ruck: ysandeep: please see https://review.rdoproject.org/r/c/rdo-jobs/+/38634/2#message-d942642017ca1572f4e0c4c6eb154d9edef7eb0c 11:47
ysandeepdviroel: sure, will check soon, in a mtg11:48
dviroelsure sure11:48
marios|ruckdviroel: ack in a few in call 11:53
ysandeepdviroel, you are right https://logserver.rdoproject.org/82/38582/4/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master/ac2c3e4/logs/undercloud/home/zuul/overcloud-hardened-uefi-full.log.txt.gz11:55
marios|ruckdviroel: yea yo are right11:57
marios|ruckdviroel: i'll work on the tox issue there 11:57
marios|ruckdviroel: gimme few 11:57
ysandeepdviroel, updating your patch - to put it back as it was11:57
ysandeepdviroel, reverted to previous patchset12:00
dviroelmarios|ruck: ysandeep ok thanks 12:04
*** ysandeep is now known as ysandeep|break12:04
rlandy|ruckpojadhav: you back today?12:11
marios|ruckfyi rlandy|ruck dasm|off https://review.rdoproject.org/r/c/rdo-jobs/+/38872 let's see if by some miracle that "just works" (tm)(patent pending)12:11
marios|ruckrlandy|ruck: will start digging at the release files for the ceph change k ? 12:12
marios|ruckrlandy|ruck: then when i finish that will check tripleo-repos side 12:12
rlandy|ruckmarios|ruck: ack12:12
dviroelmarios|ruck: worked, thanks12:17
rlandy|ruckmarios++12:17
dviroelbtw, https://codesearch.rdoproject.org/codesearch/?q=nodeset%3A%20rdo-centos-8%24&i=nope&files=&repos=12:18
marios|ruckrlandy|ruck: lets merge it dviroel https://review.rdoproject.org/r/c/rdo-jobs/+/38872/1#message-832c36809a2767221a3c46e33e61978e5b940d74 12:18
rlandy|ruckdone12:18
marios|ruckdviroel: yeah and yesterday amoralej posted that https://review.rdoproject.org/r/c/rdo-jobs/+/38813 i think we should move it back amoralej ? 12:19
marios|ruckdviroel: will check that list later need to deal sthing else now12:19
dviroelack12:19
dviroelmarios|ruck++12:19
rlandy|ruckmarios|ruck: ok - this is what is killing wallaby ...  http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos/8/opstools/x86_64/collectd-5/repodata/repomd.xml 12:20
rlandy|rucklooking for the mirror version of that repo12:20
* rlandy|ruck looks to fix that12:21
marios|ruckrlandy|ruck: ack - link /job? 12:21
marios|ruckrlandy|ruck: why is it not addressed by my workaround we already merged just now 12:21
rlandy|ruckhttps://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-containers-multinode-wallaby/63bb0e3/job-output.txt12:21
rlandy|ruckoh12:21
marios|ruckrlandy|ruck: i mean release files https://review.opendev.org/c/openstack/tripleo-quickstart/+/827111 12:21
rlandy|ruckmaybe started before merge?12:21
marios|ruckrlandy|ruck: yeah hope so :)12:21
rlandy|ruckmarios|ruck: nvm - I'll recheck those12:22
rlandy|rucktestproject coming up12:22
marios|ruckrlandy|ruck: ack thanks12:22
dviroelwallaby uses yum-config tool - but seems that release file isn't updated with marios patch12:24
dviroelchanged: [subnode-1] => (item={'type': 'generic', 'reponame': 'quickstart-centos-opstools', 'filename': 'quickstart-centos-opstools.repo', 'baseurl': '${NODEPOOL_CENTOS_MIRROR}/8/opstools/x86_64/collectd-5/'})12:24
dviroelyeah, lets wait next ones12:25
rlandy|ruckdviroel: rekicked those jobs12:25
amoralejmarios|ruck, that was just for testing12:28
amoraleji can abandon that12:28
marios|ruckack thanks amoralej 12:29
rlandy|ruckdviroel: release file not updated? https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-8/promotion-testing-hash-wallaby.yml#L173?12:34
dviroelrlandy|ruck: yeah, but in that job only12:36
dviroelin the job was still: 'baseurl': '${NODEPOOL_CENTOS_MIRROR}/8/opstools/x86_64/collectd-5/12:36
rlandy|ruckrechecked ovb patch12:38
rlandy|ruckysandeep|break: dviroel: voted on https://review.rdoproject.org/r/c/rdo-jobs/+/3863412:39
*** ysandeep|break is now known as ysandeep12:39
rlandy|ruckmarios|ruck: k - where are you with ceph patched12:40
rlandy|ruckI can help now12:40
marios|ruckrlandy|ruck: doing the release files12:40
marios|ruckrlandy|ruck: i will move to tripleo-repos but didn't start yet if you want to otherwise i will next np12:40
rlandy|ruckok - starting there12:41
dviroelfrom where we should pull ceph-nautilus now?12:41
marios|ruckhttps://trunk.rdoproject.org/centos8-master/deps/storage/nautilus/ 12:42
dviroelrdo?12:42
marios|ruckfor now 12:42
dviroelack12:42
*** dasm|off is now known as dasm13:02
dasmo/13:02
*** dasm is now known as dasm|rover13:02
rlandy|ruckdasm|rover: hey - on meeting - pls touch base with marios|ruck about current situation13:03
dasm|roverk13:04
*** rlandy|ruck is now known as rlandy|ruck|mtg13:04
marios|ruckdasm|rover: things looking better today with the 8 blockers/deps issues. 13:05
marios|ruckdasm|rover: mainly the ceph thing now blocking gates https://bugs.launchpad.net/tripleo/+bug/1959656 https://review.opendev.org/c/openstack/tripleo-quickstart/+/827279 https://review.opendev.org/c/openstack/tripleo-repos/+/827280 13:05
dasm|rover*still* 8 blockers? or 8 blockers from yesterday?13:05
dasm|roverk13:05
marios|ruckdasm|rover: maybe you can keep digging on the centos8/buildimage with release files? 13:05
marios|ruckdasm|rover: https://review.opendev.org/c/openstack/tripleo-ci/+/827157 that one (and akahat was working on his patch maybe sync with him about that)13:06
dasm|rovermarios|ruck: the one by Amol?13:06
dasm|roveryeah. this one.13:06
*** sshnaidm is now known as sshnaidm|afk13:06
dasm|rovermarios|ruck: k. i'll check that13:06
marios|ruckdasm|rover: yeah.. at least this is no longer *urgent* (since the 8 blockers/deps issue seems gone) 13:06
marios|ruckdasm|rover: but we still need it 13:06
dasm|rovermarios|ruck: ack. i'll check that13:07
marios|ruckthanks dasm|rover 13:07
*** amoralej is now known as amoralej|lunch13:13
ysandeepfrenzyfriday, hey o/ I am going through c9 cards for EL9 mtg prep,  what's left on https://issues.redhat.com/browse/TRIPLEOCI-733 ?13:13
frenzyfridayhey ysandeep , 733 is ready for review. The patches in the comments need reviews/votes13:15
ysandeepfrenzyfriday: ++ thanks13:17
ysandeepfolks please add https://review.rdoproject.org/r/c/rdo-jobs/+/38475 in your review list13:17
ysandeepthis one too https://review.rdoproject.org/r/c/rdo-jobs/+/38829/13:20
rlandy|ruck|mtgbhagyashris: hi - I may miss community call13:20
rlandy|ruck|mtgon meeting13:20
bhagyashrisrlandy|ruck|mtg, ack np13:21
akahatmarios|ruck, dasm|rover do you have some idea to optimize this: https://review.opendev.org/c/openstack/tripleo-ci/+/823688/18/roles/tripleo-build-jobs-repos/tasks/main.yaml#18 13:25
marios|ruckakahat: not right now will try check later13:25
dasm|roverakahat: should you remove python3-pip no matter the centos version?13:26
bhagyashrisarxcruz, sshnaidm, rlandy, rlandy|ruck, marios, ysandeep, bhagyashris, svyas, soniya29, pojadhav, akahat, chandankumar, frenzy_friday, anbanerj,  dviroel, rcastillo, dasm, jm1 marios|ruck dasm|rover 13:26
bhagyashrisTripleO CI community call 13:26
bhagyashrisin 4 mins13:26
dasm|roverakahat: actually i was trying to do something similar here: https://review.opendev.org/c/openstack/tripleo-ci/+/82715713:27
bhagyashrishttps://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg?both#2022-02-01-Community-Call @ line 3013:27
bhagyashrishttps://meet.google.com/bqx-xwht-wky?authuser=013:28
akahatdasm|rover, yes. i saw your patch.. and thought to include pip url in it.13:28
akahatdasm|rover, yes. we are removing it.13:28
dasm|roverakahat: that's a lot better than i thought. i wrongly assumed it's needed across the all releases.13:29
dasm|roverakahat: in that case, i think we don't need check for 8> major version13:29
akahatdasm|rover, i think you are right here.13:30
akahatlet's wait for first run13:31
bhagyashrisakahat, ^13:31
akahatbhagyashris, joining.13:32
dasm|roverakahat: ack13:32
dasm|roverakahat: i'm thinking about your change. could we add the change to centos8 in a more generic place, like https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ensure-pip/tasks/workarounds.yaml ?13:53
dasm|roversomething like ansible_python.version.major == 3; ansible_python.version.minor < 7 then upstream link would be pointing towards 3.6 pypi13:54
*** sshnaidm|afk is now known as sshnaidm13:54
dasm|roverakahat: lemme see if i can quickly prepare this kind of patch13:54
akahatdasm|rover, ack13:55
*** rlandy|ruck|mtg is now known as rlandy|ruck14:05
dviroelmarios|ruck rlandy|ruck: new cix card, for an old bug, hitting cs9 wallaby https://bugs.launchpad.net/bugs/195647514:07
* dviroel not so old bug14:07
rlandy|ruckok14:07
marios|ruckdviroel: 'thanks' ;)14:07
marios|ruckdviroel: comments on the bug looksl ike we need https://review.opendev.org/c/openstack/oslo.rootwrap/+/823571 on stable/walalby 14:08
marios|ruckThere's no W branch on rootwrap14:09
marios|ruckah 14:09
marios|ruck(dvd replied)14:09
marios|ruckrlandy|ruck: i was in the community call so onlyjust spent time ... this looks OK https://review.opendev.org/c/openstack/tripleo-repos/+/827280 14:10
marios|ruckrlandy|ruck: lets remove the WIP14:12
marios|ruckrlandy|ruck: maybe we can get some testprojec going but otherwise lgtm 14:13
*** amoralej|lunch is now known as amoralej14:13
rlandy|ruckmarios|ruck: thanks - updating14:20
dasm|roverakahat: i pushed this change: https://review.opendev.org/c/zuul/zuul-jobs/+/82729714:22
dasm|roverlet me know what do you think.14:23
dasm|roverakahat: i see at least handful of examples where it can fail. but i'm gonna give it up to cores, to make this decision.14:23
dasm|roverwhen i was drafting that, i was already not quite sure if that's the right way of doing that.14:24
marios|ruckrlandy|ruck: testproject https://review.rdoproject.org/r/c/testproject/+/38884 i can recheck when you update14:26
dviroelrlandy|ruck: added a comment https://review.opendev.org/c/openstack/tripleo-repos/+/827280/2#message-ff71fd9564490168b04456d9661c4242f542366014:31
dviroelrlandy|ruck: testing it locally14:31
marios|ruckrlandy|ruck: https://review.rdoproject.org/zuul/status#3888414:40
*** bhagyashris_ is now known as bhagyashris14:42
jm1arxcruz, rcastillo: i am setting up devstack VMs that allow us to develop and to run tests from the OpenStack Ansible collection. will ping you once i get a working setup.14:43
arxcruzjm1 cool, i'm setting up one as well14:43
dviroelmarios|ruck: https://review.opendev.org/c/openstack/tripleo-repos/+/827280/2#message-ff71fd9564490168b04456d9661c4242f5423660 14:43
arxcruzbut i think i got the wrong one, i put minimal :(14:43
jm1arxcruz: you need to enable a lot of additional services and depending on your OS you will run into unrelated bugs.14:45
dasm|roverakahat: i think my change might be even good. fungi +2'ed it. would you care running it on our CI?14:48
marios|ruckdviroel: nice catch (we are testing with https://review.rdoproject.org/r/c/testproject/+/38884 ) but rlandy|ruck can you update or busy? 14:48
marios|ruckrlandy|ruck: 16:43 < dviroel> marios|ruck: 14:48
marios|ruckhttps://review.opendev.org/c/openstack/tripleo-repos/+/827280/2#message-ff71fd9564490168b04456d9661c4242f5423660 14:48
marios|ruckrlandy|ruck: dviroel: i'll upadte sec (we can just hard code this url completely i think )14:49
rlandy|ruckdviroel: thanks - updating14:50
marios|ruckrlandy|ruck: k go ahead 14:50
* marios|ruck cancel edit 14:50
ysandeepsshnaidm, marios|ruck rlandy|ruck dviroel when you have time could you please review https://review.opendev.org/c/openstack/tripleo-ci/+/826521 , -1 from zuul due to ceph nautlius issue(I will recheck once that issue is solved) , If you have suggestion on patchset I can implement them now (passed in downstream with depends-on this patch).14:53
marios|ruckysandeep: adding to reviews 14:53
ysandeepthanks14:53
jm1arxcruz: tests passed for the first time ^^14:57
jm1arxcruz: will clean up the vm and then you can have it14:57
rlandy|ruckrechecked testproject14:58
arxcruzjm1 sorry, what are you talking about ?14:58
jm1arxcruz: a vm with devstack which is running on one of our baremetal machines. it allows you to run the tests from Sagi's Ansible OpenStack collection..14:59
rlandy|ruckysandeep: ack - thanks15:00
* dasm|rover brb15:01
arxcruzjm1 ok 15:02
dviroelysandeep: i'm updating https://issues.redhat.com/browse/TRIPLEOCI-751 with cs9-wallaby updates - do you want me to add info somewhere else?15:08
ysandeepdviroel, thanks, comment on card will work15:09
dviroelysandeep: ack15:10
ysandeepdviroel++15:10
rlandy|ruckdviroel: looking at bug you opened15:16
rlandy|ruckhttps://bugs.launchpad.net/tripleo/+bug/195958215:16
rlandy|ruckdviroel: networking team?15:17
rlandy|rucktriaging cards15:17
rlandy|ruckdviroel: k - so I see the two cards15:19
rlandy|ruckwill leave those for the CIX meeting15:19
rlandy|ruckboth are still an issue15:19
rlandy|ruck?15:19
dviroelrlandy|ruck: yes, network team15:19
*** ysandeep is now known as ysandeep|dinner15:19
rlandy|ruckhttps://bugs.launchpad.net/tripleo/+bug/1956475 is still open15:20
rlandy|ruckon master15:20
dviroelrlandy|ruck: yes, there are skiplist for those tests in master, we need to test fixes15:20
rlandy|ruckdviroel: ack - ok - you want to join the cix tomorrow to address those?15:21
dviroelrlandy|ruck: sure, will help to get in sync with other too15:21
*** dviroel is now known as dviroel|lunch15:40
ysandeep|dinnerrlandy|ruck, el9 mtg time15:45
*** ysandeep|dinner is now known as ysandeep15:45
dasm|rovermarios|ruck: when testing akahat's change, i'm seeing this in logs: "module_stderr": "/bin/sh: /usr/bin/python3: No such file or directory\n", for centos816:27
dasm|rovermarios|ruck: is it possible that centos8 is missing python installed ootb?16:28
marios|ruckdasm|rover: weird pretty sure the nodepool nodes have python already 16:30
dasm|roverhmm16:31
dasm|roveri'm gonna wait for test results to see full logs. above is just an excerpt from running job16:32
*** dviroel|lunch is now known as dviroel16:35
rlandy|ruckmarios|ruck: dasm|rover: dviroel: https://review.rdoproject.org/r/c/testproject/+/38884 green - pls vote on https://review.opendev.org/c/openstack/tripleo-repos/+/82728016:35
rlandy|ruckneeds w+16:35
marios|ruckrlandy|ruck: \o/ 16:36
rlandy|ruckmarios|ruck: maybe there is light at the end of all this16:36
marios|ruckrlandy|ruck: i'll be happy if we get a train run too 16:37
dasm|roverrlandy|ruck: let's hope it's not an incoming train16:37
dasm|rovermarios|ruck: :D16:37
rlandy|ruckdasm|rover: :)16:37
rlandy|rucklet's hope it's a passing train run16:37
marios|ruckrlandy|ruck: but yeah not as crazy as yesterday 16:37
dasm|rover:D16:37
marios|ruckdasm|rover: rlandy|ruck: going in a few 16:38
dasm|roverk16:38
marios|ruckfeels like it should be friday tomorrow16:38
marios|ruckbut it is not 16:38
marios|ruck\o/ 16:38
dasm|roverfri-yay16:38
ysandeepwhat its not friday tomorrow? 16:39
marios|ruckysandeep: right? 16:39
dasm|rovereveryday is Friday, somewhere in the world.16:39
marios|ruckdasm|rover: you should do marketing for Friday's restaurants 16:39
dasm|roverTGIF16:40
marios|ruckmmmm ribs n shrimp sounds good right now 16:40
rlandy|ruckmarios|ruck: sure - thanks for all your work today16:41
dviroelrlandy|ruck: ysandeep: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/827165 - then we can make merge ovb-fs002-wallaby16:41
dviroeltestproject: https://review.rdoproject.org/r/c/testproject/+/38639/3#message-8e3e938d281083539d7ad5c1da13b77ca589d8ae16:42
ysandeepdviroel, voted16:43
* ysandeep out now, see you tomorrow 16:43
*** ysandeep is now known as ysandeep|out16:43
dasm|roverysandeep: \o16:43
ysandeep|outdasm|rover, o/ 16:43
rlandy|ruckdasm|rover: it gets better :)16:46
marios|rucko/ have a good one folks ttyl 16:46
*** marios|ruck is now known as marios|out16:46
rlandy|ruckdviroel: merged16:46
dasm|roverone thing less to be worried about. pypi 3.6 getting installed by default: https://review.opendev.org/c/zuul/zuul-jobs/+/82729716:47
akahatdasm|rover, sorry.. i just back. That's great!! 16:53
akahatdasm|rover, let me put recheck on it once it got merged.16:54
akahatonce your patch got merged *16:54
dasm|roverakahat: there is an issue with missing python3. i'm checking what's happening there.16:55
dasm|roverakahat: i also changed your patch to include upstream pypi change16:55
akahatdasm|rover, where you are seeing that issue?16:56
dasm|roverakahat: i was looking at logs when job was running. there is an issue with pulling and installing pypi, because there is no python316:57
dasm|roverakahat: here16:57
dasm|rover16:27 <dasm|rover> marios|ruck: when testing akahat's change, i'm seeing this in logs: "module_stderr": "/bin/sh: /usr/bin/python3: No such file or directory\n", for centos816:57
dasm|roveri was waiting for a full log report16:57
akahatdasm|rover, where are you running it? do you have logs or review?16:59
dasm|roverakahat: on your review. no logs, because i updated the patch17:00
dasm|roverakahat: btw. i assumed you were done for tonight, so i didn't expect any response from you tonight :)17:00
akahatdasm|rover, oh.. i think i know what is the issue.17:01
dasm|roverthat's good! :)17:01
akahatdasm|rover, yeah.. dinner time.. :)17:01
akahatlet me update the patch.17:01
dasm|roverk17:01
dasm|roveryou can remove depends-on as well17:01
akahatdasm|rover, https://review.opendev.org/c/openstack/tripleo-ci/+/823688/20/roles/tripleo-build-jobs-repos/tasks/main.yaml#16 this is the problem.17:04
akahati just recall it.. when you are removing python3-pip from c8 it remove python3 also.17:04
dasm|rover(facepalm)17:04
akahatwe need to keep that when condition.17:05
akahat:|17:05
dasm|roverso what happens when we would try to install pip on centos9, with pip already installed?17:07
akahatyeah.. on c9 pip is already installed. I think it got install with deps.17:08
dasm|rovermhm. good to know, even though, it's very strange behavior :)17:09
rlandy|ruckdviroel: dasm|rover: please vote on https://review.opendev.org/c/openstack/tripleo-ci/+/826521/ to spring downstream17:27
dviroelwill do17:29
*** jpena is now known as jpena|off17:31
dasm|roverack17:33
*** dviroel is now known as dviroel|biab17:55
rlandy|ruckdasm|rover: let's sync for a few17:57
rlandy|ruckhttps://meet.google.com/dgi-rynt-izp?pli=1&authuser=017:57
rlandy|ruckdasm|rover: ^^17:57
rlandy|ruckdropping - ping when ready pls18:00
*** dviroel|biab is now known as dviroel18:08
dasm|roverrlandy|ruck: here. sorry, been afk18:09
rlandy|ruckdasm|rover: no worries - just wanted to see how you are doing18:09
rlandy|ruckno much we can do until all these patches are merged18:10
dasm|roverrlandy|ruck: keeping an eye on akahat's patch. there are still other issues.18:10
rlandy|ruckjust watching those for new18:10
rlandy|ruckdasm|rover: k  great18:10
rlandy|ruckwill be good to get that through18:10
dasm|roverack. i noticed some issues with ansible vars, so i'm gonna address them next18:10
rlandy|ruckcool18:11
dasm|roverheh... Error: Failed to download metadata for repo 'quickstart-centos-ceph-nautilus': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried18:16
dasm|roverdid we merge marios|out patch already rlandy|ruck ?18:16
* dasm|rover digging through reviews18:17
dasm|roverseems like we did: https://review.opendev.org/c/openstack/tripleo-quickstart/+/82711118:17
dasm|roveroh wait. that's opstools not nautilus18:17
dasm|roveroh, it's here: https://review.opendev.org/c/openstack/tripleo-quickstart/+/82727918:18
dasm|roverit's still in Q. looks decent though18:18
dasm|rovernot sure why "tripleo-ci-centos-9-content-provider" is marked as "paused"18:18
dasm|roverok. i'm gonna try depends-on on this nautilus patch. eitherway -- if it's gonna merge -- that's good. if not -- we're gonna see some progress18:19
rlandy|ruckdasm|rover: no still in testing18:20
rlandy|ruckwaiting for c9 jobs to run18:21
dasm|roverack18:21
dasm|roveranother thing failing in akahat's patch is: ""reason": "couldn't resolve module/action 'tripleo.repos.yum_config'"18:21
dasm|roverchecking this one18:21
rlandy|ruckdasm|rover: this merged: https://review.opendev.org/c/openstack/tripleo-repos/+/82728018:21
dasm|roveroh, that's good18:21
rlandy|ruckdviroel is your man for that question18:22
rlandy|ruckrcastillo: thank you for saving us on molecule yesterday18:22
dviroeldasm|rover: where? 18:24
dasm|roverdviroel: https://8cf70d2ee9f0d7f3b892-1aafb108750144ecd9565eaf1429d9e4.ssl.cf2.rackcdn.com/823688/20/check/tripleo-buildimage-overcloud-full-centos-8-wallaby/66608fb/job-output.txt18:24
dasm|roverthis patch: https://review.opendev.org/c/openstack/tripleo-ci/+/82368818:24
dviroeloh boy18:25
dasm|roverthere might be few issues on top of each other18:25
dasm|roveri'm gonna retry with depends-on on the nautilus.18:26
dviroeldasm|rover: the easy way is to disable yum_config module when building images - rcastillo and I are working on yum-config without collections (we have a patch, needs review and merge)18:31
dviroeldasm|rover: will add a comment in the patch18:31
dasm|roverack18:31
*** amoralej is now known as amoralej|off18:35
dviroeldasm|rover: 18:41
dviroelhttps://review.opendev.org/c/openstack/tripleo-ci/+/823688/21#message-bc17278065a25071f96a8c4e8391fb9cd31216ac18:42
dasm|roverdviroel: ack, thx18:42
dasm|roverdviroel: can i add depends-on on yours and rcastillo patch? or is it better to just run without it and get it merged asap?18:43
dasm|roveryours patch seems to be still WIP18:43
dviroeldasm|rover: go for disabling yum_copnfig18:44
dviroeldasm|rover: after we merge out patch, we will work on enabling again18:44
dviroelour patch*18:44
dviroeldasm|rover: we will need to enable it to work with compose pinning18:45
dasm|roverack18:45
rlandy|ruckdasm|rover: hopefully tomorrow will be more normal and we can work on promotions and component lines19:22
rlandy|rucklike normal ruck/rovers19:23
dasm|rover:D19:23
dasm|roverthat would be actually nice 19:23
dasm|roverbtw. akahat's patch has just 3 retries atm: just "wallaby". before, all centos8 jobs were failing.19:24
dasm|roverthere is a nice progress.19:24
dasm|roverstill i need to address dviroel's comments19:24
dasm|roverjust waiting for a final results to have something to work on.19:25
* dviroel periodic-tripleo-ci-centos-9-scenario010-ovn-provider-standalone-wallaby: SUCCESS in 1h 21m 26s \o/19:29
dasm|rovernice!19:30
rlandy|rucklunch - brb19:35
dasm|roverenjoy!19:39
*** dviroel is now known as dviroel|afk20:35
rlandy|ruck2022-02-01 07:26:41 | 2022-02-01 07:26:41.643576 | fa163eda-e65d-1833-4706-0000000002a7 |      FATAL | Deploy network-scripts required for deprecated network service | overcloud-controller-2 | error={"changed": false, "msg": "Failed to download metadata for repo 'quickstart-centos-opstools': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried", "rc": 1, "results": []}20:59
rlandy|ruck^^ master failure20:59
dasm|rovernot again20:59
dasm|roveri thought we're pass that one already21:00
rlandy|ruckdasm|rover: that is on c8 master21:01
dasm|roverack21:02
rlandy|ruckdasm|rover: can you check the master c8 and c9 component line21:02
dasm|roversure21:02
rlandy|ruckand wallaby c821:02
rlandy|ruckyou can try rerun failing jobs that are in criteria21:02
rlandy|ruckI will rerun the integration line failing jobs21:02
rlandy|ruckdasm|rover: you can leave notes for marios on your results21:03
dasm|roverack21:03
rlandy|ruckwe will need to start cleaning up these lines when patches are merged21:03
rlandy|ruckdasm|rover: thank you, sir!21:03
dasm|roverno worries21:04
dasm|roverbrb21:24
dasm|roverback21:25
dasm|roveri think we're still waiting on: https://review.opendev.org/c/openstack/tripleo-quickstart/+/827279/21:27
dasm|roverakahat: we're in a better shape right now. dviroel left some comments -- i didn't address them. for failing job, it seems to be similar to the issue which amoralej helped us address in the past ( kmod.rb ).22:21
dasm|roverrlandy|ruck: fyi, i'm rerunning 2 failing comoponent jobs: https://review.rdoproject.org/r/c/testproject/+/38893 22:22
* dasm|rover is leaving!22:23
dasm|rovero/22:23
*** dasm|rover is now known as dasm|off22:23
rlandy|ruckdasm|off: thanks22:23
*** dviroel|afk is now known as dviroel23:44

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!