| *** ysandeep is now known as ysandeep|ruck | 00:41 | |
| chandankumar | ysandeep|ruck: good morning | 04:57 |
|---|---|---|
| chandankumar | ysandeep|ruck: can we disable modify image role for time-being ? | 04:57 |
| chandankumar | or disable specific repo update? | 04:58 |
| chandankumar | thank you for digging dip into that. :-) | 04:59 |
| ysandeep|ruck | chandankumar, good morning o/ yes trying to confirm if older rpms fixes the issue, as issue is transient need to rerun jobs more times to be sure. | 05:19 |
| *** bhagyashris is now known as bhagyashris|ruck | 05:21 | |
| ysandeep|ruck | bhagyashris|ruck, good morning o/ | 05:25 |
| *** ysandeep|ruck is now known as ysandeep|rover | 05:26 | |
| bhagyashris|ruck | ysandeep|rover, Good morning ! :) | 05:26 |
| ysandeep|rover | bhagyashris|ruck, want to sync to divide and conquer | 05:27 |
| bhagyashris|ruck | ysandeep|rover, sure | 05:27 |
| ysandeep|rover | ack, will pm you | 05:28 |
| * pojadhav brb | 06:26 | |
| *** ysandeep|rover is now known as ysandeep|rover|afk | 06:38 | |
| chandankumar | bhagyashris|ruck: similar bug https://bugzilla.redhat.com/show_bug.cgi?id=2155466 | 08:02 |
| chandankumar | regarding selinux filed by pooja and got closed | 08:02 |
| chandankumar | sorry not this one | 08:03 |
| chandankumar | bhagyashris|ruck: this one https://bugzilla.redhat.com/show_bug.cgi?id=2154764 | 08:03 |
| *** yadnesh|away is now known as yadnesh | 08:23 | |
| bhagyashris|ruck | chandankumar, hey but we are hitting that now | 08:31 |
| *** ysandeep|rover|afk is now known as ysandeep|rover | 08:34 | |
| ysandeep|rover | bhagyashris|ruck, looks like we just fixed tripleo based jobs and we still need the fix in jenkins jobs. | 08:45 |
| ysandeep|rover | chandankumar, sry for the duplicate, but now that we have reported - would be good to keep it open so that QE folks can fix their jenkins jobs | 08:46 |
| ysandeep|rover | bhagyashris|ruck, please add a comment to your newly reported bug/cix that we faced similiar issue in tripleo jobs with older/duplicate bug in comments, and we are keeping this bug open to track jenkins job fix. | 08:47 |
| ysandeep|rover | bhagyashris|ruck, wohoo zed promoted as well :D | 08:58 |
| bhagyashris|ruck | ysandeep|rover, ack | 09:10 |
| bhagyashris|ruck | ysandeep|rover, wohoo great thanks | 09:10 |
| *** ysandeep|rover is now known as ysandeep|lunch | 09:29 | |
| chandankumar | ysandeep|lunch: sounds good! | 09:45 |
| *** ysandeep|lunch is now known as ysandeep | 10:43 | |
| ysandeep | chandankumar, bhagyashris|ruck fyi.. good news based on testing so far.. looks like issue XFS issue is coming after newer kernel. I will try just kernel downgrade next to confirm the theory. | 11:04 |
| ysandeep | There are few issues in our ovb jobs which we need to deal with. Like component ovb jobs don't use quickstart-* repos at all | 11:05 |
| ysandeep | So we can't use release file to pin | 11:06 |
| ysandeep | chandankumar, bhagyashris|ruck see: https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/1cd7430/logs/overcloud-controller-1/etc/yum.repos.d/ | 11:07 |
| ysandeep | https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/1cd7430/logs/overcloud-controller-1/var/log/extra/package-list-installed.txt.gz | 11:08 |
| ysandeep | kernel.x86_64 5.14.0-214.el9 @baseos | 11:08 |
| ysandeep | https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/1cd7430/logs/undercloud/home/zuul/build.log.txt.gz | 11:14 |
| ysandeep | ~~~ | 11:14 |
| ysandeep | 2022-12-27 00:54:07.471 | kernel x86_64 5.14.0-214.el9 baseos 2.8 M | 11:14 |
| ysandeep | ~~~ | 11:14 |
| ysandeep | ^^ weird DIB_YUM_REPO_CONF='/etc/yum.repos.d/delorean* /etc/yum.repos.d/quickstart-* /etc/yum.repos.d/*-component.repo' is right. But looks like inbuilt centos repos are conflicting. | 11:15 |
| ysandeep | inshort.. Periodic line jobs can be fixed with kernel exclude in release but need different fix for centos | 11:15 |
| ysandeep | for component line jobs* | 11:16 |
| ysandeep | ah vs periodic line job: https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-9-buildimage-overcloud-hardened-uefi-full-wallaby/4d8a9f5/build_images.sh | 11:21 |
| ysandeep | export DIB_YUM_REPO_CONF='/etc/yum.repos.d/quickstart-centos* /etc/yum.repos.d/delorean* /etc/yum.repos.d/centos.repo ' | 11:21 |
| ysandeep | we should also include /etc/yum.repos.d/centos.repo in component line jobs | 11:21 |
| ysandeep | so that disabled repos of centos.repo(on host) wins when buidling image | 11:22 |
| ysandeep | we need similiar fix(https://github.com/openstack/tripleo-ci/commit/b2ce4b4c101d68eaa71d1499609423e10f50f2d5) here: https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/build-images/defaults/main.yml#L16 | 11:39 |
| * ysandeep proposing | 11:39 | |
| ysandeep | we should really use same roles in imagebuild/container build jobs vs when we do same in ovb job using operators. | 11:40 |
| chandankumar | ysandeep: may be we can remove this line https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/build-images/tasks/main.yml#L44 | 11:44 |
| chandankumar | it will use the vars from there | 11:44 |
| ysandeep | chandankumar, I think we need that because we need to pass component.repo too | 11:45 |
| ysandeep | https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/build-images/defaults/main.yml#L17 | 11:45 |
| ysandeep | default var don't have component repo | 11:45 |
| chandankumar | yes, ok | 11:46 |
| chandankumar | so either we can add centos.repo in tqe or move the component logic in toci | 11:46 |
| ysandeep | yes..add centos.repo in tqe seems easier for now | 11:47 |
| ysandeep | but ideally yes.. we should avoid duplication of roles | 11:47 |
| ysandeep | chandankumar, bhagyashris|ruck first green run with pinned kernel and some other repos: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-zed&skip=0 :D | 11:55 |
| ysandeep | chandankumar, bhagyashris|ruck https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/868677 will ping for review after testing | 11:56 |
| bhagyashris|ruck | ysandeep, ack | 11:56 |
| *** ysandeep is now known as ysandeep|afk | 12:11 | |
| *** ysandeep|afk is now known as ysandeep | 13:05 | |
| *** ysandeep is now known as ysandeep|rover | 13:10 | |
| ysandeep|rover | bhagyashris|ruck, need help with anything? | 13:11 |
| ysandeep|rover | wohoo.. component job passed node provisioning finally passed https://review.rdoproject.org/zuul/stream/672d40b2c591408a8d1ecac02a92fa64?logfile=console.log | 13:21 |
| ysandeep|rover | bhagyashris|ruck, older kernel works, but i wonder if we are hitting this issue because we are mixing content(and this is not a kernel bug), instead of excluding kernel maybe we should fix https://logserver.rdoproject.org/37/28537/63/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-zed/7dca9fd/logs/overcloud-controller-0/etc/yum.repos.d/quickstart-centos-appstreams.repo.txt.gz to use local mirror | 13:33 |
| chandankumar | ysandeep|rover: local mirror means afs mirror? | 13:40 |
| ysandeep|rover | chandankumar, vexx/ibm mirror | 13:41 |
| chandankumar | using local mirror sounds good if it fixes the issue | 13:41 |
| ysandeep|rover | chandankumar, undercloud have correct setting: https://logserver.rdoproject.org/37/28537/63/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-zed/7dca9fd/logs/undercloud/etc/yum.repos.d/quickstart-centos-appstreams.repo.txt.gz | 13:41 |
| ysandeep|rover | overcloud have wrong entry: https://logserver.rdoproject.org/37/28537/63/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-zed/7dca9fd/logs/overcloud-controller-0/etc/yum.repos.d/quickstart-centos-appstreams.repo.txt.gz | 13:42 |
| ysandeep|rover | baseurl=http://mirror.stream.centos.org/9-stream/AppStream/x86_64/os/ | 13:42 |
| ysandeep|rover | I am checking how can we fix baseurl for overcloud nodes.. looks related to repo-setup and modify-image | 13:42 |
| ysandeep|rover | I will keep digging but incase you have pointers about ^^ | 13:43 |
| ysandeep|rover | chandankumar, yes using local mirror works as vexx mirror don't have affected kernel: http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos-stream/9-stream/BaseOS/x86_64/os/Packages/ | 13:45 |
| ysandeep|rover | yet.. but may come in future when finally mirror sync from centos.. but at that time we will know if this is kernel bug or because of content mismatch. | 13:46 |
| ysandeep|rover | leaking content because of baseurl=http://mirror.stream.centos.org/9-stream/AppStream/x86_64/os/ is wrong. | 13:46 |
| ysandeep|rover | I don't want to disable modify_image role as that would break depends-on functionality | 13:47 |
| chandankumar | ysandeep|rover: may be doing something like sudo sed -i -e "s|http://mirror.stream.centos.org|$NODEPOOL_CENTOS_MIRROR|g" /etc/yum.repos.d/delorean-master-deps.repo | 13:49 |
| chandankumar | for centos repo in release file | 13:49 |
| ysandeep|rover | chandankumar, https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-9/promotion-testing-hash-master.yml#L52-L53 maybe this is not working because we don't have /etc/ci/mirror_info.sh in image | 13:51 |
| ysandeep|rover | looks like if we copy mirror_info.sh to overcloud image.. things will work | 13:52 |
| bhagyashris|ruck | ysandeep|rover, nope | 13:52 |
| ysandeep|rover | chandankumar, regarding sed i think issue is $NODEPOOL_CENTOS_MIRROR var is not set | 13:53 |
| ysandeep|rover | bhagyashris|ruck, ack o/ | 13:53 |
| chandankumar | yes, it depends on mirror_info.sh script | 13:54 |
| chandankumar | what if we remove centos.repo from tripleo-ci then? | 13:54 |
| ysandeep|rover | for periodic line we don't have centos.repo in overcloud node | 13:55 |
| chandankumar | then we donot need to do anything | 13:55 |
| ysandeep|rover | chandankumar, quickstart repo is wrong: https://logserver.rdoproject.org/37/28537/63/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-zed/7dca9fd/logs/overcloud-controller-0/etc/yum.repos.d/quickstart-centos-appstreams.repo.txt.gz | 13:55 |
| ysandeep|rover | baseurl=http://mirror.stream.centos.org/9-stream/AppStream/x86_64/os/ | 13:55 |
| ysandeep|rover | instead of vexx mirror ^^ | 13:55 |
| ysandeep|rover | centos.repo was issue in component line jobs during image build itself. | 13:56 |
| ysandeep|rover | let me try copying mirror_info.sh to overcloud image before we execute modify-image and see if that helps | 13:59 |
| * ysandeep|rover brb food | 14:00 | |
| *** ysandeep|rover is now known as ysandeep|food | 14:00 | |
| chandankumar | on overcloud image, centos.repo is going exist already | 14:06 |
| chandankumar | if we copy mirror_info.sh then we again need to run sed on centos.repo | 14:09 |
| chandankumar | via modify image role | 14:09 |
| chandankumar | let's take a look tomorrow | 14:09 |
| chandankumar | see ya! | 14:09 |
| *** ysandeep|food is now known as ysandeep | 14:14 | |
| ysandeep | chandankumar, fyi.. because we are mounting centos.repo from host to image during image build, dib remove centos.repo from overcloud image at the end of dib execution. | 14:15 |
| ysandeep | chandankumar, please see this commit: https://github.com/openstack/tripleo-ci/commit/b2ce4b4c101d68eaa71d1499609423e10f50f2d5 | 14:16 |
| ysandeep | ~~~ | 14:16 |
| ysandeep | One change in behavior is at the end of image build, dib will delete | 14:16 |
| ysandeep | the centos.repos in the overcloud image as dib cleans up what it adds | 14:16 |
| ysandeep | ignoring what was already present. | 14:16 |
| ysandeep | ~~~ | 14:16 |
| ysandeep | ykarel, you helped me earlier with similiar patch.. when you get a chance could you please look: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/868677 | 14:18 |
| ysandeep | need for build-image role as well, we fixed oooci-build-images earlier: https://github.com/openstack/tripleo-ci/commit/b2ce4b4c101d68eaa71d1499609423e10f50f2d5 but missed the other role. | 14:19 |
| ysandeep | bhagyashris|ruck, see you tomorrow o/ | 14:42 |
| *** ysandeep is now known as ysandeep|out | 14:42 | |
| dviroel | bhagyashris|ruck: hi, let me know if there is something to follow up in my afternoon | 14:48 |
| ykarel | ysandeep ok but seems CI is broken, may be due to tox4 | 15:03 |
| bhagyashris|ruck | ysandeep, see you 0/ | 16:01 |
| bhagyashris|ruck | dviroel, not much | 16:02 |
| dviroel | bhagyashris|ruck: ok | 16:13 |
| dviroel | bhagyashris|ruck: going to propose some patches to fix tox4 issues on our repos | 16:13 |
| *** dviroel_ is now known as dviroel | 16:38 | |
| *** yadnesh is now known as yadnesh|away | 17:02 | |
| *** dviroel is now known as dviroel|out | 20:17 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!