| opendevreview | Merged openstack/tripleo-upgrade stable/wallaby: Make config-heat and config-resource options comptatible with OOOQ https://review.opendev.org/c/openstack/tripleo-upgrade/+/845680 | 00:10 |
|---|---|---|
| opendevreview | Takashi Kajinami proposed openstack/tripleo-common master: Remove password migrations for placement and rpc services https://review.opendev.org/c/openstack/tripleo-common/+/846073 | 00:13 |
| opendevreview | Takashi Kajinami proposed openstack/python-tripleoclient master: Undercloud: Deprecate hieradata_override option https://review.opendev.org/c/openstack/python-tripleoclient/+/846074 | 00:36 |
| opendevreview | Merged openstack/diskimage-builder master: Fix backward regex match https://review.opendev.org/c/openstack/diskimage-builder/+/846052 | 00:52 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Revert "CentOS 9-stream : work around selinux permissions issue" https://review.opendev.org/c/openstack/diskimage-builder/+/846078 | 01:24 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates stable/wallaby: Ironic: Use generated dnsmasq conf file https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846079 | 01:36 |
| opendevreview | John Fulton proposed openstack/tripleo-quickstart-extras master: Test Deployed Ceph in Wallaby standalone 001/004/010 https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/845054 | 02:09 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: ironic tftp: Do not format IPv6 address for bind_host https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846081 | 02:35 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: ironic: Remove tasks for image migration https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846083 | 02:47 |
| opendevreview | Brendan Shephard proposed openstack/tripleo-heat-templates master: Revert "task-core basic framework" https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841131 | 02:52 |
| opendevreview | Brendan Shephard proposed openstack/tripleo-heat-templates master: Fixing typo in HAProxyBase for DCN https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846084 | 02:52 |
| opendevreview | Brendan Shephard proposed openstack/tripleo-heat-templates master: Fixing typo in HAProxyBase for DCN https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846084 | 02:54 |
| tkajinam | hmmmm. looks like content provider job is broken now | 03:06 |
| *** ysandeep|out is now known as ysandeep | 03:45 | |
| *** soniya29 is now known as soniya29|ruck | 04:37 | |
| tkajinam | chkumar|rover, hi. are you aware of broken content provider jobs ? https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846081 | 04:44 |
| tkajinam | soniya29|ruck, ^^^ | 04:44 |
| soniya29|ruck | tkajinam, let me have a look | 04:44 |
| tkajinam | soniya29|ruck, thanks | 04:45 |
| soniya29|ruck | tkajinam, it looks like we need 2 patches get merged before we get the content-provider job stable | 04:50 |
| soniya29|ruck | https://review.rdoproject.org/r/c/testproject/+/43558 | 04:50 |
| soniya29|ruck | https://review.rdoproject.org/r/c/rdo-jobs/+/42779 | 04:50 |
| soniya29|ruck | https://review.opendev.org/c/openstack/tripleo-ci/+/841583 | 04:51 |
| soniya29|ruck | tkajinam, marios is working over them..we will try to get patches merged soon | 04:51 |
| tkajinam | ok | 04:52 |
| tkajinam | soniya29|ruck, ramishra++ found the failing package installation, which is likely to be the direct cause for the failures I'm facing https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d7e/846081/1/check/tripleo-ci-centos-9-content-provider/d7e6218/logs/undercloud/home/zuul/workspace/logs/container-builds/f4d4c433-3e5c-4187-aa35-036708efd7fa/base/redis/redis-build.log | 04:53 |
| tkajinam | loos like a mirror problem | 04:53 |
| opendevreview | Brendan Shephard proposed openstack/tripleo-heat-templates stable/wallaby: Add sample baremetal_deployment.yaml https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841114 | 05:01 |
| chkumar|rover | tkajinam: yes, it is a mirror issue, I rechecked the patch | 05:25 |
| tkajinam | chkumar|rover, thanks | 05:26 |
| *** soniya29 is now known as soniya29|ruck | 05:29 | |
| opendevreview | Pranali Deore proposed openstack/tripleo-heat-templates master: Enabling ``use_keystone_limits`` in glance to use keystone limits https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846099 | 05:43 |
| opendevreview | chandan kumar proposed openstack/tripleo-ci master: Revert "Install Python38 on CentOS-8" https://review.opendev.org/c/openstack/tripleo-ci/+/846113 | 05:53 |
| opendevreview | chandan kumar proposed openstack/tripleo-ci master: Revert "Install Python38 on CentOS-8" https://review.opendev.org/c/openstack/tripleo-ci/+/846113 | 05:56 |
| *** amoralej|off is now known as amoralej | 06:19 | |
| *** soniya is now known as soniya|ruck | 06:45 | |
| opendevreview | Francesco Pantano proposed openstack/tripleo-heat-templates master: DNM - QUINCY - Test Pending Ceph daemons in CI https://review.opendev.org/c/openstack/tripleo-heat-templates/+/778915 | 06:50 |
| opendevreview | Cedric Jeanneret proposed openstack/tripleo-heat-templates stable/wallaby: Workaround for subscription-manager https://review.opendev.org/c/openstack/tripleo-heat-templates/+/845674 | 06:56 |
| opendevreview | Cedric Jeanneret proposed openstack/python-tripleoclient stable/wallaby: Move Ansible imports to correct UX Warnings https://review.opendev.org/c/openstack/python-tripleoclient/+/841286 | 06:57 |
| opendevreview | Takashi Kajinami proposed openstack/python-tripleoclient master: Undercloud: Deprecate hieradata_override option https://review.opendev.org/c/openstack/python-tripleoclient/+/846074 | 07:02 |
| opendevreview | Cedric Jeanneret proposed openstack/tripleo-heat-templates master: Verify openstack-selinux package is properly installed https://review.opendev.org/c/openstack/tripleo-heat-templates/+/845569 | 07:05 |
| *** jpena|off is now known as jpena | 07:42 | |
| opendevreview | Cedric Jeanneret proposed openstack/validations-common master: New validation: verify_package https://review.opendev.org/c/openstack/validations-common/+/845559 | 07:43 |
| chkumar|rover | tkajinam: ramishra marios soniya|ruck fyi https://bugs.launchpad.net/tripleo/+bug/1978929 blocking cs9 jobs | 08:10 |
| chkumar|rover | Ianw replaced the corrupted file and I have rechecked the one of the affected patch, once it passes, I will let you know to recheck the other patches | 08:11 |
| ramishra | chkumar|rover: thanks for tracking that. | 08:17 |
| opendevreview | Marios Andreou proposed openstack/tripleo-quickstart-extras master: WIP add some new plays for mixed mode https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/841764 | 08:21 |
| opendevreview | Marios Andreou proposed openstack/tripleo-quickstart master: WIP add new featureset66 for mixed deployment jobs https://review.opendev.org/c/openstack/tripleo-quickstart/+/841582 | 08:21 |
| marios | thanks chkumar|rover let me know if you need votes on something | 08:22 |
| marios | ah another mirror related issue chkumar|rover :/ sorry and thanks for dealinlg | 08:24 |
| marios | (dealing) | 08:24 |
| opendevreview | Sofer Athlan-Guyot proposed openstack/tripleo-upgrade master: Fix validation inventory setup. https://review.opendev.org/c/openstack/tripleo-upgrade/+/842087 | 08:26 |
| opendevreview | Rabi Mishra proposed openstack/python-tripleoclient stable/wallaby: Only export overcloud for --stack-only/full deploy https://review.opendev.org/c/openstack/python-tripleoclient/+/846121 | 08:31 |
| opendevreview | Pooja Jadhav proposed openstack/tripleo-ci master: Move Victoria for Tripleo Repos to End Of Life https://review.opendev.org/c/openstack/tripleo-ci/+/845587 | 08:33 |
| opendevreview | Jiri Podivin proposed openstack/validations-libs stable/1.6: Man pages compatibility https://review.opendev.org/c/openstack/validations-libs/+/843783 | 08:38 |
| opendevreview | Pooja Jadhav proposed openstack/tripleo-ci master: Move Victoria for Tripleo Repos to End Of Life https://review.opendev.org/c/openstack/tripleo-ci/+/845587 | 08:39 |
| opendevreview | Cedric Jeanneret proposed openstack/validations-common master: New validation: verify_package https://review.opendev.org/c/openstack/validations-common/+/845559 | 08:40 |
| opendevreview | Jiri Podivin proposed openstack/validations-libs stable/1.6: Man pages compatibility https://review.opendev.org/c/openstack/validations-libs/+/843783 | 08:42 |
| opendevreview | Jiri Podivin proposed openstack/validations-libs stable/1.6: [Stable-Only] Refactoring the stable/1.6 CI pipeline https://review.opendev.org/c/openstack/validations-libs/+/841713 | 08:42 |
| opendevreview | Sofer Athlan-Guyot proposed openstack/tripleo-upgrade stable/wallaby: Fix validation inventory setup. https://review.opendev.org/c/openstack/tripleo-upgrade/+/846131 | 08:43 |
| opendevreview | Jiri Podivin proposed openstack/validations-libs stable/1.6: Clarifying project branching model in CONTRIBUTING.rst https://review.opendev.org/c/openstack/validations-libs/+/843782 | 08:43 |
| opendevreview | Sofer Athlan-Guyot proposed openstack/tripleo-upgrade stable/victoria: Fix validation inventory setup. https://review.opendev.org/c/openstack/tripleo-upgrade/+/846132 | 08:45 |
| opendevreview | Sofer Athlan-Guyot proposed openstack/tripleo-upgrade stable/ussuri: Fix validation inventory setup. https://review.opendev.org/c/openstack/tripleo-upgrade/+/846133 | 08:47 |
| opendevreview | Sofer Athlan-Guyot proposed openstack/tripleo-upgrade stable/train: Fix validation inventory setup. https://review.opendev.org/c/openstack/tripleo-upgrade/+/846134 | 08:50 |
| opendevreview | Gregory Thiemonge proposed openstack/tripleo-heat-templates master: Octavia Jobboard integration https://review.opendev.org/c/openstack/tripleo-heat-templates/+/820530 | 09:00 |
| opendevreview | Damien Ciabrini proposed openstack/puppet-tripleo master: Support mariabackup as a galera SST method https://review.opendev.org/c/openstack/puppet-tripleo/+/836189 | 09:02 |
| chkumar|rover | ramishra: tkajinam now you can recheck your patches | 09:09 |
| tkajinam | chkumar|rover, thanks. good news :-) | 09:10 |
| opendevreview | Sofer Athlan-Guyot proposed openstack/tripleo-upgrade stable/victoria: Fix validation inventory setup. https://review.opendev.org/c/openstack/tripleo-upgrade/+/846132 | 09:15 |
| opendevreview | Sofer Athlan-Guyot proposed openstack/tripleo-upgrade stable/ussuri: Fix validation inventory setup. https://review.opendev.org/c/openstack/tripleo-upgrade/+/846133 | 09:16 |
| opendevreview | Sofer Athlan-Guyot proposed openstack/tripleo-upgrade stable/train: Fix validation inventory setup. https://review.opendev.org/c/openstack/tripleo-upgrade/+/846134 | 09:16 |
| opendevreview | Brendan Shephard proposed openstack/tripleo-heat-templates master: Revert "task-core basic framework" https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841131 | 09:18 |
| opendevreview | Damien Ciabrini proposed openstack/tripleo-heat-templates master: Galera: initial support for mariabackup SST https://review.opendev.org/c/openstack/tripleo-heat-templates/+/836277 | 09:27 |
| opendevreview | Pooja Jadhav proposed openstack/tripleo-ci master: Move Victoria for Tripleo Repos to End Of Life https://review.opendev.org/c/openstack/tripleo-ci/+/845587 | 09:37 |
| opendevreview | Pooja Jadhav proposed openstack/tripleo-ci master: Move Victoria for Tripleo Repos to End Of Life https://review.opendev.org/c/openstack/tripleo-ci/+/845587 | 09:40 |
| opendevreview | Pooja Jadhav proposed openstack/tripleo-ci master: Move Victoria for Tripleo Repos to End Of Life https://review.opendev.org/c/openstack/tripleo-ci/+/845587 | 09:42 |
| opendevreview | Pooja Jadhav proposed openstack/tripleo-ci master: Move Victoria for Tripleo Repos to End Of Life https://review.opendev.org/c/openstack/tripleo-ci/+/845587 | 09:51 |
| opendevreview | Pooja Jadhav proposed openstack/tripleo-ci master: Move Victoria for Tripleo Repos to End Of Life https://review.opendev.org/c/openstack/tripleo-ci/+/845587 | 09:56 |
| opendevreview | Manojkatari proposed openstack/puppet-tripleo master: Support to configure cinder backend_defaults https://review.opendev.org/c/openstack/puppet-tripleo/+/845854 | 10:03 |
| opendevreview | Rabi Mishra proposed openstack/tripleo-ansible stable/train: Run all validations in check mode https://review.opendev.org/c/openstack/tripleo-ansible/+/845837 | 10:13 |
| *** rlandy|out is now known as rlandy | 10:32 | |
| *** soniya is now known as soniya29|ruck | 11:00 | |
| *** ysandeep is now known as ysandeep|afk | 11:08 | |
| *** dviroel|afk is now known as dviroel | 11:18 | |
| opendevreview | Rajesh Tailor proposed openstack/tripleo-ansible master: Remove - from endfor statement https://review.opendev.org/c/openstack/tripleo-ansible/+/845333 | 11:23 |
| opendevreview | Gregory Thiemonge proposed openstack/puppet-tripleo master: Add support for ovn clustered DB in the octavia ovn-provider https://review.opendev.org/c/openstack/puppet-tripleo/+/846059 | 11:34 |
| opendevreview | Merged openstack/tripleo-common stable/train: build_catalog should loop deeper than one level https://review.opendev.org/c/openstack/tripleo-common/+/837805 | 11:38 |
| *** soniya29 is now known as soniya29|ruck | 11:47 | |
| opendevreview | Francesco Pantano proposed openstack/tripleo-quickstart-extras master: DNM - QUINCY - Test Pending Ceph daemons in CI https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846152 | 11:49 |
| opendevreview | Daniel Bengtsson proposed openstack/tripleo-upgrade master: Changed minversion in tox to 3.18.0. https://review.opendev.org/c/openstack/tripleo-upgrade/+/835028 | 11:56 |
| *** soniya is now known as soniya29|ruck | 12:05 | |
| *** amoralej is now known as amoralej|lunch | 12:18 | |
| *** ysandeep|afk is now known as ysandeep | 12:18 | |
| opendevreview | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Only deploy ceph before the overcloud in master https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846159 | 12:29 |
| *** soniya is now known as soniya29|ruck | 12:39 | |
| opendevreview | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Only deploy ceph before the overcloud in master https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846159 | 13:00 |
| *** rlandy is now known as rlandy|rover | 13:03 | |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Swift: Disable package management by puppet-rsync explicitly https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846166 | 13:08 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Swift: Do not enable the whole rsync::server class https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846166 | 13:12 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Swift: Do not enable the whole rsync::server class https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846166 | 13:14 |
| *** amoralej|lunch is now known as amoralej | 13:26 | |
| opendevreview | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Only deploy ceph before the overcloud in master https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846159 | 13:40 |
| opendevreview | John Fulton proposed openstack/tripleo-heat-templates master: Deprecate CephPoolDefaultPgNum and CephPoolDefaultSize https://review.opendev.org/c/openstack/tripleo-heat-templates/+/844632 | 13:45 |
| opendevreview | Jiri Podivin proposed openstack/validations-libs stable/1.6: [Stable-Only] Refactoring the stable/1.6 CI pipeline https://review.opendev.org/c/openstack/validations-libs/+/841713 | 13:45 |
| opendevreview | Jiri Podivin proposed openstack/validations-libs stable/1.6: Clarifying project branching model in CONTRIBUTING.rst https://review.opendev.org/c/openstack/validations-libs/+/843782 | 13:45 |
| opendevreview | John Fulton proposed openstack/tripleo-heat-templates master: Deprecate CephPoolDefaultPgNum and CephPoolDefaultSize https://review.opendev.org/c/openstack/tripleo-heat-templates/+/844632 | 13:46 |
| *** soniya is now known as soniya|ruck | 13:58 | |
| rlandy|rover | wrt ceph changes on master: https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-scenario010-standalone&skip=0 | 13:58 |
| rlandy|rover | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_499/846059/2/check/tripleo-ci-centos-9-scenario010-standalone/4995951/job-output.txt | 13:58 |
| rlandy|rover | 2022-06-16 13:07:41.358824 | primary | TASK [tripleo.operator.tripleo_ceph_user : Enable Ceph User] ******************* | 13:58 |
| rlandy|rover | 2022-06-16 13:07:41.358915 | primary | Thursday 16 June 2022 13:07:41 +0000 (0:00:02.064) 0:09:31.171 ********* | 13:58 |
| rlandy|rover | fultonj: fmount: ^^ | 13:59 |
| rlandy|rover | soniya29|ruck: ^^ | 14:00 |
| fultonj | rlandy|rover: that's intermittent but happens with or without deployed ceph | 14:02 |
| fultonj | https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/834352 | 14:02 |
| fultonj | has more details in june 14 comment | 14:03 |
| fultonj | i'll paste here as it's a big review | 14:03 |
| fmount | thanks fultonj | 14:03 |
| rlandy|rover | ok | 14:03 |
| fultonj | https://paste.opendev.org/show/bwkF1HEb4DKraWwIhdzu/ | 14:03 |
| fultonj | rlandy|rover: i'll put up a patch to try to deal with non-idempotence of user creation role | 14:04 |
| fultonj | intermitent non-idempotence | 14:04 |
| rlandy|rover | thanks | 14:04 |
| *** ysandeep is now known as ysandeep|afk | 14:08 | |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-ansible master: Add a new role to manage certificate files https://review.opendev.org/c/openstack/tripleo-ansible/+/846178 | 14:11 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Manage CA certificates using ansible https://review.opendev.org/c/openstack/tripleo-heat-templates/+/839296 | 14:16 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-ansible master: Add a new role to manage certificate files https://review.opendev.org/c/openstack/tripleo-ansible/+/846178 | 14:21 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-ansible master: Add a new role to manage certificate files https://review.opendev.org/c/openstack/tripleo-ansible/+/846178 | 14:26 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-ansible master: Sort job order https://review.opendev.org/c/openstack/tripleo-ansible/+/846179 | 14:29 |
| *** soniya29 is now known as soniya29|ruck | 14:38 | |
| *** ysandeep|afk is now known as ysandeep | 14:57 | |
| *** amoralej is now known as amoralej|off | 15:04 | |
| rlandy|rover | https://review.rdoproject.org/r/c/testproject/+/39960 - some mixed results here | 15:13 |
| rlandy|rover | fultonj: ^^ fyi | 15:13 |
| *** soniya29|ruck is now known as soniya29|out | 15:18 | |
| opendevreview | Merged openstack/tripleo-operator-ansible master: update tripleo_operator_ansible VF hooks since CLI has changed https://review.opendev.org/c/openstack/tripleo-operator-ansible/+/845988 | 15:41 |
| opendevreview | Merged openstack/validations-common stable/1.6: Explaining stable branches in CONTRIBUTING.rst https://review.opendev.org/c/openstack/validations-common/+/843784 | 15:41 |
| opendevreview | Merged openstack/validations-libs stable/1.6: Man pages compatibility https://review.opendev.org/c/openstack/validations-libs/+/843783 | 15:41 |
| opendevreview | Ronelle Landy proposed openstack/tripleo-quickstart master: DNM - Testing downgrade of libvirt https://review.opendev.org/c/openstack/tripleo-quickstart/+/846194 | 15:42 |
| opendevreview | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Only deploy ceph before the overcloud in master https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846159 | 15:45 |
| rlandy|rover | dviroel: marios: ^^ pls vote on ... results in https://code.engineering.redhat.com/gerrit/c/testproject/+/398466 | 15:46 |
| opendevreview | Rafael Castillo proposed openstack/tripleo-quickstart master: DNM: Test building latest sdk in dependency https://review.opendev.org/c/openstack/tripleo-quickstart/+/845375 | 15:50 |
| *** ysandeep is now known as ysandeep|out | 15:58 | |
| marios | rlandy|rover: ack | 16:01 |
| fmount | rlandy|rover: late to the party but basically https://review.rdoproject.org/r/c/testproject/+/39960 confirms that we're executing "deployed ceph" only in mastr | 16:31 |
| fmount | (I looked at the logs) | 16:32 |
| fmount | master -> deployed ceph fails with -> Error: initializing source docker | 16:33 |
| fmount | wallaby -> ceph is deployed "during" the overcloud deployment -> works as expected | 16:33 |
| fmount | fultonj: ^^ | 16:33 |
| *** jpena is now known as jpena|off | 16:44 | |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-ansible master: Add a new role to manage certificate files https://review.opendev.org/c/openstack/tripleo-ansible/+/846178 | 16:47 |
| rlandy|rover | fmount: ack p only in master but stil failing | 16:47 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Manage CA certificates using ansible https://review.opendev.org/c/openstack/tripleo-heat-templates/+/839296 | 16:47 |
| rlandy|rover | following two gate blocker now - both patches in test | 16:48 |
| opendevreview | Roger Heslop proposed openstack/tripleo-docs master: Changed spelling typo: neuron > neutron in network https://review.opendev.org/c/openstack/tripleo-docs/+/846203 | 16:57 |
| opendevreview | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Revert "Temporarily freeze openssl" https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846125 | 17:13 |
| opendevreview | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Manage CA certificates using ansible https://review.opendev.org/c/openstack/tripleo-heat-templates/+/839296 | 17:24 |
| fultonj | we've got 4 +2's on https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846159 | 17:24 |
| fultonj | rlandy|rover: ^ any reason not to +w it | 17:24 |
| rlandy|rover | will do after zuul votes | 17:30 |
| fultonj | rlandy|rover: ack | 17:30 |
| rlandy|rover | fultonj: it failed on a diff issue | 17:30 |
| rlandy|rover | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Revert "Temporarily freeze openssl" https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846125 | 17:30 |
| rlandy|rover | centos-9 updated | 17:31 |
| rlandy|rover | testing three gate blockers at once | 17:31 |
| fultonj | thanks rlandy|rover | 17:32 |
| rlandy|rover | fultonj: lovely day :( | 17:32 |
| rlandy|rover | love it when centos mirror update | 17:32 |
| rlandy|rover | worst case - will update the ssl patch rather the revert it | 17:33 |
| fultonj | are centos mirror updates unpredictable? | 17:33 |
| fultonj | no worries if you're too busy fixing the problem to talk about it though | 17:35 |
| opendevreview | Rafael Castillo proposed openstack/tripleo-quickstart master: DNM: Test building latest sdk in dependency https://review.opendev.org/c/openstack/tripleo-quickstart/+/845375 | 17:51 |
| rlandy|rover | completely unpredictable | 17:53 |
| opendevreview | Jay Faulkner proposed openstack/diskimage-builder master: Add support for Python 3.10 https://review.opendev.org/c/openstack/diskimage-builder/+/844613 | 17:57 |
| rlandy|rover | https://411deec051d768e42c33-554a55c2926ce345d8a8f0805ecfe993.ssl.cf5.rackcdn.com/846159/4/check/tripleo-ci-centos-9-scenario001-standalone/a9a4889/logs/undercloud/var/log/extra/podman/containers/gnocchi_db_sync/stdout.log | 18:14 |
| rlandy|rover | fultonj: ^^ failed there | 18:14 |
| rlandy|rover | fultonj: maybe we can meet to look at all the different failures here | 18:16 |
| rlandy|rover | master ones | 18:16 |
| fultonj | hi rlandy|rover | 18:20 |
| * fultonj looking | 18:20 | |
| fultonj | is this centos9 w/ master branch? | 18:21 |
| rlandy|rover | yes - but ceph related | 18:22 |
| rlandy|rover | https://411deec051d768e42c33-554a55c2926ce345d8a8f0805ecfe993.ssl.cf5.rackcdn.com/846159/4/check/tripleo-ci-centos-9-scenario001-standalone/a9a4889/logs/undercloud/var/log/extra/podman/containers/gnocchi_db_sync/stdout.log | 18:22 |
| fultonj | gnocchi | 18:25 |
| fultonj | leifmadsen: ^ | 18:28 |
| fultonj | rlandy|rover: i'm looking at it trying to determine how gnocchi should be behaving | 18:28 |
| rlandy|rover | ok | 18:29 |
| * fultonj looking for that email about legacy telemtry | 18:29 | |
| fultonj | rlandy|rover: ceph client configuration has not changed from the TQE patch | 18:30 |
| fultonj | it still happens during overcloud configuration | 18:30 |
| rlandy|rover | yes | 18:30 |
| fultonj | so in theory it should be unrelated but let's find out why it's failing | 18:30 |
| opendevreview | Ronelle Landy proposed openstack/tripleo-quickstart master: DNM - Testing downgrade of libvirt https://review.opendev.org/c/openstack/tripleo-quickstart/+/846194 | 18:33 |
| leifmadsen | ? | 18:34 |
| fultonj | leifmadsen: would you expect gnocchi to be connecting to ceph in tripleo master jobs? | 18:35 |
| leifmadsen | umm... I don't think so? | 18:35 |
| leifmadsen | mrunge: ^^ :) | 18:35 |
| leifmadsen | Matthias will know better than I unfortunately | 18:35 |
| leifmadsen | (unfortunate in that I don't know the answer or anyone else who probably knows) | 18:35 |
| leifmadsen | I mean, I would expect it's using Ceph for data storage, which gnocchi does via myql | 18:36 |
| leifmadsen | mysql* | 18:36 |
| fultonj | i thought gnocchi was no longer maintained and no longer a part of openstack | 18:38 |
| fultonj | https://opendev.org/openstack/gnocchi | 18:41 |
| leifmadsen | not true | 18:41 |
| leifmadsen | gnocchi is required in order to provide autoscaling | 18:41 |
| leifmadsen | deprecated != removed | 18:41 |
| leifmadsen | you're correct in that it is not hosted in openstack opendev system proper | 18:41 |
| fultonj | the CI is not having a good day and the latest issue I've been told about is that gnocchi can't connect to ceph | 18:43 |
| fultonj | https://411deec051d768e42c33-554a55c2926ce345d8a8f0805ecfe993.ssl.cf5.rackcdn.com/846159/4/check/tripleo-ci-centos-9-scenario001-standalone/a9a4889/logs/undercloud/var/log/extra/podman/containers/gnocchi_db_sync/stdout.log | 18:44 |
| fultonj | i'm checking on if it's _supposed_to_be_ connecting to ceph | 18:44 |
| leifmadsen | yea I read that, and I don't understand what that really means | 18:44 |
| leifmadsen | gnocchi would connect to mysql which has a volume via ceph... what does "connect to ceph" mean? | 18:44 |
| fultonj | ah | 18:44 |
| fultonj | gnocchi as a client of a ceph server | 18:45 |
| fultonj | gnocchi.common.ceph: Ceph storage backend use 'rados' python library | 18:45 |
| fultonj | i would think the mysql db was running directly on localdisk on a controller node | 18:46 |
| fultonj | SQLAlchemyIndexer: mysql+pymysql://***:***@192.168.24.3/gnocchi? | 18:46 |
| fultonj | ^ makes sense to both of us right? | 18:46 |
| leifmadsen | yea, it's connecting to mysql at that host (that does look like a controller address) | 18:47 |
| fultonj | right | 18:47 |
| leifmadsen | my gnocchi knowledge is quite limited though. I suspect we'll have to wait for mrunge to return tomorrow (he's in Germany) | 18:47 |
| fultonj | ack | 18:48 |
| fultonj | it mentions ceph in passing but it what really fails on is redis | 18:48 |
| leifmadsen | right | 18:48 |
| leifmadsen | it also seems to be talking about an upgrade, and almost looks like a failure in the schema update | 18:49 |
| fultonj | rlandy|rover: it's failing on a redis (not ceph) call and we probably need help from mrunge when he's back online | 18:50 |
| fultonj | thanks for the info leifmadsen | 18:53 |
| leifmadsen | yea sorry I can't be of more assistance | 18:54 |
| leifmadsen | I would also ping paramite who might know but he's not in this room (Martin Magr) | 18:54 |
| * fultonj remembers when gnocchi used to have its own ceph pool to connect to | 18:55 | |
| fultonj | but that's gone | 18:55 |
| fultonj | we no longer create a pool for it | 18:55 |
| leifmadsen | yea I mean, it's very possible it connected to ceph directly for whatever reason... the job could potentially need to be adjusted | 18:55 |
| leifmadsen | I just don't understand why it would do it that way, but yea | 18:56 |
| fultonj | it shouldn't be connecting to ceph anymore though | 18:56 |
| fultonj | it was never performant enough | 18:56 |
| fultonj | w/ ceph | 18:56 |
| * fultonj going to look a little more at this | 18:58 | |
| opendevreview | Merged openstack/validations-libs stable/1.6: [Stable-Only] Refactoring the stable/1.6 CI pipeline https://review.opendev.org/c/openstack/validations-libs/+/841713 | 18:59 |
| opendevreview | Merged openstack/validations-libs stable/1.6: Clarifying project branching model in CONTRIBUTING.rst https://review.opendev.org/c/openstack/validations-libs/+/843782 | 18:59 |
| rlandy|rover | fultonj: gnocchi related to ceph or gnocchi alone | 19:02 |
| fultonj | rlandy|rover: there were performance issues w/ gnocchi and ceph when performance team tested it hard | 19:03 |
| fultonj | rlandy|rover: so our fixing patch to only test master branch failed on step5 w/ gnocchi | 19:04 |
| fultonj | connection refsued from redis | 19:04 |
| fultonj | rlandy|rover: i added mrunge to the review and left a comment about the redis connection failure https://review.opendev.org/846159 | 19:13 |
| rlandy|rover | fultonj: considering all the failures plus ones like: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario001-standalone-master/105b6ec/job-output.txt | 19:19 |
| rlandy|rover | worth a revert? | 19:19 |
| fultonj | the same ansible tasks are now being run earlier | 19:22 |
| fultonj | in theory reverting shouldn't help | 19:22 |
| fultonj | we could revert and see if it does help | 19:23 |
| fultonj | espeically if it will take the pressure off | 19:23 |
| fultonj | we saw things were green however before we merged | 19:24 |
| fultonj | in 105b6ec/job-output.txt we see it using the undercloud, not the local mirror | 19:27 |
| fultonj | though the undercloud repos sin'tset up yet | 19:28 |
| fultonj | rlandy|rover: do you want to do a temporary revert so you can be sure the centos mirror issue can be blamed and hunted down and then we'll try reverting the revert? | 19:30 |
| rlandy|rover | fultonj: let me see what we can get through first | 19:31 |
| rlandy|rover | if the libvirt downgrade works we can start there | 19:31 |
| fultonj | rlandy|rover: do you want me to open bugs for gnochi and 105b6ec/job-output.txt ? | 19:31 |
| rlandy|rover | then the ipa change | 19:31 |
| rlandy|rover | fultonj: sure | 19:32 |
| fultonj | ack | 19:32 |
| rlandy|rover | we need to track that | 19:32 |
| fultonj | do we have a doc of the recent issues? | 19:32 |
| fultonj | hackmd or something with a list of bugs that jumped out in the last 48 hours affecitng ci? | 19:32 |
| fultonj | https://bugs.launchpad.net/tripleo/+bug/1978956 | 19:33 |
| fultonj | https://bugs.launchpad.net/tripleo/+bug/1978969 | 19:33 |
| rlandy|rover | https://hackmd.io/gw8f6Y3hQcatR_l2PhhY9g | 19:33 |
| fultonj | thx | 19:34 |
| fultonj | i'll add the other two | 19:34 |
| rlandy|rover | adding the ipa one | 19:35 |
| rlandy|rover | fultonj: all on hackmd with fix patches linked | 19:42 |
| fultonj | thx | 19:43 |
| fultonj | https://bugs.launchpad.net/tripleo/+bug/1978997 | 19:43 |
| fultonj | i'm adding ^ | 19:43 |
| fultonj | i can do a WA of ^ by not deploying gnochi in tht | 19:43 |
| fultonj | i'm currious how many of these failures occur >1 | 19:44 |
| rlandy|rover | k - I need to update the ipa patch to include the second case | 19:44 |
| rlandy|rover | then will depends on the libvirt one | 19:44 |
| rlandy|rover | then we can address ceph | 19:44 |
| fultonj | so we're going to recheck 846159 w/ the libvirt as a dep? | 19:46 |
| fultonj | that would be good as we could see if gnochi fails again | 19:46 |
| * fultonj now openeing a bug for 105b6ec/job-output.txt | 19:46 | |
| fultonj | https://bugs.launchpad.net/tripleo/+bug/1978998 | 20:00 |
| fultonj | ^ is assigned to me. I took steps to prevent this issue but maybe perioidic jobs have different behavior | 20:00 |
| opendevreview | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Revert "Temporarily freeze openssl" https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846125 | 20:07 |
| opendevreview | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Revert "Temporarily freeze openssl" https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846125 | 20:09 |
| rlandy|rover | dviroel: ^^ pls review | 20:10 |
| fultonj | https://bugs.launchpad.net/tripleo/+bug/1978998/comments/3 | 20:37 |
| fultonj | dviroel: ^ do you know why containers-prepare-parameters.yaml is different in periodic standalone001 vs non-periodic? | 20:38 |
| dviroel | fultonj: non-periodic jobs consume it from content-provider | 20:43 |
| fultonj | so it's a known thing that they differ | 20:44 |
| fultonj | can you point me at the code that configures periodic job containers-prepare-parameters.yaml ? | 20:46 |
| fultonj | dviroel: ^ | 20:46 |
| fultonj | i need to make an update | 20:46 |
| fultonj | to address LP 1978998 | 20:46 |
| dviroel | fultonj: not sure if what you want but - https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/standalone/tasks/containers.yml#L128 | 20:53 |
| dviroel | fultonj: the difference is that on non-periodic jobs, we enable consumer | 20:54 |
| dviroel | this is the place that we replace by content-provider ip | 20:55 |
| * dviroel looks at yout LP bug | 20:55 | |
| fultonj | dviroel: that' | 20:56 |
| fultonj | dviroel: that's fine, to pull them from content provider but we need to get rid of push_destination for the ceph contianer | 20:56 |
| fultonj | we can do an exclude to address that | 20:57 |
| fultonj | https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/deployment/container_image_prepare.html#layering-image-preparation-entries | 20:58 |
| fultonj | so i now understand why/how this is happening and how to fix | 20:59 |
| *** dviroel is now known as dviroel|biab | 21:07 | |
| fultonj | dviroel: in periodic jobs is standalone_container_ceph_updates|bool true? | 21:08 |
| opendevreview | Rafael Castillo proposed openstack/tripleo-quickstart master: DNM: Test building latest sdk in dependency https://review.opendev.org/c/openstack/tripleo-quickstart/+/845375 | 21:09 |
| fultonj | https://github.com/openstack/tripleo-quickstart-extras/commit/78b758851678a4df86248185dbd0fae0979b1494 | 21:12 |
| fultonj | ^ is all good, i just want to make a tweak so deployed-ceph can handle this | 21:13 |
| rlandy|rover | fultonj: ok - some hope for the libvirt downgrade - updating that patch | 21:22 |
| rlandy|rover | then will updated the ipa one on to depends- on | 21:23 |
| fultonj | rlandy|rover: the perioidic job failure is understood, i can make a patch | 21:23 |
| rlandy|rover | those should spring t-q-e ceph patch free | 21:23 |
| fultonj | is periodic failure as important? | 21:23 |
| rlandy|rover | sure - you can propose that | 21:23 |
| rlandy|rover | it will take some time to get this all through | 21:23 |
| fultonj | rlandy|rover: https://hackmd.io/gw8f6Y3hQcatR_l2PhhY9g?both | 21:24 |
| rlandy|rover | we may as well have the patches in line | 21:24 |
| fultonj | i updated it to have ceph and non-ceph | 21:24 |
| rlandy|rover | I see | 21:24 |
| rlandy|rover | thank you for that exaplanation | 21:24 |
| rlandy|rover | a lot going on | 21:24 |
| fultonj | np! | 21:24 |
| fultonj | the MaybeCeph issue came up when testing branch-overrides | 21:24 |
| fultonj | but we haven't seen the gnochi issue reproduced in other jobs AFAICT | 21:25 |
| * fultonj thinks branch-overrides will fix downstream faster and isn't sure other issues are releated | 21:27 | |
| fultonj | we're bleeding, branch-overrides makes us bleed less while we address other wounds nad branch-overrides is a simple patch | 21:27 |
| rlandy|rover | ack | 21:28 |
| rlandy|rover | the libvirt patch is the worst bleeder right now | 21:28 |
| rlandy|rover | then ipa | 21:28 |
| rlandy|rover | and ceph | 21:28 |
| fultonj | sort the bloody bugs | 21:29 |
| fultonj | ;) | 21:29 |
| fultonj | by how much blood they produce | 21:29 |
| fultonj | rlandy|rover: how critical is periodic? | 21:30 |
| fultonj | and how can i test a patch to periodic? | 21:30 |
| rlandy|rover | not crazy | 21:30 |
| rlandy|rover | we promoted w c9 today | 21:30 |
| rlandy|rover | and master yesterday | 21:30 |
| rlandy|rover | so go ahead | 21:30 |
| rlandy|rover | gate is the only burning thing | 21:31 |
| fultonj | bleeding and burning | 21:31 |
| * fultonj is reminded of a song | 21:31 | |
| rlandy|rover | :) | 21:32 |
| fultonj | dviroel: in periodic jobs is standalone_container_ceph_updates|bool true? | 21:40 |
| * fultonj thinks yes | 21:43 | |
| opendevreview | Ronelle Landy proposed openstack/tripleo-quickstart master: Temp downgrade of libvirt 8.4.x - failing tempest https://review.opendev.org/c/openstack/tripleo-quickstart/+/846194 | 22:02 |
| opendevreview | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Revert "Temporarily freeze openssl" https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846125 | 22:03 |
| rlandy|rover | dviroel|biab: ^^ pls vote when you are back ... | 22:03 |
| rlandy|rover | testing https://review.opendev.org/c/openstack/tripleo-quickstart/+/846194 with container builds in periodic | 22:04 |
| opendevreview | John Fulton proposed openstack/tripleo-quickstart-extras master: Override Ceph --container-namespace for periodic jobs https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846231 | 22:08 |
| fultonj | rlandy|rover: dviroel|biab ^ | 22:11 |
| fultonj | rlandy|rover: how can i test ^ in a periodic job? | 22:12 |
| rlandy|rover | fultonj: testproject | 22:14 |
| rlandy|rover | getting | 22:14 |
| rlandy|rover | fultonj: which job? | 22:15 |
| fultonj | https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario001-standalone-master/105b6ec/logs/undercloud/home/zuul/ansible.log.txt.gz | 22:16 |
| rlandy|rover | https://review.rdoproject.org/r/c/testproject/+/36256 | 22:17 |
| rlandy|rover | fultonj: ^^ | 22:17 |
| rlandy|rover | - periodic-tripleo-ci-centos-9-scenario001-standalone-master: | 22:17 |
| rlandy|rover | vars: | 22:17 |
| rlandy|rover | force_periodic: true | 22:17 |
| rlandy|rover | featureset_override: | 22:17 |
| rlandy|rover | dlrn_hash_tag: c9787f8a14d76665fbfafd3395275cdc | 22:17 |
| rlandy|rover | Depends-On: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846231 | 22:17 |
| fultonj | thanks | 22:18 |
| * fultonj heading out | 22:37 | |
| rlandy|rover | fultonj: thanks - have a good night | 22:52 |
| *** dviroel|biab is now known as dviroel | 23:07 | |
| *** dviroel is now known as dviroel|out | 23:16 | |
| *** rlandy|rover is now known as rlandy|out | 23:36 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!