| rdogerrit | rdo-trunk rdo-trunk proposed deps/python-prometheus_client c9s-caracal-rdo: Update to 0.17.1 https://review.rdoproject.org/r/c/deps/python-prometheus_client/+/51165 | 00:52 |
|---|---|---|
| rdogerrit | rdo-trunk rdo-trunk proposed deps/python-semantic_version c9s-caracal-rdo: Update to 2.10.0 https://review.rdoproject.org/r/c/deps/python-semantic_version/+/51166 | 00:52 |
| rdogerrit | rdo-trunk rdo-trunk proposed deps/python-suds c9s-caracal-rdo: Update to 1.1.2 https://review.rdoproject.org/r/c/deps/python-suds/+/51167 | 00:53 |
| tkajinam | hmm. there seems to be something wrong with sudo in c9s? I see most of the commands executed with sudo takes 2 minutes which is causing very frequent timeout in integration jobs. | 04:50 |
| rdogerrit | Merged rdo-infra/ansible-role-weirdo-common master: Enable EPEL repository for Ceph https://review.rdoproject.org/r/c/rdo-infra/ansible-role-weirdo-common/+/51131 | 07:21 |
| rdogerrit | Alfredo Moralejo proposed rdoinfo master: Add TCIB to antelope cloudsig repo https://review.rdoproject.org/r/c/rdoinfo/+/51168 | 07:22 |
| rdogerrit | Merged rdoinfo master: Update maintainers for Heat https://review.rdoproject.org/r/c/rdoinfo/+/51140 | 07:25 |
| *** gthiemon1e is now known as gthiemonge | 07:41 | |
| tkajinam | amoralej, jcapitao[m] fyi... I've seen p-o-i jobs and packstack jobs consistently time out over the weekend. I'm struggling to debug the issue because logs are not captured, but console output indicates something is causing 2 minutes delay in every sudo command which is strange. | 07:49 |
| amoralej | yes, i was working on it on friday | 07:49 |
| amoralej | it's something related to pam - dbus integration that starts failing after enabling libvirt services | 07:50 |
| amoralej | i suspect from some update in centos | 07:50 |
| tkajinam | ah ok | 07:50 |
| tkajinam | that explains why even litmus job in puppet-nova fails while it does not fail in the other modules | 07:51 |
| tkajinam | (only litmus job in puppet-nova enables libvirt afair | 07:51 |
| amoralej | i couldn't reproduce it by manually enabling services | 07:51 |
| amoralej | i'll keep working on it today, i'll probably be able to give you access if you want | 07:52 |
| tkajinam | amoralej, thanks. If you can hold a node and give me access to it then I'll check the node tonight. | 07:55 |
| amoralej | ok, sure | 07:56 |
| rdogerrit | Joel Capitao proposed rdo-infra/releng master: Add edit_source_gpg_sign https://review.rdoproject.org/r/c/rdo-infra/releng/+/47530 | 08:10 |
| rdogerrit | Joel Capitao proposed rdo-infra/releng master: Edit pin_puppet_modules script https://review.rdoproject.org/r/c/rdo-infra/releng/+/47672 | 08:11 |
| rdogerrit | Joel Capitao proposed rdo-infra/releng master: Edit sources GPG signature before sending new version https://review.rdoproject.org/r/c/rdo-infra/releng/+/50131 | 08:11 |
| rdogerrit | Joel Capitao proposed openstack/neutron-distgit antelope-rdo: openstack-neutron: failed to build be09dd089a https://review.rdoproject.org/r/c/openstack/neutron-distgit/+/51164 | 08:13 |
| rdogerrit | Joel Capitao proposed openstack/neutron-distgit antelope-rdo: Lowercase the GPG signature https://review.rdoproject.org/r/c/openstack/neutron-distgit/+/51164 | 08:15 |
| rdogerrit | Merged deps/python-prometheus_client c9s-caracal-rdo: Update to 0.17.1 https://review.rdoproject.org/r/c/deps/python-prometheus_client/+/51165 | 08:15 |
| rdogerrit | Merged rdoinfo master: Add TCIB to antelope cloudsig repo https://review.rdoproject.org/r/c/rdoinfo/+/51168 | 08:41 |
| rdogerrit | Joel Capitao proposed rdoinfo master: Promote CBS tags update for antelope-9s-release https://review.rdoproject.org/r/c/rdoinfo/+/51135 | 08:43 |
| rdogerrit | Merged rdoinfo master: Promote CBS tags update for bobcat-9s-release https://review.rdoproject.org/r/c/rdoinfo/+/51118 | 08:48 |
| rdogerrit | Merged rdoinfo master: Promote CBS tags update for antelope-9s-release https://review.rdoproject.org/r/c/rdoinfo/+/51135 | 10:06 |
| rdogerrit | Joan Francesc Gilabert proposed rdo-jobs master: Point to pod logs in zuul artifacts for adoption https://review.rdoproject.org/r/c/rdo-jobs/+/51169 | 11:31 |
| rdogerrit | Merged rdo-infra/releng master: Add edit_source_gpg_sign https://review.rdoproject.org/r/c/rdo-infra/releng/+/47530 | 11:51 |
| rdogerrit | Merged rdo-infra/releng master: Edit sources GPG signature before sending new version https://review.rdoproject.org/r/c/rdo-infra/releng/+/50131 | 12:49 |
| rdogerrit | Merged rdo-jobs master: Remove unused data-plane adoption periodic job https://review.rdoproject.org/r/c/rdo-jobs/+/51152 | 13:05 |
| rdogerrit | Joan Francesc Gilabert proposed rdo-jobs master: Point to pod logs in zuul artifacts for adoption https://review.rdoproject.org/r/c/rdo-jobs/+/51169 | 13:37 |
| *** blarnath is now known as d34dh0r53 | 15:00 | |
| rdogerrit | Joan Francesc Gilabert proposed rdo-jobs master: Remove and clean up after unused adoption job https://review.rdoproject.org/r/c/rdo-jobs/+/51170 | 15:04 |
| amoralej | tkajinam, we just found some more info. The problem is caused by latest version of systemd. When dbus is restarted systemd-logind does not reconnect unless you restart systemd-logind. | 15:05 |
| amoralej | I'll report it as a bug to centos stream 9 | 15:05 |
| amoralej | but i was wondering if we may avoid restarting dbus | 15:06 |
| amoralej | i think we shouldn't need to restart, and according to some systemd specialist, restarting dbus is usually not a good practice and usually is not needed | 15:06 |
| amoralej | karolinku[m] ^ | 15:07 |
| karolinku[m] | what is the reason of restarting it currently? | 15:13 |
| tkajinam | amoralej, The restart is not very intentional. let me try fixing it | 15:14 |
| amoralej | karolinku[m], ^ dunno | 15:14 |
| tkajinam | https://review.opendev.org/c/openstack/puppet-nova/+/905000 | 15:17 |
| tkajinam | let's see if this helps | 15:18 |
| rdogerrit | Joan Francesc Gilabert proposed rdo-jobs master: Remove and clean up after unused adoption job https://review.rdoproject.org/r/c/rdo-jobs/+/51170 | 15:19 |
| amoralej | tkajinam, karolinku[m] reported in https://issues.redhat.com/browse/RHEL-21004 | 15:20 |
| tkajinam | current implementation restarts dbus when libvirt packages are updated or libvirt config files are modified. This is sideeffect of using the libvirt-service to align the resource to the expected phase... | 15:20 |
| tkajinam | amoralej, thanks ! | 15:21 |
| tkajinam | ah, now c9s bugs should be reported in jira. ok | 15:22 |
| amoralej | yes | 15:22 |
| amoralej | but i understood that that would only restart it if it was not running | 15:22 |
| amoralej | and in this case it was | 15:23 |
| amoralej | actually, i guess dbus is always running in cs9, i doubt it can run without it | 15:23 |
| tkajinam | if that's true then we can even remove that service resource | 15:25 |
| tkajinam | ok I noticed my explanation above was not very accurate > current implementation restarts ... | 15:26 |
| tkajinam | the problem is caused by https://github.com/openstack/puppet-nova/blob/master/manifests/compute/libvirt/services.pp#L132 and | 15:26 |
| tkajinam | https://github.com/openstack/puppet-nova/blob/master/manifests/compute/libvirt/services.pp#L158 | 15:26 |
| tkajinam | the problem is caused by these two... after libvirt package is installed initially, dbus is restarted | 15:26 |
| tkajinam | because of that notification at L132 | 15:26 |
| tkajinam | we have only limited and intentional notifications for libvirt resources but that is the only one relation which can notify the service resource for dbus | 15:28 |
| amoralej | i think we can safely remove it tbh | 15:28 |
| tkajinam | yeah | 15:29 |
| tkajinam | if the proposed change does not work then I'll try removing it. We can probably remove it from master while we tweak only notifications (as proposed now) in stable branches | 15:30 |
| amoralej | sounds good | 15:30 |
| rdogerrit | Dariusz Smigiel proposed rdo-infra/ci-config master: Replaced track_integration_promotion with simpler function https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/51163 | 16:28 |
| amoralej | tkajinam, the litmus job passed | 16:31 |
| amoralej | so i guess it will be fine | 16:31 |
| tkajinam | yeah I was checking the same :-) | 16:32 |
| rdogerrit | Joan Francesc Gilabert proposed rdo-jobs master: Remove and clean up after unused adoption job https://review.rdoproject.org/r/c/rdo-jobs/+/51170 | 16:36 |
| rdogerrit | Joan Francesc Gilabert proposed rdo-jobs master: Remove and clean up after unused adoption job https://review.rdoproject.org/r/c/rdo-jobs/+/51170 | 16:44 |
| rdogerrit | Dariusz Smigiel proposed rdo-infra/ci-config master: Replaced track_integration_promotion with simpler function https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/51163 | 16:47 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!