| admin1 | controllers and hypervisors, basically all servers /etc/issue shows Ubuntu 22.04.3 LTS .. but when trying to do an upgrade from 26->27, I get "You are trying to build wheels for the distribution or architecture that you don't have build host for. Please, ensure that you have at least 1 host in repo_all group with the following distribution: ubuntu | 08:15 |
|---|---|---|
| admin1 | 20.04 x86_64 Otherwise, consider adding matching OS to the group or set `venv_wheel_build_enable: False`. Note, that you can run only in a serial manner when wheel build is disabled. Found venv_build_targets: {'ubuntu-22.04-x86_64': 'c1_repo_container-83f07d5e'}"" | 08:15 |
| admin1 | \o | 08:15 |
| admin1 | do i need to delete the utility container and have it create it again ? | 08:15 |
| noonedeadpunk | admin1: so, you did upgrade of OS on controllers, but didn't re-create containers afterwards? | 08:16 |
| admin1 | after upgrade, just ran the playbook in full to ensure that things still work .. and when they worked, moved for the upgrade | 08:17 |
| admin1 | 24->25->26 did not complain .. 26-27 caught this | 08:17 |
| noonedeadpunk | Yeah, so I assume all your containers are still running 20.04 | 08:17 |
| admin1 | so delete all except galera ? | 08:18 |
| noonedeadpunk | You at least need to re-create 1 repo container | 08:18 |
| noonedeadpunk | to have it on 22.04 | 08:18 |
| admin1 | oh .. got it .. i can delete the repo container and have it re-cerate it | 08:18 |
| noonedeadpunk | but eventually - I think you'd need to rebuild all of them before moving to the release that don't have 20.04 among supported versions | 08:18 |
| noonedeadpunk | (which is bobcat) | 08:19 |
| admin1 | well, i can delete all except galera .. | 08:19 |
| noonedeadpunk | You can delete them controller-by-controller actually | 08:19 |
| noonedeadpunk | like --limit infra01-host_containers | 08:20 |
| noonedeadpunk | Probably worth to mark all these services as DRAIN/MAINT in haproxy first though | 08:21 |
| noonedeadpunk | we have thing like that for this specific purpose: https://paste.openstack.org/show/bSS4ao3i9xDSkt42evVD/ | 08:22 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_keystone master: Cleanup upgrade to ssh_keypairs step https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/898801 | 08:43 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Cleanup upgrade to ssh_keypairs step https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/898802 | 08:44 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Fix logic of discovering hosts by service https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/898749 | 08:47 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Drop ssh_keypairs_install_authorized_keys reference https://review.opendev.org/c/openstack/openstack-ansible/+/898804 | 09:18 |
| noonedeadpunk | jrosser: hey! can you kindly help me to understand ssh_keypairs role better? I'm right now trying to add it to the functional test for plugins repo | 09:59 |
| jrosser | sure i can try :) | 10:00 |
| noonedeadpunk | So there's a check, when you try to access a container from localhost through SSH | 10:00 |
| noonedeadpunk | And I wonder what kinda you need to set to allow SSH from localhost to that container, but not vice versa | 10:00 |
| noonedeadpunk | So it's smth like that atm https://paste.openstack.org/show/bGHuAmV5wqmcmiQHar3x/ | 10:01 |
| noonedeadpunk | What I'm missing to make it work?:) | 10:01 |
| noonedeadpunk | or principal should be installed everywhere? | 10:02 |
| jrosser | hmm i just have a meeting for 30mins | 10:03 |
| noonedeadpunk | ++ | 10:03 |
| jrosser | but yes keystone role would be the kind of prototype for this | 10:03 |
| jrosser | principal is what decides if it is allowed to login | 10:03 |
| jrosser | on the localhost should need the public/private key/signed cert | 10:04 |
| jrosser | on the container only the CA, proper config in sshd.conf and principal which matches what the key was signed with | 10:04 |
| noonedeadpunk | Yeah, that seems to match.... | 10:05 |
| noonedeadpunk | maybe cert is missing... | 10:06 |
| noonedeadpunk | Really don't see what's off... | 10:19 |
| noonedeadpunk | Oh, well/.. | 10:20 |
| noonedeadpunk | Could it be that `TrustedUserCAKeys /etc/ssh/trusted_ca` but CA is actually /etc/ssh/trusted_ca.d/OpenStack-Ansible-SSH-Signing-Key | 10:21 |
| noonedeadpunk | yup, that's the reason | 10:23 |
| noonedeadpunk | But how things do work then? | 10:23 |
| noonedeadpunk | ah | 10:30 |
| noonedeadpunk | I had some failure that didn't trigger `Regenerate trusted_ca file` handler :) | 10:30 |
| noonedeadpunk | (I guess that was it) | 10:30 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Generate SSH certificates for delegation test https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/898809 | 11:06 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Simplfy addition of keystone users to roles https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/896017 | 11:06 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Add openstack_resources role skeleton https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/878794 | 11:06 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/2023.1: Define install_method default when hosts resolution depend on it https://review.opendev.org/c/openstack/openstack-ansible/+/898072 | 11:43 |
| opendevreview | Merged openstack/openstack-ansible-os_aodh stable/2023.1: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/892355 | 11:43 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts stable/2023.1: Add ability to define a config for journald https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/898769 | 11:50 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts stable/2023.1: Add ability to define a config for journald https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/898769 | 11:51 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/2023.1: Apply rate limit for journald in AIO builds https://review.opendev.org/c/openstack/openstack-ansible/+/898770 | 11:52 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_gnocchi stable/2023.1: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/892356 | 11:54 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_ceilometer master: Fix example playbook linters https://review.opendev.org/c/openstack/openstack-ansible-os_ceilometer/+/898814 | 11:57 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_ceilometer master: Enable Ceilometer resource cache https://review.opendev.org/c/openstack/openstack-ansible-os_ceilometer/+/888032 | 11:57 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_ceilometer master: Add quorum support for service https://review.opendev.org/c/openstack/openstack-ansible-os_ceilometer/+/895696 | 11:58 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cloudkitty master: Add quorum queues support for service https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/898816 | 12:15 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_designate master: Add quorum queues support for service https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/898818 | 12:31 |
| opendevreview | Merged openstack/openstack-ansible stable/2023.1: Run nova db post setup from nova playbook https://review.opendev.org/c/openstack/openstack-ansible/+/898494 | 12:37 |
| *** Adri2000_ is now known as Adri2000 | 13:19 | |
| spatel | Folks.. need help for ceph | 13:25 |
| spatel | what is the command to backfill setting ? | 13:25 |
| spatel | I want to add OSD in ceph but want to keep data trashing low... | 13:26 |
| spatel | last time it fill my pipe :( | 13:26 |
| jamesdenton | noonedeadpunk have you seen instances where the conditional check is being ignored? https://paste.opendev.org/show/bVSNa9kBIl5LBKf0nkXV/ - in this case, user.role is a string and the role assignment was made accordingly, but then it's failing when trying to perform the 'add keystone roles' task because it's not a list. But the conditional is the run that task "when not string" | 14:23 |
| jrosser | i have a patch for this i think | 14:24 |
| noonedeadpunk | yeah | 14:24 |
| noonedeadpunk | https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/896017 | 14:24 |
| jamesdenton | cool cool | 14:24 |
| jamesdenton | that's one way of doing it :D | 14:25 |
| jrosser | grmbl grmbl more rocky repo shennanigans there | 14:26 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Generate SSH certificates for delegation test https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/898809 | 14:26 |
| jrosser | in the end we can remove that logic | 14:26 |
| jrosser | but i think you can't fix all the role stuff to be lists without needing to handle both | 14:27 |
| jrosser | as it's throughout all the roles everywhere | 14:27 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Generate SSH certificates for delegation test https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/898809 | 15:28 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Generate SSH certificates for delegation test https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/898809 | 15:30 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Generate SSH certificates for delegation test https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/898809 | 16:43 |
| noonedeadpunk | I really dunno why this is failing.... asked for a hold | 16:45 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Generate SSH certificates for delegation test https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/898809 | 16:49 |
| NeilHanlon | jrosser: hm. at least that one seems like the jobs just got a mirror that was offline, which could be any of them in the community. I will start the conversation again with infra about getting a local rocky mirror. i think that will 'solve' it. i do apologize for the noise and crap :( | 17:26 |
| noonedeadpunk | Yeah, problematic mirrors are kind of "fine" | 17:41 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Generate SSH certificates for delegation test https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/898809 | 17:43 |
| noonedeadpunk | that happens everywhere actually | 17:44 |
| noonedeadpunk | we see it less with other distros only because of using local infra repos. | 17:44 |
| noonedeadpunk | But even then couple of times per year they might get broken, especially EPEL ones, because there's hardly a reliable mirror that can be used for infra... | 17:45 |
| noonedeadpunk | From what I got from mirror maintainers, is that during some new version being released it's too much changes that mirrors can hardly deal with | 17:46 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Generate SSH certificates for delegation test https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/898809 | 18:06 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Generate SSH certificates for delegation test https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/898809 | 18:30 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Generate SSH certificates for delegation test https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/898809 | 18:47 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Add openstack_resources role skeleton https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/878794 | 18:47 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Simplfy addition of keystone users to roles https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/896017 | 18:48 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!