| opendevreview | Merged openstack/openstack-ansible stable/2023.2: Enable RabbitMQ feature flags pre-upgrade https://review.opendev.org/c/openstack/openstack-ansible/+/920998 | 00:18 |
|---|---|---|
| opendevreview | Merged openstack/openstack-ansible master: Bump SHAs for 2024.1 (Caracal) RC1 https://review.opendev.org/c/openstack/openstack-ansible/+/921271 | 00:52 |
| noonedeadpunk | Some reviews for https://review.opendev.org/c/openstack/openstack-ansible/+/918979 would be appreciated | 07:34 |
| noonedeadpunk | until it passes CI and branches not dropped.... | 07:34 |
| hamidlotfi_ | Hi there, | 07:35 |
| hamidlotfi_ | An instance whose volume is on CEPH, after abnormally restarting the Compute server where that instance is located, that instance does not come up and displays the following error: | 07:35 |
| hamidlotfi_ | https://www.irccloud.com/pastebin/lQoVeUo0/ | 07:35 |
| hamidlotfi_ | The unsaid point is that I put the disk_cachemodes parameter in all three mentioned cases, but the problem persists. | 07:35 |
| hamidlotfi_ | How can I solve this problem? | 07:35 |
| noonedeadpunk | hamidlotfi_: and you tried disk_cachemodes = network=directsync ? | 07:39 |
| noonedeadpunk | (or network=none) | 07:39 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_trove master: DNM https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/915052 | 07:44 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_keystone master: Set X-Forwarded-Proto header in apache conditionally https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/921312 | 07:46 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_keystone master: Set X-Forwarded-Proto header in apache conditionally https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/921312 | 07:52 |
| jrosser | good morning | 07:54 |
| noonedeadpunk | o/ | 07:57 |
| noonedeadpunk | I've proposed branching of integrated repo: https://review.opendev.org/c/openstack/releases/+/919484 | 07:58 |
| noonedeadpunk | and I think there was at least 1 backport to 2024.1 for cloudkitty | 07:59 |
| hamidlotfi_ | noonedeadpunk: I try these "file=directsync,block=none,network=writeback" | 08:00 |
| hamidlotfi_ | but I didn't check the one you mentioned, | 08:00 |
| hamidlotfi_ | let me check it. | 08:00 |
| noonedeadpunk | actually 2: https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/921284 and https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/921285 | 08:00 |
| noonedeadpunk | hamidlotfi_: um, so writeback will crash FS :) | 08:00 |
| jrosser | andrew is away for today/tomorrow and all of next week so we are going to be struggling for reviews a bit | 08:00 |
| noonedeadpunk | and only `network` is valid for Ceph backend | 08:00 |
| noonedeadpunk | Damian is also having vacation :/ | 08:00 |
| noonedeadpunk | well, will summon folks then | 08:01 |
| hamidlotfi_ | noonedeadpunk: Oh, this is an important point. | 08:01 |
| noonedeadpunk | hamidlotfi_: there's quite good explanation what each mode means: https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.disk_cachemodes | 08:01 |
| noonedeadpunk | and what you defined is just default value | 08:02 |
| noonedeadpunk | so yeah, basically you did not change any behaviour :) | 08:02 |
| hamidlotfi_ | noonedeadpunk: Yes ;) | 08:06 |
| noonedeadpunk | hamidlotfi_: but frankly, we're pretty much fine with writethrough for Ceph | 08:32 |
| noonedeadpunk | which kinda balances risk and performance | 08:33 |
| jrosser | it might be better trying out the ceph caching stuff too | 08:33 |
| noonedeadpunk | yeah, true | 08:33 |
| jrosser | as you get to optimise if you care about caching the underlying image, or writes | 08:33 |
| noonedeadpunk | though you'd need extra drives on computes afaik | 08:34 |
| jrosser | true | 08:34 |
| jrosser | and also tbh ceph was never about perforamance to a single vm | 08:34 |
| opendevreview | OpenStack Release Bot proposed openstack/openstack-ansible stable/2024.1: Update .gitreview for stable/2024.1 https://review.opendev.org/c/openstack/openstack-ansible/+/921357 | 08:46 |
| opendevreview | OpenStack Release Bot proposed openstack/openstack-ansible stable/2024.1: Update TOX_CONSTRAINTS_FILE for stable/2024.1 https://review.opendev.org/c/openstack/openstack-ansible/+/921358 | 08:46 |
| opendevreview | OpenStack Release Bot proposed openstack/openstack-ansible master: Update master for stable/2024.1 https://review.opendev.org/c/openstack/openstack-ansible/+/921359 | 08:46 |
| noonedeadpunk | also, this would be nice fix kinda: https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/921314 | 08:55 |
| noonedeadpunk | mgariepy: NeilHanlon if/when around can you check on it? ^ | 08:55 |
| noonedeadpunk | and also these backports to 2024.1: https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/921285 and https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/921284 | 09:03 |
| noonedeadpunk | and also, as I've scrwwed deploy guide - we need to land and backport https://review.opendev.org/c/openstack/openstack-ansible/+/921237 | 09:05 |
| noonedeadpunk | and https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/921314?tab=change-view-tab-header-zuul-results-summary fixes distro installation of horizon for EL as it;'s broken now | 09:09 |
| jrosser | i was looking at that and its a bit 8-0 | 09:10 |
| noonedeadpunk | yeah.... | 09:15 |
| noonedeadpunk | it kinda is. | 09:15 |
| noonedeadpunk | as rdo started (at some point?) to place static files and openstack_dashboard not inside site-packages (as that python packages after all), but in /usr/share... | 09:16 |
| noonedeadpunk | the only question I had if we wanna have condition or always add libdir to manage.py script | 09:25 |
| noonedeadpunk | this kinda doesn't hurt overall | 09:26 |
| noonedeadpunk | ah, and when horizon_lib_dir is not defined - it will get detected on it's own by python path for lib files | 09:29 |
| opendevreview | Merged openstack/openstack-ansible stable/2024.1: Update .gitreview for stable/2024.1 https://review.opendev.org/c/openstack/openstack-ansible/+/921357 | 09:40 |
| opendevreview | Merged openstack/openstack-ansible stable/2024.1: Update TOX_CONSTRAINTS_FILE for stable/2024.1 https://review.opendev.org/c/openstack/openstack-ansible/+/921358 | 09:42 |
| opendevreview | Merged openstack/openstack-ansible master: Update master for stable/2024.1 https://review.opendev.org/c/openstack/openstack-ansible/+/921359 | 09:54 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Start 2024.2 (Dalmatian) development https://review.opendev.org/c/openstack/openstack-ansible/+/921367 | 09:57 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Start 2024.2 (Dalmatian) development https://review.opendev.org/c/openstack/openstack-ansible/+/921367 | 10:04 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Switch services SHAs to track master https://review.opendev.org/c/openstack/openstack-ansible/+/921368 | 10:06 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Drop support for Debian 11 (bullseye) https://review.opendev.org/c/openstack/openstack-ansible/+/921370 | 10:30 |
| jrosser | do we need to go back through the stable/unmaintained branches and strip out centos 8-stream? | 10:38 |
| opendevreview | Merged openstack/openstack-ansible-os_cloudkitty stable/2023.2: Define region and interface for collector/fetcher https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/921054 | 11:00 |
| noonedeadpunk | we do... | 11:03 |
| opendevreview | Merged openstack/openstack-ansible-openstack_hosts stable/2023.2: Do not mark all directories as safe for git clone by default https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/921323 | 11:14 |
| opendevreview | Merged openstack/openstack-ansible-openstack_hosts stable/2023.1: Do not mark all directories as safe for git clone by default https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/921324 | 11:24 |
| hamidlotfi_ | noonedeadpunk: I checked all of those values for the disk_cachemodes parameter, | 11:31 |
| hamidlotfi_ | network=directsync | 11:31 |
| hamidlotfi_ | network=none | 11:31 |
| hamidlotfi_ | network=writethrough | 11:31 |
| hamidlotfi_ | but in all of the values, the result is the same and I had that error. | 11:31 |
| frickler | hamidlotfi_: none of these can repair your broken fs, those option only help to repeat the issue from happening again on the next host crash | 11:34 |
| noonedeadpunk | yes, absolutely ^ | 11:37 |
| hamidlotfi_ | Yes, I know, I have nothing to do with the previous file system in that instance, deleted it, the test I am doing is on a newly created instance and I expect this will not happen to the new instance, but this will happen again. I cold boot my compute host to simulate this event. | 11:40 |
| hamidlotfi_ | After changing the value of this parameter every time, I create an instance and cold reboot the compute server. | 11:42 |
| hamidlotfi_ | I hope I have explained well. | 11:42 |
| noonedeadpunk | aha, ok, then it's quite surprising frankly speaking... | 11:45 |
| hamidlotfi_ | unfortunately | 11:46 |
| hamidlotfi_ | Do you have any other recommendations? Maybe another parameter? | 11:58 |
| noonedeadpunk | frankly - not really | 12:13 |
| hamidlotfi_ | Ok, Thanks | 12:13 |
| mgariepy | good momrning ! went through the reviews you posted 3 hours ago noonedeadpunk. | 12:37 |
| noonedeadpunk | awesome! | 12:37 |
| noonedeadpunk | thanks | 12:37 |
| noonedeadpunk | so, upgrade jobs looks quite borked for 2024.1 | 12:52 |
| noonedeadpunk | ie https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/921284 | 12:52 |
| noonedeadpunk | fails on missing tasks/rabbitmq_feature_flags.yml | 12:53 |
| noonedeadpunk | wonder if we need to have sha bump on stable branches for that... | 12:54 |
| noonedeadpunk | as highly likely we do | 12:54 |
| mgariepy | 2023.1 works but not 2023.2 ? | 13:00 |
| mgariepy | upgrade from ^^ | 13:01 |
| mgariepy | ha. the patch for 2023.1 is not merged. | 13:04 |
| noonedeadpunk | yeah, I guess we need to bump shas to get the rabbitmq role with the task | 13:07 |
| jrosser | i guess we should be careful about merging feature flag things further back | 13:13 |
| mgariepy | the bump should have been added to :https://review.opendev.org/c/openstack/openstack-ansible/+/920998 | 13:13 |
| jrosser | else the upgrade jobs will also be broken further back and prevent merging a sha bump | 13:14 |
| jrosser | yeah | 13:14 |
| mgariepy | since the tests are not quite complete.. we missed it. | 13:15 |
| mgariepy | upgrade job are always tested on tip of the previous release i guess ? | 13:16 |
| jrosser | well i guess that is chicken/egg because you cant test the bumped sha until you merge the bump? | 13:17 |
| mgariepy | indeed | 13:18 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/2023.2: Update RabbitMQ SHA https://review.opendev.org/c/openstack/openstack-ansible/+/921379 | 13:26 |
| noonedeadpunk | hopefully this will fix it ^ | 13:26 |
| mgariepy | was the depends-on root cause of the upgrade test pass on merge? in the 2023.2 branch ? | 13:47 |
| noonedeadpunk | well, CI always kinda gets zuul-prepared repos for current branch | 13:49 |
| noonedeadpunk | but for N-1 it does bootstrap process as user | 13:49 |
| noonedeadpunk | and then on N uses what zuul has | 13:49 |
| noonedeadpunk | and zuul has latest for current branch | 13:49 |
| mgariepy | okie | 13:50 |
| opendevreview | Merged openstack/openstack-ansible-os_horizon master: Fix horizon distro installation for EL https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/921314 | 14:58 |
| opendevreview | Merged openstack/openstack-ansible-rabbitmq_server stable/2023.1: Enable feature flags post-upgrade https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/920999 | 15:28 |
| deflated | Hi all, trying to deploy 29.0.0rc1 on debian bookworm but getting an apt key depreciated errorr, found the bug report but its still wip, is there a way round this or is it not possible to deploy 29 on bookworm? TiA | 15:49 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible stable/2023.1: Update RabbitMQ SHA https://review.opendev.org/c/openstack/openstack-ansible/+/921396 | 15:51 |
| mgariepy | noonedeadpunk, copied your commit message :) hehe | 15:51 |
| noonedeadpunk | deflated: hey, it's weird you;re getting error. can you please paste it? | 15:53 |
| noonedeadpunk | mgariepy: I think it's better to update https://review.opendev.org/c/openstack/openstack-ansible/+/921000 ? | 15:54 |
| deflated | sure, where should i paste it? | 15:56 |
| deflated | Nevermind, got it | 15:56 |
| noonedeadpunk | ie https://paste.openstack.org/ | 15:56 |
| mgariepy | hmm sure. | 15:57 |
| mgariepy | sorry for that. | 15:57 |
| deflated | the catchpa seems broken on there | 15:58 |
| deflated | so its here instead: https://pastebin.com/6kmZjhN5 | 15:58 |
| deflated | that goes on for miles and happens on every container it seems | 15:59 |
| noonedeadpunk | deflated: so, apt key depreciated is warning from what I see | 15:59 |
| noonedeadpunk | but it's followed by some different error I've never seen before | 15:59 |
| noonedeadpunk | "E: Repository 'http://osbpo.debian.net/osbpo bullseye-zed-backports-nochange InRelease' changed its 'Origin' value from 'Infomaniak' to 'osbpo'" | 16:00 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible stable/2023.1: Enable RabbitMQ feature flags pre-upgrade https://review.opendev.org/c/openstack/openstack-ansible/+/921000 | 16:00 |
| noonedeadpunk | deflated: also I kinda wonder how you ended up with both bullseye and bookworm repos | 16:01 |
| deflated | it was an upgrade from it but i have checked all repos and removed everything related to it so i'll quickly chgeck my configs | 16:02 |
| noonedeadpunk | um, maybe you left ansible facts? | 16:03 |
| deflated | in /etc/opensdtack_deploy? | 16:03 |
| noonedeadpunk | yeah | 16:04 |
| jrosser | (before you delete them just take a look at one of the files to properly understand this) | 16:04 |
| noonedeadpunk | so, we do test bookworm in CI, and that's recent results for LXC test: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_98b/921367/2/check/openstack-ansible-deploy-aio_lxc-debian-bookworm/98bfb20/logs/etc/ | 16:04 |
| noonedeadpunk | you can see configured repos there and compare with what you have | 16:04 |
| noonedeadpunk | but the one that's role not happy with is https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_98b/921367/2/check/openstack-ansible-deploy-aio_lxc-debian-bookworm/98bfb20/logs/etc/openstack/aio1-horizon-container-3e2f35c5/apt/sources.list.d/osbpo.list.txt | 16:05 |
| noonedeadpunk | also, keep in mind that some "base" repos are copied to containers from host IIRC | 16:05 |
| jrosser | we basically don't support having containers that are not the same as the host OS | 16:06 |
| noonedeadpunk | well... I guess, that facts are container specific? | 16:09 |
| noonedeadpunk | so if you've spawned containers on bullseye, then upgraded underlying OS, ansible will correctly manage existing bullseye on top of bookworm | 16:10 |
| noonedeadpunk | but yes, in case you re-create containers, there's no OS choice - it will be same as host OS | 16:10 |
| noonedeadpunk | smth I wanted to work to improve at some point... | 16:10 |
| noonedeadpunk | another vote on this one would be nice: https://review.opendev.org/c/openstack/openstack-ansible/+/921379 | 16:12 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ops master: Update ansible-collection-kubernetes version pin https://review.opendev.org/c/openstack/openstack-ansible-ops/+/921399 | 16:45 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-ops master: Revert "Ensure that python3-cryptography is present in k8s control plane hosts" https://review.opendev.org/c/openstack/openstack-ansible-ops/+/921400 | 16:50 |
| deflateder | ok so i've removed all containers as well as files from /etc/openstack_deploy (apart from what is needed anyway), checked apt/sources for anything relating to bullseye (there's nothing), deleted unused apt keys, ran cleanup_venvs from ansible-ops just in case its held there and reran, same issue, did the same again, rebooted, updated, same issue. | 18:11 |
| deflateder | Just to be clear i deleted /openstack /opt/*ansible* and a few others, not sure why at this point, nothing on my system points towards bullseye | 18:14 |
| mgariepy | deflateder, anything in /etc/openstack_deploy/ansible_facts ? | 18:20 |
| deflateder | not the second time, i deleted the folder and everything i could find relating to openstack and ansible | 18:20 |
| mgariepy | after the run what are the fact for nova container? | 18:21 |
| mgariepy | like: ansible_distribution_release | 18:22 |
| deflateder | give me 5 and i'll let you know | 18:22 |
| deflateder | well. 15 mor elike | 18:23 |
| mgariepy | no worries | 18:24 |
| deflateder | "ansible_distribution_release": "bookworm" _version "12.4" | 18:44 |
| mgariepy | still have the zed repo in the container? | 18:45 |
| deflateder | yes | 18:46 |
| deflateder | which is just weird at this point | 18:47 |
| mgariepy | when was the repo added ? | 18:47 |
| mgariepy | might come from the lxc cached imagE? | 18:47 |
| deflateder | whenever i tried zed so howver many months/years ago that was, i have successfully ran 2023 at one point tho, the cached image you mention is not something i would of removed, where wo7uld i find it? | 18:48 |
| mgariepy | if you run stat in the apt directory when does the files were last modified? | 18:51 |
| deflateder | Today, osbo.list points to bookworm-caracal-backports main | 18:54 |
| deflateder | only other list i have is ceph which points to reef | 18:54 |
| mgariepy | huh. wasn't the issue some weird repos in your containers ? | 19:16 |
| deflateder | yes, but you didnt specify if the check you asked me to do was in the container or on the host, i checked the host | 19:17 |
| deflateder | can check the container if thats what you meant | 19:17 |
| mgariepy | yes please. | 19:19 |
| deflateder | ok, i'll recreate them, just wiped everything openstack/ansible related that i could find | 19:20 |
| jrosser | wierd ara behaviour in my failing capi jobs | 19:22 |
| deflateder | used nova container, dates are all today apart from modify which states 2023/12/13 | 19:38 |
| mgariepy | the novacontainer is bookworm as well ? | 19:52 |
| mgariepy | what playbook have you ran ? | 19:53 |
| deflateder | git clone, then bootstrap, then setup_hosts.yml | 19:56 |
| mgariepy | flush the container and rerun with `-e lxc_image_cache_refresh=true ` | 20:01 |
| deflateder | not sure what you mean by flush, i'm assuming purge/delete? | 20:02 |
| mgariepy | or simply destroy the containers. | 20:03 |
| mgariepy | there is a playbook for this. | 20:03 |
| deflateder | yeah i am running containers_destroy | 20:04 |
| jrosser | deflateder: can you restate what is happening - i cant quite follow what actually is wrong | 20:06 |
| deflateder | trying to deploy 29.0.0rc1 on debian bookworm but getting an apt key depreciated error, paste is here: https://pastebin.com/6kmZjhN5 Seems a bullseye repo is getting added toi my containters during setup_hosts | 20:10 |
| deflateder | Running setup_hosts with -e lxc_image_cache_refresh=true now | 20:11 |
| deflateder | quick question while i'm waiting for ity to complete/fail, does ceph-rgw-install.yml setup the nedpoint for a pre existing ceph cluster? Just checking before i contemplate running it to connect an existing ceph installation | 20:33 |
| deflateder | yes! it just finished, thanks for the solution! Adding that to my notes so i know for next time | 20:34 |
| jrosser | deflateder: well it means we have a bug somewhere | 20:34 |
| jrosser | the container image should be automatically rebuilt if the OS is upgraded | 20:35 |
| jrosser | I exp | 20:35 |
| jrosser | I expect most people redeploy/PXEboot rather than upgrade in place | 20:36 |
| deflateder | possibly, i've never had the need to do so with debian, its always worked flawlessly even upgrading, i had the same problem on 2023.2 as well, dont recall it occuring in 2023.1 | 20:37 |
| jrosser | ok no worries - we have no test coverage for this so the most ideal case would be submitting a patch to fix it | 20:39 |
| deflateder | i'll see about how i do that shortly | 20:45 |
| jrosser | there will be some condition in the lxc_hosts role that does not correctly see that the base image in /var/lib/machines is out of date | 20:49 |
| mgariepy | weird indeed. | 21:36 |
| mgariepy | is ntp working correcly on your server ? | 21:36 |
| jrosser | the image in /var/lib/machines should have the OS release name in it | 21:50 |
| jrosser | and that getting updated should trigger the rootfs to be compressed and pushed into lxc | 21:51 |
| deflateder | it is, after that command setup_hosts ran ok, now running setup_openstack | 21:51 |
| deflateder | New error, An exception occurred during task execution. To see the full traceback, use -vvv. The error was: AttributeError: 'int' object has no attribute 'items' | 22:11 |
| deflateder | fatal: [ost01-cinder-volumes-container-25ad06a8]: FAILED! => {"msg": "Unexpected failure during module execution: 'int' object has no attribute 'items'", "stdout": ""} | 22:11 |
| deflateder | Full trace here: https://pastebin.com/nkQRnDAy | 22:11 |
| opendevreview | Merged openstack/openstack-ansible stable/2023.2: Update RabbitMQ SHA https://review.opendev.org/c/openstack/openstack-ansible/+/921379 | 22:52 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!