| mgariepy | wait.. what ? https://access.redhat.com/security/cve/cve-2023-20593 | 00:07 |
|---|---|---|
| noonedeadpunk | wonder why it is "moderate" if you can "access sensitive information" | 06:14 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Add openstack_resources role skeleton https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/878794 | 06:29 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Add openstack_resources role skeleton https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/878794 | 07:17 |
| anskiy | noonedeadpunk: nah, it's fine. I wonder if that role could be either unit-tested or pyargv could be moved to uwsgi ini-file. | 07:56 |
| noonedeadpunk | yup, that's fair, we're talking about adding molecule testing for quite a while now, but quite little time to really work on that | 07:57 |
| jrosser | anskiy: can pyargv go in the ini file? | 07:58 |
| noonedeadpunk | The main "blocker" or better say smth we need to figure out, is how to "centrally" define molecule environemnt | 07:58 |
| noonedeadpunk | like ansible versions, required collections, etc | 07:59 |
| noonedeadpunk | molecule version itself | 07:59 |
| anskiy | jrosser: I'm not sure, this change would be better made separate, I guess, so I haven't really looked into that | 07:59 |
| jrosser | iirc it was pretty tricky to find a way to do what pyargv was needed for | 08:00 |
| jrosser | aaah that line in the uwsgi role was origianlly a var in quotes itself "{{ }}" and needed to include quotes in the rendered output - i see whats happened there now | 08:04 |
| noonedeadpunk | yeah, as I said - my bad :( | 08:13 |
| noonedeadpunk | And really wonder how much such bugs I've introduced.... | 08:14 |
| noonedeadpunk | I tried to be careful though... | 08:14 |
| noonedeadpunk | It's good we're not _that_ close to the release | 08:14 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Add openstack_resources role skeleton https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/878794 | 08:58 |
| damiandabrowski | hi folks, so regarding upgrade job failures on stable/2023.1: | 09:05 |
| damiandabrowski | tonight i successfully executed upgrade jobs on my two AIO VMs(scenario=aio_metal, branch=stable/2023.1, os=ubuntu_jammy). | 09:05 |
| damiandabrowski | I have no clue why these jobs are failing in CI. These problems seem to be intermittent(but with high probability to occur). | 09:05 |
| damiandabrowski | Problems started around 13.07.2023. | 09:05 |
| damiandabrowski | openstack-ansible-upgrade-aio_metal-ubuntu-jammy: https://zuul.opendev.org/t/openstack/builds?job_name=openstack-ansible-upgrade-aio_metal-ubuntu-jammy&branch=stable%2F2023.1&skip=0 | 09:05 |
| damiandabrowski | openstack-ansible-upgrade_yoga-aio_metal-ubuntu-focal: https://zuul.opendev.org/t/openstack/builds?job_name=openstack-ansible-upgrade_yoga-aio_metal-ubuntu-focal&branch=stable%2F2023.1&skip=0 | 09:05 |
| damiandabrowski | openstack-ansible-upgrade-aio_metal-rockylinux-9: https://zuul.opendev.org/t/openstack/builds?job_name=openstack-ansible-upgrade-aio_metal-rockylinux-9&branch=stable%2F2023.1&skip=0 | 09:06 |
| noonedeadpunk | damiandabrowski: can you try setting keystone passwords to 64 symbols for key services during initial (N-1) deployment? | 09:10 |
| noonedeadpunk | and then upgrade? | 09:10 |
| noonedeadpunk | Ah! And ensure that your ansible-collections-requirements track top of the branch for openstack-ansible-plugins | 09:10 |
| noonedeadpunk | specifically this patch https://opendev.org/openstack/openstack-ansible-plugins/commit/88a8bfcd62fe7bb027ca7a8636fbe943bfda88c1 | 09:11 |
| noonedeadpunk | on N-1 and N | 09:11 |
| damiandabrowski | 1. keystone passwords - but currently they have 16-64 symbols. So they need to have exactly 64 chars? Why does it change anything? | 09:15 |
| damiandabrowski | 2. um, how to do that in CI for N-1? | 09:15 |
| jrosser | damiandabrowski: https://opendev.org/openstack/keystone/src/branch/master/keystone/common/password_hashing.py#L71 | 09:28 |
| damiandabrowski | ah ok i get i know, so on my local AIO i will set password length >54 chars for all services | 09:34 |
| jrosser | doesnt it need to be less than / equal to 54? | 09:38 |
| jrosser | i beleive that this is something to do with greater than 54 char length passwords being truncated and this then breaking idempotency | 09:38 |
| jrosser | noonedeadpunk: ^ is this correct understanding? | 09:39 |
| jrosser | damiandabrowski: ah you mean to reproduce the CI failure you will try passwords > 54 chars? | 09:40 |
| noonedeadpunk | But to catch error you need it to be >54 | 09:40 |
| noonedeadpunk | so understanding is correct, just depends on what you wanna do :) | 09:40 |
| jrosser | :) | 09:40 |
| noonedeadpunk | damiandabrowski: it would be also great to understand on how to migratre from bcache | 09:41 |
| damiandabrowski | jrosser: yeah, i want to reproduce this issue on my AIO :D | 09:41 |
| damiandabrowski | noonedeadpunk: bcache or bcrypt? | 09:42 |
| noonedeadpunk | ah, yes, bcrypt | 09:42 |
| noonedeadpunk | to scrypt | 09:42 |
| noonedeadpunk | As this is smth we might wanna do internally | 09:43 |
| noonedeadpunk | (and might wanna change default upstream) | 09:43 |
| noonedeadpunk | (or at least in osa) | 09:43 |
| jrosser | what would happen to existing bcrypt passwords? | 09:44 |
| noonedeadpunk | I can recall some option to re-hash | 09:45 |
| * jrosser surprised | 09:46 | |
| noonedeadpunk | at least that was done once when keystone has changed default algo back in.... Queens? | 09:46 |
| noonedeadpunk | https://opendev.org/openstack/keystone/src/commit/8ad765e0230ceeb5ca7c36ec3ed6d25c57b22c9d/releasenotes/notes/bug_1543048_and_1668503-7ead4e15faaab778.yaml | 09:47 |
| damiandabrowski | alternatively, we can leave them as is and just encrypt new passwords with scrypt | 09:48 |
| noonedeadpunk | yes, or that. but this all should be tested/verified first | 09:48 |
| jrosser | yeah, just surprises me that without the original password plaintext you can rehash it | 09:48 |
| jrosser | obviously do-able for service users managed by OSA | 09:48 |
| jrosser | but anything outside of that feels tricky | 09:49 |
| noonedeadpunk | maybe you can't and I got previous change wrong. I guess damiandabrowski is right, and old will continue working "as is" | 09:49 |
| noonedeadpunk | but I was not digging there deep enough | 09:49 |
| amarao | Hello. I found that if external_lb_vip_address points to domain with A and AAAA record, haproxy is listening only on ipv6 address (effectively ignoring A record). Is this a known bug or should I report it? (26.0.0). | 10:07 |
| noonedeadpunk | amarao: well, we have haproxy_bind_external_lb_vip_address and haproxy_bind_internal_lb_vip_address for such cases | 10:21 |
| amarao | Oh, so it's a feature, not a bug. Thanks. | 10:22 |
| noonedeadpunk | Well. That's kinda space for improvement ofc. | 10:23 |
| noonedeadpunk | But it's somehow known | 10:23 |
| noonedeadpunk | Also we have https://docs.openstack.org/openstack-ansible-haproxy_server/latest/configure-haproxy.html#adding-additional-global-vip-addresses that we usually use for IPv6 | 10:24 |
| noonedeadpunk | amarao: OR, you can bind to the interface and don't care about IPs at all. | 10:25 |
| noonedeadpunk | for example - haproxy_bind_external_lb_vip_address: * haproxy_bind_external_lb_vip_interface: bond0 | 10:26 |
| noonedeadpunk | I probably should patch documentation to reflect that.. | 10:26 |
| opendevreview | Merged openstack/ansible-role-systemd_mount stable/zed: Installing systemd-udev with NVR https://review.opendev.org/c/openstack/ansible-role-systemd_mount/+/889352 | 10:28 |
| opendevreview | Merged openstack/ansible-role-uwsgi master: Fix pyargv value rendering https://review.opendev.org/c/openstack/ansible-role-uwsgi/+/889643 | 10:35 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server master: [doc] Document usage of binding to interface https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/889691 | 10:36 |
| opendevreview | Merged openstack/openstack-ansible-plugins stable/zed: Installing systemd-udev with NVR https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/889349 | 10:41 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/yoga: Pin version of setuptools https://review.opendev.org/c/openstack/openstack-ansible/+/889022 | 10:47 |
| hamidlotfi_ | Hi there, | 11:09 |
| hamidlotfi_ | I have a new group with the name `all_lxc_containers` but the before of time it was `all_containers`, what's the difference between these? | 11:09 |
| hamidlotfi_ | And now when running the `setup-hosts.yml --syntax-check` show me this warning: `[WARNING]: Could not match supplied host pattern, ignoring: all_lxc_containers` | 11:09 |
| hamidlotfi_ | In the `Stable/Zed` | 11:10 |
| opendevreview | Merged openstack/openstack-ansible-plugins stable/2023.1: Installing systemd-udev with NVR https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/889348 | 11:14 |
| hamidlotfi_ | @noonedeadpunk | 11:22 |
| noonedeadpunk | hamidlotfi_: I don't think this is smth we have by default but rather smth that was created according to configuration | 11:28 |
| noonedeadpunk | So I have no idea what is that, as it's not present neither in our production envs nor in sandbox | 11:28 |
| noonedeadpunk | damiandabrowski: I have a question to you regarding tempest, as you was the last touching resource creation there:) | 11:30 |
| noonedeadpunk | How does it work at all in case tempest_projects_create: False? As if it's not executed - it will fail all futher tasks? https://opendev.org/openstack/openstack-ansible-os_tempest/src/branch/master/tasks/tempest_resources.yml#L175-L179 | 11:31 |
| hamidlotfi_ | https://www.irccloud.com/pastebin/jGnLAR3P/ | 11:32 |
| noonedeadpunk | ah! | 11:32 |
| hamidlotfi_ | @noonedeadpunk | 11:32 |
| noonedeadpunk | ok, that is dynamic group :) | 11:32 |
| noonedeadpunk | I think it was kinda introduced back in times when we had nspawn containers | 11:34 |
| noonedeadpunk | to produce empty group when there was no lxc_hosts/lxc_containers in inventory | 11:35 |
| noonedeadpunk | Likely, we can drop that now | 11:36 |
| noonedeadpunk | As I don't really see how it could be useful | 11:36 |
| hamidlotfi_ | doesn't pose a problem for deployment? | 11:37 |
| noonedeadpunk | No, it should not? | 11:39 |
| hamidlotfi_ | OK, Thanx | 11:40 |
| noonedeadpunk | that is like that for last 6 years and never was problematic from what I know | 11:40 |
| hamidlotfi_ | ok | 11:41 |
| opendevreview | Merged openstack/openstack-ansible master: nova/haproxy: fix typo in detection of 'serialconsole' https://review.opendev.org/c/openstack/openstack-ansible/+/889417 | 11:44 |
| opendevreview | Merged openstack/openstack-ansible stable/yoga: Include proper vars_file for rally https://review.opendev.org/c/openstack/openstack-ansible/+/888656 | 11:44 |
| opendevreview | Merged openstack/openstack-ansible stable/yoga: Disable upgrade jobs from Xena https://review.opendev.org/c/openstack/openstack-ansible/+/889310 | 11:44 |
| anskiy | jrosser: well, adding `pyargv = --debug` to `/etc/uwsgi/neutron-server.ini` does work. | 11:57 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible stable/2023.1: nova/haproxy: fix typo in detection of 'serialconsole' https://review.opendev.org/c/openstack/openstack-ansible/+/889358 | 12:01 |
| lsudre | Hi, I'm trying to run setup-opentack.yml and I have a task failed https://paste.openstack.org/show/bPsADCP6nymVCm6bwTkP/ this task try to cp /etc/nova/api-paste.ini but my directory has no api-paste.ini file. Have you any idea why this file is missing in my setup? | 12:23 |
| noonedeadpunk | lsudre: I'd say that your venv installation might fail during previous run | 12:27 |
| lsudre | noonedeadpunk: should I rerun setup-infrastructure.yml? | 12:28 |
| noonedeadpunk | I'd suggest to re-run os-nova-install.yml -e venv_rebuild=true | 12:29 |
| lsudre | Ok I'm re-running this file | 12:34 |
| anskiy | noonedeadpunk: ceph-ansible is unusable in Zed, as it requires Ansible 2.12. But next, they're gonna require 2.14 in this PR: https://github.com/ceph/ceph-ansible/pull/7432. | 12:40 |
| anskiy | and with this PR, they're removing group name variables, for example: https://github.com/ceph/ceph-ansible/pull/7432/files#diff-3bdc3933d528a6a3547962f750000ce4ef0980aeebe19c7caeff9820069096e9L721-L726 | 12:42 |
| jrosser | anskiy: i think you have to use very specific versions of ceph-ansible with each OSA release if you want them to use the same ansible | 12:42 |
| noonedeadpunk | anskiy: I'd say it depends on what you mean under "unusable". OSA playbooks does not include their verification of ansible version, and 2.13 is quite compatible with 2.12, so things "just work" if not their verification | 12:42 |
| anskiy | for Zed it's stable-7.0 now, prior to that it was stable-6.0: both of them require the same Ansible verion -- 2.12 | 12:42 |
| noonedeadpunk | anskiy: and then for 2023.2 we're working on merging 2.15 | 12:43 |
| noonedeadpunk | https://review.opendev.org/q/topic:osa%252Fcore-2.15 | 12:43 |
| anskiy | noonedeadpunk: by "unusable" I mean out of the box -- you have to patch ceph-ansible | 12:43 |
| jrosser | anskiy: but we run CI jobs for ceph - those would be broken surely? | 12:44 |
| noonedeadpunk | So, if you're using ceph-ansible playbooks, likely you need to have an independent venv to run it | 12:44 |
| noonedeadpunk | or use it completely independently | 12:44 |
| jrosser | ^ yes thats what we do | 12:44 |
| noonedeadpunk | if you're using OSA playbooks - they work | 12:44 |
| jrosser | totally separate ansible venv for ceph | 12:44 |
| noonedeadpunk | and it's passing CI even | 12:44 |
| jrosser | i think it's good to remember that the OSA<>ceph-ansible integration is primarily for CI / illustration purposes | 12:45 |
| noonedeadpunk | I kinda tried to promote separation of ceph-ansible playbooks to avoid this confusion and level of support/compatibiltiy | 12:45 |
| noonedeadpunk | but quite some folks were against that | 12:45 |
| jrosser | and personally i would totally not use it like that in a prod environment | 12:45 |
| anskiy | well, it was me and damiandabrowski, actually :) | 12:45 |
| noonedeadpunk | and that specific reason was exactly one of several why I wanted to do that | 12:46 |
| noonedeadpunk | As we can't really expect to sync to such narrow requirements of ansible | 12:46 |
| jrosser | the coupling between ansible <> ceph <> openstack versions is far far too tight and you soon find impossible situations at upgrade | 12:46 |
| noonedeadpunk | In PR they release requirement to be >= 2.14 which is way more nicer then current one | 12:47 |
| anskiy | yeah, but at the same time, they have this thing: https://github.com/ceph/ceph-ansible/pull/7432/files#diff-b8a67f35d97d561cb8bdb2bcdfa147c3c4d6fcf91e6d5d589deffb4cf3abe7b3R9 | 12:48 |
| lsudre | Now I found this error when I run os-nova-install.yml -e venv_rebuild=true https://paste.openstack.org/show/bmpigL8X4j9rPC3YrFlo/ | 12:48 |
| anskiy | I was probably wrong, and it's broken only on upgrade, which includes that check, I guess | 12:48 |
| jrosser | right - but in OSA we just use a subset of the ceph-ansible roles | 12:48 |
| anskiy | yeah-yeah, my bad, sorry for that | 12:49 |
| noonedeadpunk | lsudre: os-nova-install.yml -e venv_rebuild=true -e venv_wheels_rebuild=true | 12:50 |
| noonedeadpunk | anskiy: to be frank, I would propose a patch to ceph ansible to introduce a variable or a tag that would allow to avoid this specific version check, saying that it's done at your own risk | 12:51 |
| noonedeadpunk | but dunno... | 12:51 |
| noonedeadpunk | maybe you already can do that | 12:52 |
| anskiy | we have a couple of patches for ceph-ansible already... like the one that prevents ALL OSDs from restart when new one is added | 12:52 |
| anskiy | nah, you can't as of now: https://github.com/ceph/ceph-ansible/blob/stable-7.0/infrastructure-playbooks/rolling_update.yml#L105-L106 | 12:55 |
| noonedeadpunk | yeah, I already checked and they have no tags or anything that would allow to do that | 12:57 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Define default value for _service_adminuri_insecure https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/889707 | 13:06 |
| lsudre | noonedeadpunk: I got this error: https://paste.openstack.org/show/bXrPRhK2BIuVELopIWqX/ | 13:09 |
| noonedeadpunk | lsudre: that is actually weird. Can you kindly show more output? | 13:09 |
| noonedeadpunk | lsudre: also was you using limits when running the playbook? | 13:10 |
| noonedeadpunk | (but I guess it should work nicely with limits as well) | 13:10 |
| noonedeadpunk | lsudre: also can you provide output of `curl http://172.16.12.69:8181/os-releases/27.0.0/ubuntu-22.04-x86_64/wheels`? | 13:12 |
| lsudre | noonedeadpunk: for the curl command the return is https://paste.openstack.org/show/bS15eRK4KdNs6YeYYs4V/ | 13:14 |
| noonedeadpunk | well, then you should look inside your repo-containers and if nginx is running there | 13:15 |
| lsudre | noonedeadpunk: i dont use containers | 13:16 |
| noonedeadpunk | then the host where repo server should run | 13:16 |
| lsudre | where I run the playbooks? | 13:17 |
| lsudre | you can find more logs: https://paste.openstack.org/show/bbFIlJAWmjx9tS2zEP6Q/ | 13:18 |
| noonedeadpunk | lsudre: um, no, your infra nodes likely. | 13:21 |
| noonedeadpunk | You can check that by IP that's configured in haproxy backend | 13:22 |
| noonedeadpunk | or smth like `echo "show stat" | nc -U /run/haproxy.stat | grep repo_all-back` might point you to the host as well | 13:23 |
| noonedeadpunk | or even `echo "show stat" | nc -U /run/haproxy.stat | grep repo_all-back | cut -d ',' -f 2` | 13:23 |
| lsudre | noonedeadpunk: in my infra1 i have nginx and I run th curl command and got a 301 Moved permanently | 13:29 |
| noonedeadpunk | ah, add backslash at the end | 13:30 |
| lsudre | i got a prompt | 13:30 |
| noonedeadpunk | ok, but do you still get 503 when asking through haproxy? | 13:30 |
| noonedeadpunk | or better say - thorugh internal VIP | 13:31 |
| lsudre | I got this html : https://paste.openstack.org/show/bcaBM2m6yhXR1dEIZke7/ | 13:32 |
| noonedeadpunk | lsudre: have you done smth to nginx? As 503 means that no backend is available basically | 13:33 |
| lsudre | systemctl status nginx return me an active and running process | 13:34 |
| noonedeadpunk | as nova version in nginx is same as being asked in your output here https://paste.openstack.org/show/bXrPRhK2BIuVELopIWqX/ | 13:34 |
| noonedeadpunk | But do you still get this https://paste.openstack.org/show/bS15eRK4KdNs6YeYYs4V/ when curling http://172.16.12.69:8181/os-releases/27.0.0/ubuntu-22.04-x86_64/wheels/ ? | 13:35 |
| lsudre | from where? | 13:35 |
| noonedeadpunk | um, let's say from compute01? | 13:35 |
| noonedeadpunk | from where you got it at the first place? | 13:36 |
| jrosser | where is the 503? /me confused | 13:36 |
| noonedeadpunk | yeah. me too | 13:36 |
| lsudre | ok so from my compute I got a : No server is available | 13:36 |
| noonedeadpunk | ok, so that is the issue basically | 13:37 |
| noonedeadpunk | you should be able to access nginx from your computes | 13:37 |
| jrosser | lsudre: it's pretty helpful if you also paste the command you tried and the output too | 13:37 |
| lsudre | I got a 503 when I curl from my deploy host (where I run ansible playbooks) | 13:37 |
| noonedeadpunk | through private VIP | 13:37 |
| jrosser | lsudre: the thing is "I got a 503 when I curl from my deploy host" <- curl what | 13:38 |
| lsudre | 172.16.12.69 is the VIP | 13:38 |
| jrosser | either the VIP or the repo server IP | 13:38 |
| jrosser | it's imporant to be really clear and specific here then we can help the best | 13:38 |
| lsudre | curl this http://172.16.12.69:8181/os-releases/27.0.0/ubuntu-22.04-x86_64/wheels/ | 13:38 |
| jrosser | so where did the HTML output come from? | 13:38 |
| lsudre | jrosser: noonedeadpunk ask me to curl this url | 13:38 |
| lsudre | controller | 13:38 |
| jrosser | and with the same command? | 13:39 |
| noonedeadpunk | sounds like... keepalived split brain? | 13:40 |
| jrosser | if this is a metal deploy then the services need to be bound to !VIP as well | 13:41 |
| lsudre | To avoid some missunderstood: https://pasteboard.co/DppQyfpsnM3F.png | 13:41 |
| jrosser | lsudre: you can only run the haproxy stat command on the host that is running haproxy | 13:42 |
| jrosser | it connects to a local unix socket | 13:42 |
| noonedeadpunk | and please provide output of that :) preferably from each haproxy host | 13:42 |
| lsudre | I think I have only one haproxy host | 13:44 |
| lsudre | It's pretty difficult for me to understand everything of the OSA stack. Before trying to use OSA I installed nova/cinder/horizon/placement on 3 vms (controller,compute,storage host) Now I should have proxy, with LB I don't understand a lot of things in OSA. I was thinking to have one host 172.16.12.69 for the haproxy would be enough | 13:46 |
| lsudre | It's a openstack_user_config problem? | 13:49 |
| jrosser | lsudre: did you start by building an OSA all-in-one? | 13:51 |
| lsudre | no | 13:52 |
| jrosser | the reason there is HAProxy is to do a few things, terminate a proper SSL certificate for the external endpoint, loadbalance between multiple infra hosts so that you get high availability backend services, and also to be able to run multiple HAProxy instances with keepalived (or other) failover between them, again to achieve H/A | 13:53 |
| jrosser | a lot of this is optional, so if you don't want to have H/A then you can run with just one HAProxy | 13:53 |
| lsudre | jrosser: I want to be as close as possible to my final production infrastructure | 13:53 |
| jrosser | then i would recommend that you have 3 infra hosts, each running HAProxy and use keepalived for the VIP | 13:54 |
| jrosser | but a lot of these decisions depend on what you want to acheive, everyone has different requirements | 13:54 |
| lsudre | this is my user_config: https://paste.openstack.org/show/bs11N8kgYuY2gx0gLMUi/ | 13:55 |
| jrosser | OSA is very much a toolbox that lets you construct your own deployment, rather than having only one possible approach | 13:55 |
| lsudre | It's very difficult to find good (and recent) ressources to help like a step by step tutorial with this mega tools | 13:57 |
| jrosser | that is why the all-in-one exists | 13:57 |
| jrosser | have a look at this https://docs.openstack.org/openstack-ansible/2023.1/user/aio/quickstart.html | 13:57 |
| jrosser | it will auto-generate the config for you in a single host test environment | 13:57 |
| jrosser | it should be possible to start with an empty VM / actual host and deploy a test environment in a couple of hours | 13:58 |
| lsudre | "I have already completed this step and installed OpenStack on different machines, tested integrating Ceph, added a second compute node, and ran VMs on both Ubuntu and Windows. Everything was perfect. Now, I want to make the deployment tool (OpenStack-Ansible) work to try to deploy OpenStack automatically on my infrastructure. | 14:01 |
| anskiy | lsudre: you mean, you've installed OpenStack manually before, and now you're trying to use OSA, right? | 14:05 |
| lsudre | yes | 14:06 |
| opendevreview | Merged openstack/openstack-ansible-os_rally stable/yoga: Include proper commit in rally_upper_constraints_url https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/887681 | 14:06 |
| anskiy | could you please show the output of running `echo "show stat" | nc -U /run/haproxy.stat | grep repo_all-back` (the command from before) from your `lb1` host -- the host with HAProxy. | 14:07 |
| lsudre | anskiy: here: https://paste.openstack.org/show/bDb6U9JFDnqtYRtLdvI5/ | 14:09 |
| noonedeadpunk | lsudre: yeah, so somehow haproxy do not mark repo backend as UP | 14:10 |
| anskiy | lsudre: there is `no route to host` error | 14:10 |
| noonedeadpunk | I kinda wonder if the IP for infra1 is correct in haproxy | 14:11 |
| anskiy | looks like correct, according to https://paste.openstack.org/show/bs11N8kgYuY2gx0gLMUi/ | 14:11 |
| noonedeadpunk | but then why haproxy_hosts has same IP as internal_lb_vip_address? | 14:12 |
| noonedeadpunk | lsudre: ^ | 14:12 |
| lsudre | Maybe because I don't understand well what im doing | 14:12 |
| noonedeadpunk | internal_lb_vip_address is supposed to be vritual address, that can failover between hosts | 14:13 |
| noonedeadpunk | It's brought up by keepalived | 14:13 |
| jrosser | well in this case there is only one haproxy? | 14:13 |
| jrosser | so no keepalived is deployed | 14:13 |
| noonedeadpunk | Yeah, but that's conceptually weird | 14:13 |
| lsudre | I think | 14:13 |
| noonedeadpunk | I'm not saying that it is root cause, but it will be wrong for real deployment | 14:14 |
| jrosser | lsudre: do any of the high availabilty things matter for you? | 14:14 |
| lsudre | yes | 14:14 |
| noonedeadpunk | jrosser: I'm also not sure if that is same host or not as well | 14:14 |
| jrosser | yeah | 14:14 |
| lsudre | right now im trying to use OSA in proxmox env with an host and vms | 14:14 |
| jrosser | oh we have people have all sorts of difficulty with this approach before :/ | 14:15 |
| noonedeadpunk | lsudre: so, is haproxy is a separate VM there or it's same? | 14:15 |
| lsudre | 172.16.12.69 is a separate vm | 14:15 |
| noonedeadpunk | ok, then this VM should have yet another IP that will be in the same network as 172.16.12.69 | 14:16 |
| noonedeadpunk | but not used as `internal_lb_vip_address` | 14:16 |
| noonedeadpunk | ideally | 14:16 |
| noonedeadpunk | and then, you should ensure that you can access 172.16.12.71 from VM 172.16.12.69 | 14:17 |
| jrosser | also given that this is proxmox i would do some really basic tests that you can ping the IP of the other VM from the haproxy VM | 14:17 |
| noonedeadpunk | as seems there's smth wrong with networking between these 2 | 14:17 |
| jrosser | and that the traffic actually runs properly br-mgmt -> br-mgmt and not accidentally via some default route or other | 14:18 |
| jrosser | lsudre: if you say that the high-availabity things are important - then you should have 3 infra nodes, and run haproxy on each of them, not necessarily as a separate node | 14:20 |
| lsudre | I made a basic network diagram: https://paste.openstack.org/show/bBchQGS4EoVQ37NfagI9/ | 14:24 |
| jrosser | well, from HAProxy perspective you have a `no route to host` error | 14:27 |
| jrosser | this is the most obvious problem to address first | 14:27 |
| lsudre | where I can investigate? | 14:31 |
| jrosser | well as i said, this is basic networking checks from the haproxy node | 14:36 |
| jrosser | can you ping the .69 address from there, does to routing table look like you expect, etc | 14:36 |
| jrosser | oh not .69, whatever the br-mgmt IP of the infra node is | 14:36 |
| lsudre | I found a pb in my controller node, the br-mgmt bridge has now multiples ip and one is the haproxy node https://paste.openstack.org/show/bgHrFwZNDsPTa5s6PPtG/ | 14:38 |
| jrosser | have you checked where actually haproxy is running? is it on the infra node for some reason? | 14:40 |
| lsudre | no, I think is a missconfiguration, my plan was to have the haproxy on a dedicated host the (.69) | 14:42 |
| damiandabrowski | noonedeadpunk: sorry I missed your question | 15:16 |
| damiandabrowski | hmm, i didn't test it but in fact i think you are right and when you explicitly define tempest_projects_create: False, you won't be able create tempest resources because tasks will fail | 15:17 |
| damiandabrowski | that's why its default value looks like: https://opendev.org/openstack/openstack-ansible-os_tempest/src/branch/master/defaults/main.yml#L63 | 15:18 |
| damiandabrowski | it would be good to cover this case here: https://opendev.org/openstack/openstack-ansible-os_tempest/src/branch/master/tasks/tempest_resources.yml#L26-L84 | 15:19 |
| noonedeadpunk | well, but you can kinda override tempest_projects and have them pre-created. IMO tempest_projects_create - that shouldn't be a requirement, but I am unsure | 15:20 |
| damiandabrowski | ah right, most likely you can define keystone_demo_tenant_id with your project id, but it doesn't have very obvious structure: https://opendev.org/openstack/openstack-ansible-os_tempest/src/branch/master/tasks/tempest_resources.yml#L177 | 15:38 |
| damiandabrowski | i remember that it was quite hard to cover all potential use cases for tempest resource creation - but it's always worth trying :D | 15:40 |
| damiandabrowski | noonedeadpunk: I have a theory why 2023.1 upgrade jobs work fine on my local AIO but fail in CI and in fact it's related to https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/888152 | 15:49 |
| damiandabrowski | it was merged on 13.07.2023 when the problems started so it makes sense | 15:49 |
| noonedeadpunk | haven't I pointed to it earlier today? | 15:49 |
| damiandabrowski | yes, but you didn't share details, so: | 15:50 |
| noonedeadpunk | `[11:11] <noonedeadpunk> specifically this patch https://opendev.org/openstack/openstack-ansible-plugins/commit/88a8bfcd62fe7bb027ca7a8636fbe943bfda88c1` | 15:50 |
| damiandabrowski | my AIO uses e6ce7a42e1282d3b80f422feacf32f6301bbc6b1 version of openstack-ansible-plugins - that's expected behavior | 15:50 |
| damiandabrowski | but for some reason, CI seems to use 88a8bfcd62fe7bb027ca7a8636fbe943bfda88c1: https://zuul.opendev.org/t/openstack/build/4eb51e63841e4eea928c4292987e7324/log/job-output.txt#5099 | 15:51 |
| noonedeadpunk | that's actually also expected | 15:51 |
| noonedeadpunk | we test "latest" on the branch | 15:51 |
| damiandabrowski | ah, i had no idea about it | 15:51 |
| noonedeadpunk | so that depends-on work properly and we detect when we break things before bump | 15:52 |
| noonedeadpunk | But that doesn't explain why it fails. Yes, now we do not forcefully update password on upgrade | 15:52 |
| noonedeadpunk | but why it becomes invalid - that's the question | 15:52 |
| noonedeadpunk | so during password upgrade password can get re-hashed or smth | 15:53 |
| noonedeadpunk | but it's kinda weird | 15:53 |
| damiandabrowski | for me it makes sense now | 15:53 |
| noonedeadpunk | there might be more issues then regarding idempotency of keystone playbooks | 15:54 |
| damiandabrowski | so with service_update_password: true passwords get rehashed and everything works fine | 15:54 |
| noonedeadpunk | yup | 15:54 |
| damiandabrowski | btw. do you remember where exactly do we override collection requirements in CI? | 15:54 |
| damiandabrowski | i'm curious | 15:55 |
| noonedeadpunk | but this does not explain why it doesn't work without that and what exactly breaks | 15:55 |
| noonedeadpunk | we use zuul prepared repos with required_projects | 15:55 |
| noonedeadpunk | and then here | 15:55 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/get-ansible-collection-requirements.yml#L59-L67 | 15:55 |
| damiandabrowski | "but this does not explain why it doesn't work without that and what exactly breaks" because password may have more than 54 characters and since 2023.1 keystone trims it? that's my theory | 15:56 |
| noonedeadpunk | nah, it trims since Pike. Like it always was trimming them | 15:56 |
| noonedeadpunk | And then non-upgrade jobs work nicely with trimming | 15:57 |
| noonedeadpunk | as well as zed/antelope | 15:57 |
| noonedeadpunk | and I saw alerts on Xena that password is too long, so it was trimmed | 15:57 |
| *** me is now known as Guest7047 | 15:58 | |
| damiandabrowski | um, this one was implement in 2023.1: https://opendev.org/openstack/keystone/commit/3288af579de8ee312c36fb78ac9309ce8c554827 | 15:58 |
| damiandabrowski | implemented* | 15:58 |
| Guest7047 | hi team i am upgrading from 24.1.0 to 25.4.0 , and i am getting this error | 15:58 |
| Guest7047 | how to fix it TASK [python_venv_build : Slurp up the constraints file for later re-deployment] **************************************************************************** fatal: [t1w_nova_api_container-0f76010b -> t1w_repo_container-1bd52331(172.29.239.196)]: FAILED! => {"changed": false, "msg": "file not found: /var/www/repo/os-releases/25.4.0/ubuntu-20.04-x86_64/requirements/nova-25.4.0-constraints.txt"} | 15:59 |
| noonedeadpunk | Guest7047: have repo-install.yml finished without issues? | 16:00 |
| noonedeadpunk | ah | 16:01 |
| noonedeadpunk | it's constraint file | 16:01 |
| noonedeadpunk | not u-c | 16:01 |
| damiandabrowski | noonedeadpunk: so if i understand correctly: before 2023.1 password was truncated to `CONF.identity.max_password_length` and since 2023.1 it is truncated to `BCRYPT_MAX_LENGTH` (if `CONF.identity.password_hash_algorithm == 'bcrypt'`) | 16:02 |
| noonedeadpunk | ah | 16:02 |
| noonedeadpunk | damiandabrowski: have you change somewhere handy? | 16:03 |
| noonedeadpunk | Guest7047: try to run openstack-ansible os-nova-install.yml -e venv_rebuild=true | 16:03 |
| damiandabrowski | what? :D | 16:04 |
| noonedeadpunk | damiandabrowski: as wondering when exactly that change landed | 16:04 |
| Guest7047 | ok | 16:04 |
| damiandabrowski | this one? https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/888152 | 16:05 |
| damiandabrowski | or this one? https://review.opendev.org/c/openstack/keystone/+/828595 | 16:05 |
| noonedeadpunk | second :D | 16:05 |
| noonedeadpunk | so.... upgrades to Antelope should also fail then | 16:06 |
| damiandabrowski | and they fail? that is our main issue :D | 16:09 |
| damiandabrowski | example: https://zuul.opendev.org/t/openstack/build/4eb51e63841e4eea928c4292987e7324/ | 16:09 |
| noonedeadpunk | ah, ok then :D | 16:25 |
| noonedeadpunk | I'd say - maybe we set algo to scrypt and add update_password variable? | 16:26 |
| noonedeadpunk | *to upgrade scripts | 16:26 |
| damiandabrowski | for master - yes, but for 2023.1 I suggest to revert https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/888152 | 16:26 |
| damiandabrowski | I'm just writing commit message to explain it more clearly | 16:27 |
| damiandabrowski | what do you think? | 16:27 |
| noonedeadpunk | well, you've seen the bug it was fixing, right? | 16:27 |
| damiandabrowski | yeah, but i think it's either reverting fix for this bug for 2023.1 or change encryption algorithm for stable release | 16:29 |
| damiandabrowski | idk, i'm think with both solutions | 16:29 |
| damiandabrowski | but changing encryption alg for stable release also doesn't sound good :D | 16:29 |
| noonedeadpunk | I'm not saying about changing algo for stable branch | 16:30 |
| noonedeadpunk | but defining update_password for upgrade process is fair requirement | 16:30 |
| noonedeadpunk | for major upgrades at least | 16:30 |
| noonedeadpunk | you still will get all services restarted | 16:30 |
| noonedeadpunk | and regardless of that - this is potentially prolonged API downtime we need to cover in release notes | 16:31 |
| damiandabrowski | ah, makes sense then | 16:31 |
| Guest7047 | still it failed with the same error | 16:32 |
| noonedeadpunk | I'm not saying what's best to do though, but there's totally smth to check on | 16:32 |
| noonedeadpunk | Guest7047: can you check that file `/var/www/repo/os-releases/25.4.0/ubuntu-20.04-x86_64/requirements/nova-25.4.0-constraints.txt` is not present in any of your repo containers? | 16:34 |
| noonedeadpunk | assuming you have 3 of them - worth checking all | 16:34 |
| noonedeadpunk | As I wonder if that is related to potentiall issues with gluster or smth | 16:34 |
| noonedeadpunk | also would be helpful to get, say output including previous 10-15 tasks | 16:35 |
| opendevreview | Merged openstack/ansible-role-zookeeper master: Do not use notify inside handlers https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/888760 | 16:35 |
| Guest7047 | it is there | 16:36 |
| Guest7047 | -rw-r--r-- 1 nginx www-data 162 Jul 26 14:48 nova-25.4.0-requirements.txt | 16:36 |
| Guest7047 | sry that file is not there | 16:37 |
| Guest7047 | nova-25.4.0-constraints.txt | 16:37 |
| noonedeadpunk | you can paste output using https://paste.openstack.org/ just in case | 16:38 |
| noonedeadpunk | but having more output from the role would be awesome to understand what was going on before that lead to the issue | 16:38 |
| Guest7047 | so what should we do , if it is not there as for other services it is there, except neutron,heat which still has to run | 17:07 |
| noonedeadpunk | um, I can't say without seeing more output | 17:15 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Add openstack_resources role skeleton https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/878794 | 17:22 |
| Guest7047 | ok i will copy paste the error in the link | 17:22 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tempest master: Adopt for usage openstack_resources role https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/889741 | 17:25 |
| noonedeadpunk | jrosser: damiandabrowski That's first attempt to use openstack_resources ^ Likely will fail, but that should be a minor failure. | 17:26 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Add openstack_resources role skeleton https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/878794 | 17:27 |
| Guest7047 | i have paste the complete error | 17:28 |
| noonedeadpunk | where? | 17:30 |
| noonedeadpunk | when you paste a unique link is generated | 17:30 |
| * noonedeadpunk checking out for todat | 17:31 | |
| Guest7047 | in https://paste.openstack.org/ | 17:33 |
| Guest7047 | Paste #bxNKPk2jBDgJuaP9rR8x | 17:33 |
| noonedeadpunk | Guest7047: I'm interested in previous tasks. Can you supply whole run of the role to the paste? | 17:35 |
| noonedeadpunk | I got that file is not there, but it does not disclose why it's not there and what tasks run and what were skipped during the run | 17:35 |
| Guest7047 | i will try , but it is too big if i enable verbose | 17:40 |
| Guest7047 | There is a very good article written on that, but not sure | 17:43 |
| Guest7047 | how to get this in to action | 17:43 |
| Guest7047 | https://bugs.launchpad.net/openstack-ansible/+bug/1989506 | 17:43 |
| Guest7047 | Paste #bLSrpnlu2pEUQvJ0MzsI | 17:45 |
| * damiandabrowski will upload changes to fix stable/2023.1 upgrade jobs later this evening | 17:46 | |
| noonedeadpunk | Guest7047: ok, try pls `openstack-ansible os-nova-install.yml -e venv_rebuild=true venv_wheels_rebuild=true` | 17:55 |
| Guest7047 | ok | 17:57 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tempest master: Adopt for usage openstack_resources role https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/889741 | 18:01 |
| noonedeadpunk | Guest7047: oh sorry | 18:02 |
| noonedeadpunk | `openstack-ansible os-nova-install.yml -e venv_rebuild=true -e venv_wheels_rebuild=true` | 18:02 |
| noonedeadpunk | I've forgotten another `-e` | 18:02 |
| Guest7047 | yeah it is running, i have added it | 18:04 |
| Guest7047 | i will update you may be trmw since it will take time | 18:04 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tempest master: Adopt for usage openstack_resources role https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/889741 | 18:40 |
| -opendevstatus- NOTICE: The Gerrit service on review.opendev.org will be offline briefly for a minor upgrade at 21:00 utc, approximately an hour from now | 20:02 | |
| -opendevstatus- NOTICE: The Gerrit service on review.opendev.org will be offline briefly for a minor upgrade, but should return shortly | 21:00 | |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible master: Fix issues with truncated keystone passwords https://review.opendev.org/c/openstack/openstack-ansible/+/889781 | 21:19 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible stable/2023.1: Fix issues with truncated keystone passwords https://review.opendev.org/c/openstack/openstack-ansible/+/889801 | 21:28 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
