| opendevreview | Merged openstack/openstack-ansible stable/2024.1: Remove ansible-role-qdrouterd from zuul required-projects https://review.opendev.org/c/openstack/openstack-ansible/+/939914 | 03:24 |
|---|---|---|
| hamidlotfi_ | Hi there, I apologize; this may not be the right place to ask this question. | 04:49 |
| hamidlotfi_ | Is enabling or disabling SNAT for each port individually in the router definition possible (OVS)? Please advise. | 04:49 |
| noonedeadpunk | hamidlotfi_: no, I think you can do that only to the router as a whole | 10:50 |
| birbilakos | Hi everyone. I'd like to hook up an existing ceph storage cluster in my openstack. Cluster is already in the storage network. I'm thinking to opt of the second approach as explained here: https://docs.openstack.org/openstack-ansible/latest/user/ceph/full-deploy.html which is basically defining the ceph mons in user_variables | 11:29 |
| birbilakos | If i understand it right, I also need this in my user_variables: https://docs.openstack.org/openstack-ansible-ceph_client/latest/config-from-file.html | 11:30 |
| noonedeadpunk | birbilakos: so I think second approach is not defining ceph_mons but instead provide ceph.conf and keyrings into /etc/openstack_deploy folder | 11:31 |
| noonedeadpunk | and I'd indeed suggest using this method | 11:31 |
| birbilakos | hmm | 11:31 |
| birbilakos | so it's either-or, right? | 11:31 |
| noonedeadpunk | As the one with defining ceph_mons would require osa deploy host to SSH to these hosts to fetch data from the directly | 11:31 |
| birbilakos | I though I need both | 11:31 |
| noonedeadpunk | no, it's either or | 11:32 |
| noonedeadpunk | if you have ceph.conf provided - you don't need ceph_mons | 11:32 |
| birbilakos | got it | 11:32 |
| noonedeadpunk | and also - defining ceph_mons will trigger logic to go to these mons and fetch ceph.conf from them | 11:32 |
| birbilakos | so I need to fix user_variables.yml based on the second page | 11:33 |
| birbilakos | what would be the playbooks to run? I need integration with cinder, glance for sure | 11:33 |
| noonedeadpunk | ceph_client is triggered from inside of service roles | 11:34 |
| noonedeadpunk | so it will be part of os_glance, os_cinder, os_nova | 11:34 |
| birbilakos | and btw, I would like to keep my already defined cinder & glance storage config | 11:34 |
| birbilakos | here's the current config: https://paste.opendev.org/show/byFYutEIfEeLW479MQkf/ | 11:35 |
| noonedeadpunk | so they all do support working with multiple backends. But depending on your use-case, you still might need to be cautious with configuration to get intended behaviour | 11:35 |
| birbilakos | i want to keep the nfs backends since it has stuff stored | 11:35 |
| noonedeadpunk | one things with cinder and Ceph/NFS, is that NFS driver does not support active/active mode, while Ceph does support | 11:36 |
| birbilakos | i just want to 'append' the ceph option on top (and make it default backend going forward) | 11:36 |
| noonedeadpunk | so what we did when were migrating from NFS to Ceph - defined a second group of cinder_volume for ceph specifically (inside lxc) | 11:36 |
| birbilakos | let me clarify: the existing config uses NFS but its not backed by Ceph | 11:37 |
| birbilakos | I'd like to move away from NFS for cinder & glance (eventually) | 11:37 |
| birbilakos | but need both options for now in them | 11:38 |
| noonedeadpunk | yeah, sure, I got that. We had quite same issue | 11:38 |
| noonedeadpunk | and that is why I'm telling you about separate set of cinder_volume for Ceph | 11:38 |
| noonedeadpunk | Glance is not an issue at all | 11:38 |
| noonedeadpunk | It is smart enough and stores path for all existing images | 11:38 |
| noonedeadpunk | So once you change default store - old images will keep working | 11:39 |
| noonedeadpunk | and for cinder - it's better to separate cinder-volume working with ceph and with nfs, imo | 11:40 |
| birbilakos | hmm, how can this be done? | 11:40 |
| noonedeadpunk | I don't have exact config with me, as we've wiped nfs leftovers years ago, but we've defined extra groups which are part of cinder_volume and used group_vars to control which one will have what backend | 11:41 |
| noonedeadpunk | but we were also running NFS one on bare metal rather then LXC | 11:42 |
| birbilakos | ok | 11:42 |
| birbilakos | I think I'll just get rid of the current cinder backend and just keep the NFS backed glance | 11:42 |
| birbilakos | i reckon just removing the cinder config from openstack_user_config would be sufficient, right? | 11:43 |
| noonedeadpunk | um | 11:45 |
| noonedeadpunk | given you have no VMs running volumes on NFS - yes | 11:46 |
| noonedeadpunk | so we did defined new group through env.d for sure: https://docs.openstack.org/openstack-ansible/latest/reference/inventory/understanding-inventory.html#understanding-container-groups-env-d-structure | 11:48 |
| noonedeadpunk | but I really can't find good enough example right now in our docs... | 11:49 |
| noonedeadpunk | probably that: https://docs.openstack.org/openstack-ansible/latest/user/prod/provnet_groups.html#custom-groups | 11:51 |
| jrosser | if you use boot-from-volume there are advantages to having cinder and glance on the same ceph cluster | 12:05 |
| jrosser | it cann be set up so that there is no data copying / just a snapshot to get from your glance image to boot volume | 12:05 |
| noonedeadpunk | hm, I've tried to just roll-out capi_proxy, and I also have error like was reported by somebody a while back | 12:14 |
| noonedeadpunk | requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://internal.cloud.io:6443/apis/infrastructure.cluster.x-k8s.io/v1beta1/namespaces/magnum-system/openstackclusters | 12:14 |
| noonedeadpunk | while it obviously does exist | 12:15 |
| noonedeadpunk | `kubectl -n magnum-system get openstackclusters` returns non-empty reply | 12:15 |
| noonedeadpunk | kubectl -n magnum-system get openstackclusters uses slightly different url though.... | 12:45 |
| noonedeadpunk | https://internal.cloud.io:6443/apis/infrastructure.cluster.x-k8s.io/v1alpha7/namespaces/magnum-system/openstackclusters vs https://internal.cloud.io:6443/apis/infrastructure.cluster.x-k8s.io/v1beta1/namespaces/magnum-system/openstackclusters | 12:45 |
| noonedeadpunk | so v1alpha7 vs v1beta1 | 12:47 |
| noonedeadpunk | ok, so this: https://github.com/vexxhost/magnum-cluster-api/commit/8e375c414ab3e2027486b105b12765891850c8ac | 12:51 |
| noonedeadpunk | aha | 12:52 |
| noonedeadpunk | so mcapi_vexxhost_proxy_install_branch is not really aligned with magnum_magnum_cluster_api_git_install_branch | 12:52 |
| noonedeadpunk | ok, makes sense now | 12:52 |
| opendevreview | Merged openstack/openstack-ansible stable/2023.2: Remove ansible-role-qdrouterd from zuul required-projects https://review.opendev.org/c/openstack/openstack-ansible/+/939915 | 12:55 |
| noonedeadpunk | jrosser: do you know from top of your head - which interface magnum capi tries to reach the proxy from? | 13:32 |
| jrosser | the proxy talks to mcapi through the internal vip to update capi idea of where the workload cluster api endpoint is, I think | 13:34 |
| jrosser | then mcapi calls direct to the proxy ip it was given | 13:35 |
| noonedeadpunk | so I'm thinking about how to limit haproxy binding | 13:36 |
| noonedeadpunk | as * is - ugh. There's an env var PROXY_BIND which can be used | 13:36 |
| noonedeadpunk | and my guess would be - that it should be management_address, as capi don't have anything else to communicate with haproxy... | 13:37 |
| noonedeadpunk | but I'm not 100% sure | 13:37 |
| noonedeadpunk | as the only network capi master cluster and compute share - is management network kinda... | 13:39 |
| noonedeadpunk | yeah, sure internal vip can be used for communication | 13:39 |
| jrosser | this should be in the example? | 13:39 |
| noonedeadpunk | there's no doc for proxy part fwiw | 13:40 |
| noonedeadpunk | or you mean vexxhost docs? | 13:40 |
| jrosser | proxy_bind should yes be mgmt address on the network/compute node | 13:40 |
| noonedeadpunk | ++ | 13:40 |
| noonedeadpunk | thanks! | 13:40 |
| jrosser | oh dog maybe I didn’t finish that | 13:40 |
| jrosser | also port | 13:40 |
| noonedeadpunk | I will write some docs then | 13:40 |
| jrosser | there’s a var for that otherwise it s random | 13:40 |
| noonedeadpunk | or well... we figure it out ) | 13:40 |
| noonedeadpunk | yeah, saw also port | 13:40 |
| noonedeadpunk | but wasn't sure if it might need to be random | 13:41 |
| jrosser | the probably the “k8s way” :) | 13:41 |
| noonedeadpunk | ok, backend is identified by sni, so it can be very much static | 13:41 |
| noonedeadpunk | yeah, I hate env vars.... | 13:41 |
| hamidlotfi_ | noonedeadpunk: Thanks. | 13:41 |
| noonedeadpunk | I see no good reason why this can not be in conf as well... | 13:42 |
| jrosser | oh well, i added the env vars | 13:43 |
| jrosser | before it was just wired to 0.0.0.0:random | 13:43 |
| jrosser | as i am not sure there is a config file for this right now? | 13:44 |
| noonedeadpunk | so proxy does try to load config file, but it's not required | 13:44 |
| noonedeadpunk | and rest goes from ENV vars | 13:44 |
| jrosser | ah ok so we could make this be better | 13:45 |
| jrosser | though of course there is no magnum.conf on the computes/network nodes....... | 13:45 |
| noonedeadpunk | and the role has magnum_cluster_api_proxy_environment | 13:45 |
| noonedeadpunk | which passes them to systemd unit | 13:45 |
| jrosser | yes | 13:45 |
| noonedeadpunk | well. role does create /etc/capi_proxy but it's an empty folder | 13:47 |
| noonedeadpunk | hamidlotfi_: btw, was it you who struggled with CAPI driver for Magnum? | 13:51 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Respect defined version and source of mcapi driver https://review.opendev.org/c/openstack/openstack-ansible-ops/+/940217 | 14:05 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Fix a typo in mcapi_vexxhost_proxy_git_constraints https://review.opendev.org/c/openstack/openstack-ansible-ops/+/940218 | 14:06 |
| noonedeadpunk | though I have a different more weird issue now with haproxy not marking backend as up :( | 14:11 |
| noonedeadpunk | anyway... | 14:11 |
| jrosser | is this ovn? | 14:13 |
| jrosser | linuxbridge/ovs it goes on the network nodes | 14:13 |
| jrosser | ovn it needs to be on *all* the computes | 14:13 |
| noonedeadpunk | yeah, it's on all computes | 14:15 |
| noonedeadpunk | but I've just spotted that it goes to loadbalancer rather then spawned controller | 14:16 |
| noonedeadpunk | and I was checking connection to controller. Though LB is indeed unreachable from ovnmeta namespace for some reason | 14:16 |
| noonedeadpunk | so likely it's smth off with env inself | 14:17 |
| jrosser | I think it will always go to the lb rather than the controller | 14:24 |
| jrosser | as the controller ip will be unknown, and change during upgrade or self-healing | 14:25 |
| birbilakos | noonedeadpunk: I went ahead and deployed the cinder changes - now I see the following in the cinder volumes container: Jan 26 14:34:56 infra1-cinder-volumes-container-d4c1df53 cinder-volume[64]: 2025-01-26 14:34:56.075 64 ERROR cinder.service [-] Manager for service cinder-volume infra1-cinder-volumes-container-d4c1df53@rbd is reporting problems, not sending heartbeat. Service will appear "down". | 14:35 |
| birbilakos | sudo ceph -n client.cinder status works in the container so this doesn't seem like a connection issue | 14:36 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Move variables defenition from playbook level for mcapi proxy https://review.opendev.org/c/openstack/openstack-ansible-ops/+/940220 | 14:41 |
| noonedeadpunk | jrosser: yeah, that's fair. though I somehow don't have access from the namespace to the loadbalancer. While loadbalancer seems doing fine and I don't see issues reaching k8s controller from amphora... so it's somehow weird... I kinda wonder in metadata namespace even gonna have access if vm spawned with disk-config | 14:45 |
| birbilakos | Full log: https://paste.opendev.org/show/bBmv7etvZrttKQiVe5vd/ | 14:45 |
| noonedeadpunk | as then metadata isn't really used | 14:45 |
| noonedeadpunk | birbilakos: and does `rbd -n client.cinder ls -p volumes` work? | 14:46 |
| noonedeadpunk | as it looks you didn't give enough permissions | 14:46 |
| birbilakos | root@infra1-cinder-volumes-container-d4c1df53:~# rbd -n client.cinder ls -p volumroot@infra1-cinder-volumes-container-d4c1df53:~# rbd -n client.cinder ls -p volumes rbd: error opening pool 'volumes': (2) No such file or directory rbd: listing images failed: (2) No such file or directoryes rbd: error opening pool 'volumes': (2) No such file or directory rbd: listing images failed: (2) No such file or directory | 14:46 |
| noonedeadpunk | ok, I in fact don't know what was the pool name you've configured in ceph | 14:47 |
| noonedeadpunk | and in cinder | 14:47 |
| birbilakos | default config | 14:47 |
| noonedeadpunk | but I'm pretty sure it's a premission issue for the client to the pool | 14:47 |
| birbilakos | do I need to create the pool on the ceph side? | 14:47 |
| noonedeadpunk | there's no such thing as "default cofig" | 14:48 |
| noonedeadpunk | yes, sure | 14:48 |
| noonedeadpunk | and ensure client.cinder has enough permissions to it | 14:48 |
| birbilakos | well, I gave admin perms to client.cinder | 14:48 |
| noonedeadpunk | um | 14:48 |
| noonedeadpunk | it doesn't need them for sure | 14:48 |
| birbilakos | client.cinder key: <key> caps: [mds] allow * caps: [mgr] allow * caps: [mon] allow * caps: [osd] allow * | 14:49 |
| birbilakos | rbd_pool: volumes | 14:50 |
| noonedeadpunk | so we set permissions like that to ceph we setup with ceph-ansible: https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/all/ceph.yml#L53-L60 | 14:50 |
| noonedeadpunk | andpools like that: https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/ceph_all.yml#L51-L71 | 14:50 |
| noonedeadpunk | but neverthe less - with external ceph you need to ensure existance of clients and pools and pretty much everything | 14:51 |
| noonedeadpunk | (caps) | 14:51 |
| noonedeadpunk | as cinder/glance just consume what are given | 14:51 |
| birbilakos | just added the pool volumes and enabled application rbd to it | 14:53 |
| birbilakos | restarted cinder volumes and still getting same errors. Seems perms related | 14:53 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Move variables defenition from playbook level for mcapi proxy https://review.opendev.org/c/openstack/openstack-ansible-ops/+/940220 | 14:54 |
| birbilakos | rados.PermissionDeniedError: [errno 13] RADOS permission denied (error connecting to the cluster) | 14:54 |
| noonedeadpunk | seems like this, yes | 14:54 |
| opendevreview | Merged openstack/openstack-ansible stable/2023.1: Remove ansible-role-qdrouterd from zuul required-projects https://review.opendev.org/c/openstack/openstack-ansible/+/939941 | 14:54 |
| birbilakos | here's the relevant config I see in the cinder volumes container: https://paste.opendev.org/show/bwGt5aUtAYEXDbu72NkL/ | 14:57 |
| noonedeadpunk | I kind of wonder how you was setting permissions | 14:58 |
| birbilakos | sudo ceph auth get-or-create client.cinder mon 'allow *' mgr 'allow *' osd 'allow *' mds 'allow *' -o client.cinder | 14:59 |
| birbilakos | this should create the client for cinder and the keyring | 15:00 |
| birbilakos | is it possible that rados is trying to access with a different client? | 15:05 |
| birbilakos | is the rbd_secret_uuid something I should fill in in the config? | 15:07 |
| noonedeadpunk | yeah, that should be enough... | 15:08 |
| noonedeadpunk | rbd_secret_uuid - um, I think it's mainly for qemu | 15:08 |
| noonedeadpunk | also - you still should be able to do some listing/oprations to the pool with the client | 15:09 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Move variables defenition from playbook level for mcapi proxy https://review.opendev.org/c/openstack/openstack-ansible-ops/+/940220 | 15:13 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Move variables defenition from playbook level for mcapi proxy https://review.opendev.org/c/openstack/openstack-ansible-ops/+/940220 | 15:15 |
| birbilakos | this works: rados --id cinder --keyring cinder.keyring -p volumes ls | 15:18 |
| birbilakos | and this also works form within the container: rados --id cinder --keyring /etc/ceph/ceph.client.cinder.keyring -p volumes ls | 15:21 |
| noonedeadpunk | I kind of wonder about keyring naming though | 15:46 |
| noonedeadpunk | yeah, looks like it's correct one | 15:47 |
| noonedeadpunk | ok, but that is both rados | 15:48 |
| noonedeadpunk | what about rbd? | 15:48 |
| noonedeadpunk | as rados is not _that_ relevant for cinder, it mostly operates as rbd | 15:48 |
| noonedeadpunk | (though it would be still needed) | 15:49 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Move variables defenition from playbook level for mcapi proxy https://review.opendev.org/c/openstack/openstack-ansible-ops/+/940220 | 15:49 |
| birbilakos | is it a probelm that ceph version is quincy in my cluster and reef in the container? | 15:50 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: [doc] Add brief documentation for mcapi proxy https://review.opendev.org/c/openstack/openstack-ansible-ops/+/940222 | 15:50 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: [doc] Add brief documentation for mcapi proxy https://review.opendev.org/c/openstack/openstack-ansible-ops/+/940222 | 15:53 |
| noonedeadpunk | I don't think it's actually a problem, as clients do have backwards compatability | 15:53 |
| birbilakos | root@infra1-cinder-volumes-container-d4c1df53:~# rbd --keyring /etc/ceph/ceph.client.cinder.keyring ls volumes 2025-01-26T15:53:36.528+0000 7fe0917f7640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] rbd: couldn't connect to the cluster! rbd: listing images failed: (13) Permission denied | 15:54 |
| birbilakos | any ideas what is that? | 15:54 |
| noonedeadpunk | not really, no | 15:54 |
| noonedeadpunk | only guess could be monitor api version 1 vs 2 | 15:54 |
| birbilakos | rbd -n client.cinder ls volumes works from within the container | 16:17 |
| birbilakos | still getting rados.PermissionDeniedError: [errno 13] RADOS permission denied (error connecting to the cluster) from cinder volumes though :( | 16:19 |
| birbilakos | noonedeadpunk: well, I ended up using client admin with the appropriate keyring. It worked :) | 16:42 |
| birbilakos | another question: i noticed that new instances store their disk in the vm pool now, even though I don't create a volume for them. Why is that? Can I avoid it and keep the disk local to the compute unless block storage mapping is requested? | 17:30 |
| birbilakos | would simply removing this line be enough? nova_libvirt_images_rbd_pool: vms | 17:37 |
| birbilakos | it worked, though it would be a nice feature to have on some occassions. Wonder if it could be configured when creating an instance... | 19:10 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!