| mnasiadka | morning | 07:57 |
|---|---|---|
| mnasiadka | trivial patches for this week - ebbex's assert usage (instead of fail: module) - https://review.opendev.org/q/topic:assert | 07:58 |
| *** ralonsoh_ is now known as ralonsoh | 08:37 | |
| guesswhat[m] | Can anyone help me with integrating Cinder backend for Glances? Seems that it does not work correctly. In my installation is Cinder enabled, even internalURL endpoint is set in glance-api.conf https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/glance/templates/glance-api.conf.j2#L23, but it fails to https://pastebin.com/raw/mLjx48ue , even if I use kolla_copy_ca_into_containers: "yes" or if publicURL is | 09:18 |
| guesswhat[m] | certificate is signed with Lets Encrypt. I guess single node and devstacks can benefit from Cinder backend. | 09:18 |
| guesswhat[m] | Another question is why it is trying to communitace to publicURL endpoint, when internalURL is explicitly set. Ideas? Thanks | 09:18 |
| kevko | guesswhat[m]: why are u copy ca into container if you are using letsencrypt ... you need to copy ca into container if it is your ca ... | 09:24 |
| kevko | guesswhat[m]: debug with cul | 09:25 |
| kevko | curl | 09:25 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Add support for CentOS Stream 9 https://review.opendev.org/c/openstack/kolla/+/869906 | 09:49 |
| opendevreview | Merged openstack/kolla stable/xena: CI: store only templated out Dockerfiles in work_dir https://review.opendev.org/c/openstack/kolla/+/873005 | 10:19 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support https://review.opendev.org/c/openstack/kolla/+/873549 | 10:27 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Add support for CentOS Stream 9 https://review.opendev.org/c/openstack/kolla/+/869906 | 10:33 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Add support for CentOS Stream 9 https://review.opendev.org/c/openstack/kolla/+/869906 | 10:42 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support https://review.opendev.org/c/openstack/kolla/+/873549 | 10:42 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Add support for CentOS Stream 9 https://review.opendev.org/c/openstack/kolla/+/869906 | 10:44 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support https://review.opendev.org/c/openstack/kolla/+/873549 | 10:48 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support https://review.opendev.org/c/openstack/kolla/+/873549 | 10:48 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support https://review.opendev.org/c/openstack/kolla/+/873549 | 10:49 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support https://review.opendev.org/c/openstack/kolla/+/873549 | 10:50 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support https://review.opendev.org/c/openstack/kolla/+/873549 | 10:51 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support https://review.opendev.org/c/openstack/kolla/+/873549 | 10:52 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support https://review.opendev.org/c/openstack/kolla/+/873549 | 10:52 |
| guesswhat[m] | kevko: I tried also selfsigned certificate generated via kolla-ansible, same errror.. | 11:03 |
| kevko | guesswhat[m]: post here pastebin with your configuration related to tls and also directory structure of your kolla config via pastebin or something similar | 11:04 |
| guesswhat[m] | problem would be in some cacert, not sure which component tho, probably glance I guess, or maybe even in glance python client | 11:04 |
| guesswhat[m] | I tested two cases: 1. haproxy.pem generated from ACME and 2. haproxy.pem default one ( kolla_enable_tls_external: "yes" ) generated via kolla-ansible | 11:06 |
| guesswhat[m] | Both cases fails to TLS validation error, but only for glance component, other components/services are running correctly | 11:06 |
| guesswhat[m] | Can be reproduced with default Cinder & Glance configuration. I changed https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/glance/templates/glance-api.conf.j2#L66 to cinder and deleted file and http backends https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/glance/templates/glance-api.conf.j2#L16 | 11:08 |
| guesswhat[m] | Otherwise Glance fallbacks to working backends, in this case It would be http or file backend | 11:08 |
| guesswhat[m] | Its weird, because Glance is calling public URL of Cinder, I believe this is not necessary | 11:09 |
| guesswhat[m] | kevko: its almost default installation with cinder lvm backend, but thats not related I guess | 11:15 |
| kevko | guesswhat[m]: and where did you read that glance support cinder as backend ? :D | 11:20 |
| kevko | guesswhat[m]: ah yes, you are right | 11:20 |
| kevko | guesswhat[m]: # * file | 11:21 |
| kevko | # * http | 11:21 |
| kevko | # * swift | 11:21 |
| kevko | # * rbd | 11:21 |
| kevko | # * cinder | 11:21 |
| kevko | # * vmware | 11:21 |
| kevko | # * s3 | 11:21 |
| kevko | guesswhat[m]: firstly you should leave default_backends as is i think because it is only default backends ... enabled backends is configured by glance_backends | 11:24 |
| kevko | guesswhat[m]: ok now i got a point ...you set default backend for cinder and deleted configured backends just to verify cinder will be one and only glance will try to contact and you found an issue with tls ...right ? | 11:26 |
| guesswhat[m] | yes | 11:26 |
| guesswhat[m] | this way its easily reproducible | 11:26 |
| kevko | guesswhat[m]: ok, i got it ...and glance is contacting cinder via public which is already under tls ..right ? | 11:27 |
| guesswhat[m] | yes | 11:27 |
| guesswhat[m] | it shouldnt, cuz there is cinder_catalog_info = volume:cinder:internalURL | 11:28 |
| kevko | guesswhat[m]: ok, so when you exec into glance_api container and touch cinder api with curl standalone ..you will get same results right ? | 11:28 |
| guesswhat[m] | kevko: thats working | 11:36 |
| kevko | could you paste entire log error ? | 11:37 |
| guesswhat[m] | other services are respecting cacert or/and custom certificates | 11:37 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support https://review.opendev.org/c/openstack/kolla/+/873549 | 11:39 |
| guesswhat[m] | kevko: https://pastebin.com/raw/fc6qgaBh | 11:41 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support https://review.opendev.org/c/openstack/kolla/+/873549 | 11:41 |
| kevko | guesswhat[m]: what images for kolla do you use ? | 11:42 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support https://review.opendev.org/c/openstack/kolla/+/873549 | 11:43 |
| guesswhat[m] | kevko: zed/jammy | 11:44 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: Put etcd behind HTTP loadbalancer https://review.opendev.org/c/openstack/kolla-ansible/+/852391 | 11:45 |
| guesswhat[m] | `docker exec -it glance_api curl -IL -H "Host: openstack.host.cloud" https://172.17.0.12` working correctly, thats public interface | 11:46 |
| kevko | guesswhat[m]: for me it looks like configuration of openstack glance-> cinder connection is somehow broken | 11:47 |
| kevko | guesswhat[m]: try it without -H "Host: ....." | 11:47 |
| kevko | guesswhat[m]: why do you sending Host header if it is resolvable ? | 11:48 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Drop failing Zun jobs https://review.opendev.org/c/openstack/kolla-ansible/+/873551 | 11:50 |
| kevko | guesswhat[m]: get token from keystone .... openstack token issue ... and run this in glance-api curl -v -H "X-Auth-Token: YOUR_TOKEN" https://openstack.public.cloud:8776/v3/ | 11:50 |
| kevko | guesswhat[m]: it should fail i think | 11:50 |
| guesswhat[m] | Got 200 response code without -H header | 11:54 |
| guesswhat[m] | AFAIK its not neeeded to test it with different header, curl would report problem with SSL | 11:55 |
| guesswhat[m] | Maybe cinder_ca_certificates_file ( https://docs.openstack.org/glance/latest/configuration/configuring.html ) have to be specified | 11:58 |
| guesswhat[m] | But that does not explain why its calling public URL endpoint | 12:00 |
| guesswhat[m] | even cinder_api_insecure = true in [DEFAULT] table in glance-api.conf does not help | 12:01 |
| guesswhat[m] | Its somehow broken | 12:01 |
| opendevreview | Ivan Halomi proposed openstack/kolla-ansible master: Splitting docker_worker into container_worker https://review.opendev.org/c/openstack/kolla-ansible/+/863414 | 12:03 |
| kevko | guesswhat[m]: try to setup this cinder_ca_certificates_file=/path/of/ca | 12:15 |
| kevko | guesswhat[m]: https://docs.openstack.org/glance/latest/configuration/configuring.html#configuring-the-cinder-storage-backend | 12:17 |
| kevko | guesswhat[m]: or try to play with cinder_endpoint_template or cinder_catalog_info | 12:18 |
| kevko | guesswhat[m]: or, do you tried to set cinder_api_insecure = false ? | 12:20 |
| *** priteau_ is now known as priteau | 12:25 | |
| opendevreview | Ivan Halomi proposed openstack/kolla-ansible master: Splitting docker_worker into container_worker https://review.opendev.org/c/openstack/kolla-ansible/+/863414 | 12:34 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Add support for CentOS Stream 9 https://review.opendev.org/c/openstack/kolla/+/869906 | 12:35 |
| opendevreview | Ivan Halomi proposed openstack/kolla-ansible master: Splitting docker_worker into container_worker https://review.opendev.org/c/openstack/kolla-ansible/+/863414 | 12:41 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support https://review.opendev.org/c/openstack/kolla/+/873549 | 12:43 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support https://review.opendev.org/c/openstack/kolla/+/873549 | 12:48 |
| mnasiadka | kevko, bbezak: https://review.opendev.org/c/openstack/ansible-collection-kolla/+/872626 - if we could merge this, we could pin Zun jobs to docker 22 instead of removing them | 12:57 |
| opendevreview | Michal Nasiadka proposed openstack/ansible-collection-kolla master: docker: Add docker_systemd_reload functionality https://review.opendev.org/c/openstack/ansible-collection-kolla/+/871293 | 12:57 |
| kevko | mnasiadka: looks ok, i will check after zuul | 12:59 |
| kevko | *chec again | 12:59 |
| opendevreview | Michal Nasiadka proposed openstack/ansible-collection-kolla master: Add support for pinning docker package https://review.opendev.org/c/openstack/ansible-collection-kolla/+/872626 | 13:02 |
| mnasiadka | ok, rebased that one as well | 13:02 |
| opendevreview | Verification of a change to openstack/kayobe master failed: CI: Use nested nodepools for jobs that are spawning vms https://review.opendev.org/c/openstack/kayobe/+/872647 | 13:03 |
| guesswhat[m] | @kevko: cinder_api_insecure = false/true same error... , seems its broken somehow | 13:03 |
| kevko | mnasiadka: would you recommend ovn for production ? Because we still using openvswitch for customers ..but I am thinking about ovn-based deployment | 13:28 |
| kevko | mnasiadka: for the new one | 13:30 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: Fix deploy/genconfig in check mode https://review.opendev.org/c/openstack/kolla-ansible/+/870014 | 14:03 |
| kevko | guesswhat[m]: well, investigate code :P | 14:12 |
| guesswhat[m] | kevko: got some progress https://pastebin.com/raw/dhtAq0hB , now getting 2023-02-13 15:24:28.578 667 ERROR glance.common.wsgi os_brick.exception.VolumeDeviceNotFound: Volume device not found at . | 14:26 |
| kevko | guesswhat[m]: When i will have some time ..i will try...but we are using glance with rbd backend | 14:27 |
| opendevreview | Matt Crees proposed openstack/kolla-ansible master: Use the upgraded image to run Nova upgrade checks https://review.opendev.org/c/openstack/kolla-ansible/+/871279 | 14:51 |
| guesswhat[m] | kevko: after adding cinder_volume_type = lvm-1 ( default from kolla ), glance is not starting to ERROR: 'NoneType' object has no attribute 'user_id' I guess glance-api.conf must contain cinder connection | 14:55 |
| kevko | guesswhat[m]: i am not sure if i can help you because i never tried to setup before | 14:57 |
| opendevreview | Matt Crees proposed openstack/kolla-ansible master: Use the upgraded image to run Nova upgrade checks https://review.opendev.org/c/openstack/kolla-ansible/+/871279 | 15:02 |
| opendevreview | Matt Crees proposed openstack/kolla-ansible master: Use the upgraded image to run Nova upgrade checks https://review.opendev.org/c/openstack/kolla-ansible/+/871279 | 15:16 |
| opendevreview | Merged openstack/ansible-collection-kolla master: Add support for pinning docker package https://review.opendev.org/c/openstack/ansible-collection-kolla/+/872626 | 15:26 |
| guesswhat[m] | @kevko heres my config, seems that cinder connection string must be defined https://pastebin.com/raw/7L0az7pb | 15:35 |
| guesswhat[m] | Still getting glance.common.wsgi os_brick.exception.VolumeDeviceNotFound: Volume device not found at . | 15:35 |
| guesswhat[m] | DEFAULT should be default volume type | 15:35 |
| kevko | guesswhat[m]: haha, yeah that might have attacked me, sorry ... enabled backens have to be defined as it is in cinder for example | 15:36 |
| guesswhat[m] | Its still does not work :/ | 15:36 |
| guesswhat[m] | Maybe You will find some dragons there | 15:36 |
| guesswhat[m] | Its pitty that it does not work for kollla ansible out of the box | 15:38 |
| kevko | guesswhat[m]: well, I think we don't have scenario for this ? w8 a minute i will check ... Most people using openstack together with ceph | 15:49 |
| opendevreview | Ivan Halomi proposed openstack/kolla-ansible master: Splitting docker_worker into container_worker https://review.opendev.org/c/openstack/kolla-ansible/+/863414 | 15:51 |
| guesswhat[m] | Small scale, single node deployment based on LVM, NFS | 15:52 |
| kevko | guesswhat[m]: specify cinder_volume_type = TYPE_OF_VOLUME into cinder block | 15:52 |
| opendevreview | Martin Hiner proposed openstack/kolla-ansible master: Refactor DockerWorker into ContainerWorker https://review.opendev.org/c/openstack/kolla-ansible/+/863414 | 15:53 |
| guesswhat[m] | Default Glance backend ( file ) is consuming space on primary host disk | 15:53 |
| kevko | guesswhat[m]: did you try to define new volume type and not use __DEFAULT__ ? | 15:54 |
| guesswhat[m] | While Cinder ( with LVM ) forces You to use additional disk | 15:54 |
| guesswhat[m] | Yes. | 15:54 |
| kevko | guesswhat[m]: maybe some examples help you ? -> https://opendev.org/openstack/glance/src/branch/master/doc/source/configuration/configuring.rst | 15:57 |
| guesswhat[m] | Thanks, saw that.. back to the single node ceph I guess. Its 5G of ram wasted.. | 16:04 |
| guesswhat[m] | Maybe thats why its not implemented in kolla, | 16:05 |
| opendevreview | Merged openstack/kolla-ansible master: zun: Use assert on checks for readability https://review.opendev.org/c/openstack/kolla-ansible/+/869589 | 16:19 |
| opendevreview | Merged openstack/kolla-ansible master: loadbalancer: Use assert on checks for readability https://review.opendev.org/c/openstack/kolla-ansible/+/869594 | 16:19 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support https://review.opendev.org/c/openstack/kolla/+/873549 | 16:52 |
| opendevreview | Michal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support https://review.opendev.org/c/openstack/kolla/+/873549 | 17:16 |
| *** bodgix9 is now known as bodgix | 20:19 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!