Monday, 2023-02-13

mnasiadkamorning07:57
mnasiadkatrivial patches for this week - ebbex's assert usage (instead of fail: module) - https://review.opendev.org/q/topic:assert07:58
*** ralonsoh_ is now known as ralonsoh08:37
guesswhat[m]Can anyone help me with integrating Cinder backend for Glances? Seems that it does not work correctly. In my installation is Cinder enabled, even internalURL endpoint is set in glance-api.conf https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/glance/templates/glance-api.conf.j2#L23, but it fails to https://pastebin.com/raw/mLjx48ue , even if I use  kolla_copy_ca_into_containers: "yes" or if publicURL is09:18
guesswhat[m]certificate is signed with Lets Encrypt. I guess single node and devstacks can benefit from Cinder backend. 09:18
guesswhat[m]Another question is why it is trying to communitace to publicURL endpoint, when internalURL is explicitly set. Ideas? Thanks09:18
kevkoguesswhat[m]: why are u copy ca into container if you are using letsencrypt ... you need to copy ca into container if it is your ca ...09:24
kevkoguesswhat[m]: debug with cul 09:25
kevkocurl09:25
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add support for CentOS Stream 9  https://review.opendev.org/c/openstack/kolla/+/86990609:49
opendevreviewMerged openstack/kolla stable/xena: CI: store only templated out Dockerfiles in work_dir  https://review.opendev.org/c/openstack/kolla/+/87300510:19
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support  https://review.opendev.org/c/openstack/kolla/+/87354910:27
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add support for CentOS Stream 9  https://review.opendev.org/c/openstack/kolla/+/86990610:33
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add support for CentOS Stream 9  https://review.opendev.org/c/openstack/kolla/+/86990610:42
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support  https://review.opendev.org/c/openstack/kolla/+/87354910:42
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add support for CentOS Stream 9  https://review.opendev.org/c/openstack/kolla/+/86990610:44
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support  https://review.opendev.org/c/openstack/kolla/+/87354910:48
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support  https://review.opendev.org/c/openstack/kolla/+/87354910:48
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support  https://review.opendev.org/c/openstack/kolla/+/87354910:49
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support  https://review.opendev.org/c/openstack/kolla/+/87354910:50
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support  https://review.opendev.org/c/openstack/kolla/+/87354910:51
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support  https://review.opendev.org/c/openstack/kolla/+/87354910:52
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support  https://review.opendev.org/c/openstack/kolla/+/87354910:52
guesswhat[m]kevko:  I tried also selfsigned certificate generated via kolla-ansible, same errror..11:03
kevkoguesswhat[m]: post here pastebin with your configuration related to tls and also directory structure of your kolla config via pastebin or something similar11:04
guesswhat[m]problem would be in some cacert, not sure which component tho, probably glance I guess, or maybe even in glance python client11:04
guesswhat[m]I tested two cases: 1. haproxy.pem generated from ACME and 2. haproxy.pem default one ( kolla_enable_tls_external: "yes" ) generated via kolla-ansible11:06
guesswhat[m]Both cases fails to TLS validation error, but only for glance component, other components/services are running correctly11:06
guesswhat[m]Can be reproduced with default Cinder & Glance configuration. I changed https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/glance/templates/glance-api.conf.j2#L66 to cinder and deleted file and http backends https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/glance/templates/glance-api.conf.j2#L16 11:08
guesswhat[m]Otherwise Glance fallbacks to working backends, in this case It would be http or file backend11:08
guesswhat[m]Its weird, because Glance is calling public URL of Cinder, I believe this is not necessary11:09
guesswhat[m]kevko: its almost default installation with cinder lvm backend, but thats not related I guess11:15
kevkoguesswhat[m]: and where did you read that glance support cinder as backend ? :D 11:20
kevkoguesswhat[m]: ah yes, you are right 11:20
kevkoguesswhat[m]: #         * file11:21
kevko#         * http11:21
kevko#         * swift11:21
kevko#         * rbd11:21
kevko#         * cinder11:21
kevko#         * vmware11:21
kevko#         * s311:21
kevkoguesswhat[m]: firstly you should leave default_backends as is i think because it is only default backends ... enabled backends is configured by glance_backends11:24
kevkoguesswhat[m]: ok now i got a point ...you set default backend for cinder and deleted configured backends just to verify cinder will be one and only glance will try to contact and you found an issue with tls ...right ? 11:26
guesswhat[m]yes11:26
guesswhat[m]this way its easily reproducible11:26
kevkoguesswhat[m]: ok, i got it ...and glance is contacting cinder via public which is already under tls ..right ? 11:27
guesswhat[m]yes11:27
guesswhat[m]it shouldnt, cuz there is cinder_catalog_info = volume:cinder:internalURL11:28
kevkoguesswhat[m]: ok, so when you exec into glance_api container and touch cinder api with curl standalone ..you will get same results right ? 11:28
guesswhat[m]kevko:  thats working11:36
kevkocould you paste entire log error ? 11:37
guesswhat[m]other services are respecting cacert or/and custom certificates11:37
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support  https://review.opendev.org/c/openstack/kolla/+/87354911:39
guesswhat[m]kevko: https://pastebin.com/raw/fc6qgaBh11:41
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support  https://review.opendev.org/c/openstack/kolla/+/87354911:41
kevkoguesswhat[m]: what images for kolla do you use ? 11:42
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support  https://review.opendev.org/c/openstack/kolla/+/87354911:43
guesswhat[m]kevko:  zed/jammy11:44
opendevreviewMark Goddard proposed openstack/kolla-ansible master: Put etcd behind HTTP loadbalancer  https://review.opendev.org/c/openstack/kolla-ansible/+/85239111:45
guesswhat[m]`docker exec -it glance_api curl -IL -H "Host: openstack.host.cloud"  https://172.17.0.12` working correctly, thats public interface11:46
kevkoguesswhat[m]: for me it looks like configuration of openstack glance-> cinder connection is somehow broken 11:47
kevkoguesswhat[m]: try it without -H "Host: ....."11:47
kevkoguesswhat[m]: why do you sending Host header if it is resolvable ? 11:48
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: CI: Drop failing Zun jobs  https://review.opendev.org/c/openstack/kolla-ansible/+/87355111:50
kevkoguesswhat[m]: get token from keystone .... openstack token issue ... and run this in glance-api curl -v -H "X-Auth-Token: YOUR_TOKEN" https://openstack.public.cloud:8776/v3/11:50
kevkoguesswhat[m]: it should fail i think 11:50
guesswhat[m]Got 200 response code without -H header11:54
guesswhat[m]AFAIK its not neeeded to test it with different header, curl would report problem with SSL11:55
guesswhat[m]Maybe cinder_ca_certificates_file ( https://docs.openstack.org/glance/latest/configuration/configuring.html ) have to be specified11:58
guesswhat[m]But that does not explain why its calling public URL endpoint12:00
guesswhat[m]even cinder_api_insecure = true in [DEFAULT] table in glance-api.conf does not help12:01
guesswhat[m]Its somehow broken12:01
opendevreviewIvan Halomi proposed openstack/kolla-ansible master: Splitting docker_worker into container_worker  https://review.opendev.org/c/openstack/kolla-ansible/+/86341412:03
kevkoguesswhat[m]: try to setup this cinder_ca_certificates_file=/path/of/ca12:15
kevkoguesswhat[m]: https://docs.openstack.org/glance/latest/configuration/configuring.html#configuring-the-cinder-storage-backend12:17
kevkoguesswhat[m]: or try to play with cinder_endpoint_template or cinder_catalog_info12:18
kevkoguesswhat[m]: or, do you tried to set cinder_api_insecure = false ? 12:20
*** priteau_ is now known as priteau12:25
opendevreviewIvan Halomi proposed openstack/kolla-ansible master: Splitting docker_worker into container_worker  https://review.opendev.org/c/openstack/kolla-ansible/+/86341412:34
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add support for CentOS Stream 9  https://review.opendev.org/c/openstack/kolla/+/86990612:35
opendevreviewIvan Halomi proposed openstack/kolla-ansible master: Splitting docker_worker into container_worker  https://review.opendev.org/c/openstack/kolla-ansible/+/86341412:41
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support  https://review.opendev.org/c/openstack/kolla/+/87354912:43
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support  https://review.opendev.org/c/openstack/kolla/+/87354912:48
mnasiadkakevko, bbezak: https://review.opendev.org/c/openstack/ansible-collection-kolla/+/872626 - if we could merge this, we could pin Zun jobs to docker 22 instead of removing them12:57
opendevreviewMichal Nasiadka proposed openstack/ansible-collection-kolla master: docker: Add docker_systemd_reload functionality  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/87129312:57
kevkomnasiadka: looks ok, i will check after zuul12:59
kevko*chec again12:59
opendevreviewMichal Nasiadka proposed openstack/ansible-collection-kolla master: Add support for pinning docker package  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/87262613:02
mnasiadkaok, rebased that one as well13:02
opendevreviewVerification of a change to openstack/kayobe master failed: CI: Use nested nodepools for jobs that are spawning vms  https://review.opendev.org/c/openstack/kayobe/+/87264713:03
guesswhat[m]@kevko: cinder_api_insecure = false/true same error... , seems its broken somehow13:03
kevkomnasiadka: would you recommend ovn for production ? Because we still using openvswitch for customers ..but I am thinking about ovn-based deployment 13:28
kevkomnasiadka: for the new one 13:30
opendevreviewMark Goddard proposed openstack/kolla-ansible master: Fix deploy/genconfig in check mode  https://review.opendev.org/c/openstack/kolla-ansible/+/87001414:03
kevkoguesswhat[m]: well, investigate code :P 14:12
guesswhat[m]kevko: got some progress https://pastebin.com/raw/dhtAq0hB , now getting 2023-02-13 15:24:28.578 667 ERROR glance.common.wsgi os_brick.exception.VolumeDeviceNotFound: Volume device not found at .14:26
kevkoguesswhat[m]: When i will have some time ..i will try...but we are using glance with rbd backend14:27
opendevreviewMatt Crees proposed openstack/kolla-ansible master: Use the upgraded image to run Nova upgrade checks  https://review.opendev.org/c/openstack/kolla-ansible/+/87127914:51
guesswhat[m]kevko:  after adding cinder_volume_type = lvm-1 ( default from kolla ), glance is not starting to  ERROR: 'NoneType' object has no attribute 'user_id' I guess glance-api.conf  must contain cinder connection 14:55
kevkoguesswhat[m]: i am not sure if i can help you because i never tried to setup before 14:57
opendevreviewMatt Crees proposed openstack/kolla-ansible master: Use the upgraded image to run Nova upgrade checks  https://review.opendev.org/c/openstack/kolla-ansible/+/87127915:02
opendevreviewMatt Crees proposed openstack/kolla-ansible master: Use the upgraded image to run Nova upgrade checks  https://review.opendev.org/c/openstack/kolla-ansible/+/87127915:16
opendevreviewMerged openstack/ansible-collection-kolla master: Add support for pinning docker package  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/87262615:26
guesswhat[m]@kevko heres my config, seems that cinder connection string must be defined https://pastebin.com/raw/7L0az7pb15:35
guesswhat[m]Still getting glance.common.wsgi os_brick.exception.VolumeDeviceNotFound: Volume device not found at .15:35
guesswhat[m]DEFAULT should be default volume type15:35
kevkoguesswhat[m]: haha, yeah that might have attacked me, sorry ... enabled backens have to be defined as it is in cinder for example 15:36
guesswhat[m]Its still does not work :/ 15:36
guesswhat[m]Maybe You will find some dragons there15:36
guesswhat[m]Its pitty that it does not work for kollla ansible out of the box15:38
kevkoguesswhat[m]: well, I think we don't have scenario for this ? w8 a minute i will check ... Most people using openstack together with ceph 15:49
opendevreviewIvan Halomi proposed openstack/kolla-ansible master: Splitting docker_worker into container_worker  https://review.opendev.org/c/openstack/kolla-ansible/+/86341415:51
guesswhat[m]Small scale, single node deployment based on LVM, NFS15:52
kevkoguesswhat[m]: specify cinder_volume_type = TYPE_OF_VOLUME into cinder block15:52
opendevreviewMartin Hiner proposed openstack/kolla-ansible master: Refactor DockerWorker into ContainerWorker  https://review.opendev.org/c/openstack/kolla-ansible/+/86341415:53
guesswhat[m]Default Glance backend ( file ) is consuming space on primary host disk15:53
kevkoguesswhat[m]: did you try to define new volume type and not use __DEFAULT__ ? 15:54
guesswhat[m]While Cinder ( with LVM ) forces You to use additional disk15:54
guesswhat[m]Yes. 15:54
kevkoguesswhat[m]: maybe some examples help you ? -> https://opendev.org/openstack/glance/src/branch/master/doc/source/configuration/configuring.rst15:57
guesswhat[m]Thanks, saw that.. back to the single node ceph I guess. Its 5G of ram wasted..16:04
guesswhat[m]Maybe thats why its not implemented in kolla,16:05
opendevreviewMerged openstack/kolla-ansible master: zun: Use assert on checks for readability  https://review.opendev.org/c/openstack/kolla-ansible/+/86958916:19
opendevreviewMerged openstack/kolla-ansible master: loadbalancer: Use assert on checks for readability  https://review.opendev.org/c/openstack/kolla-ansible/+/86959416:19
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support  https://review.opendev.org/c/openstack/kolla/+/87354916:52
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: Add RockyLinux 9 support  https://review.opendev.org/c/openstack/kolla/+/87354917:16
*** bodgix9 is now known as bodgix20:19

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!