| opendevreview | Elvis Ratzlaff proposed openstack/kolla-ansible stable/2023.2: cast port string to number in kolla_url https://review.opendev.org/c/openstack/kolla-ansible/+/910498 | 02:26 |
|---|---|---|
| mnasiadka | samcat116: that's bcrypt limitation and it was always there | 07:14 |
| mnasiadka | frickler, kevko: seems redis got broken in cinder - https://a8c5389f08768dc97af2-76b70ec5cd2ca2755a9230dd387d9dfa.ssl.cf5.rackcdn.com/911591/1/check/kolla-ansible-rocky9-cephadm/3d6ec78/primary/logs/kolla/cinder/cinder-api-error.txt | 07:19 |
| mnasiadka | and that's affecting all multinode jobs | 07:19 |
| mnasiadka | argh | 07:19 |
| mnasiadka | seems that's tooz 6.0.0 related | 07:24 |
| mnasiadka | but that was last week | 07:24 |
| mnasiadka | so a bit weird | 07:24 |
| frickler | maybe it took a bit for new images to be built? but I also wonder why this isn't failing upstream | 07:44 |
| frickler | likely nobody testing with redis backend | 07:44 |
| frickler | I'll still copy this to cinder and oslo channels, maybe they have an idea | 07:45 |
| frickler | seems indeed to be a bug in tooz that will require a patch and a new release. not sure whether we should just wait a bit or block 6.0.0 locally | 08:23 |
| frickler | https://review.opendev.org/c/openstack/tooz/+/911648 | 08:46 |
| SvenKieske | thanks frickler | 08:49 |
| opendevreview | Darin Chakalov proposed openstack/kolla-ansible master: Enable TLS support for Manila https://review.opendev.org/c/openstack/kolla-ansible/+/905010 | 09:14 |
| mnasiadka | frickler: thanks | 09:23 |
| mnasiadka | frickler: I'll raise a patch to disable coordination in cephadm jobs for now | 10:12 |
| kevko | morning ! | 10:18 |
| kevko | mnasiadka: can u send some review to check the redis ? (i don't know how to get info about revied id from log url only :( ) | 10:20 |
| SvenKieske | kevko: I guess frickler posted the fix url above, here it is again: https://review.opendev.org/c/openstack/tooz/+/911648 | 10:22 |
| kevko | SvenKieske: thanks | 10:22 |
| kevko | hmm, for this situations I have downstream mechanism to patch python files :( | 10:23 |
| opendevreview | Matt Crees proposed openstack/kayobe master: Support saving baremetal compute inspection data https://review.opendev.org/c/openstack/kayobe/+/911688 | 10:44 |
| opendevreview | Matt Crees proposed openstack/kayobe master: Support saving baremetal compute inspection data https://review.opendev.org/c/openstack/kayobe/+/911688 | 10:45 |
| mnasiadka | kevko: checking things like this is complicated in CI... | 10:48 |
| opendevreview | Ivan Halomi proposed openstack/kolla-ansible master: Refactor of kolla_container_facts https://review.opendev.org/c/openstack/kolla-ansible/+/911417 | 10:57 |
| opendevreview | Ivan Halomi proposed openstack/kolla-ansible master: Move actions to kolla_container_facts https://review.opendev.org/c/openstack/kolla-ansible/+/911505 | 11:17 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: Support returning <network>_interface_address from kolla_address https://review.opendev.org/c/openstack/kolla-ansible/+/911695 | 11:21 |
| opendevreview | Jakub Darmach proposed openstack/kayobe master: Use new collections in Kayobe https://review.opendev.org/c/openstack/kayobe/+/910742 | 11:21 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Disable redis in cephadm jobs https://review.opendev.org/c/openstack/kolla-ansible/+/911697 | 11:50 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix images pull in ovs-dpdk role https://review.opendev.org/c/openstack/kolla-ansible/+/899626 | 11:56 |
| kevko | mnasiadka: what do you mean it's complicated ? | 12:06 |
| kevko | mnasiadka: when I have some bug in images (most of the time it's one function or one line etc. ...) and fix is in gerrit ...i take tha patch from gerrit and add to kolla as patch ... test itself is that images are working after patch ..tests are passing ... | 12:08 |
| mnasiadka | kevko: as in overriding the package list, good thing is tooz is something we directly install - not a dependency | 12:08 |
| kevko | mnasiadka: if it's more and it's service ..i am using git install from forked git ... | 12:08 |
| opendevreview | Will Szumski proposed openstack/kayobe master: Adds seed service destroy https://review.opendev.org/c/openstack/kayobe/+/911702 | 12:10 |
| mnasiadka | kevko: yes, I was thinking about proper depends-on support, which might be complicated ;-) | 12:12 |
| kevko | mnasiadka: i am thinking about this some weeks already ..like some kolish way ...and have something written on more fields .... | 12:15 |
| kevko | mnasiadka: point is that if we are providing kolla as software for building images ...we should provide real way how to "modify" images ... patch the code, specific version, proper dependency resolution ..etc etc | 12:16 |
| mnasiadka | It doesn't seem that anybody except you needs that feature, nobody else has brought that up in the past - so if you want to propose something - I'm fine with that, but if you're able to manage that downstream - I'm also fine with that | 12:18 |
| mnasiadka | hmm, actually we could pin tooz to older version | 12:20 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: openstack-base: Pin tooz to 5.0.0 https://review.opendev.org/c/openstack/kolla/+/911703 | 12:21 |
| kevko | mnasiadka: nobody ? can u propose this discussion on PTG - I am curious | 12:23 |
| kevko | mnasiadka: Perhaps we have demanding clients, and we're doing it well :) | 12:25 |
| mnasiadka | kevko: https://etherpad.opendev.org/p/kolla-dalmatian-ptg - feel free to propose it here | 12:25 |
| kevko | mnasiadka: yeah, downstream is OK for now ...but I really like when I can send something to upstream :) | 12:25 |
| mnasiadka | I still don't think patching or using forked git repos should be easy - we've made overriding u-c easy already ;) | 12:34 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: WIP: Fix arm builds https://review.opendev.org/c/openstack/kolla/+/909915 | 12:38 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: WIP: Switch to Ubuntu 24.04 https://review.opendev.org/c/openstack/kolla/+/907589 | 12:41 |
| kevko | mnasiadka: why not ? it is ...not really easy ..but it's not so hard | 12:43 |
| kevko | https://review.opendev.org/c/openstack/kolla-ansible/+/899626 << mnasiadka can u please approve after zuul verify ? i will create backports then | 12:45 |
| kevko | or any cores | 12:50 |
| opendevreview | Ivan Halomi proposed openstack/ansible-collection-kolla master: Bump min. version of docker-py https://review.opendev.org/c/openstack/ansible-collection-kolla/+/910751 | 13:01 |
| opendevreview | Matúš Jenča proposed openstack/kolla-ansible master: Implement TLS for Redis https://review.opendev.org/c/openstack/kolla-ansible/+/911712 | 13:24 |
| opendevreview | Matúš Jenča proposed openstack/kolla-ansible master: Implement TLS for Redis https://review.opendev.org/c/openstack/kolla-ansible/+/909188 | 13:29 |
| kevko | :( glance probably broken ? | 13:39 |
| kevko | hh, funny is that i am fixing glance (because some CADF work on it ...) and I really didn't understand why I am getting error - and that's because I regenerate images with my change ...but downloaded new glance master version where it's broken :D | 13:42 |
| kevko | 2024-03-06T21:59:08.752716897Z ERROR: expected str, bytes or os.PathLike object, not NoneType | 13:42 |
| kevko | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_475/911591/1/check/kolla-ansible-centos9s/4755314/primary/logs/container_logs/glance_api.txt | 13:43 |
| mnasiadka | lol | 13:43 |
| mnasiadka | is there anything that is not broken now? :D | 13:43 |
| mnasiadka | how are those projects testing their stuff... | 13:43 |
| kevko | aaaaaaa... this is happening to me always when i have really good work drive :D | 13:45 |
| kevko | what can I do ... i am going to fix glance :D | 13:45 |
| SvenKieske | is there already the pin for tooz somewhere? if not you could provide that, that should fix glance, no? | 13:46 |
| mnasiadka | I don't think glance is affected by tooz | 13:47 |
| SvenKieske | the mail is also out that tooz 6.0.0 is broken and 6.0.1 will be soon(TM) released | 13:47 |
| SvenKieske | mnasiadka: ah sorry then; I misunderstood that then. | 13:47 |
| SvenKieske | and I see there is the pin already: https://review.opendev.org/c/openstack/kolla/+/911703 | 13:48 |
| kevko | And i really love bugs like this :D | 13:50 |
| kevko | (glance-api)[root@controller0 /]# glance-api | 13:50 |
| kevko | ERROR: expected str, bytes or os.PathLike object, not NoneType | 13:50 |
| kevko | no traceback ..nothing ... | 13:50 |
| SvenKieske | the usual helpful python output when interacting with openstack ;) | 13:50 |
| SvenKieske | tbf half the time you get a traceback (which sometimes tells you where the problem is) | 13:51 |
| kevko | The only thing left for me is to open Vim and put messages like print("k**va") there and go by the interval method. :D | 13:51 |
| SvenKieske | print() debugging is best debugging | 13:51 |
| opendevreview | Merged openstack/kolla-ansible master: Fix typo in release note https://review.opendev.org/c/openstack/kolla-ansible/+/910703 | 13:52 |
| mnasiadka | kevko: on the other front, I love how keystone produces craploads of python traceback because Ansible module creating a domain is checking first if the domain exists | 13:53 |
| mnasiadka | :) | 13:53 |
| kevko | SvenKieske: yep, especially with "k**va" :D | 13:55 |
| mnasiadka | so, tooz, glance, keystone, anything more? :D | 13:55 |
| kevko | I don't know ..but it looks like all returned from vacation or what :D | 13:57 |
| kevko | or left to some | 13:57 |
| kevko | another funny thing is that it looks like it is a bug in function i was trying to fix ... :D | 14:01 |
| kevko | mnasiadka: SvenKieske: do you know what fixed glance ? :D :D :D | 14:36 |
| kevko | cp -av /etc/glance/glance-api-paste.ini /etc/glance/glance-cache-paste.ini | 14:36 |
| kevko | haha | 14:36 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Fix aarch64 builds https://review.opendev.org/c/openstack/kolla/+/909915 | 14:37 |
| SvenKieske | mhm, what's in glance-cache-paste.ini? do we patch that? or is this upstreams bug? | 14:37 |
| kevko | SvenKieske: glance code has very ugly code (which I was fixing ) ...and they changed upstream code ...so that ugly code start to be buggy :D | 14:38 |
| kevko | SvenKieske: https://github.com/openstack/glance/commit/9c7820740a5ceaf31336da39624226b9e398e84d | 14:40 |
| kevko | SvenKieske: this line is problematic now https://github.com/openstack/glance/blame/master/glance/common/config.py#L660 | 14:41 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Fix aarch64 builds https://review.opendev.org/c/openstack/kolla/+/909915 | 14:41 |
| SvenKieske | I don't understand, it's first reading glance-api and then glance-cache config, seems reasonable? | 14:45 |
| SvenKieske | we template both files in glance role, so maybe there is a bug? | 14:46 |
| kevko | There is a function in common config I think ( not near PC right now) which is incorrectly reading config file and try to find bad file which is concentrated from glance service name and api paste suffix | 14:47 |
| kevko | find_conf_file or something like that ..will write when I will be back | 14:48 |
| kevko | SvenKieske: we can fix by kolla ... But it's a bug | 14:48 |
| kevko | In glance | 14:48 |
| SvenKieske | kevko: there is also this patch: https://review.opendev.org/c/openstack/glance/+/681787 (yes old, but actually active recently) | 14:50 |
| SvenKieske | and no tests :D | 14:51 |
| frickler | glance did some cache changes recently, like a new default backend, that might well be related https://review.opendev.org/q/topic:%22centralized-cache-db%22 | 14:55 |
| SvenKieske | ah thanks for the pointer, did see that one just yet (there's a lot of activity in glance) | 14:58 |
| kevko | SvenKieske: tldr -> this line is not working as described above the line https://github.com/openstack/glance/blob/519ce0b94dcc6dad229fbe08d8d21ccdefb6c99e/etc/glance-api.conf#L5989 << | 15:00 |
| SvenKieske | frickler: the change kevko mentioned above is actually from that caching backend rework, see https://review.opendev.org/c/openstack/glance/+/899872/18/glance/common/config.py | 15:05 |
| kevko | this function https://github.com/openstack/glance/blob/519ce0b94dcc6dad229fbe08d8d21ccdefb6c99e/glance/common/config.py#L694-L705 | 15:05 |
| SvenKieske | I didn't see the topic because I only looked at the changes via the github links. | 15:05 |
| mnasiadka | kevko: do we need to change the file name? ;-) | 15:06 |
| SvenKieske | I think we should adjust our templates to what upstream does, no? first using the api-paste.ini and then overwriting it - where it makes sense - with stuff from api-cache.ini? | 15:09 |
| SvenKieske | so not really overwriting, glance does that already. I guess you know what I mean.. | 15:12 |
| opendevreview | Kevin Tindall proposed openstack/kolla-ansible master: Add TLS proxy for novncproxy https://review.opendev.org/c/openstack/kolla-ansible/+/911141 | 15:25 |
| kevko | mnasiadka: There is also some another issue ...let me fix glance probably ..and we will see | 15:26 |
| opendevreview | Merged openstack/kayobe-config-dev stable/2023.1: Use dummy1 as bridge port instead of eth1 https://review.opendev.org/c/openstack/kayobe-config-dev/+/907940 | 15:37 |
| opendevreview | Merged openstack/kayobe-config-dev stable/2023.2: Use dummy1 as bridge port instead of eth1 https://review.opendev.org/c/openstack/kayobe-config-dev/+/907939 | 15:52 |
| opendevreview | Will Szumski proposed openstack/kayobe master: Add podman support https://review.opendev.org/c/openstack/kayobe/+/909686 | 15:55 |
| samcat116 | Is there anyway to configure the PowerDNS4 backed with designate in K-A? I know it only officially supports bind9 and infoblox and that i'll need to do all my own config overrides, but just need to make sure the right components are getting deployed | 16:04 |
| opendevreview | Verification of a change to openstack/kayobe master failed: Fix: configure etc-hosts for overcloud group https://review.opendev.org/c/openstack/kayobe/+/907306 | 16:05 |
| SvenKieske | samcat116: it could work, afaik it's untested. be aware that designate only supports AXFR as a transfer option, so you need to configure that in powerdns accordingly for the designate services. | 16:07 |
| samcat116 | yeah designate has pretty good docs on actually configuring that, just concerned about kolla deploying the right designate images and copying over config | 16:08 |
| samcat116 | also is infoblox tested? I was never able to get that to work as Infoblox requires the designate mdns container to bind to port 53 and k-a doesn't run the container as root so that always seemed to fail | 16:09 |
| SvenKieske | it was agreed that this is working as intended, it is documented how to make your setup more insecure so infoblox works: https://review.opendev.org/c/openstack/kolla/+/886636 https://docs.openstack.org/kolla-ansible/latest/reference/networking/designate-guide.html#infoblox-backend | 16:14 |
| samcat116 | Ah, missed that box. Thank you! | 16:15 |
| SvenKieske | we didn't want to make everyone more insecure just so a closed source backend works out of the box :) | 16:16 |
| SvenKieske | so regarding tests: afaik we don't - rather can't - test infoblox because it's proprietary | 16:16 |
| SvenKieske | but I really don't know infoblox, why would you want to use a closed source dns provider if no external forces are forcing you? genuine question :) | 16:17 |
| kevko | fg | 16:20 |
| samcat116 | External forces :) | 16:23 |
| samcat116 | but thats why I'm pivoting to the PDNS4 backend now | 16:23 |
| opendevreview | Sven Kieske proposed openstack/kolla-ansible master: [doc] document --limit limitations https://review.opendev.org/c/openstack/kolla-ansible/+/911082 | 16:23 |
| kevko | SvenKieske: okay, fixed | 16:25 |
| kevko | SvenKieske, mnasiadka: we need also temporary fix for glance ...or configure cache shared | 16:25 |
| kevko | with current config for glance it's trying to concentrate url for sqlite db None + cache.db | 16:26 |
| kevko | which is error | 16:26 |
| kevko | frickler: you were right with cache | 16:28 |
| spatel | hey folks, I have push neutron policy.yaml in 2023.1 release which is not working but same time I push that policy to zed which works. | 16:43 |
| spatel | Anything changed in 2023.1 release to handle policy. I didn't find anything unusual in docs. | 16:44 |
| kevko | spatel: check the log | 16:45 |
| spatel | looking.. but does policy.yaml required to restart neutron server? | 16:45 |
| spatel | I am seeing this thing - 2024-03-06 20:42:33.756 25 WARNING oslo_policy.policy [req-c576d538-5b91-4a22-ba99-2867af82100b req-902a6aa4-71b5-4090-b665-33d6e4f353eb d8dc1d082d5d4f27a5c178932b6458b5 275e3127e26143049cbd54babba9dd5a - - default default] Policies ['delete_floatingip'] reference a rule that is not defined. | 16:47 |
| spatel | But no errors anywhere | 16:48 |
| kevko | Policies ['delete_floatingip'] reference a rule that is not defined. check this | 16:49 |
| kevko | but it is only warning | 16:49 |
| spatel | hmm I found very strange thing.. policy.yaml only available on controller01 but not on 02 and 03 | 16:51 |
| spatel | let me re-run playbook | 16:51 |
| -opendevstatus- NOTICE: Jobs that fail due to being unable to resolve mirror.dfw.rackspace.opendev.org can be rechecked. This error was an unexpected side effect of some nodepool configuration changes which have been reverted. | 16:54 | |
| mnasiadka | kevko: if there's an easy fix - then sure | 16:57 |
| SvenKieske | at least it's a warning and not a debug message :) | 16:58 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Fix aarch64 builds https://review.opendev.org/c/openstack/kolla/+/909915 | 17:00 |
| spatel | kevko it works :) look like my playbook didn't work first time and just push policy.yaml to single node | 17:06 |
| kevko | mnasiadka: let me send some | 17:20 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Fix aarch64 builds https://review.opendev.org/c/openstack/kolla/+/909915 | 17:21 |
| mnasiadka | kevko: tooz pin is not critical (only non-voting jobs are failing) - so I think glance fix would improve things a lot | 17:22 |
| mnasiadka | (unless there are some other breakages in other services we haven't seen yet) | 17:22 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Remove setuptools pin https://review.opendev.org/c/openstack/kolla/+/901828 | 17:56 |
| opendevreview | Will Szumski proposed openstack/kayobe master: Fix issue removing docker volumes https://review.opendev.org/c/openstack/kayobe/+/909594 | 18:05 |
| opendevreview | Will Szumski proposed openstack/kayobe master: Support configuring persistent systemd journal https://review.opendev.org/c/openstack/kayobe/+/909685 | 18:05 |
| opendevreview | Will Szumski proposed openstack/kayobe master: Add podman support https://review.opendev.org/c/openstack/kayobe/+/909686 | 18:05 |
| opendevreview | Will Szumski proposed openstack/kayobe master: Support configuring persistent systemd journal https://review.opendev.org/c/openstack/kayobe/+/909685 | 18:06 |
| opendevreview | Will Szumski proposed openstack/kayobe master: Add podman support https://review.opendev.org/c/openstack/kayobe/+/909686 | 18:07 |
| mnasiadka | kevko: it’s interesting that devstack does not fail on glance | 18:08 |
| jovial | I've started seeing: "'<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f749bc58460>: Failed to establish a new connection: [Errno -2] Name or service not known')': /wheel/rocky-9.3-x86_64/pip/" in kayobe jobs. Is that an infra issue? | 18:10 |
| jovial | e.g here: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f41/909686/26/check/kayobe-seed-upgrade-rocky9/f415651/primary/ansible/install-pre-upgrade | 18:10 |
| kevko | mnasiadka: https://review.opendev.org/c/openstack/glance/+/911955 | 18:15 |
| kevko | jovial: can u send a review url ? | 18:31 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix glance not running https://review.opendev.org/c/openstack/kolla-ansible/+/912025 | 21:09 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fixing glance not running after upstream cache changes. https://review.opendev.org/c/openstack/kolla-ansible/+/912025 | 23:22 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix glance not running after upstream cache changes. https://review.opendev.org/c/openstack/kolla-ansible/+/912025 | 23:23 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix glance not running after upstream cache changes. https://review.opendev.org/c/openstack/kolla-ansible/+/912030 | 23:23 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix glance not running after upstream cache changes https://review.opendev.org/c/openstack/kolla-ansible/+/912030 | 23:24 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix creation of ovs bridges https://review.opendev.org/c/openstack/kolla-ansible/+/911591 | 23:26 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix glance not running after upstream cache changes https://review.opendev.org/c/openstack/kolla-ansible/+/912030 | 23:31 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix creation of ovs bridges https://review.opendev.org/c/openstack/kolla-ansible/+/911591 | 23:31 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!