| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Refactor external ceph https://review.opendev.org/c/openstack/kolla-ansible/+/907166 | 00:22 |
|---|---|---|
| opendevreview | Kien Nguyen proposed openstack/kolla-ansible master: Support Fluentd per-host custom config https://review.opendev.org/c/openstack/kolla-ansible/+/924382 | 03:59 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Refactor external ceph https://review.opendev.org/c/openstack/kolla-ansible/+/907166 | 05:56 |
| kevko | frickler: can i somehow get access to nodes ? as i am unable to check why cephdm jobs failing | 06:01 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Add support for docker_image_name_prefix https://review.opendev.org/c/openstack/kolla-ansible/+/924450 | 06:10 |
| mnasiadka | kevko: libvirt.libvirtError: Secret not found: no secret with matching uuid '25a6a64e-9810-4c19-b92e-541c01d1ae86' | 06:35 |
| opendevreview | Roman Krček proposed openstack/kolla-ansible master: Rewrite kolla-ansible CLI to python https://review.opendev.org/c/openstack/kolla-ansible/+/923110 | 06:58 |
| opendevreview | Roman Krček proposed openstack/kolla-ansible master: Rewrite kolla-ansible CLI to python https://review.opendev.org/c/openstack/kolla-ansible/+/923110 | 07:11 |
| kevko | mnasiadka: hmmm...that's weird | 07:20 |
| mnasiadka | kevko: yeah, that file is in secrets, so maybe some rights are wrong or something else | 07:20 |
| kevko | mnasiadka: can u send me a log ? | 07:21 |
| kevko | mnasiadka: but thank you very much ... this is very weird | 07:24 |
| mnasiadka | it's in nova-libvirt log on secondary01 | 07:25 |
| mnasiadka | (I closed the tab already) | 07:25 |
| sylvr | good morning! | 07:25 |
| kevko | morning | 07:25 |
| frickler | kevko: if you want a hold: ask in #opendev, state the patch in question, the jobname, some short desc of what you want to look at and your public ssh key to give you access with | 07:29 |
| kevko | frickler: thank you, i will try to find out what is going on as mnasiadka helped me a lot i would say ...but it's still weird ..as i didn't touch the code around secrets ... | 07:33 |
| kevko | or better said ..i touched ..but everything is there ..as i only refactored | 07:35 |
| kevko | frickler: mnasiadka: btw, i have fresh caracal stack deployed ...and after some time my rabbitmq is unhealthy ..and i can see bunch of errors in rabbitmq log https://paste.openstack.org/show/b8Cm4MWuHGGiTum5vaLz/ | 07:39 |
| kevko | any ideas ? | 07:40 |
| kevko | quorum queues turned on | 07:40 |
| mnasiadka | kevko: seems it's only heat? | 07:40 |
| mnasiadka | missing heartbeats? | 07:40 |
| mnasiadka | if that's only one service with such errors, then you can't really blame RMQ ;-) | 07:41 |
| kevko | mnasiadka: but my rabbitmq is slow, unresponsive ...and even docker healthchecks not working | 07:41 |
| mnasiadka | stop heat and check if it comes back to normal - it might be some connection flood that is making your RMQ not happy? | 07:42 |
| mnasiadka | Or it's an RMQ bug | 07:42 |
| mnasiadka | I haven't used Caracal longer than a CI run - so can't tell | 07:42 |
| kevko | mnasiadka: for me it looks like rmq bug ... | 07:42 |
| kevko | mnasiadka: it's fine for hours ...then rabbitmq all three nodes are broken | 07:43 |
| kevko | mnasiadka: why do you think it's heat ? ...i think logs around heat-engine is ok ... i meant the error logs | 07:44 |
| mnasiadka | are you running rmq 3.13.4 or earlier? | 07:44 |
| kevko | mnasiadka: debian images caracal stable | 07:45 |
| kevko | mnasiadka: (rabbitmq)[root@controller0 /]# dpkg -l | egrep 'rabbitmq-server|erlang-base' | 07:45 |
| kevko | ii erlang-base 1:26.2.5.2-1rmq1ppa1~ubuntu22.04.1 amd64 Erlang/OTP virtual machine and base applications | 07:45 |
| kevko | ii rabbitmq-server 3.13.4-1 all Multi-protocol messaging and streaming broker | 07:45 |
| mnasiadka | so that's newest | 07:45 |
| kevko | i was thinking if it wouldn't be some repo issue ... | 07:46 |
| kevko | https://www.rabbitmq.com/docs/install-debian saying bookworm -> bullseye repo ...but ppalaunchpadcontent has only jammy | 07:50 |
| mnasiadka | kevko: or open a discussion in the rabbitmq GH repo - they are usually helpful, just don't use the word OpenStack multiple times ;-) | 07:52 |
| kevko | mnasiadka: haha :D :D | 07:53 |
| mnasiadka | https://github.com/rabbitmq/rabbitmq-server/discussions | 07:53 |
| kevko | mnasiadka: this looks promising https://github.com/rabbitmq/rabbitmq-server/discussions/11712 | 08:02 |
| kevko | mnasiadka: okay, i need to discuss again repos for rmq | 08:17 |
| kevko | mnasiadka: we switched several times from ppa to novemberrain and vice versa | 08:18 |
| kevko | mnasiadka: now rmq is from novemberrain and erlang from ppa ....why ? | 08:18 |
| mnasiadka | erlang is from ppa | 08:18 |
| mnasiadka | it was always from ppa | 08:18 |
| mnasiadka | and they don't provide erlang on any other mirror | 08:18 |
| mnasiadka | at least that's how it was in the past | 08:18 |
| kevko | https://www.rabbitmq.com/docs/install-debian | 08:18 |
| kevko | doc saying something different | 08:19 |
| mnasiadka | even aarch64 erlang? | 08:19 |
| kevko | don't know | 08:19 |
| mnasiadka | well, not different | 08:20 |
| mnasiadka | Debian packages of Erlang from Team RabbitMQ on Launchpad | 08:20 |
| mnasiadka | this is the ppa we're using | 08:20 |
| kevko | but i saw some commit ... probably from you ... that we are using amd64 packages because aarch64 not exist | 08:20 |
| mnasiadka | Frankly speaking it's been long ago, I don't remember all the details :) | 08:20 |
| kevko | ah, that's probably for rmq only | 08:21 |
| kevko | https://review.opendev.org/c/openstack/kolla/+/896616 | 08:21 |
| kevko | mnasiadka: also merged only one +2 :D | 08:22 |
| sylvr | Hello ! when trying to validate my nodes deploy interface return false and indicate that it's missing 'instance_info.image_source'. this seems very odd to me has I did build the image using `kayobe overcloud host image build --force-rebuild` /httpboot on the seed does contain deployment_image.qcow2 and I even hardcoded the correct name in overcloud.yml just to be sure, I'm reading the doc trying to figure out the issue but it seems | 08:27 |
| sylvr | encountering a bug (doesn't seem very likely, it used to work on my previous tests) | 08:27 |
| mnasiadka | kevko: because everything stopped to work :D | 08:31 |
| kevko | mnasiadka: i know ..but still ...i think it's better to install erlang and rabbitmq from novemberrain ... because we already installing amd64 for aarch64 | 08:32 |
| mnasiadka | we need erlang for aarch64 | 08:32 |
| mnasiadka | rabbitmq is in reality arch-independent | 08:32 |
| mnasiadka | and unless you can prove it helps in anything - I don't think we need to move | 08:33 |
| mnasiadka | especially that it's the first repo on the erlang list on their website ;-) | 08:33 |
| kevko | okay, nevermind ... i'am going to install 3.13.3 rabbitmq on my test cloud and will monitor several hours and i will see if fixed ... | 08:33 |
| kevko | mnasiadka: it looks like it helped ...it's also faster | 09:47 |
| kevko | mnasiadka: i will see in couple of hours | 09:47 |
| opendevreview | Matt Crees proposed openstack/kayobe master: Bump stackhpc.linux collection to v1.2.3 https://review.opendev.org/c/openstack/kayobe/+/924480 | 09:57 |
| opendevreview | Ivan Vnučko proposed openstack/kolla-ansible master: Add backend TLS encryption between RabbitMQ management and HAProxy https://review.opendev.org/c/openstack/kolla-ansible/+/919086 | 10:27 |
| opendevreview | Franciszek Przewoźny proposed openstack/kolla-ansible master: Add /v3 suffix to OPENSTACK_KEYSTONE_URL https://review.opendev.org/c/openstack/kolla-ansible/+/924487 | 10:42 |
| opendevreview | Doug Szumski proposed openstack/kolla-ansible master: Support custom Nova Compute Ironic host names https://review.opendev.org/c/openstack/kolla-ansible/+/912483 | 10:53 |
| opendevreview | Will Szumski proposed openstack/kayobe master: WIP: Build Rocky based IPA images https://review.opendev.org/c/openstack/kayobe/+/924498 | 11:52 |
| opendevreview | Will Szumski proposed openstack/kolla-ansible master: Adds the neutron_service_limit option https://review.opendev.org/c/openstack/kolla-ansible/+/877776 | 12:04 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Fix cases where port is not available yet https://review.opendev.org/c/openstack/kolla-ansible/+/924506 | 12:33 |
| opendevreview | Franciszek Przewoźny proposed openstack/kolla-ansible master: Add /v3 suffix to OPENSTACK_KEYSTONE_URL https://review.opendev.org/c/openstack/kolla-ansible/+/924487 | 12:36 |
| kevko | mnasiadka: hmm...that libvirt secret error is not my review fault ... this patch non related to ceph https://review.opendev.org/c/openstack/kolla-ansible/+/924450 << is also failing | 12:37 |
| Fl1nt | Hi everyone | 13:13 |
| Fl1nt | quick question | 13:13 |
| Fl1nt | When building a binary based image | 13:13 |
| Fl1nt | if the upstream repo doesn't contain appropriate package release | 13:13 |
| Fl1nt | I still can use the upper-constraints.txt to override pip installed packages right? | 13:14 |
| Fl1nt | or is there any other mecanism that would let me instruct neutron-openvswitch-agent to upgrade its pyroute2 package to 0.6.11 ? I've tried to use the image footer, it does execute the pip upgrade but the image still use the old 0.6.6 release. | 13:20 |
| opendevreview | Matúš Jenča proposed openstack/kolla-ansible master: Add backend TLS between MariaDB and ProxySQL https://review.opendev.org/c/openstack/kolla-ansible/+/909912 | 13:25 |
| mnasiadka | kevko: libvirt bug or what? | 13:56 |
| opendevreview | Franciszek Przewoźny proposed openstack/kolla-ansible master: Add /v3 suffix to OPENSTACK_KEYSTONE_URL https://review.opendev.org/c/openstack/kolla-ansible/+/924487 | 14:18 |
| kevko | mnasiadka: nope, hardcoded value ... not good implementation in kolla i would say | 14:34 |
| mnasiadka | kevko: nice | 14:40 |
| kevko | mnasiadka: i think if i revert rendering cinder/nova user in secrets for libvirt ..it will work | 14:42 |
| kevko | mnasiadka: but i need to find a way how to do it properly .... if user for vms and volumes is the same ...there should be one secret ...or define in configs somewhere ...still analyzing | 14:42 |
| kevko | mnasiadka: but if you check the libvirt secrets ...even if you set in kolla nova_ceph_user: foo and cinder_ceph_user: bar ... it is client.nova and client.cinder ... | 14:43 |
| kevko | when i refactored ... it's client.cinder and client.nova | 14:44 |
| kevko | so it's somehow working ... | 14:44 |
| kevko | i would say it's somewhere configurable ...and if user is same ..there should be one secret ... | 14:44 |
| kevko | it's about the logic and little bit templating ..but i will handle it | 14:44 |
| kevko | i need this patch to be merged ...as we have third client which needs several cephs for several AZs ...etc ... | 14:45 |
| kevko | you can set keyrings to be the same for all cephs ...but it's not nice ...and not good .. | 14:45 |
| kevko | moreover ..this refactor let you fake several cephs with different ceph.confs and different pools and different users ... | 14:46 |
| Fl1nt | so, anyone about that pyroute2 shit thing ?? ^^ | 14:56 |
| kevko | Fl1nt: let me advice you :) | 14:56 |
| Fl1nt | kevko, go ahead! | 14:57 |
| kevko | Fl1nt: ah, i am not good in kolla overrides and blocks :D i just have my branch where i modify it directly in git from which i am building | 14:57 |
| kevko | Fl1nt: it is an option ? | 14:58 |
| Fl1nt | fine for me, I can override whatever I want ^^ The real deal here is I can't grasp my head around how on binary images, the dnf subsystem is calling pip. | 14:58 |
| Fl1nt | To my understanding | 14:58 |
| Fl1nt | it does through upper-constraints | 14:59 |
| Fl1nt | BUT | 14:59 |
| Fl1nt | the only place I set that upper-constraints.txt file is on kolla-toolbox image | 14:59 |
| Fl1nt | so I'm a bit confused | 14:59 |
| kevko | Fl1nt: i am not sure if in versions which supported binary packages ..the system packages and potentionally created venv 's packages are in same PYTHONPATH ... | 15:00 |
| kevko | it's a long time | 15:01 |
| kevko | Fl1nt: what about to build your own package ? | 15:01 |
| Fl1nt | right know, I've found the correct pythonpath, it's fine, what I don't get is how a working (log show me it was successfully upgraded) pip install --upgrade <package> at image footer location end up not being the package version I found on the pythonpath. | 15:03 |
| Fl1nt | I mean, neutron_openvswitch_agent footer build step show me pip finding 0.6.6 and upgrading it to 0.6.11, how is that possible that I found pyroute2.netcore.0.6.6 on that path once the image deployed?? Oo | 15:04 |
| Fl1nt | Could be a cache thing tho... | 15:04 |
| Fl1nt | let me check that one more time. | 15:05 |
| kevko | It can | 15:06 |
| kevko | Add sleep infinity somewhere to dockerfile and exec to container ...you will see 🙈 | 15:07 |
| Fl1nt | kevko, yep, I'll test that out in order to find out if it correctly install everything at appropriate location as the log tell me so. | 15:25 |
| Fl1nt | kevko, ok, the issue was indeed the pythonpath... I need to find a way to install the packages on /usr/lib/python instead of /usr/local/lib/python now ^^ | 15:54 |
| Fl1nt | thanks! | 15:54 |
| opendevreview | Will Szumski proposed openstack/kolla-ansible master: Adds the neutron_service_limit option https://review.opendev.org/c/openstack/kolla-ansible/+/877776 | 15:55 |
| kevko | Fl1nt: no problem | 19:07 |
| opendevreview | Franciszek Przewoźny proposed openstack/kolla-ansible master: Add /v3 suffix to OPENSTACK_KEYSTONE_URL https://review.opendev.org/c/openstack/kolla-ansible/+/924487 | 20:21 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!