Friday, 2025-04-25

« Thursday, 2025-04-24 Index Saturday, 2025-04-26 »
opendevreviewOpenStack Proposal Bot proposed openstack/openstack-ansible master: Imported Translations from Zanata  https://review.opendev.org/c/openstack/openstack-ansible/+/94803802:26
f0onoonedeadpunk: so I guess all is well for me since I use NFS06:20
f0oI need to read into gnocchi how it actually works and where it dumps data... 06:21
noonedeadpunkwell, you can use swift for data06:22
noonedeadpunka nasty thing about NFS are dead locks in case of any network interruptions, which are extremely annoying06:23
noonedeadpunkbut, you can migrate to ceph anytime more or less (with some effort ofc)06:23
f0othe main concern with ceph is the knowledge gap and unknown compute requirements for it06:36
f0obut one step at a time; let's get coordination_hosts deployed and just gnocchi with swift. then deal with ceilometer later since i probably need that override for coordinator entries06:42
noonedeadpunkf0o: fwiw, you can also push ceilometer data to prometheus or opentelemetry07:10
noonedeadpunkno multi-tenancy though07:10
f0odebugging why gnocchi fails to install now xD08:09
f0ooddly enough it failed at the package installtion step; some large pip errors... just gonna double check that the containers have connectivity and all08:09
f0odidnt know gnocchi doesnt support distro installation08:10
f0oat least ubuntu does have gnocchi packages tho08:14
noonedeadpunkoh, well...08:16
noonedeadpunkthat can be the case08:16
noonedeadpunkwe don't have distro path everywhere indeed08:17
f0oZK installed fine tho; so that's neat!08:26
f0o:D08:26
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: wip  https://review.opendev.org/c/openstack/openstack-ansible/+/94811108:49
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: wip  https://review.opendev.org/c/openstack/openstack-ansible/+/94811108:58
f0onoonedeadpunk: https://paste.opendev.org/show/b5c0nV3S50zKWLEyv42B/09:01
f0oSeems like its not installing build-essentials on debian/ubuntu09:01
f0ohttps://github.com/openstack/openstack-ansible-os_gnocchi/blob/master/vars/debian.yml << its listed as gnocchi_devel_distro_packages but when I check dpkg its not installed09:05
noonedeadpunkwell, this is what is installed on repo container to build wheels for source install09:05
f0ohuh09:06
noonedeadpunkunless wheels build is disabled09:06
noonedeadpunkwhich can be smth you can try09:06
f0owell we got distro as installation method; so I guess the repo container wont build wheels then at all09:06
noonedeadpunktry `venv_wheel_build_enable: false`09:07
f0oand gnocchi container is now trying to install things with pip which fails since build-essentials arent installed09:07
noonedeadpunkyeah09:07
noonedeadpunkbut I guess it's already false..09:07
f0oso I either install those dev packages in all gnocchi containers or I implement distro pacakges into the gnocchi role...09:08
f0oit's swift all over again xD09:08
f0ocan I force the repo container to build wheels?09:09
noonedeadpunkif you have it even...09:09
f0oI do have a repo container09:09
noonedeadpunkbut rly, try venv_wheel_build_enable: false09:11
noonedeadpunkbut I can't tell more without seeing ansible output tbh09:11
f0oTASK [python_venv_build : Show venv install failure message] ******************************************************************************************************************************************************09:12
f0ofatal: [h1_2-gnocchi-container-32776452]: FAILED! => {"changed": false, "msg": "The python packages have failed to install, please check the log file located at /var/log/python_venv_build.log for more information.\n"}09:12
f0oand the file it refers to is just a big log where the error is that it cannot build python-systemd because pkg-config doesnt exist on gnocchi contianer09:12
noonedeadpunkand venv_wheel_build_enable: false was set?09:12
f0onot set yet, setting now09:13
f0oTASK [python_venv_build : Install distro packages for venv build]  << this is new09:13
f0owell it's compiling now09:14
f0oit passed!09:17
noonedeadpunknice :)09:28
noonedeadpunkI think there's some logic which will prevent wheels build for distro setups regardless09:28
noonedeadpunkI'm not sure what it is though...09:28
f0odo I need any additional config to add ceilometer to horizon?11:19
f0oor should it "just work"?11:19
f0oreason I ask is because I'm not seeing ceilometer in horizon... but I got os-ceilometer installed correctly11:20
noonedeadpunkthere's no horizon ui afaik11:22
f0ooh11:22
noonedeadpunkceilometer does not have any api or client11:22
noonedeadpunkthere's an api for gnocchi11:22
noonedeadpunkbut I don't think there;s a horizon dashboard for it11:22
f0ohrm and aodh?11:22
f0oI could swear there used to be a UI in mitaka times11:23
f0oAKA an eternity ago11:23
noonedeadpunkoh, well. back then there was an API part of ceilometer11:23
noonedeadpunkit was dropped around rocky11:23
f0ohaha11:23
f0ojust gotta find a way to verify ceilometer is working now11:24
noonedeadpunkI don't see aodh dashboard with quick googling11:24
f0outility container doesnt have metrics command so probably some client missing like I had with siwft 11:24
noonedeadpunkwell, you can try fetching data from gnocchi 11:24
noonedeadpunkthere should be gnocchiclient11:24
noonedeadpunkbut again, dunno for distro setup...11:25
noonedeadpunkit might be missing indeed11:25
f0oyessir!11:25
noonedeadpunkyeah: https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/utility_all/defaults.yml#L43-L4911:25
f0oa quick python3-gnocchiclient later and boom metric list shows stuff11:26
f0oSo I assume it's working then :D11:27
noonedeadpunkjust for source we grep upper-constraints for clients11:27
noonedeadpunkso we don't need t odefine the list explicitly11:27
f0ohrm12:31
f0osomething went fubar12:32
f0onot sure if it's cinder after coordination hosts or if glance just never recovered12:32
f0owell `openstack image save ...` works so I guess it's ciner12:33
f0oyeah cant even create empty volumes12:37
f0ocant see any errors12:37
f0oany pointers?12:39
f0oit's not erroring its just stuck in creating12:40
f0oand logs dont show anything useful12:40
noonedeadpunkzookeeper effect?12:42
noonedeadpunkare cinder-volume service is running and reporting healthy12:42
noonedeadpunkas I'd guess it's smth between cinder-scheduler and cinder-volume 12:43
f0oah12:43
f0ooslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.20.1.114:5671 after inf tries: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'cinder-backup_fanout' in vhost 'cinder': received 'false' but current is 'true'12:43
f0oso Ceilometer nuked cinder amqp it seems12:43
f0oor at least something nuked it12:44
f0oI had this shit before12:44
f0ostupid backups12:44
f0oxD12:44
noonedeadpunkyou had indeed12:47
noonedeadpunkbut I wonder why this happens12:47
f0ono clue but deleting the exchange and restarting the scheduler always fixes it12:49
noonedeadpunkMy guess is that you might be getting older rabbitmq then expected by some services...12:50
f0oall rabbit nodes are identical tho12:50
noonedeadpunkas with distro install from some releases before, we use rabbitmq/mariadb from native repos12:51
noonedeadpunkwhat version of rabbit/osa?12:51
f0orabbitmq-server             4.0.3-112:51
f0oosa is stable/2024.2 afaik12:51
noonedeadpunkyeah, it's modern enough for sure12:51
f0oI think ti's actually cinder-backup that creates it wrong12:51
noonedeadpunkwell, I'm using cinder-backup with 2024.2 and it's fine12:52
f0obut that is same version as all other cinders12:52
f0ohrm12:52
f0oI also noticed that ceilometer notification agent is hammering the DB like hell12:54
noonedeadpunkDB? Does ceilometer even has an access to DB?12:54
f0omariadb went from ~3-10% to 200+12:54
f0owhen I turn off the ceilometer-central lxc it stops12:55
f0osomething might be crashlooping there12:55
f0opossibly also amqp related since the fanout was borken12:55
noonedeadpunkit's probably gnocchi12:55
f0ounfortunately I'm not at the office anymore so i cant easily check12:55
noonedeadpunkas I think it uses mysql for coordination by default12:56
noonedeadpunkbut it makes total sense to re-configure gnocchi to use zookeeper instead12:56
f0owell I got ZK installed and running12:57
f0oah it doesnt configure it by defalt12:57
f0oI see12:57
noonedeadpunkI think it was missed toghether with ceilometer...12:59
f0ohehe12:59
f0oalso 12:59
f0oh1_2-swift-proxy-container-a9384c0712:59
noonedeadpunkWill try to check on that for 2025.1....12:59
f0oeh12:59
f0oswift-proxy-server[177020]: ModuleNotFoundError: No module named 'ceilometermiddleware'12:59
f0oit also nuked siwft xD12:59
noonedeadpunkright... doh12:59
noonedeadpunkI'd expect it being in swift requirements rather then needed to be defined explicilty13:00
noonedeadpunkbut this one is an easy patch13:00
f0o:D13:00
f0ojust a fun cat and mouse again13:00
f0oneed ceilo for swift accounting; ceilo breaks swift13:00
f0oneed swift for gnocchi13:01
f0oneed gnocchi for ceilo13:01
f0oso many catch.2213:01
f0o:D13:01
noonedeadpunkoh, wait: https://opendev.org/openstack/openstack-ansible-os_swift/src/branch/master/defaults/main.yml#L32713:01
noonedeadpunkbut you;'re having distro setup...13:01
noonedeadpunkah damn it13:01
f0oyep13:01
noonedeadpunkso apt install python3-ceilometermiddleware ?13:02
f0othat's what I'm doing now13:02
noonedeadpunkwill you be able to push the patch to add it here? https://opendev.org/openstack/openstack-ansible-os_swift/src/branch/master/vars/debian.yml#L32 (and also for redhat)13:03
f0othat seems to work13:04
f0owill do when in the office again13:04
f0oon phone its tedious13:04
f0oso that explains gnocchi hammering the DB too since it failed continously to fetch some gnocci-config from swift and restarted13:04
noonedeadpunkyeah...13:06
f0ofun :D13:06
NeilHanlonnoonedeadpunk: btw i added some docs review for https://review.opendev.org/c/openstack/openstack-ansible-ops/+/943866/1113:08
NeilHanlonalso good morning.. i hope you're feeling better! 13:08
f0onoonedeadpunk: for later; how can I "detect" if ceilometer is being pulled here? or should it just install the middleware no matter what?13:11
f0oalso how do I set gnocchi_coordination_url to use zookeeper correctly? I see https://opendev.org/openstack/openstack-ansible-os_gnocchi/src/branch/master/defaults/main.yml#L94-L98 and wouldnt even know how to replicate this for ZK13:14
f0ohttps://opendev.org/openstack/openstack-ansible-os_cinder/src/branch/master/vars/main.yml#L151-L158 << even worse but maybe I can just copy paste that? :D13:16
noonedeadpunkf0o: yeah, I guess you can just install it13:25
noonedeadpunkNeilHanlon: quite some edits13:25
noonedeadpunkand that time I even used a spell checker13:26
NeilHanlon😅 sorry! 13:28
NeilHanlonmy docs team has been beating me about using active voice in technical writing so I can't leave well enough alone13:28
noonedeadpunkyeah, right...13:29
noonedeadpunkI actually can recall someone from my team discouraging me to use passive voice13:30
noonedeadpunkbut my english is bad, so I did not get nuances of it I think13:30
noonedeadpunkLike Rule4: https://xahteiwi.eu/resources/presentations/devopsdays-berlin-2022/13:31
opendevreviewDaniel Preussker proposed openstack/openstack-ansible-os_swift master: Add Ceilometer middleware to distro packages  https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/94819413:31
f0oregarding https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/948194 ; I have no idea if the package name is correct for redhat; I tried looking on pkgs.org but I realized that it doesnt have the OpenStack repos13:32
NeilHanlonEnglish is an awful, awful language. :P 13:34
f0obetter than suomi 💀13:34
f0oluckily I only need to remember that once a year 😅13:35
noonedeadpunkf0o: yeah, it's fine https://mirror.stream.centos.org/SIGs/9-stream/cloud/x86_64/openstack-caracal/Packages/p/13:39
f0o<313:44
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Add a collection for managing encryption of secret data  https://review.opendev.org/c/openstack/openstack-ansible-ops/+/94386613:45
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Add a collection for managing encryption of secret data  https://review.opendev.org/c/openstack/openstack-ansible-ops/+/94386613:46
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-ops master: Fix conditional on ansible_vault_in_place_copy  https://review.opendev.org/c/openstack/openstack-ansible-ops/+/94783413:46
opendevreviewMerged openstack/openstack-ansible master: Imported Translations from Zanata  https://review.opendev.org/c/openstack/openstack-ansible/+/94803814:16
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: Add CI jobs for Ceph on Ubuntu 24.04  https://review.opendev.org/c/openstack/openstack-ansible/+/94825516:47
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: Add CI jobs for Ceph on Ubuntu 24.04  https://review.opendev.org/c/openstack/openstack-ansible/+/94825516:54
noonedeadpunkpotentially, we'd need to do partial backport to 2024.216:56
f0onoonedeadpunk: ceilometer-agent-notification is absolutely insane; it's not only spamming gnocchi with Kpps but also erroring like hell about all sorts of things17:32
f0oare you sure this is stable?17:32
f0o2025-04-25 17:31:56.559 5403 ERROR cotyledon._utils oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.20.1.114:5671 after inf tries: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'heat' in vhost 'swift': received 'false' but current is 'true'17:33
f0oI dont even have Heat17:33
f0oand why would Heat be on vhost Swift?17:33
f0oit's really spamming it too17:34
f0oon gnocchi I get the error17:34
f0oApr 25 17:34:27 h1-2-gnocchi-container-32776452 uwsgi[205]: Fri Apr 25 17:34:27 2025 - *** uWSGI listen queue of socket "10.20.3.251:8041" (fd: 3) full !!! (100/100) ***17:34
f0obut all of this is the ceilometer-agent-notification going berserker17:34
f0oand yes these logs are on all controllers, not just that singular one17:40
f0o2025-04-25 17:42:39.841 92 ERROR cotyledon._utils oslo_messaging.exceptions.MessageDeliveryFailure: Unable to connect to AMQP server on 10.20.2.175:5671 after inf tries: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'nova' in vhost 'swift': received 'false' but current is 'true' << also this... 17:42
opendevreviewIvan Anfimov proposed openstack/ansible-role-httpd master: tox: Remove basepython  https://review.opendev.org/c/openstack/ansible-role-httpd/+/94825917:44
opendevreviewIvan Anfimov proposed openstack/ansible-role-httpd master: tox: Bump minimum version and remove basepython  https://review.opendev.org/c/openstack/ansible-role-httpd/+/94825917:45
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: tox: Bump minimum version and remove basepython  https://review.opendev.org/c/openstack/openstack-ansible/+/94826117:47
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: tox: Bump minimum version and remove basepython  https://review.opendev.org/c/openstack/openstack-ansible/+/94826117:48
opendevreviewIvan Anfimov proposed openstack/openstack-ansible-os_horizon master: Loop around module list in horizon_translations_update  https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/65604518:09
opendevreviewIvan Anfimov proposed openstack/openstack-ansible-os_horizon master: Loop around module list in horizon_translations_update  https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/65604518:09
opendevreviewIvan Anfimov proposed openstack/openstack-ansible-os_horizon master: Loop around module list in horizon_translations_update  https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/65604518:11
opendevreviewIvan Anfimov proposed openstack/openstack-ansible-os_horizon master: Loop around module list in horizon_translations_update  https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/65604518:13
opendevreviewIvan Anfimov proposed openstack/openstack-ansible-os_horizon master: Loop around module list in horizon_translations_update  https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/65604518:15
-opendevstatus- NOTICE: Gerrit is getting restarted to pick up container image updates. It should only be gone for a moment.18:15
f0oI've rerun setup-everything again and I still need to hard kill ceilometer-agent-notification.service on all ceilo containers because it's just absolutely going insane and throwing the oddest errors like above18:41
f0oany openstack service on the vhost swift... sahara? sure thing, nova? ofc neutron? probably18:41
f0oand while it does it, it's also ddos'ing gnocchi to a point where it's just crashing. all of 5*16 instances of it18:42
f0oalso after using ZK for coordinator on ceilometer the ceilometer-polling service fails with ModuleNotFoundError: No module named 'kazoo'18:44
f0oso os-ceilometer needs python3-kazoo in distro packages at least18:47
f0ono clue what to do about the ceilometer-agent-notification.service going berserker18:48
f0ohrm i now have all those exchanges in all my vhosts... i never installed sahara or trove or similar and yet I got those exchanges on all vhosts19:06
f0oguess ill need to manually clean those up and hope not to delete some exchange thats needed19:06
opendevreviewMerged openstack/openstack-ansible master: Maximize amount of ANSIBLE_FORKS for CI  https://review.opendev.org/c/openstack/openstack-ansible/+/94712820:07
opendevreviewIvan Anfimov proposed openstack/openstack-ansible-os_horizon master: Loop around module list in horizon_translations_update  https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/65604520:40
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: wip  https://review.opendev.org/c/openstack/openstack-ansible/+/94811121:14
opendevreviewIvan Anfimov proposed openstack/openstack-ansible master: wip  https://review.opendev.org/c/openstack/openstack-ansible/+/94811121:29
« Thursday, 2025-04-24 Index Saturday, 2025-04-26 »

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!