Thursday, 2023-10-05

opendevreviewPierre Riteau proposed openstack/kayobe stable/yoga: Remove upgrade jobs following Xena EOL  https://review.opendev.org/c/openstack/kayobe/+/89740007:10
opendevreviewPierre Riteau proposed openstack/kayobe stable/zed: CI: Enable Ubuntu jobs again  https://review.opendev.org/c/openstack/kayobe/+/89740107:33
zigokevko: Hey, what was the Salsa URL of python-podman, so I fix your issue?08:30
zigo(and sorry for the delay, the Bobcat release kept me busy...)08:31
mnasiadkakevko: well, erlang 24 doesn't work on aarch64 :)08:50
mnasiadkaso basically aarch64 is to remain broken09:04
kevkozigo: https://salsa.debian.org/python-team/packages/python-podman , and it is already in unstable -> https://packages.debian.org/sid/python3-podman  << 09:35
opendevreviewMichal Arbet proposed openstack/kolla master: Switch from antelope to bobcat APT repository  https://review.opendev.org/c/openstack/kolla/+/89740609:40
zigokevko: FYI, I installed it on bookworm-zed up to bookworm-bobcat. Rsync is currently running...09:41
zigoI'm nearly all done with bookworm-bobcat.09:42
zigoAll services done, only a few dashboard and neutron plugins are remaining.09:42
opendevreviewMichal Nasiadka proposed openstack/kolla master: Revert "CentOS/Rocky: use CentOS Cloud SIG repo instead of Delorean"  https://review.opendev.org/c/openstack/kolla/+/89085109:57
kevkozigo:  well, kolla don't support deb packages anymore 09:59
opendevreviewMichal Arbet proposed openstack/kolla master: Debian: Switch from antelope to bobcat APT repository  https://review.opendev.org/c/openstack/kolla/+/89740610:04
guesswhat[m]Hey, what openstack-exporter is used in latest stable kolla? 1.6.0 ? Thanks10:28
kevkoguesswhat[m]: you can check yourself 10:29
guesswhat[m]did exec and openstack-exporter --version, but its empty10:29
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: [yoga-only]: Fix erlang versions  https://review.opendev.org/c/openstack/kolla/+/89735210:29
guesswhat[m]docker exec -it prometheus_openstack_exporter /opt/openstack-exporter/openstack-exporter --version10:30
kevkoguesswhat[m]: haha, so why you don't use git ? kolla is public accessible and tagged10:30
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: [yoga-only]: Fix erlang versions  https://review.opendev.org/c/openstack/kolla/+/89735210:31
guesswhat[m]asking cuz I have few metrics missing from this list https://github.com/openstack-exporter/openstack-exporter#metrics-collected10:31
opendevreviewVerification of a change to openstack/kayobe stable/2023.1 failed: Fix data file path detection with new pip  https://review.opendev.org/c/openstack/kayobe/+/89637910:32
guesswhat[m]its 1.6.0 https://github.com/openstack/kolla/blob/stable/2023.1/docker/prometheus/prometheus-openstack-exporter/Dockerfile.j2#L9, not sure whats wrong then10:32
guesswhat[m]holy moly, its info log level is freaking crazy10:36
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: [yoga-only]: Fix erlang versions  https://review.opendev.org/c/openstack/kolla/+/89735210:36
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: [yoga-only]: Fix erlang versions  https://review.opendev.org/c/openstack/kolla/+/89735210:37
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: [yoga-only]: Fix erlang versions  https://review.opendev.org/c/openstack/kolla/+/89735210:39
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: [yoga-only]: Fix erlang versions  https://review.opendev.org/c/openstack/kolla/+/89735210:39
opendevreviewPierre Riteau proposed openstack/kayobe stable/yoga: Remove upgrade jobs following Xena EOL  https://review.opendev.org/c/openstack/kayobe/+/89741010:39
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: [yoga-only]: Fix erlang versions  https://review.opendev.org/c/openstack/kolla/+/89735210:40
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: [yoga-only]: Fix erlang versions  https://review.opendev.org/c/openstack/kolla/+/89735210:40
mnasiadkahopefully this passes now10:40
guesswhat[m]seems https://github.com/openstack-exporter/openstack-exporter/issues/268#issuecomment-1484789977 fixes the bug with nova metrics10:42
kevkoguesswhat[m]: fill the bug, test it and propose patch :) 10:44
mnasiadkacontributors welcome :)10:46
guesswhat[m]dont have enough good experience with contributing here in kolla project :D10:50
guesswhat[m]mnasiadka: https://bugs.launchpad.net/kolla-ansible/+bug/2036301 this one is mine , the answer is unfortunatelly no 10:50
mnasiadkaguesswhat[m]: Is that our fault you're not interested  (lack of reviews or something similar) - or there's some different reason?10:52
kevkomnasiadka: what about use Kerl and compile erlang ourself ? :D 11:01
kevkooverkill ? :D 11:01
mnasiadkawell, I think erlang for RPM was built in COPR by hrw11:02
mnasiadkaI would normally ask the RMQ guys to have a repo with 25.2, but given their welcoming response last time, I'll pass11:02
mnasiadkaUnless we want to copycat their Launchpad setup and do a PPA for 25.211:03
kevkomeh, anyway split aarch64 from rest is good step I think ...11:04
kevkomost of the time aarch64 is a thorn in the heel11:05
kevkoas a reson to switch repositories etc etc 11:05
opendevreviewMerged openstack/kayobe stable/zed: CI: Enable Ubuntu jobs again  https://review.opendev.org/c/openstack/kayobe/+/89740111:05
mnasiadkawell, we could just drop it if nobody is using it, but feels a bit of shame11:07
kevkoPTG theme  ? 11:08
kevkoi would like to know who is using aarch64 11:08
kevkobut yeah, better to not remove 11:09
kevkobut it's bit hard to maintain aarch64 if package sources missing some versions and another repos missing completly aarch6411:10
mnasiadkaespecially hrw is not really around here anymore to care for these11:10
mnasiadkaand I don't even have an aarch64 system to check out anything11:10
Continuityguesswhat[m], I can confirm that the bug you ref above for openstack-exporter does indeed fix the issues with nova metrics11:11
kevkomnasiadka: now? or do you mean it in general?11:12
kevkoregarding hrw11:12
mnasiadkakevko: I mean in general, Linaro does not care about OpenStack anymore I think - but if we drop aarch64 from Kolla - then I don't know what happens with Linaro's OpenDev nodes (which use Kolla) :)11:13
hrwkevko: at Linaro we ended work on OpenStack.11:13
hrwmnasiadka: there are no plans on upgrading Linaro developer cloud11:13
mnasiadkahrw: I think developer cloud and the nodepool provider are two different things11:14
mnasiadkaAt least that's what clarkb mentioned one day11:14
hrwkevko: I moved to other tasks. Kolla/aarch64 was quite time consuming11:14
kevkoaaa , i didn't know this ..11:14
hrwmnasiadka: nodepool is pool of hardware11:14
hrwiirc11:15
mnasiadkahrw: well, I'll rephrase that - the OpenStack cloud that Opendev CI uses via nodepool is some separate cloud that uses Kolla11:15
hrwok11:15
mnasiadkaso if we end that - it might be that OpenStack has no place to test ARM stuff11:15
mnasiadkaand for now the only core problem I see is RMQ :)11:16
hrwwhat this time?11:16
guesswhat[m]Continuity:  theres probably one more: level=error msg="Failed to collect metric for exporter: nova, error: failed to collect metric: security_groups, error: Resource not found: [GET http://192.168.1.1:8774/v2.1/os-security-groups], error message: {\"itemNotFound\": {\"code\": 404, \"message\": \"The resource could not be found.\"}}" source="exporter.go:123"11:16
hrwmnasiadka: I thought that kolla's policy for rmq is "use latest everywhere as upstream gives us only latest"11:16
mnasiadkahrw: RMQ 3.9 (which we use in Yoga) supports max erlang 25.2.*11:16
mnasiadkaand all repos have only 25.3.*11:17
mnasiadkaso we would need to build latest 25.2 in COPR and I guess Launchpad11:17
hrwmnasiadka: migrate yoga to 3.10 then?11:17
hrwor EOL yoga11:17
mnasiadkahrw: it was migrated from 3.8, so it's a bit complicated - because we can't jump from 3.8 to 3.10 directly (if somebody deployed 3.8 in the past and didn't update to 3.9)11:18
hrwEOL then11:18
mnasiadkayeah, I don't really mind marking Yoga as EOL soon11:18
hrwif we cannot maintain it then EOL is clear sign to users11:18
mnasiadkaExtended Maintenance estimated 2023-11-0211:18
Continuityguesswhat[m]: yeah thats been throwing errors for as long as i have been using the exporter :D. One day i might get round to seeing if i can work out a fix. We dont use those metrics so it hasnt been an issue for us11:19
hrwthis way if someone is on 3.8 then latest yoga images will be 'old, unmaintained' 3.9 version. Which can be updated to 3.10 in Zed11:19
mnasiadkaand we should surely look into delivering multiple RMQ version tagged images for each release11:20
guesswhat[m]Continuity: btw, what metrics / monitoring are You using? Unfortunutally a lot of public Grafana dashboards are "unbaked"11:20
mnasiadkaor even go back to the infra images approach and have a different repo that is non-cycle-bound11:20
hrwimagine user X having cloud on wallaby rmq 3.8(?). they can upgrade to EOL yoga, get rmq 3.9 and then move to Zed with 3.10 (which should be upgraded to Bobcat anyway)11:21
kevkoI understand that EOL is the solution to almost all possible problems, but it is still not a solution. On the one hand, we strictly take care not to upgrade erlang/rabbitmq (because someone may have an old version), and on the other hand, we drop the entire branch - which is the same11:21
Continuityguesswhat[m]: Mostly in house developed stuff. 11:21
Continuitya mix of openstack exporter, container exporter, and libvirt exporter11:21
Continuityto give a fully rounded picture.11:21
hrwkevko: how many devs kolla has? 4-5?11:21
guesswhat[m]Continuity: nothing publicly available, right? :D I have bunch of exporters running ( there are nice dashboards for cadvisor, node-exporter, etc, but nothing perfect for libvirt and openstack-exporter )11:22
hrwkevko: I was always a fan or 'do devel branch, care about latest stable, may look at anything older if there is a time'11:22
kevkohrw:  i totally understand you 11:23
hrwkevko: I am in Kolla since 2017 and this project was always overloaded with work11:23
kevkomee too :) 11:23
kevkomaybe 2018 ? 11:23
Continuityguesswhat[m]: not currently. there is nothing *secret* about what we are doing, its just very tailored to our environment. 11:23
hrwso if current-4 branch needs work then it is time to EOL it to leave migration path11:23
hrwmnasiadka: add 'one RMQ/Erlang upgrade per stable branch' to PTG topics?11:24
mnasiadkawell, we're already doing it like that - without a PTG topic11:24
mnasiadkabut yeah, we should discuss the way forward11:25
hrwmnasiadka: so stable gets rmq 3.X, upgrade to 3.X+1 during maintaince and 3.X+2 == EOL11:25
kevkoWell, i am ok with it ...  but i think there should be a way how to fix even that branches ...something as  need only one +2 core 11:25
mnasiadkaI guess we'll fix Yoga for x86_64 and do EOL11:25
hrwmnasiadka: what about 3.8 -> 3.10 upgrade???11:25
mnasiadkakevko: Add a topic to the PTG about this and we can discuss ;)11:26
guesswhat[m]Continuity: okay, thanks :)11:26
mnasiadkahrw: basically from my perspective we should be able to bump to latest RMQ irrelevant of the OpenStack release11:26
kevkofor example we have also customers with older versions (because they are waiting and waiting and don't want to upgrade) ...but then ..when it comes ... upgrade is painfull :D 11:27
kevkomnasiadka: +1 +1 +1 11:27
hrwmnasiadka: Xena, Wallaby are on rmq 3.8, right? Moving Yoga to rmq 3.10 will break migration path11:28
opendevreviewRafal Lewandowski proposed openstack/kolla-ansible master: Add a separate interface address for tgtd  https://review.opendev.org/c/openstack/kolla-ansible/+/89741911:29
mnasiadkahrw: yes, but if we produce separate image for 3.9 and separate for 3.10 and separate for 3.11 - we can have a variable in kolla-ansible that denotes the version you want to use11:29
mnasiadkaand do an upgrade check11:30
hrwmnasiadka: if you have time to maintain it11:31
hrwand to write, test, backport, test in 5 branches11:31
mnasiadkahrw: well, there are people like me and kevko that use Kolla in day-to-day work11:31
mnasiadkaso it would just make our lifes easier11:32
hrwok11:32
kevkothis should make you think about whether tagging components with the openstack version makes sense (?) Kolla should be tagged with real tags of the given components, and kolla ansible should define the versions of these components. Then each version of kolla-ansible can define a specific version of component11:32
mnasiadkakevko: I still think we should be release specific for OpenStack components, but we could break out infra components out of that OpenStack release loop11:33
kevkoagree 11:33
hrwsplit kolla into kolla-infra and kolla-openstack11:33
mnasiadkayup11:33
kevkomy thinking was about another components 11:33
kevkorabbit, maria, proxysql ..etc ..etc 11:33
kevkohrw +1 11:33
hrwkevko: anything !openstack-base based is infra11:34
kevkoyep ... idea behing is the same 11:34
kevkobehind 11:34
hrwcode for it is there11:34
mnasiadkaok, added that to PTG11:35
hrwopenstack.kolla/mariadb:11 openstack.kolla/rabbitmq:3.1311:41
hrwopenstack.kolla/neutron-server:bobcat11:41
hrwthis kind?11:42
mnasiadkayup11:42
hrwmnasiadka: makes a lot of sense11:42
SvenKieskeinteresting discussion, was in meetings the whole morning11:43
SvenKieskemnasiadka: why is this marked as "invalid"? https://bugs.launchpad.net/kolla/+bug/202532111:43
opendevreviewVerification of a change to openstack/kayobe master failed: Revert "CI: Disable bare metal testing on RL9/c9s"  https://review.opendev.org/c/openstack/kayobe/+/89726012:11
mnasiadkaSvenKieske: because it's targeted to Yoga and it's not invalid there?12:23
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: [yoga-only]: Fix erlang versions  https://review.opendev.org/c/openstack/kolla/+/89735212:24
SvenKieskeokay, I hate launchpad UX :D12:39
hrwSvenKieske: a way to say 'Yoga only'12:56
mnasiadkahrw, kevko, frickler: https://review.opendev.org/c/openstack/kolla/+/890851 - can we get that merged, and then I'll post a patch to use bobcat from rdo (which we can later revert in master after branching)? :)12:58
SvenKieskehrw: yeah, as I said, I hate launchpad UX. it should say: master|antelope etc. invalid and yoga valid, instead it's invalid against the core component. it's okay, just confusing UX, to me, so ymmv.13:00
SvenKiesketo me at least*13:00
hrwmnasiadka: +213:00
mnasiadkathanks13:00
hrwmnasiadka: next time send as patch series: one reverting, second moving to bobcat13:01
hrwmnasiadka: easier to notice plans13:01
mnasiadkahrw: too many things on my plate, but sure :)13:02
hrwmnasiadka: it makes it easier for you too13:02
hrwmnasiadka: 'git checkout -b move-centos-ones-to-bobcat', 2 commits, git review -y, done13:03
*** hrww is now known as hrw13:15
opendevreviewMichal Nasiadka proposed openstack/kolla stable/zed: ubuntu: mark collectd and telegraf as buildable  https://review.opendev.org/c/openstack/kolla/+/89590113:27
opendevreviewMichal Nasiadka proposed openstack/kolla stable/zed: ubuntu: mark collectd and telegraf as buildable  https://review.opendev.org/c/openstack/kolla/+/89590113:27
kevkohrw: https://review.opendev.org/c/openstack/kolla/+/897406 can u ? 13:30
hrwdone13:30
kevkothank you 13:36
kevkomnasiadka: i am preparing my downstream repo for upgrade ...do you think we can backport some usefull patches from master to zed ? 13:37
kevko(so i don't need to include it in my downstream repo ..i will just rebase against our zed upstream )13:37
kevkothis type of patches , for example this https://review.opendev.org/c/openstack/kolla-ansible/+/86543413:39
kevkonot big , not killer 13:39
mnasiadkaIf those are usability improvements, I guess we could merge the backports - as long as they don't change any default behaviour13:40
kevkookay13:50
kevkothanks 13:50
opendevreviewMichal Arbet proposed openstack/kolla-ansible stable/zed: Add ability to configure rabbitmq  https://review.opendev.org/c/openstack/kolla-ansible/+/89744113:53
opendevreviewMichal Arbet proposed openstack/kolla-ansible stable/zed: Add ability to configure rabbitmq  https://review.opendev.org/c/openstack/kolla-ansible/+/89744113:56
opendevreviewMichal Arbet proposed openstack/kolla-ansible stable/zed: Trivial: Add connection: local for keystone-fernet cron generate task  https://review.opendev.org/c/openstack/kolla-ansible/+/89744213:58
guesswhat[m]What about adopting https://github.com/inovex/prometheus-libvirt-exporter instead of Tinkoff`s exporter, see https://github.com/prometheus-community/community/issues/50 for more info.13:59
opendevreviewPierre Riteau proposed openstack/kayobe stable/yoga: Fix data file path detection with new pip  https://review.opendev.org/c/openstack/kayobe/+/89638114:04
kevkoguesswhat[m]: we have libvrit exporter if i remember well14:05
priteauI think guesswhat[m] means replacing the current one14:06
opendevreviewMichal Arbet proposed openstack/kolla-ansible stable/zed: Fix issue with octavia security group rules creation  https://review.opendev.org/c/openstack/kolla-ansible/+/89744314:06
priteauI didn't realise Tinkoff/libvirt-exporter had been archived :/ 14:07
opendevreviewMichal Arbet proposed openstack/kolla-ansible stable/zed: Add support for multiple ceph files  https://review.opendev.org/c/openstack/kolla-ansible/+/89744414:10
opendevreviewMichal Arbet proposed openstack/kolla-ansible stable/zed: Configure coordination in default for masakari-api  https://review.opendev.org/c/openstack/kolla-ansible/+/89744514:14
opendevreviewMerged openstack/kayobe stable/2023.1: Fix data file path detection with new pip  https://review.opendev.org/c/openstack/kayobe/+/89637914:35
guesswhat[m]priteau: it has also lot of bugs, https://github.com/inovex/prometheus-libvirt-exporter has these fixed and its based on better library 14:49
opendevreviewMerged openstack/kayobe stable/yoga: Remove upgrade jobs following Xena EOL  https://review.opendev.org/c/openstack/kayobe/+/89741014:50
priteauFeel free to propose a patch. It would be nice to know which metrics might change.14:50
mnasiadkaThere's also https://github.com/vexxhost/libvirtd_exporter - which looks like it shouldn't vanish ;)14:51
guesswhat[m]wondering what they are using for openstack part, if openstack-exporter/openstack-exporter or exporter by cannonical, or something else? 14:51
mnasiadkabut it would be good to compare metrics14:51
guesswhat[m]openstack-exporter/openstack-exporter has lastest release year ago..14:52
opendevreviewMerged openstack/kolla stable/zed: Revert "bifrost: mark unbuidable for Ubuntu"  https://review.opendev.org/c/openstack/kolla/+/89684914:53
mnasiadkaand last commit two weeks ago14:53
priteauguesswhat[m]: It's still active, there may be a new release soon14:53
mnasiadkahttps://github.com/openstack-exporter/openstack-exporter/issues/29914:53
guesswhat[m]priteau: may be..  :) 14:55
priteaumnasiadka: I saw the vexxhost libvirtd exporter a while ago, but last commit in 2020…14:58
priteauIt's even less maintained than other exporters14:59
guesswhat[m]priteau: also https://github.com/inovex/prometheus-libvirt-exporter is maintained by a company15:17
priteauWell, vexxhost is a company too, as were tinkoff and kumina15:18
kevkobtw, what about vexxhost magnum driver in kolla ? i remember that i've seen some discussion here :) 15:26
kevkoi think then magnum start to be usable :D 15:26
opendevreviewMerged openstack/kolla master: Debian: Switch from antelope to bobcat APT repository  https://review.opendev.org/c/openstack/kolla/+/89740615:34
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: [yoga-only]: Fix erlang versions  https://review.opendev.org/c/openstack/kolla/+/89735216:12
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: [yoga-only]: Fix erlang versions  https://review.opendev.org/c/openstack/kolla/+/89735216:13
opendevreviewMichal Nasiadka proposed openstack/kolla stable/yoga: [yoga-only]: Fix erlang versions  https://review.opendev.org/c/openstack/kolla/+/89735216:14
mnasiadkakevko: I think SCS/OSISM is interested in getting that in (cc: SvenKieske frickler)16:15
opendevreviewMerged openstack/kolla stable/zed: ubuntu: mark collectd and telegraf as buildable  https://review.opendev.org/c/openstack/kolla/+/89590116:18
kevkomnasiadka: well, we are using kubespray and rancher ...but personally i am interested to get in ..because without it i think magnum is not usable :D 16:38
guesswhat[m]CAPO works just ok, theres no need to use Magnum at all, https://github.com/vexxhost/magnum-cluster-api still reuquires a k8s cluster with CAPI installed, its just a API proxy..16:41
opendevreviewVerification of a change to openstack/kayobe master failed: Use importlib.metadata instead of importlib_metadata  https://review.opendev.org/c/openstack/kayobe/+/89730616:41
guesswhat[m]Also vexxhost is using k8s as underlying orchestrator https://github.com/vexxhost/atmosphere16:42
opendevreviewMerged openstack/kayobe stable/2023.1: Use importlib.metadata instead of importlib_metadata  https://review.opendev.org/c/openstack/kayobe/+/89731316:43
opendevreviewMerged openstack/kayobe stable/zed: Use importlib.metadata instead of importlib_metadata  https://review.opendev.org/c/openstack/kayobe/+/89731416:43
guesswhat[m]Oh, Horizon is now supporting OTP https://docs.openstack.org/releasenotes/horizon/2023.2.html ?16:48
mnasiadkaguesswhat[m]: there are people that like using Horizon or standardized OpenStack APIs (e.g. via Terraform) - so they are happy with Magnum supporting CAPO16:52
kevkocapo ? 16:53
kevkomnasiadka: regarding erlang ..what about build erlang in image ? :D 16:53
kevkovia kerl 16:53
kevkoleave it as is for x86_64 and build for aarch6416:54
mnasiadkakevko: Cluster API Provider Openstack (the openstack driver for CAPI)17:27
kevkoaaa ok 17:53
greatgatsbykevko: mnasiadka:  Thanks for the work on the rabbitmq/erlang PR.  Just a heads up, we built our own rabbitmq image with the correct versions (as per the PR) and pushed those out to one of our existing clusters.  Every VM went down.  We have another cluster where we already had the VM shutdown issue, where we then rolled out the new images, and it's been rock solid, running deploy after deploy, nothing goes down.  Might not be related at 19:46
greatgatsbyall, but I thought I should just mention in case changing the erlang version, even if to the correct one, will itself cause VMs to go down the first time. 19:46
greatgatsbysorry if this is incorrect, but I thought better to mention than not19:46
opendevreviewPierre Riteau proposed openstack/kayobe master: Speed up migrate_rabbitmq_queues  https://review.opendev.org/c/openstack/kayobe/+/89749321:08
opendevreviewVerification of a change to openstack/kayobe master failed: Use importlib.metadata instead of importlib_metadata  https://review.opendev.org/c/openstack/kayobe/+/89730622:03

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!