Thursday, 2024-04-11

SvenKieskemnasiadka, kevko: for opensearch I tried this but it failed then with grafana: https://review.opendev.org/c/openstack/kolla/+/915322 should be good to merge imho06:51
opendevreviewPierre Riteau proposed openstack/kayobe master: CI: Run kayobe-tox-ansible using Ubuntu Focal  https://review.opendev.org/c/openstack/kayobe/+/91533007:23
SvenKieskekevko, mnasiadka: I raised also a bug yesterday, but "only" to pin opensearch https://bugs.launchpad.net/kolla/+bug/206066807:52
SvenKieskeand what do we do for the grafana situation? it seems grafana has it's own conference ongoing so it seems unlikely someone will respond soon, given it's been broken for a week now07:54
SvenKieskedoes someone know anyone at grafana labs per chance?07:54
SvenKieskeelse I would just say mark the whole monitoring stuff as unbuildable and non-voting for now? I know that's a rather harsh reaction, but CI is blocked.07:55
SvenKieskemhm, there are reports coming in that it works again, will test locally07:57
SvenKieske\o/ seems to work again (tested in debian latest container)08:01
SvenKieskefwiw there was an incident but only related to the grafana-agent apt package: https://status.grafana.com/incidents/xblqfhgkz0sg08:05
opendevreviewPierre Riteau proposed openstack/kayobe master: Revert "Mark kayobe-tox-ansible job as non-voting"  https://review.opendev.org/c/openstack/kayobe/+/91546508:21
opendevreviewMichal Arbet proposed openstack/kolla master: [DNM] Add apt-cache policy to identify problems with install package  https://review.opendev.org/c/openstack/kolla/+/91546608:38
SvenKieskekevko: the grafana stuff seems to have just been silently fixed on their part, that's why your local installation works again. :)08:45
kevkokevko: I don't think08:51
kevkoSvenKieske: what does it mean they just silently fixed ? I've tried to build grafana right after build failed in CI ..and worked ...  so it can be problem in a mirror i would say ...as my local build refers directly to their repo while CI not 08:52
SvenKieskekevko: I pinged you multiple time on issues raised and fixes proposed, did you read any of that?08:53
SvenKieskekevko: one last try: https://github.com/grafana/grafana/issues/85827 many people had this issue with direct upstream repo officially from grafana08:53
SvenKieskekevko: I also could reproduce this issue locally yesterday, with upstream repo, no mirrors involved08:54
SvenKieskekevko: if you read this issue till the end it started working again today, one or two hours ago08:54
SvenKieskekevko: I also verified this locally. so it really really was broken upstream. for some days at least.08:55
SvenKieskeHTH :)08:55
kevkoSvenKieske: haha, you are right :D ...working now 09:03
SvenKieskekevko: yeah, good that it works now. I was actually also quite surprised that it was really an upstream issue09:04
kevkoSvenKieske: hack with policy .... btw ... useful no ? https://8c8da465335a8767dcb7-4a7e6fa54cc64c7867564a09e7ad64c3.ssl.cf1.rackcdn.com/915466/1/check/kolla-build-debian/001517e/kolla/build/grafana.log 09:04
opendevreviewMichal Arbet proposed openstack/kolla master: Fix handling configs in base image  https://review.opendev.org/c/openstack/kolla/+/91544009:06
opendevreviewMichal Arbet proposed openstack/kolla master: [DNM] Add apt-cache policy to identify problems with install package  https://review.opendev.org/c/openstack/kolla/+/91546609:06
SvenKieskemhm, maybe a little long? it would be nice if we could just toggle this via some debug switch, maybe?09:06
kevkoSvenKieske: long ? :D ...we can also drop the log and write OK or FAIL into the log :D 09:08
SvenKieskeI mean it's not thaaat long, but not short? :D It's okayish I guess.09:16
kevkohow is it possible that https://zuul.opendev.org/t/openstack/build/4e86289c85434261a24b6eca03716795 failing but locally working ? :D 09:21
kevkoah, ok now failing :) 09:27
SvenKieskeso: do we want to keep opensearch non voting or do the pinning on 2.12 instead until we have auth working in CI?09:34
kevkohow i hate unit tests :( 09:45
SvenKieskeI like them :)09:51
priteauSvenKieske: I would think we need pinning since it also needs to be backported?10:40
opendevreviewMerged openstack/kayobe master: Revert "Mark kayobe-tox-ansible job as non-voting"  https://review.opendev.org/c/openstack/kayobe/+/91546511:12
SvenKieskepriteau: my idea was to not extend the pinning outside master because it's a CI only issue and we would pin users to a possibly older release than what they are already running, also 2.13 fixed some cves, which I don't know the impact of.11:22
SvenKieskeso afaik the idea was: workaround the CI issue for now by pinning and then implement proper auth between opensearch and haproxy so we can revert the pin asap11:23
mnasiadkaPinning opensearch is complicated in stable branches, we will break people already using 2.1311:27
mnasiadkaI’ll have a good look next week how to fix that drama11:27
kevkoSvenKieske: to write them ? 11:30
SvenKieskekevko: to write what? Sorry, I don't understand?11:35
kevkoSvenKieske: how i hate unit tests -> I like them :) -> to write them ? 11:36
kevkodo anybody know ? https://6a83944802eed202968f-4cf4bb096d4774f141f32d30757edf40.ssl.cf2.rackcdn.com/915440/3/check/kolla-ansible-rocky9-upgrade/b6af6a4/primary/logs/kolla/libvirt/libvirtd.txt11:36
SvenKieskeah! :D well I like working unit tests that prevent regressions. ;) I can also write them, yes.11:36
SvenKieskeyes there's a bug open about /dev/kvm11:37
kevkoSvenKieske: I can also write them is not I like write them :D 11:38
SvenKieskekevko: ah no, sorry, that one is slightly different: https://bugs.launchpad.net/kolla-ansible/+bug/204971611:38
kevkothis is upgrade jobs for both debuntu and rocky 11:39
SvenKieskekevko: we could probably run this before to check if everything works? https://libvirt.org/manpages/virt-host-validate.html11:39
SvenKieskeI stole that idea from this issue :D https://github.com/kubevirt/kubevirt/issues/1060511:39
SvenKieskemight be an issue with nested-vt and/or our qemu-kvm driver config, or some distro package? but if it's affecting both rpm and deb it's either upstream bug or our config/env is buggy I think, without deep analysis.11:41
opendevreviewPierre Riteau proposed openstack/kayobe stable/2023.2: Revert "Mark kayobe-tox-ansible job as non-voting"  https://review.opendev.org/c/openstack/kayobe/+/91535511:47
opendevreviewPierre Riteau proposed openstack/kayobe stable/2023.1: Revert "Mark kayobe-tox-ansible job as non-voting"  https://review.opendev.org/c/openstack/kayobe/+/91535611:47
opendevreviewMartin Hiner proposed openstack/kolla-ansible master: Add container engine migration scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/83694113:09
opendevreviewMark Goddard proposed openstack/kolla-ansible master: Remove "distutils" library  https://review.opendev.org/c/openstack/kolla-ansible/+/85700313:32
opendevreviewAlex Welsh proposed openstack/kolla-ansible master: Automate prometheus blackbox configuration  https://review.opendev.org/c/openstack/kolla-ansible/+/91242014:39
opendevreviewSven Kieske proposed openstack/kolla master: CI/Master only: pin opensearch{-dashboards}  https://review.opendev.org/c/openstack/kolla/+/91532215:28
opendevreviewMatúš Jenča proposed openstack/kolla-ansible master: Add backend TLS between MariaDB and ProxySQL  https://review.opendev.org/c/openstack/kolla-ansible/+/90991215:31
chris218Hi I've been wondering recently, does Kolla support virtualized control plane setup? As in control plane being inside VMs managed by itself?17:06
fricklerchris218: well kolla uses docker or podman containers, so that's kind of virtualized. I don't see how you could run the control plane within VMs in nova, how would you bootstrap or cold start such a setup?17:44
chris218Create 3 VMs manually with QEMU  on the same network that Neutron will use, then deploy on all hosts + 3 VMs, and lastly manually shut down and import one-by-one each VM to Nova keeping all the static IPs17:47
chris218That would be the rough idea17:47
chris218I've inherited 7 pretty beefy servers to manage which have a year old and a bit broken Kolla installation and am now evaluating different ways I can setup it most efficiently 17:52
chris218Hmm the only issue might be if I stop all the containers and then want to start them I won't be able to connect to the control plane to start services on it back 🤔21:53

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!