Thursday, 2024-03-21

mnasiadkaSvenKieske: not really, last time they sent us a link to a package07:05
mnasiadkaSvenKieske: replied there07:06
mnasiadkajovial: re https://bugs.launchpad.net/kolla-ansible/+bug/2058512 - is the CI in kolla-ansible failing or is it kayobe?07:08
mnasiadkafantastic, seems docker 26 broke fluentd image label check07:15
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: common: Fix fluentd labels when using Docker 26  https://review.opendev.org/c/openstack/kolla-ansible/+/91386807:21
fricklernice one07:23
mnasiadkawe could actually drop the usage of labels, since we moved to v5 in B07:25
mnasiadkabut let's see if that succeeds07:28
mnasiadkabecause we'd need to backport the fix07:36
mnasiadkayay, upgrade jobs are failing07:48
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/2023.2: common: Fix fluentd labels when using Docker 26  https://review.opendev.org/c/openstack/kolla-ansible/+/91380507:49
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/2023.1: common: Fix fluentd labels when using Docker 26  https://review.opendev.org/c/openstack/kolla-ansible/+/91380607:49
mnasiadkaso 2032.2 will need to go in first07:50
SvenKieskeo/08:01
SvenKieskeprometheus-opensearch job in above stable/2023.2 docker 26 fix fails in the precheck with: "08:25
SvenKieskeTASK [rabbitmq : Check if RabbitMQ quorum queues need to be configured]"08:25
SvenKieske""msg": "om_enable_rabbitmq_quorum_queues is True but non-quorum queues have been found. Currently the procedure to migrate to quorum queues is manual."08:26
SvenKieskethat test should not fail imho.08:26
SvenKieskeadded the information to the change itself08:29
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Just test rally baby  https://review.opendev.org/c/openstack/kolla-ansible/+/91372808:31
kevko\o/08:31
kevkoshouldn't we include wrapper into fluentd image instead of labels use ? 08:33
SvenKieskewhat's the benefit?08:35
SvenKieskeMy guess would be that the mechanism for retrieving a container label maybe changes every 5-10 years. a wrapper around fluentd I expect I need to change every 6 month - 3 years, because they like to change everything08:36
SvenKieskeand I have more code with more bugs -> less code, coder happy :)08:36
SvenKieskethis reminds me of this excellent blog post ;) https://grugbrain.dev/08:38
mnasiadkawe fix it as it is now, and we drop label check in master afterwards08:45
mnasiadkait lives only in stable/2023.208:46
mnasiadkano sense in wasting more energy on it08:46
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/2023.2: Rework quorum queues precheck  https://review.opendev.org/c/openstack/kolla-ansible/+/91380708:46
mnasiadkaSvenKieske: ^^ this will fix prometheus jobs in 2023.208:46
SvenKieskeah, nice! this was missing, I thought about it, wondering "didn't we have something to fix this somewhere"? didn't realize it was missing in that branch,ty!08:48
mnasiadkawhat is funny, we have that in 2023.108:48
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/2023.2: Rework quorum queues precheck  https://review.opendev.org/c/openstack/kolla-ansible/+/91380708:48
mnasiadkarebased on the docker 26 fix08:48
SvenKieskeI have seen this pattern now at least more than once. do we have something missing in the process when cutting a release? do we need some freeze period or something? we had multiple changes missing from stable branche, but only some each time.08:51
SvenKieskebranches*08:51
SvenKieskebtw, something seems seriously broken when you just set "enable_ovn: 'yes'" in the ovn scenario: https://review.opendev.org/c/openstack/kolla-ansible/+/913838?tab=change-view-tab-header-zuul-results-summary08:54
SvenKieskeso as an addition, the usual ovn flags of course also apply, I just added this one and more than 50% of jobs failed (yesterday, so hopefully not related to docker 26)08:55
SvenKieskelooking into why now :)08:56
SvenKieskemhm, some of this might indeed be related to the docker breakage :(09:06
mnasiadkaenable_ovn will only deploy OVN09:06
mnasiadkaNeutron will not use it09:06
mnasiadkaneutron_plugin_agent is the variable you want to use09:07
mnasiadkahttps://docs.openstack.org/kolla-ansible/latest/reference/networking/neutron.html#ovn-ml2-ovn09:07
SvenKieskemnasiadka: I just added it to the ovn scenario in globals-default.j2, neutron_plugin_agent is already set there, pls look at the actual code I post before giving advice :P09:08
mnasiadkaenable_ovn is not needed then09:08
SvenKieskebut it also shouldn't break anything if it's added09:09
mnasiadkaI'm not going to speculate, but there's no way it can break anything :)09:10
SvenKieskeI honestly wonder what the flag itself is good for, I know it's used internally in the playbooks to check for ovn etc. but this whole construct of "when is ovn enabled" seems to be a little brittle09:10
SvenKieskemaybe I just remove it from the ovn-exporter change, as it should not be needed there.09:10
SvenKieskeafaik neutron_plugin_agent also enables enable_ovn right?09:11
SvenKieskeyes, just looked at it again09:11
mnasiadkayes, enable_ovn is enabled when neutron_plugin_agent == ovn09:12
mnasiadka(by default)09:12
SvenKieskeand neutron is enabled :D09:13
opendevreviewMatt Crees proposed openstack/kolla-ansible master: CI: drop RMQ reconfigure step in queue migrations  https://review.opendev.org/c/openstack/kolla-ansible/+/91387609:14
SvenKieskeI guess I'm one step closer to fix the ovn-exporter now09:14
opendevreviewMatt Crees proposed openstack/kolla-ansible master: CI: Only migrate RMQ queues during SLURP  https://review.opendev.org/c/openstack/kolla-ansible/+/90997109:20
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Just test rally baby  https://review.opendev.org/c/openstack/kolla-ansible/+/91372809:22
opendevreviewMatt Crees proposed openstack/kayobe master: CI: drop RMQ upgrade step in queue migrations  https://review.opendev.org/c/openstack/kayobe/+/91387809:25
opendevreviewMatt Crees proposed openstack/kolla-ansible master: CI: Only migrate RMQ queues during SLURP  https://review.opendev.org/c/openstack/kolla-ansible/+/90997109:39
jovialmnasiadka: re https://bugs.launchpad.net/kolla-ansible/+bug/2058512 , I was just seeing it kayobe CI in the patch trying to bump the ansible version. There was an oddity around the way we were doing the rabbit queue migration which meant we running new kayobe-config (which has references to kolla_ansible_source_version) with the old version of kayobe. When I tied to workaround that, this one popped up :D09:49
kevkoSvenKieske: mnasiadka: To which patch i can rebase to fix fluentd ? 10:07
kevkoah, I see10:10
mnasiadkafirst the 2023.2 one needs to merge to fix upgrade jobs10:10
SvenKieskekevko: this is the 2023.2 backport: https://review.opendev.org/c/openstack/kolla-ansible/+/913805 this is master: https://review.opendev.org/c/openstack/kolla-ansible/+/913868?usp=cherry-pick10:10
mnasiadkafrickler: redis license changed, seems the sensible thing to do is to support zookeeper for coordination?10:24
opendevreviewMatt Crees proposed openstack/kayobe master: CI: rework RMQ steps for queue migrations  https://review.opendev.org/c/openstack/kayobe/+/91387810:24
opendevreviewMatt Crees proposed openstack/kayobe master: CI: rework RMQ steps for queue migrations  https://review.opendev.org/c/openstack/kayobe/+/91387810:26
fricklermnasiadka: SvenKieske mentioned this internally, too. zookeeper, etcd, some redis clone? guess we need some research and discussion at the PTG?10:29
opendevreviewMatt Crees proposed openstack/kayobe master: Bump KA Ansible versions to match new defaults  https://review.opendev.org/c/openstack/kayobe/+/91357110:31
opendevreviewMatt Crees proposed openstack/kayobe master: Bump up Ansible supported versions to 8.x/9.x  https://review.opendev.org/c/openstack/kayobe/+/91051310:31
opendevreviewMatt Crees proposed openstack/kayobe master: Use new collections in Kayobe  https://review.opendev.org/c/openstack/kayobe/+/91074210:31
mnasiadkafrickler: we tried etcd in multinode jobs, it has issues with slow disks10:33
mnasiadkafrickler: probably easiest is to revert zookeeper removals and bump up to newer version10:33
SvenKieskemhm, really sad, we have all the recent work to enable TLS for redis as caching backend. I don't know if zookeeper in openstack can be a complete replacement? from the oslo.cache perspective afaik it's their main implementation, so that part seems good10:33
mnasiadkayeah, let's discuss on PTG10:34
fricklerin running zookeeper for our local zuul setup I'm seeing issues with slow disks, too, so not sure whether that makes much of a difference10:36
mnasiadkaI thought zookeeper is similar to redis in mainly keeping data in memory?10:37
fricklerI have not digged deeper yet, might totally be possible that it's just a matter of (im)proper configuration10:40
opendevreviewMatt Crees proposed openstack/kayobe master: CI: rework RMQ steps for queue migrations  https://review.opendev.org/c/openstack/kayobe/+/91387810:44
opendevreviewMatt Crees proposed openstack/kayobe master: Bump KA Ansible versions to match new defaults  https://review.opendev.org/c/openstack/kayobe/+/91357110:44
opendevreviewMatt Crees proposed openstack/kayobe master: Bump up Ansible supported versions to 8.x/9.x  https://review.opendev.org/c/openstack/kayobe/+/91051310:44
opendevreviewMatt Crees proposed openstack/kayobe master: Use new collections in Kayobe  https://review.opendev.org/c/openstack/kayobe/+/91074210:44
kevkois it really needed ? 10:51
kevko4. 10:52
kevko You may convey verbatim copies of the Program's source code as you10:52
kevko  receive it, in any medium, provided that you conspicuously and10:52
kevko  appropriately publish on each copy an appropriate copyright notice; keep10:52
kevko  intact all notices stating that this License and any non-permissive terms10:52
kevko  added in accord with section 7 apply to the code; keep intact all notices10:52
kevko  of the absence of any warranty; and give all recipients a copy of this10:52
kevko  License along with the Program.  You may charge any price or no price for10:52
kevko  each copy that you convey, and you may offer support or warranty10:52
kevko  protection for a fee.10:52
kevkoIt's still free but it just means that we can't provide service to third person ...which we don't ...as it's internal system ...10:53
kevkoIt would be a pity to replace it with something else - zooekeeper or etcd3.10:54
opendevreviewMark Goddard proposed openstack/kayobe master: Prevent running from a different Kayobe configuration repository  https://review.opendev.org/c/openstack/kayobe/+/90313910:55
kevkoQ: I am the author of an open-source project that uses Redis in a non-competitive way. If someone else uses my project to produce a competitive product or hosted service (e.g., starts using my project in their SaaS solution), am I at risk of being considered competitive and violating the dual license? Do I need to track all users of my project and11:00
kevkoreport suspected infringing use?11:00
kevkoA: Only those who are embedding or hosting Redis products in a competitive manner are in violation of the license. The violation does not extend to a project owner who is not doing so and does not require asking others to do so on its behalf. The project owner has no obligation to monitor or report on how others are using their project.11:00
kevkoSo we really don't need to replace redis ...as it is really working nice11:00
opendevreviewMark Goddard proposed openstack/kayobe master: Support dict format IP routing rules on CentOS/Rocky  https://review.opendev.org/c/openstack/kayobe/+/89994111:19
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Just test rally baby  https://review.opendev.org/c/openstack/kolla-ansible/+/91372811:31
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Just test rally baby  https://review.opendev.org/c/openstack/kolla-ansible/+/91372811:33
opendevreviewMerged openstack/kayobe master: CI: Make SLURP jobs non-voting  https://review.opendev.org/c/openstack/kayobe/+/91382511:36
fricklerkevko: isn't that essentially the same change that elasticsearch made?11:43
opendevreviewMerged openstack/kolla-ansible stable/2023.2: common: Fix fluentd labels when using Docker 26  https://review.opendev.org/c/openstack/kolla-ansible/+/91380511:51
kevkofrickler: yeah similar - we didn't need to replace elastic i would say ... 12:04
kevkofrickler: It's mainly because they don't want to let big companies take open-source products, slap their branding on them, and offer them strictly as a paid service12:07
fricklerkevko: yes, which IMO is a reason to consider them to be non-free and move away12:09
opendevreviewMatt Crees proposed openstack/kayobe master: Use new collections in Kayobe  https://review.opendev.org/c/openstack/kayobe/+/91074212:10
kevkoRSALv2 is a permissive, non-copyleft license created by Redis Ltd., allowing users the right to “use, copy, distribute, make available, and prepare derivative works of the software” and has only two primary limitations. You may not:12:10
kevkoCommercialize the software or provide it to others as a managed service12:10
kevkoRemove or obscure any licensing, copyright, or other notices12:10
kevkoNote that RSALv2 has not been approved by the OSI, and we do not refer to it as an Open Source license.12:10
kevkofrickler: okay, let's drop also redhat derivatives ..because who knows how long there will be OSes as rocky ,almalinux ...etc12:11
fricklerbig +1 to that12:11
kevkofrickler: mee too :D 12:11
kevkomy opinion is just don't do anything if it is still opensource - if they will close in some point ..there will be still enough time to reimplement in one cycle12:12
kevkoor in 2 years - as our images are build from stable os releases - ubuntu, debian .... which has redis as package inside with older version12:13
kevkofrickler: btw, do you know how to live migrate centos libvirt -> debian libvirt ? ... i think it's not possible even if I will build libvirt with machine aliases (as Redhat guys modifiing machine types every version :D ...) because there is a different size of ... (i don't remember what ...) ...so i think there is only option to cold migrate ...am12:15
kevkoI right ? 12:15
kevkotox broken :) 12:21
kevkohttps://zuul.opendev.org/t/openstack/build/ecf58304cbc24b58818da664def21d0512:21
fricklerah, that's the new py312 job, nice. and I heard this about centos multiple times, but have not first hand experience with it12:23
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Just test rally baby  https://review.opendev.org/c/openstack/kolla-ansible/+/91372812:23
kevkofrickler: for now we have thousands VMs running on centos stream libvirt and don't know how to finally migrate ...whole cloud is on ubuntu but libvirt is centos :D 12:25
kevkofrickler: it's also inpossible for now to migrate to newer libvirt from centos ..because they releases buggy libvirt :D 12:25
kevkofrickler: podman rabbitmq failing https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a28/913868/1/check/kolla-ansible-ubuntu-podman/a28112d/primary/logs/kolla/rabbitmq/rabbit%40primary.txt << fluentd patch 12:55
opendevreviewMark Goddard proposed openstack/kolla-ansible master: Use the running MariaDB server image for backups  https://review.opendev.org/c/openstack/kolla-ansible/+/91390113:34
opendevreviewMerged openstack/kolla-ansible stable/2023.2: Rework quorum queues precheck  https://review.opendev.org/c/openstack/kolla-ansible/+/91380713:43
SvenKieskefrickler (or anybody core): this change has w+1 and should imho be gated since 2 days but gerrit doesn't move? https://review.opendev.org/c/openstack/kolla/+/90518914:04
fricklerSvenKieske: doesn't need a core to see the relation chain ;)14:15
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/2023.2: CI: Increase galera node timeouts  https://review.opendev.org/c/openstack/kolla-ansible/+/91381014:18
SvenKieskeah right14:19
kevkoSvenKieske: relation chain 14:27
kevkoaa..sorry blind 14:27
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible stable/2023.1: CI: Increase galera node timeouts  https://review.opendev.org/c/openstack/kolla-ansible/+/91381114:28
kevkodo anyone know why we are not installing cinder by default ? 14:29
kevkohttps://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_10f/913728/13/check/kolla-ansible-ubuntu/10f47a9/primary/logs/kolla/index.html14:29
mnasiadkawe are, in multinode jobs14:32
mnasiadkabut true, by default we don't deploy cinder - it's not required ;-)14:32
opendevreviewGaël THEROND proposed openstack/kolla-ansible master: Fix keystone service configuration for haproxy when federation is enabled.  https://review.opendev.org/c/openstack/kolla-ansible/+/91390814:33
SvenKieskemight be a rather fringe setup though, openstack without cinder, maybe for some bootstrap scenarios/underclouds relevant14:33
opendevreviewGaël THEROND proposed openstack/kolla-ansible master: Fix keystone configuration for haproxy.  https://review.opendev.org/c/openstack/kolla-ansible/+/91390814:35
Fl1nthum... Am I missing something or the CI is broken for kolla? https://review.opendev.org/c/openstack/kolla/+/913795 14:42
SvenKieskeyeah, there are some hickups currently (docker has removed some functionality, but that should afaik be fixed)14:47
SvenKieskeFl1nt: that being said please don't use so called "bare rechecks" (so just writing "recheck"). there should always be a reason stated because the release team actually checks why rechecks are being done and tries to improve CI based on this information14:48
SvenKieskeso e.g. write "recheck unrelated breakage in docker-engine" if that is the case, that would be most kind14:49
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Just test rally baby  https://review.opendev.org/c/openstack/kolla-ansible/+/91372814:50
SvenKieskeI still wanted to ask the guys doing this work if they just can't completely disable bare rechecks..would be more efficient imho14:50
opendevreviewVerification of a change to openstack/kayobe master failed: Fix the glob for the custom RabbitMQ configuration  https://review.opendev.org/c/openstack/kayobe/+/90911314:51
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Just test rally baby  https://review.opendev.org/c/openstack/kolla-ansible/+/91372814:53
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Just test rally baby  https://review.opendev.org/c/openstack/kolla-ansible/+/91372814:55
opendevreviewRafal Lewandowski proposed openstack/kayobe master: Add Redfish rules to Ironic and Bifrost introspection  https://review.opendev.org/c/openstack/kayobe/+/90277214:58
Fl1ntSvenKieske, thanks!15:18
Fl1ntSvenKieske, problem is we can't really do that sometimes as when IPs break or remote repository just deny the requests etc.15:19
SvenKieskecan't do what?15:22
Fl1ntLike, in here, seems like container timed out but... why I can't really know15:22
SvenKieskeyou can always write "recheck remote mirror $foo has an network issue" no?15:22
SvenKieskethen write container timed out :)15:22
SvenKieskekevko: this is the short form why we can't use SSPL, it's forbidden by openstack governance: https://governance.openstack.org/tc/reference/licensing.html15:23
Fl1ntSvenKieske, Well, yes, but ain't really help them that much, they'll see shit ton of: "FluentD container didn't restarted properly" or "FluentD container restart task failed contacting host" but are there someone actually looking at the error?15:25
SvenKieskethey can start yell at fluentd :D or we can start looking for alternatives if we realize 90% of CI breakage is in one program (only hypothetically speaking)15:26
Fl1ntLike, in this one, if those remote host not responding is happening often, how could we have a working zuul process?15:26
Fl1ntI mean, zuul workers are benevolent offers from OVH & co right?15:27
SvenKieskethe first step is always to collect data what even fails and how often, then people can look into why and alternatives. without data (and recheck without any further info is no data) we can't do much. seems rather straight forward to me?15:27
Fl1ntSvenKieske, I know that, but I'm not working on the CI, if I need to debug OS/KOLLA + the CI... I'll not have that much of free time after work :D15:28
SvenKieskeno, all that was ask that you can provide the people who do the debugging a little bit more info by sacrificing 10 seconds maybe to just write the concrete failure when doing rechecks15:30
SvenKieskeI also wish we had more automatic failure analytics (any at all), but well15:30
Fl1ntSvenKieske, Of course I'll do it, I mean, I'm not that in a hurry, but I'm already having way too much time involved on OS (overall, not just kolla/ka) to my taste right know, so, I'm contributing back upstream patches and bugs we catch on, but I'll not make extra effort on that specific topic tho.15:32
SvenKieskesure, everybody does what they can/want :)15:34
SvenKieskeit's stll a rather cheap solution I'd say, if you compare to alternatives ;) I mean you are probably big enough that broadcom might consider giving you a vmware licence ;)15:35
SvenKieskebut I guess there you just need to sit around and wait until they fix reported bugs 3 years later15:35
Fl1ntIndeed! I'm a bit fed up at the moment as stakeholders are starting really heavily questioning our involvment on OS/Linux/HW right now, nobody is indeed willing to understand what you've just write.15:36
Fl1ntBroadcom is killing VMWare, we're doubling down on OS because of that15:37
Fl1ntBUT15:37
Fl1ntYou know how it is15:37
Fl1nteveryone want everything, right know, for free, but can't spend a dime on it...15:38
Fl1ntI'll have strategic meetings next week about that specific topic as Broadcom as for a x10 increase in licensing price, I'm all in to push OS, but yet the stakeholders need to understand we need budget. But as they freezed any spends for 2024... I'm... a bit concerned.15:39
Fl1nt*Broadcom asked for15:40
SvenKieske:D15:40
SvenKieskeyeah, I feel you.15:40
Fl1ntAnyway, I'll try have more explicit rechecks ;-)15:41
SvenKieskethanks, and I try to improve CI so we don't need that many rechecks :)15:42
Fl1nt;_)15:42
SvenKieskeno guarantees, though ;)15:42
Fl1ntyeah no worries, I was just asking as I saw twice in a row issues that aren't related to the code/bugfixes themselves.15:43
kevkoSvenKieske: what is SSPL  ? 15:48
kevkoSvenKieske: document you've sent >> https://governance.openstack.org/tc/reference/licensing.html <<   If you say that we must comply with licensing according to that document where it is written "Licenses considered incompatible with this requirement include GPLv2, GPLv3, and AGPL." ... so  does that mean we cannot use even the Linux kernel, which15:50
kevkois licensed under GPLv2? :D that's what you are trying to say ? 15:50
kevkoSvenKieske: and what about mariadb ? are we going to drop it ? https://github.com/MariaDB/server?tab=GPL-2.0-1-ov-file#readme << it's also GPLv2 15:51
kevkoSvenKieske: so, we are fine I would say 15:57
SvenKieskekevko: redis is under sspl: https://redis.com/legal/server-side-public-license-sspl/ and sspl is not OSI approved: https://opensource.org/blog/the-sspl-is-not-an-open-source-license16:00
kevkoSvenKieske: where you can find it is under SSPL ? 16:01
SvenKieskekevko: what you cite is about external libraries. not a lawyer but afaik you need to read further "Projects run as part of the OpenStack Infrastructure (in order to produce OpenStack software) may be licensed under any OSI-approved license."16:01
kevkoSvenKieske: there is this 16:01
kevko"After this date, contributions are subject to the user's choice of16:01
kevkothe Redis Source Available License v2 (RSALv2) or the Server Side Public License v1 (SSPLv1), as16:01
kevkofollows:"16:01
kevkoOR is important 16:01
SvenKieskekevko: https://redis.com/blog/redis-adopts-dual-source-available-licensing/ and their git repo has the licence change already :)16:02
SvenKieskethe RSALv2 afaik is even more weird and afaik not OSI approved, but I won't honestly bother to check. the copyright system in it's current form still seems to be doomed.16:03
SvenKieskethe old licence was 3 clause bsd, that is what e.g. keydb still uses (a redis fork)16:03
SvenKieskethat _is_ an OSI approved license.16:04
kevkohttps://redis.com/blog/redis-adopts-dual-source-available-licensing/ << here >> Under the new license, cloud service providers hosting Redis offerings will no longer be permitted to use the source code of Redis free of charge. For example, cloud service providers will be able to deliver Redis 7.4 only after agreeing to licensing terms with Redis,16:05
kevkothe maintainers of the Redis code. These agreements will underpin support for existing integrated solutions and provide full access to forthcoming Redis innovations.16:05
SvenKieskeI guess this is really a question for the TC, if at all :) I'm grateful if I don't need to dump more time into this useless stuff16:05
mnasiadkaSvenKieske: wonder if keydb is a drop-in replacement for tooz or not16:05
SvenKieskemnasiadka: they are a real fork of redis source code, afaik 2-4 years ago. they have added multithreading and stuff but the protocol should be the same. but I have never used it myself, only read about it online.16:06
kevkoSo, simply said by them -  YOU as cloud provider who earn money ... you need to agree licensing ....  , you who are maintain your cloud inhouse as private cloud with our redis -- you are fine 16:06
SvenKieskekevko: yeah but we - as openinfra -  have rules what licensed software we use and what we don't use.16:07
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Just test rally baby  https://review.opendev.org/c/openstack/kolla-ansible/+/91372816:07
SvenKieskethat being said, I wonder how vmware support in nova ever worked, last time I looked most of their stuff was completely proprietary. apart from some low level stuff16:07
mnasiadkathe python libraries they wrote were also proprietary?16:08
SvenKieskeanyway I really will now stop with this discouraging topic and move to the other discouraging topic of ovn-exporter again :D16:08
mnasiadkaoslo.vmware was Apache license, right? :)16:08
SvenKieskeI don't know. I don't know how the integration worked. I also don't care. But to provide shims to avoid licensing issues smells wrong to me. I'm more in line with Torvalds on that I guess.16:09
SvenKieskeand you see where it end, commercial entities just want the free lunch, most of the time. vmware support is dying.16:10
opendevreviewSven Kieske proposed openstack/kolla-ansible master: Add ovn-exporter  https://review.opendev.org/c/openstack/kolla-ansible/+/85549816:11
opendevreviewMartin Hiner proposed openstack/kolla-ansible master: Add container engine migration scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/83694116:14
kevkoSvenKieske: like ? https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3d4/913728/16/check/kolla-ansible-ubuntu/3d49338/primary/logs/tempest/reports/tempest-smoke.1.html16:15
SvenKieskekevko: I don't understand the link without context, do you have any context, please? This seems to neither be about vmware, redis, or ovn-exporter.16:21
kevkoSvenKieske: tempest testing :) 16:21
SvenKieskethat's nice, did you see mnasiadkas wip patch for that?16:21
SvenKieskeI don't want to have two different tempest tests one day :D16:22
kevkoSvenKieske: https://review.opendev.org/c/openstack/kolla-ansible/+/91372816:22
SvenKieskekevko: yeah I saw your change, I'm talking about https://review.opendev.org/c/openstack/kolla-ansible/+/87882616:24
SvenKieskedon't know if this will conflict or if it is abandoned effectively. I'd suggest you two talk about it, maybe :)16:24
kevkowell, i really prefer rally ... why ? because it's very simple ...can generate HTML reports which is usefull as you can see above ^  ...and you can easily setup for example  job for run, boot, remove server  ..and write to 5 lines json which you run with rally and create report ...that's it 16:26
SvenKieskeyeah, looks nice! I remember back in the day we had tempest running at $oldjob and also looked into using rally16:27
kevkorally is cool yeah ...16:27
kevkonow i am looking into code and will replace some bashes with scenarios from rally 16:27
kevkoall will be in one folder as report in html ..with traceback ...everything 16:28
SvenKieskehow do they still run on an old site layout though? :D https://docs.openstack.org/rally/latest/16:28
SvenKieskeah their canonical docs url is https://rally.readthedocs.io/en/latest/16:29
kevkoSvenKieske: It will be fragrant, in pale pink.16:29
kevkoSvenKieske: i don't know ... once I've seen the docs ...i closed it and just read the code :D 16:30
kevkobecause it's totally obsolete ...16:30
kevko*outdated 16:31
SvenKieske:D16:31
mnasiadkakevko: if you could write an Ansible role instead of running bash, then probably me and SvenKieske would be happier ;)16:39
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Template out kolla-build.conf using role  https://review.opendev.org/c/openstack/kolla-ansible/+/90377116:42
opendevreviewMichal Nasiadka proposed openstack/kolla-ansible master: WIP: CI: Template out kolla-build.conf using role  https://review.opendev.org/c/openstack/kolla-ansible/+/90377116:43
kevkomnasiadka: haha 😂16:44
kevkoFirstly I wanted to spin rally containers on controllers and write just Ansible role to run_once and write python flask api to get report ...as it is in db which we have under galera ..so you can run test from Kolla-Ansible 16:45
kevkoBut it was overkill for now :) 16:45
kevko*to play against group with run_once flag16:46
kevkoBenchmarkAndTestAsService 🤭16:46
Fl1ntrally and tempest where deprecated and removed a while back, do you really want to reintroduce them using bash, then tight those two services to the "framework"? That is really heading away from original kolla idea, and IMHO, not that clever.16:54
Fl1ntTBH, I really think kolla/k-a should focus on being really good at OS lifecycle in a first place and let everything else satellite to be managed outside of it.16:56
Fl1ntlike... Grafana, Opensearch, Prometheus, etc.16:56
SvenKieskecan someone kick this change over the line? https://review.opendev.org/c/openstack/kolla-ansible/+/913868 looking at CI all voting jobs passed, except debian slurp upgrade, which should finish any moment. sorry for being impatient but the unrelated CI failures are annoying today :)16:58
Fl1ntbasically get a OS lifecycle framework as: kolla-build (Create images only) / kolla-install (Deploy/Upgrade services from images) / kolla-config (Configure Openstack such as flavors/images/settings).16:58
SvenKieskeI don't know, smaller shops are not very good with setting up their own ELK stack, from a big corp background I can understand that, because you likely already have your huge ELK stack running somewhere else17:00
Fl1ntthen add kolla-satellite to the count and tada magic happens17:00
Fl1ntbut at least that would avoid confusion and the code base to become cluttered and very tricky to maintain/test17:00
SvenKieskenot sure what you really mean by that? what kind of satellite? who maintains that?17:01
Fl1ntFor instance17:01
Fl1ntlibvirt exporter is a living dead17:01
mnasiadkaSvenKieske: it has w+1, once it passes it will gate again...17:01
Fl1ntforks over forks over deprecation and yet the latest is now diying again.17:01
Fl1ntSvenKieske, Satellite like the concept, not like RHEL stuff ^^17:02
SvenKieskehttps://github.com/inovex/prometheus-libvirt-exporter is dying?17:02
SvenKieskeseems alive and kicking to me: https://github.com/prometheus-community/community/issues/5017:04
Fl1nthold on a sec, I'll gave you the whole picture of this exporter and you'll understand why companies are fed up with it.17:05
opendevreviewKevin Tindall proposed openstack/kolla-ansible master: Add TLS proxy for novncproxy  https://review.opendev.org/c/openstack/kolla-ansible/+/91114117:07
Fl1ntSo, inovex, come from tinkoff, that came from rumanzo itself forked from kumina itself coming from an obscure shell script before that. In here we're at the fith iteration of this exporter, it broke any upgrade process on any serious company that are a bit regarding about security.17:08
Fl1ntSvenKieske, we should probably update the exporter on master with the new exporter I guess tho :D17:09
SvenKieskebut none of them helped maintain it I guess. and I still don't see how isolating "third party" stuff from openstack deployment is a net benefit to providing a prod deployment ready solution to provision openstack? at least you need an alternate libvirt exporter then17:09
Fl1ntsorry sixth iteration, did forget about the zhangjianweibj one :D17:09
SvenKieskeso your solution to maintenance is "don't ship anything" whic is no solution at all. only for large corps who have apparently their own internal tools for that17:10
SvenKieskeso either you have an internal proprietary exporter that does better that you don't open source. or you integrate it yourself in a different way or you just don't bother with monitoring.17:11
Fl1ntSvenKieske, nope, you don't, logging and metrics are supposed to be provided by the telemetry project on OS if you really want to be OS oriented, everything Grafana/Prometheus is a plus that should be external and companies should manage it on their own as they most of time do it anyway, that was the reason behind the "external" concept a few years back.17:11
Fl1ntYou want to strip optionals, not make them tightly coupled with the core.17:12
SvenKieskeso are you saying you use telemetry project (the actual projects would be aodh and ceilometer I guess)?17:13
Fl1ntNope, we use both, telemetry on some clusters and grafana/prom on others.17:14
Fl1ntbut the point is17:14
Fl1ntnot to deprecate or unsupport for grafana/opensearch/prometheus, the point is to bundle externalities on a specific kolla project17:15
SvenKieskethis is imho about real world oss deployments. real world deployments use ELK, not aodh. we also use docker and podman, because those are popular. by the same logic you could want us to provide the infrastructure containers via openstack zun instead of docker17:15
Fl1ntcall it kolla-addons if you prefer ^^17:15
SvenKieskelinux is also external, it's certainly not an openstack project, as are all the used distros.17:16
Fl1ntSvenKieske, I'm a real world deployment, we use both, depends on many more than just technicals inclination over solutions.17:16
Fl1ntLinux is a requirement...17:16
Fl1ntDon't mix everything, without linux nothing works, without those services your OS is perfectly fine.17:16
SvenKieskeIf you want to propose a solution I guess PTG would be the right point to prepare a talk or something - you don't really need to prepare anything - and then put in the work and show that it's beneficial I guess17:17
SvenKieskeI guess the rest of us is busy keeping the lights on17:18
Fl1ntYep, I think that thinking about kolla as a framework with different specialized part would be beneficial for a lot of people17:18
SvenKieskeit might be worth to split it up, it also might not be, I don't know. I know you can easily not deploy opensearch today with a single toggle `enable_central_logging: no`17:19
SvenKieskeand whatever else the solution is, it should at least be that easy to enable/disable something17:19
Fl1ntThat is just a discussion and suggestions, that is not a rant or anything tho, I'm more than able to patch/maintain kolla on our own, it's just that we participate back a bit to upstream from time to time.17:20
SvenKieskesure. but if you propose: "hey if we do this huge amount of work, x, y and z will be better" than you need to: a) convince many people b) pay people c) do the work yourself17:21
SvenKieskethat's a universal rule of open source development or even any development I would say. nothing special with regards to openstack or kolla17:21
Fl1ntSvenKieske, the problem isn't the switch on, switch off, we could start another kolla-X repo the same way kolla-collection was made or even the kayobe and other dependencies.17:22
SvenKieskeany combination of a), b) or c) will suffice17:22
SvenKieskesure17:22
kevkofrickler: rally, tempest are deprecated ? 17:22
Fl1ntMy main issue is, who can create new repo with kolla's name on opendev?17:22
SvenKieskelast time I looked at least tempest was not?oO17:22
kevkoSvenKieske: i think frickler meant it's deprecated in kolla, kolla-ansible as we removed it "because it is not a service" 17:23
SvenKieskeFl1nt: well that's not really a hard requirement, is it? the governance board has some rules around that. you could always start with a github repo17:23
Fl1ntI was refering to kolla 17:23
Fl1ntSvenKieske, no, the idea is to stick with kolla, if I'm starting a kolla fork it doesn't worth anything.17:24
SvenKieskeFl1nt: there's a doc for that: https://docs.opendev.org/opendev/infra-manual/latest/creators.html17:24
fricklerI'm lacking context, where did I say that? but rally is hanging on the edge of being abandoned with just a single contributor mostly. tempest isn't doing much better though17:24
SvenKieskespecifically under the kolla namespace I guess you would need buy in from PTL (mnasiadka) and possibly some core reviewers17:24
Fl1ntSvenKieske, that's my point, hence why a PTG discussion/presentation around this topic would be nice before starting to put any effort on this.17:25
kevkofrickler: okay , Fl1nt said that above :) .... ^^ "rally and tempest where deprecated and removed a while back, do you really want to reintroduce them using bash, then tight those two services to the "framework"? That is really heading away from original kolla idea, and IMHO, not that clever."17:26
kevkosorry17:26
kevkoFl1nt: yeah, that's the reason why i dropped it :D 17:26
Fl1ntfrickler, kevko was referring to the fact I told him tempest and rally were deprecated few releases cycles ago, now he want to reintroduce them in, that triggered that whole discussion about responsability splits :D17:26
SvenKieskeFl1nt: sure! discussion is always good! add your topic (with name or nick please, so we know who wanted to talk about it) here: https://etherpad.opendev.org/p/kolla-dalmatian-ptg17:27
SvenKieskewell, my idea with tempest was, that it has a decent openstack testsuite (afaik is still run on many upstream CIs) so we don't need to rewrite that many tests if we want to get rid of "test-core-openstack.sh" and friends17:28
SvenKieskealso it has afaik way deeper test coverage than our handwritten bash stuff. I know writing tests is a lot of work, so I really appreciate every single test that we have, but imho we have not really deep test coverage, especially when it comes to end-to-end and integration testing.17:30
Fl1ntI'm not really against it tho, I just point out the lack of maintainers on it back in time and even on the upstream project itself.17:30
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Just test rally baby  https://review.opendev.org/c/openstack/kolla-ansible/+/91372817:52
Fl1ntmnasiadka, added the tooz documentation about API compatibility for info on etherpad as choice of replacement "Might" broke services that even using tooz call for specific API features not supported on all backends.18:01
Fl1ntThinking about designate for instance. I don't know if the 'Only redis is currently supported as a backend for designate coordination' is still relevant.18:02
opendevreviewVerification of a change to openstack/kolla-ansible stable/2023.2 failed: CI: Increase galera node timeouts  https://review.opendev.org/c/openstack/kolla-ansible/+/91381018:25
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Just test rally baby  https://review.opendev.org/c/openstack/kolla-ansible/+/91372818:52
opendevreviewMerged openstack/kolla-ansible stable/2023.1: CI: Increase galera node timeouts  https://review.opendev.org/c/openstack/kolla-ansible/+/91381119:03
opendevreviewMerged openstack/kolla-ansible master: common: Fix fluentd labels when using Docker 26  https://review.opendev.org/c/openstack/kolla-ansible/+/91386819:32
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [DNM] Py312 test  https://review.opendev.org/c/openstack/kolla-ansible/+/91392819:34
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Just test rally baby  https://review.opendev.org/c/openstack/kolla-ansible/+/91372821:03
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Just test rally baby  https://review.opendev.org/c/openstack/kolla-ansible/+/91372823:20

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!