| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Bump mariadb_monitor_ping_timeout to max https://review.opendev.org/c/openstack/kolla-ansible/+/950434 | 05:14 |
|---|---|---|
| opendevreview | Michal Nasiadka proposed openstack/kolla master: WIP: Move to CentOS Stream 10 https://review.opendev.org/c/openstack/kolla/+/950392 | 05:25 |
| kevko | mnasiadka: if you trying to tune proxysql somehow in CI when container is down during the mariadb upgrade ...I am afraid that it's impossible ..because there are another mechanisms which are checking availability of node ...not only ping | 07:50 |
| mnasiadka | No, it's not the case - many times we just have constrained CI and networking/CPU/whatever causes issues - bumping up ping values is helping | 07:51 |
| mnasiadka | but it seems our multinode is not only suffering from that... | 07:51 |
| mnasiadka | (sometimes proxysql, sometimes rabbitmq, sometimes something else) | 07:51 |
| mnasiadka | trying to get to a stable state with these jobs | 07:51 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Bump mariadb_monitor_ping_timeout to max https://review.opendev.org/c/openstack/kolla-ansible/+/950434 | 07:52 |
| kevko | Ah, I don't think proxysql issue is problem in multinode ... (But I can't say if it is not only in CI) ....but we have production multinodes clusters ..without problems for years now with proxysql ... | 07:53 |
| kevko | mnasiadka: so it will be something different | 07:53 |
| kevko | I would say .. | 07:53 |
| mnasiadka | yeah, that only happens in CI - you're not running controllers on 8GB RAM VMs with oversubscribed everything | 07:53 |
| mnasiadka | and lousy network | 07:53 |
| kevko | mnasiadka: I have 16 :) on my tests | 07:58 |
| kevko | mnasiadka how much CPUs have nodes with 16 ? | 07:58 |
| mnasiadka | we don't have 16GB nodes, at least not yet. | 07:58 |
| kevko | Ah, sorry I meant 8 - how many CPUs ? | 08:00 |
| mnasiadka | 8 | 08:22 |
| frickler | or sometimes 4, but modern, fast ones | 08:35 |
| mnasiadka | we could try switching multinode jobs to nested-virt labels - we would then get the modern ones - let me try | 08:43 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: DNM: Test multinode on nested virt labels https://review.opendev.org/c/openstack/kolla-ansible/+/950492 | 08:45 |
| kevko | mnasiadka: well, I think the issue is too much RAM against CPU | 09:18 |
| kevko | mnasiadka: i think it should be 1/4 2/8 4/16 | 09:19 |
| kevko | frickler: can I ask you for review please ? https://review.opendev.org/c/openstack/kolla-ansible/+/949812 second +2 | 09:21 |
| opendevreview | Michal Nasiadka proposed openstack/kolla-ansible master: DNM: Test multinode on nested virt labels https://review.opendev.org/c/openstack/kolla-ansible/+/950492 | 09:23 |
| kevko | mnasiadka: where I can see nodesets ( or labels ) ? To check ram/cpu | 09:27 |
| mnasiadka | You can't, these are only flavor names on respective clouds - which sometimes have these values in the name, and sometimes not | 09:28 |
| mnasiadka | And I don't think we can influence anything | 09:28 |
| mnasiadka | With new Zuul-launcher instead of nodepool there will be some more options with 16G of RAM | 09:28 |
| mnasiadka | But frickler is closer to these definitions ;-) | 09:28 |
| kevko | i think you need 1/4 CPU/RAM | 09:29 |
| kevko | ideally 4/16 | 09:30 |
| kevko | I would try 2/8 for example | 09:31 |
| kevko | ah, ok i found it in openstack infra repo | 09:36 |
| kevko | is it 8vcpu and 8gb ram ? :D https://786b6091616ef4dbd1b1-3bbb48847b672b5467267c1efe6f2622.ssl.cf1.rackcdn.com/openstack/14193b183f33422c8d68f3e68371fdb1/primary/logs/facts.json | 09:39 |
| kevko | we need v3-standard-2 or v3-standard-4 got from vexxhost https://vexxhost.com/pricing/ | 09:48 |
| kevko | but there is only https://paste.openstack.org/show/bPcuLkuV6IyOfS5qagOv/ v3-standard-8 :( in project-config repo | 09:49 |
| kevko | mnasiadka: you've set it to v3-standard-8 ...so it should be stable as it is 8/32 | 09:52 |
| kevko | but still...i think 2/8 can be enough | 09:52 |
| kevko | mnasiadka: How do you actually estimate whether it's OK? | 09:54 |
| kevko | bbezak: can I ask for review please :) ? https://review.opendev.org/c/openstack/kolla-ansible/+/949812 | 09:56 |
| bbezak | sure, you can ask! :) | 09:57 |
| bbezak | let me see | 09:57 |
| kevko | bbezak: thank you | 09:59 |
| kevko | bbezak: it's nice patch i think ... and with ubuntu 24.04 useful as I can bootstrap and create venv with k-a | 10:00 |
| kevko | mnasiadka: btw, is it hard to ask for 4/16 flavor or 2/8 ? | 10:11 |
| kevko | mnasiadka: it's really better than 8/8 ... | 10:11 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: Revert "[release] Use OpenStack 2025.1 release" https://review.opendev.org/c/openstack/kayobe/+/950512 | 10:14 |
| mnasiadka | kevko: I see you don't understand - ask every company delivering resources for adding a flavor? I guess so :) | 10:20 |
| mnasiadka | And I doubt it would change anything | 10:20 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: CI: Remove openstack-tox-py39 override https://review.opendev.org/c/openstack/kayobe/+/950517 | 10:24 |
| opendevreview | Pierre Riteau proposed openstack/kolla-ansible master: CI: Use openstack-python3-jobs template https://review.opendev.org/c/openstack/kolla-ansible/+/950519 | 10:36 |
| kevko | mnasiadka: of course I understand :) | 10:39 |
| kevko | mnasiadka: michalarbet@pixla:/tmp/project-config$ | 10:40 |
| kevko | grep -Ri nested-virt-debian-bookworm | awk -F ':' '{print $1}' | sort | uniq | sort | xargs grep -Ri providers: -A10 | grep cloud: | 10:40 |
| kevko | nodepool/nl05.opendev.org.yaml- cloud: rax | 10:40 |
| kevko | nodepool/nl06.opendev.org.yaml- cloud: openmetal | 10:40 |
| kevko | nodepool/nl07.opendev.org.yaml- cloud: vexxhost | 10:40 |
| kevko | nodepool/nl08.opendev.org.yaml- cloud: ovh | 10:40 |
| kevko | mnasiadka: I just thought it might be in their interest to provide fewer resources, and at the same time we would have the correct CPU/RAM ratio :) … the wolf would be full and the sheep intact … | 10:41 |
| kevko | mnasiadka: btw, I've been thinking about our eternal struggle with RabbitMQ, rate limiting, Cloudsmith APT... versions... etc... | 11:11 |
| kevko | mnasiadka: What about downloading Erlang as one big package, downloading RabbitMQ as one big package, and just installing those two packages? | 11:11 |
| kevko | mnasiadka: I mean Instead of using the APT repo? https://www.erlang-solutions.com/downloads-2/# and also https://github.com/rabbitmq/rabbitmq-server/releases for rabbitmq | 11:12 |
| mnasiadka | and get more issues with github rate limits? | 11:12 |
| mnasiadka | Let's not break something that works today ;) | 11:13 |
| kevko | mnasiadka: does github have also some rate limits ? :) | 11:15 |
| mnasiadka | who doesn't :) | 11:15 |
| kevko | mnasiadka: and include deb into git ? :D | 11:16 |
| kevko | mnasiadka: or rpm ? | 11:16 |
| kevko | bad idea i know ... but ? :D | 11:17 |
| kevko | 60 mb + 30mb :D | 11:17 |
| kevko | just asking | 11:17 |
| mnasiadka | Let's just leave it, there's a lot of other things we need to do this cycle ;) | 11:23 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: CI: Update command to match allowlist_externals https://review.opendev.org/c/openstack/kayobe/+/950528 | 11:32 |
| opendevreview | Michal Arbet proposed openstack/kolla master: debian: Use Epoxy extrepo release https://review.opendev.org/c/openstack/kolla/+/950530 | 11:50 |
| kevko | haha ... we should build against bookworm backport epoxy not dalmatian :D | 11:51 |
| kevko | mnasiadka frickler bbezak https://review.opendev.org/c/openstack/kolla/+/950530 | 11:51 |
| mnasiadka | kevko: https://review.opendev.org/c/openstack/kolla/+/949752 | 11:52 |
| kevko | mnasiadka: ahaaaa, thanks | 11:57 |
| opendevreview | Michal Arbet proposed openstack/kolla master: debian: Use bookworkm suite for rabbitmq APT repo https://review.opendev.org/c/openstack/kolla/+/950531 | 11:57 |
| opendevreview | Michal Arbet proposed openstack/kolla master: debian: Use Epoxy extrepo https://review.opendev.org/c/openstack/kolla/+/949752 | 12:02 |
| opendevreview | Michal Arbet proposed openstack/kolla master: debian: Use bookworkm suite for rabbitmq APT repo https://review.opendev.org/c/openstack/kolla/+/950531 | 12:02 |
| opendevreview | Michal Arbet proposed openstack/kolla master: debian: Use bookworkm suite for rabbitmq APT repo https://review.opendev.org/c/openstack/kolla/+/950531 | 12:08 |
| opendevreview | Merged openstack/ansible-collection-kolla master: Improve virtualenv handling and pip installation for SDK roles https://review.opendev.org/c/openstack/ansible-collection-kolla/+/949767 | 12:14 |
| opendevreview | Merged openstack/kolla-ansible master: Standardize Python interpreter handling https://review.opendev.org/c/openstack/kolla-ansible/+/949812 | 12:14 |
| opendevreview | Michal Arbet proposed openstack/kolla master: Fix mock_open() expectations for Python 3.13+ https://review.opendev.org/c/openstack/kolla/+/950533 | 12:29 |
| opendevreview | Michal Arbet proposed openstack/kolla master: debian: Use Epoxy extrepo https://review.opendev.org/c/openstack/kolla/+/949752 | 12:30 |
| opendevreview | Michal Arbet proposed openstack/kolla master: debian: Use bookworkm suite for rabbitmq APT repo https://review.opendev.org/c/openstack/kolla/+/950531 | 12:30 |
| opendevreview | Amir Hossein Ahmadi proposed openstack/kolla master: swift: Reintroduce support for Swift deployment https://review.opendev.org/c/openstack/kolla/+/950534 | 12:33 |
| opendevreview | Amir Hossein Ahmadi proposed openstack/kolla master: swift: Reintroduce support for Swift deployment https://review.opendev.org/c/openstack/kolla/+/950534 | 12:34 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible stable/2025.1: [CI] Fix bootstrap testing https://review.opendev.org/c/openstack/kolla-ansible/+/950295 | 12:38 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible stable/2025.1: Standardize Python interpreter handling https://review.opendev.org/c/openstack/kolla-ansible/+/949813 | 12:38 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible stable/2025.1: [CI] Enable virtualenv testing for Debian/Ubuntu https://review.opendev.org/c/openstack/kolla-ansible/+/950405 | 12:38 |
| opendevreview | Michal Arbet proposed openstack/kolla master: debian: Use bookworkm suite for rabbitmq APT repo https://review.opendev.org/c/openstack/kolla/+/950531 | 12:52 |
| msalari | amir58118: Did you get a response to your email sent to the core team? | 12:56 |
| kevko | mnasiadka: haha, today I am fixing the stuff and then I am noticing that I have a merge conflict and when I open ..it's the same fix from you :D :D :D ^^ 3.13 python | 12:58 |
| mnasiadka | lol | 12:59 |
| amir58118-2 | msalari: Unfortunately, no. i sent an email to Dr. Harbott , but i haven't recieved a responce, yet. | 12:59 |
| mnasiadka | Just a small reminder - there's no regular IRC meeting today - we're meeting at https://meetpad.opendev.org/kolla-flamingo-ptg for a PTG recap | 13:00 |
| opendevreview | Michal Arbet proposed openstack/kolla master: Fix mock_open() expectations for Python 3.13+ https://review.opendev.org/c/openstack/kolla/+/950398 | 13:01 |
| opendevreview | Michal Arbet proposed openstack/kolla master: debian: Use Epoxy extrepo https://review.opendev.org/c/openstack/kolla/+/949752 | 13:01 |
| opendevreview | Michal Arbet proposed openstack/kolla master: debian: Use bookworkm suite for rabbitmq APT repo https://review.opendev.org/c/openstack/kolla/+/950531 | 13:01 |
| kevko | hh, thanks | 13:01 |
| frickler | msalari: amir58118-2: is there anything that cannot be discussed in public? I personally have zero interest in swift at all | 13:22 |
| kevko | mnasiadka: I took the liberty of changing the commit message and adding more information here in your patch -> https://review.opendev.org/c/openstack/kolla/+/950398/5 | 13:37 |
| amir58118-2 | frickler: that's totally fine, if the door is open for us to take over maintenance, we're happy to take full responsibility for developing and supporting swift in kolla. | 13:45 |
| amir58118-2 | we think there’s still a valid use case for Swift in containerized environments, especially for operators who rely on Kolla and don’t have alternative deployment methods in place | 13:45 |
| amir58118-2 | we believe the existing issues with Swift in Kolla can be addressed, and we’re fully committed to doing the work. | 13:48 |
| kevko | amir58118-2: I would say that drop of swift will be good start, and you can send new master patches for review ..including tests ...and if it will be done in current cycle ...flammingo will be with your pure new swift, right ? | 13:51 |
| amir58118-3 | The only technical issue raised so far has been the incompatibility between HAProxy and the Swift proxy server. I can say we have a solution for that — we can use nginx or traefik as the load balancer for Swift to avoid interfering with the rest of the OpenStack deployment. | 13:52 |
| frickler | as long as there isn't broader interest in swift, I would suggest for you to maintain it in a downstream fork | 13:53 |
| msalari | kevko: That would be great. But naturally, we'll need some guidance along the way as we prepare the patch. What worries me is the statement that there's no interest in Swift — it makes me concerned that our efforts might end up being in vain. | 13:54 |
| msalari | frickler: Are the technical reasons behind the lack of interest in Swift not something that can be shared? Is it just because there were serious issues with it in the past? | 13:55 |
| frickler | well to me ceph is the better solution, so swift is kind of useless | 13:57 |
| msalari | frickler: So, does the fact that you dont like swift means that if we propose Swift role (modernized and without problems) from scratch and propose it, you won’t even review it? | 14:04 |
| kevko | frickler: ceph is always better ...BUT ...every user is different ..for example swift is much benevolent with HW ... | 14:05 |
| msalari | kevko: and swift is better for multi-region data centers. swift is eventual consistent and ceph is strong consistent. use cases are different | 14:06 |
| kevko | in overall Swift is less demanding on CPU and hardware than Ceph , simple replcation , plain disk files , run well on older HW or lowend ... | 14:06 |
| kevko | i am not saying that we are using it ...just wondering about pros/cons | 14:07 |
| msalari | Eventual consistency of swift gets better availability in network partition conditions and in multi-region clusters | 14:07 |
| kevko | yep | 14:07 |
| amir58118-2 | +1 | 14:07 |
| kevko | i will review ..from scratch ... but i am only one it looks like :D | 14:07 |
| mnasiadka | It's not about that we dislike Swift, it's about why Kolla should support deploying Swift. | 14:08 |
| kevko | as kolla is orchestration project for openstack projects which swift is still part of ? :) I would say | 14:08 |
| kevko | and still usable ...not like freezer or panko or watcher or those craps | 14:08 |
| mnasiadka | For Swift to be sustainable and performant in Kolla - it would need to have support for another load balancer than the API HAProxy | 14:08 |
| msalari | kevko: frickler: I just need a clear answer to this question: if we invest time in redesigning Swift support for Kolla Ansible, will you consider it? I need this to define our company’s OKRs for this quarter. | 14:09 |
| msalari | mnasiadka: exact. our plan is right this | 14:09 |
| kevko | +1 | 14:10 |
| amir58118-2 | mnasiadka: exactly, As mentioned earlier, I did point that out — and we already have a working solution using nginx or traefik instead of HAProxy for Swift. | 14:10 |
| frickler | right, I won't spend time reviewing it, just like I don't for other topics like LE or rocky | 14:10 |
| mnasiadka | Well, I would prefer it being second haproxy - we would need to build nginx/traefik in Kolla and there's no sane reason for doing that. | 14:11 |
| amir58118-2 | mnasiadka: That's fair — using a second HAProxy is totally doable for us as well. if a second HAProxy instance is more aligned with Kolla's direction, we're happy to adapt to that. | 14:13 |
| mnasiadka | amir58118-2, msalari: I can't promise it will be merged as I can't promise I'll have time to review it - but if kevko is happy to guide you to success - then you can try. | 14:13 |
| amir58118-2 | Thanks, that’s all we needed. | 14:13 |
| amir58118-2 | Kevko: thanks for being the Yoda of our Swift mission =) | 14:15 |
| msalari | kevko: Are you going to give us a more encouraging answer? | 14:18 |
| kevko | amir58118-2: Only what I believe is right, I do. The path of open source, this is. | 14:19 |
| msalari | kevko: sorry. just ignore my last message. thank you for support | 14:24 |
| kevko | amir58118-2 msalari : Look, I personally don't need Swift support right now, but who knows – maybe in the future I will, if one of our clients requests it. For this reason, I’ll do the review for you and help guide you. I imagine Swift as being separate from OpenStack services. Maybe we can use the HAProxy role and introduce a second VIP, or | 14:24 |
| kevko | set up Nginx/Traefik as mnasiadka suggested. Swift shouldn't affect the current services and should be installable using Kolla. If you add me as a reviewer, I’ll definitely keep an eye on it. That said, the policy requires two +2 votes... and I’m just one | 14:24 |
| kevko | amir58118-2 msalari: I just want to see new patches, with tests ...from scratch ... | 14:25 |
| amir58118-2 | kevko: Thanks a lot for your support. We’re actively working on the patch for Swift and making sure it meets all requirements, including tests. We’ll share the updates with you as soon as it’s ready for review. | 14:29 |
| amir58118-2 | You can count on us for other Kolla-related matters as well. thanks to everyone. | 14:31 |
| kevko | amir58118-2: do you have some work in progress patches now ? | 14:32 |
| amir58118-2 | yes | 14:33 |
| amir58118-2 | https://review.opendev.org/c/openstack/kolla-ansible/+/945791 | 14:33 |
| amir58118-2 | https://review.opendev.org/c/openstack/kolla/+/950534 | 14:33 |
| amir58118-2 | Our goal is to reintroduce swift role from scratch. | 14:35 |
| amir58118-2 | Therefore, we may submit a new patch specifically for Kolla-Ansible. | 14:36 |
| PrzemekK | If You could look at changes/bugs from Piotr https://review.opendev.org/q/project:openstack/kolla-ansible https://bugs.launchpad.net/kolla-ansible/+bugs?orderby=-id&start=0 . Finally we upgraded first enviroment to 2025.1 | 14:38 |
| kevko | PrzemekK: done | 15:05 |
| kevko | mnasiadka bbezak: now backport 2025.1 please ? https://review.opendev.org/c/openstack/kolla-ansible/+/949813 | 15:06 |
| kevko | whole stack of course .. | 15:09 |
| kevko | so I can remove venv creation from Zuul CI in next patch and we have working auto venv creation | 15:09 |
| opendevreview | Piotr Milewski proposed openstack/kolla-ansible master: Fix missing vendoradata.json file for nova-metadata https://review.opendev.org/c/openstack/kolla-ansible/+/950406 | 15:19 |
| opendevreview | Piotr Milewski proposed openstack/kolla-ansible master: Unnecessary comma in kolla-tolbox configuration - external rabbitmq https://review.opendev.org/c/openstack/kolla-ansible/+/950311 | 15:29 |
| PrzemekK | @kevko <3 | 15:46 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!