| fkr | o/ | 07:00 |
|---|---|---|
| gtema | morning | 07:01 |
| joek-office | hello together | 07:01 |
| fkr | how is everyone? | 07:02 |
| tobberydberg | o/ | 07:02 |
| tobberydberg | All good here thanks. How about you? | 07:02 |
| joek-office | here too | 07:02 |
| fkr | yes, very well | 07:03 |
| tobberydberg | FYI .. I have a conflicting mandatory meeting that popped up at 9:30 today, so I will have split focus by that time :-) | 07:05 |
| tobberydberg | Lets kick it off | 07:05 |
| fkr | alright. let's get going. | 07:05 |
| fkr | (last time we forgot the meetingbot) | 07:06 |
| tobberydberg | #startmeeting publiccloud_sig | 07:06 |
| opendevmeet | Meeting started Wed Aug 30 07:06:26 2023 UTC and is due to finish in 60 minutes. The chair is tobberydberg. Information about MeetBot at http://wiki.debian.org/MeetBot. | 07:06 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 07:06 |
| opendevmeet | The meeting name has been set to 'publiccloud_sig' | 07:06 |
| tobberydberg | Slow bot today or did I misspell? ;-) | 07:06 |
| tobberydberg | There it is :-) | 07:06 |
| tobberydberg | Agenda to be found: #link https://etherpad.opendev.org/p/publiccloud-sig-meeting | 07:07 |
| fkr | ok. first item on the agenda: "0. Quick introduction of any new people" | 07:07 |
| tobberydberg | Please put yourname in there | 07:07 |
| fkr | my hope is still strong (new people), since I did send out a reminder to the scs list as well :) | 07:08 |
| gtema | :) | 07:08 |
| tobberydberg | Do we have any new people in here? :-) | 07:08 |
| zigo | o/ | 07:08 |
| fkr | hey zigo! | 07:08 |
| joek-office | i'm new, but not brand new ;-) | 07:08 |
| fkr | joek-office: do you want to intro yourself? | 07:09 |
| joek-office | ok. | 07:09 |
| zigo | frickler: Hey! :) | 07:09 |
| zigo | whoops, meant for fkr... | 07:09 |
| fkr | ;) | 07:09 |
| joek-office | Like i said in the past, we are at the moment building up a mixed public/private Cloud infrastructure based on openstack. | 07:10 |
| joek-office | destination is running services in the logistcs area with the software that is build inhouse. | 07:10 |
| tobberydberg | Welcome (again) joek-office ;-) | 07:11 |
| fkr | joek-office: what was the reason for choosing openstack? | 07:11 |
| joek-office | at the moment we are running from testing to preprod stage. | 07:11 |
| joek-office | fkr. i'm new in the company since a year. the decision is made before. but i think is the same as everywhere. Make the things more inhouse than just leave on the public hyper scalers. | 07:13 |
| tobberydberg | Sounds like a sound choice :-) | 07:14 |
| tobberydberg | If no more intros, lets move on in the agenda... | 07:15 |
| fkr | ack | 07:15 |
| fkr | (to both ;) | 07:15 |
| tobberydberg | #topic 1. Topic for lightning talk next time? | 07:15 |
| joek-office | the "make more inhouse" mindset is very common in the company. there are divisions from building up warehouse to running the complete lifecycle | 07:15 |
| joek-office | sorry, what is a lightning talk? | 07:15 |
| fkr | a short (5 - 15 ? minute) talk | 07:16 |
| tobberydberg | Tha plan is to have a "video call" next time, and the proposal is to have a lightning talk if we have a topic and someone willing to present something | 07:16 |
| fkr | and the idea was to make it a lightning talk to have it short and entertaining | 07:16 |
| fkr | as well as less burden to prepare :) | 07:16 |
| tobberydberg | Anyone having an interesting topic they would like to present? | 07:17 |
| joek-office | ok, thank you | 07:17 |
| gtema | maybe I have something from SDK/CLI area to present what might be interesting for the cloud operators: OpenAPI support for OpenStack and making a new faster CLI on Rust | 07:18 |
| fkr | woot | 07:18 |
| fkr | +1 | 07:18 |
| gtema | not sure whether I would have enough time to prepare though | 07:18 |
| joek-office | sounds good and interesting | 07:19 |
| fkr | from my perspective, it does not need to be with slides, bells and whistles | 07:19 |
| fkr | the idea that puck, tobberydberg and me had was to let this be up for the person giving the lightning talk so demo, code, etherpad contents whatever is imho perfectly fine | 07:20 |
| gtema | sure | 07:20 |
| tobberydberg | Agreed, super interesting and bells and whistles not needed :-) | 07:20 |
| tobberydberg | +1 | 07:20 |
| tobberydberg | Cool! Lets go with that then, great! | 07:20 |
| tobberydberg | Will you create a Jitsi meeting that is open under scs jitsi fkr? | 07:21 |
| fkr | yes | 07:21 |
| fkr | https://conf.scs.koeln:8443/OIF-public-cloud-sig | 07:22 |
| fkr | that is going to be the place | 07:22 |
| tobberydberg | Perfect, thanks! | 07:22 |
| fkr | (this way we already have it in the meeting log) | 07:22 |
| tobberydberg | next topic... | 07:22 |
| fkr | Metering & Billing | 07:22 |
| tobberydberg | #topic 2. Meetering and billing... | 07:22 |
| tobberydberg | (oooops, too many eee's) | 07:23 |
| fkr | last time I had to depart early and after I left there was a discussion on metering and billing with distil being thrown into the conversation | 07:23 |
| fkr | that I found very interesting, since as part of SCS this topic was raised a year ago as well | 07:23 |
| joek-office | yes, as alternative or side by side of cloudkitty | 07:23 |
| fkr | and as such there was a tender being created (VP13 - https://scs.community/tenders/lot13) | 07:24 |
| fkr | and Cloud & Heat successfully bid on that tender and is currently working with gnocchi and ceilometer to provide something that emits json to a billing stack | 07:24 |
| fkr | https://github.com/SovereignCloudStack/issues/issues/269 | 07:25 |
| fkr | that is the epic with the user stories referenced | 07:25 |
| fkr | they've been reaching out to mrunge to coordinate this with ceilometer as well | 07:25 |
| fkr | why I wanted to bring it back to the table here: I'm sure you do billing :) | 07:26 |
| fkr | how are you (looking at this nice round ;) / how have you been doing this? | 07:26 |
| zigo | FYI, we use cloudkitty (plus https://salsa.debian.org/openstack-team/services/ceilometer-instance-poller/ that I wrote to bill Windows images...). | 07:26 |
| joek-office | we use the combination prometheus/cloudkitty | 07:27 |
| tobberydberg | We did use ceilometer in the past, but had to much issues with that, loosing events etc that caused the "billing" to be incorrect. This is some time ago though.... | 07:28 |
| joek-office | have tested the combination of ceilometer/gnocchi/cloudkitty as well, but this stack looks like not so easy and wide supported as the easy combination of the prometheus exporters | 07:28 |
| tobberydberg | I'm keen on somthing that is more "prometheus based", but I haven't dug my self deep into it for quite some time | 07:28 |
| zigo | The main issue we have, is the Gnocchi API and MySQL being super busy even with as few as 3K VMs. We have dedicated 3x 128 core server for it, and a dedicated Ceph cluster too, but even that isn't enough... | 07:29 |
| tobberydberg | I know we had a discussion regarding kind of forming a new metering project within openstack a bunch of years ago, prometheus based was already then a strong contender | 07:30 |
| joek-office | the 3x128 core servers for compute or just for gnocchi/mysql? i hope for compute | 07:30 |
| tobberydberg | performance was a huge issue for us as well | 07:31 |
| fkr | jeiks (3x128 core servers) | 07:31 |
| zigo | For the role we call "messaging" node. It contains: MySQL, Gnocchi-api, and cloudkitty-processor. | 07:31 |
| joek-office | ok, thanks zigo. that is much ressources for "just" computing billing. but is a nice forecast for my future resource demand | 07:32 |
| zigo | joek-office: I'm telling you, that is *not* overprovision ... :P | 07:33 |
| fkr | tobberydberg: interesting (initiative of new metering, prometheus based). is there any trace of that? | 07:34 |
| zigo | Correction: 64 cores, not 128, and 784 GB of RAM, but still... for "only" 3K VMs... | 07:34 |
| tobberydberg | Didn't kick off at all...just one meeting kind of... | 07:34 |
| fkr | but it reads to me as "everyone does it themselves by using similar building blocks but there is nothing that really works out of the box" | 07:34 |
| joek-office | yes belief it. in the past i have seen in our test env that the cloudkitty processor shoot out the processors with just some (10-15) vms that have deployed and destroyed in short time | 07:35 |
| zigo | Sometimes, there's so many Gnocchi threads that there's no source port left to connect !!! | 07:35 |
| fkr | maybe not "works out of the box" but is "suiteable out of the box" | 07:35 |
| zigo | How does the prometheus solution scales? Better? | 07:36 |
| fkr | joek-office: can you share a bit more on how you built that? | 07:36 |
| joek-office | like i mentioned. i have at the moment no stage/production systems with many vms deployed. but in the past i see many load for just ten to fifteen vms. | 07:37 |
| tobberydberg | Haven't played with it in this use case, but I would assume so since it scales super good for "other" metrics | 07:38 |
| joek-office | we have build up the mobnitoring cluster before based on prometheus. then we evaluated the combinations of gnocchi/ceilometer/cloudkitty and prometheus/cloudkitty | 07:38 |
| joek-office | we deployed both with kolla-ansible | 07:38 |
| joek-office | at the end we decide to use prometheus/cloudkitty because of the fewer services that had to run | 07:39 |
| joek-office | but at the moment no scale projection can be made | 07:40 |
| joek-office | tobberydberg: my feeling sounds also good for scaling with prometheus | 07:41 |
| tobberydberg | I hope to get my team to prioritize looking in to that at some point... (point = year) ;-) | 07:42 |
| fkr | is there an interest in this group to have a breakout to dive into more of this subject? | 07:43 |
| joek-office | for me: yes. But the broader view of the billing infrastructure is very interesting. | 07:44 |
| tobberydberg | Yes, if there is "enough" operators in the "call". But, I don't think I can pull the resources to to actual work unfortunate | 07:44 |
| joek-office | are there best practices for the billing journey with OpenStack | 07:45 |
| fkr | tobberydberg: I can go fishing for operators that join such a breakout | 07:45 |
| fkr | and collect interest beforehand | 07:45 |
| fkr | action item for me would be to also write a mail to openstack-discuss then | 07:46 |
| tobberydberg | From what I know, the ceilometer/gnocchi/cloudkitty is the "official" and only recommendation upstream | 07:46 |
| tobberydberg | How many operators that actually do it that way, not sure... | 07:46 |
| tobberydberg | We don't, we are not using any of those projects | 07:47 |
| joek-office | what do you use? | 07:47 |
| joek-office | or how do you solve this | 07:47 |
| joek-office | fkr: sounds good for me. | 07:48 |
| tobberydberg | Self developed hack if I'm going to honest. It is solid and well proven for our use case and how we charge our customers. But, not perfect for that either | 07:48 |
| fkr | tobberydberg: my gut feeling is that you're in good company with that | 07:48 |
| tobberydberg | Yes, that is my understanding as well | 07:49 |
| tobberydberg | Catalyst is probably an exception, at least from public cloud perspective. | 07:49 |
| tobberydberg | As public cloud it is super important to be fully accurate with the numbers (which we had issues to reach with ceilometer) | 07:50 |
| joek-office | tobberydberg: yes, this is it. would be nice to hear from puk how they work with that. | 07:52 |
| fkr | i'll re-add it to the agenda for next time as well (maybe puck is around then) | 07:52 |
| joek-office | and how it scale | 07:52 |
| tobberydberg | They seam to be successful using that though, no clue about the scale there. | 07:53 |
| tobberydberg | But a separate session regarding this is probably a good starting point, at least to share experiences | 07:55 |
| tobberydberg | Would be awesome if that leads to something concrete, but sharing experience is always good if it stops with that :-) | 07:56 |
| tobberydberg | Almost out of time.... Should we postpone the last topic until next time or start it now? :-) | 07:56 |
| fkr | i'll add it for next time | 07:56 |
| fkr | tobberydberg: +1 on the concrete! | 07:57 |
| tobberydberg | Perfect! Something else we should bring up now or suggestion for topics moving ahead? | 07:58 |
| joek-office | here in germany are several providers with public clouds out there that have to do billing and use OpenStack. Like Syseleven or Open Telekom Cloud. Anyone knows someone from this companys? | 07:59 |
| fkr | yes | 07:59 |
| joek-office | tobberydberg: from my side there is nothing. | 08:00 |
| tobberydberg | Some representatives are already in here, but lets continue to try to get more of them in here on a regular basis... | 08:00 |
| fkr | i'll now dive into the SCS Team IaaS call. | 08:02 |
| tobberydberg | Ok, so time is unfortunate up here. We look forward to your lightning talk in two weeks gtema :-) | 08:02 |
| tobberydberg | Thanks everyone for today and we will see (!) each other in two weeks :-) | 08:02 |
| gtema | :) | 08:02 |
| fkr | was a really nice meeting today, thanks for the participation! | 08:02 |
| gtema | see ya | 08:02 |
| tobberydberg | Have a great rest of the day! | 08:03 |
| tobberydberg | #endmeeting | 08:03 |
| opendevmeet | Meeting ended Wed Aug 30 08:03:15 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 08:03 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/publiccloud_sig/2023/publiccloud_sig.2023-08-30-07.06.html | 08:03 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/publiccloud_sig/2023/publiccloud_sig.2023-08-30-07.06.txt | 08:03 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/publiccloud_sig/2023/publiccloud_sig.2023-08-30-07.06.log.html | 08:03 |
| joek-office | buy, nice to read you ;-) | 08:03 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!