| opendevreview | Roman Krček proposed openstack/kolla-ansible master: Refactor services' check-containers and optimise https://review.opendev.org/c/openstack/kolla-ansible/+/773243 | 07:51 |
|---|---|---|
| opendevreview | Roman Krček proposed openstack/kolla-ansible master: Fix unintentional trigger of ansible handlers https://review.opendev.org/c/openstack/kolla-ansible/+/924145 | 07:53 |
| SvenKieske | frickler: mhm, we might have a little licence problem here: https://opendev.org/openstack/kolla-ansible/src/branch/master/tests/j2lint.py at least I can't find dbad on the OSI list and the language looks very ambigous to me. I filed an upstream issue already if they can maybe dual licence this.. | 09:30 |
| SvenKieske | I filed a bug for easier tracking: https://bugs.launchpad.net/kolla-ansible/+bug/2075316 | 09:39 |
| deflated | Morning people, getting errors trying to deploy cloudkitty, doesnt matter if i use influx/opensearch/elasticsearch theres always some error (aiming to use opensearch if it matters, tried the others just to see if any worked), pasted the errors here:https://paste.openstack.org/show/brXI10k9ZnlwFZoSf35d/ Any ideas where i am going wrong? | 09:50 |
| SvenKieske | deflated: on what version of openstack are you running? is this test or prod deployment? are you sure the network connection works? | 10:01 |
| deflated | Pre prod, when i get this working it'll be ready for prod, 2024.1, everything else works and i can see the right stuff in networking (eg ovs-vsctl show etc) | 10:02 |
| deflated | Well, i say it'll be ready, i havent got internal tls working yet (int/ext i have) so this and that and a rebuild for final production | 10:03 |
| SvenKieske | is this a single influx/opensearch node or cluster? | 10:05 |
| deflated | its a multinode deployment, influx was justa test, but opensearch is clustered | 10:06 |
| SvenKieske | well I have seen working cloudkitty deployments I think but it has been some time, at least I'm currently not aware if it works and I fear there are not many (none?) integration tests actually testing it works (well deployment should work I guess) | 10:07 |
| SvenKieske | Do you happen to have a look at openstack logs as well? | 10:08 |
| SvenKieske | it's probably worth it to file a bug and send an email to the mailing list about this, as many people are on vacation I think. I'm not deeply familiar with cloudkitty, just have used it in test environments myself. | 10:11 |
| SvenKieske | deflated: https://bugs.launchpad.net/kolla-ansible | 10:11 |
| SvenKieske | a bug report with as much information as you can provide would be very helpful :) | 10:12 |
| deflated | I have and there is some extended error info in cloudkitty-api.log, i'll get a bug report posted with as much info as i can shortly | 10:13 |
| opendevreview | Maximilian Stinsky proposed openstack/kolla-ansible master: Implement neutron-ovn-vpn-agent https://review.opendev.org/c/openstack/kolla-ansible/+/924575 | 10:14 |
| oli | I have enabled login with keycloak and it works for the horizon dashboard but doesn't for the skyline dashboard, I get a 401 unauthorized, and the skyline dashboard is in the keycloak.conf as a trusted dashboad. Any pointers on where to look next? | 10:27 |
| oli | I can login to skyline using the keystone credentials but no the openid ones, it doens't even redirect me to the provider login | 10:29 |
| deflated | So, on to internal tls, im pretty much in the same boat as this guy on the mailing list https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/NLWURK5LBF5ZU4KLUMFNOTAOZGTS7FCZ/ How do we use official certs for internal tls? | 11:22 |
| opendevreview | Merged openstack/kayobe master: Stop replacing dashes with underscores for Kolla interfaces https://review.opendev.org/c/openstack/kayobe/+/922480 | 12:07 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2024.1: Stop replacing dashes with underscores for Kolla interfaces https://review.opendev.org/c/openstack/kayobe/+/925390 | 12:39 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2023.2: Stop replacing dashes with underscores for Kolla interfaces https://review.opendev.org/c/openstack/kayobe/+/925391 | 12:39 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2023.1: Stop replacing dashes with underscores for Kolla interfaces https://review.opendev.org/c/openstack/kayobe/+/925392 | 12:40 |
| bbezak | Hi. meeting in 5 - mgoddard mnasiadka bbezak frickler kevko SvenKieske mmalchuk gkoper jangutter jsuazo jovial osmanlicilegi mattcrees dougszu darmach | 12:55 |
| bbezak | #startmeeting kolla | 13:00 |
| opendevmeet | Meeting started Wed Jul 31 13:00:38 2024 UTC and is due to finish in 60 minutes. The chair is bbezak. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:00 |
| opendevmeet | The meeting name has been set to 'kolla' | 13:00 |
| bbezak | #topic rollcall | 13:00 |
| SvenKieske | o/ | 13:01 |
| r-krcek | \o/ | 13:01 |
| mhiner | o/ | 13:01 |
| bbezak | not many of us, vacation period I guess ;) | 13:02 |
| bbezak | #topic agenda | 13:02 |
| mmalchuk | o/ | 13:02 |
| mattcrees | o/ | 13:02 |
| bbezak | * Roll-call | 13:02 |
| bbezak | * Agenda | 13:02 |
| bbezak | * Announcements | 13:02 |
| bbezak | * Review action items from the last meeting | 13:02 |
| bbezak | * CI status | 13:02 |
| bbezak | * Release tasks | 13:02 |
| bbezak | * Regular stable releases (first meeting in a month) | 13:02 |
| bbezak | * Current cycle planning | 13:02 |
| bbezak | * Additional agenda (from whiteboard) | 13:02 |
| bbezak | * Open discussion | 13:02 |
| bbezak | #topic CI status | 13:03 |
| Fl1nt | Hi everyone | 13:03 |
| bbezak | looks green | 13:03 |
| Fl1nt | bbezak, The men I was looking for! | 13:03 |
| bbezak | :) | 13:03 |
| bbezak | we're on CI status topic btw :) | 13:04 |
| SvenKieske | periodic pipeline looks not so green? https://bugs.launchpad.net/kolla-ansible/+bug/2075316 | 13:04 |
| SvenKieske | ah sorry, wrong link | 13:04 |
| SvenKieske | https://zuul.openstack.org/builds?project=openstack%2Fkolla&pipeline=periodic&result=FAILURE&result=RETRY_LIMIT&result=POST_FAILURE&result=NODE_FAILURE&result=SKIPPED&skip=0 | 13:04 |
| Fl1nt | bbezak, just saw that np | 13:04 |
| Fl1nt | I'll wait :D | 13:04 |
| SvenKieske | periodic-weekly looks better | 13:05 |
| SvenKieske | gateway timeout, probably infra related? https://zuul.openstack.org/build/9e07eb72ee3f44c3b00e13468bfe2e83 I still need to subscribe to the opendev infra ML. don't know if there was some expected outage | 13:06 |
| bbezak | except arm64 it doesn't look bad | 13:07 |
| Fl1nt | SvenKieske, OVH had a lot of trouble this week with various HW/Fiber incident. | 13:07 |
| bbezak | last several days are ok | 13:08 |
| bbezak | #topic Release tasks | 13:08 |
| bbezak | we're r-9 | 13:09 |
| SvenKieske | Fl1nt: ah good to know | 13:09 |
| bbezak | time flies ;) | 13:09 |
| bbezak | not much to do now, next week there will be some tasks | 13:09 |
| bbezak | #topic Regular stable releases | 13:09 |
| bbezak | I've raised change to have a new minor releases | 13:10 |
| bbezak | as we didn't do that for several months already | 13:10 |
| * frickler needs to review that one | 13:10 | |
| bbezak | https://review.opendev.org/c/openstack/releases/+/925182 | 13:10 |
| bbezak | yeap :) | 13:10 |
| bbezak | #topic Current cycle planning | 13:11 |
| bbezak | I was on vacation last couple of weeks, so I didn't progress much ;) | 13:11 |
| bbezak | I'll check with darmach the ubuntu noble progress | 13:11 |
| bbezak | oh, he is now. how is it doing darmach? | 13:12 |
| bbezak | in any case, we need to finish that soon'ish | 13:12 |
| bbezak | if you need anything around this cycle priorities, let us know - https://etherpad.opendev.org/p/KollaWhiteBoard#L227 | 13:13 |
| bbezak | #topic Additional agenda (from whiteboard) | 13:14 |
| bbezak | https://etherpad.opendev.org/p/KollaWhiteBoard#L70 | 13:14 |
| bbezak | mhiner and r-krcek changes are there | 13:15 |
| mhiner | yes, additionally on behalf of ihalomi and regarding failed tests in 908295: | 13:16 |
| mhiner | Upgrade tests are failing because we use stable release for ansible-collection-kolla that doesn't include repository that contains docker 6.0.0 and during upgrade bootstrap-container role is called only once before first deploy, that means even we change later to correct ansible-collection-kolla it doesn't effect installed version of docker so test fails with cgroupns missing. | 13:16 |
| SvenKieske | I +1 the question wrt to bandit from r-krcek as I still need to make myself more familiar with it. this is in the context of the kolla-ansible "binary" python rewrite. | 13:16 |
| mhiner | My proposed solution is to cherrypick this commit to stable branch | 13:16 |
| mhiner | https://github.com/openstack/ansible-collection-kolla/commit/42116ded107f99983ebdcb0c70e8a8c4cd6fdc52 | 13:16 |
| bbezak | so something that will be fixed when frickler will merge release change :) | 13:17 |
| SvenKieske | ah right iahlomi asked me about that one a few days ago, I agree we need to fix the docker stuff | 13:17 |
| SvenKieske | bbezak: mhm, wait, shouldn't we follow our own suggestions to the users and install from stable git branches instead of released artifacts? ;) (only half joking) | 13:18 |
| bbezak | yeah, there is some inconsistency there | 13:19 |
| bbezak | I'll look into the docker stuff mhiner - hopefully tomorrow | 13:20 |
| mhiner | thanks | 13:20 |
| bbezak | and to review some patches as well | 13:21 |
| SvenKieske | good to know that was already fixed, I thought we had some error in our upgrade testing at first | 13:22 |
| bbezak | ok, I'll book some time for those patches, let's move on | 13:23 |
| bbezak | #topic Open discussion | 13:23 |
| chembervint | Hi, may I up my patch and kindly ask to review it? https://review.opendev.org/c/openstack/kolla-ansible/+/920377. | 13:23 |
| chembervint | and 1 more - it's already merged to master, but cherry-picks has stolen .. https://review.opendev.org/c/openstack/kolla-ansible/+/924322 | 13:24 |
| SvenKieske | well I hate to be "that guy", but I filed https://bugs.launchpad.net/kolla-ansible/+bug/2075316 because I think we are violating upstreams licence, I asked upstream if they can possibly relicence/dual-licence but I don't think our changes to their code are substantial, IANAL. | 13:24 |
| SvenKieske | chembervint: I guess the backports will eventually be processed :) | 13:25 |
| SvenKieske | the octavia stuff is annoying, I don't know if we can actually implement a better solution than what you proposed, but I guess mnasiadka is on vacation | 13:26 |
| bbezak | yeah, will push those backports | 13:26 |
| Fl1nt | bbezak, I really like the DON'T BE A DICK public license :D | 13:26 |
| Fl1nt | Especially the share the love money :D | 13:26 |
| bbezak | imho looks pretty permissive :) | 13:26 |
| bbezak | just don't be a dick | 13:26 |
| Fl1nt | yep | 13:27 |
| chembervint | SvenKieske: ok, thank you! will wait mnasiadka | 13:27 |
| SvenKieske | well, personally I think it's a "funny" licence (humor is subjective). professionally it's annoying because it's not OSI approved and I don't think we can just wrap it in ASL 2.0, so well, yeah. | 13:27 |
| frickler | for octavia I still think configuring the interface statically is the better solution, just a bit more work | 13:28 |
| Fl1nt | SvenKieske, I like introducing those license to my colleagues of the licensing department, they usually lol a lot and then send a simple: Approved because... Why not. | 13:28 |
| SvenKieske | the TC guidelines are pretty clear on that, I also went around, there are multiple other jinja linters which are better maintained, maybe I can add one of those, but that would incur more overhead.. | 13:28 |
| SvenKieske | the original question why I looked at it came up in https://review.opendev.org/c/openstack/kolla-ansible/+/909912/comment/4108e5e2_ad15207a/ | 13:29 |
| chembervint | frickler: ok, but how to do it in a proper way? I've described my thoughts in a ticket. if you have some ideas - I'll be glad to try it | 13:29 |
| SvenKieske | because I think we need to extend it, and we are also missing some commits from upstream which we might/might not want | 13:29 |
| Fl1nt | Can't we use the one used by ansible? | 13:29 |
| SvenKieske | afaik we already do and this is just an additional linter | 13:31 |
| Fl1nt | Does this additional linter brings something special/desirable? | 13:31 |
| SvenKieske | so we certainly use "ansible-lint". in python/ansible lint there is a ton of different linters with varying degrees of coverage.. | 13:31 |
| chembervint | frickler: I mean, first off all in any case it will be still a ovs port, so we have to wait OVS container is up and running, and handle it somehow in systemd. and then - we are support different linux distros with different network configuration staff | 13:31 |
| SvenKieske | Fl1nt: I'm not certain; maybe I take some more minutes to investigate if we can just rip it out for good :) | 13:31 |
| Fl1nt | Yeah, could be a much more simpler way to solve the issue :D | 13:32 |
| SvenKieske | I guess the octavia discussion is more interesting/pressing and has more merit :) | 13:32 |
| SvenKieske | so we would need a dedicated play I guess, which does the right thing with the ip config stuff depending on the distro (sounds like a thing ansible was basically invented for?) | 13:33 |
| frickler | chembervint: maybe if you have so much trouble with ovs, you can use a vlan interface instead? anyway, yes, I was understating the effort when saying "a bit of work" | 13:33 |
| SvenKieske | :D | 13:34 |
| frickler | solving your questions would actually be part of that task | 13:34 |
| Fl1nt | Octavia is an heavy used service, with very poor architectural design, we have deployed it on our clusters... I'm always having questions about it that I can't solve without having to extensively discuss with the Octavia guys. | 13:34 |
| chembervint | One more question - one of our clients requires us to avoid storing passwords in plain-text in config files. We've tested oslo.config + Castellan stuff and already have a working prototype. And now working on non-openstack services as well. Could it be interesting to upstream such security stuff? | 13:34 |
| Fl1nt | chembervint, yes, very much | 13:35 |
| SvenKieske | chembervint: I think so, yes | 13:35 |
| Fl1nt | I have a lot of our customers that are specifically asking us for such solution. | 13:36 |
| frickler | I've started to look at using application credentials instead, but not sure whether the secret being used there still fits the same category | 13:36 |
| Fl1nt | and end up implementing vault from hashicorp... which isn't really integrated. | 13:36 |
| chembervint | Fl1nt: exactly. castellan + vault | 13:36 |
| bbezak | castellan can also talk to hashi | 13:37 |
| bbezak | yeap | 13:37 |
| Fl1nt | yep | 13:37 |
| SvenKieske | mhm, what do you guys think about castellan vs barbican, if it's not too off topic? | 13:37 |
| Fl1nt | castellan is barbican :D | 13:38 |
| Fl1nt | barbican leverage castellan lib | 13:38 |
| chembervint | frickler: in the past we've used vlans, before it was automated in k-a to config tenant network for octavia :) but it a bit harder to automate in in common case ... | 13:38 |
| mattcrees | Just to jump in for open discussion, we're getting close to rabbit being ready for slurp a->c. I'm looking for a second core review for merging this patch chain please: https://review.opendev.org/c/openstack/kolla/+/918974 then any reviews welcome here: https://review.opendev.org/c/openstack/kolla-ansible/+/918976 | 13:39 |
| chembervint | castellan in just a supported backend for oslo.config ... barbican also could use it as an option | 13:39 |
| bbezak | we're using vlans via kayobe for octavia network - so automated there | 13:40 |
| bbezak | thx mattcrees, we need second core outside of stackhpc kevko frickler please take a look | 13:41 |
| Fl1nt | using vlans with Octavia really is the easiest way to go BUT it bring a bit of an issue as it add another L2 on infrastructure that doesn't really need that. | 13:41 |
| Fl1nt | on the underlay I mean | 13:41 |
| bbezak | yeah, unnecessary complexity, but vlans are cheap | 13:42 |
| chembervint | bbezak: but in case of VLANs we are depends on L2 for computes aggregate for amphoras ... | 13:42 |
| SvenKieske | well if you want to have a "pure" L3 network it might not work | 13:42 |
| bbezak | chembervint: true, amphorae needs this vlan. additional vlan on physnet for vms indeed | 13:43 |
| chembervint | I agree. l2 could be an option. but in common case we have to support "pure" l3 | 13:43 |
| bbezak | some people even can't mix those traffics | 13:44 |
| bbezak | (security) | 13:44 |
| bbezak | I meant fabrics | 13:44 |
| Fl1nt | yep | 13:44 |
| SvenKieske | soo..maybe not that simple? what deployment scenarios do we want to "support"/care about? ;) | 13:45 |
| Fl1nt | and Octavia teams just keep repeating everyone pure L3 is supported so it's a bit hard to fight in a meeting about that. | 13:45 |
| SvenKieske | we could of course start with l2 vlan and add more stuff later, maybe. | 13:45 |
| Fl1nt | yes, L2 scenario then L3 then mix ? | 13:47 |
| chembervint | I suggest to start with merging my patch, which will fix current logic and current deployments ;) and continue to investigate on it | 13:47 |
| Fl1nt | Of course | 13:47 |
| bbezak | ok, let's continue outside of the meeting, thank you all! | 13:49 |
| bbezak | #endmeeting | 13:49 |
| opendevmeet | Meeting ended Wed Jul 31 13:49:55 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:49 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-07-31-13.00.html | 13:49 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-07-31-13.00.txt | 13:49 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-07-31-13.00.log.html | 13:49 |
| SvenKieske | thanks bbezak | 13:52 |
| Fl1nt | bbezak, I've a question for you | 13:52 |
| bbezak | shoot Fl1nt | 13:53 |
| chembervint | thanks to all! :) | 13:53 |
| Fl1nt | bbezak, I've catch a bug on CEPH RadosGW Pacific with Swift StaticWeb and it seems you've got the same issue on Reef telling by the #48382 bugreport, do you have a workaround so far? I've my RGW that throw 404 half time and I think its related to web-index of the container and sub folders. | 13:54 |
| Fl1nt | chembervint, you're welcome | 13:54 |
| bbezak | unfortunately I don't have a solution for you :/ Fl1nt | 13:55 |
| bbezak | half time only? | 13:55 |
| Fl1nt | bbezak, If it can help, I've the same RGW instance that always 200 and 2/3 that 404, if I force only to use this one through sticky session on haproxy VIP it kinda work. | 13:55 |
| Fl1nt | bbezak, yes 50% time 100% reproducible on a 3 node. | 13:56 |
| bbezak | maybe fill that to bug report, but probably nobody will reply anyway | 13:56 |
| bbezak | I've also sent a mail to mailing list | 13:57 |
| Fl1nt | I've registered on redmine but I need an administrator approval... | 13:57 |
| bbezak | https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/3C4JRQ4HMYFBD3U6JYIKAY2ULJ3AQI5O/ | 13:58 |
| Fl1nt | rgw01 always answer correctly 200 on object, rgw03 always answer with 404 NoSuchKey would it be on / or /subsection/ where rgw01 correctly translate 100% time / to /index.html and fail to do the same on /subsection. | 13:58 |
| Fl1nt | bbezak, gonna answer on the mailing to add a bit of support. | 13:58 |
| bbezak | cool, thx Fl1nt | 13:59 |
| Fl1nt | fun fact, I'm having a bit different issue as my 404 isn't about NoSuchBucket but NoSuchKey which is related to access_key on code but swift is using #anonymous as key. | 14:00 |
| Fl1nt | bbezak, done, let's hope someone will catch the bottle :D | 14:34 |
| Fl1nt | but I think I've tracked down the issue already, I'll test few things on tomorrow. | 14:34 |
| opendevreview | Matúš Jenča proposed openstack/kolla-ansible master: Add backend TLS between MariaDB and ProxySQL https://review.opendev.org/c/openstack/kolla-ansible/+/909912 | 15:36 |
| opendevreview | Matúš Jenča proposed openstack/kolla-ansible master: Add backend TLS between MariaDB and ProxySQL https://review.opendev.org/c/openstack/kolla-ansible/+/909912 | 15:40 |
| wncslln | hey kolla o/, about this mail https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/IYI3KHNDG2VODNYBBGVOCOF7SOKZVWH5/ | 16:37 |
| wncslln | I did not find where all_computes_in_batch was definied | 16:38 |
| wncslln | anyone can show me? | 16:38 |
| Fl1nt | wncslln, It is calculated by the task | 17:58 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!