Wednesday, 2024-07-31

opendevreviewRoman Krček proposed openstack/kolla-ansible master: Refactor services' check-containers and optimise  https://review.opendev.org/c/openstack/kolla-ansible/+/77324307:51
opendevreviewRoman Krček proposed openstack/kolla-ansible master: Fix unintentional trigger of ansible handlers  https://review.opendev.org/c/openstack/kolla-ansible/+/92414507:53
SvenKieskefrickler: mhm, we might have a little licence problem here: https://opendev.org/openstack/kolla-ansible/src/branch/master/tests/j2lint.py at least I can't find dbad on the OSI list and the language looks very ambigous to me. I filed an upstream issue already if they can maybe dual licence this..09:30
SvenKieskeI filed a bug for easier tracking: https://bugs.launchpad.net/kolla-ansible/+bug/207531609:39
deflatedMorning people, getting errors trying to deploy cloudkitty, doesnt matter if i use influx/opensearch/elasticsearch theres always some error (aiming to use opensearch if it matters, tried the others just to see if any worked), pasted the errors here:https://paste.openstack.org/show/brXI10k9ZnlwFZoSf35d/ Any ideas where i am going wrong?09:50
SvenKieskedeflated: on what version of openstack are you running? is this test or prod deployment? are you sure the network connection works?10:01
deflatedPre prod, when i get this working it'll be ready for prod, 2024.1, everything else works and i can see the right stuff in networking (eg ovs-vsctl show etc)10:02
deflatedWell, i say it'll be ready, i havent got internal tls working yet (int/ext i have) so this and that and a rebuild for final production10:03
SvenKieskeis this a single influx/opensearch node or cluster?10:05
deflatedits a multinode deployment, influx was justa  test, but opensearch is clustered10:06
SvenKieskewell I have seen working cloudkitty deployments I think but it has been some time, at least I'm currently not aware if it works and I fear there are not many (none?) integration tests actually testing it works (well deployment should work I guess)10:07
SvenKieskeDo you happen to have a look at openstack logs as well?10:08
SvenKieskeit's probably worth it to file a bug and send an email to the mailing list about this, as many people are on vacation I think. I'm not deeply familiar with cloudkitty, just have used it in test environments myself.10:11
SvenKieskedeflated: https://bugs.launchpad.net/kolla-ansible10:11
SvenKieskea bug report with as much information as you can provide would be very helpful :)10:12
deflatedI have and there is some extended error info in cloudkitty-api.log, i'll get a bug report posted with as much info as i can shortly10:13
opendevreviewMaximilian Stinsky proposed openstack/kolla-ansible master: Implement neutron-ovn-vpn-agent  https://review.opendev.org/c/openstack/kolla-ansible/+/92457510:14
oliI have enabled login with keycloak and it works for the horizon dashboard but doesn't for the skyline dashboard, I get a 401 unauthorized, and the skyline dashboard is in the keycloak.conf as a trusted dashboad. Any pointers on where to look next?10:27
oliI can login to skyline using the keystone credentials but no the openid ones, it doens't even redirect me to the provider login 10:29
deflatedSo, on to internal tls, im pretty much in the same boat as this guy on the mailing list https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/NLWURK5LBF5ZU4KLUMFNOTAOZGTS7FCZ/ How do we use official certs for internal tls?11:22
opendevreviewMerged openstack/kayobe master: Stop replacing dashes with underscores for Kolla interfaces  https://review.opendev.org/c/openstack/kayobe/+/92248012:07
opendevreviewPierre Riteau proposed openstack/kayobe stable/2024.1: Stop replacing dashes with underscores for Kolla interfaces  https://review.opendev.org/c/openstack/kayobe/+/92539012:39
opendevreviewPierre Riteau proposed openstack/kayobe stable/2023.2: Stop replacing dashes with underscores for Kolla interfaces  https://review.opendev.org/c/openstack/kayobe/+/92539112:39
opendevreviewPierre Riteau proposed openstack/kayobe stable/2023.1: Stop replacing dashes with underscores for Kolla interfaces  https://review.opendev.org/c/openstack/kayobe/+/92539212:40
bbezakHi. meeting in 5 - mgoddard mnasiadka bbezak frickler kevko SvenKieske mmalchuk gkoper jangutter jsuazo jovial osmanlicilegi mattcrees dougszu darmach12:55
bbezak#startmeeting kolla13:00
opendevmeetMeeting started Wed Jul 31 13:00:38 2024 UTC and is due to finish in 60 minutes.  The chair is bbezak. Information about MeetBot at http://wiki.debian.org/MeetBot.13:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.13:00
opendevmeetThe meeting name has been set to 'kolla'13:00
bbezak#topic rollcall13:00
SvenKieskeo/13:01
r-krcek\o/13:01
mhinero/13:01
bbezaknot many of us, vacation period I guess ;)13:02
bbezak#topic agenda13:02
mmalchuko/13:02
mattcreeso/13:02
bbezak* Roll-call13:02
bbezak* Agenda13:02
bbezak* Announcements13:02
bbezak* Review action items from the last meeting13:02
bbezak* CI status13:02
bbezak* Release tasks13:02
bbezak* Regular stable releases (first meeting in a month)13:02
bbezak* Current cycle planning13:02
bbezak* Additional agenda (from whiteboard)13:02
bbezak* Open discussion13:02
bbezak#topic CI status13:03
Fl1ntHi everyone13:03
bbezaklooks green13:03
Fl1ntbbezak, The men I was looking for!13:03
bbezak:)13:03
bbezakwe're on CI status topic btw :)13:04
SvenKieskeperiodic pipeline looks not so green? https://bugs.launchpad.net/kolla-ansible/+bug/207531613:04
SvenKieskeah sorry, wrong link13:04
SvenKieskehttps://zuul.openstack.org/builds?project=openstack%2Fkolla&pipeline=periodic&result=FAILURE&result=RETRY_LIMIT&result=POST_FAILURE&result=NODE_FAILURE&result=SKIPPED&skip=013:04
Fl1ntbbezak, just saw that np13:04
Fl1ntI'll wait :D13:04
SvenKieskeperiodic-weekly looks better13:05
SvenKieskegateway timeout, probably infra related? https://zuul.openstack.org/build/9e07eb72ee3f44c3b00e13468bfe2e83 I still need to subscribe to the opendev infra ML. don't know if there was some expected outage13:06
bbezakexcept arm64 it doesn't look bad13:07
Fl1ntSvenKieske, OVH had a lot of trouble this week with various HW/Fiber incident.13:07
bbezaklast several days are ok13:08
bbezak#topic Release tasks13:08
bbezakwe're r-913:09
SvenKieskeFl1nt: ah good to know13:09
bbezaktime flies ;)13:09
bbezaknot much to do now, next week there will be some tasks13:09
bbezak#topic Regular stable releases13:09
bbezakI've raised change to have a new minor releases13:10
bbezakas we didn't do that for several months already13:10
* frickler needs to review that one13:10
bbezakhttps://review.opendev.org/c/openstack/releases/+/92518213:10
bbezakyeap :)13:10
bbezak#topic Current cycle planning13:11
bbezakI was on vacation last couple of weeks, so I didn't progress much ;)13:11
bbezakI'll check with darmach the ubuntu noble progress13:11
bbezakoh, he is now. how is it doing darmach?13:12
bbezakin any case, we need to finish that soon'ish13:12
bbezakif you need anything around this cycle priorities, let us know - https://etherpad.opendev.org/p/KollaWhiteBoard#L22713:13
bbezak#topic Additional agenda (from whiteboard)13:14
bbezakhttps://etherpad.opendev.org/p/KollaWhiteBoard#L7013:14
bbezakmhiner and r-krcek changes are there13:15
mhineryes, additionally on behalf of ihalomi and regarding failed tests in 908295:13:16
mhinerUpgrade tests are failing because we use stable release for ansible-collection-kolla that doesn't include repository that contains docker 6.0.0 and during upgrade bootstrap-container role is called only once before first deploy, that means even we change later to correct ansible-collection-kolla it doesn't effect installed version of docker so test fails with cgroupns missing.13:16
SvenKieskeI +1 the question wrt to bandit from r-krcek as I still need to make myself more familiar with it. this is in the context of the kolla-ansible "binary" python rewrite.13:16
mhinerMy proposed solution is to cherrypick this commit to stable branch13:16
mhinerhttps://github.com/openstack/ansible-collection-kolla/commit/42116ded107f99983ebdcb0c70e8a8c4cd6fdc5213:16
bbezakso something that will be fixed when frickler will merge release change :)13:17
SvenKieskeah right iahlomi asked me about that one a few days ago, I agree we need to fix the docker stuff13:17
SvenKieskebbezak: mhm, wait, shouldn't we follow our own suggestions to the users and install from stable git branches instead of released artifacts? ;) (only half joking)13:18
bbezakyeah, there is some inconsistency there13:19
bbezakI'll look into the docker stuff mhiner - hopefully tomorrow13:20
mhinerthanks13:20
bbezakand to review some patches as well13:21
SvenKieskegood to know that was already fixed, I thought we had some error in our upgrade testing at first13:22
bbezakok, I'll book some time for those patches, let's move on13:23
bbezak#topic Open discussion13:23
chembervintHi, may I up my patch and kindly ask to review it? https://review.opendev.org/c/openstack/kolla-ansible/+/920377. 13:23
chembervintand 1 more - it's already merged to master, but cherry-picks has stolen .. https://review.opendev.org/c/openstack/kolla-ansible/+/92432213:24
SvenKieskewell I hate to be "that guy", but I filed https://bugs.launchpad.net/kolla-ansible/+bug/2075316 because I think we are violating upstreams licence, I asked upstream if they can possibly relicence/dual-licence but I don't think our changes to their code are substantial, IANAL.13:24
SvenKieskechembervint: I guess the backports will eventually be processed :)13:25
SvenKieskethe octavia stuff is annoying, I don't know if we can actually implement a better solution than what you proposed, but I guess mnasiadka is on vacation13:26
bbezakyeah, will push those backports13:26
Fl1ntbbezak, I really like the DON'T BE A DICK public license :D13:26
Fl1ntEspecially the share the love money :D13:26
bbezakimho looks pretty permissive :)13:26
bbezakjust don't be a dick13:26
Fl1ntyep13:27
chembervintSvenKieske: ok, thank you! will wait mnasiadka13:27
SvenKieskewell, personally I think it's a "funny" licence (humor is subjective). professionally it's annoying because it's not OSI approved and I don't think we can just wrap it in ASL 2.0, so well, yeah.13:27
fricklerfor octavia I still think configuring the interface statically is the better solution, just a bit more work13:28
Fl1ntSvenKieske, I like introducing those license to my colleagues of the licensing department, they usually lol a lot and then send a simple: Approved because... Why not.13:28
SvenKieskethe TC guidelines are pretty clear on that, I also went around, there are multiple other jinja linters which are better maintained, maybe I can add one of those, but that would incur more overhead..13:28
SvenKieskethe original question why I looked at it came up in https://review.opendev.org/c/openstack/kolla-ansible/+/909912/comment/4108e5e2_ad15207a/13:29
chembervintfrickler: ok, but how to do it in a proper way? I've described my thoughts in a ticket. if you have some ideas - I'll be glad to try it13:29
SvenKieskebecause I think we need to extend it, and we are also missing some commits from upstream which we might/might not want13:29
Fl1ntCan't we use the one used by ansible?13:29
SvenKieskeafaik we already do and this is just an additional linter13:31
Fl1ntDoes this additional linter brings something special/desirable?13:31
SvenKieskeso we certainly use "ansible-lint". in python/ansible lint there is a ton of different linters with varying degrees of coverage..13:31
chembervintfrickler: I mean, first off all in any case it will be still a ovs port, so we have to wait OVS container is up and running, and handle it somehow in systemd. and then - we are support different linux distros with different network configuration staff13:31
SvenKieskeFl1nt: I'm not certain; maybe I take some more minutes to investigate if we can just rip it out for good :)13:31
Fl1ntYeah, could be a much more simpler way to solve the issue :D13:32
SvenKieskeI guess the octavia discussion is more interesting/pressing and has more merit :)13:32
SvenKieskeso we would need a dedicated play I guess, which does the right thing with the ip config stuff depending on the distro (sounds like a thing ansible was basically invented for?)13:33
fricklerchembervint: maybe if you have so much trouble with ovs, you can use a vlan interface instead? anyway, yes, I was understating the effort when saying "a bit of work"13:33
SvenKieske:D13:34
fricklersolving your questions would actually be part of that task13:34
Fl1ntOctavia is an heavy used service, with very poor architectural design, we have deployed it on our clusters... I'm always having questions about it that I can't solve without having to extensively discuss with the Octavia guys.13:34
chembervintOne more question - one of our clients requires us to avoid storing passwords in plain-text in config files. We've tested oslo.config + Castellan stuff and already have a working prototype. And now working on non-openstack services as well. Could it be interesting to upstream such security stuff?13:34
Fl1ntchembervint, yes, very much13:35
SvenKieskechembervint: I think so, yes13:35
Fl1ntI have a lot of our customers that are specifically asking us for such solution.13:36
fricklerI've started to look at using application credentials instead, but not sure whether the secret being used there still fits the same category13:36
Fl1ntand end up implementing vault from hashicorp... which isn't really integrated.13:36
chembervintFl1nt: exactly. castellan + vault13:36
bbezakcastellan can also talk to hashi13:37
bbezakyeap13:37
Fl1ntyep13:37
SvenKieskemhm, what do you guys think about castellan vs barbican, if it's not too off topic?13:37
Fl1ntcastellan is barbican :D13:38
Fl1ntbarbican leverage castellan lib13:38
chembervintfrickler: in the past we've used vlans, before it was automated in k-a to config tenant network for octavia :) but it a bit harder to automate in in common case ... 13:38
mattcreesJust to jump in for open discussion, we're getting close to rabbit being ready for slurp a->c. I'm looking for a second core review for merging this patch chain please: https://review.opendev.org/c/openstack/kolla/+/918974 then any reviews welcome here: https://review.opendev.org/c/openstack/kolla-ansible/+/91897613:39
chembervintcastellan in just a supported backend for oslo.config ... barbican also could use it as an option13:39
bbezakwe're using vlans via kayobe for octavia network - so automated there13:40
bbezakthx mattcrees, we need second core outside of stackhpc kevko frickler please take a look13:41
Fl1ntusing vlans with Octavia really is the easiest way to go BUT it bring a bit of an issue as it add another L2 on infrastructure that doesn't really need that.13:41
Fl1nton the underlay I mean13:41
bbezakyeah, unnecessary complexity, but vlans are cheap13:42
chembervintbbezak: but in case of VLANs we are depends on L2 for computes aggregate for amphoras ... 13:42
SvenKieskewell if you want to have a "pure" L3 network it might not work13:42
bbezakchembervint: true, amphorae needs this vlan. additional vlan on physnet for vms indeed13:43
chembervintI agree. l2 could be an option. but in common case we have to support "pure" l313:43
bbezaksome people even can't mix those traffics13:44
bbezak(security)13:44
bbezakI meant fabrics13:44
Fl1ntyep13:44
SvenKieskesoo..maybe not that simple? what deployment scenarios do we want to "support"/care about? ;)13:45
Fl1ntand Octavia teams just keep repeating everyone  pure L3 is supported so it's a bit hard to fight in a meeting about that.13:45
SvenKieskewe could of course start with l2 vlan and add more stuff later, maybe.13:45
Fl1ntyes, L2 scenario then L3 then mix ?13:47
chembervintI suggest to start with merging my patch, which will fix current logic and current deployments ;) and continue to investigate on it13:47
Fl1ntOf course13:47
bbezakok, let's continue outside of the meeting, thank you all!13:49
bbezak#endmeeting13:49
opendevmeetMeeting ended Wed Jul 31 13:49:55 2024 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)13:49
opendevmeetMinutes:        https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-07-31-13.00.html13:49
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-07-31-13.00.txt13:49
opendevmeetLog:            https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-07-31-13.00.log.html13:49
SvenKieskethanks bbezak13:52
Fl1ntbbezak, I've a question for you13:52
bbezakshoot Fl1nt13:53
chembervintthanks to all! :)13:53
Fl1ntbbezak, I've catch a bug on CEPH RadosGW Pacific with Swift StaticWeb and it seems you've got the same issue on Reef telling by the #48382 bugreport, do you have a workaround so far? I've my RGW that throw 404 half time and I think its related to web-index of the container and sub folders.13:54
Fl1ntchembervint, you're welcome13:54
bbezakunfortunately I don't have a solution for you :/ Fl1nt13:55
bbezakhalf time only?13:55
Fl1ntbbezak, If it can help, I've the same RGW instance that always 200 and 2/3 that 404, if I force only to use this one through sticky session on haproxy VIP it kinda work.13:55
Fl1ntbbezak, yes 50% time 100% reproducible on a 3 node.13:56
bbezakmaybe fill that to bug report, but probably nobody will reply anyway13:56
bbezakI've also sent a mail to mailing list13:57
Fl1ntI've registered on redmine but I need an administrator approval...13:57
bbezakhttps://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/3C4JRQ4HMYFBD3U6JYIKAY2ULJ3AQI5O/13:58
Fl1ntrgw01 always answer correctly 200 on object, rgw03 always answer with 404 NoSuchKey would it be on / or /subsection/ where rgw01 correctly translate 100% time / to /index.html and fail to do the same on /subsection.13:58
Fl1ntbbezak, gonna answer on the mailing to add a bit of support.13:58
bbezakcool, thx Fl1nt13:59
Fl1ntfun fact, I'm having a bit different issue as my 404 isn't about NoSuchBucket but NoSuchKey which is related to access_key on code but swift is using #anonymous as key.14:00
Fl1ntbbezak, done, let's hope someone will catch the bottle :D14:34
Fl1ntbut I think I've tracked down the issue already, I'll test few things on tomorrow.14:34
opendevreviewMatúš Jenča proposed openstack/kolla-ansible master: Add backend TLS between MariaDB and ProxySQL  https://review.opendev.org/c/openstack/kolla-ansible/+/90991215:36
opendevreviewMatúš Jenča proposed openstack/kolla-ansible master: Add backend TLS between MariaDB and ProxySQL  https://review.opendev.org/c/openstack/kolla-ansible/+/90991215:40
wncsllnhey kolla o/, about this mail https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/IYI3KHNDG2VODNYBBGVOCOF7SOKZVWH5/16:37
wncsllnI did not find where all_computes_in_batch was definied16:38
wncsllnanyone can show me?16:38
Fl1ntwncslln, It is calculated by the task17:58

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!