| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2023.2: Remove default value from sample configuration https://review.opendev.org/c/openstack/kayobe/+/911074 | 07:58 |
|---|---|---|
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/2023.1: Remove default value from sample configuration https://review.opendev.org/c/openstack/kayobe/+/911075 | 07:58 |
| opendevreview | Pierre Riteau proposed openstack/kayobe stable/zed: Remove default value from sample configuration https://review.opendev.org/c/openstack/kayobe/+/911076 | 07:59 |
| mnasiadka | kevko: I have bad memories with backporting features that require new inventory groups, otherwise I guess it's ok ;-) | 08:48 |
| kevko | mnasiadka: I understand that it can be a problem ..but also inventory change can be problematic or not ... if it is totally new group which works only with some enable_something ..it should be also OK... | 09:14 |
| mnasiadka | problem is it just doesn't run/apply and user doesn't know why | 09:17 |
| mnasiadka | we could add prechecks if inventory exists - but then it will fail on people that didn't update inventory | 09:17 |
| mnasiadka | nothing is ideal, so I prefer to not backport such changes | 09:17 |
| mnasiadka | but all others are ok from my perspective | 09:17 |
| opendevreview | Ivan Halomi proposed openstack/kolla-ansible master: Refactor of kolla_container_facts https://review.opendev.org/c/openstack/kolla-ansible/+/911417 | 09:57 |
| opendevreview | Matt Crees proposed openstack/kolla-ansible stable/2023.1: Add precheck for RabbitMQ quorum queues https://review.opendev.org/c/openstack/kolla-ansible/+/909967 | 10:02 |
| opendevreview | Matt Crees proposed openstack/kolla-ansible stable/2023.1: Rework quorum queues precheck https://review.opendev.org/c/openstack/kolla-ansible/+/909968 | 10:02 |
| opendevreview | Matt Crees proposed openstack/kolla-ansible stable/2023.1: RabbitMQ: correct docs on Quorum Queue migrations https://review.opendev.org/c/openstack/kolla-ansible/+/909969 | 10:02 |
| opendevreview | Matt Crees proposed openstack/kolla-ansible master: CI: Only migrate RMQ queues during SLURP https://review.opendev.org/c/openstack/kolla-ansible/+/909971 | 10:12 |
| *** mrunge_ is now known as mrunge | 10:22 | |
| opendevreview | Merged openstack/kolla-ansible stable/2023.2: Configure missing nova services to expose vendordata over configdrive https://review.opendev.org/c/openstack/kolla-ansible/+/911067 | 10:42 |
| opendevreview | Will Szumski proposed openstack/ansible-collection-kolla master: Add feature to use binary version of crun https://review.opendev.org/c/openstack/ansible-collection-kolla/+/911422 | 10:54 |
| opendevreview | Will Szumski proposed openstack/kayobe master: WIP: Add podman support https://review.opendev.org/c/openstack/kayobe/+/909686 | 11:11 |
| opendevreview | Ivan Halomi proposed openstack/kolla-ansible master: Refactor of kolla_container_facts https://review.opendev.org/c/openstack/kolla-ansible/+/911417 | 11:13 |
| opendevreview | Will Szumski proposed openstack/ansible-collection-kolla master: Add feature to use binary version of crun https://review.opendev.org/c/openstack/ansible-collection-kolla/+/911422 | 11:24 |
| opendevreview | Will Szumski proposed openstack/kayobe master: Add podman support https://review.opendev.org/c/openstack/kayobe/+/909686 | 11:25 |
| opendevreview | Merged openstack/kayobe stable/zed: Remove default value from sample configuration https://review.opendev.org/c/openstack/kayobe/+/911076 | 11:35 |
| WJeffs7508 | Morning, Anyone had any experience in tuning for 100GBE cards on kolla? We appear to be hitting a 15Gbit/s limit inside VMs/OVS, but host to host get ~98Gbit/s | 11:46 |
| mnasiadka | are you using SRIOV, or just virtio? | 11:49 |
| WJeffs7508 | just virtio, but I'm wondering if this is part of the issue. | 11:49 |
| mnasiadka | it is, vif multiqueue might help a bit, usually SRIOV is the way though | 11:55 |
| WJeffs7508 | Yea I was starting to get down that line to think it was the option - but was hoping it wasn't the only way. | 11:55 |
| kevko | mnasiadka: btw, this is nice idea ..add some precheck for inventory change ... | 11:58 |
| opendevreview | Ivan Halomi proposed openstack/kolla-ansible master: Move actions to kolla_container_facts https://review.opendev.org/c/openstack/kolla-ansible/+/911505 | 11:59 |
| kevko | mnasiadka: btw, yesterday i was walking around setuptools, pbr, wsgi-scripts stuff , pip resolver, poetry ...etc ... do you have a plan how to deal with this in kolla images ? If i am correct .. we are not updating setuptools (and maybe others) for now because of that ...am I right ? | 12:02 |
| mnasiadka | we have setuptools pinned because it broke horizon | 12:05 |
| mnasiadka | I don't know if that's still the case | 12:05 |
| opendevreview | Ivan Halomi proposed openstack/kolla-ansible master: Refactor of kolla_container_facts https://review.opendev.org/c/openstack/kolla-ansible/+/911417 | 12:05 |
| opendevreview | Michal Nasiadka proposed openstack/kolla master: Remove setuptools pin https://review.opendev.org/c/openstack/kolla/+/901828 | 12:06 |
| mnasiadka | let's see | 12:06 |
| kevko | mnasiadka: I meant email chain I've added you into right now | 12:14 |
| mnasiadka | ah, that drama | 12:15 |
| kevko | mnasiadka: yep | 12:18 |
| opendevreview | Ivan Halomi proposed openstack/kolla-ansible master: Refactor of kolla_container_facts https://review.opendev.org/c/openstack/kolla-ansible/+/911417 | 12:19 |
| opendevreview | Ivan Halomi proposed openstack/kolla-ansible master: Move actions to kolla_container_facts https://review.opendev.org/c/openstack/kolla-ansible/+/911505 | 12:19 |
| opendevreview | Merged openstack/kayobe stable/2023.2: Remove default value from sample configuration https://review.opendev.org/c/openstack/kayobe/+/911074 | 12:21 |
| opendevreview | Merged openstack/kayobe stable/2023.1: Remove default value from sample configuration https://review.opendev.org/c/openstack/kayobe/+/911075 | 12:21 |
| opendevreview | Merged openstack/kolla-ansible master: prometheus: Add friendly instance labels for ironic and alertmanager https://review.opendev.org/c/openstack/kolla-ansible/+/899614 | 12:27 |
| opendevreview | Merged openstack/kolla-ansible master: CI: Replace etcd with redis in GATE_IMAGES for cephadm scenario https://review.opendev.org/c/openstack/kolla-ansible/+/911143 | 12:28 |
| kevko | mnasiadka: hmm, it seems that change which is creating bridges in ovs not working if I am using two bridges :/ | 13:20 |
| kevko | mnasiadka: weird https://paste.openstack.org/show/brVDt8fDDsrmFmuvzNNR/ | 13:21 |
| SvenKieske | mhm, I wonder if I hit the same problem in ovs-exporter | 13:21 |
| SvenKieske | ah no, that is a different task (I guess) | 13:21 |
| kevko | https://ara.master.ultimum.cloud/results/4049.html | 13:24 |
| kevko | it looks like loop didn't work | 13:25 |
| kevko | mnasiadka: do we have somewhere two bridges configured ? | 13:30 |
| kevko | i mean in CI | 13:30 |
| SvenKieske | mhm wait a second, you said loop, that reminds me of something.. | 13:36 |
| SvenKieske | wondering if this is somehow related again to https://github.com/ansible/ansible/issues/80848 | 13:39 |
| kevko | SvenKieske: I am using containerized kolla-ansible container and I have this patch included for several releases and I am very glad I have patched ansible ... | 13:42 |
| kevko | SvenKieske: I've already check it ...no it's not it | 13:43 |
| SvenKieske | yes, just rechecked the kolla_toolbox.py and it seems it can't be the culprit, mhm. | 13:44 |
| SvenKieske | well, it's really curious that I have also a problem in a similar area, when creating ovs-bridges, exactly in this task: https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/common/defaults/main.yml#L110 | 13:45 |
| SvenKieske | but it fails because it can't connect to the ovs db socket because it isn't there, but it should be there (see https://review.opendev.org/c/openstack/kolla-ansible/+/855498/comments/1e150cc2_74f26b6f for some logs) | 13:46 |
| SvenKieske | so there's a difference to your problem | 13:46 |
| kevko | SvenKieske: I think it's somewhere in kolla-toolbox ... already debugging | 13:47 |
| SvenKieske | the /run (/var/run is just a symlink) directory is part of kolla-toolbox and I don't understand why kolla wouldn't be able to connect. that being said I wonder what are the permission bits on the socket.. | 13:47 |
| SvenKieske | the thing is; I'm pretty sure this whole stuff won't work in podman because we simply don't mount /run there shared, because that is not possible, by deliberate design in podman (I linked afaik a bugreport in the comments above somewhere) | 13:48 |
| SvenKieske | so I'm also thinking on how to share this socket in a multi r/w fashion with different containers in the podman world | 13:48 |
| kevko | SvenKieske: if I remember i was also reading documentation a lot and it was everything about r/w shared ...etc etc :D | 13:50 |
| SvenKieske | yes, but that's just not possible from podmans pov, what makes it worse, I can even understand the reasoning :D | 13:54 |
| SvenKieske | https://github.com/containers/podman/issues/16305#issuecomment-1299810004 | 13:55 |
| SvenKieske | mhm, I guess I need to look at it from another angle. So the /run can be mounted r/w/shared, but of course only once, because shared means it gets mounted on the host mount namespace, and once that is done it doesn't make sense to mount it again | 13:56 |
| SvenKieske | so I guess we could "just" add templating logic to detect if this is already mounted and if yes don't try to mount, if no mount it? | 13:57 |
| SvenKieske | the logic involved might get rather complex though :D | 13:57 |
| kevko | the problem is with ovn exporter right ? https://review.opendev.org/c/openstack/kolla-ansible/+/855498 | 13:59 |
| SvenKieske | yes | 13:59 |
| SvenKieske | I want to finally get it finished, but the containerized deployment still has some issues. | 14:00 |
| kevko | SvenKieske: I will check also right after my issue will be solved | 14:01 |
| SvenKieske | ty, so your issues is that the loop doesn't really create the two bridges? I haven't really seen an helpful error message there though? | 14:02 |
| frickler | mnasiadka: no warning, no meeting? | 14:02 |
| kevko | I am already here :D | 14:03 |
| SvenKieske | ah right, it's already 2 past 3 | 14:03 |
| spatel | mnasiadka hey! | 14:04 |
| spatel | quick question, How does kolla mount NFS share on compute nodes when using NFS backend for cinder-volume? | 14:05 |
| mnasiadka | meeting time | 14:06 |
| mnasiadka | sorry, train was delayed | 14:06 |
| mnasiadka | #startmeeting kolla | 14:06 |
| opendevmeet | Meeting started Wed Mar 6 14:06:40 2024 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:06 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:06 |
| opendevmeet | The meeting name has been set to 'kolla' | 14:06 |
| frickler | train is EOL *scnr* | 14:06 |
| mnasiadka | mgoddard mnasiadka bbezak frickler kevko SvenKieske mmalchuk gkoper jangutter jsuazo jovial osmanlicilegi mattcrees dougszu - meeting now | 14:07 |
| mnasiadka | #topic rollcall | 14:07 |
| mnasiadka | o/ | 14:07 |
| frickler | \o | 14:07 |
| mmalchuk | \o | 14:07 |
| mattcrees | o/ | 14:07 |
| mhiner | o/ | 14:07 |
| r-krcek | o/ | 14:07 |
| SvenKieske | o/ | 14:07 |
| dougszu | |o | 14:07 |
| jangutter | o/ | 14:07 |
| jovial | 0/ | 14:08 |
| mnasiadka | #topic agenda | 14:09 |
| mnasiadka | * CI status | 14:09 |
| mnasiadka | * Release tasks | 14:09 |
| mnasiadka | * Regular stable releases (first meeting in a month) | 14:09 |
| mnasiadka | * Current cycle planning | 14:09 |
| mnasiadka | * Additional agenda (from whiteboard) | 14:09 |
| mnasiadka | * Open discussion | 14:09 |
| kevko | \o/ | 14:09 |
| mnasiadka | #topic CI status | 14:09 |
| mnasiadka | I think it's green-ish, there was some breakage due to switch from etcd to redis in cephadm jobs (kolla side image build regex was not updated) - but should be good now | 14:10 |
| jovial | Kayobe is green again this week | 14:10 |
| mnasiadka | aarch jobs got broken with sha256 checking of sources.py, I'll have a look | 14:10 |
| mnasiadka | otherwise looks good | 14:10 |
| mnasiadka | #topic Release tasks | 14:11 |
| mnasiadka | This week is R-4 - Cycle highlights | 14:11 |
| mnasiadka | Any volunteer to propose cycle highlights? | 14:11 |
| mnasiadka | bbezak: ? | 14:11 |
| mnasiadka | Ok, I'll do that ;-) | 14:12 |
| mnasiadka | jovial: can you do that for Kayobe? | 14:12 |
| jovial | Sure | 14:12 |
| mnasiadka | #topic Current cycle planning | 14:13 |
| jovial | Is there an example patch from last time? | 14:13 |
| mnasiadka | jovial: let me find it | 14:13 |
| frickler | it is in the deliverables file in the release repo | 14:13 |
| mnasiadka | #link https://review.opendev.org/c/openstack/releases/+/902420 | 14:13 |
| kevko | mnasiadka: well it's greenish ...but ovs is not creating second bridge if defines :) | 14:14 |
| WJeffs7508 | Anyone know why 2023.1 containers have been removed from Docker? Or if there is a archive somewhere? | 14:14 |
| mnasiadka | kevko: well, we can't test that in CI - I can have a look | 14:14 |
| kevko | mnasiadka: will be fixed in minutes i think | 14:14 |
| mnasiadka | WJeffs7508: we mainly publish to quay.io/openstack.kolla | 14:15 |
| jovial | thanks | 14:15 |
| mnasiadka | kevko: sure, if you need me to have a look in that just let me know | 14:15 |
| kevko | okay, fixed .. I will added you to review | 14:15 |
| frickler | kevko: thats sound more like a bug than a CI issue | 14:15 |
| SvenKieske | I guess s/mainly/only/ with regards to quay.io? | 14:15 |
| kevko | frickler: mmm...yes it's ...sorry bad interpretation | 14:15 |
| mnasiadka | SvenKieske: not only, we still have docker publish jobs, but I really don't look at them ;-) | 14:15 |
| mnasiadka | docker weekly I think | 14:16 |
| mnasiadka | quay daily | 14:16 |
| WJeffs7508 | mnasiadka: Just found them there :) Just wondered if there was a reason it wasn't on docker mostly. | 14:16 |
| WJeffs7508 | Wasn't sure if there was some major reason they got pulled I had missed. | 14:16 |
| mnasiadka | WJeffs7508: the reason is Docker in the past tried to screw all open source projects and required to move to paid teams | 14:16 |
| mnasiadka | they backed off that later - but the stench remains | 14:16 |
| frickler | WJeffs7508: also note those images are only meant for testing, build your own for production use | 14:16 |
| mnasiadka | that's another thing :) | 14:17 |
| WJeffs7508 | Yea agreed. We are :) | 14:17 |
| mnasiadka | ok then, back to the topic - current cycle planning | 14:17 |
| mnasiadka | #link https://etherpad.opendev.org/p/KollaWhiteBoard#L222 | 14:17 |
| mnasiadka | I added RMQ 3.13 | 14:17 |
| mnasiadka | Although it's only 3.13.0 now - I prefer to bump it now, than next cycle | 14:18 |
| mnasiadka | we still have some months to release | 14:18 |
| SvenKieske | yes | 14:18 |
| mnasiadka | and as promised on the PTG, we'll (SHPC) work on getting 3.13 to older stable releases | 14:18 |
| SvenKieske | did anybody look at some of the TLS patches? | 14:18 |
| mnasiadka | which ones? | 14:19 |
| frickler | best link them in the etherpad, using a common topic | 14:19 |
| mnasiadka | mattcrees: how is the Quorum queues in Antelope? | 14:19 |
| SvenKieske | #link https://review.opendev.org/c/openstack/kolla-ansible/+/909188 | 14:20 |
| SvenKieske | but the "redis-cache" topic doesn't have all afaik | 14:20 |
| mattcrees | There's some backports in a chain here: https://review.opendev.org/c/openstack/kolla-ansible/+/909967 and I've also changed Caracal to only migrate queues in SLURP: https://review.opendev.org/c/openstack/kolla-ansible/+/909971 | 14:20 |
| SvenKieske | where exactly should I put caracal PTG links in the whiteboard? | 14:21 |
| SvenKieske | ah I see, line 222ff | 14:21 |
| SvenKieske | put the link to the redis-cache topic there, but it's only a fraction of all the internal TLS stuff | 14:22 |
| SvenKieske | frickler, you promised to review last week afaik ;) | 14:23 |
| mnasiadka | promises, promises ;) | 14:23 |
| frickler | yes, sorry, I'll leave that page open now until I review | 14:23 |
| mnasiadka | frickler: are you going to work on split-glance this cycle? | 14:23 |
| SvenKieske | my promised doc patch also took 2 weeks until I got around to it.. | 14:24 |
| frickler | mnasiadka: I still intend to do so, yes | 14:24 |
| mnasiadka | good, if you change your mind - I might want to find somebody else to pick it up, it would be good security-wise to do that this cycle | 14:25 |
| frickler | well if someone else wants to work on that, I certainly wouldn't object | 14:25 |
| mnasiadka | let's go to next topic | 14:25 |
| mnasiadka | frickler: I didn't say that :) | 14:25 |
| mnasiadka | #topic Additional agenda (from whiteboard) | 14:26 |
| mnasiadka | SvenKieske: your TLS topic I assume was already mentioned | 14:26 |
| mnasiadka | (mhiner) Please review: | 14:26 |
| mnasiadka | Pointers on where and how to create documentation for ce migration? | 14:26 |
| mnasiadka | action option introduction to kolla_container_facts: https://review.opendev.org/c/openstack/kolla-ansible/+/911417 | 14:26 |
| mnasiadka | docker-py version bump: https://review.opendev.org/c/openstack/ansible-collection-kolla/+/910751 | 14:26 |
| mnasiadka | transition to high-level docker api: https://review.opendev.org/c/openstack/kolla-ansible/+/908295 | 14:26 |
| mhiner | Specifically, I would like to know how I can create that documentation page | 14:27 |
| SvenKieske | ah "ce migration" referse to container engine migration, that took me a while to parse. | 14:28 |
| mnasiadka | all the docs are under doc/source/ | 14:28 |
| mhiner | I see, I thought it was supposed to be a page on the docs.openstack.org site | 14:28 |
| kevko | sorry ..i need to go ..i will be around ..but i need to go to another VDI ..we have some incident ... | 14:28 |
| frickler | mhiner: a new page is just a new file. it will get pushed to the docs site automatically | 14:29 |
| mnasiadka | mhiner: it's in sphinx format, it gets generated in the docs job in CI - and then when merged published to docs.openstack.org | 14:29 |
| mnasiadka | mhiner: commented on the docker-py version bump, will have a look in the rest later | 14:30 |
| jovial | Also, you can use `tox -e docs` to generate locally | 14:30 |
| mnasiadka | ok, let's go forward | 14:30 |
| mnasiadka | #topic Open discussion | 14:30 |
| mnasiadka | Anybody anything? | 14:30 |
| frickler | mhiner: did that answer your question? | 14:31 |
| mhiner | yes, thank you | 14:31 |
| mnasiadka | mhiner: of course in any doubt - ask questions here :) | 14:31 |
| SvenKieske | just my small doc patch regarding --limit: https://review.opendev.org/c/openstack/kolla-ansible/+/911082 | 14:31 |
| r-krcek | Hi I would like to ask about my patch. It has been inactive for a while. Is there anything else I need to do? https://review.opendev.org/c/openstack/kolla-ansible/+/905831 | 14:31 |
| SvenKieske | and if anyone has any insight in ovs socket sharing across containers and how to make this sane I would appreciate any feedback, details are in the comments here: https://review.opendev.org/c/openstack/kolla-ansible/+/855498 | 14:32 |
| SvenKieske | I'm still working on it myself of course, but maybe need a different perspective to look at it. the CI fails at creating ovs-bridge. need also to rework how we mount /run/ so it works in podman (currently we just ignore this in podman) | 14:33 |
| r-krcek | (followup to my previous comment) SvenKieske mentioned backporting. Would that also be my responsibility to facilitate as part of the patch? | 14:34 |
| frickler | SvenKieske: iirc you can only mount explicit subdirs of /run with podman? | 14:34 |
| mhiner | I think that's how it's currently done for podman | 14:34 |
| SvenKieske | yeah, the problem, as I understood, is that the "share" option means sharing the "/run" dir with the complete host, which only makes sense if you do it once, not twice.. so we need basically a singleton mechanism that ensures this is only run once and then not again. or mount countless subdirs, but I doubt that is a solution because multiple containers need to mount the ovs.socket run dir, namely: kolla_ | 14:37 |
| SvenKieske | toolbox, prometheus-ovn-exporter, ovn(?) itself, probably more | 14:37 |
| SvenKieske | but r-krcek asked before I did and is possibly easier to answer as well :) | 14:37 |
| frickler | I don't think a patch introducing a new role would be backportable | 14:37 |
| mnasiadka | Yeah, if you want to backport that, then fix it without introducing a role | 14:38 |
| SvenKieske | currently it's not a role, so imho should be fine :) we can maybe move it later to a dedicated role. | 14:38 |
| mnasiadka | yeah, refactor to a separate role afterwards | 14:39 |
| r-krcek | Okay, so in this patch I should fix it without introducing a new role, which will be backported to stable releases and open a new patch that would break it out to separate role. | 14:41 |
| SvenKieske | correct. I think the patch is also in a decent shape, at least I don't see anything preventing a merge, but the core reviewers have the ultimate decision :) | 14:41 |
| r-krcek | okay, thank you :) | 14:42 |
| mnasiadka | ok then | 14:43 |
| mnasiadka | seems we're done | 14:43 |
| mnasiadka | thanks for coming! | 14:43 |
| mnasiadka | #endmeeting | 14:43 |
| opendevmeet | Meeting ended Wed Mar 6 14:43:55 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:43 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-03-06-14.06.html | 14:43 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-03-06-14.06.txt | 14:43 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/kolla/2024/kolla.2024-03-06-14.06.log.html | 14:43 |
| mmalchuk | thanks mnasiadka | 14:44 |
| SvenKieske | thanks! | 14:44 |
| SvenKieske | kevko: when you're back from your emergency: do you have a link to a patch regarding your ovs issue or is this just an issue you hit in a dev/live env? maybe a bugreport would be nice so we can properly debug it (if it exists in vanilla openstack) | 14:45 |
| kevko | SvenKieske: it's regular patch ..i will send a patch | 14:52 |
| kevko | SvenKieske: it's vanilla openstack bug | 14:52 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix creation of ovs bridges https://review.opendev.org/c/openstack/kolla-ansible/+/911591 | 15:23 |
| kevko | SvenKieske: ^^ https://bugs.launchpad.net/kolla-ansible/+bug/2056332 | 15:23 |
| SvenKieske | ty | 15:23 |
| kevko | SvenKieske: funny no ? :D | 15:24 |
| SvenKieske | yeah, was wondering if we should have dedicated tests for this stuff.. (could in theory checked with a "virtual" precheck I guess) | 15:26 |
| kevko | Well, this can be tested with Octavia for example ..as I have ... Not octavia_auto_configure | 15:45 |
| kevko | It's also more production setup ... | 15:46 |
| kevko | *production like | 15:46 |
| mnasiadka | uhh, keystone is broken by passlib | 16:00 |
| mnasiadka | 2024-03-06 13:39:50.483974 2024-03-06 13:39:50.482 38 ERROR passlib.handlers.bcrypt AttributeError: module 'bcrypt' has no attribute '__about__' | 16:00 |
| mnasiadka | frickler: https://review.opendev.org/c/openstack/requirements/+/910534 - that seems to be related ;-) | 16:00 |
| frickler | mnasiadka: nice, I have heard about that issue but not actually seen it happen yet | 16:01 |
| frickler | do you have a link to a failed build? I think we may need to patch reqs within kolla for that for now | 16:02 |
| mnasiadka | all jobs in https://review.opendev.org/c/openstack/kolla/+/911093 failed because of keystone | 16:02 |
| frickler | so there goes our greenish CI \o/ | 16:03 |
| SvenKieske | but was passlib updated, or keystone? looking at passlib, is this the still official website, https://passlib.readthedocs.io/en/stable/history/ ? it hasn't really got any updates? | 16:06 |
| mnasiadka | bcrypt was updated | 16:07 |
| mnasiadka | passlib is not maintained, or at least lousely maintained | 16:07 |
| mnasiadka | basically keystone needs to stop using passlib | 16:07 |
| SvenKieske | no update since 3 years, a little concerning for a security lib: https://foss.heptapod.net/python-libs/passlib | 16:07 |
| SvenKieske | pynacl would be better I guess, I don't know about the integration in keystone though. | 16:11 |
| SvenKieske | https://review.opendev.org/c/openstack/requirements/+/910534 | 16:12 |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/2023.1 failed: Add precheck for RabbitMQ quorum queues https://review.opendev.org/c/openstack/kolla-ansible/+/909967 | 16:13 |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/2023.1 failed: Rework quorum queues precheck https://review.opendev.org/c/openstack/kolla-ansible/+/909968 | 16:13 |
| opendevreview | Verification of a change to openstack/kolla-ansible stable/2023.1 failed: RabbitMQ: correct docs on Quorum Queue migrations https://review.opendev.org/c/openstack/kolla-ansible/+/909969 | 16:13 |
| SvenKieske | ansible apparently also relies on this unmaintained mess: https://github.com/ansible/ansible/issues/81949 | 16:15 |
| SvenKieske | nvm, it's afaik deprecated. | 16:18 |
| SvenKieske | no it's not..ansible source code is a mess | 16:18 |
| kevko | have anyone seen issue when some router just switching between nodes without the reason ? | 16:32 |
| kevko | btw ..router keepalived logging to syslog but we as kolla dropping this log :( | 16:33 |
| mnasiadka | kevko: it’s usually network problem or your host is too busy to respond to vrrp keepalives | 16:48 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: prometheus: Support overriding address of scrape targets https://review.opendev.org/c/openstack/kolla-ansible/+/899615 | 16:51 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: Revert "Allow setting any_errors_fatal true for gather-facts" https://review.opendev.org/c/openstack/kolla-ansible/+/910601 | 17:00 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: Support reducing scope of delegated fact gathering https://review.opendev.org/c/openstack/kolla-ansible/+/910503 | 17:00 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: Allow to continue when some hosts are unreachable https://review.opendev.org/c/openstack/kayobe/+/910509 | 17:04 |
| opendevreview | Mark Goddard proposed openstack/kayobe master: DNM: Test --continue-on-unreachable https://review.opendev.org/c/openstack/kayobe/+/910511 | 17:04 |
| mnasiadka | Hmm, maybe keystone is not always broken, weird | 17:24 |
| kevko | mnasiadka: well, OK, but how can I be sure what it is ? | 17:38 |
| frickler | mnasiadka: it seems the bcrypt error also appears in passing jobs like https://75f846b58e02305c95b0-4c4be73445f57ce8e9e80fbec5b1bb43.ssl.cf5.rackcdn.com/911093/2/check/kolla-ansible-ubuntu/f6cc93d/primary/logs/kolla/keystone/keystone-apache-public-error.txt | 17:40 |
| frickler | so the actual reason for the failure might be something completely different | 17:41 |
| SvenKieske | from that log it seems there's a workaround for bcrypt: | 17:57 |
| SvenKieske | 2024-03-05 16:06:22.664213 2024-03-05 16:06:22.663 1012 DEBUG passlib.handlers.bcrypt [None req-65773efd-6242-43ac-9a44-8144861c0726 - - - - - -] 'bcrypt' backend lacks $2$ support, enabling workaround _finalize_backend_mixin /var/lib/kolla/venv/lib/python3.10/site-packages/passlib/handlers/bcrypt.py:406\x1b[00m | 17:57 |
| opendevreview | Merged openstack/kolla-ansible stable/2023.1: Configure missing nova services to expose vendordata over configdrive https://review.opendev.org/c/openstack/kolla-ansible/+/911068 | 18:34 |
| opendevreview | Merged openstack/kolla-ansible stable/zed: Configure missing nova services to expose vendordata over configdrive https://review.opendev.org/c/openstack/kolla-ansible/+/911069 | 18:34 |
| mnasiadka | frickler: interesting | 18:48 |
| samcat116 | I'm working on a bobcat deployment and my keystone is constantly logging warnings about "Truncating password to algorithm specific maximum length 72 characters.". Has anyone else seen this? I don't have any users with passwords due to sso so I have to imagine its some internal service users that k-a is creating | 21:35 |
| samcat116 | I do see that in my passwords.yaml none of them are over 72 characters, so slightly confused | 21:35 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!