| *** atmark is now known as Guest1107 | 07:51 | |
| SvenKieske | fungi: re: OSSA-2023-003 just some general feedback for the next time: it might be worth to highlight it more upfront that non iscsi/FC deployments need to make config changes as well, people sadly tend to skim such announcements. I had people asking and experienced people answering "this is only about iscsi". so they obviously missed the config change part for everyone. | 08:41 |
|---|---|---|
| SvenKieske | tbf, you(?) clearly wrote that in the scope section | 08:42 |
| SvenKieske | communication is so difficult | 08:42 |
| sean-k-mooney | yes | 10:00 |
| SvenKieske | it at least mentions oslos "service_token_roles", the question is: which service token roles do we need to assign to nova and how do I enable that cinder checks this role in kolla-ansible? | 10:01 |
| SvenKieske | I'll dig into the oslo config docs, I'm not that familiar with the rbac stuff just yet.. | 10:02 |
| sean-k-mooney | so the service role existing in keystone for service users prior to our reuse of it for rbac | 10:02 |
| sean-k-mooney | so i belive its the service role that needs to be added to nova | 10:02 |
| sean-k-mooney | ill take a look at the cinder config ref and see if that line up | 10:03 |
| SvenKieske | so in oslo this should look like this: https://docs.openstack.org/oslo.policy/latest/user/usage.html#how-to-register but I agree, better to look up the cinder docs | 10:03 |
| sean-k-mooney | im just looking for that now but im not sure that cider as a singel unifed config referince doc lik noav | 10:04 |
| SvenKieske | ah there is some rbac stuff here: https://docs.openstack.org/cinder/latest/configuration/block-storage/policy-personas.html | 10:05 |
| SvenKieske | and i guess we need this, basically: https://docs.openstack.org/cinder/latest/configuration/block-storage/policy.html | 10:05 |
| sean-k-mooney | ya im trying to find the cinder version of https://docs.openstack.org/nova/latest/configuration/config.html | 10:05 |
| sean-k-mooney | but it might be quicker ot look at the code | 10:06 |
| sean-k-mooney | i dong think this is policy related | 10:06 |
| sean-k-mooney | its in the keystone_authtoken section | 10:06 |
| sean-k-mooney | so its part of keystone middlewaare | 10:06 |
| sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#keystone_authtoken.service_token_roles | 10:07 |
| sean-k-mooney | that should be common for all openstack serivces since its actullly handeld by the ketone libs not in the proejcts directly | 10:07 |
| sean-k-mooney | so yes the 'service' role is what is required | 10:08 |
| SvenKieske | found it for cinder: https://docs.openstack.org/cinder/latest/configuration/block-storage/service-token.html#configuration | 10:08 |
| SvenKieske | point 4: If Cinder is going to receive service tokens from other services it needs to have two options configured in the [keystone_authtoken] section of the configuration file: | 10:09 |
| sean-k-mooney | so we need to add [keystone_authtoken]service_token_roles_required=true | 10:09 |
| SvenKieske | and "service_token_roles_required" true | 10:09 |
| SvenKieske | ah sorry, you wrote that | 10:09 |
| sean-k-mooney | heh no worries | 10:10 |
| SvenKieske | we must list the appropriate roles also with "service_token_roles" it seems | 10:10 |
| sean-k-mooney | so ya that is missing form the patch i wrote to actully do the enforcement. the patch i wrote will send the token but cidner will not check that it has the service role | 10:10 |
| SvenKieske | ah default is "service" should be fine | 10:10 |
| SvenKieske | thank you for the patch anyway! highly appreciated :) | 10:11 |
| sean-k-mooney | yep | 10:11 |
| SvenKieske | and thanks for helping figuring this out | 10:11 |
| sean-k-mooney | well this converstaion also just told me i have more work to do on another installer | 10:11 |
| sean-k-mooney | cause i did hte same fix there. i just added the nova section but i have not 1 ensured that it has the service role or two modifed the deployemnt of cinder et al to actully validate that | 10:12 |
| sean-k-mooney | so this has been useful for me too | 10:12 |
| sean-k-mooney | i can proably quickly update the kolla-ansible patch ot add the cinder bits if you like althoug really it woudl good to do this for all services | 10:12 |
| sean-k-mooney | so that is proably more work then i have time to do | 10:13 |
| sean-k-mooney | actull for nova its proably a good idea to enable the role validation on nova itslef too | 10:13 |
| SvenKieske | I'll check with mnasiadka to fix other services, or make at least a list of work to do, depending on the scope of this | 10:13 |
| sean-k-mooney | SvenKieske: im a littel sad about hte upgrade impact of this change in nova/cinder requiremnts | 10:14 |
| sean-k-mooney | i understand why this was the only solution they could come up with to close all the angeles | 10:14 |
| sean-k-mooney | its just sad tha it will require peopel to update there installer to be able to update the deploy openstack | 10:15 |
| sean-k-mooney | well or for kolla you can use the config override mecahimue to od this without ansibel changes | 10:15 |
| SvenKieske | mhm, well, just thinking: wouldn't it suffice to restart nova/cinder? then they should validate the service roles? of course you need to handover stuff for HA scenarios, but should be possible? | 10:15 |
| SvenKieske | what impact are you talking about? | 10:16 |
| sean-k-mooney | ya its just a config update and restart (after you add the role to nova and the other services.) | 10:16 |
| SvenKieske | yeah sure, you also need to update the installer, I guess | 10:16 |
| SvenKieske | this only tangentially affects myself, as I'm not really using iscsi/FC, as far as I'm aware at least. | 10:17 |
| sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/882847/1/releasenotes/notes/service-user-token-421d067c16257782.yaml | 10:19 |
| sean-k-mooney | so we dont currently have a check to hard stop nova starting | 10:19 |
| sean-k-mooney | but i think on the cinder side you will need the serviec token to do some operations | 10:20 |
| SvenKieske | mhm, thinking further: I would propose it would also be good to switch the defaults in the services: that is: if service_tokens are send "service_token_roles_required=true" should imho be the default | 10:20 |
| SvenKieske | that like not checking https certs for validity, that was a huge fuzz on oss-sec just some days ago, regarding some popular perl library | 10:21 |
| sean-k-mooney | ya it proably shoudl be but until now this cofnig was optional | 10:21 |
| SvenKieske | yeah, I understand that; also the topic of backwards compatibility needs to be thought through, but over 1-2 releases we should be able to change that | 10:21 |
| sean-k-mooney | """For backwards compatibility reasons we must let valid service tokens pass that don’t pass the service_token_roles check as valid. Setting this true will become the default in a future release and should be enabled if possible""" | 10:21 |
| SvenKieske | it's easy to miss, don't want to know how many admins wanted to enforce service tokens but never validated them by accident | 10:22 |
| sean-k-mooney | thats in the docs so it looks like fliping that to true is the plan anyway | 10:22 |
| SvenKieske | ah nice! | 10:22 |
| sean-k-mooney | i assume this will go to true as part of the rbac goal | 10:22 |
| sean-k-mooney | once we get to the point where we can start relying on the service role | 10:23 |
| sean-k-mooney | the orginal usecasue for the service_user cofnig in case you were not familar was so that service A can say (the users token was valid when i recived it) so that operation in service B do not fail due to time outs | 10:24 |
| sean-k-mooney | for any long runnign requests | 10:24 |
| SvenKieske | that would be great; I'll be out for lunch for now; thanks for all the help and information again :) | 10:24 |
| sean-k-mooney | in the future with SRBAC it will also be used for service to service calls to drop admin form the services | 10:24 |
| *** mmalchuk_ is now known as mmalchuk | 11:03 | |
| fungi | SvenKieske: thanks for the excellent point. i'm hoping there's no "next time" though, this was a very unusual circumstance and the first time in 13 years of the project that a vulnerability fix has backported a behavior that required configuration/deployment changes. if there had been *any* other way i would have preferred we didn't take this approach | 11:30 |
| fungi | there is a reason why our policy for security advisories is that just applying patches should suffice | 11:31 |
| fungi | i'm similarly quite worried that ossa-2023-003 is going to cause confusion due to the sheer complexity | 11:32 |
| fungi | it was by far the most "wordy" advisory we've ever issued, and that's after splitting about half of it into a separate ossn-0092 | 11:33 |
| fungi | i had to merge a change to our site generator, because the sphinx extension was written to assume the description would only ever be a single paragraph with no special formatting | 11:33 |
| fungi | as for not checking https certs, we still have a lot of that too in various places, we've always considered it a hardening opportunity because in most cases it's vendor drivers designed to communicate with internal interfaces of things like disk array controllers that operators traditionally just relied on tofu for a self-signed cert the device autogenerated on first boot | 11:37 |
| fungi | at least i know when i was in that position, we never bought real certs for devices which were only reachable on our management network. that's probably less common now that free sources of certs abound, but still it's complicated to get those issued for devices that can't/won't ever be connected to the internet | 11:39 |
| SvenKieske | sure, that stuff only changed very recently with the adoption of let's encrypt and the general policy changes regarding usage of https and correct certs even in internal environments, it's still a work in progress for many systems, but overall a good move in the right direction | 11:44 |
| SvenKieske | and also agreed that this is a very unusual patch/problem; but I wouldn't bet on stuff never occuring again. This might also just be a side effect from people paying more attention to security problems today, at least that is my optimistic take. | 11:51 |
| fungi | yeah, i'm not betting on that, not much of one to gamble after all, but it is a once-so-far in the lifetime of the project | 12:01 |
| sean-k-mooney | fungi: for what its worth just applying the patch will fix most of the CVE edgaces but not all of them | 12:08 |
| sean-k-mooney | SvenKieske: on this topic one of the recomendation was to also modify the multipathd config | 12:08 |
| sean-k-mooney | that is also something that should be handeled in kolla and other installer. i think there are patchs for this for tripleo | 12:09 |
| SvenKieske | ah, well I need to take an hour and read the complete advisory I guess, I still just skimmed it, am in a meeting now though. If I can assist further, just ping me here or in private. | 12:09 |
| sean-k-mooney | but i dont know if a comunication has been done to OSA or or other installers like kolla | 12:09 |
| fungi | sean-k-mooney: i took the opportunity to proactively add the ptls of all official deployment projects to our advance notification list at the end of last month (previously we just added them if they asked), and also pinged them all privately in irc to alert them to the fact that they were going to be receiving copies of the advisory/note and patches | 12:36 |
| sean-k-mooney | fungi: ack that sound like a good improvement in process | 12:37 |
| fungi | so in theory at least the ptls for kolla, osa, openstack-helm, puppet-openstack, openstack-chef got copies about a week ahead | 12:37 |
| sean-k-mooney | fungi: on the nova side we have not implemnted a hard bock on nova starting if the service user is not there | 12:37 |
| fungi | openstack-charms too | 12:37 |
| sean-k-mooney | but i have not looked at the cidner patch to see if they change the requriemetns | 12:38 |
| sean-k-mooney | so i dont know if we will automatically se job failures or not | 12:38 |
| sean-k-mooney | in etiher case its imporant to make sur that this is adress in the installer | 12:38 |
| fungi | and yeah, for tripleo i have no idea what the plan is since the current "maintained" stable branches have no maintainers | 12:38 |
| fungi | but maybe they also have ~0 users | 12:39 |
| sean-k-mooney | there are definely downstream patches for ooo | 12:39 |
| sean-k-mooney | i hope there are upstream versions of the same | 12:39 |
| fungi | right, for the old versions rh is supporting downstream | 12:39 |
| sean-k-mooney | ya so i saw patches for the downswtream wallaby and train equivalnets | 12:39 |
| sean-k-mooney | i hope now the embargo has passed those will be upstream but not sure about master | 12:40 |
| sean-k-mooney | or zed | 12:40 |
| fungi | but the tripleo leadership said wallaby was the most recent red hat still had interest in supporting | 12:40 |
| sean-k-mooney | ya so i expect it to get fixed ther ebut tripleo historicall did not alwasy requrie patches to merge on master first | 12:41 |
| fungi | given that they basically didn't maintain versions after wallaby anyway, i have a feeling whatever they patch in wallaby could be trivially forward-ported to the later branches | 12:41 |
| sean-k-mooney | since they never have follow the stable backport policy | 12:41 |
| sean-k-mooney | for what its worth ooo has been using the service_user since at least train | 12:41 |
| fungi | but also, i'm only concerned about patches in newer tripleo branches if they actually have users, which is doubtful | 12:42 |
| sean-k-mooney | i dont know if the validation for roles was ever enabled | 12:42 |
| sean-k-mooney | or if it adds the servie role to the relevnet acounts | 12:42 |
| sean-k-mooney | os i think that would be the likely gap currently | 12:42 |
| sean-k-mooney | also the multipatd config change i guess | 12:42 |
| sean-k-mooney | fungi: this is where a lot of our installer focus is now https://issues.redhat.com/browse/OSPRH-191 | 12:43 |
| sean-k-mooney | im trying to capture all the changes that are rquired and make sure they get applied to the new golang k8s operator based installer | 12:44 |
| fungi | yeah, i gathered that was the thrust of the replacement | 12:45 |
| fungi | basically make openshift be the new rdo installer | 12:45 |
| sean-k-mooney | effectivly | 12:45 |
| sean-k-mooney | or you can think of it liek make openstack be an app you can install on an openshift cluster like any other applciaiton | 12:46 |
| sean-k-mooney | perhaps just a tad more complex the the typical wordpress example | 12:46 |
| fungi | which is effectively the approach canonical took with charmed openstack too | 12:47 |
| sean-k-mooney | ya using juju as the task orchstrtor | 12:47 |
| sean-k-mooney | instead of openshift | 12:47 |
| fungi | much different implementation obviously, but the overall goal is the same | 12:47 |
| sean-k-mooney | but similar or like what starling x does | 12:47 |
| fungi | sure | 12:47 |
| sean-k-mooney | they use airship and helm to deploy openstack on vanilla k8s | 12:48 |
| fungi | i want to say they replaced their use of airship recently, but maybe that's still in progress | 12:48 |
| sean-k-mooney | the core idea is create an entity that know how to take a declaritive input and make it so in our case a operator writteen in go in thres a charm | 12:48 |
| sean-k-mooney | hopefully i can still peak at there code cause i wanted to see if i could port how they run wsgi servies under grunicorn to devstack and maybe our operator stuff | 12:49 |
| sean-k-mooney | fungi: basically so we can move devstack of uwsgi to something maintaiend and perhapse make it simpler to use in a venv | 12:50 |
| sean-k-mooney | but the fact they used grunicorn ot run all the wsgi services in starlinx means in principal all the openstack service should work with it | 12:51 |
| sean-k-mooney | i kind of wish more or the experiment/inotvations they come up with in starlingx could propagate back ot openstack in general | 12:52 |
| fungi | well, uwsgi is maintained afaik, just not getting further development... or did that change more recently? | 12:52 |
| sean-k-mooney | https://uwsgi-docs.readthedocs.io/en/latest/# | 12:52 |
| sean-k-mooney | https://github.com/unbit/uwsgi/commit/5838086dd4490b8a55ff58fc0bf0f108caa4e079 | 12:54 |
| sean-k-mooney | so it went into maintaince mode in april 2022 | 12:54 |
| sean-k-mooney | so its not dead or anything but a signal has at least been sent that its lifetime is proably limited | 12:55 |
| fungi | maintenance mode sounds maintained to me ;) | 12:56 |
| fungi | anyway, having more options is definitely good, yes | 12:56 |
| sean-k-mooney | yes im just not sure it that include suppprot for new python version | 12:56 |
| fungi | easy enough to compile a fresh cpython 3.12.0a7 and find out | 12:58 |
| fungi | i always build the latest prerelease and have it available in my path for similar reasons | 12:59 |
| sean-k-mooney | i have not looked at teh devstack venv work in a while | 13:01 |
| sean-k-mooney | im hoping that will help here once that is merged | 13:01 |
| sean-k-mooney | its much simpler to this tyep of testing with a venv then system wide install | 13:01 |
| fungi | absolutely | 13:01 |
| sean-k-mooney | evne if its just create a ven and pip install everytin in gloabl requirement and see what explodes | 13:02 |
| sean-k-mooney | although that has much less value then atcully runing the code | 13:02 |
| sean-k-mooney | cool well i beter go fix the go installler chat to ye later o/ | 13:03 |
| fungi | thanks! | 13:03 |
| sean-k-mooney | fungi: by the way this is actully the second cve to require a config change to fix | 13:49 |
| sean-k-mooney | but two in 14 years is still a good record | 13:49 |
| fungi | yeah, though in that case it was a simple toggle on a default i think? not significant additional requirements to how things are deployed | 13:50 |
| sean-k-mooney | to fully fix https://bugs.launchpad.net/os-vif/+bug/1734320 with ml2/ovs https://opendev.org/openstack/os-vif/src/branch/master/vif_plug_ovs/ovs.py#L96 to true | 13:51 |
| fungi | and it wasn't a config change that was mandatory to keep things from breaking on upgrade | 13:51 |
| sean-k-mooney | ya so just adding [os_vif_ovs]isolate_vif=ture | 13:51 |
| fungi | also we didn't do that one as an official security advisory | 13:52 |
| sean-k-mooney | no? it had a cve assocated with it i tought | 13:54 |
| sean-k-mooney | but ya its https://review.opendev.org/c/openstack/cinder/+/882835/2/cinder/volume/api.py#2543 | 13:55 |
| sean-k-mooney | that is diffent here. | 13:55 |
| fungi | anyone can get a cve assigned for anything they like. the openstack vmt didn't request a cve for it, in 2018 we declared it an incomplete fix | 13:55 |
| sean-k-mooney | fair enough i am pretty sure downstream thsi was treated as a cve but i have ejected it form my brain at this point | 13:56 |
| fungi | well, like i said, cve != security advisory | 13:56 |
| fungi | cve is merely a unique tracking id. it doesn't even mean that the project considers what that's tracking to be a vulnerability at all | 13:57 |
| fungi | anyone can ask mitre to assign a cve for anything, with or without the input of the people responsible for the software | 13:57 |
| fungi | there is a process for projects to request that mitre revoke a cve or for marking it as disputed upstream, but we (openstack vmt) don't bother to do that because we don't consider the existence of a cve to necessarily mean there's an actual vulnerability anyway | 13:59 |
| fungi | some people do assume the existence of a cve means there's a vulnerability, but those people are gravely misunderstanding the entire point of mitre's cve system | 14:00 |
| * SvenKieske feels this same topic was just recently discussed on oss-sec as well. | 14:01 | |
| fungi | yes, it comes up there with some regularity | 14:01 |
| SvenKieske | fungi: just a shout out: if I had read your announcement completely - and other people as well - it would have answered all questions regarding the service token stuff. | 14:02 |
| fungi | in the case of bug 1734320 i agree it represents an actual vulnerability, just one we couldn't backport fixes for | 14:02 |
| fungi | a not-on-by-default fix for that bug did eventually get backported to some stable branches, but many years later and for versions that didn't even exist at the time we made the call | 14:03 |
| SvenKieske | I mean the heuristic cve ~ vuln is a pretty decent one. people think in abstractions, after all. I also treat the system and it's workings more like a feature. I can request a cve for a vuln, even if $vendor disagrees. and how many vendors disagree just because they want to avoid bad press and not on factual grounds? sadly still a lot. | 14:03 |
| sean-k-mooney | this is one of the reasons i like to fix things upstream | 14:04 |
| SvenKieske | it's of course a terribly complex system for end users, which they don't understand. | 14:04 |
| fungi | yeah, like i said, as a general rule we don't disagree on cve requests others make for our bugs, we just ask they update the bug with the cve that was assigned so there don't end up being duplicates | 14:04 |
| sean-k-mooney | because even if they disagree you still get the fix evenutlly | 14:04 |
| SvenKieske | sure :) | 14:05 |
| fungi | but by the same token, we don't treat a bug any differently just because someone got a cve assignment issued for it | 14:05 |
| fungi | if we (vmt) decide we'll be issuing an advisory then we ask mitre to assign a cve for it, assuming someone else didn't do that already | 14:06 |
| fungi | in the case of ossa-2023-003 the cve was assigned by red hat, because one of the developers felt compelled to loop rh's security team into the discussion ahead of schedule | 14:08 |
| fungi | but also sometimes the reporter of a bug works for an organization which is a cna and so assigns a cve themselves before they even contact us (we've had some from cisco talos like that, for example) | 14:16 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!