| opendevreview | Takashi Kajinami proposed openstack/os-vif master: Clean up Windows support https://review.opendev.org/c/openstack/os-vif/+/932436 | 00:56 |
|---|---|---|
| opendevreview | Takashi Kajinami proposed openstack/placement master: Switch python version for placement-nova-tox-functional https://review.opendev.org/c/openstack/placement/+/932472 | 02:03 |
| opendevreview | Takashi Kajinami proposed openstack/os-vif master: Drop kuryr-kubernetes job https://review.opendev.org/c/openstack/os-vif/+/932474 | 02:17 |
| opendevreview | Takashi Kajinami proposed openstack/placement master: Switch python version used for periodic jobs https://review.opendev.org/c/openstack/placement/+/932472 | 04:17 |
| opendevreview | melanie witt proposed openstack/nova master: WIP trait, filter, request spec https://review.opendev.org/c/openstack/nova/+/932153 | 04:23 |
| opendevreview | melanie witt proposed openstack/nova master: api: Add microversion 2.97 to add image_type to block_device_mapping_v2 https://review.opendev.org/c/openstack/nova/+/930968 | 06:58 |
| auniyal | sean-k-mooney can you please have another look on these | 09:07 |
| auniyal | https://review.opendev.org/c/openstack/nova/+/929858 | 09:07 |
| auniyal | https://review.opendev.org/c/openstack/nova/+/926521 - melwittis +2 on this already | 09:07 |
| tkajinam | so for some reason placement-nova-tox-functional-py39 uses python 3.8 actually. I wonder if we can migrate that job to noble which uses 3.12 by default. https://review.opendev.org/c/openstack/placement/+/932472 | 10:45 |
| sean-k-mooney | ah is that why that failed | 10:45 |
| tkajinam | though we may want to fix that inconsistency in stable branches (especially 2024.2 ) | 10:45 |
| tkajinam | yeah | 10:45 |
| sean-k-mooney | +2 | 10:46 |
| tkajinam | that's truly wired | 10:46 |
| sean-k-mooney | i have approved the nova and os-vif bumps | 10:46 |
| tkajinam | thanks ! | 10:46 |
| sean-k-mooney | i wonder if its related to the min tox version we are using | 10:47 |
| sean-k-mooney | minversion = 3.18.0 vs minversion = 4.6.0 in nova | 10:48 |
| tkajinam | minversion only determines min and would not decide the version actually used. | 10:49 |
| tkajinam | tox-4.21.2-py3-none-any.whl is what is pulled in the job | 10:49 |
| sean-k-mooney | https://github.com/openstack/placement/commit/1c8afcd3f10a2ace8fdbada4f758b27bb2774cec | 10:49 |
| sean-k-mooney | so when we bumped to 4.0 we were able to drop ignore_basepython_conflict = True | 10:49 |
| sean-k-mooney | on ok | 10:50 |
| sean-k-mooney | ya that makes sense you cant use tox to install tox | 10:50 |
| gibi | sean-k-mooney: could you please check my test results in https://bugs.launchpad.net/nova/+bug/2077070/comments/7 regarding the pci_devices cleanup at compute_node delete. | 11:28 |
| gibi | sean-k-mooney: other than the discover_hosts complication I will dig later, I don't see any real bugs. | 11:29 |
| sean-k-mooney | are you testign a fix or trying to repoduce | 11:29 |
| gibi | I only trying to reproduce | 11:30 |
| sean-k-mooney | so "nova-manage db archive_deleted_rows" shoudl not be requried to be able to crate a new compute node | 11:30 |
| gibi | it is not required | 11:30 |
| sean-k-mooney | ok so the compute node is marked as deleted? | 11:31 |
| sean-k-mooney | when the compute service is deleted? | 11:31 |
| sean-k-mooney | ill read your comment poeprly | 11:31 |
| gibi | yes, the compute_node is marked deleted | 11:32 |
| gibi | the pci_devices rows are not marked deleted but they are pointing to a compute that is marked deleted so they does not interfere | 11:32 |
| gibi | and archive deletes both the compute_node and the pci_devices rows | 11:33 |
| gibi | this might clarify it further https://bugs.launchpad.net/nova/+bug/2077070/comments/9 | 11:34 |
| sean-k-mooney | reading your comemnt ohter then it not workint without --by-service | 11:34 |
| sean-k-mooney | it seams reasonable | 11:34 |
| sean-k-mooney | i susppect the --by-service thing is what was really the issue then | 11:34 |
| sean-k-mooney | tripleo or the end user were problly not using that | 11:34 |
| gibi | yeah, I will dig into that side of the issue as we might fix the discover_hosts to work with undeleted computes | 11:35 |
| sean-k-mooney | im sure you had but just double chekcing you had pci devices in the pci_device table when checking this right | 11:37 |
| opendevreview | Merged openstack/nova master: Libvirt: updates resource provider trait list https://review.opendev.org/c/openstack/nova/+/926521 | 11:39 |
| sean-k-mooney | im wonderign if an instance was using one and you evacuated it will the archive still work | 11:39 |
| sean-k-mooney | or the undelete | 11:39 |
| gibi | yes, I had pci device even allocated :) | 11:40 |
| gibi | I can try the evac case, as I local deleted the instance after the simulated compute failure | 11:41 |
| sean-k-mooney | so in the evacuate case wehre your keepign the identity | 11:42 |
| sean-k-mooney | there is extra logic in the compute agent to properly clean up after the evacuated instnaces | 11:42 |
| sean-k-mooney | so im wondering if that could be relevent here | 11:42 |
| sean-k-mooney | it proably does not hurt to test wtih the removal of the compute uuid file too but im more concerned about the ohter case | 11:43 |
| gibi | OK. I will check the two case with evac today... | 11:47 |
| gibi | thanks for the review on the bug | 11:47 |
| opendevreview | Merged openstack/nova master: Remove Python 3.8 support https://review.opendev.org/c/openstack/nova/+/931047 | 12:57 |
| opendevreview | Merged openstack/os-vif master: Remove Python 3.8 support https://review.opendev.org/c/openstack/os-vif/+/931480 | 12:59 |
| tkajinam | My I ask for 2nd vote for https://review.opendev.org/c/openstack/placement/+/932472 to unblock py38 removal ? | 13:03 |
| tkajinam | (I guess the existing job fails because now nova requires python >= 3.9 | 13:03 |
| gibi | tkajinam: +A, thanks | 13:04 |
| tkajinam | gibi, thank you, too ! | 13:05 |
| gibi | tkajinam: also thanks for the reviews on my igb series. I will quickly respin it to implement you suggestion about value/values | 13:08 |
| tkajinam | gibi, ah, yes. thanks. | 13:13 |
| tkajinam | alternatively we can implement raise_on_too_new_value as a wrapper of raise_on_too_new_values (then we don't have to maintain the same logic twice) but I feel like it's still redundant | 13:15 |
| tkajinam | hopefully we can add typing to avoid wrong usage of raise_on_too_new_values | 13:16 |
| tkajinam | (to prevent someone from using value"s" for a single string | 13:17 |
| rpittau | hi all! I was looking at the PTG schedule for next week and wondering when we could have a joint session between ironic and nova, looks like thursday 24 may be a good day, at 13 UTC, wdyt? | 13:23 |
| gibi | tkajinam: adding typing is not easy as I cannot just require that the passed in value is an iterable as if a single string is passed in it is still an iterable | 13:28 |
| tkajinam | gibi, ah, yes. that's a good point | 13:29 |
| gibi | but unit test should catch these | 13:31 |
| gibi | so I'm not that worried | 13:31 |
| tkajinam | anyway this is rather a global issue and so far the "values" naming should help identifying the expected type | 13:31 |
| tkajinam | yeah | 13:31 |
| tkajinam | rpittau, I think that works. some of the people may be busy on Monday and Wednesday for tc discussion or cross project discussions but may be flexible on Tuesday or Thursday | 13:34 |
| tkajinam | rpittau, though I'd defer the decision to bauzas | 13:34 |
| rpittau | thanks, I chose thursday as for ironic is the less busy day | 13:35 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Refactor obj_make_compatible to reduce complexity https://review.opendev.org/c/openstack/nova/+/928590 | 13:43 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: [ovo]Add igb value to hw_vif_model image property https://review.opendev.org/c/openstack/nova/+/928456 | 13:43 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: [libvirt]Support hw_vif_model = igb https://review.opendev.org/c/openstack/nova/+/928584 | 13:43 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: [doc]Developer doc about PCI and SRIOV testing https://review.opendev.org/c/openstack/nova/+/928834 | 13:43 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Refactor obj_make_compatible to reduce complexity https://review.opendev.org/c/openstack/nova/+/928590 | 13:43 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: [ovo]Add igb value to hw_vif_model image property https://review.opendev.org/c/openstack/nova/+/928456 | 13:43 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: [libvirt]Support hw_vif_model = igb https://review.opendev.org/c/openstack/nova/+/928584 | 13:43 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: [doc]Developer doc about PCI and SRIOV testing https://review.opendev.org/c/openstack/nova/+/928834 | 13:43 |
| gibi | tkajinam: fixed your comment :) ^^ | 13:44 |
| tkajinam | gibi, thanks ! | 13:46 |
| bauzas | rpittau: fair enough, we can do this at Thursday 13OOUTC | 13:56 |
| bauzas | rpittau: adding it now in https://etherpad.opendev.org/p/nova-2025.1-ptg | 13:57 |
| rpittau | bauzas: thanks, I'll update our schedule! | 13:57 |
| bauzas | in our room or yours ? | 13:57 |
| rpittau | we don;t have one booked at that time, maybe better yours | 13:57 |
| bauzas | okay so please join the austin room :) | 13:58 |
| rpittau | we will, thanks :) | 13:58 |
| bauzas | rpittau: added it in https://etherpad.opendev.org/p/nova-2025.1-ptg#L69 | 13:58 |
| rpittau | perfect, I've added it at https://etherpad.opendev.org/p/ironic-ptg-october-2024#L10 | 14:00 |
| admiyo | sean-k-mooney, I see that you closed bug 698696 as won;t fix: https://bugs.launchpad.net/keystone/+bug/968696 I think that is a mistake. All role assignments are scoped. I had a proposal a few years back that addresses your concerns. Basically, for "global" actions that nova has scoped to a proejcct (due to history) you confirm that the project on the token is an "administrative" project. It requires minimal changes | 14:15 |
| admiyo | to Nova. https://adam.younglogic.com/2017/05/fixing-bug-96869/ | 14:15 |
| sean-k-mooney | admiyo: there is no such thing as a project scoped admin | 14:19 |
| sean-k-mooney | admiyo: there is a governance document to descibe how roels shoudl work consitnatnly across all of openstack | 14:20 |
| admiyo | sean-k-mooney, as of when? When Openstack started, that was all there was. It was a bug in the initial design, before Keystone was a project/ | 14:20 |
| sean-k-mooney | the admin role is not scoped to a project based on the desgin agreed in yoga | 14:20 |
| admiyo | And the reason why it has stuck around is that Nova scoped APIs to projects that needed the Aadmin role. It can't be fixed in openstack without being fixed in Nova | 14:20 |
| admiyo | What I wrote up was the "hard" way | 14:21 |
| admiyo | the even harder way was to fix all of the APIs in Nova | 14:21 |
| sean-k-mooney | amorin: have you read https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html | 14:21 |
| sean-k-mooney | amorin: you cant have a proejct scoped admin role in any project | 14:21 |
| sean-k-mooney | its not just nova | 14:21 |
| admiyo | We just killed the Admin role? | 14:21 |
| sean-k-mooney | no | 14:22 |
| sean-k-mooney | the admin role is defiend as a role that allows the bearer to performe admin operations on any service agaisnt any project | 14:22 |
| sean-k-mooney | which is what it has alwasys ment | 14:23 |
| admiyo | "we decided to postpone the scope implementation" | 14:23 |
| admiyo | Heh....scope existed from the get go | 14:23 |
| sean-k-mooney | not in anything other then keystone | 14:23 |
| sean-k-mooney | we not only persponed it we removed it form almost every project that impmented it | 14:23 |
| admiyo | Keystone split off from Nova. It was originally implemented there. The bug is Nova's original sin | 14:24 |
| sean-k-mooney | admiyo: to be clear i was not makeign the deciosn to close it form a nova point of view | 14:24 |
| admiyo | And the problem was that there WERE nova apis that required admin scoped to the project | 14:24 |
| sean-k-mooney | as a comuntiy a collect dission was made not to go in that direction | 14:24 |
| admiyo | so you HAD to split it up somehow. So you can fix it, and maybe it is fixed now | 14:25 |
| sean-k-mooney | admiyo: have you read https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html or partipated in any of the discussion that have been taking place for the last 5 years on this topic | 14:25 |
| admiyo | I got so burnt out by chasing this that I had to step away from Keystone. | 14:26 |
| admiyo | And I have popped in periodically. The bug was still assigned to me for a while. | 14:26 |
| admiyo | And I think it is ok to say that it is fixed if you cannot have a admin scoped project token | 14:27 |
| sean-k-mooney | admiyo: when we explained to operator that they would not be able to do openstack service list --all-tenant with the proposed implemetion they said hell no | 14:27 |
| admiyo | And there was a solution to that | 14:27 |
| admiyo | That was why we had implied roles. | 14:27 |
| sean-k-mooney | we still have implied roels | 14:27 |
| sean-k-mooney | admin impiles member implies reader | 14:28 |
| admiyo | I know. I wrote that implementation and I can see it has been maintained. | 14:28 |
| sean-k-mooney | that does nto solve the problem when a servier is a proejct scoped resoucece | 14:28 |
| admiyo | Yes it does.... | 14:28 |
| sean-k-mooney | well in 5 years noone has been able to expalin how to make it owrk in a way that people can accpet and implemtnet | 14:29 |
| admiyo | You need the appropriate role, but implied roles work down the hierarchy, So if I have the correct role on the parent proejct (or domain) it works for all projects underneath it. | 14:29 |
| sean-k-mooney | sure | 14:29 |
| sean-k-mooney | but we dont supprot domain scoped tokens and you cant have nested projects | 14:30 |
| admiyo | And you could even do it on a domain, but it looks like they have made it work bettern | 14:30 |
| sean-k-mooney | even if we did use the domain scopet token to list all the project you shoudl have acceess too that still means you cant see things across multiple domains | 14:30 |
| admiyo | It used to work. | 14:31 |
| admiyo | I think we called it the default domain.... | 14:31 |
| sean-k-mooney | admiyo: i was just doing clean up of old bugs by the way | 14:31 |
| sean-k-mooney | as far as i can tell what was being asked for in that bug is not compatible with the governance doc | 14:32 |
| sean-k-mooney | admiyo: gmann: has been following srbac more closely then i | 14:32 |
| sean-k-mooney | admiyo: but the only work i am aware of in this are is what descibe in that doc | 14:32 |
| admiyo | It sounds like you have a fix in place. I think that is better to write up there than "won't fix." This was a deal breaker for giving people direct access to the API in a lot of cases, and lead to everyone putting APIs in front of OpenStack. I would love to see it closed properly. | 14:33 |
| opendevreview | Merged openstack/placement master: Switch python version used for periodic jobs https://review.opendev.org/c/openstack/placement/+/932472 | 14:33 |
| sean-k-mooney | admiyo: i really dont get the issue with acess to the apis | 14:33 |
| sean-k-mooney | there are verly few things that actully need admin in nova | 14:33 |
| admiyo | And do those HAVE to be on project tokens? | 14:34 |
| sean-k-mooney | specificly creating flavors, managing compute servcices/host aggreates or things like live migration | 14:34 |
| admiyo | I'll explein the issue after, but I am more interested in the solution, sounds like you have one | 14:34 |
| sean-k-mooney | admiyo: yes if you want to create a service via nova you need to use a project scoped token | 14:35 |
| admiyo | Right, and the resources you list there are not scoped to projects anymore? | 14:35 |
| sean-k-mooney | they were system scoped and now they are project scoped again | 14:35 |
| admiyo | but why are the "admin" then if admin is global role? | 14:36 |
| sean-k-mooney | becuase flavors/hostaggated compute service are not somethign a normal user shoudl be able to modify | 14:36 |
| sean-k-mooney | they are admin because only admins of the openstack cloud should be able to interact with them | 14:36 |
| admiyo | but should a project admin be able to change flavors that are specific to a different project? | 14:37 |
| sean-k-mooney | no | 14:37 |
| sean-k-mooney | you are misussing that term based on the current agreed usage | 14:38 |
| sean-k-mooney | a a use with an admin role is a global admin | 14:38 |
| admiyo | can I have a project specific flavor ? | 14:38 |
| sean-k-mooney | technially yes private flavor are a thing and only admins can create them on behayce fo end users | 14:39 |
| sean-k-mooney | a normal user of a public cloud should not be allowed ot od that with the member role | 14:39 |
| sean-k-mooney | admiyo: a new role that is seperat eform the curent heriacy was proposed if to have project scoped elevated access | 14:40 |
| sean-k-mooney | https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html#implement-support-for-project-manager-personas | 14:40 |
| dansmith | flavors are also a lot more involved than they were in 2010 and require knowledge of the backend to some degree to set up in a lot of real-world cases, like knowing what hardware is available and in what configuration | 14:40 |
| sean-k-mooney | its called the project manager role | 14:40 |
| sean-k-mooney | that is part of phase 3 and we are in the midel of pahse 2 right now | 14:40 |
| dansmith | and as sean-k-mooney has been saying, I think this is not nova-specific and also pretty much a sailed ship at this point | 14:41 |
| admiyo | So it looks like you DO have a plan in place to fix the bug. It should not be closed as won't-fix | 14:41 |
| sean-k-mooney | the bug is honestly invlid | 14:41 |
| sean-k-mooney | we have specificly decied to not scope the admin role | 14:42 |
| dansmith | agree | 14:42 |
| admiyo | Well, no. It is not invalid. | 14:42 |
| sean-k-mooney | we are keeping it as a global role and providing a new manager role | 14:42 |
| sean-k-mooney | the manager role can be scoped to project or domains | 14:42 |
| sean-k-mooney | when ever we implemet it | 14:42 |
| admiyo | The admin role has been removed from being a project scoped role, that is fine. But do those APIs still look for the project ID on the token? That was the problem before. If nova has removed that restriction, then you have fixed it. | 14:43 |
| sean-k-mooney | thats a diffent question. | 14:43 |
| admiyo | Becuase other projects beside Nova "reqquired" if for day to day operations. It was a logicial contradiction | 14:43 |
| sean-k-mooney | you realise that cinder or glance until recently reuiqred a proejct in the url | 14:44 |
| sean-k-mooney | not just in the headers | 14:44 |
| admiyo | Oh yes I do | 14:44 |
| admiyo | that actually made certain things easier to do, so I understand why they did it. I am still not certain they were wrong, but having it consistent across all OpenStack was more important | 14:45 |
| sean-k-mooney | to answer your question, weither or not we actuly need a project id depends on a few things, the policy file/default and if we are suign it for auiting | 14:45 |
| ratailor | sean-k-mooney, gibi could you please review this patch https://review.opendev.org/c/openstack/nova/+/928933 and let me know why api-samples tests are failing, I assume something is missing, but not sure what ? I tried my best to find why its failing, but couldn't get any further. | 14:46 |
| sean-k-mooney | if we look at the policy for managing flavor https://github.com/openstack/nova/blob/master/nova/policies/flavor_manage.py#L28 we use the base admin check https://github.com/openstack/nova/blob/master/nova/policies/base.py#L40 which does not check for the project id | 14:47 |
| sean-k-mooney | admiyo: however much of our logging does expect a project id | 14:47 |
| admiyo | policy file overrides are a fix, but not required, so I would say that case if OK. The real issue is the defaults. If the code requires a both a project ID and the Admin role, you have a problem. | 14:47 |
| sean-k-mooney | admiyo: for context we moved them to be system scoped and undit that https://github.com/openstack/nova/commit/066e1e69d1394839a9f0bde4ca8c3a0db2d52396 | 14:48 |
| admiyo | So how do you limit who can get the ADMIN role? | 14:48 |
| sean-k-mooney | you do that via keystone | 14:48 |
| sean-k-mooney | a keyston admin is needed to asing an admin role to a user on a project/domian | 14:49 |
| admiyo | From my perspective, system scoped was a mistake. I understand why they did it (to make things more obvious) but it was fighting all of the services. | 14:49 |
| sean-k-mooney | right which is why we went back to everyting in nova being project scoped again | 14:49 |
| sean-k-mooney | admiyo: if you have a devstack you can proably jsut create a non project scoped token iwth the admin role and call nova to see what the behaivor is | 14:50 |
| admiyo | What I had proposed was that we mark certain projects as ADMIN projects. Thus, if you wanted to be selective about what ADMINs could perform, say, a flavor modification, you put an additional check on those operations. But they still used project scoped tokens | 14:50 |
| sean-k-mooney | admiyo: that not a nova dicussion | 14:51 |
| admiyo | It waws. And it was accepted way-back-why byt the Nova PTL.... | 14:51 |
| sean-k-mooney | if you give someone admin they have admin when you call nova or neutron | 14:51 |
| sean-k-mooney | are you usign the term project to refer to openstack serivce or keystone pojects | 14:52 |
| admiyo | No, I was specifically using it for projects. Because the APIs all required project scoped tokens | 14:52 |
| admiyo | For keystone projects | 14:53 |
| dansmith | admiyo: you seem to be missing a lot of context and community-wide discussion that happened in recent years... | 14:53 |
| admiyo | and, I agreee, the name project was not the right one. | 14:53 |
| dansmith | rehashing it with "it was agreed by the nova PTL in 2012" is not a very compelling argument in 2024 | 14:53 |
| admiyo | it was in 2018, right before the Keystone PTL changed the direction and I rage quit. But the issue still stands that if it is broken in Nova it is broken everywhere | 14:54 |
| admiyo | and the issue was that Admin was required on a project scoped token. | 14:54 |
| sean-k-mooney | admiyo: for what its worth there was a discussion abou thaving a maping of roles ot service endpoints | 14:54 |
| admiyo | And elsewhere in openstack, that was required to do things that you would assume an end user could do | 14:54 |
| sean-k-mooney | admiyo: i.e. give someone admin on the neutron api but not on others. keystone did not want to do that | 14:55 |
| admiyo | to be clear. I was never in support of service scoped tokens. I just stepped away from the discussion after a year of my life went in to gettin a nova acceptable solution....so all I am trying to do now is confirm that you do actually have a solution, and I am...55% convinced of that right now | 14:55 |
| sean-k-mooney | admiyo: do you have an exampel of a call to noava that required admin on a diffent service | 14:56 |
| admiyo | Back in the day it was stuff like add/remove userse from projects and grant role assignments. | 14:57 |
| sean-k-mooney | which are all part of keystone | 14:57 |
| admiyo | so in keystone to assign a user to a project, you HAD to have admin, becuase one of the roles you could grant them was Admin | 14:57 |
| admiyo | and thus..... | 14:57 |
| sean-k-mooney | right so the manager role is proposed to be used to do that | 14:58 |
| sean-k-mooney | a porject manager would be able ot add a user to a project | 14:58 |
| admiyo | TO be honest, I have not tracked what was done in Cinder or Glance ever since then. | 14:58 |
| sean-k-mooney | a domain manager would be able to create proejcts and users in there domain | 14:58 |
| admiyo | I know that kn heat you used to need it to clean up things that were wedged.... | 14:59 |
| sean-k-mooney | again that just soudn like a bug | 14:59 |
| admiyo | What you are saying is that a requirement in Nova dictated how all the other projects had to implement policy. | 15:00 |
| sean-k-mooney | admiyo: no im not | 15:00 |
| admiyo | And you then needed to audit and get agreement from all the other projects on how to fix it. | 15:00 |
| sean-k-mooney | again please read that document | 15:01 |
| admiyo | Yeah, you are. Because nova both required admin and that it be on a project scoped token | 15:01 |
| sean-k-mooney | you are mising a lot of contenxt and missrepreenting what i have said | 15:01 |
| admiyo | I have read it. And I get it, and you did get that agree ment, but I was not able to do that back in 2017 | 15:01 |
| sean-k-mooney | to be clear neutron clinder glance and most other service had simeilr objects when they spoke to operatores about there needs | 15:02 |
| admiyo | You might notice that glance, cinder and neutron all made efforts to fix the bug | 15:02 |
| sean-k-mooney | admiyo: your framing the histroy as if nova was the one that pushed for this change when it was our user that push back | 15:02 |
| sean-k-mooney | admiyo: so did we and then we reverted it based on teh compuntiy change of direction | 15:03 |
| admiyo | Now go an look at my proposal again, and realize that it still addresses the operator concern, maintains backwards compat, and fixes the bug. | 15:03 |
| admiyo | But all that is water under the bridge. It sounds like you do have a fix. | 15:04 |
| admiyo | If that is the case, then don't mark the bug "won't fix." because that bug is the reason why end users did not get access to openstack APIs | 15:04 |
| admiyo | the operators could not risk it, and they wrapped it with horrible things like CLoudForms and made the APIs operator only. | 15:05 |
| opendevreview | Amit Uniyal proposed openstack/nova stable/2024.2: Reproducer test for image property hw_architecture https://review.opendev.org/c/openstack/nova/+/932521 | 15:05 |
| opendevreview | Amit Uniyal proposed openstack/nova stable/2024.2: Libvirt: updates resource provider trait list https://review.opendev.org/c/openstack/nova/+/932522 | 15:05 |
| admiyo | But it was not a wishlist item, and it should not be marked as won't fix | 15:05 |
| admiyo | And by the way, we should never have had domains either. And we should have kept the term tennants instead of calling them projects. Mistakes of the past.... | 15:07 |
| sean-k-mooney | admiyo: i can change it to invalid if you would liek. but what i was litrually trying ot convay by updating the bug was no one is working on changing the scope fo the admin role and its not inline wiht the projects direction | 15:08 |
| sean-k-mooney | and by project i litraly mean openstack | 15:08 |
| sean-k-mooney | admiyo: if you want to change the direciton you need to do that at the openstack level and change the grovernace resolution | 15:09 |
| admiyo | Can you state that none of Novas APIs require Admin in order to perform operations that an end user needs to be able to do? To include project self maintainence like remove some resource where the owner has quit | 15:10 |
| sean-k-mooney | as far as i am aware that has been true for at least 10 years | 15:10 |
| sean-k-mooney | maybe more | 15:10 |
| admiyo | That was not the case when we opened the bug, and it was a critical security issue. | 15:10 |
| admiyo | Then you fixed it | 15:11 |
| sean-k-mooney | in nova all end user creatable resocues are owned by teh project not the user with the sole excption of the ssh keypair | 15:11 |
| admiyo | Yeah, let us avoid that rabbit hole | 15:12 |
| sean-k-mooney | the reason is say at leat 10 years is i have worked on openstack since havana and i belive it was true for nova even then | 15:12 |
| admiyo | I think that the issue I remember was flavors. | 15:12 |
| sean-k-mooney | flavor s shoudl nto eb managebale by end users | 15:13 |
| admiyo | And images in glance was an issue that they addressed. | 15:13 |
| admiyo | So you can see, it was an Openstack wide issue, and you DID address it. And it certainly affected Nova. | 15:14 |
| admiyo | BTW, the original implementation was done by termie, the solution of having admin projects was also suggested by him. I was justtrying to keep things consistent with how the main projects viewed things. | 15:15 |
| opendevreview | Amit Uniyal proposed openstack/nova stable/2024.1: Reproducer test for image property hw_architecture https://review.opendev.org/c/openstack/nova/+/932531 | 16:11 |
| opendevreview | Amit Uniyal proposed openstack/nova stable/2024.1: Libvirt: updates resource provider trait list https://review.opendev.org/c/openstack/nova/+/932532 | 16:11 |
| opendevreview | Amit Uniyal proposed openstack/nova stable/2023.2: Reproducer test for image property hw_architecture https://review.opendev.org/c/openstack/nova/+/932533 | 16:11 |
| opendevreview | Amit Uniyal proposed openstack/nova stable/2023.2: Libvirt: updates resource provider trait list https://review.opendev.org/c/openstack/nova/+/932534 | 16:11 |
| opendevreview | Amit Uniyal proposed openstack/nova stable/2023.1: Reproducer test for image property hw_architecture https://review.opendev.org/c/openstack/nova/+/932536 | 16:13 |
| opendevreview | Amit Uniyal proposed openstack/nova stable/2023.1: Libvirt: updates resource provider trait list https://review.opendev.org/c/openstack/nova/+/932537 | 16:13 |
| opendevreview | Merged openstack/placement master: Remove Python 3.8 support https://review.opendev.org/c/openstack/placement/+/925648 | 17:46 |
| gmann | admiyo: we have not touched the admin role due to not breaking the users. and like sean-k-mooney mentioned project manager is something (going to be implemented in Nova soon) which can serve as higher authority than member/reader role within project | 18:44 |
| opendevreview | Merged openstack/nova master: VMware: updates resource provider trait list https://review.opendev.org/c/openstack/nova/+/926522 | 19:04 |
| admiyo | gmann, you could start by acking that there is a problem and not marking the bug as a woishlist item tagged will not fix. It is a real problem and requires a united effort across openstack to address. I can appreciate that the systems roles approach doesn't work for people. I had stepped back when the Keystone leadership pivoted to that approach instead of "is_admin_project" and now seeing that the other approach fell | 20:30 |
| admiyo | apart, I am trying to make sense of the landscape. | 20:30 |
| admiyo | If I understand correctly, you have taken the approach of "make it impossible to assign the role admin except to system wide admin users" which is a feature that was not in Keystone when I left the project, and which would have broken other projects at the time. I think it is still not correct, but it might be good-enough. | 20:32 |
| sean-k-mooney[m] | admiyo: the pivort away form that happend many many years before the change in direction. the idea of a specal adming project was not how openstack worked before we started supporting python 3 | 20:37 |
| sean-k-mooney[m] | admiyo: ill also note that you are taking a very agressive approach there in blaming people for a descision taht was made collectivily over the course of many years | 20:38 |
| admiyo | sean-k-mooney[m], let me take a look at the Nova API policy enforcement to make sure I understand the approach. | 20:39 |
| sean-k-mooney[m] | while i dan and gman all took part in the discusion the direction was not set by the nova comunity | 20:39 |
| admiyo | sean-k-mooney[m], I am aggressive on it, because it has been met with ignorance for a long time, and I put a lot of effort in to it. And because it was one of the reasons that the project was long term hurt in adoption. But I am going to do my due diligence now to see if I understand the state of it. | 20:40 |
| admiyo | I did talk with the Keystone devs and there are aspects to this solution that they did not understand, and so I don't trust that the solution is in place yet. | 20:40 |
| gmann | I agree that we might have solved all the cases for access control but seeing the back and forth approach for so many years, we are trying to mainly solve 1. project personas (mainly ready role and new one called project manager )2. better service to service interaction | 20:44 |
| gmann | there is idea of global reader also but its spec not yet up | 20:45 |
| gmann | and i think doing these in bit and pieces taking mostly asked personas as first step is the feasible solution otherwise we could have stuck on it for another 5 years | 20:46 |
| sean-k-mooney[m] | most of the intiall work was completed in https://specs.openstack.org/openstack/nova-specs/specs/train/approved/policy-default-refresh.html in 2019 | 20:48 |
| sean-k-mooney[m] | we then had to rip out and redo a subset of that | 20:48 |
| gmann | yeah | 20:48 |
| sean-k-mooney[m] | mainly the systemscopted parts | 20:48 |
| sean-k-mooney[m] | and since hten we have been focusing on the personas that operators were actully asking for | 20:48 |
| sean-k-mooney[m] | the reader roles was the highest priorty | 20:49 |
| sean-k-mooney[m] | then the servicer role gained impoartnace for CVE reasons and now we are working on the manager role | 20:49 |
| gmann | exactly. for system scope, I will say we tried to sell this new feature to operators when they did not require it :) | 20:49 |
| sean-k-mooney[m] | for that soped hiher privaldged user that is not global admin | 20:49 |
| gmann | everyone mainly asked for reader role and we tried to give them system scoped concept also which made them confused and break | 20:50 |
| sean-k-mooney[m] | but even before the work in train you could have had custom policy it just was not in tree so this has evovled alot over the last 10 years | 20:50 |
| gmann | true, main idea is to make our default as close as what operator override to | 20:51 |
| gmann | we cannot match each and every cases but match with the mostly used one | 20:51 |
| sean-k-mooney[m] | which is always a blance between public cloud usecases and private cloud | 20:52 |
| admiyo | You have put a lot of work in to this. | 20:52 |
| sean-k-mooney[m] | gmann has alot more then i as they have also been invovled in the tempest testing of this not just enabling it in one porject | 20:53 |
| admiyo | It is 7 years since I stepped away from the project, and you are still working on it. | 20:53 |
| sean-k-mooney[m] | yep we littally impemetned project scopes released it and operators said no | 20:54 |
| gmann | :) progress is slow I agree but If I count we have ~12 projects implemented the project personas reader role which is good | 20:54 |
| sean-k-mooney[m] | so we had to rip it out as we had no path to enable it by defualt | 20:54 |
| admiyo | And they implemented system scopes based on operator feedback. | 20:54 |
| gmann | this cycle, I am going to make more progress on remaining projects and manager role in core services | 20:54 |
| admiyo | It used to be possible to generate the default policy file from the code. I am assuming that is still the case, and that I need a proper tox set up to do so. As I recall, that was tox genpolicy? | 20:56 |
| sean-k-mooney[m] | i belive nova,neutron,placement, ironic and maybe glance and cidner + obviously keystoen had system scope | 20:56 |
| sean-k-mooney[m] | now it has been removed form all but keystone and ironic i think | 20:56 |
| gmann | yes, genpolicy tox env | 20:56 |
| sean-k-mooney[m] | that is still possible yes althoguh you dont need a policy file anymore | 20:56 |
| sean-k-mooney[m] | you only need it if you want to change it | 20:57 |
| admiyo | It just allows me to see what the actual policy enforcement is, though | 20:57 |
| admiyo | as opposed to crawling through all the apis. | 20:57 |
| sean-k-mooney[m] | we now also use yaml instead of json for reasons | 20:57 |
| admiyo | I think comments were the primary driver | 20:57 |
| gmann | yeah | 20:57 |
| * gmann going for lunch | 20:58 | |
| *** gmann is now known as gmann_afk | 20:58 | |
| sean-k-mooney[m] | if you do want to look at the code its all in https://github.com/openstack/nova/tree/master/nova/policies | 20:58 |
| sean-k-mooney[m] | and this is our doc on this topic https://docs.openstack.org/nova/latest/configuration/policy-concepts.html | 21:00 |
| sean-k-mooney[m] | i think we had a rednered policy file soemwhere in the docs but you can also gernerated it locally | 21:01 |
| sean-k-mooney[m] | ah its here https://docs.openstack.org/nova/latest/configuration/policy-concepts.html | 21:01 |
| admiyo | I was, but the logic aaaround deprecations and such made it hard for me to determine exactly what was meant by ADMIN. So I figure that the policy generator is going to evaluate it down to how it is actuall implemented. | 21:01 |
| sean-k-mooney[m] | admin is now just does the token have the admin role | 21:02 |
| admiyo | yeah, I think it is still broken. Hold on. | 21:02 |
| sean-k-mooney[m] | https://github.com/openstack/nova/blob/ed2bf3699ddc360fb646b895560fa9dbcfd6beba/nova/policies/base.py#L40 | 21:03 |
| sean-k-mooney[m] | the rule is defiend here https://github.com/openstack/nova/blob/ed2bf3699ddc360fb646b895560fa9dbcfd6beba/nova/policies/base.py#L40 | 21:03 |
| admiyo | you can't have it both ways. either ADMIN is scoped to the project, or it is global. If there are cases where it needs to be scoped to a project, you can never assign that role to anything other than a system power user, and thus you have an un-delegatable operation. | 21:05 |
| sean-k-mooney[m] | admin is global the resouces in the api are porject scoped | 21:05 |
| sean-k-mooney[m] | yep | 21:06 |
| sean-k-mooney[m] | it can only be asgined to the sytem power user i.e. the root of openstack | 21:06 |
| sean-k-mooney[m] | that is however that role is defiend now | 21:06 |
| admiyo | And that is OK, so long as you don't also need the other. So anything which is rule:context_is_admin" is essentially something reserved for system administators | 21:07 |
| admiyo | for example (first one I saw) #"os_compute_api:os-admin-actions:reset_state": "rule:context_is_admin" | 21:08 |
| admiyo | so the only people that can perform a reset_state operation on a server is a system administrator, right? | 21:08 |
| admiyo | #"os_compute_api:os-aggregates:set_metadata": "rule:context_is_admin" | 21:09 |
| admiyo | and so on? | 21:09 |
| sean-k-mooney[m] | yes reset state is very dangourse | 21:10 |
| admiyo | #"os_compute_api:os-assisted-volume-snapshots:delete": "rule:context_is_admin" | 21:10 |
| sean-k-mooney[m] | it basically bypassing all oru saftey checks and has a high probability of requireing db surgery | 21:10 |
| sean-k-mooney[m] | so reset state is not someing that and end user should every be able to do | 21:10 |
| sean-k-mooney[m] | the assisted volume snapshot api | 21:11 |
| sean-k-mooney[m] | is not ment to be used for humans | 21:11 |
| sean-k-mooney[m] | its reseved exclustivly for cinder | 21:11 |
| sean-k-mooney[m] | in the future admin will not be able to call it and it will require the service role instead | 21:11 |
| sean-k-mooney[m] | aggerate metadata is also systems scoped | 21:12 |
| sean-k-mooney[m] | so all of the above are examples of things only global admisn should be able to do | 21:12 |
| sean-k-mooney[m] | aggreate metadata is basically to allow openstack admins to map capablity to speicifc hosts | 21:12 |
| sean-k-mooney[m] | i.e. to reserve hyperviors for specific tenants | 21:13 |
| admiyo | OK I think I see how things are going. | 21:13 |
| sean-k-mooney[m] | or partions kvm hosts for hyperv ectra | 21:13 |
| sean-k-mooney[m] | service can escualte on behalf of users in some cases where appriate. i.e. if you do a cinder snapshot of a volume as a normal user cinder willl call novas assisted snapshot api as an admin if and only if the volume is attached to a nova instace | 21:14 |
| admiyo | sean-k-mooney[m], understand that in order to implement this, Keystone had to break its users. | 21:14 |
| sean-k-mooney[m] | where that is required we are trying to move those endpoint to the service role | 21:15 |
| admiyo | I still think this is fragile, but I can see where it is sufficient | 21:15 |
| sean-k-mooney[m] | as far as i am ware the usage of admin i am descibeing has been there since 2013 | 21:15 |
| sean-k-mooney[m] | i think longer but i started working on quantom in 2013 at the end of havana | 21:16 |
| admiyo | Yes, it predated essex. It was designed broken | 21:16 |
| sean-k-mooney[m] | so we now have 12 years of oeprators expecting this global admin behvior | 21:17 |
| admiyo | Yep. But it also had serious side effects | 21:17 |
| sean-k-mooney[m] | yep both good and bad | 21:18 |
| admiyo | don't think of the people that use the system. Think of the people that decided not to because they could not expose their APIs to their end users | 21:18 |
| admiyo | I think it seriously affected adoption. | 21:18 |
| sean-k-mooney[m] | what i dont get is you could alwasy use custom policy | 21:18 |
| admiyo | It is the same problem as having to do everything as root in a linux system. Fine if you are the only one using it.... | 21:18 |
| admiyo | And I did serious education on using custom policy at the time. | 21:19 |
| sean-k-mooney[m] | so downstream we impemented the reader roled using custom policy in wallaby | 21:19 |
| admiyo | But it was hard to make custom policy that worked across a whole deployment consistently | 21:19 |
| admiyo | The defaults varied from project to project | 21:19 |
| sean-k-mooney[m] | right so since the reader role was not aviable in wallaby in every proejct we litally wrote custome policy for every service to make it consitnt | 21:20 |
| admiyo | https://www.youtube.com/watch?v=5G-LYCfABJY | 21:20 |
| admiyo | I am so sorry | 21:20 |
| admiyo | It was made harder when everyone decided to move policy in to code, and hard-code the defaults. Before, when everyone had to use a policy file, it was simpler to say "replace yours with this one" | 21:22 |
| admiyo | And when Red Hat went with TripleO and everything had to be generated from a Puppet manifest, I made an honest effort, and then gave up | 21:22 |
| admiyo | I really felt like people wanted to keep it broken. | 21:22 |
| admiyo | #"os_compute_api:os-migrations:index": "rule:context_is_admin" | 21:23 |
| sean-k-mooney[m] | well redhat also offcialy does not supprot custom policy | 21:23 |
| admiyo | Well admiyo officially does not support Red Hat, and now you know why | 21:23 |
| sean-k-mooney[m] | mainly because custoemr have litrally currpted nova’s db before by messing up neutrons policy file | 21:23 |
| sean-k-mooney[m] | such that all instance lost all ports | 21:23 |
| JayF | sean-k-mooney[m]: I'm waiting for the first bug report w/ironic enabling RBAC defaults, along those lines | 21:23 |
| admiyo | Yeah, as I recall there were some interesting policy enforcement in Neutron.... | 21:24 |
| sean-k-mooney[m] | it was actully a tirdparty vendor who i wont name who provide a core plugin with custom policy | 21:24 |
| admiyo | So migrations should never be done by an end user....I think I get that. They don't know about physical hosts....but that does seem like a draconian setup. Seems like there could be cases where you want to let users trigger their own migrations. But I get it. | 21:25 |
| sean-k-mooney[m] | we dont today because its a potinal ddos vector | 21:26 |
| sean-k-mooney[m] | but in the future we do plan to allow that via the manager role | 21:26 |
| sean-k-mooney[m] | so member would not be able to but if you cant some one manager in a project they would be able to trigger a live migation | 21:27 |
| sean-k-mooney[m] | we are also considring if we could allow them to select a host but thats tbd | 21:27 |
| sean-k-mooney[m] | admiyo: ond of our custoemer does nto allow normal admins to live migrate by the way. they have a super admin role for that | 21:28 |
| admiyo | OK. Remember, you also have implied roles, so admin can imply a lower level role and that gives them the ability to move forward. | 21:28 |
| admiyo | so superadmin implies admin | 21:28 |
| sean-k-mooney[m] | basically they had one team that did all the openstack admin and another team that did all the hardware/software mantahcne and patching | 21:28 |
| sean-k-mooney[m] | they actully impmented it by making live migration require a dedicatd live_migration role and used a custom policy overried just for that | 21:29 |
| admiyo | And by the way, according to Keystone, projects can still be nested, so if you assign an inheritable role on a parent or domain, you get it for the full hierarchy, but that means that administrators need to get a token specifically scoped to the project they want to affect | 21:30 |
| admiyo | and I can see how that would be annoying in Horizon.... | 21:30 |
| sean-k-mooney[m] | so i dont remember the details but with unifed limists there have been other changes to how nesting works | 21:31 |
| admiyo | heh...yeah. Another lost battle. | 21:32 |
| admiyo | Did you end up going with the two level approach? | 21:32 |
| sean-k-mooney[m] | yes | 21:32 |
| sean-k-mooney[m] | although its not on by default yet | 21:32 |
| sean-k-mooney[m] | but we have supprot innova | 21:32 |
| sean-k-mooney[m] | glance now only uses unifed limits | 21:33 |
| admiyo | even if a quota is subdivided, you still need to check the overall quota of the project and all the parent projects. | 21:33 |
| sean-k-mooney[m] | we are trying to swap to it by defaut in the next release or two | 21:33 |
| admiyo | there is no real benefit or reason to limit it to two levels. | 21:33 |
| sean-k-mooney[m] | so i dont think any of the service really supprot nested projects | 21:33 |
| sean-k-mooney[m] | at least we dont have special logic to walk the tree | 21:34 |
| admiyo | But for access control, especially admin functionality, you can make use of it. Remember, a domain IS a project. And even domain have a root, the defult project, so you can make roles that roll-on-down-the-hill\ | 21:34 |
| sean-k-mooney[m] | so depending on how much of that is common in the keyston auth middleware ectra your milage will vary | 21:35 |
| sean-k-mooney[m] | you can but you need to use a project scoped token becaue nova does not know about domains | 21:35 |
| sean-k-mooney[m] | nor does neutron or glance ectra | 21:36 |
| sean-k-mooney[m] | heat might | 21:36 |
| admiyo | yeah, you can use a project sccoped token | 21:38 |
| admiyo | but they don't know about nesting | 21:39 |
| admiyo | so you need a token scoped to the project | 21:39 |
| sean-k-mooney[m] | yep | 21:39 |
| admiyo | but you automatically have access to that if you have the role assignment on the domain-as-aproject | 21:39 |
| admiyo | and yes, domains were a mistake | 21:39 |
| admiyo | OK, let me look at the keystone role assignemnt api. that is the real question | 21:40 |
| sean-k-mooney[m] | well domains are kind for for the VPC/reseller usecse | 21:40 |
| sean-k-mooney[m] | but that is not very common | 21:40 |
| admiyo | NO, domains should have just been nested proejcts (tennants) on the proejct side, and IdPs on the user side | 21:40 |
| admiyo | domains as groupings of projects was a mistake | 21:41 |
| *** gmann_afk is now known as gmann_ | 21:41 | |
| sean-k-mooney[m] | probably | 21:41 |
| *** gmann_ is now known as gmann | 21:41 | |
| admiyo | but they predated Federated Identity. But IdP is the logical grouping for users, not domains | 21:41 |
| sean-k-mooney[m] | im not really aware of domains being used by any of our custoemrs | 21:41 |
| sean-k-mooney[m] | it would be a good question for the openstack user survay | 21:41 |
| admiyo | this is all 20.20 hindsight | 21:41 |
| admiyo | Although I do remember rendering insufficient resistenace to domains, thinking they should be projects. And I do rmeember being to person reminding people they were projects, not tennants. So I am very much to blame | 21:42 |
| sean-k-mooney[m] | there is one feature that i have always wanted in keystone since your here. i wish there was a way to get a token form an sshkey like i want to just use the openstack client but assocating a public key with my user and using my private key to sign in | 21:44 |
| sean-k-mooney[m] | the other feature is more a nova one but people have asked use to provide a token to vms liek k8s does | 21:44 |
| sean-k-mooney[m] | i.e have a way via the metadata api to provide a project scoped token or proably applciation credital to the vm | 21:45 |
| admiyo | And I think I just spotted how things are broken. I am working backwards here. But there is no specific check in the keystone role assignment API that does anything like say "you can only assign a role that you yourself have" which was one of the reasons for implied roles. And to do a role assignment for a project I need the role": | 21:45 |
| admiyo | "identity:create_grant": "(rule:admin_required) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id) | 21:46 |
| admiyo | s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" | 21:46 |
| sean-k-mooney[m] | usually people ask for that in JWT format | 21:46 |
| admiyo | Ugh...really. | 21:46 |
| admiyo | OK, I think I see what they are doing there. | 21:46 |
| admiyo | And I think it is broken by that first or | 21:46 |
| sean-k-mooney[m] | there is some form of delegation check for applciation credentials | 21:46 |
| admiyo | yeah, but it is skipped | 21:47 |
| sean-k-mooney[m] | it might be enforced outside the policy layer | 21:47 |
| admiyo | (rule:admin_required) means that anyone with an admin role can execute that policy | 21:47 |
| admiyo | none of the others will be checked | 21:47 |
| sean-k-mooney[m] | right | 21:48 |
| admiyo | #"admin_required": "role:admin or is_admin:1 | 21:48 |
| sean-k-mooney[m] | but thats in line with how admin works in all other proejcts | 21:48 |
| sean-k-mooney[m] | * all other openstack services | 21:48 |
| sean-k-mooney[m] | admiyo: i will note that keystones docs still document what you want | 21:49 |
| sean-k-mooney[m] | but i dont think that has been true for a very very long time | 21:49 |
| sean-k-mooney[m] | admiyo: https://docs.openstack.org/keystone/latest/admin/service-api-protection.html#admin | 21:49 |
| admiyo | I just generated the policy file from the code | 21:49 |
| sean-k-mooney[m] | that is not consistent with the standard definition of admin | 21:50 |
| sean-k-mooney[m] | per the SRBAC comunity goal | 21:50 |
| sean-k-mooney[m] | the keystone doc imples it is scoped but that has never been the case as far as i was aware | 21:50 |
| sean-k-mooney[m] | so i think it documented the intent you had in 2015 | 21:51 |
| admiyo | We are going to have to break the operators one way or another to fix this | 21:51 |
| sean-k-mooney[m] | but not what was implemnted | 21:51 |
| admiyo | EIther in their keystone workflows or in their Nova | 21:51 |
| sean-k-mooney[m] | perhaps but i dont think any deployemnt cna be using the workflow you are descibing | 21:52 |
| admiyo | I think it might be less painful to do it in Keystone, and it will at a minimum make those policy rules simpler | 21:52 |
| admiyo | OK let me talk it through with the Keystone folks. I think I can see a way forward. Bascially, if the policy enforcement for Keystone uses a role other than admin for performing the operation when you want it limited to a project, everything else will be ok | 21:53 |
| sean-k-mooney[m] | ya so it looks like the keytone doc was never updated after yoga | 21:53 |
| sean-k-mooney[m] | admiyo: jsut remember the defintion of the personas and role means were effectivly moved out of keystone with the goverance resolution | 21:54 |
| admiyo | Yeah, but they are effectively implemented there, which means if Keystone is wrong, everyone suffers | 21:56 |
| admiyo | and I think that is what I am seeing. At least according to the policy generate | 21:56 |
| admiyo | d on my system | 21:56 |
| admiyo | ADMIN_OR_DOMAIN_ADMIN = ( | 21:57 |
| admiyo | '(' + base.RULE_ADMIN_REQUIRED + ') or ' | 21:57 |
| admiyo | '(' + GRANTS_DOMAIN_ADMIN + ') and ' | 21:57 |
| admiyo | '(' + DOMAIN_MATCHES_ROLE + ')' | 21:57 |
| admiyo | ) | 21:57 |
| admiyo | yeah it is right there in the code. OK. | 21:57 |
| sean-k-mooney[m] | admiyo: https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rbac.html#legacy-admin-continues-to-work-as-it-is is why i marked the bug as wont fix by the way | 22:00 |
| sean-k-mooney[m] | we collectivly commited to not change admin going forward and instead add manager for more scoped privaldged access | 22:01 |
| sean-k-mooney[m] | which is pahse 3 which is where we are now | 22:01 |
| sean-k-mooney[m] | phase 2 and 3 are basiclaly happening in parallel | 22:02 |
| sean-k-mooney[m] | we already use the servie role in some cases but have not removed admin so they were admin_required and not they are admin_or_serivce for upgrade reasons | 22:03 |
| admiyo | sean-k-mooney[m], ok, so the only point you and I now disagree on is the status of the bug. You should make it "in progress" and not "wish list" because you have made changes to fix it. And if you do that, and fix it, I will grant a copy of the t shirt I had made up to both you and to whomever did the lionshare of the work for fixing it. | 22:04 |
| sean-k-mooney[m] | so the reason i set it as wont fix is we wont ever limit the scope of admin to a project or domain | 22:06 |
| sean-k-mooney[m] | so we wont fix the bug in the way it was stated in the deciption | 22:07 |
| sean-k-mooney[m] | we are fixing it by intoducing a new role for “scoped” admin | 22:07 |
| admiyo | OK. No T-shirt for you. But I can see your argument. | 22:10 |
| JayF | admiyo: you'd be surprised at the number of people who read the title/first lines of an issue, see "fix released" and assumed they got exactly what they wanted rather than whatever compromise item got fixed instead :) | 22:11 |
| admiyo | I suspect, however, that things are broken in other services. Admin needs to go away everywhere, and have special enforcement in Keystone | 22:11 |
| JayF | admiyo: are you an ironic user? Would be interested to see what you think about how we took on rbac | 22:11 |
| admiyo | JayF, I think you misspelled Jaded | 22:12 |
| admiyo | JayF, I have been an Ironic user in the past. I will take a look | 22:12 |
| JayF | I don't think we'd likely change anything, but I'm curious how it comes across to someone who has old openstack experience back when Ironic was an admin-only api | 22:12 |
| admiyo | But the short of it is that Ironic is almost all admin functions as defined by Nova. Which may or may not be how you view the world | 22:12 |
| JayF | not anymore :) | 22:12 |
| admiyo | Yeah, I am a big Bifrost fan | 22:13 |
| JayF | You can 100% use a project manager as your nova service user today, as long as the nodes being managed are node.owner=that_project | 22:13 |
| admiyo | https://adam.younglogic.com/2022/05/keystone-ldap-with-bifrost/ | 22:13 |
| sean-k-mooney[m] | well not exactly its perfectly reasonable for ironic to be accessed with member or reader if a node has been assigned to a user/project | 22:13 |
| JayF | admiyo: authenticating ironic, which has a component called ipa, using freeipa, is just devious :D | 22:13 |
| admiyo | We had the name first | 22:14 |
| admiyo | and I worked on both | 22:14 |
| JayF | sean-k-mooney[m]: yeah, I think he's commenting on the old ironic world before we had a concept of non-admin | 22:14 |
| JayF | admiyo: beer had the name first | 22:14 |
| JayF | admiyo: Did I know you as another nickname? Or is my old man brain just being forgetful | 22:14 |
| admiyo | https://adam.younglogic.com/2010/07/talking-to-freeipa-json-web-api-via-curl/ | 22:14 |
| admiyo | ayoung | 22:14 |
| JayF | aha | 22:14 |
| sean-k-mooney[m] | admiyo: for what its worth you have also been able to use ironci via nova with just member for a decade or more | 22:15 |
| JayF | sean-k-mooney[m]: not a project-scoped member, but yes :) | 22:16 |
| admiyo | sean-k-mooney[m], the problem was (and I suspect still is) that other projects are using admin the same way that keystone was. | 22:16 |
| sean-k-mooney[m] | the real delta now is ironic nativly supprot assinging host to project/user adn partitioning the privlsaged info(ipmi creds) form the ablity to do things like power on/off a system | 22:17 |
| JayF | sean-k-mooney[m]: I <3 that our model allows a central ironic to have multiple novas talking to it that all are just isolated by rbac project. How many people use this arch? Hell if I know. But it's a green flag that it's possible :D | 22:17 |
| admiyo | i.e. you as the local admin, not a global one | 22:17 |
| admiyo | JayF, that was something I dreamt of in the past | 22:17 |
| JayF | sean-k-mooney[m]: yeah, that's default in devstack now too, it's really really nice to see all the redacted stuff :D | 22:17 |
| admiyo | very cool. and how it should be | 22:17 |
| JayF | admiyo: we basically support now, at provision time, we will assign the project that owns the instance on the node to "node.lessee", allowing that project minimal access (mainly reader) to that node | 22:18 |
| admiyo | "dear google, I want the actual git repo, not github, for openstack. Please at least return that in the second or third search result." | 22:18 |
| JayF | admiyo: combine that with our new runbooks feature to basically encapsulate sets of steps to perform maintenance + our service feature (basically perform manual cleaning on an active node) == fully self-serve, curated maintenance by the project that runs the node | 22:18 |
| JayF | (this is my downstreams' story: now they can perform firmware updates by providing a runbook to their downstream teams, and those teams can call Ironic API directly to run those runbooks so they can coordinate their own maintenance without ever having to talk to another human) | 22:19 |
| sean-k-mooney[m] | admiyo: they are all ato opendev.org/openstack/whatever but untile we mvoed to gitea away for git web i always wanted the github result | 22:19 |
| admiyo | I know. ..and I am just in grumpy old man mode | 22:20 |
| sean-k-mooney[m] | now that we actully have a nice web ui im fine with either to be honest | 22:20 |
| JayF | sean-k-mooney[m]: admamcarthur7 (someone who works at GR-OSS part time on Ironic) actually said they /prefer/ the gitea opendev.org instance over github | 22:20 |
| admiyo | I wanted the upstream repo URL to clone | 22:20 |
| JayF | sean-k-mooney[m]: I was shocked | 22:20 |
| JayF | s/gitea/whatever_the_hell_that_web_ui_is/ | 22:20 |
| JayF | lol | 22:20 |
| sean-k-mooney[m] | JayF: the search was worse on gitea btu github has been getting worse at that lately | 22:21 |
| JayF | since discovering ripgrep, I almost never search using anything but get a local clone -> rg string | 22:21 |
| sean-k-mooney[m] | rg is fast so is ag the_silver_searcher | 22:22 |
| admiyo | "is_admin": "rule:admin_api or (rule:is_member and role:baremetal_admin)" | 22:22 |
| JayF | sean-k-mooney[m]: https://github.com/ggreer/the_silver_searcher last commit 4 years ago; ripgrep is still maintained | 22:23 |
| admiyo | JayF, so that looks sane. It avoids the problem that I saw on the keystone one. It looks like the onus is on Keystone. And I think the tirck there is to allow a user to only assign roles that they themselves already have, to include implied roles | 22:23 |
| JayF | sean-k-mooney[m]: although knowing ggreer it's probably just that it doesn't need any more code written lol | 22:23 |
| sean-k-mooney[m] | ya it kind of does what it needs too i still use vainall grep | 22:23 |
| admiyo | so id admin implies member, someone with the admin role can assign member. But if I want to make a project manager, they should not (CANNOT!) be allowed to assign admin to anyone in their project | 22:24 |
| sean-k-mooney[m] | but i should swap | 22:24 |
| sean-k-mooney[m] | admiyo: manager would imply member and member would imply reader | 22:24 |
| admiyo | Exactly. | 22:24 |
| sean-k-mooney[m] | you will only be able to delgate roles you have | 22:24 |
| sean-k-mooney[m] | so not admin | 22:24 |
| admiyo | ++ | 22:25 |
| admiyo | That was one of the reasons I wrote that feature... | 22:25 |
| sean-k-mooney[m] | we dont plan to remove admin by the way but we do plan to reduce what its used for as much as possibel by making it doable via manager or service depending on if a human should do it | 22:26 |
| sean-k-mooney[m] | nova’s external evnets api should be service only | 22:27 |
| sean-k-mooney[m] | because humans even the global adim should not send an event impersonating a openstack service | 22:27 |
| JayF | honestly service scope is still the one thing that doesn't make a lot of sense to me | 22:28 |
| JayF | how it's different than system scope in any context except keystone | 22:28 |
| sean-k-mooney[m] | the diffent is scopes apply to the resouce not the token | 22:29 |
| sean-k-mooney[m] | a port is owned by a project nto the system | 22:29 |
| sean-k-mooney[m] | bindign a port to a host is a privaldaged action | 22:29 |
| JayF | so if I take off my Ironic hat for a moment | 22:29 |
| JayF | (since we have a history as admin-only I think we are exceptional a little) | 22:29 |
| sean-k-mooney[m] | only servies shoudl be able to do that not humans | 22:29 |
| JayF | essentially: a system-scoped admin shouldn't be able to do things to project-scoped items, generally | 22:30 |
| JayF | only a service-scoped admin would, since service implies the ability to operate cross-project | 22:30 |
| sean-k-mooney[m] | correct | 22:30 |
| JayF | that makes so much sense why it doesn't make sense in Ironic then | 22:30 |
| JayF | because we have to carry system-scope forward for everything due to $historical_baggage more or less | 22:30 |
| JayF | at least until we're willing to require all new node creations to be scoped to a project, which will likely make a lot of our users unhappy | 22:31 |
| sean-k-mooney[m] | another example form recent cves | 22:31 |
| admiyo | JayF, that was my understanding, but the nova users seem to balk at that change. I am not certain it is alwyas wise tolisten to your users. I suspect in this case it will be good-enough. | 22:31 |
| sean-k-mooney[m] | its safe for a normal user to detach a volume form a nova instance via the nova api | 22:31 |
| sean-k-mooney[m] | its not safe to do that via cinders api | 22:31 |
| sean-k-mooney[m] | so the cinder api requires the service role now | 22:31 |
| admiyo | The problem is how role assignments are done in keystone. If you can do a role assignemnt, you can assign any role. | 22:31 |
| sean-k-mooney[m] | but the nova detach volume api is member | 22:31 |
| admiyo | So that needs to be fixed post haste | 22:31 |
| JayF | oh no, this is venturing close to your favorite beef with the ironic model. Iceberg, dead ahead! Hard to port! LOL | 22:32 |
| admiyo | I think that each API should actually have its own role, and then you use implied role assignments to link to the roles that users actually get | 22:32 |
| sean-k-mooney[m] | well ironci is technically a service so it gets some leway because ye should be able to manage voluem in standalone mode | 22:32 |
| admiyo | so ironic_volume could be a role, and admin implies ironic_volume | 22:32 |
| JayF | that kills a big part of the operator value of the RBAC model | 22:33 |
| admiyo | The issue is that Nova users are using project scoped tokens to do system wide operations. | 22:33 |
| sean-k-mooney[m] | admiyo: it could btu we dont want to have service specific roles | 22:33 |
| sean-k-mooney[m] | we want to have common personas | 22:33 |
| JayF | I would never want, e.g., a reader role that didn't work for all services | 22:33 |
| admiyo | personas are roles. | 22:33 |
| sean-k-mooney[m] | that have a reasonable meeaning across all services | 22:33 |
| JayF | sean-k-mooney[m]: ++ and that persona-based stuff is the value of the RBAC model to operators imho | 22:34 |
| sean-k-mooney[m] | they are but personas are not service specific | 22:34 |
| sean-k-mooney[m] | yep i agree | 22:34 |
| JayF | the Ironic access granted to a nova user who just provisioned an instance is an A++ example of that tbh | 22:34 |
| admiyo | i think in 3 levels | 22:34 |
| JayF | a great example of how we carry the idea of that persona cross-project now in ways we didn't before | 22:34 |
| sean-k-mooney[m] | the fact that you dont need ot give peopel createor on barbican and stack_operator? on heat and just use member was a big win | 22:34 |
| admiyo | implied roles | 22:35 |
| admiyo | reader implies nova_reader | 22:35 |
| admiyo | or something .... | 22:35 |
| sean-k-mooney[m] | we dont want role explsoiton or the imped roels to depend on which service are deployed | 22:35 |
| sean-k-mooney[m] | having sepcific roels that are common across all clouds was part of the interop story | 22:36 |
| admiyo | ah the old explosion argyument | 22:36 |
| sean-k-mooney[m] | so instead of having nova_reader and ironic_reader we have jsut reader which works the same for each proejct | 22:37 |
| admiyo | yeah, that is fine for somethings. | 22:37 |
| admiyo | I think 3 levles: top level persona. bottom level resource. Middle level workflow | 22:37 |
| JayF | Part of what I like about the policy model is that if you need more specific things than that, you can self-serve. It allows us flexibility without making our users get bombarded with a ton of information up front (e.g. the role explosion). | 22:38 |
| admiyo | Execpt that the largest deployer, Red Hat, does not allow policy customization. | 22:38 |
| JayF | I don't work there, I'm an OpenStack upstream contributor :) | 22:39 |
| JayF | find someone with a red hat to complain to about red hat policies :) | 22:39 |
| admiyo | I don;t work there anymore either. Because of things like this.... | 22:39 |
| sean-k-mooney[m] | we dont allow it by defualt because we cant reasonably test it. | 22:39 |
| sean-k-mooney[m] | i work there and we allow people to use it under a supprot excption | 22:39 |
| admiyo | And that is because we mix too many things into policy enforcement | 22:39 |
| JayF | admiyo: I don't speak for RH, but I assume like all things with enterprise software "not supported" means "ask us how much it'd cost" :D | 22:39 |
| sean-k-mooney[m] | that basicallly say fi you want to diveragne and it breaks thats not covered by our support | 22:39 |
| admiyo | Right, which is another way of saying "you can'tdo that" | 22:40 |
| admiyo | and that is due to the existing mechanisms being too limited. | 22:40 |
| sean-k-mooney[m] | i will nether confirm or deniy that we added a tripleo flag to generate the cutome role for a cutomer in a testable way :) | 22:40 |
| JayF | what's tripleo? ;) | 22:41 |
| admiyo | The best thing to come out of TripleO was the Owl T Shirt | 22:41 |
| admiyo | openstack on openstack. OOO | 22:41 |
| JayF | the best thing to come outta tripleo was ironic :D | 22:41 |
| sean-k-mooney[m] | lol | 22:41 |
| admiyo | ironic predated TripleO | 22:41 |
| sean-k-mooney[m] | ironic predates it | 22:41 |
| JayF | I didn't realize that, and bluntly, based on the community when I joined up with it in icehouse, the community was laser-focused on tripleo needs | 22:41 |
| sean-k-mooney[m] | it was nova baremental and then ironci granduated before tripleo | 22:41 |
| JayF | so I assumed it was always a tripleo-y service | 22:42 |
| sean-k-mooney[m] | nope | 22:42 |
| * JayF 's first anything in the OpenStack community was arguing with lifeless and aeva about how cleaning needed to be a first-class Ironic feature and not a workload Ironic ran | 22:42 | |
| sean-k-mooney[m] | my prefered way to use it is via bifrost/kolla-ansible to be honett | 22:42 |
| sean-k-mooney[m] | *honest | 22:42 |
| sean-k-mooney[m] | keep it simple and debugablle | 22:43 |
| admiyo | Yeah, Ironic was first done as an extension to Nova, and then it split out as the code was different enough...If I recall the process correctly | 22:43 |
| sean-k-mooney[m] | it was the nova baremental driver | 22:43 |
| admiyo | Yeah, bifrost is a good thing. | 22:43 |
| admiyo | Right | 22:43 |
| JayF | I knew about nova-bm, just didn't realize it was in use outside of tripleo use cases | 22:43 |
| JayF | I did work on a cloud at $prev_purple_employer that I believe used that driver at some point though | 22:44 |
| sean-k-mooney[m] | admiyo: redhat has replaced tripleo with a set of golang operators to deploy oepsntack as an applciation on openshift now | 22:46 |
| admiyo | https://bugs.launchpad.net/keystone/+bug/1284922 is probably a better way to express the problem now | 22:46 |
| admiyo | And openshift uses Ironic under the covers. | 22:46 |
| admiyo | So we are finally at the "sandwich" | 22:47 |
| admiyo | kube at the bottom. openstack in the middle. Kube at the top | 22:47 |
| JayF | admiyo: also metal3 is secretly Ironic too :) | 22:48 |
| JayF | admiyo: it's pretty funny how everyone is slowly realizing hardware is a pain and just integrating with us instead of building it out from scratch lol | 22:49 |
| sean-k-mooney[m] | admiyo: yep we support deploy openshift on openstack which is istelf on a differnet openshift which uses ironci via metal3 to deploy the openshift 0’s workers | 22:49 |
| JayF | you must pay architects by the layer over there :P | 22:49 |
| JayF | each time we add an abstraction layer, a sales engineer gets their wings? | 22:50 |
| sean-k-mooney[m] | JayF: the thing that makes me sad about metal3 is it limits the scalablity of ironic because the operator cant keep up | 22:50 |
| JayF | (/s obviously) | 22:50 |
| JayF | sean-k-mooney[m]: What I hear you say is that OpenStack is more scalable than K8s CAPI, nice to hear /s | 22:50 |
| sean-k-mooney[m] | im sure there are ways to fix it but i belive they were having to do batches for 25 phsyical hosts when scaling out | 22:50 |
| JayF | metal3 just doesn't expose any of our scaling features, really | 22:51 |
| sean-k-mooney[m] | right but even with one conductor and no sharding i would expct ironci to be able to provision more then 25 servers at a time | 22:51 |
| sean-k-mooney[m] | liek biforst can | 22:51 |
| sean-k-mooney[m] | so im just sad metal3 cant, when direct ironic or even ironci via nova can scale better | 22:52 |
| JayF | sean-k-mooney[m]: it really can, I'm curious what the limitation is there | 22:54 |
| JayF | sean-k-mooney[m]: oh, probably sqlite? | 22:54 |
| JayF | if metal3 still uses sqlite by defaultt | 22:54 |
| sean-k-mooney[m] | JayF: i think its ectd and how its monitoring the proress of the deployment | 22:55 |
| JayF | oh, so *completely* outside of Ironic's pervue then | 22:56 |
| JayF | and likely in a way that could be fixed (e.g. notifications->some consuming service which updates etcd via push instead of polling | 22:56 |
| sean-k-mooney[m] | right ironic can scale more but the baremetal operator that is managing it cant keep up | 22:56 |
| sean-k-mooney[m] | if you ask it to do 100 ndoes it will eventually catch up | 22:56 |
| JayF | when onmetal was first being created; the principle engineer on our team would just drop 100 builds on us occassionally as a test | 22:57 |
| JayF | that is actually the reason we redid the compute manager in ironic to do peer list, which we are just now ripping out | 22:58 |
| JayF | because the old clusteredcomputemanager we used for ironic was so racey | 22:58 |
| sean-k-mooney[m] | right but that predats palcement | 23:00 |
| sean-k-mooney[m] | and like the admin roles in highnsigt we proably never would have done the peer list | 23:01 |
| sean-k-mooney[m] | anyway i should calll it a day and go to sleep o/ | 23:02 |
| JayF | o/ | 23:02 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!