| gibi | _jralbert: you need bauzas to wake up (he is in EU timezone) | 06:45 |
|---|---|---|
| bauzas | good morning Nova | 07:17 |
| bauzas | gibi, _jralbert: what's up ? | 07:17 |
| bauzas | oh the reboot case | 07:17 |
| bauzas | there is an open bug I need to correctly care https://bugs.launchpad.net/nova/+bug/1900800 | 07:18 |
| gibi | bauzas: I knew that you know more | 08:37 |
| bauzas | gibi: I know you knew I know ;) | 08:38 |
| * bauzas starts to work on some agenda for the PTG | 08:38 | |
| gibi | :) | 08:38 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: DNM Trigger tests with new eventlet and dnspython https://review.opendev.org/c/openstack/nova/+/812456 | 08:55 |
| opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Move live migration jobs back to voting https://review.opendev.org/c/openstack/nova/+/812392 | 09:46 |
| opendevreview | Lee Yarwood proposed openstack/nova master: Revert "zuul: Skip block migration with attached volumes tests due to bug #1931702" https://review.opendev.org/c/openstack/nova/+/812473 | 09:46 |
| lyarwood | ^ okay this is looking good now | 09:47 |
| lyarwood | I can't believe I didn't see this earlier but there we go | 09:47 |
| gibi | better later than never ;) | 09:47 |
| bauzas | lyarwood: I promise I'll look at your changes | 10:28 |
| bauzas | lyarwood: actually, I had a comment for the devstack change https://review.opendev.org/c/openstack/devstack/+/812391/4/lib/nova#302 | 10:34 |
| lyarwood | done | 10:36 |
| * lyarwood -> lunch/fuel hunting | 10:39 | |
| lyarwood | actually I'm going to drop for a while this afternoon, could use the air ahead of being locked down until surgery next Tuesday, back online this evening UK time | 11:24 |
| lyarwood | if anything comes up with the initiator change feel free to respin before I'm back | 11:25 |
| kashyap | gibi: For later: Does chengsheng's response on this help any bit? -- https://review.opendev.org/c/openstack/nova/+/762330/ | 13:12 |
| kashyap | I'd like to make some time to get this to completion. The lack of these two newer and better APIs is causing subtle live migration bugs :-( | 13:13 |
| gibi | kashyap: added to my queue but wont reach it today | 13:17 |
| kashyap | Does not have to be today :) | 13:18 |
| *** artom_ is now known as artom | 13:18 | |
| gibi | kashyap: but thank you for the ping, I already forgot about his patch | 13:19 |
| *** dpawlik2 is now known as dpawlik | 13:31 | |
| bauzas | artom_: sean-k-mooney: looks to me some SRIOV live migration issue https://bugs.launchpad.net/nova/+bug/1944619 | 13:53 |
| sean-k-mooney | ill take a look but this is technially hardwoar offloade ovs if they are usign switch dev | 13:54 |
| artom_ | bauzas, hrmm, looks we're not cleaning up properly in some cases | 13:55 |
| sean-k-mooney | currently we do not support live migration with vdpa | 13:55 |
| artom_ | Though I don't like "provoke an error in pre_live_migration" | 13:55 |
| bauzas | artom_: can we triage this one ? | 13:55 |
| *** artom_ is now known as artom | 13:55 | |
| sean-k-mooney | so if they are usign the sriov nic agent they may not use switchdev mode | 13:55 |
| sean-k-mooney | if they are using switchdev mode then they have to use ovs hardware offload instead | 13:55 |
| artom | bauzas, it'd be cool if they could upload full nova compute logs that include the error, ideally without the failure being "provoked" manually | 13:58 |
| bauzas | they gave us some pastebin https://paste.ubuntu.com/p/ThQmDYtdSS/ | 13:59 |
| bauzas | artom: ^ | 13:59 |
| artom | bauzas, yeah, I'm not a fan of them triggering the failure by manually creating the destination instance disk | 14:02 |
| artom | But I suppose it *could* happen organically, so Nova needs to handle it correctly? | 14:02 |
| bauzas | artom: here, I don't really want to discuss about the bug, but rather about its upstream triage | 14:03 |
| bauzas | artom: if you think they need to give us some other verification, could you ask it in the bug report and punt it to the 'Incomplete' status ? | 14:04 |
| artom | bauzas, valid, I suppose, then | 14:05 |
| bauzas | artom: ack, could you then triage it ? | 14:05 |
| artom | Tbh, I don't know the gritty hardware details to understand what sean-k-mooney was saying about various levels of support in variou backends | 14:05 |
| sean-k-mooney | im currently responding to it to level set what shoudl be supproted and what is not ssupproted | 14:06 |
| sean-k-mooney | Check the logs for: libvirt.libvirtError: Requested operation is not valid: PCI device 0000:03:04.1 is in use by driver QEMU, domain instance-00000001 | 14:06 |
| sean-k-mooney | ^ that seams to be the error they are reporting that breaks the roolback | 14:07 |
| sean-k-mooney | that is not related to sriov live migration | 14:07 |
| sean-k-mooney | that is cause by previou incorrect unshleves and possible cold migrations | 14:07 |
| sean-k-mooney | bauzas: artom is aredally working on a fix for the in use device issue | 14:08 |
| bauzas | kk | 14:08 |
| artom | The unshelve fix? It merged in master, currently sitting on Victoria in the backport train | 14:08 |
| artom | Depending on lyarwood's "controversial" helper backport here: https://review.opendev.org/c/openstack/nova/+/791481/1 | 14:09 |
| sean-k-mooney | artom: that is part of it yes | 14:09 |
| sean-k-mooney | the unsleve fix | 14:09 |
| sean-k-mooney | although gibi also has another upstream bug for it which i am looking for currently | 14:09 |
| sean-k-mooney | oh actully no that was a different issue | 14:14 |
| sean-k-mooney | so the in use port i am pretty sure was cause by the unshelve of a vm previously | 14:14 |
| sean-k-mooney | its not related to the sriov live migration | 14:15 |
| * bauzas needs to disappear for getting her kid, see you in 20 mins-ish | 14:18 | |
| sean-k-mooney | bauzas: artom im 99% sure this is hardware offload ovs not standard sriov by the way | 14:29 |
| artom | sean-k-mooney, what's the difference? | 14:38 |
| sean-k-mooney | artom: they have very different code paths | 15:03 |
| sean-k-mooney | artom: for standard sriov all the interface configuretion is doen by libvirt | 15:03 |
| opendevreview | Merged openstack/nova master: nova-manage: Ensure mountpoint is passed when updating attachment https://review.opendev.org/c/openstack/nova/+/811713 | 15:03 |
| sean-k-mooney | for hardware offload ovs os-vif need to create ovs ports and add representor netdevs to ovs and the neutron l2 agent or ovn need to configure ovs to do thinks like vlan tagging | 15:04 |
| sean-k-mooney | so the neutron contol path gose via ovn or the l2 agent not the sriov nic agent | 15:05 |
| sean-k-mooney | the nic is in swtichdev mode not in legacy | 15:05 |
| sean-k-mooney | vlan enfromanet is done via ovs not libvirt and you can use geneve/vxlan networks not just flat/vlan | 15:06 |
| sean-k-mooney | there are some other difference too but those are the main ones | 15:11 |
| yoctozepto | frickler: as centos has 5.1 - did you check the qemu cmdline there? | 15:12 |
| frickler | sean-k-mooney: continuing about the qemu on debian discussion: these are instances with the same flavor (256M ram) in ubuntu and debian https://paste.opendev.org/show/809797/ | 15:13 |
| frickler | yoctozepto: didn't hold that job yet, can do tomorrow | 15:13 |
| *** whoami-rajat__ is now known as whoami-rajat | 15:13 | |
| yoctozepto | frickler: ok | 15:13 |
| yoctozepto | I feel it might be reserving an extra chunk of mem for no reason | 15:13 |
| yoctozepto | as it specifies the size there twice explicitly | 15:14 |
| frickler | sean-k-mooney: the machine types look similar, but the added memory-backend option looks suspicous. but I assume that is added by libvirt, not nova? | 15:14 |
| yoctozepto | << If "-machine memory-backend" is explicitly provided, "-m X" value must match specified backend's size >> | 15:15 |
| yoctozepto | so by those semantics, it's just an unwieldy cmdline parsing assumption | 15:15 |
| frickler | (context: devstack jobs on debian are ooming when tempest runs non-serial, see https://review.opendev.org/c/openstack/devstack/+/789083 ) | 15:15 |
| yoctozepto | and should not consume more | 15:15 |
| yoctozepto | but who knows (-: | 15:16 |
| bauzas | reminder, nova meeting in 44 mins here in this channel #openstack-nova | 15:16 |
| sean-k-mooney | frickler: yoctozepto sorry was updating a bug reading back | 15:33 |
| sean-k-mooney | frickler: the macine types are similar but different version pc-i440fx-4.2 vs pc-i440fx-5.2 | 15:35 |
| sean-k-mooney | frickler: yoctozepto regarding sepcifying the memory twice | 15:37 |
| sean-k-mooney | -machine pc-i440fx-5.2,accel=tcg,usb=off,dump-guest-core=off,memory-backend=pc.ram -cpu qemu64 -m 256 -object memory-backend-ram,id=pc.ram,size=268435456 | 15:37 |
| sean-k-mooney | the -m 256 is the old way to specify it and memory-backend=pc.ram -object memory-backend-ram,id=pc.ram,size=268435456 | 15:37 |
| sean-k-mooney | is the new way but you should be able to specify both | 15:38 |
| sean-k-mooney | without it increase the memory allcoation | 15:38 |
| sean-k-mooney | yoctozepto: frickler those commandline should result in the same behavior as far as i can see but this delta is cause by differnt libvirt versions | 15:39 |
| sean-k-mooney | the nova generated xml shoudl be identicaly | 15:39 |
| sean-k-mooney | debian is using libvirt 7.0.0 and ubuntu is using 6.0.0 | 15:40 |
| sean-k-mooney | frickler: do you have the actual job logs so we can compare the nova xmls and ensure they are the same | 15:46 |
| sean-k-mooney | if the xml created by nova is the same in both cases this is a libvirt/qemu bug most likely | 15:47 |
| bauzas | nova meeting starting in 1 min | 15:59 |
| bauzas | #startmeeting nova | 16:00 |
| opendevmeet | Meeting started Tue Oct 5 16:00:20 2021 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| opendevmeet | The meeting name has been set to 'nova' | 16:00 |
| bauzas | howdy folks | 16:00 |
| dansmith | o/ | 16:00 |
| elodilles | o/ | 16:00 |
| gibi | o/ | 16:00 |
| bauzas | #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting | 16:01 |
| bauzas | should hopefully be quickier than last week | 16:01 |
| bauzas | let's start, people will come along | 16:01 |
| bauzas | #topic Bugs (stuck/critical) | 16:01 |
| bauzas | One Critical bug | 16:02 |
| bauzas | #link https://bugs.launchpad.net/nova/+bug/1945983 | 16:02 |
| bauzas | that said, it's triaged incomplete for us and I think the root cause is on devstack | 16:02 |
| sean-k-mooney | o/ | 16:02 |
| bauzas | I voted on lyarwood's change, we're waiting for devstack cores to look at it | 16:03 |
| sean-k-mooney | ah the iscsi issue | 16:03 |
| bauzas | correct | 16:03 |
| gibi | so we can get back the live migration coverage | 16:03 |
| bauzas | yup, that's what I was about to say | 16:03 |
| bauzas | there is a Depends-On this change | 16:03 |
| bauzas | for enabling again the job | 16:04 |
| bauzas | #link https://review.opendev.org/c/openstack/devstack/+/812391 | 16:04 |
| bauzas | #link https://review.opendev.org/c/openstack/nova/+/812392 | 16:04 |
| bauzas | I'm gonna approve the nova change | 16:04 |
| bauzas | but we need the devstack change first | 16:05 |
| bauzas | any questions about this one ? I think most of us covered this one | 16:05 |
| sean-k-mooney | i think it has a bug | 16:05 |
| sean-k-mooney | i dont see where iscsi-iname is set | 16:05 |
| sean-k-mooney | unless that is a command? | 16:06 |
| sean-k-mooney | ah | 16:06 |
| bauzas | I expect this to be a command | 16:06 |
| sean-k-mooney | it is | 16:06 |
| sean-k-mooney | https://linux.die.net/man/8/iscsi-iname | 16:06 |
| sean-k-mooney | ok then it looks fine | 16:06 |
| bauzas | I verified our docs | 16:06 |
| bauzas | and it generates the FQDN and the port | 16:06 |
| sean-k-mooney | yep | 16:06 |
| bauzas | something like example.org:8008 | 16:06 |
| bauzas | ok, moving on then | 16:07 |
| bauzas | #link 13 new untriaged bugs (+0 since the last meeting): #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New | 16:07 |
| bauzas | we did a few triage | 16:07 |
| bauzas | thanks to the ones who did that | 16:07 |
| bauzas | we still have a backlog and I'll try to see how to reduce it | 16:08 |
| bauzas | more to discuss next week if I can't | 16:08 |
| bauzas | No open bug marked with xena-rc-potential tag #link https://bugs.launchpad.net/nova/+bugs?field.tag=xena-rc-potential | 16:08 |
| bauzas | actually this one is no longer needed, we're past RC | 16:08 |
| bauzas | #topic Gate status | 16:08 |
| bauzas | Nova gate bugs #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure | 16:09 |
| bauzas | as we discussed, we're going to enable the live-migration job back | 16:09 |
| bauzas | congrats to lyarwood for his effort | 16:09 |
| bauzas | Placement periodic job status #link https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly | 16:09 |
| bauzas | placement periodic jobs look fine, nothing to say | 16:09 |
| bauzas | any other point to address about the gate status ? | 16:10 |
| bauzas | most of the wonky issues we had last week were fixed (thanks gmann for tracking it, btw.) | 16:10 |
| bauzas | nothing ? okay, let's continue | 16:11 |
| bauzas | #topic Release Planning | 16:11 |
| bauzas | Release tracking etherpad #link https://etherpad.opendev.org/p/nova-xena-rc-potential | 16:11 |
| bauzas | Final release candidates for both Nova and Placement are RC2 | 16:11 |
| bauzas | The new Xena OpenStack release will be on Wednesday (based on our RC2s) | 16:11 |
| bauzas | (or maybe Thursday, I'm unsure about the exact date) | 16:12 |
| bauzas | anything to mention about Xena ? | 16:12 |
| sean-k-mooney | ack is the review to the release repo proposed yet | 16:12 |
| bauzas | we'll do a retrospective at the PTG | 16:12 |
| bauzas | sean-k-mooney: you mean the final tags ? | 16:13 |
| gibi | https://review.opendev.org/c/openstack/releases/+/812251 | 16:13 |
| bauzas | thanks gibi | 16:13 |
| sean-k-mooney | yes | 16:13 |
| bauzas | I'll look at it tonight | 16:13 |
| bauzas | #topic Review priorities | 16:14 |
| bauzas | #link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement)+label:Review-Priority%252B1 | 16:15 |
| bauzas | I made a few runs over some | 16:15 |
| bauzas | I'll continue this week | 16:15 |
| bauzas | if people have changes they'd like to get reviews, ping me | 16:15 |
| bauzas | ok, I assume no asks | 16:17 |
| bauzas | #topic PTG Planning | 16:17 |
| bauzas | every info is in the PTG etherpad #link https://etherpad.opendev.org/p/nova-yoga-ptg | 16:17 |
| bauzas | I started to look at it | 16:17 |
| bauzas | for the moment, I haven't moved the topics to group themselves by common interest, we only have a few of them | 16:17 |
| bauzas | in the last week, I'll shuffle them if needed | 16:18 |
| bauzas | nothing more boring than a placement-then-something-else-then-placement session :) | 16:18 |
| gibi | bauzas: will there be TC - PTL meeting during the PTG? | 16:19 |
| bauzas | I guess americans would appreciate to wake up later if placement was grouped early in the morning their time :) | 16:19 |
| bauzas | gibi: excellent question, haven't heard yet | 16:19 |
| gibi | hm If my memory serves we had such last time but I might be mistake | 16:19 |
| gibi | n | 16:19 |
| bauzas | I even don't know if the TC came up with an agenda | 16:19 |
| bauzas | dansmith: do you know if there will be some session like it ? | 16:20 |
| dansmith | um, | 16:20 |
| bauzas | maybe about the next upstream priorities ? | 16:20 |
| dansmith | there's a ptl interaction session | 16:20 |
| dansmith | there's a goals session | 16:20 |
| dansmith | I don't think there's a clear schedule yet, but the days (mon, thu, fri) are defined | 16:21 |
| dansmith | https://etherpad.opendev.org/p/tc-yoga-ptg | 16:21 |
| bauzas | ok, I need to do homework then and look at them | 16:21 |
| bauzas | dansmith: thanks | 16:21 |
| gibi | thanks | 16:21 |
| bauzas | I also need to look at the goals candidates | 16:21 |
| bauzas | (if we have some of them already) | 16:21 |
| dansmith | rbac, I think, at least | 16:22 |
| dansmith | should be uncontentious | 16:22 |
| dansmith | but the session doesn't mention specific | 16:22 |
| dansmith | *specifics | 16:22 |
| bauzas | I'd appreciate if we could exactly define the needed efforts for rbac into our project | 16:22 |
| bauzas | but I'll take a look | 16:22 |
| bauzas | one thing I wanted to stress on, | 16:23 |
| bauzas | we're missing at the moment cinder, neutron and keystone cross-project discussions | 16:23 |
| sean-k-mooney | i think most of the rbac effrort will be with interacting with other project | 16:23 |
| artom | I thought we were done with rbac? | 16:23 |
| sean-k-mooney | e.g. making sure we can call neutron with system scopted tokens | 16:23 |
| dansmith | artom: we're done getting ready for rbac :) | 16:23 |
| bauzas | artom: sean-k-mooney: maybe, that's why I want to exactly know the key objectives for this proposed goal | 16:23 |
| artom | dansmith, right, I meant Nova is read, other projects aren't yet, hence the custom policy "workaround" that's currently up for review | 16:24 |
| artom | *ready | 16:24 |
| dansmith | artom: ready, but not completed, AIUI | 16:24 |
| bauzas | me too | 16:24 |
| sean-k-mooney | nova can be called with the secure rbac | 16:24 |
| sean-k-mooney | that does not mean it can call other project properly with it | 16:25 |
| sean-k-mooney | or that they can call nova with it | 16:25 |
| dansmith | can you manage aggregates with a non-project-scoped admin token? | 16:25 |
| bauzas | we need at least some CI job running | 16:25 |
| sean-k-mooney | proably not | 16:25 |
| dansmith | and my next question was, if so, is it tested? :) | 16:25 |
| bauzas | to identify the potential gaps | 16:25 |
| sean-k-mooney | bauzas: yes that is part of the scope of the goal | 16:25 |
| gibi | also we have the specify-target-host-with-project-admin issue open | 16:25 |
| sean-k-mooney | so likely we have work to do | 16:26 |
| dansmith | gibi: yeah, so beyond "meh, it probably works" I expect there are lots of corner cases like that | 16:26 |
| bauzas | gibi: correct, hence my "we need to understand the goal's objectives" | 16:26 |
| gibi | exactly | 16:26 |
| dansmith | hence, I said "done getting ready" :) | 16:26 |
| gibi | :) | 16:26 |
| bauzas | lol | 16:26 |
| bauzas | ok, I think we're done with that for now :) | 16:26 |
| gibi | we are ready to do the _real_ work :D | 16:26 |
| sean-k-mooney | we can discuss more at the ptg but i think it would be nice if we aimed to have a RBAC job running by m1 | 16:26 |
| sean-k-mooney | to give use time to fix the issue it will find | 16:27 |
| artom | Can we do a nova-only job for that? | 16:27 |
| artom | I keep hearing it's "all or nothing" | 16:27 |
| gibi | sean-k-mooney: I'm not even sure tempest is ready to add RBAC testing | 16:27 |
| dansmith | gibi: I think it is | 16:27 |
| sean-k-mooney | lance has been working on some limited testing | 16:27 |
| gibi | ahh OK | 16:27 |
| artom | Actually, do we even need tempest? | 16:27 |
| sean-k-mooney | with tempest | 16:27 |
| sean-k-mooney | artom: yes we do | 16:28 |
| dansmith | gibi: glance is testing it with its tempest plugin | 16:28 |
| artom | Smells like something we can do in functional tests, no? | 16:28 |
| dansmith | but it's much easier to test glance in isolation than nova | 16:28 |
| sean-k-mooney | artom: no | 16:28 |
| gibi | dansmith: cool, so we have an example | 16:28 |
| sean-k-mooney | we need to test interservice interaction | 16:28 |
| dansmith | definitely need tempest, IMHO | 16:28 |
| bauzas | oh yes | 16:28 |
| dansmith | functional will definitely not cut it | 16:28 |
| sean-k-mooney | lances intiall testing show we cant boot with a neutron port | 16:28 |
| bauzas | can't see how we could achieve this without tempest | 16:28 |
| sean-k-mooney | because neturon as configured by devstack at lest currently cant send the network plugged event correctly | 16:28 |
| dansmith | sean-k-mooney: is that their fault or ours? | 16:29 |
| sean-k-mooney | so we definetly need to do tempeest integration testing | 16:29 |
| dansmith | seems like it's likely ours | 16:29 |
| bauzas | we need a job | 16:29 |
| sean-k-mooney | dansmith: not sure yet proably a mix of devstack config and our policy | 16:29 |
| dansmith | or maybe a combo I guess. if they use a system-scoped token but need to augment with project maybe | 16:29 |
| dansmith | sean-k-mooney: ack | 16:29 |
| bauzas | and first and foremost, we need people working on it, if so :) | 16:29 |
| sean-k-mooney | i think it enabeld the new policy on our side but did not create the nova user with the right scope and neutron config | 16:29 |
| dansmith | sean-k-mooney: honestly, I probably need to think on how that event interface should work | 16:30 |
| sean-k-mooney | so we enforced scope but the token neutron used did not have system scope but was an admin token | 16:30 |
| dansmith | like maybe a system-scoped token that looks up any instance on the system is okay | 16:30 |
| dansmith | I would normally think that should be project-scoped, because instances are project-scoped and events are tied to instances | 16:31 |
| dansmith | but it's intended to mostly be used by other services, so .. I dunno | 16:31 |
| sean-k-mooney | i think it should be system scope | 16:31 |
| bauzas | do we have sort of guidance from the keystone team about those events ? | 16:31 |
| sean-k-mooney | becasue as you said this is for service to service interaction | 16:31 |
| sean-k-mooney | but ya its tricky | 16:31 |
| bauzas | or is it us just picking what we want ? | 16:31 |
| dansmith | sean-k-mooney: but it's not something you can ever do without a project-scoped resource ... | 16:31 |
| dansmith | bauzas: we should probably consult a bit | 16:31 |
| sean-k-mooney | dansmith: yep which is why its tricky | 16:32 |
| dansmith | this is kinda my problem with system scope, is that it actually doesn't apply to a lot of stuff, because almost everything is a project-scoped resource | 16:32 |
| artom | Can ports ever be system-scope? | 16:32 |
| artom | Instances are obviously project-scope, but Neutron external events have to do with ports as well | 16:32 |
| sean-k-mooney | im a little relucted to say that api should be project-admin however | 16:32 |
| bauzas | if that becomes a goal, we need some owner of this goal, just sayin' :) | 16:32 |
| dansmith | aggregates are the one example of a system-scoped resource I use a lot | 16:32 |
| artom | Is there some funky network topology that can have system-scoped ports? | 16:32 |
| dansmith | sean-k-mooney: that's another thing, definitely doesn't need admin | 16:32 |
| sean-k-mooney | dansmith: it proably shoudl be system-admin with project-ide set | 16:32 |
| dansmith | sean-k-mooney: that's project-scoped, AFAIK | 16:33 |
| dansmith | events don | 16:33 |
| dansmith | don't need to be admin either | 16:33 |
| sean-k-mooney | the event api is admin only | 16:33 |
| dansmith | they don't need to be, and I don't think they initially were | 16:33 |
| sean-k-mooney | since enduser including operators are not ment to call it | 16:33 |
| dansmith | but admin != scope | 16:34 |
| dansmith | anyway | 16:34 |
| dansmith | clearly needs some discussion and thinking | 16:34 |
| sean-k-mooney | we could make it system member | 16:34 |
| sean-k-mooney | so yes | 16:34 |
| sean-k-mooney | we could drop admin but ya | 16:34 |
| bauzas | could we now put our thoughts into https://etherpad.opendev.org/p/nova-yoga-ptg L204 and move to other things ? | 16:34 |
| sean-k-mooney | lets defer for now | 16:34 |
| sean-k-mooney | am sure but we might want a sperate etherpad | 16:34 |
| sean-k-mooney | linked form there to go though this in more detail | 16:35 |
| bauzas | sean-k-mooney: feel free to link to it | 16:35 |
| bauzas | also, this ties to my other point | 16:35 |
| bauzas | for the moment, we don't have cinder, neutron and keystone cross project sessions at the PTG | 16:35 |
| bauzas | I'd recommend us to engage some talks with the keystone team so we could wrap some stuff about RBAC during the PTG :) | 16:35 |
| bauzas | just sayin' | 16:35 |
| sean-k-mooney | if we had a neutorn one i have one potential topic | 16:35 |
| sean-k-mooney | https://etherpad.opendev.org/p/ovn_live_migration | 16:36 |
| bauzas | ok, so, I guess I can ask the neutron folks and the keystone folks at least | 16:36 |
| bauzas | for timeslots | 16:36 |
| gibi | I don't have any neutron topic at the momement | 16:36 |
| bauzas | lyarwood or others, do people feel wanting to discuss some cinder topics ? | 16:36 |
| artom | I believe he dropped off early today, might want to check with him async | 16:37 |
| sean-k-mooney | i think lee had one topic but lee wont be able to attend the PTG | 16:37 |
| bauzas | gibi: I'll ask the neutron team if they have nova-specific concerns too | 16:37 |
| bauzas | I know lyarwood wants to address the Manila thing | 16:37 |
| gibi | sure | 16:37 |
| sean-k-mooney | Removing os-volume proxy API during Yoga | 16:37 |
| sean-k-mooney | that was the one that i tought lee might bring up | 16:37 |
| bauzas | but here, the question is, do we need the Cinder team for discussing those topics ? | 16:38 |
| artom | That doesn't really affect Cinder, does it? | 16:38 |
| bauzas | artom: that's my question | 16:38 |
| bauzas | do we have topics to discuss that engage the cinder team ? | 16:38 |
| bauzas | I can leave this for next week | 16:38 |
| sean-k-mooney | the manila thing beign supprot for virtio-fs | 16:39 |
| sean-k-mooney | if so then no there is no interation nwith cinder in that case | 16:39 |
| bauzas | for the keystone x-p session, do you think it would be nice for us getting the whole crew or only publicize our discussions about Secure RBAC so people like lance would join ? | 16:39 |
| sean-k-mooney | assuming unified limits work will continue we might want to discuss that too but im not sure if there is anything left on the keystone side | 16:40 |
| sean-k-mooney | dansmith: you wanted some changes to the lib interface i think but not sure if that needs to be covered at the ptg | 16:41 |
| dansmith | no, I don't think soi | 16:41 |
| dansmith | I also think that the issues have been resolved with some of my recent changes to oslo.limit, | 16:41 |
| dansmith | but I need to circle back | 16:41 |
| dansmith | regardless, nothing needing lots of discussion I don't think | 16:42 |
| bauzas | yeah | 16:42 |
| bauzas | so this is rbac only | 16:42 |
| bauzas | we'll figure that out | 16:42 |
| bauzas | moving on | 16:42 |
| bauzas | #topic Stable Branches | 16:42 |
| bauzas | Nova Wallaby 23.1.0 release is proposed #link https://review.opendev.org/c/openstack/releases/+/809000 | 16:42 |
| bauzas | Nova Victoria 22.3.0 release is proposed #link https://review.opendev.org/c/openstack/releases/+/812291 | 16:43 |
| bauzas | Nova Ussuri 21.2.3 release is proposed #link https://review.opendev.org/c/openstack/releases/+/812292 | 16:43 |
| bauzas | elodilles: other things to mention ? | 16:43 |
| elodilles | i'll ping release folks to review those tomorrow | 16:43 |
| bauzas | cool | 16:43 |
| elodilles | and maybe we could merge this: https://review.opendev.org/c/openstack/nova/+/810461 | 16:43 |
| bauzas | oh | 16:44 |
| elodilles | as far as i remember we agreed to accept this | 16:44 |
| bauzas | yup | 16:44 |
| elodilles | and I can backport it to older branches then | 16:44 |
| elodilles | and unblcok train-stein-rocky-queens-pike :) | 16:44 |
| bauzas | I guess you revisioned it because of the pin version ? | 16:44 |
| elodilles | tox min version needed a bump as stephenfin marked | 16:45 |
| bauzas | yeah, that | 16:45 |
| bauzas | ok, I'll +2 it | 16:45 |
| elodilles | thanks \o/ | 16:45 |
| elodilles | that's it i think | 16:45 |
| bauzas | cool | 16:45 |
| bauzas | #topic Sub/related team Highlights | 16:45 |
| bauzas | Libvirt (lyarwood) | 16:45 |
| bauzas | well, he's off | 16:45 |
| bauzas | but I haven't heard anything specifc | 16:46 |
| bauzas | moving on | 16:46 |
| bauzas | #topic Open discussion | 16:46 |
| bauzas | nothing in the agenda, floor is yours | 16:46 |
| gibi | it was a productive meeting | 16:47 |
| bauzas | I can't promise this everyweek | 16:48 |
| bauzas | #endmeeting | 16:48 |
| opendevmeet | Meeting ended Tue Oct 5 16:48:07 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:48 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2021/nova.2021-10-05-16.00.html | 16:48 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2021/nova.2021-10-05-16.00.txt | 16:48 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2021/nova.2021-10-05-16.00.log.html | 16:48 |
| bauzas | oh I forgot to mention the Big News | 16:48 |
| bauzas | shit | 16:48 |
| bauzas | https://twitter.com/openinfradev/status/1445011468248920071 | 16:49 |
| bauzas | bribe me if you wanna get my opinion | 16:49 |
| bauzas | but I'm 99% sold on the destination | 16:49 |
| bauzas | clue #3 was easy | 16:49 |
| gibi | lol | 16:53 |
| dansmith | when will that be? | 16:54 |
| dansmith | spring or fall? | 16:54 |
| sean-k-mooney | dansmith: presumably spring but good question | 16:56 |
| sean-k-mooney | i had been assuming march/april | 16:56 |
| dansmith | was hoping for fall to give a little more time | 16:56 |
| gmann | in Oct/Nov | 16:56 |
| gmann | yeah | 16:56 |
| sean-k-mooney | oh ok | 16:56 |
| dansmith | I can't imagine being ready to travel by march | 16:56 |
| dansmith | gmann: ah cool | 16:56 |
| sean-k-mooney | so the A release ptg then | 16:57 |
| sean-k-mooney | not the Z release | 16:57 |
| sean-k-mooney | so the next 2 PTGs will be virtual | 16:57 |
| sean-k-mooney | dansmith: i very quickly added somethign for the external event on lin 84 https://etherpad.opendev.org/p/nova-yoga-ptg feel free to edit as you see fit | 16:58 |
| gmann | I think PTG are not yet included/planned but it may be as per previous format it might be summit+PTG | 16:58 |
| sean-k-mooney | oh this was explictly for the summit | 16:58 |
| sean-k-mooney | ok | 16:58 |
| dansmith | yeah just summit | 16:58 |
| sean-k-mooney | i assuem it was both | 16:58 |
| sean-k-mooney | well the summit and fourm ware normally together right | 16:59 |
| sean-k-mooney | then ptg is later | 16:59 |
| sean-k-mooney | i guess we will see how it plays out | 16:59 |
| gmann | sean-k-mooney: dansmith bauzas just saw rbac things for PTG, I am going to check projects sessions/topics on RBAC with lbragstad (friday or next week) and then we can plan cross project if needed | 16:59 |
| bauzas | gmann: cool thankq | 17:00 |
| bauzas | sean-k-mooney: dansmith: oh I'd be disappointed if that'd be fall 2022 | 17:00 |
| sean-k-mooney | gmann: i was thinking it might make sense to have a singel cross project session on that goal and then if needed also cover it in the indiviual cross project seesion too | 17:00 |
| gmann | sean-k-mooney: good idea. | 17:01 |
| dansmith | bauzas: not me, I'd be happy if we just assume no travel before 2025 | 17:01 |
| bauzas | that said, the destination I found would be set for fall if COVID-19 wasn't a thing | 17:01 |
| gmann | bauzas: but it would not be so far from your place :) | 17:01 |
| sean-k-mooney | gmann: e.g. have nova/neutron breakouts if we need to talk about a spciric interction but cover the high level rbac topci is a comunity/multi project sesssion | 17:01 |
| bauzas | gmann: honestly, I'm desperate to move | 17:02 |
| bauzas | dansmith: hah, don't know what to say about this then :) | 17:02 |
| sean-k-mooney | dansmith: i dont really like to travel but i do miss the inperson hallway chats and withboarding work we get done with an inperson event | 17:03 |
| bauzas | that ^ | 17:03 |
| sean-k-mooney | unless we go back to having midcycle or other checking events per milestone its hard to cover everything in a virutal event at least the way we do them currently | 17:04 |
| bauzas | fun fact is, if that'd be April, I could literrally go get my car from this place and return to home | 17:04 |
| bauzas | my *new | 17:04 |
| sean-k-mooney | are you really planning on getting a tesla model y | 17:05 |
| gmann | :) | 17:05 |
| bauzas | sean-k-mooney: let's wait for the OIF announcement either way | 17:05 |
| bauzas | sean-k-mooney: I do | 17:05 |
| bauzas | my Superb is nice but I wanna change | 17:05 |
| dansmith | bauzas: I'm disappointed in you | 17:05 |
| dansmith | cars that don't burn dinosaurs = boring | 17:05 |
| bauzas | dansmith: this car can litterally make you fart | 17:06 |
| dansmith | boring. | 17:06 |
| bauzas | so I wouldn't call it "boring" | 17:06 |
| bauzas | :p | 17:06 |
| dansmith | soul-less. | 17:06 |
| bauzas | I'm pretty sure I can find some Spotify playlist that would groan | 17:06 |
| bauzas | dansmith: and automatic transmission, I know :p | 17:07 |
| sean-k-mooney | dansmith: i dont know some of the hydrogen eletic hybrid look pretty fun | 17:07 |
| bauzas | the clutch is past story to me | 17:07 |
| dansmith | sean-k-mooney: has nothing to do with speed or performance | 17:07 |
| sean-k-mooney | you just dont like the generic stylig or is it the feel of the drive | 17:08 |
| dansmith | has nothing to do with being ugly, has entirely to do with being soul-less | 17:09 |
| * sean-k-mooney is currently looking to replace there 6 speed mini but not seeing many car i like to replace it with | 17:09 | |
| bauzas | dansmith: :) | 17:10 |
| bauzas | I understand people's reluctance | 17:11 |
| bauzas | it's just me who wants a car that can drive full electric with some pleasure :) | 17:11 |
| * bauzas looked at the Ioniq 5 and the new EV6 but eventually decided to go with the TMY | 17:12 | |
| sean-k-mooney | i dont like the ioniq 5 really | 17:13 |
| sean-k-mooney | it looks too much like an estate and the sharp line remind me of the telsla pickup | 17:14 |
| bauzas | I don't like the interior and the trunk is small | 17:15 |
| bauzas | I tested it tho, nice to drive | 17:15 |
| bauzas | the EV6 is on its way | 17:15 |
| bauzas | but it costs nearly the TMY without the fun | 17:15 |
| bauzas | and... that's a Kia | 17:16 |
| sean-k-mooney | kia at least has a 7 year warrenty | 17:19 |
| sean-k-mooney | i much prefer the ev6's styling but i dont need a car that size | 17:20 |
| sean-k-mooney | im cosidering a peugeot e-208 but not sure i can justify the price gien how little i drive | 17:21 |
| sean-k-mooney | the 6 speed petrol is about 5-7k cheaper for similar specs | 17:22 |
| bauzas | sean-k-mooney: Sophie drives a e208 | 17:22 |
| bauzas | we got a 7k€ incentive for buying it | 17:23 |
| bauzas | so yeah, without this, I totally agree with you on the fact this is horribly expensive | 17:23 |
| sean-k-mooney | does sophie like the e208 was it worht the investment | 17:26 |
| sean-k-mooney | we can get part of the veical registration tax back and there is an ev grant too | 17:26 |
| sean-k-mooney | but even with that factored in its still several grand more to get teh e versions | 17:27 |
| bauzas | let's discuss this off this topic | 17:27 |
| bauzas | tl;dr: yes she likes it | 17:28 |
| sean-k-mooney | hehe sure :) | 17:28 |
| dansmith | sean-k-mooney: you know that I *wrote* the external events API right? :) | 17:34 |
| sean-k-mooney | dansmith: yes i do | 17:43 |
| sean-k-mooney | i was adding that cotext for other reading it | 17:43 |
| dansmith | sean-k-mooney: okay just trying not to be offended by you 'splaining what it was originally intended for on the etherpad :) | 17:44 |
| dansmith | mkay | 17:44 |
| sean-k-mooney | you orginally worte it for the dhcp server race when using quantum instead of nova network right | 17:44 |
| sean-k-mooney | the reason i put the "must not be called by humans part is really the warnign we have in the api ref by the way" | 17:45 |
| * sean-k-mooney well i mess up those quotes | 17:45 | |
| sean-k-mooney | mainly the last line | 17:45 |
| sean-k-mooney | "Unless you are writing Neutron, Cinder or Ironic code you should not be using this API." | 17:46 |
| dansmith | sean-k-mooney: yeah I know, and that's what is making me think maybe just saying that network-* events come only from system-scoped things, but I dunno | 17:46 |
| sean-k-mooney | ya althoght we have had a usecase propsed int he past where an operator might want to send a network-changed event to referesh the network info cache | 17:47 |
| dansmith | well, we used to have a dedicated API for that, but yes, that's kinda what I mean... those sorts of things | 17:47 |
| sean-k-mooney | its an interesting problem. i dont think we have any other cases where we might want to very the policy based on a sub feild of the request body | 17:47 |
| sean-k-mooney | in this case the event type | 17:48 |
| dansmith | well, I think we can just manually enforce that in code | 17:48 |
| dansmith | I dunno, I just don't want to call this system scope only and eliminate the possibility to use it for human-triggered things in the future | 17:48 |
| sean-k-mooney | i dont think oslo policy can do this today so ya we would have to do it on the nova side | 17:48 |
| sean-k-mooney | well if we assume it remaind amin only the the operator could create a system scoped token ot interact with it | 17:49 |
| sean-k-mooney | but its not exactly a user freindly approch | 17:49 |
| dansmith | but that's not necessarily always an admin thing | 17:50 |
| gmann | default value or scope cannot by handle dynamically in oslo but yes a new policy enforcement based on request field can be added in code | 17:50 |
| sean-k-mooney | this is one api that is currently not usable by nova client or osc by the way | 17:51 |
| dansmith | I know | 17:51 |
| sean-k-mooney | the reason i bring that up is i have wondered if we should close that gap in the past | 17:51 |
| sean-k-mooney | but i have never actully looked at doing that since right now we have no usage that is human trigggered | 17:51 |
| lbragstad | could you rewrite the policy to be service specific? | 17:52 |
| lbragstad | instead of system-specific? | 17:52 |
| sean-k-mooney | lbragstad: how woudl we tell which service called it? do you mean make ti depened on the event type | 17:53 |
| gmann | is that mean the very original design of having multiple system and system_id in enforcement ? | 17:53 |
| dansmith | sean-k-mooney: I think novaclient has an api for this, you just mean there's no CLI right? | 17:54 |
| gmann | like system per service or so | 17:54 |
| sean-k-mooney | dansmith: yes no cli but we likely have the python bindigns for it | 17:54 |
| dansmith | yeah okay | 17:54 |
| dansmith | no CLI makes sense while there's no CLI-able event I think | 17:54 |
| sean-k-mooney | i would expect neutron to just deletagte to novaclinet internally | 17:54 |
| lbragstad | sean-k-mooney does nova need to know which service called that API/ | 17:56 |
| dansmith | no | 17:56 |
| artom | Though currently only Neutron does, IIRC | 17:56 |
| dansmith | it doesn't need to know that neutron only sends network events, but... that's kinda the point | 17:56 |
| artom | And it's obvious that neutron does from the ebent name | 17:56 |
| sean-k-mooney | artom: cinder and cyborg also call it | 17:57 |
| dansmith | it's waiting for a thing, doesn't really need to know that neutron sent it, it just only really works if that's the case :) | 17:57 |
| sean-k-mooney | artom: cinder for swap volume i think and cybrog for the arq binding completion | 17:57 |
| artom | sean-k-mooney, I thought cinder swap volume was another "proper" API? | 17:57 |
| artom | volume-update... | 17:57 |
| sean-k-mooney | perhaps but there are cinder volume event | 17:58 |
| sean-k-mooney | dansmith: so ya neutron is using cinder client https://github.com/openstack/neutron/blob/master/neutron/notifiers/nova.py#L84-L91 | 17:58 |
| sean-k-mooney | *novaclient | 17:58 |
| artom | Yeah https://docs.openstack.org/api-ref/compute/?expanded=update-a-volume-attachment-detail#update-a-volume-attachment | 17:58 |
| sean-k-mooney | artom: it uses for volume extetnions not swap sorry | 17:59 |
| artom | Anyways, we're off point here | 17:59 |
| sean-k-mooney | artom: this is the list of events currently https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/objects/external_event.py#L18-L36 | 18:00 |
| dansmith | currently neutron is using a project token to send these, | 18:00 |
| artom | "power-update" is weird | 18:00 |
| artom | When/how does that happen? | 18:00 |
| dansmith | is that because that's the token and client that neutron uses to interact with nova or other things in a sensible way, or just because it has no knowledge that a system scoped token would be right? | 18:01 |
| dansmith | artom: ironic, IIRC | 18:01 |
| dansmith | artom: it's not just neutron :) | 18:01 |
| * artom stands corrected | 18:01 | |
| lbragstad | yeah - i think it's using the nova service user, which has the admin role | 18:01 |
| lbragstad | and we haven't really gone through that migration yet, from a policy perspective | 18:02 |
| dansmith | lbragstad: meaning it's just naive and not like that user/client is project-scoped for other stuff? | 18:03 |
| dansmith | even still, if neutron is sending an event for an instance or group of ports or something, having it be project-scoped in that call to the event thing also helps to prevent any missteps in sending events to other instances | 18:05 |
| dansmith | like if there was some port change thing going on, during ownership transfer, or anything else | 18:05 |
| dansmith | this goes back to my "the interface is specific to project-scoped resources, so it feels like it too should be project-scoped" | 18:05 |
| lbragstad | iiuc - it's using the nova service user, which just happens to have the admin role (probably for legacy reasons) and that's why it works | 18:07 |
| lbragstad | so - because the nova user has the 'admin' role on the service project, it can update external server events for instance 'foo' in project bar... | 18:07 |
| dansmith | yeah, well the default policy makes the api admin-only anyway | 18:07 |
| sean-k-mooney | yes so its relyin gon haveing a user with admin rights | 18:08 |
| sean-k-mooney | that can be the nova user or a common service user | 18:09 |
| lbragstad | correct - but it's not enforcing tenancy in any way - from what i can tell | 18:09 |
| sean-k-mooney | i dont think it is either today | 18:09 |
| sean-k-mooney | since this is mainly a service to service api | 18:09 |
| dansmith | the api? | 18:10 |
| dansmith | it uses the context to look up the instances it creates the events for, so it should be scoped to the project of the token | 18:11 |
| sean-k-mooney | the external events api does not assert that the user that created teh event is a has any relationship with the project reosuce that is being modified | 18:11 |
| dansmith | or I guess if you're admin then it lets you see them all, but it should still be tight if you're not admin | 18:12 |
| dansmith | you won't be able to look up instances you don't have access to, and you'll fail with a 404 I think | 18:12 |
| sean-k-mooney | well this has been admin only for as long as i can rememebr it exsiting | 18:12 |
| dansmith | or just not send those events, I guess, I'm not sure, but.. | 18:13 |
| dansmith | sean-k-mooney: by default you mean :) | 18:13 |
| sean-k-mooney | yes by default | 18:13 |
| dansmith | I'm just saying I don't think that the api is not tenant-safe | 18:13 |
| sean-k-mooney | well while you can get a responce form tha tapi when you do a post we dont supprot get request on it correct | 18:14 |
| dansmith | there's nothing to get | 18:14 |
| sean-k-mooney | and the info you get back is pretty limited in general | 18:14 |
| dansmith | there's no persistence | 18:14 |
| dansmith | I'm saying you can't POST events for instances you don't own | 18:14 |
| sean-k-mooney | right so since it most a write only api | 18:14 |
| dansmith | it is entirely write-only | 18:14 |
| sean-k-mooney | sure you can | 18:14 |
| gmann | dansmith: sean-k-mooney I remember the discussion of need of service specific role for such cross service API. system scope in this API was not concluded/discussed permission for this at the time of moving to new defaults. | 18:15 |
| sean-k-mooney | well ok you saying if you change the policy to allow anyone call it | 18:15 |
| sean-k-mooney | then it wont work | 18:15 |
| dansmith | objects.InstanceList.get_by_filters( | 18:15 |
| dansmith | cctxt, {'uuid': instance_uuids_by_cell[cell_uuid]}, | 18:15 |
| dansmith | sean-k-mooney: this ^ will not work for instances you don't own if you are not admin | 18:15 |
| gmann | either project scope or both or need service user specific role for such operation | 18:15 |
| sean-k-mooney | do we have a db level check to enforce that | 18:15 |
| dansmith | sean-k-mooney: that's how that interface works right? | 18:16 |
| dansmith | I mean, it certainly was when the code was written :) | 18:16 |
| sean-k-mooney | in the port bindign case the event are all submited as tyepically the nova user regradesss of who owns the port | 18:17 |
| sean-k-mooney | dansmith: the user token is never used | 18:17 |
| dansmith | ...right, as admin yes? | 18:17 |
| sean-k-mooney | also even if it was the even it decupled form a user action | 18:18 |
| sean-k-mooney | yes as admin | 18:18 |
| dansmith | what are we arguing about again? | 18:18 |
| sean-k-mooney | wether the resouce that is being evented on is own by the user/project assocated with teh keyton token used to send the event to nova i think | 18:19 |
| sean-k-mooney | i kind of got confused by what you were saying would and would not work and im not sure you were following me either | 18:19 |
| dansmith | https://github.com/openstack/nova/blob/7967ad78649a1f8d7ffc34ae28274ee89b0011cf/nova/db/main/api.py#L1413-L1416 | 18:20 |
| dansmith | this ^ is where we enforce you can only see your own instances in the DB layer, if not admin | 18:20 |
| dansmith | which makes that API not allow non-admins (if so granted) able to send events to instances they do not own | 18:20 |
| sean-k-mooney | i see | 18:21 |
| dansmith | if the nova user neutron uses has admin, that's why it can send events to any instance, as expected | 18:21 |
| dansmith | I'm talking about if someone were to drop the admin requirement from the policy, or if we later add a human-initiated event, the API is not insecure | 18:21 |
| sean-k-mooney | right i tought we were trying to remove those low level db enfrocements form the code | 18:22 |
| dansmith | in response to this: | 18:22 |
| dansmith | [11:09:14] <lbragstad> correct - but it's not enforcing tenancy in any way - from what i can tell | 18:22 |
| dansmith | [11:09:33] <sean-k-mooney> i dont think it is either today | 18:22 |
| gmann | dansmith: so your proposal is to allow only project scoped (owner or admin) not system right? | 18:22 |
| dansmith | sean-k-mooney: not those, AFAIK.. tons of stuff would need to change at the top, and always provide a project-level filter down or we'd be querying tons of information all the time that we need to filter in python | 18:22 |
| sean-k-mooney | dansmith: right so its the db laywer not the api that enforcing that | 18:22 |
| gmann | sean-k-mooney: we have that as TODO to move admin-checks from DB to API layer side | 18:23 |
| dansmith | gmann: no I'm saying we allow either and enforce scope type based on the event_type in the event, which are probably all system right now (but need to look) | 18:23 |
| dansmith | gmann: you're not talking about removing the project filtering from the DB queries entirely right? | 18:24 |
| gmann | yeah it is all system-admin for now | 18:24 |
| dansmith | (not that it matters for this conversation, because this API behaves the same as all our other ones at the moment, which rely on db filtering) | 18:24 |
| gmann | dansmith: not filtering but like if admin then we check admin policy and call separate DB interface itself at API layer and do not check is_admin in DB | 18:25 |
| dansmith | gmann: right ++ | 18:25 |
| gmann | on event type: keeping system scope is for human-initiated event ? | 18:27 |
| dansmith | no | 18:28 |
| gmann | making them system-only was not right thing at initial implementation | 18:28 |
| dansmith | honestly, I don't even know what to say anymore: I think neutron declaring a project id when making the call is not bad, | 18:28 |
| sean-k-mooney | i think we just converted admin_api to system_admin_api as an itital step | 18:28 |
| dansmith | which makes it a project-scoped interface It hink | 18:28 |
| gmann | yeah | 18:29 |
| dansmith | I'm just saying I don't want this interface to just be forever tagged as system-only | 18:29 |
| dansmith | if we want to make some events system-only, then let's do that per-event, | 18:29 |
| dansmith | but even in the neutron case, I think scoping it by project is probably not a bad idea anyway | 18:29 |
| sean-k-mooney | dansmith: so currently neutron get the credentails form the config so if we wanted to make it non admin they would need to have some other way to generate the token that is used | 18:30 |
| dansmith | sean-k-mooney: not saying non-admin for network events | 18:30 |
| dansmith | I'm saying probably not system | 18:30 |
| dansmith | there's like a 3D chart here I think :) | 18:30 |
| dansmith | or maybe 7D I dunno | 18:30 |
| gmann | dansmith: agree, and that per event check we can do in code itself as you mentioned earlier. if x_event then context.scope == 'system' | 18:31 |
| dansmith | gmann: exactly | 18:31 |
| dansmith | btw, | 18:31 |
| dansmith | I thought at one point we were supposed to get the ability for neutron to say "here's the user's token for context, and here's my token for auth" | 18:32 |
| sean-k-mooney | right but the network-vif-pluged event for example is sent when but the dhcp server has configured the enttry for the ip and the l2 agent has installed flow rules ectra. so it has to create the token form a config file | 18:32 |
| sean-k-mooney | it cannot use the user token | 18:32 |
| dansmith | so we could scope to the user's project from their token, but elevate the ability to do systemy things with the service's token | 18:32 |
| lbragstad | i think there is a policy check for that? | 18:32 |
| dansmith | sean-k-mooney: it doesn't have to use the user's token | 18:32 |
| dansmith | sean-k-mooney: are you saying it doesn't know anything about the project that owns the port? | 18:33 |
| dansmith | because all it needs is the project_id | 18:33 |
| sean-k-mooney | no the port has a project_id assocatied with it | 18:34 |
| dansmith | right, so it can use an admin token generated from creds from the conf, scoped to that project yeah? | 18:34 |
| sean-k-mooney | but the xproejct_override work that lbragstad has been workign on only works with system tokens | 18:34 |
| dansmith | fine? | 18:34 |
| sean-k-mooney | so its a system admin toke with the project_id also set which is what we started with | 18:35 |
| sean-k-mooney | your just saying the project_id should come form the port | 18:35 |
| sean-k-mooney | not be hardcoded in the config | 18:35 |
| dansmith | what we started with was the assertion that it *has* to be that way | 18:35 |
| dansmith | actuall,y | 18:35 |
| dansmith | I think what we started with was "let's discuss this high-bandwidth at ptg" which is *really* what I'd like | 18:36 |
| sean-k-mooney | ok sure. although this is the second deiscussion i have had on this topic as i started talking to lbragstad about it a few weeks ago | 18:38 |
| dansmith | yeah, clearly he needs to be in said high-bandwidth discussion | 18:39 |
| sean-k-mooney | so i was strating form an assumtion that the neutron config would have the creds for a system scoped token and possibly use the project override mechanium | 18:39 |
| sean-k-mooney | im totally fine with and actully like the idea that neutron would overried the project_id with the ide of the port or network owner | 18:40 |
| lbragstad | override == use the project id pass-through functinality? | 18:40 |
| sean-k-mooney | but we woudl need code change to neutron to do that. today the best we can do via the config would be a system_admin token | 18:41 |
| sean-k-mooney | lbragstad: yes | 18:41 |
| sean-k-mooney | project_id passtough is what i mean by overried | 18:41 |
| hemna | can you boot an instance on behalf of another tenant (as admin) ? | 18:41 |
| sean-k-mooney | lbragstad: neutron currently uses the same nova client to send all external events to nova https://github.com/openstack/neutron/blob/master/neutron/notifiers/nova.py#L84-L91 | 18:42 |
| sean-k-mooney | hemna: as in server create no | 18:43 |
| lbragstad | yeah - i hit that issue when i configured nova to enforce scope and worked on the project-id passthrough functionality in ksa and ksm | 18:43 |
| sean-k-mooney | hemna:you can start a server | 18:43 |
| hemna | ok dang. thanks | 18:43 |
| sean-k-mooney | lbragstad: neutron also batchs sending events today | 18:44 |
| sean-k-mooney | so i think they reuse the client for multiple events for diffeenr resouces | 18:44 |
| lbragstad | ok - so that's the PUT portion of that API, right? | 18:45 |
| sean-k-mooney | its a post but yes https://docs.openstack.org/api-ref/compute/#create-external-events-os-server-external-events | 18:45 |
| dansmith | sean-k-mooney: they never batched before, are you sure? | 18:46 |
| lbragstad | ok - nevermind, i thought i saw that API expose a PUT method | 18:46 |
| sean-k-mooney | https://github.com/openstack/neutron/blob/master/neutron/notifiers/nova.py#L146 | 18:46 |
| dansmith | sean-k-mooney: or do you mean batches of ports for a single instance? | 18:46 |
| sean-k-mooney | dansmith: ^ | 18:46 |
| dansmith | okay I thought you said you didn't know the interface was intended for batches? :) | 18:47 |
| dansmith | pretty sure that's new (since I last looked) because I was surprised how long after the interface was written that it was still doing singles | 18:47 |
| dansmith | oh, this is network-changed I think | 18:48 |
| sean-k-mooney | this was network changed yes | 18:48 |
| lbragstad | ok - and the goal is to continue allowing this stuff work to work without having to refactor neutron to use system-scope? | 18:49 |
| dansmith | ack, I would have been looking at vif-plugged | 18:49 |
| dansmith | (which looks to be single-shot) | 18:50 |
| dansmith | lbragstad: I don't think that's a goal | 18:50 |
| dansmith | lbragstad: my intent on arguing here is about declaring this API as forever system-scope-only | 18:50 |
| dansmith | I don't really care if neutron uses system scope for its actions here, although I think providing a project_id is a good idea so we're only hitting instances for that project in a given post | 18:51 |
| lbragstad | oh - ok | 18:52 |
| lbragstad | so your concern is more that we're using a really big hammer to only interact with project resources | 18:53 |
| gmann | and it can be default to project-member (owner), project-admin is not needed as such | 18:53 |
| dansmith | gmann: the actual policy rule default I'm not so concerned about | 18:54 |
| dansmith | I think we could probably set that rule to default to "scope:system and role:admin" or whatever, | 18:54 |
| dansmith | but I just don't want that to be a constant | 18:54 |
| sean-k-mooney | gmann: im not sure we really want normaly project_member to be able to call it by default | 18:55 |
| frickler | sean-k-mooney: those instances are not from a job but I created them manually, so I can do further debugging as needed. "xml generated by nova" is what I can see with "virsh dumpxml"? or do I need to get it from the logs? | 18:55 |
| sean-k-mooney | at least for how it work today | 18:55 |
| dansmith | gmann: by the way, what do you see as being the avenue if we change the allowed scopes in the future for an interface? | 18:55 |
| gmann | or I am thinking if adding new scope 'service_scope' can help in making those API to cross-service based interaction-only. | 18:56 |
| dansmith | meaning, what if something is scope_types=['system'] and we add project to that? how do we signal or microversion that? | 18:56 |
| gmann | dansmith: not microversion but only in releasenotes as it change the configuration only. like we do for every policy change | 18:56 |
| sean-k-mooney | frickler: i was thinking if from the logs | 18:56 |
| sean-k-mooney | frickler: nova prints the xml we send to libvirt at debug level | 18:57 |
| dansmith | gmann: but that won't always work right? like if we go from "you have to be project scoped" to "or you can be system scoped but with a projectid override".. certainly a client needs to know that the behavior has changed right? | 18:57 |
| sean-k-mooney | frickler: we will need that xml if we want to have some of our libvirt folks look at the qemu output and node if its valide or not | 18:57 |
| dansmith | gmann: because that's my concern here over deciding scope for this interface now and wanting to change it later, because it seems like it's something that will be very hard to change later without confusing behavior effects | 18:58 |
| sean-k-mooney | gmann: chanign policy with out a microversion has always made me uncomfortable for what its worth | 18:58 |
| dansmith | if something is fundamentally system-scoped, then fine, | 18:58 |
| dansmith | but if it's not, then it's just adding a layer of potential breakage later | 18:59 |
| dansmith | like I keep saying, aggregates are clearly system-scoped to me, but few other things are, IMHO | 18:59 |
| gmann | dansmith: yes for 'project scope'->'system with project_id' is something we need to discuss if we need to add microvesion or not. with NOTE: new policies are not microversioned. | 18:59 |
| dansmith | (or at least, I have few other examples that I think are legit) | 18:59 |
| gmann | but supporting both for a release or so | 18:59 |
| dansmith | gmann: welp, FWIW, that makes me rather nervous :/ | 19:00 |
| dansmith | especially knowing that our customers don't run adjacent releases | 19:00 |
| opendevreview | Lance Bragstad proposed openstack/nova master: DNM: Attempt to expose external events API to project users https://review.opendev.org/c/openstack/nova/+/812601 | 19:00 |
| opendevreview | Lance Bragstad proposed openstack/nova master: DNM: Update external events policy to allow for project-id passthrough https://review.opendev.org/c/openstack/nova/+/812602 | 19:00 |
| opendevreview | Lance Bragstad proposed openstack/nova master: DNM: Update policy to allow for service-specific auth targets https://review.opendev.org/c/openstack/nova/+/812603 | 19:00 |
| lbragstad | dansmith cover your eyes - fstrings on the way | 19:00 |
| sean-k-mooney | gmann: new policyes i can may see as not requiring a microverson. any change in default policy behaivor for any endpoint however i kind of feell like it should be a microversion | 19:00 |
| dansmith | lbragstad: *shudder* | 19:00 |
| gmann | sean-k-mooney: new policy are change in default behavior too | 19:00 |
| dansmith | sean-k-mooney: policies can be changed by the operators, so I don't think that means they need microversions, | 19:01 |
| sean-k-mooney | e.g. if add a new policy has no visable effect then fine but if it changes the behaivor in any way by default i think it needs a microversion | 19:01 |
| dansmith | sean-k-mooney: but new core unchangeable policy things like "system-scope plus project id" are different to me | 19:01 |
| sean-k-mooney | dansmith: i agree that operators can certly change policy and alter behavior and that does not affect the microversion today | 19:02 |
| gmann | yeah "system-scope plus project id" is completely different new thing | 19:02 |
| sean-k-mooney | but im not really sure where the line is | 19:02 |
| sean-k-mooney | gmann: i might be missing something but im not really convicned it is | 19:03 |
| dansmith | gmann: that's my concern over deciding the fate of this event api, FWIW | 19:03 |
| sean-k-mooney | what makes that different then changing an api form project member to say project admin | 19:03 |
| dansmith | sean-k-mooney: it is definitely a different thing to me | 19:03 |
| dansmith | sean-k-mooney: it requires you know that the thing be new enough *and* that you need a different client behavior to make it work | 19:04 |
| dansmith | neither of which are discoverable from the outside | 19:04 |
| dansmith | sean-k-mooney: because of the client behavior change required | 19:04 |
| sean-k-mooney | is there not a client behaivor change requried with enabling scope enforce too | 19:04 |
| gmann | sean-k-mooney: project member to say project admin, operator can always change it back via policy.yaml | 19:04 |
| lbragstad | https://review.opendev.org/c/openstack/nova/+/812601/1/nova/policies/server_external_events.py would be example of how to expose this to project users | 19:04 |
| lbragstad | but https://review.opendev.org/c/openstack/nova/+/812602/1/nova/policies/server_external_events.py would be using project id pass-through | 19:05 |
| dansmith | sean-k-mooney: yep, that's definitely a change in line with the one described above, but less concerning than just a single policy change to use a different role I think | 19:05 |
| sean-k-mooney | lbragstad: thats assuming we standardise an new service role? or does that already exist | 19:06 |
| lbragstad | it's not standardized - just an example | 19:06 |
| dansmith | lbragstad: yeah, the second is my preference | 19:07 |
| sean-k-mooney | the resaond i was asking is the idea of a service role is a new construct to me adn would have to be standarised i think if we were to use it in code | 19:07 |
| gmann | dansmith: added sysmte+project-id case in PTG etherpad L227 | 19:07 |
| dansmith | lbragstad: and then perhaps we limit it per event, so to say "we know neutron uses system scope, so we're limiting network-* events to system scoped contexts to avoid users being able to send those" | 19:07 |
| dansmith | gmann: thanks | 19:07 |
| sean-k-mooney | the second i think is just the system_amdin_or_owner policy right | 19:07 |
| dansmith | sean-k-mooney: no | 19:08 |
| dansmith | sean-k-mooney: it's system scope WITH a project override, or a regular project token | 19:08 |
| gmann | yeah | 19:08 |
| sean-k-mooney | sorry not admin or owner | 19:08 |
| sean-k-mooney | ok | 19:08 |
| lbragstad | dansmith yeah - if nova can pack that as target data - you could do something like role:service and networking:%(event_type)s | 19:08 |
| sean-k-mooney | ya i see the differnce | 19:08 |
| dansmith | also, lbragstad I'm fine with the policy not allowing regular users to hit the API until/unless we have a human-oriented event | 19:08 |
| dansmith | lbragstad: because it's just a default | 19:08 |
| dansmith | lbragstad: oh we could do that too I guess | 19:09 |
| dansmith | lbragstad: I don't think we apply the policy like that today, but we could | 19:09 |
| gmann | but that need policies per event | 19:09 |
| dansmith | lbragstad: I was thinking more like hard-coded python to check the scope after we look at the event type | 19:09 |
| dansmith | gmann: right, I like it less, but it's an option | 19:09 |
| lbragstad | o.p should generalize the target data | 19:10 |
| lbragstad | s/should generalize/generalizes/ | 19:10 |
| lbragstad | i'm not sure if it supports regular expressions though | 19:10 |
| dansmith | I dunno what your point is | 19:10 |
| gmann | dansmith: lbragstad oh or we can make check_str with OR-per-event and like what lbragstad mentioned | 19:10 |
| dansmith | I'm saying I think there's something to be said for it not being in the actual rule | 19:11 |
| frickler | sean-k-mooney: https://paste.opendev.org/show/809803/ , I can also give you or others access to the held nodes if you want to have a look yourselves | 19:11 |
| gmann | so single policy, check str with event_x AND project OR event_y AND system | 19:11 |
| dansmith | tbh, I think some of these overly complex policy rules are asking for admins to get confused, write a policy wrong and break or expose something they don't expect | 19:11 |
| lbragstad | yeah - i was just throwing it out there as an option if you wanted a way to restrict some users from posting networking events | 19:11 |
| dansmith | which makes me wonder if putting this in the policy rule is the right thing, but.. it's an option and I don't totally hate it | 19:11 |
| dansmith | lbragstad: ack | 19:12 |
| lbragstad | otherwise - if you consider it business logic, putting it in the service itself makes sense | 19:12 |
| sean-k-mooney | frickler: so in both cases the nova generated xml is more or less identical as we woudl expect meanign any delta in behavior is a libvirt bug | 19:12 |
| dansmith | yeah, "unless the admin could/would need to change it, put it in code" | 19:12 |
| sean-k-mooney | frickler: we are simplely setting the memory with <memory>262144</memory> we are not speficying any of the advance memory backing config | 19:13 |
| dansmith | lbragstad: another question: would it be rude for me to say I'm officially burned out on policy stuff for like at least 24 hours? | 19:13 |
| lbragstad | join the club | 19:13 |
| dansmith | haha | 19:14 |
| lbragstad | :) | 19:14 |
| lbragstad | it's only tuesday, too | 19:14 |
| dansmith | I know, I was going to say "for the week" but realized there's a lot of week left :/ | 19:14 |
| sean-k-mooney | well i for one am planning on not thining about this until the ptg | 19:15 |
| dansmith | sean-k-mooney: I think you are a LIAR | 19:15 |
| sean-k-mooney | lbragstad: that said im going to try and re review the latest version of your ooo changes | 19:15 |
| lbragstad | every time i try *not* thinking about policy, i think about policy | 19:16 |
| sean-k-mooney | dansmith: :) i still find our approch to policy to be very very non intuitive | 19:16 |
| dansmith | sean-k-mooney: you just proved me right! | 19:16 |
| lbragstad | going out on a limb here, but if a sub-system takes 5 years to change, it's probably more than just non-intuitive :) | 19:18 |
| dansmith | heh | 19:20 |
| sean-k-mooney | isnt the replacemnt for it ment to be simpler and more intunitieve to show forward progress :) | 19:21 |
| *** tamas_erdei is now known as terdei | 20:32 | |
| opendevreview | Merged openstack/nova stable/ussuri: [stable-only] Pin virtualenv and setuptools https://review.opendev.org/c/openstack/nova/+/810461 | 21:08 |
| opendevreview | melanie witt proposed openstack/nova stable/train: [stable-only] Pin virtualenv and setuptools https://review.opendev.org/c/openstack/nova/+/812553 | 21:30 |
| opendevreview | melanie witt proposed openstack/nova stable/train: [stable-only] Pin virtualenv and setuptools https://review.opendev.org/c/openstack/nova/+/812553 | 21:34 |
| opendevreview | melanie witt proposed openstack/nova stable/train: [stable-only] Pin virtualenv and setuptools https://review.opendev.org/c/openstack/nova/+/812553 | 21:40 |
| opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Add PCI VPD Capability Handling https://review.opendev.org/c/openstack/nova/+/808199 | 21:51 |
| opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Support remote-managed SmartNIC DPU ports https://review.opendev.org/c/openstack/nova/+/812111 | 21:51 |
| opendevreview | melanie witt proposed openstack/nova master: Add logic to enforce local api and db limits https://review.opendev.org/c/openstack/nova/+/712139 | 23:41 |
| opendevreview | melanie witt proposed openstack/nova master: Enforce api and db limits https://review.opendev.org/c/openstack/nova/+/712142 | 23:41 |
| opendevreview | melanie witt proposed openstack/nova master: Update quota_class APIs for db and api limits https://review.opendev.org/c/openstack/nova/+/712143 | 23:41 |
| opendevreview | melanie witt proposed openstack/nova master: Update limit APIs https://review.opendev.org/c/openstack/nova/+/712707 | 23:41 |
| opendevreview | melanie witt proposed openstack/nova master: Update quota sets APIs https://review.opendev.org/c/openstack/nova/+/712749 | 23:41 |
| opendevreview | melanie witt proposed openstack/nova master: Tell oslo.limit how to count nova resources https://review.opendev.org/c/openstack/nova/+/713301 | 23:41 |
| opendevreview | melanie witt proposed openstack/nova master: Enforce resource limits using oslo.limit https://review.opendev.org/c/openstack/nova/+/615180 | 23:41 |
| opendevreview | melanie witt proposed openstack/nova master: Add legacy limits and usage to placement unified limits https://review.opendev.org/c/openstack/nova/+/713498 | 23:41 |
| opendevreview | melanie witt proposed openstack/nova master: Update quota apis with keystone limits and usage https://review.opendev.org/c/openstack/nova/+/713499 | 23:41 |
| opendevreview | melanie witt proposed openstack/nova master: Add reno for unified limits https://review.opendev.org/c/openstack/nova/+/715271 | 23:41 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!