| *** macz_ has quit IRC | 00:06 | |
| *** ociuhandu has joined #openstack-nova | 00:33 | |
| *** ociuhandu has quit IRC | 00:37 | |
| *** macz_ has joined #openstack-nova | 00:50 | |
| *** tosky has quit IRC | 00:51 | |
| *** macz_ has quit IRC | 00:55 | |
| *** mlavalle has quit IRC | 01:24 | |
| openstackgerrit | sean mooney proposed openstack/nova master: libvirt: Add vDPA nodedev parsing https://review.opendev.org/c/openstack/nova/+/770533 | 01:26 |
|---|---|---|
| openstackgerrit | sean mooney proposed openstack/nova master: libvirt: Add guest generation for vDPA https://review.opendev.org/c/openstack/nova/+/770532 | 01:26 |
| openstackgerrit | sean mooney proposed openstack/nova master: pci: Add vDPA vnic to PCI request mapping and filtering https://review.opendev.org/c/openstack/nova/+/778350 | 01:26 |
| openstackgerrit | sean mooney proposed openstack/nova master: add hw:mlock extra spec https://review.opendev.org/c/openstack/nova/+/778347 | 01:26 |
| *** martinkennelly has quit IRC | 01:31 | |
| *** xek_ has joined #openstack-nova | 01:33 | |
| *** xek has quit IRC | 01:36 | |
| *** macz_ has joined #openstack-nova | 02:04 | |
| *** k_mouza has joined #openstack-nova | 02:07 | |
| *** macz_ has quit IRC | 02:09 | |
| *** k_mouza has quit IRC | 02:12 | |
| *** ociuhandu has joined #openstack-nova | 02:21 | |
| *** psachin has joined #openstack-nova | 02:22 | |
| *** spatel has joined #openstack-nova | 02:25 | |
| *** ociuhandu has quit IRC | 02:26 | |
| *** artom has quit IRC | 02:46 | |
| *** whoami-rajat_ has joined #openstack-nova | 02:55 | |
| *** whoami-rajat_ is now known as whoami-rajat | 03:16 | |
| *** macz_ has joined #openstack-nova | 03:17 | |
| *** macz_ has quit IRC | 03:22 | |
| *** sapd1 has joined #openstack-nova | 03:23 | |
| *** vishalmanchanda has joined #openstack-nova | 03:27 | |
| *** spatel has quit IRC | 03:30 | |
| *** jamesdenton has quit IRC | 03:33 | |
| *** jamesdenton has joined #openstack-nova | 03:34 | |
| *** rcernin has quit IRC | 03:48 | |
| *** macz_ has joined #openstack-nova | 03:59 | |
| *** macz_ has quit IRC | 04:04 | |
| *** sapd1 has quit IRC | 04:07 | |
| *** spatel_ has joined #openstack-nova | 04:09 | |
| *** rcernin has joined #openstack-nova | 04:15 | |
| *** rcernin has quit IRC | 04:19 | |
| *** rcernin has joined #openstack-nova | 04:19 | |
| *** lifeless has quit IRC | 04:42 | |
| *** k_mouza has joined #openstack-nova | 04:45 | |
| *** ratailor has joined #openstack-nova | 04:47 | |
| *** k_mouza has quit IRC | 04:50 | |
| *** sapd1 has joined #openstack-nova | 04:58 | |
| *** macz_ has joined #openstack-nova | 05:00 | |
| *** macz_ has quit IRC | 05:05 | |
| *** whoami-rajat has quit IRC | 05:28 | |
| *** sapd1 has quit IRC | 05:32 | |
| openstackgerrit | Yongli He proposed openstack/nova master: smartnic support - functional tests https://review.opendev.org/c/openstack/nova/+/780147 | 05:34 |
| *** macz_ has joined #openstack-nova | 05:56 | |
| *** ociuhandu has joined #openstack-nova | 05:58 | |
| *** macz_ has quit IRC | 06:00 | |
| *** ociuhandu has quit IRC | 06:02 | |
| *** LinPeiWen has quit IRC | 06:02 | |
| *** spatel_ has quit IRC | 06:10 | |
| *** LinPeiWen has joined #openstack-nova | 06:13 | |
| *** ralonsoh has joined #openstack-nova | 06:18 | |
| *** rcernin has quit IRC | 06:47 | |
| *** lifeless has joined #openstack-nova | 06:49 | |
| *** k_mouza has joined #openstack-nova | 06:54 | |
| *** whoami-rajat_ has joined #openstack-nova | 06:56 | |
| *** rcernin has joined #openstack-nova | 06:58 | |
| *** k_mouza has quit IRC | 06:59 | |
| *** khomesh24 has joined #openstack-nova | 06:59 | |
| *** ociuhandu has joined #openstack-nova | 07:03 | |
| *** ociuhandu has quit IRC | 07:07 | |
| *** slaweq has joined #openstack-nova | 07:11 | |
| *** sapd1 has joined #openstack-nova | 07:18 | |
| *** rcernin has quit IRC | 07:33 | |
| *** rcernin has joined #openstack-nova | 07:34 | |
| *** rcernin has quit IRC | 07:40 | |
| *** rcernin has joined #openstack-nova | 07:42 | |
| *** belmoreira has joined #openstack-nova | 07:44 | |
| *** ociuhandu has joined #openstack-nova | 07:45 | |
| *** ociuhandu has quit IRC | 07:45 | |
| *** sapd1 has quit IRC | 07:46 | |
| *** rcernin has quit IRC | 07:47 | |
| *** dklyle has quit IRC | 07:47 | |
| *** sapd1 has joined #openstack-nova | 07:50 | |
| *** jamesdenton has quit IRC | 07:55 | |
| *** jamesdenton has joined #openstack-nova | 07:56 | |
| *** macz_ has joined #openstack-nova | 07:57 | |
| *** ociuhandu has joined #openstack-nova | 07:57 | |
| *** xinranwang has joined #openstack-nova | 07:57 | |
| *** ociuhandu has quit IRC | 07:57 | |
| *** ociuhandu has joined #openstack-nova | 07:58 | |
| *** gryf is now known as gryf_ | 07:58 | |
| *** _gryf is now known as gryf | 07:59 | |
| *** macz_ has quit IRC | 08:01 | |
| *** rcernin has joined #openstack-nova | 08:06 | |
| *** zzzeek has quit IRC | 08:07 | |
| *** zzzeek has joined #openstack-nova | 08:11 | |
| *** gyee has quit IRC | 08:11 | |
| gibi | sean-k-mooney: ack, looking | 08:11 |
| *** rcernin has quit IRC | 08:11 | |
| gibi | yonglihe: hi! Is it possible to test the smartnic series in a devstack with the fake cyborg driver? | 08:13 |
| yonglihe | functional test it is. devstack, we need real hw for that. | 08:14 |
| yonglihe | except that, the devstack have some feature to accept the VM stuck to a none-exit pci devices as succeefuly state(that expected off cause). | 08:15 |
| yonglihe | If we also use a faked libvirt, that workable. | 08:16 |
| yonglihe | gibi: btw, othe comments gonna take me 2 or 3 days hard work. | 08:20 |
| gibi | yonglihe: so the devstack + cyborg fake driver solution doesn't work as the fake driver returns nonexistent pci device and that cannot be added to the VM. thank make sense. | 08:21 |
| gibi | yonglihe: which comment is the hard one? | 08:21 |
| yonglihe | code reactors. | 08:22 |
| yonglihe | refactors | 08:22 |
| yonglihe | but we have solutions for that. | 08:22 |
| *** tesseract has joined #openstack-nova | 08:23 | |
| *** rcernin has joined #openstack-nova | 08:23 | |
| yonglihe | The one need decouple the flavor arq from port-arq need more force, that cofused everyone. | 08:25 |
| gibi | I see | 08:25 |
| gibi | alex_xu: will you be around in the next week to review the smartnic series ? | 08:26 |
| yonglihe | but we could use port uuid instead of instace uuid as arq consumer, that will much more clear | 08:26 |
| gibi | hm that sounds like a good idea | 08:26 |
| *** ociuhandu has quit IRC | 08:26 | |
| *** ociuhandu has joined #openstack-nova | 08:27 | |
| *** rcernin has quit IRC | 08:28 | |
| alex_xu | gibi: yes, I will be there to review the smartnic series | 08:31 |
| gibi | alex_xu: thanks | 08:31 |
| alex_xu | np | 08:32 |
| gibi | yonglihe: how do you feel can you propose the fixes not later than Wednesday next week? | 08:32 |
| *** andrewbonney has joined #openstack-nova | 08:33 | |
| *** rcernin has joined #openstack-nova | 08:33 | |
| yonglihe | Your Wednesday, thats sounds ok for me. | 08:33 |
| *** rcernin has quit IRC | 08:35 | |
| gibi | Ok | 08:35 |
| *** ociuhandu has quit IRC | 08:50 | |
| bauzas | good morning Nova | 08:50 |
| *** tosky has joined #openstack-nova | 08:51 | |
| bauzas | have we discussed on FFEs ? | 08:51 |
| bauzas | nvm http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020772.html | 08:52 |
| gibi | bauzas: yepp, and we have two requested on the ML | 08:53 |
| bauzas | I see the smartnic ones | 08:53 |
| bauzas | hence my question | 08:53 |
| gibi | bauzas: sean-k-mooney requested one for the vdpa too | 08:53 |
| bauzas | oh, the vdpa one | 08:54 |
| gibi | if you read the scrollback then you see my discussion with yonglihe and alex_xu about the smartnic one | 08:54 |
| gibi | from this morning | 08:54 |
| bauzas | the vdpa one got eyes on it yesterday AFAICT | 08:54 |
| bauzas | but what about the smartnic one ? | 08:54 |
| gibi | bauzas: yes, and we found a bug | 08:54 |
| gibi | bauzas: so the vdpa one is pretty close as sean-k-mooney fixed the bug during the night | 08:55 |
| gibi | bauzas: for vdpa we need a patch that blocks unsupported operations like live-migrate, and needs a reno | 08:55 |
| bauzas | merging stuff on today seems reasonable to me provided we kinda verify we don't really change a lot | 08:55 |
| bauzas | like, adding new stuff looks good to me, but for example, asking to have a new ovo field, no | 08:56 |
| gibi | I think we have a good chance to approve the vdpa one today | 08:56 |
| bauzas | gibi: ok, and for smartnic ? | 08:56 |
| gibi | that is a bigger step | 08:56 |
| bauzas | gibi: I can try to look at it | 08:56 |
| bauzas | hah | 08:56 |
| gibi | I had various comments yesterday | 08:56 |
| bauzas | I'll quickly look at the series | 08:56 |
| gibi | yonglihe thinks the hard parts of that can be fixed not later than Wednesday | 08:57 |
| gibi | bauzas: and alex_xu confirmed that he will be around to review | 08:57 |
| bauzas | because as i said, if they want to modify some RPC APIs or want to change ov.o objects or DB, then I wouldn't be super happy | 08:57 |
| bauzas | gibi: well, Wednesday is a bit late, no ? :- | 08:57 |
| bauzas | :( | 08:57 |
| gibi | it is a stretch | 08:58 |
| bauzas | again, my concern is not really about when, but rather about what's modified | 08:58 |
| gibi | bauzas: when you say no new ovo field do you mean we should not merge anything after tomorrow that changes an ovo? | 08:58 |
| bauzas | yes | 08:58 |
| bauzas | or a RPC API | 08:58 |
| bauzas | or a DB upgrade | 08:58 |
| bauzas | or... | 08:58 |
| bauzas | or a API microversion | 08:58 |
| gibi | there is no DB/RPC change in vdpa or smartnic | 08:59 |
| gibi | but both changes ovo | 08:59 |
| bauzas | because merging those while we're already close to RC1 means that if we see problems, it could be difficult to just revert the changes | 08:59 |
| gibi | smartnic adds field https://review.opendev.org/c/openstack/nova/+/771363/13/nova/objects/network_request.py | 08:59 |
| bauzas | /o\ | 08:59 |
| bauzas | if we merge those ovo changes today, I'm OK | 08:59 |
| bauzas | what I'm not OK it to merge ovo changes like next week | 09:00 |
| gibi | vdpa adds enum value https://review.opendev.org/c/openstack/nova/+/777481/8/nova/objects/fields.py | 09:00 |
| bauzas | I know for vdpa | 09:00 |
| gibi | bauzas: fair point | 09:00 |
| gibi | then I think smartnic needs to be deferred | 09:00 |
| gibi | I'm not happy to merge the ovo change today without seeing the whole series coming together | 09:00 |
| bauzas | again, it's more a question about how to be reverting if we merge them next week and we see problems | 09:00 |
| *** tesseract has quit IRC | 09:00 | |
| bauzas | gibi: yeah :( | 09:01 |
| gibi | and you have a valid point about the risk in ovo | 09:01 |
| bauzas | but for example, I could be OK with merging a new config option on Monday | 09:01 |
| bauzas | Wednesday is late | 09:01 |
| bauzas | but, at least if we see problems, it's simple to just revert | 09:02 |
| bauzas | people could tell it's simple to revert ovo changes | 09:02 |
| bauzas | but the problem here is that master is down then | 09:02 |
| gibi | what do you mean by down? | 09:03 |
| *** lucasagomes has joined #openstack-nova | 09:03 | |
| *** tesseract has joined #openstack-nova | 09:04 | |
| *** xek_ is now known as xek | 09:06 | |
| *** ociuhandu has joined #openstack-nova | 09:06 | |
| *** derekh has joined #openstack-nova | 09:07 | |
| bauzas | gibi: I mean that if we would want to revert an ovo patch, this would mean that we absolutely need to be sure that we don't merge other ovo changes after this one | 09:08 |
| *** martinkennelly has joined #openstack-nova | 09:08 | |
| bauzas | we can pretend it never existed but there is a high risk of tangling | 09:08 |
| bauzas | hence me being super conservative about such changes to be merged while we're so close from RC1 | 09:09 |
| gibi | tanglig by having two different ovo object with the same version | 09:09 |
| gibi | yonglihe: bauzas had a point above about the ovo change being risky close to RC1 and I think his point is valid. I'm affraid we have to defer the smartnic series to X | 09:12 |
| gibi | alex_xu: ^^ | 09:12 |
| bauzas | I'll review the series this morning | 09:12 |
| bauzas | it's fair to look at the change before cutting the rope | 09:13 |
| yonglihe | gibi: got, that's reasonable. likely, we could merge smartnic in very ealry stage of X | 09:13 |
| bauzas | but since I haven't reviewed the spec, it'll take time for me to load the context in mind :) | 09:13 |
| bauzas | yonglihe: if you don't mind then, I can propose sponsoring your series as soon as we branch RC1 | 09:14 |
| bauzas | which will be in two weeks | 09:14 |
| yonglihe | bauzas, sure, that sound good for me, and thanks. | 09:15 |
| gibi | yonglihe: sorry for the bad news and thanks for the flexibility | 09:15 |
| gibi | I will summarize this to the ML | 09:15 |
| bauzas | yonglihe: thanks for your understanding :( | 09:15 |
| bauzas | yonglihe: ping me once RC1 is cut, and then I'll review your patches | 09:15 |
| *** ociuhandu has quit IRC | 09:16 | |
| yonglihe | bauzas: we all responsible to minimize risk, that's right way to go. thanks. | 09:16 |
| bauzas | yonglihe: don't forget to ping me as I could forget | 09:17 |
| yonglihe | bauzas, sure. | 09:18 |
| openstackgerrit | Wenping Song proposed openstack/nova master: Remove get_device_profile_request_groups function in cyborg.py https://review.opendev.org/c/openstack/nova/+/780206 | 09:19 |
| *** ociuhandu has joined #openstack-nova | 09:23 | |
| alex_xu | gibi: bauzas got it, thanks for the review and feedback anyway | 09:29 |
| *** kd has quit IRC | 09:32 | |
| openstackgerrit | Jinsheng Zhang proposed openstack/nova stable/victoria: Add nova support ironic instance port group network metadata https://review.opendev.org/c/openstack/nova/+/780209 | 09:44 |
| *** dtantsur|afk is now known as dtantsur | 09:51 | |
| *** k_mouza has joined #openstack-nova | 09:56 | |
| *** macz_ has joined #openstack-nova | 09:57 | |
| gibi | sean-k-mooney, stephenfin: I finished reading the new patches of vdpa. Left some small comments and question inline. I will be off for a while between 13:00 - 16:00 CET but I will review whatever needs to be reviewed. just let me know | 10:00 |
| *** macz_ has quit IRC | 10:02 | |
| stephenfin | gibi: ack, working on a functional tests atm to prove it out before giving my final review | 10:03 |
| gibi | stephenfin: cool | 10:03 |
| *** k-s-dean has joined #openstack-nova | 10:15 | |
| bauzas | I can cycle a bit of reviews for the vdpa stuff | 10:16 |
| *** sapd1 has quit IRC | 10:21 | |
| gibi | bauzas: your help might be needed on the functional test patch as that is written by stephenfin so he cannot +2 it | 10:30 |
| *** ratailor_ has joined #openstack-nova | 10:32 | |
| sean-k-mooney | o/ | 10:33 |
| sean-k-mooney | thanks ill take a look and answer them as best i can shortly | 10:34 |
| gibi | sean-k-mooney: \o | 10:34 |
| *** ratailor has quit IRC | 10:34 | |
| sean-k-mooney | ah damb it i dropt the vdpa path in the wrong patch | 10:37 |
| *** ratailor has joined #openstack-nova | 10:37 | |
| sean-k-mooney | ya that should be in the previous one i did it at the end and did an interactive rebase to suash it in but obvioulsy picked the wong patch | 10:37 |
| *** smcginnis has joined #openstack-nova | 10:38 | |
| *** ratailor_ has quit IRC | 10:39 | |
| sean-k-mooney | gibi: while im fixing the pep8 issue in pci: Add vDPA vnic to PCI request mapping and filtering ill fix the odd indenting too | 10:40 |
| gibi | cool | 10:40 |
| sean-k-mooney | one thing im debating is shoudl i rebase this seriese on top of the pci/socket and port numa changes or wait until we get to the final functional patch which is the only one that is in conflict with those | 10:41 |
| *** jangutter has joined #openstack-nova | 10:42 | |
| gibi | I think you can wait until the functional patch | 10:43 |
| sean-k-mooney | cool that avoid needing to rebase the frist couple of patches | 10:44 |
| sean-k-mooney | lyarwood: have your devstack changes merged for the cinder issue | 10:44 |
| sean-k-mooney | the tgtadm WWN issue | 10:45 |
| *** jangutter_ has quit IRC | 10:45 | |
| lyarwood | sean-k-mooney: no, they wanted to hold off until after FF as we don't have any proof that it is causing the detach issue | 10:46 |
| * lyarwood wanted to try to reproduce again today FWIW | 10:46 | |
| sean-k-mooney | ok | 10:47 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Replace grenade and nova-grenade-multinode with grenade-multinode https://review.opendev.org/c/openstack/nova/+/778885 | 10:47 |
| lyarwood | one more change to go and I'll try to reproduce the issue in your cloud again | 10:47 |
| lyarwood | I *think* I fsck'd up earlier in the week and forgot to upgrade my tempest.conf to allow volume attached LM | 10:48 |
| lyarwood | update* | 10:48 |
| sean-k-mooney | ah. well in the last 7 days we have hit the detach issue 275 times although my current match seems to trigger 3 times on each event os closer to 90 failed jobs | 10:50 |
| sean-k-mooney | we also seam to have some other libvirt issue too that i have seen intermitently but i cant recall it now | 10:51 |
| *** slaweq has quit IRC | 11:00 | |
| *** slaweq has joined #openstack-nova | 11:02 | |
| *** jangutter_ has joined #openstack-nova | 11:06 | |
| *** jangutter_ has quit IRC | 11:07 | |
| *** jangutter_ has joined #openstack-nova | 11:08 | |
| *** jangutter has quit IRC | 11:09 | |
| *** brinzhang0 has quit IRC | 11:11 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: block_device: Use initialize APIs to refresh when reported as idempotent https://review.opendev.org/c/openstack/nova/+/720769 | 11:15 |
| *** ratailor has quit IRC | 11:16 | |
| lyarwood | elod: https://review.opendev.org/c/openstack/nova/+/780014 - thoughts on this? | 11:17 |
| *** ociuhandu_ has joined #openstack-nova | 11:18 | |
| *** ociuhandu has quit IRC | 11:21 | |
| *** ociuhandu_ has quit IRC | 11:22 | |
| elod | lyarwood: oh, sorry, I've lost it in my TODOs :S +2+W'd | 11:24 |
| lyarwood | elod: np thanks | 11:24 |
| openstackgerrit | sean mooney proposed openstack/nova stable/train: add functional regression test for bug #1888395 https://review.opendev.org/c/openstack/nova/+/759533 | 11:31 |
| openstack | bug 1888395 in OpenStack Compute (nova) train "live migration of a vm using the single port binding work flow is broken in train as a result of the introduction of sriov live migration" [High,In progress] https://launchpad.net/bugs/1888395 - Assigned to Billy Olsen (billy-olsen) | 11:31 |
| *** tesseract has quit IRC | 11:36 | |
| *** tesseract has joined #openstack-nova | 11:36 | |
| *** jamesdenton has quit IRC | 11:50 | |
| *** jamesdenton has joined #openstack-nova | 11:52 | |
| *** xinranwang has quit IRC | 11:57 | |
| stephenfin | functional tests works. hurrah | 11:58 |
| stephenfin | sean-k-mooney: are you addressing gibi's nits and mine or will I? | 11:58 |
| *** macz_ has joined #openstack-nova | 11:58 | |
| stephenfin | I don't mind. I have to push the functional test anyway | 11:59 |
| sean-k-mooney | i am yes | 11:59 |
| stephenfin | okay, great | 11:59 |
| sean-k-mooney | if you push with -R it wont rebase my stuff and i can cherry pick in your test when i push | 11:59 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Add functional test for vDPA device https://review.opendev.org/c/openstack/nova/+/780112 | 11:59 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: tests: Make mdev stubs work like vDPA https://review.opendev.org/c/openstack/nova/+/780234 | 11:59 |
| stephenfin | already done ^ | 11:59 |
| sean-k-mooney | yep | 12:00 |
| stephenfin | bauzas: gibi: ^ | 12:00 |
| gibi | ack | 12:00 |
| *** macz_ has quit IRC | 12:03 | |
| *** ociuhandu has joined #openstack-nova | 12:09 | |
| *** ociuhandu has quit IRC | 12:15 | |
| *** smcginnis has quit IRC | 12:28 | |
| sean-k-mooney | so artoms socket patch hit the block detach failure again https://review.opendev.org/c/openstack/nova/+/772779 | 12:34 |
| sean-k-mooney | this time in the gate queue | 12:34 |
| sean-k-mooney | shoudl we keep rechecking this or do we have another solution? | 12:34 |
| lyarwood | looks like there are other failures in there as well | 12:35 |
| lyarwood | nova-next failed with a ssh timeout | 12:35 |
| sean-k-mooney | yep but the same tests passed in check | 12:35 |
| sean-k-mooney | granted its mixed with other patchs in gate | 12:35 |
| lyarwood | nova-ceph-multistore failed with a volume backup failure | 12:35 |
| sean-k-mooney | but i dont think those failures are related | 12:36 |
| sean-k-mooney | i think all the failures it hit are intermitent failure in the jobs | 12:36 |
| lyarwood | nova-live-migration failed but not because of a detach issue AFAICT | 12:36 |
| lyarwood | the instance just didn't migrate | 12:36 |
| sean-k-mooney | glanceclient.exc.HTTPNotFound: HTTP 404 Not Found: No image found with ID 202b34e0-db15-437c-8a19-ab391dfcf6e0 | 12:37 |
| sean-k-mooney | although that might be a differnt test? | 12:38 |
| lyarwood | yeah I think so | 12:41 |
| lyarwood | https://zuul.opendev.org/t/openstack/build/98ef805affe54cdd96e32a57b6b87ea1/log/compute1/logs/screen-n-cpu.txt#10011 | 12:41 |
| lyarwood | that's why the LM failed | 12:41 |
| sean-k-mooney | whats annoying about those test failures in general is artoms code changes only take effect if your using pci passthough or sriov which we dont do in any gate jobs. | 12:41 |
| lyarwood | but why is that a different request-id | 12:42 |
| sean-k-mooney | libvirt.libvirtError: unable to connect to server at 'ubuntu-focal-vexxhost-ca-ymq-1-0023456556:49152': Connection refused | 12:43 |
| sean-k-mooney | tempest-LiveMigrationTest-1563426117 tempest-LiveMigrationTest-1563426117-project-admin] Could not generate host nqn: [Errno 2] No such file or directory | 12:44 |
| *** artom has joined #openstack-nova | 12:44 | |
| sean-k-mooney | https://zuul.opendev.org/t/openstack/build/98ef805affe54cdd96e32a57b6b87ea1/log/compute1/logs/screen-n-cpu.txt#10481 | 12:44 |
| sean-k-mooney | nvme gen-hostnqn | tee /etc/nvme/hostnqn | 12:45 |
| sean-k-mooney | that odd | 12:45 |
| lyarwood | yeah I think that's ignored | 12:45 |
| lyarwood | it's the os-brick connector | 12:45 |
| sean-k-mooney | why are we doing nvme things | 12:45 |
| sean-k-mooney | ya i think its ignored too but the gate cant test nvme as far as im aware | 12:46 |
| sean-k-mooney | so not sure why the nvmeof connector is trying to do anything | 12:46 |
| lyarwood | yeah the os-brick connector doesn't know ahead of time what the actual volume type is | 12:46 |
| lyarwood | so it just gathers info on all of the possible IQN/HBAs etc it can | 12:47 |
| sean-k-mooney | thats not ideal but ok | 12:47 |
| lyarwood | yeah it's a really old wrinkle in the APIs | 12:47 |
| lyarwood | ideally you'd want cinder API to tell the caller what the type is first and what connector info it needs to map the volume | 12:47 |
| sean-k-mooney | artom recheked this so im going to go back to fixing nits | 12:48 |
| sean-k-mooney | lyarwood: yep | 12:48 |
| sean-k-mooney | like with os-vif | 12:48 |
| sean-k-mooney | we use the vif-type and vnic-type to only call the correct plugin | 12:48 |
| sean-k-mooney | we never do any kind of probing | 12:48 |
| *** tkajinam has quit IRC | 12:54 | |
| *** smcginnis has joined #openstack-nova | 13:05 | |
| *** dosaboy_ is now known as dosaboy | 13:06 | |
| *** macz_ has joined #openstack-nova | 13:10 | |
| sean-k-mooney | stephenfin: by the way a possible gotch ya with pre-commit | 13:11 |
| sean-k-mooney | i dont think it runs if you do a git add during an interactive rebase and then do git rebase --continue | 13:12 |
| sean-k-mooney | you have to do git commit --amend first | 13:12 |
| bauzas | sean-k-mooney: you got a merge conflict ? | 13:14 |
| sean-k-mooney | for? | 13:14 |
| *** macz_ has quit IRC | 13:15 | |
| sean-k-mooney | im interactive rebaseing to adress the fact i removed the devpath in the wong commit and im adress all the style nits in the rest of the following patchs since they are going to be rebased as a result anyway | 13:16 |
| sean-k-mooney | with precommit if you just to rebase --continue it just does not run the commit hook i think | 13:16 |
| sean-k-mooney | since it technially not a commit | 13:17 |
| sean-k-mooney | so if you want precommit to run when doing interactive rebases and your editing files then make sure to comit before moving on | 13:17 |
| *** tesseract has quit IRC | 13:19 | |
| sean-k-mooney | bauzas: if you are asking will there be a merge conflict then yes | 13:19 |
| *** tesseract has joined #openstack-nova | 13:19 | |
| sean-k-mooney | a very minor one with the pci socket policy patch | 13:19 |
| sean-k-mooney | its a one line conflict in a test | 13:19 |
| openstackgerrit | sean mooney proposed openstack/nova master: libvirt: Add vDPA nodedev parsing https://review.opendev.org/c/openstack/nova/+/770533 | 13:21 |
| openstackgerrit | sean mooney proposed openstack/nova master: libvirt: Add guest generation for vDPA https://review.opendev.org/c/openstack/nova/+/770532 | 13:21 |
| openstackgerrit | sean mooney proposed openstack/nova master: pci: Add vDPA vnic to PCI request mapping and filtering https://review.opendev.org/c/openstack/nova/+/778350 | 13:21 |
| openstackgerrit | sean mooney proposed openstack/nova master: tests: Add functional test for vDPA device https://review.opendev.org/c/openstack/nova/+/780112 | 13:21 |
| openstackgerrit | sean mooney proposed openstack/nova master: WIP: tests: Make mdev stubs work like vDPA https://review.opendev.org/c/openstack/nova/+/780234 | 13:21 |
| sean-k-mooney | stephenfin: that should adress all the nits too | 13:21 |
| lyarwood | I can't recall ever seeing this written down anywhere but do we have a policy against passing context around, if only to improve logging by ensuring a request-id is logged alongside each message? | 13:22 |
| lyarwood | I'm trying to debug the volume detach failure and am getting pissed with the lack of context in some places | 13:23 |
| lyarwood | I've added instance=instance to all of the LOG calls in the volume drivers within libvirt now | 13:23 |
| sean-k-mooney | i personally try not to pass it if i can avoid it | 13:23 |
| lyarwood | but as we don't pass context down into these there's no request-id | 13:23 |
| sean-k-mooney | but i almost never use request id | 13:23 |
| lyarwood | yeah, I'm a heavy heavy user of it now | 13:24 |
| lyarwood | and while I get that it's pointless if it isn't used | 13:24 |
| lyarwood | the request-id is soooooooo useful | 13:24 |
| sean-k-mooney | if you have the instance uuid that normally enough | 13:24 |
| sean-k-mooney | it can be yes | 13:24 |
| lyarwood | you just end up making assumptions I find | 13:25 |
| sean-k-mooney | but unless we can come up with a better way then pass it to every function i would prefer not to | 13:25 |
| lyarwood | I'll have to read the oslo.log code again to recall what it actually looks for | 13:25 |
| lyarwood | it might look for other things aside from context | 13:25 |
| artom | lyarwood, I thought the request ID was stashed in a thread global or something? At least that's what I was told... | 13:25 |
| sean-k-mooney | artom: i dont think so | 13:26 |
| lyarwood | hmm I thought oslo.log pulled it out of context tbh | 13:26 |
| sean-k-mooney | that would not work well with eventlests | 13:26 |
| artom | *shrug* I have to do the dad taxi thing anyways | 13:26 |
| sean-k-mooney | unless you mean a green thread local variable of some kind | 13:26 |
| sean-k-mooney | bu it i still would assume its not form that | 13:27 |
| lyarwood | oh it's both | 13:27 |
| lyarwood | https://github.com/openstack/oslo.log/blob/51324b276a8a5f69847d3a17fa89924dabad4759/oslo_log/formatters.py#L61-L78 | 13:27 |
| *** ociuhandu has joined #openstack-nova | 13:28 | |
| lyarwood | ah right cool | 13:29 |
| lyarwood | okay so passing context isn't required | 13:29 |
| * lyarwood wonders why it wasn't logged when calling disconnect_volume in that case | 13:29 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Ensure all volume drivers log the instance whenever possible https://review.opendev.org/c/openstack/nova/+/780260 | 13:30 |
| sean-k-mooney | https://github.com/openstack/oslo.context/blob/0d02866365bc8b779aef9ebd0b79a52c96ae40e5/oslo_context/context.py#L508-L513 | 13:30 |
| sean-k-mooney | so it can retrun None | 13:31 |
| sean-k-mooney | _request_store is a thread local | 13:31 |
| sean-k-mooney | https://github.com/openstack/oslo.context/blob/0d02866365bc8b779aef9ebd0b79a52c96ae40e5/oslo_context/context.py#L40 | 13:31 |
| sean-k-mooney | so if the green thread resoumes on a different thread becaue we are usign a thread pool it might be empty | 13:32 |
| lyarwood | kk makes sense, both connect and disconnect wait behind a lock | 13:32 |
| lyarwood | so I assume that's where we end up losing it | 13:32 |
| lyarwood | ew | 13:32 |
| sean-k-mooney | do i want to know what you just saw | 13:33 |
| lyarwood | that was more at the logging changing because of thread local storage | 13:33 |
| lyarwood | I'm looking at yet another example failure of the detach thing | 13:34 |
| lyarwood | another test attaches a volume | 13:34 |
| lyarwood | the failing test attaches a volume | 13:34 |
| lyarwood | the other test detaches | 13:34 |
| lyarwood | and then our failing test attempts to detach | 13:34 |
| sean-k-mooney | thats also ew :) | 13:34 |
| lyarwood | well it's all locked and serialised so that's fine | 13:35 |
| lyarwood | but the failing test just times out trying to detach the device | 13:35 |
| sean-k-mooney | can we just chnage this locally in nova to workaround the issue | 13:35 |
| lyarwood | there's some stuff in syslog | 13:35 |
| lyarwood | but I can't get my head around it tbh | 13:35 |
| sean-k-mooney | you still think this is due to conflicting WWNs right | 13:36 |
| lyarwood | yeah I don't think that's helping but I can't find a smoking gun here | 13:37 |
| *** jangutter has joined #openstack-nova | 13:45 | |
| *** psachin has quit IRC | 13:45 | |
| belmoreira | hi, we are live migrating thousands of instances because hardware retirement and hit an old issue :) this was discussed in the past for instance creation, but now I'm facing it at live migration, when the migration allocation is created. https://bugs.launchpad.net/nova/+bug/1918419 | 13:47 |
| openstack | Launchpad bug 1918419 in OpenStack Compute (nova) "vCPU resource max_unit is hardcoded" [Undecided,New] | 13:47 |
| *** jangutter_ has quit IRC | 13:48 | |
| sean-k-mooney | belmoreira: it would be incorrect to factor in allocation ration when setting max_unit | 13:48 |
| belmoreira | I feel that for the vcpu particular case the operator could take the responsibility to defined the max_unit value. | 13:48 |
| sean-k-mooney | i dont think i agree | 13:49 |
| sean-k-mooney | you might be abel to use provider.yaml to cahnge it | 13:49 |
| sean-k-mooney | but its not valid for a vm to oversubsribe against itself | 13:49 |
| sean-k-mooney | so the max_unit should never exceed total | 13:50 |
| sean-k-mooney | if you are disabling hyperthread thats havlfing your total cpus so the max_unit should also be reduced | 13:50 |
| belmoreira | "its not valid for a vm to oversubsribe against itself" I agree with that | 13:50 |
| sean-k-mooney | max_unit in placment is there to prevent that | 13:51 |
| sean-k-mooney | belmoreira: the correct approch in your case would be to resize the vms before the migration | 13:51 |
| sean-k-mooney | form the 32 core flavor to 16 i assume | 13:51 |
| sean-k-mooney | or 32 but across 2 numa nodes | 13:52 |
| sean-k-mooney | likely depending on the vm | 13:52 |
| *** smcginnis has quit IRC | 13:52 | |
| belmoreira | of course in the general I agree with you. And it's a good default. What I'm talking about is to give the operator the freedom to change this sensible default for very particular operations. | 13:52 |
| sean-k-mooney | that the thing its all or nothing | 13:53 |
| sean-k-mooney | if you change the max_unit in the placment inventory | 13:53 |
| sean-k-mooney | it will apply to all operations | 13:53 |
| sean-k-mooney | which could allow vms to boot that were over subscibing againt themselves | 13:53 |
| sean-k-mooney | we dont suport setting max_unit for indivigual resouce in the allocation candiates request | 13:54 |
| sean-k-mooney | belmoreira: if you realy need too you can use https://specs.openstack.org/openstack/nova-specs/specs/ussuri/approved/provider-config-file.html | 13:55 |
| sean-k-mooney | to orverride the max_unit | 13:55 |
| sean-k-mooney | assuming you have at least ussuri | 13:55 |
| belmoreira | I'm not familiar with that spec. Let me have a look. (still in Stein for Nova) | 13:57 |
| sean-k-mooney | you would basically do | 13:57 |
| sean-k-mooney | meta: | 13:57 |
| sean-k-mooney | schema_version: 1.0 | 13:57 |
| sean-k-mooney | providers: | 13:57 |
| sean-k-mooney | # List of dicts | 13:57 |
| sean-k-mooney | - identification: | 13:57 |
| sean-k-mooney | uuid: $COMPUTE_NODE | 13:57 |
| sean-k-mooney | inventories: | 13:57 |
| sean-k-mooney | additional: | 13:57 |
| sean-k-mooney | VCPU: | 13:57 |
| sean-k-mooney | max_unit: 32 | 13:57 |
| *** spatel_ has joined #openstack-nova | 13:58 | |
| belmoreira | If we can do that in Ussuri it looks good enough to me for any special case | 13:59 |
| belmoreira | thanks sean-k-mooney | 13:59 |
| sean-k-mooney | actully it wont work for your usecase | 13:59 |
| sean-k-mooney | i forgot we can only use this for CUSTOM_ resouces | 13:59 |
| *** mlavalle has joined #openstack-nova | 13:59 | |
| *** khomesh24 has quit IRC | 14:00 | |
| sean-k-mooney | we intentionally blocked overwriding sthe standard ones form the virt driver | 14:00 |
| belmoreira | my thinking on all of this is how an operator should move forward when (for whatever reason) the resource inventory changes. | 14:05 |
| sean-k-mooney | well form a downstream and i thnk upstream point of vew we do not supprot changing the number of hyper treads on a host with vms | 14:06 |
| belmoreira | For my particular case, vCPUs were cut in half, but we need to keep the oversubcribed instances running. Now to live migrate then we need "patch" the compute nodes | 14:06 |
| sean-k-mooney | the same aslo gos for numa nodes e.g. by enable cluster on die ot changeing numa per socket in the bios | 14:06 |
| sean-k-mooney | so today the only supported way it to do a resize to a differnt flavor | 14:07 |
| sean-k-mooney | belmoreira: yep i understand unfortuetly the senario your are attempting to do is not currenlty supported by nova | 14:07 |
| belmoreira | sean-k-mooney :) fair enough. I didn't think we would ever disable SMT in production nodes. | 14:07 |
| *** macz_ has joined #openstack-nova | 14:07 | |
| sean-k-mooney | the "keep vms alive requirement is really the toughest" | 14:08 |
| * sean-k-mooney messed up the quotes | 14:08 | |
| sean-k-mooney | belmoreira: for live migration we cant really change the toplogy of the guest | 14:09 |
| sean-k-mooney | we can change teh mapping the host but putting that 32 core vm on a 16 core host would not be a valid schudlign decision based on our current oversubscpiton rules | 14:10 |
| sean-k-mooney | belmoreira: actully i hate to say this but did you attempt a force live migration | 14:10 |
| sean-k-mooney | i assume that still fails because placment will block it | 14:10 |
| sean-k-mooney | when we try to update the allcoations | 14:11 |
| belmoreira | sean-k-mooney it's fine. And in most part I agree with you, I just wanted to raise this issue here because other operators may suffer from the same thing. And these are the "use cases" that we didn't think about... Like I said I never thought we would need to disable SMT in production nodes. | 14:11 |
| sean-k-mooney | it is certenly something we could put in the docs somewhwere | 14:12 |
| belmoreira | sean-k-mooney actually I didn't... but in the case is placement, so it should fail | 14:12 |
| sean-k-mooney | i think the allocation update will fail but i have never tried this | 14:12 |
| sean-k-mooney | nova with the old microversion will skip the schduler fileter if you force it | 14:13 |
| sean-k-mooney | but i think we always do the placment update | 14:13 |
| *** macz_ has quit IRC | 14:13 | |
| sean-k-mooney | belmoreira: if you want to hack around it without changing the code there is one thing you could try | 14:14 |
| sean-k-mooney | you could increase the interval taht we update placment at in the config on a node temporally so say once an hour | 14:14 |
| sean-k-mooney | and you could manully chagne the value in the placment inventory with osc-placment | 14:15 |
| sean-k-mooney | then migrate | 14:15 |
| sean-k-mooney | that will end up with an invalidly placed vm but you could force it that way | 14:15 |
| belmoreira | sean-k-mooney yeah, that would work. What i'm doing is just putting a fake value for max_int in the update placement. These nodes will be removed anyway. | 14:17 |
| sean-k-mooney | ya its basiclaly the same just via the api for those that cant hack the code direcly in prodcution | 14:17 |
| belmoreira | sean-k-mooney thanks a lot for your comments. | 14:18 |
| belmoreira | Having something in the docs may help others. I think I can move this bug forward and update the docs. | 14:18 |
| sean-k-mooney | i hope you dont mind that i marked it as invalid but you could bring it up at the ptg | 14:19 |
| sean-k-mooney | or in a nova team meeting/mailing list to get more input form others | 14:19 |
| belmoreira | sean-k-mooney I think having something in the docs is reasonable | 14:20 |
| sean-k-mooney | yep i agree if you wanted to convert that to a docs bug i think it would be good | 14:21 |
| *** jamesdenton has quit IRC | 14:24 | |
| belmoreira | I need some guidance... should I mentioned in the bug that we discussed this and we agreed that a docs change could be enough? or this bug needs to be submitted somewhere else? | 14:24 |
| *** jamesdenton has joined #openstack-nova | 14:24 | |
| sean-k-mooney | you can link to the irc convo | 14:24 |
| sean-k-mooney | http://eavesdrop.openstack.org/irclogs/%23openstack-nova/latest.log.html#t2021-03-12T13:47:17 | 14:25 |
| sean-k-mooney | then we can triage it as valid | 14:25 |
| belmoreira | ok thanks a lot | 14:25 |
| sean-k-mooney | so you can reuse the same bug if you like althotuhg you might want to also update the titile to refelct its a docs change now | 14:26 |
| *** smcginnis has joined #openstack-nova | 14:28 | |
| belmoreira | will do | 14:28 |
| sean-k-mooney | stephenfin: im going to put my func test for the api block in a different class. they way you are currently mocking the vdpa devices is more complex then i would like and since i dont actully need them since im blocking these ops at the api level im going to skip mocking them for now | 14:32 |
| sean-k-mooney | stephenfin: when the op moves form unsupported to supported ill need the vdpa devices mocked but for these negitive test i dont | 14:33 |
| sean-k-mooney | that sound ok to you | 14:33 |
| *** lpetrut has joined #openstack-nova | 14:35 | |
| sean-k-mooney | actully i might need to mock some fo them they way you are but ill cross that bridge when i come to it | 14:35 |
| sean-k-mooney | i guess i could do api unit tests instead of functional too. | 14:36 |
| stephenfin | sean-k-mooney: go for it | 14:37 |
| stephenfin | though I take offence to the suggestion it's too complex :-P | 14:37 |
| stephenfin | (tbf, I actually thought it was reasonably understandable) | 14:37 |
| sean-k-mooney | well i cant just do fakelibvirt.HostPCIDevicesInfo(num_pfs=1, num_vf=0, num_vdpa=2) | 14:38 |
| sean-k-mooney | like i made work in the unit tests | 14:38 |
| sean-k-mooney | the fact you are manually creating the devices in the test is why is complex | 14:38 |
| stephenfin | oh, well it's easy to add that | 14:39 |
| stephenfin | I just didn't need it since I'd a single test | 14:39 |
| sean-k-mooney | yep that the only thing i was unhappy with really | 14:39 |
| *** whoami-rajat_ is now known as whoami-rajat | 14:40 | |
| sean-k-mooney | well i didnt review them too closely so there mihgt be something else but that made me go ill look at this when i have more time | 14:41 |
| * gibi is reviewing vdpa | 14:58 | |
| *** artom has quit IRC | 15:04 | |
| openstackgerrit | Merged openstack/nova stable/train: [stable-only] gate: Pin CEPH_RELEASE to nautilus in LM hook https://review.opendev.org/c/openstack/nova/+/780014 | 15:04 |
| *** k_mouza has quit IRC | 15:13 | |
| *** k_mouza has joined #openstack-nova | 15:14 | |
| *** artom has joined #openstack-nova | 15:16 | |
| *** lpetrut has quit IRC | 15:16 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/stein: [stable-only] gate: Pin CEPH_RELEASE to nautilus in LM hook https://review.opendev.org/c/openstack/nova/+/780277 | 15:17 |
| elod | gibi lyarwood : i've uploaded the release patches: https://review.opendev.org/q/project:openstack/releases+status:open+intopic:nova | 15:17 |
| gibi | will check soon | 15:18 |
| elod | thx o/ let me know if you disagree with the minor version bump | 15:19 |
| *** sapd1 has joined #openstack-nova | 15:24 | |
| *** ociuhandu has quit IRC | 15:32 | |
| lbragstad | dansmith opened up a bug against nova to document what we found https://bugs.launchpad.net/nova/+bug/1918945 | 15:33 |
| openstack | Launchpad bug 1918945 in OpenStack Compute (nova) "Nova API fails with 500s when called with non-project-scoped keystone tokens" [Undecided,New] | 15:33 |
| dansmith | lbragstad: sweet | 15:33 |
| *** macz_ has joined #openstack-nova | 15:35 | |
| *** macz_ has quit IRC | 15:36 | |
| sean-k-mooney | lbragstad: that is kind fo expected depending on what you are calling | 15:37 |
| *** macz_ has joined #openstack-nova | 15:37 | |
| sean-k-mooney | lbragstad: server create for example should not work wit | 15:37 |
| sean-k-mooney | domain or system tokens | 15:37 |
| lbragstad | sean-k-mooney yeah - i don't expect it to work | 15:37 |
| lbragstad | i just didn't expect a 500 | 15:37 |
| dansmith | sean-k-mooney: it's failing in object field validation, | 15:37 |
| sean-k-mooney | it should be a 400 or 403 | 15:37 |
| dansmith | which is...not the right place :) | 15:37 |
| sean-k-mooney | yep agreed | 15:38 |
| sean-k-mooney | we should be handeling it in the policy check level or similar in the api | 15:38 |
| dansmith | yeah I think we probably need a decorator on anything that might create resources to @require_project or something | 15:39 |
| dansmith | because I think even with something like instance shutdown, we might go to write an instance action record and fail if we have no project :( | 15:39 |
| dansmith | I haven't looked at that yet, but.. there's likely a lot of potential for those types of things | 15:39 |
| *** ociuhandu has joined #openstack-nova | 15:47 | |
| gibi | stephenfin: I have couple of things in the func test for the vdpa https://review.opendev.org/c/openstack/nova/+/780112 | 15:50 |
| stephenfin | good timing - I was just fixing the test failure | 15:50 |
| *** jangutter has quit IRC | 15:51 | |
| gibi | ohh, I only run the vdpa test locally not the whole suite | 15:51 |
| gibi | good that we have the CI :) | 15:51 |
| *** jangutter has joined #openstack-nova | 15:51 | |
| stephenfin | yeah, me too /o\ | 15:52 |
| gibi | sean-k-mooney: I will look at the ops blocking patch before I go to bed today. Now I have to step out before the curfew | 15:52 |
| sean-k-mooney | gibi: im still working on it so it may or may not be ready but hopefully will | 15:53 |
| sean-k-mooney | ill push what i have before the end of the day but might need more time to actully get test for all of them | 15:53 |
| *** ociuhandu has quit IRC | 15:53 | |
| sean-k-mooney | dansmith: we proably should be blcoking all isntace actions | 15:54 |
| sean-k-mooney | at least for now | 15:54 |
| dansmith | sean-k-mooney: agreed | 15:54 |
| dansmith | sean-k-mooney: kinda makes me wonder what the point of system-admin is in a lot of cases, if they can't shutdown or migrate instances | 15:55 |
| openstackgerrit | Sylvain Bauza proposed openstack/nova master: Bump the Compute RPC API to version 6.0 https://review.opendev.org/c/openstack/nova/+/761452 | 15:55 |
| sean-k-mooney | if we actully add support for isntace been own by a domain or somethin later then cool but for now even our logging assuems user and project id | 15:55 |
| dansmith | I guess it's useful for aggregate type things | 15:55 |
| bauzas | dansmith: just provided a new rev for your nits | 15:55 |
| bauzas | and will work on a new rev for trying to only accept >=5.12 on Monday | 15:56 |
| sean-k-mooney | dansmith: we might be able to support some of them but things like volumn attach might be weried | 15:56 |
| dansmith | bauzas: yeah, was replying when I noticed... | 15:56 |
| gibi | sean-k-mooney: ack | 15:56 |
| sean-k-mooney | well actully not volumn atach snap shot is a better example | 15:56 |
| dansmith | sean-k-mooney: yeah, I mean all of neutron would have to support this as well | 15:56 |
| sean-k-mooney | if a system-admin creates a snapshot who would own it | 15:57 |
| dansmith | sean-k-mooney: yep | 15:57 |
| dansmith | sean-k-mooney: the slop becomes slippery quite fast | 15:57 |
| dansmith | *slope | 15:57 |
| *** k_mouza has quit IRC | 15:58 | |
| sean-k-mooney | do we have a topic for the ptg for outstanding RBAC tasks | 15:59 |
| *** dklyle has joined #openstack-nova | 15:59 | |
| sean-k-mooney | or do you think we will punt on domain users other thne fixing where it fails in Xena | 15:59 |
| sean-k-mooney | or system for that mater | 15:59 |
| *** LinPeiWen has quit IRC | 16:00 | |
| dansmith | I dunno, I got the impression from lbragstad that domain users are not so widely used and maybe don't have a bright future | 16:01 |
| sean-k-mooney | we would obviously need a spec to alter the api to support them but not sure we coudl do that without collaberating with other peoject too to line everything up | 16:01 |
| sean-k-mooney | dansmith: i think only keystone has support for them today | 16:01 |
| dansmith | yeah I kinda .. don't want to do that | 16:01 |
| dansmith | coming up with a strategy for what to do if a sysadmin does an instance migrate is one thing, | 16:02 |
| sean-k-mooney | but no one else | 16:02 |
| gmann | yah domain is in keystone noly afaik | 16:02 |
| lbragstad | keystone uses them - but it's unclear if other projects will fully adopt them | 16:02 |
| dansmith | but making sysadmin able to create and snapshot instances, not so much | 16:02 |
| sean-k-mooney | if i was to summerise. operation that create/consume reousces proably are not easy/desireable to support but other operations that change state of exising resouce may be ok | 16:03 |
| sean-k-mooney | for system admin that is | 16:04 |
| dansmith | sean-k-mooney: as a future goal, yeah.. as a short-term bug, probably need to just validate and reject anything related to instances | 16:04 |
| dansmith | aggregates are fine, service actions are okay, etc | 16:05 |
| sean-k-mooney | gmann:lbragstad: any of that ^ shocking or concerning to you ? | 16:05 |
| sean-k-mooney | or was that also what ye were expecting. its more or less where my mind is at too | 16:06 |
| lbragstad | sorry - catching up | 16:06 |
| sean-k-mooney | not that i have spent that much time thinking about it | 16:06 |
| lbragstad | ok - so domains | 16:06 |
| *** ociuhandu has joined #openstack-nova | 16:07 | |
| lbragstad | since domains are containers of projects, i can see a case where calling GET /v2.0/servers with domain-scoped token would yield all servers across all projects within that domain... but, | 16:07 |
| dansmith | sure | 16:07 |
| lbragstad | you can also tell keystone to inherit a domain role assignment to all containing projects | 16:07 |
| gmann | but system admin can also do same right? | 16:07 |
| lbragstad | in which case, you do that and it works today | 16:07 |
| lbragstad | and then nova doesn't have to fetch a hierarchy of projects from keystone - at least not right now | 16:08 |
| dansmith | not sure how that will yield nova showing you all instances in your sub-projects, | 16:09 |
| dansmith | because we would have to filter for multiples | 16:09 |
| dansmith | we could do it (hence the sure) but I think we'd need the hierarchy | 16:09 |
| lbragstad | right - i think so, too | 16:09 |
| lbragstad | which nova doesn't support today | 16:10 |
| lbragstad | afaict | 16:10 |
| dansmith | right | 16:10 |
| lbragstad | and that might just seem like a lot of extra processing | 16:10 |
| dansmith | you said "works today" so wanted to clarify | 16:10 |
| dansmith | yes | 16:10 |
| sean-k-mooney | did we talk about keystone midelware portentaly being able to provide us with the set of projects | 16:10 |
| lbragstad | what i meant there was that that same domain-admin could have an inherited role assignment on all containing project (this works today in keystone) | 16:10 |
| lbragstad | allowing them to get project-scoped tokens for each project within their domain | 16:11 |
| dansmith | domain admin is really a side thing though, right? unrelated to system_admin and no current focus on supporting it | 16:11 |
| lbragstad | dansmith correct - imho domain-support is on the back-back-burner | 16:11 |
| *** k_mouza has joined #openstack-nova | 16:11 | |
| sean-k-mooney | so with keystone domains today you can have full admin on a subset of a cloud right and create falvors and such | 16:11 |
| dansmith | ack, so back to sean-k-mooney's question about how to handle system admin | 16:11 |
| gmann | yeah, i am also not so clear on use case of it | 16:12 |
| lbragstad | so - so for system-scope, here are my thoughts | 16:12 |
| lbragstad | i was open to leaving it up to the service to determine if they allow system-users to create resources within projects - but it would absolutely require an alternative method for supplying the project_id | 16:13 |
| lbragstad | (e.g., like what glance does with the owner property in the request body) | 16:13 |
| dansmith | I just don't think providing a project_id for every operation is reasonable | 16:13 |
| dansmith | an *alternate* project_id I mean | 16:14 |
| sean-k-mooney | neutorn added suport for project_id for networks and some other reseouce i htink | 16:14 |
| lbragstad | my firm opinion is that i think it would be a bad idea if nova, for example, started using fake project ids to continue allowing system-administrators to create servers on behalf of other users | 16:14 |
| gmann | yeah, we required that for create server on specific host use case but that could be achieving by opening the policy for get hypervisors and project admin or so do it | 16:15 |
| gmann | lbragstad: is there any plan to support system-ids in system scope token instead of 'all' ? i remember that was original proposal for system scope. | 16:15 |
| sean-k-mooney | i could maybe see this being done via midelwhere. where keystone alloed a system-admin toek to be create with a project_id | 16:15 |
| lbragstad | or - if servers could be created under a project called 'system' or something liek that | 16:15 |
| gmann | or that was dropped ? | 16:15 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/rocky: [stable-only] gate: Pin CEPH_RELEASE to nautilus in LM hook https://review.opendev.org/c/openstack/nova/+/780286 | 16:15 |
| sean-k-mooney | and then we would use that in nova as if it was a poject token | 16:15 |
| lbragstad | gmann not dropped - just not enough time to implement it | 16:16 |
| dansmith | maybe this is a dumb question, but if I have sysadmin powers and I want to create an instance for someone, can't I get a project token for that project before I create it? | 16:16 |
| gmann | i see | 16:16 |
| sean-k-mooney | lbragstad: basically allow one token type to issuee a lesser scoped token | 16:16 |
| lbragstad | dansmith yes - exactly | 16:16 |
| gmann | dansmith: yes | 16:16 |
| gmann | same way for project user, | 16:16 |
| dansmith | lbragstad: okay so system-scoped tokens need not be able to create resources directly right? | 16:16 |
| lbragstad | if you're an admin - you have access to all of keystone's role assignment API | 16:16 |
| gmann | if they want some system info in their operation then get system token or so.. | 16:16 |
| sean-k-mooney | lbragstad: can you do it with out updateing asignemnt | 16:17 |
| sean-k-mooney | lbragstad: e.g. can i use a system admin credential to issue a proejct scoped token directly | 16:17 |
| *** ociuhandu has quit IRC | 16:17 | |
| sean-k-mooney | without assing a rule to myslef in the project | 16:17 |
| lbragstad | dansmith correct - i think we'd really need to think through the ramifications of doing that consistently if we decided to open that up | 16:17 |
| sean-k-mooney | *role | 16:17 |
| lbragstad | sean-k-mooney no | 16:17 |
| sean-k-mooney | so that would be a non invasive way to do it | 16:18 |
| lbragstad | but it would bypass keystone's delegation mechanism | 16:18 |
| sean-k-mooney | just adding proejct member and create a project toeknb does not feel like an improvment over what we have today | 16:18 |
| dansmith | lbragstad: I would much rather make sure system admins (the people) can get a token to do what they need in a project, vs make every operation take a project (i.e. osc server stop --project $foo) | 16:18 |
| lbragstad | which may have adverse side-effects on auditing | 16:18 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: [stable-only] gate: Pin CEPH_RELEASE to nautilus in LM hook https://review.opendev.org/c/openstack/nova/+/780287 | 16:19 |
| lyarwood | elod: ack sorry irc dropped, looking now | 16:19 |
| sean-k-mooney | lbragstad: well im thinki of it kind of like app crentials | 16:19 |
| lyarwood | elod / melwitt ; https://review.opendev.org/q/I1d029ebe78b16ed2d4345201b515baf3701533d5 - btw that should sort the remaining ceph jobs out | 16:19 |
| sean-k-mooney | i have a super set of permision acroos all doemains/proejct as a system admin | 16:19 |
| lbragstad | sean-k-mooney application credentials are currently tied directly to projects and by default prevent you from using them to bypass authorization | 16:20 |
| sean-k-mooney | so i should be able to create a scoped token of a different type | 16:20 |
| sean-k-mooney | lbragstad: this would not be a bypsss though | 16:20 |
| sean-k-mooney | it would be a explict keystoen feature to allow higer privladge tokens ot issue a lower coped token | 16:20 |
| sean-k-mooney | but it sound like that cant be done with keystone current delegation design | 16:21 |
| dansmith | sounds like we should figure out what we need to do for W to not return 500 and then have a video conf about this right? | 16:21 |
| dansmith | i.e. ptg | 16:21 |
| sean-k-mooney | yep proably or wait for our PMs to say we need to support it | 16:21 |
| lbragstad | yeah - introducing a new way to delegate authorization makes me nervous without thinking through it over a longer period of time | 16:22 |
| lbragstad | agree - a video call would help | 16:22 |
| dansmith | we clearly need to understand what the expected usage pattern is, how that will or won't work and go from there | 16:22 |
| gmann | yeah | 16:22 |
| lbragstad | for now though - a system-admin can explicitly give themselves access to a project and do whatever they need to | 16:23 |
| dansmith | because it seems like there's maybe some confusion here (more than just my ignorance) | 16:23 |
| sean-k-mooney | lbragstad: right but today they dont have too | 16:23 |
| sean-k-mooney | lbragstad: i do not need to be a meber of a project as a system admin to stop a vm | 16:23 |
| dansmith | lbragstad: ack, so cutting off resource create for them makes sense for the 500 case currently | 16:23 |
| lbragstad | ok- so in that case | 16:24 |
| lbragstad | if i'm an admin on project `admin` and i use my token to stop a vm in project `foo` | 16:24 |
| lbragstad | what does nova do with my `admin` project id? | 16:24 |
| sean-k-mooney | for vm stop we just need the vm uuid but if we look in the logs we likely will log the action as done by the admin project | 16:25 |
| gmann | so it will be 403 right? | 16:25 |
| sean-k-mooney | no it should be a 200 | 16:26 |
| gmann | with new default project_id is compared for all | 16:26 |
| dansmith | the action has a project_id, I'm looking to see where we get it from, but normally it's context | 16:26 |
| dansmith | because it's the person doing the action | 16:26 |
| dansmith | so that would mean system_admin has to be refused | 16:26 |
| gmann | system admin case there is no proejct id so would not be compared | 16:26 |
| dansmith | which means they can't shut down an instance | 16:27 |
| gmann | i mean system admin power for all projects | 16:27 |
| gmann | they can | 16:27 |
| sean-k-mooney | im going to chefck it now | 16:27 |
| gmann | i mean with our current default policy, system admin can stop any instance, project member/admin can stop their own instance | 16:28 |
| dansmith | https://github.com/openstack/nova/blob/ab07507e5cfce6232fef373d07ff92ea704541da/nova/objects/instance_action.py#L56-L63 | 16:28 |
| sean-k-mooney | gmann: that is what im expecting | 16:28 |
| dansmith | gmann: they can't | 16:28 |
| dansmith | gmann: if their project_id is going to be None, they will not be able to create an InstanceAction | 16:28 |
| dansmith | which likely will fail before we even begin the call to stop the instance | 16:29 |
| gmann | ah i see, they fail later even pass the policy... | 16:29 |
| dansmith | right | 16:30 |
| dansmith | this is what I'm saying, | 16:30 |
| dansmith | looots of stuff in nova will not allow (or break) if project_id is None, | 16:30 |
| dansmith | and fixing all of that seems silly to me | 16:30 |
| sean-k-mooney | i just did it with horzon | 16:30 |
| gmann | ah yeah, same case of "POST server" by system | 16:30 |
| dansmith | I'd rather say "no tenant-isolated resource manip if you're a system_admin with no project" | 16:30 |
| sean-k-mooney | it shut down fine | 16:30 |
| dansmith | you have to get a project-scoped token to do those things | 16:30 |
| melwitt | lyarwood++ thanks for getting that all fixed up. will go through and review | 16:30 |
| sean-k-mooney | so i create a vm on my k8s project, remove my user form that proejct and then stoped it from the admin view in horizon | 16:31 |
| lbragstad | sean-k-mooney are you using a system-scoped token though? | 16:31 |
| dansmith | sean-k-mooney: is horizon using a system-scoped token where context.project_id is None? | 16:31 |
| sean-k-mooney | no the old style ones | 16:31 |
| dansmith | because that's the new thing here | 16:32 |
| dansmith | right, so that's not the problem | 16:32 |
| gmann | I do not think Horizon use system scope token | 16:32 |
| gmann | yeah | 16:32 |
| lbragstad | correct - they're still working on it | 16:32 |
| sean-k-mooney | my point is system scoped tokens should work identiclaly to that | 16:32 |
| dansmith | they don't | 16:32 |
| sean-k-mooney | then that a problem | 16:32 |
| gmann | dansmith: i agree on not putting project_id in all such APIs request body | 16:32 |
| dansmith | sean-k-mooney: did you look at the bug? | 16:32 |
| sean-k-mooney | yes and i suspect i know why it does not work | 16:32 |
| dansmith | sean-k-mooney: it's pretty gruesomely obvious with the project_id=None being.. a big deal | 16:32 |
| dansmith | gmann: ++ | 16:33 |
| gmann | yeah, if API operate on project_id form context then system token is not useful | 16:33 |
| sean-k-mooney | yep | 16:33 |
| gmann | which is our most server APIs/action | 16:33 |
| sean-k-mooney | unless we can populate the context in some way | 16:33 |
| gmann | but some of them we might be able to make it work if we use 'id' itself instead of context | 16:34 |
| gmann | like in stop sevrer case, | 16:34 |
| gmann | except POST server, we should be able to fix those right? | 16:34 |
| dansmith | well, I suggested one way is to have a default project id in config that we use for system-scoped tokens, but even still, that's pretty gross | 16:34 |
| sean-k-mooney | i dont like that but maybe | 16:35 |
| dansmith | I don't either | 16:35 |
| gmann | yeah | 16:35 |
| sean-k-mooney | i would prefer to have a midelware level populatie it | 16:35 |
| dansmith | which is why I'd rather say that you can't dick with an instance as a system admin | 16:35 |
| dansmith | you have to get a token | 16:35 |
| sean-k-mooney | that is why i was suggesting the new token delegation flow | 16:35 |
| dansmith | sean-k-mooney: populate it with what? doesn't matter where it comes from.. if the user doesn't have a project.. you need to fake it or reject | 16:36 |
| sean-k-mooney | e.g. sudo give me a project token | 16:36 |
| dansmith | right | 16:36 |
| dansmith | that is what I think needs to happen | 16:36 |
| gmann | lbragstad: when we said system_id things, does system)id means some project id? or some new logical system ids | 16:37 |
| lbragstad | today? | 16:37 |
| lbragstad | today system means a super special string 'all' | 16:37 |
| gmann | i mean in old design when keystone thought on system_id | 16:37 |
| gmann | instead of 'all' | 16:38 |
| lbragstad | we never had the 'system' construct in the old days? | 16:38 |
| lbragstad | if i'm understanding your question | 16:38 |
| lbragstad | we just used the 'admin' role for everything and didn't evaluate tenancy | 16:38 |
| sean-k-mooney | lbragstad: i think the direction gmann was going was coudl we stash a project id in it | 16:39 |
| gmann | lbragstad: i mean if system_id is supported then what it will be like ? some new project id or a new system id generated | 16:39 |
| lbragstad | oh- ok | 16:39 |
| lbragstad | i *could* be a service ID, in the future | 16:39 |
| gmann | sean-k-mooney: i was thinking like this - context.project_id or context.system_id way ? | 16:39 |
| sean-k-mooney | lbragstad: its ment to be like compute, storage or networking | 16:39 |
| lbragstad | so - i could do something like $ openstack role add --user sean-k-mooney --system compute admin | 16:39 |
| lbragstad | right | 16:40 |
| lbragstad | so - then sean-k-mooney can manage nova, but can't add users to keystone | 16:40 |
| *** belmoreira has quit IRC | 16:40 | |
| lbragstad | or create public images in glance | 16:40 |
| lbragstad | it would allow us to implementing constrained RBAC | 16:40 |
| dansmith | does that mean system_id holds an actual project, or is system_id something else? | 16:40 |
| lbragstad | system_id could be a service ID | 16:40 |
| lbragstad | but not a project ID | 16:40 |
| dansmith | that seems bad to me, | 16:40 |
| lbragstad | the service ID seems bad? | 16:41 |
| lbragstad | or the project ID seems bad? | 16:41 |
| gmann | yeah but then we lose the concept of system vs project in nova ight? at least if we end up asking giving project token a sysmte role | 16:41 |
| dansmith | because then if I'm going to try to audit "who shut down my instance" and I query for a project matching that ID, there is no project, right? | 16:41 |
| sean-k-mooney | its really jsut a sting as proposed today | 16:41 |
| sean-k-mooney | the porject would also need to have the same string in there policy right | 16:41 |
| sean-k-mooney | e.g. we would have to tag all the compute apis as part of the compute system | 16:41 |
| lbragstad | today - the system scope policies don't understand that concept, but neither does keystone | 16:41 |
| gmann | yeah | 16:42 |
| lbragstad | again - i'm talking way down the road | 16:42 |
| lbragstad | but it might be important before we get to that point and realize projects are using system by overloading it with project information | 16:42 |
| lbragstad | dansmith yeah - in that case, you'd need to query the projects and service | 16:43 |
| lbragstad | services* | 16:43 |
| dansmith | lbragstad: well, we call it a project_id, so you just "need to know" that it could be either right? | 16:44 |
| gmann | dansmith: sean-k-mooney why we cannot remove the project_id = context.project_id from here ? https://github.com/openstack/nova/blob/ab07507e5cfce6232fef373d07ff92ea704541da/nova/objects/instance_action.py#L60 | 16:44 |
| gmann | at least that will allow action APIs to eb operable form system and keep project_id in instance same as old | 16:45 |
| lbragstad | yeah - so you're saying it could be an actual project_id or a service_id (assuming all that work actually happens in keystone) | 16:45 |
| dansmith | gmann: we have to do it everywhere | 16:45 |
| dansmith | gmann: and, it hides who did the thing | 16:45 |
| gmann | yeah, that is some work we have to do | 16:45 |
| dansmith | gmann: that object is for auditing, and if we just lie about who shut down your instance, that's... like, bad and stuff? :) | 16:45 |
| gmann | dansmith: we can add some specific field of shut_down_by= and 'system admin' if system admin does | 16:46 |
| dansmith | gmann: -2 on that :) | 16:46 |
| dansmith | all of this comes because we have project-scoped resources, and now we have this user that is in "no project" | 16:47 |
| gmann | i mean at some stage we have to consume system admin info in resources DB or so | 16:47 |
| lbragstad | would nova be opposed to a mutually exclusive group for project_id|system_id? | 16:47 |
| dansmith | which is why I think we shouldn't allow you to act on project-scoped resources with out.. some sort of project scope :) | 16:47 |
| gmann | but that shrink the system token use. | 16:48 |
| gmann | or the overall purpose | 16:48 |
| dansmith | why not just give all system admins a single project? we can still use the system-admin part to authorize, but then we have a project_id to account for things | 16:48 |
| gmann | i will say system_id we should add and then do context.project_id or context.system_id | 16:49 |
| gmann | or mutual exclusive as lbragstad mentioned before | 16:50 |
| sean-k-mooney | i think we need 1 coffee 2 a whiteborad or virtual alternitve and 3 verbal realtime comunication | 16:50 |
| dansmith | yes, I would really like to do this via voice | 16:51 |
| sean-k-mooney | and just fix the bug to fail eailer untile ^ | 16:51 |
| gmann | yeah, +1 on voice call | 16:51 |
| * lbragstad nods | 16:51 | |
| sean-k-mooney | i think the main issue is what keysotn is provideign via this mechanim is not how i woudl expect RBAC to work from other systems i have looked at | 16:52 |
| lbragstad | sean-k-mooney can you give me some example references for how you expect this to work? i'd like to dig into those | 16:52 |
| sean-k-mooney | im not sure i can directly. i was more thinkin of a set of scope privalge delegations rather then 3 indipened types of tokens | 16:54 |
| sean-k-mooney | e.g. i was still expectin git to be possibel to create a singel system token that could be used like the old admin role | 16:55 |
| sean-k-mooney | but that cant be done as far as i can see | 16:55 |
| dansmith | so in kerberos, | 16:55 |
| dansmith | I can use my same account to get a token, an admin token, and I think a token in any group I have access to right? | 16:55 |
| dansmith | it's been a long time for me, but..I think that's right | 16:55 |
| sean-k-mooney | by the way i was thinking of my litel experince with oath2 | 16:55 |
| sean-k-mooney | dansmith: i think so an di was expecting to be able to do something similar | 16:56 |
| dansmith | I would expect that if I have sysadmin powers, I should be able to effectively newgrp my way into any project to make a request as that project, without having to add myself to that project | 16:56 |
| dansmith | sean-k-mooney: right, that's how this should work IMHO | 16:56 |
| dansmith | don't act like you have no affiliation, just allow system admins to have any affiliation they want that makes sense | 16:56 |
| sean-k-mooney | basicaly i was hopeing for an "openstack token issue -type project -id <whatever>" | 16:57 |
| *** lucasagomes has quit IRC | 16:57 | |
| dansmith | EXACTLY | 16:57 |
| dansmith | EFFING EXACTLY what I've been trying to say | 16:57 |
| sean-k-mooney | then my user would still be me for audinging | 16:58 |
| dansmith | EFFING EXACTLY | 16:58 |
| sean-k-mooney | but the project in the contex would be the one that was specifed | 16:58 |
| dansmith | assuming you mean "auditing" :) | 16:58 |
| dansmith | right | 16:58 |
| sean-k-mooney | hehe yes | 16:58 |
| sean-k-mooney | seanspeak is flexible | 16:58 |
| elod | lyarwood: thank you both for checking the release patches and backporting your gate fixing patch! \o/ | 16:58 |
| gmann | but how you handle for multiple project token issue needed | 16:58 |
| dansmith | and I left my decoder ring in the other room | 16:58 |
| dansmith | gmann: not sure what you mean.. when do you ever do one thing as two projects? | 16:59 |
| gmann | i mean as system admin i want multiple project resource to operate | 16:59 |
| sean-k-mooney | in one api call? | 17:00 |
| gmann | no independent | 17:00 |
| lbragstad | gmann like listing all instances in a deployment? | 17:00 |
| dansmith | lbragstad: no that's different | 17:00 |
| gmann | and without storing or passing projct_id in request | 17:00 |
| sean-k-mooney | that we can support directly with a system_admin token | 17:00 |
| dansmith | gmann: you get a token before each call, just like you would in unix: newgrp teamA; do thing; newgrp teamB; do thing | 17:01 |
| gmann | hummm, i think let's do voice call | 17:01 |
| dansmith | ++ voice call :) | 17:01 |
| sean-k-mooney | lbragstad: anyway i hope you see where i was coming form | 17:02 |
| sean-k-mooney | better go work on the last vdpa patch | 17:02 |
| lbragstad | sean-k-mooney yep - i think i get the general idea | 17:02 |
| gmann | dansmith: sean-k-mooney lbragstad added this in ptg etherpad too (L213)- https://etherpad.opendev.org/p/nova-xena-ptg | 17:06 |
| gmann | when we can have voice call before ptg? | 17:07 |
| sean-k-mooney | i would suggest not until after RC1 | 17:08 |
| gmann | sure, | 17:09 |
| *** ociuhandu has joined #openstack-nova | 17:11 | |
| dansmith | yep, a pre-call would be good | 17:12 |
| dansmith | post-RC1 seems good | 17:12 |
| lyarwood | elod: np :) | 17:18 |
| lyarwood | elod: thanks for sorting the releases out | 17:18 |
| lyarwood | gmann: https://review.opendev.org/c/openstack/tempest/+/775630 - would you mind hitting this to unblock https://review.opendev.org/c/openstack/nova/+/708701 | 17:21 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Add functional test for vDPA device https://review.opendev.org/c/openstack/nova/+/780112 | 17:25 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: tests: Make mdev stubs work like vDPA https://review.opendev.org/c/openstack/nova/+/780234 | 17:25 |
| stephenfin | gibi: Thanks for the review. That should be good to go now ^ | 17:26 |
| *** vishalmanchanda has quit IRC | 17:26 | |
| *** jangutter has quit IRC | 17:26 | |
| *** jangutter has joined #openstack-nova | 17:27 | |
| *** tesseract has quit IRC | 17:27 | |
| gmann | lyarwood: sure checking | 17:29 |
| lyarwood | gmann: thanks | 17:33 |
| gmann | lyarwood: +A | 17:34 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Document UEFI secure boot feature https://review.opendev.org/c/openstack/nova/+/776684 | 17:42 |
| *** ociuhandu has quit IRC | 17:43 | |
| stephenfin | melwitt: lyarwood: Collective nits addressed ^ | 17:43 |
| *** ociuhandu has joined #openstack-nova | 17:44 | |
| lyarwood | stephenfin: ack LGTM | 17:48 |
| stephenfin | thanks | 17:48 |
| *** k_mouza has quit IRC | 17:48 | |
| *** ociuhandu has quit IRC | 17:49 | |
| kashyap | stephenfin: How come emulated-tpm.rst got touched? | 17:50 |
| * kashyap clicks | 17:50 | |
| kashyap | This bit https://review.opendev.org/c/openstack/nova/+/776684/9/doc/source/admin/emulated-tpm.rst#b41 | 17:50 |
| stephenfin | kashyap: I copied the "show me the trait" snippet from that for the secure boot guide and both lyarwood and melwitt suggested changes to it, so it made sense to fix the original source also | 17:51 |
| stephenfin | *suggested changes to the secure boot variant of the snippet | 17:51 |
| kashyap | stephenfin: Ah, right; just now caught up w/ Mel's comments | 17:52 |
| kashyap | stephenfin: Fair enough. I just rubbing my eyes twice to see if I something else | 17:52 |
| kashyap | s/just/was/ | 17:52 |
| * kashyap --> hungry & grumpy; back later | 17:55 | |
| *** ralonsoh has quit IRC | 17:55 | |
| *** gyee has joined #openstack-nova | 17:59 | |
| *** spatel_ has quit IRC | 18:02 | |
| *** sapd1 has quit IRC | 18:02 | |
| *** artom has quit IRC | 18:06 | |
| *** derekh has quit IRC | 18:07 | |
| *** artom has joined #openstack-nova | 18:08 | |
| *** dtantsur is now known as dtantsur|afk | 18:10 | |
| *** ociuhandu has joined #openstack-nova | 18:10 | |
| *** hamalq has joined #openstack-nova | 18:14 | |
| openstackgerrit | Merged openstack/nova master: apidb: Compact Stein database migrations https://review.opendev.org/c/openstack/nova/+/759406 | 18:20 |
| openstackgerrit | Merged openstack/nova master: pci: implement the 'socket' NUMA affinity policy https://review.opendev.org/c/openstack/nova/+/772779 | 18:21 |
| *** k-s-dean has quit IRC | 18:21 | |
| artom | \o/ | 18:22 |
| *** smcginnis has quit IRC | 18:25 | |
| *** smcginnis has joined #openstack-nova | 18:30 | |
| *** jangutter_ has joined #openstack-nova | 18:30 | |
| *** jangutter has quit IRC | 18:34 | |
| sean-k-mooney | oh finally | 18:38 |
| sean-k-mooney | stephenfin: why are you creating device of dev_type='VF' | 18:41 |
| sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/780112/4/nova/tests/functional/libvirt/test_pci_sriov_servers.py#738 | 18:41 |
| *** ociuhandu has quit IRC | 18:41 | |
| stephenfin | cos each vDPA device is associated with a VF? | 18:42 |
| sean-k-mooney | for a start that should be dev_type='type-VF' no if you wanted VFs but in this case it should be dev_type='VDPA' | 18:42 |
| stephenfin | no, that's not creating a PciDevice object | 18:42 |
| stephenfin | that's creating fake libvirt xml | 18:42 |
| sean-k-mooney | right | 18:42 |
| sean-k-mooney | we dont have VF object for the VDPA devices | 18:42 |
| *** ociuhandu has joined #openstack-nova | 18:43 | |
| stephenfin | each vdpa device has a '<parent>' element | 18:43 |
| sean-k-mooney | in the DB we will have rows of dev_type='type-PF' and dev_type='vdpa' | 18:43 |
| sean-k-mooney | stephenfin: yep but we dont have VFs in the db | 18:44 |
| stephenfin | Yeah, I know. Again, I'm not creating PciDevice objects here | 18:45 |
| stephenfin | that's just generating libvirt XML for a number of VFs | 18:45 |
| stephenfin | which we do have | 18:45 |
| sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/770533/12/nova/virt/libvirt/host.py#1236 | 18:45 |
| stephenfin | e.g. 1 PF, 4 VFS, and 1 vDPA dev per VF | 18:45 |
| sean-k-mooney | is this generating the nodedev xmls | 18:45 |
| stephenfin | yes | 18:46 |
| stephenfin | and then we let the libvirt driver generate the PciDevice objects | 18:46 |
| sean-k-mooney | ok so not the data rutrined by _get_pcidev_info or the PciDevice objects | 18:46 |
| stephenfin | no | 18:46 |
| stephenfin | so I'd expect to see see five devices in the database | 18:47 |
| stephenfin | one with type-PF, because I'm whitelisting that | 18:47 |
| stephenfin | and four with 'vdpa' | 18:47 |
| sean-k-mooney | yes | 18:47 |
| stephenfin | we can expand the test to verify that if you like. That would be a good addition | 18:47 |
| sean-k-mooney | no i just want to fiture out how to reuse | 18:47 |
| sean-k-mooney | this | 18:47 |
| *** ociuhandu has quit IRC | 18:47 | |
| sean-k-mooney | i want to create 1 PF and 1 vdpa device | 18:48 |
| sean-k-mooney | can you tell me how to do that | 18:48 |
| stephenfin | you'll want to copy lines 722 - 751 | 18:48 |
| stephenfin | drop the for loop | 18:48 |
| stephenfin | and replace idx with '1' | 18:48 |
| stephenfin | sorry, 0 | 18:48 |
| sean-k-mooney | what lines? | 18:49 |
| stephenfin | so what that'll do is generate 1 PCI device with PF caps, 1 PCI device with VF caps, and 1 vDPA device | 18:49 |
| stephenfin | https://review.opendev.org/c/openstack/nova/+/780112/4/nova/tests/functional/libvirt/test_pci_sriov_servers.py#722 | 18:49 |
| stephenfin | from 'pci_info.add_device(' | 18:49 |
| sean-k-mooney | oh your looking at a differnt version then i was | 18:50 |
| stephenfin | ah yeah, I fixed the broken tests and gibi's comments | 18:50 |
| stephenfin | so that is complete now | 18:50 |
| stephenfin | I think | 18:50 |
| sean-k-mooney | so dev_type PF is not a thing right it should be type-PF that is the value of the constant | 18:50 |
| sean-k-mooney | PF is not a valid PF | 18:50 |
| sean-k-mooney | *dev_type value | 18:51 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/objects/fields.py#L753-L761 | 18:51 |
| sean-k-mooney | which is what the pcidevice object uses https://github.com/openstack/nova/blob/63bba50f4336f4b8bf0609b0cbe4717e5d0591d7/nova/objects/pci_device.py#L112 | 18:52 |
| stephenfin | we're not generating PCI devices here | 18:52 |
| stephenfin | that's just a marker we use to tell the fakelibvirt fixture what type of fake XML to generate | 18:53 |
| stephenfin | https://review.opendev.org/c/openstack/nova/+/780112/4/nova/tests/unit/virt/libvirt/fakelibvirt.py#340 | 18:53 |
| stephenfin | we could also pass 'MDEV_TYPES', in which case it would generate XML for a PCI device with mdev capabilities | 18:53 |
| sean-k-mooney | ugh ya | 18:53 |
| sean-k-mooney | we really should not use those contansts though | 18:53 |
| sean-k-mooney | can we make them the same as the real ones | 18:53 |
| sean-k-mooney | i guess we have been using these like this for a while | 18:54 |
| stephenfin | Not really, because we'd need a way to indicate mdevs | 18:54 |
| *** andrewbonney has quit IRC | 18:54 | |
| stephenfin | for XML generation purposes | 18:54 |
| sean-k-mooney | i ocationllay modify this but its really annoying that we have PF and type- | 18:54 |
| sean-k-mooney | type-PF | 18:54 |
| sean-k-mooney | ya im going to sotp using the pci_info as the name | 18:55 |
| stephenfin | I'd rather we moved away from passing the type and instead had e.g. 'add_pci_device', 'add_pci_device_with_mdev', 'add_pci_device_with_vf' etc. helpers | 18:55 |
| stephenfin | that would be clearer IMO | 18:55 |
| sean-k-mooney | ya | 18:55 |
| stephenfin | That would be a good follow-up, but I think my changes there are already too invasive without tacking on even more | 18:56 |
| sean-k-mooney | althoguh i still want fakelibvirt.HostPCIDevicesInfo( num_pci=0, num_pfs=1, num_vfs=0, num_vdpa=2) | 18:56 |
| stephenfin | by all means, go for it. It should be relatively easy to add | 18:57 |
| stephenfin | I only had one caller so I stuck with explicit | 18:57 |
| stephenfin | if we've got more, we can make it more generic | 18:57 |
| sean-k-mooney | im finding it quite hard to follow what that test is doing if im honest | 18:58 |
| sean-k-mooney | espcailly the self.stub_out( | 18:59 |
| sean-k-mooney | 'nova.virt.libvirt.guest.Guest.create', | 18:59 |
| stephenfin | that's a pattern I copied from the SR-IOV test. I want to inspect what the XML being used to create the guest is | 18:59 |
| sean-k-mooney | is vdpa_info.add_device(f'vdpa_vdpa{idx}', idx, vf) | 18:59 |
| stephenfin | so I'm spying on it | 18:59 |
| sean-k-mooney | what makes it a vdpa device | 18:59 |
| sean-k-mooney | right but i dont think we should be doing that in this test | 19:00 |
| sean-k-mooney | or at least i dont think my test should do that | 19:00 |
| stephenfin | no, yours shouldn't. We only need it once | 19:00 |
| stephenfin | it's just to make sure everything is wired up correctly | 19:00 |
| sean-k-mooney | right i would not have put that in the basic create | 19:00 |
| sean-k-mooney | i would have put that explicy in its own test | 19:00 |
| stephenfin | I didn't want two tests that did the exact same thing and only changed what they were looking at | 19:01 |
| *** MrClayPole has quit IRC | 19:01 | |
| stephenfin | If I had another test for e.g. resizing, I wouldn't bother with this | 19:01 |
| stephenfin | I just want to make sure what we're handing off to neutron and libvirt is correct | 19:02 |
| stephenfin | i.e. black box testing, looking at only at the inputs and outputs (from the nova service to other services) | 19:02 |
| sean-k-mooney | yep i know anyway i think i have figured out what is needed an not thanks | 19:03 |
| sean-k-mooney | stephenfin: it just a case of you are asserting two thing in one test and i was trying to fiture out what was the minium i needed | 19:03 |
| *** MrClayPole has joined #openstack-nova | 19:03 | |
| stephenfin | gotcha | 19:03 |
| stephenfin | yeah, to recap you don't need the libvirt spy | 19:03 |
| stephenfin | nor do you really need to look at the neutron requests | 19:04 |
| stephenfin | or PCI device counts in the DB | 19:04 |
| stephenfin | just create a port, create a server, and then try your various and see that they're correctly rejects, I guess? | 19:04 |
| sean-k-mooney | and i should not need to pass the libvirt and qemu version explictly either | 19:04 |
| sean-k-mooney | since you are passing the default | 19:05 |
| * gibi is back for a round of review :) | 19:05 | |
| stephenfin | you will | 19:05 |
| stephenfin | the FakeLibvirtFixture defaults to the minimums | 19:05 |
| stephenfin | which are too low | 19:05 |
| sean-k-mooney | oh ok | 19:05 |
| stephenfin | sean-k-mooney: btw, the 'test_vtpm' module has examples you can use | 19:05 |
| sean-k-mooney | i though it was max | 19:05 |
| stephenfin | e.g. test_live_migrate_server | 19:06 |
| stephenfin | test_shelve_server | 19:06 |
| sean-k-mooney | i have some working | 19:06 |
| sean-k-mooney | ya | 19:06 |
| stephenfin | great :) | 19:06 |
| sean-k-mooney | ill take a look | 19:06 |
| sean-k-mooney | im trying to use/extend the integrated helpers | 19:06 |
| sean-k-mooney | so added attach/detach interface with calls to the fakenotifier to wait | 19:06 |
| sean-k-mooney | that kind of thing | 19:07 |
| sean-k-mooney | for shelve i think we already have that there | 19:07 |
| *** ociuhandu has joined #openstack-nova | 19:12 | |
| openstackgerrit | Merged openstack/nova stable/ussuri: Prevent archiving of pci_devices records because of 'instance_uuid' https://review.opendev.org/c/openstack/nova/+/760977 | 19:14 |
| gibi | stephenfin: thanks for the update of the vdpa func test, it looks good now | 19:16 |
| *** spatel has joined #openstack-nova | 19:17 | |
| *** ociuhandu has quit IRC | 19:17 | |
| sean-k-mooney | stephenfin: what did you fix between v3 and v4 | 19:22 |
| sean-k-mooney | i might need to rebase my patch to pick up the v4 changs | 19:23 |
| sean-k-mooney | ya you changed enough that i should | 19:25 |
| *** jamesdenton has quit IRC | 19:26 | |
| *** jamesdenton has joined #openstack-nova | 19:27 | |
| sean-k-mooney | one thing that is confusiton me is i am seeing different pci address then i expect to see | 19:28 |
| *** belmoreira has joined #openstack-nova | 19:41 | |
| *** zzzeek has quit IRC | 19:45 | |
| *** ociuhandu has joined #openstack-nova | 19:46 | |
| *** zul has quit IRC | 19:46 | |
| *** zzzeek has joined #openstack-nova | 19:46 | |
| *** k_mouza has joined #openstack-nova | 19:49 | |
| *** k_mouza has quit IRC | 19:54 | |
| *** ociuhandu has quit IRC | 20:01 | |
| *** ociuhandu has joined #openstack-nova | 20:01 | |
| sean-k-mooney | .... stephenfin before i rebased on v4 of you patch my 2 test passed and my first test failed | 20:04 |
| sean-k-mooney | after my first test passed and my second test failed | 20:04 |
| sean-k-mooney | and i did not notice fo the last hour and was editing the wrong thing | 20:05 |
| *** macz_ has quit IRC | 20:28 | |
| *** macz_ has joined #openstack-nova | 20:28 | |
| openstackgerrit | Merged openstack/nova master: apidb: Compact Train database migrations https://review.opendev.org/c/openstack/nova/+/771420 | 20:38 |
| lbragstad | sean-k-mooney gmann not that you need to do anything with this now, but i had to start getting ideas on paper from today's discussion - https://etherpad.opendev.org/p/consuming-system-scope | 20:47 |
| sean-k-mooney | cool ill book mark it | 20:47 |
| *** ociuhandu has quit IRC | 20:47 | |
| sean-k-mooney | one thing i tought about afer is when issuing the token we might also want to list the roles | 20:48 |
| *** ociuhandu has joined #openstack-nova | 20:48 | |
| sean-k-mooney | so you had openstack token issue --os-cloud system-admin --for-project foo | 20:49 |
| sean-k-mooney | but we might want "openstack token issue --os-cloud system-admin --for-project foo --roles project_member" | 20:49 |
| openstackgerrit | Merged openstack/nova stable/train: replace the "hide_hypervisor_id" to "hw:hide_hypervisor_id" https://review.opendev.org/c/openstack/nova/+/768736 | 20:49 |
| *** ociuhandu has quit IRC | 20:53 | |
| *** belmoreira has quit IRC | 20:54 | |
| *** zzzeek has quit IRC | 21:04 | |
| *** zzzeek has joined #openstack-nova | 21:05 | |
| *** artom has quit IRC | 21:09 | |
| *** whoami-rajat has quit IRC | 21:13 | |
| *** ociuhandu has joined #openstack-nova | 21:19 | |
| sean-k-mooney | ok i now have the block patch written jsut need to add doc and releasenote then ill push | 21:20 |
| *** ociuhandu has quit IRC | 21:23 | |
| *** smcginnis has quit IRC | 21:38 | |
| *** smcginnis has joined #openstack-nova | 21:44 | |
| openstackgerrit | sean mooney proposed openstack/nova master: block unsupported actions with vdpa. https://review.opendev.org/c/openstack/nova/+/780333 | 21:45 |
| sean-k-mooney | gibi: stephenfin ^ proably need more work but that is basicly the blocker patch | 21:45 |
| sean-k-mooney | i think i might be able to rewrite it to use the network info cache in some cases instead | 21:46 |
| *** ociuhandu has joined #openstack-nova | 21:46 | |
| sean-k-mooney | of calling neutron but that is more or less what it will look like | 21:46 |
| *** hoonetorg has joined #openstack-nova | 21:49 | |
| *** ociuhandu has quit IRC | 21:50 | |
| *** smcginnis has quit IRC | 21:53 | |
| *** ociuhandu has joined #openstack-nova | 21:54 | |
| *** ociuhandu has quit IRC | 21:59 | |
| *** ociuhandu has joined #openstack-nova | 22:05 | |
| *** ociuhandu has quit IRC | 22:09 | |
| *** ociuhandu has joined #openstack-nova | 22:22 | |
| *** ociuhandu has quit IRC | 22:36 | |
| openstackgerrit | Merged openstack/nova master: Add generate schemas tool https://review.opendev.org/c/openstack/nova/+/769796 | 22:42 |
| *** amodi has quit IRC | 23:08 | |
| *** mgariepy has quit IRC | 23:21 | |
| openstackgerrit | melanie witt proposed openstack/nova master: Add --task-log option to nova-manage db archive_deleted_rows https://review.opendev.org/c/openstack/nova/+/780395 | 23:21 |
| *** mgariepy has joined #openstack-nova | 23:34 | |
| openstackgerrit | Merged openstack/nova master: nova-next: Start testing the q35 machine type https://review.opendev.org/c/openstack/nova/+/708701 | 23:38 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!