Friday, 2021-01-22

*** jamesdenton has quit IRC00:02
brinzhangsean-k-mooney, gibi: got it, thanks, will check00:02
*** jamesdenton has joined #openstack-nova00:03
gmannlbragstad: left comment for placement policy - https://review.opendev.org/c/openstack/placement/+/760240/14/placement/tests/functional/fixtures/gabbits.py#76300:03
gmannlbragstad: if wee can run same or new tests with fixture to check if old  token still work or not when enforce_scopy is false00:04
gmannwhich is what we ship as default.00:06
*** tosky has quit IRC00:09
*** spatel has joined #openstack-nova00:13
brinzhangsean-k-mooney: the os-simple-tenant-usage action will replace with os-simple-project-usage action, the tempest check the os-simple-tenant-usage action, if I dont skip the tempest check, it seems cannot pass00:17
brinzhangsean-k-mooney: is it right?00:17
brinzhangthe nova change: https://review.opendev.org/c/openstack/nova/+/768852/7/nova/api/openstack/compute/routes.py#68900:18
*** spatel has quit IRC00:18
brinzhanggmann: I know you are an expert in tempest, can you help to check?00:20
brinzhanggmann: the tempest patch https://review.opendev.org/c/openstack/tempest/+/76912000:20
gmannbrinzhang: sure, you want to move to openstack-qa ?00:22
brinzhanggmann: of course00:24
*** swp20 has joined #openstack-nova00:30
*** rcernin has quit IRC00:32
*** brinzhang_ has joined #openstack-nova00:34
*** songwenping_ has joined #openstack-nova00:37
*** rcernin has joined #openstack-nova00:37
*** brinzhang has quit IRC00:37
*** lbragstad_ has joined #openstack-nova00:38
*** songwenping_ has quit IRC00:38
*** songwenping_ has joined #openstack-nova00:39
*** swp20 has quit IRC00:39
*** lbragstad has quit IRC00:40
gmannbrinzhang_: replied on gerrit for your query about os-simple-tenant-usage action https://review.opendev.org/c/openstack/nova/+/76885200:50
gmannbrinzhang_: we still nee old API route and controller/sample/test etc as this is still valid for old microversion00:51
gmannwe do not have microversion knowledge in routes so we can add new one with old one still there and in API method in controller we can return 404 for old API for >2.8900:53
brinzhang_gmann: greate, will check00:53
gmannwhich is nothing but this     @wsgi.Controller.api_version("2.1",  "2.89")00:54
gmannand new API method can be decorated with     @wsgi.Controller.api_version("2.90")00:55
gmannso any user requesting new API will get 404 with <2.9000:55
gmannbasically keep the old API things (test/sample etc) as it is but with capping it with microversion 2.89.00:57
brinzhang_you mean, I should add @wsgi.Controller.api_version("2.90") in the index and show api?00:58
brinzhang_s/in/on/00:59
gmannfor new API yes00:59
brinzhang_https://review.opendev.org/c/openstack/nova/+/768509/10/nova/api/openstack/compute/simple_tenant_usage.py00:59
brinzhang_gmann: I think I should add two api for 2.90, and add the max version limit the old version01:00
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for resource providers  https://review.opendev.org/c/openstack/placement/+/76024001:00
gmannlike ServerTopology API added in 2.78 https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/server_topology.py#L2601:00
brinzhang_gmann: I am not understand waht you say about the router change01:01
gmannbrinzhang_: in routes, you just need to add the new controller and url path and leave old one as it is01:01
brinzhang_ack01:02
gmannbecause old API still valid for older microversion.01:02
brinzhang_gmann: do we need to rename the tenant file?01:02
gmannwhich one?01:02
gmannthis one? https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/simple_tenant_usage.py01:03
brinzhang_this patch I add the 2.90 support https://review.opendev.org/c/openstack/nova/+/768509/10/nova/api/openstack/compute/simple_tenant_usage.py01:03
brinzhang_and https://review.opendev.org/c/openstack/nova/+/768852 renamed the tenant* to project*01:04
gmannbrinzhang_: there are two part 1. changing the 'tenant' in response etc which is 768509 this is good as you are doing, I had a quick glance01:06
brinzhang_https://review.opendev.org/c/openstack/nova/+/768509/10/nova/api/openstack/compute/simple_tenant_usage.py#29501:06
gmannand 2nd is about adding new API method which is 768852 for this we can add new file or new method using old method in existing file01:07
brinzhang_this patch I should add the @wsgi.Controller.api_version("2.1",  "2.89") in the old index and show api, and add @wsgi.Controller.api_version("2.90") to as the new api for index and show, right?01:07
gmannright01:07
brinzhang_but this change in the 76885201:08
brinzhang_as your suggestion01:08
gmann have two option here 1. if rename file then add new controller class which can reuse the old APi method 2. have complete new file with new method which can also use existing method01:08
*** _mlavalle_1 has quit IRC01:09
gmannmay be 1st option is better. you can rename file and keep old controller class in that file with all API method as @wsgi.Controller.api_version("2.1",  "2.89")01:09
gmannand new controller class with new method with  @wsgi.Controller.api_version("2.90") and add these in routes.py01:10
gmannand same way in tests also we can do the same to reuse the code01:12
brinzhang_gmann: yes, in 768852, we renamed the file, and reused the old test01:13
brinzhang_I need to keep the old action in the router and add the new action controller01:14
gmannbrinzhang_: but you are changing old API route and controller which means old API is completely gone - https://review.opendev.org/c/openstack/nova/+/768852/7/nova/api/openstack/compute/simple_project_usage.py01:14
*** dave-mccowan has quit IRC01:15
brinzhang_is it need to restort the nova/api/openstack/compute/simple_tenant_usage.py01:15
brinzhang_ file?01:15
gmannfor example: GET /os-simple-tenant-usage wil return 404 even for older microversion01:15
brinzhang_and add the nova/api/openstack/compute/simple_project_usage.py?01:15
gmannthat is one option01:16
gmannor other option is01:16
*** macz_ has quit IRC01:16
gmannrename and add SimpleProjectUsageController(SimpleTenantUsageController) as new class there01:17
gmannand in routes.py you can add new path like     ('/os-simple-project-usage', {01:18
gmann                                                                                                'GET': [simple_project_usage_controller, 'index']01:18
gmanndef index() in new controller will just call the old SimpleTenantUsageController.index() and have  @wsgi.Controller.api_version("2.90")01:19
brinzhang_gmann: ok, we will try to use the two option01:19
brinzhang_a question01:20
brinzhang_@wsgi.Controller.api_version("2.1",  "2.89") and @wsgi.Controller.api_version("2.90") for the api change, is it need to change with the router change?01:21
brinzhang_I mean keep this change in 768852 patch01:21
brinzhang_gmann: what do you think?01:21
gmannbrinzhang_: in routes.py if you rename the file then 1. for old API route, change import (from nova.api.openstack.compute import simple_tenant_usage -> from nova.api.openstack.compute import simple_project_usage )01:25
gmannand for new API add new route as separate01:25
gmannbrinzhang_: like this  http://paste.openstack.org/show/801854/01:29
*** dave-mccowan has joined #openstack-nova01:30
brinzhang_gmann: yeah, get it01:30
brinzhang_gmann: above question is which patch I shuold change the max or min version changing of the index and show usages apis?01:31
gmannbut other file like request/response sample file can exist in separate if it become very lengthy and unreadable01:32
gmannbrinzhang_: ah that is good question but tricky also.01:32
brinzhang_ok, I can seperate the route change from 76885201:33
gmannbrinzhang_: it has to be route change in 768852 and 768509 changing the response etc. so 768509 as first will not change anything for older microversion01:35
gmannso you current order of patches is good01:35
gmannthis way in 768852  we expose the new API url with new response01:36
gmannand restrict old API with 2.89 and return old response01:36
brinzhang_ok, just need to do these changes in 768852 ^01:37
brinzhang_and keep the 768509 not change01:37
*** lbragstad_ is now known as lbragstad01:37
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for aggregates  https://review.opendev.org/c/openstack/placement/+/76023501:37
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for allocations  https://review.opendev.org/c/openstack/placement/+/76023601:37
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for allocation candidates  https://review.opendev.org/c/openstack/placement/+/76023701:37
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for inventories  https://review.opendev.org/c/openstack/placement/+/76023801:37
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for resource classes  https://review.opendev.org/c/openstack/placement/+/76023901:37
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for traits  https://review.opendev.org/c/openstack/placement/+/76024101:37
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for usage  https://review.opendev.org/c/openstack/placement/+/76024201:37
lbragstadgmann ^ those should have all the bits for testing the deprecated policies01:37
gmannlbragstad: thanks. I will review tomorrow with results. basically almost good to go just need to have new test green01:39
gmannbrinzhang_: yes. for this route things. I will review the complete series tomorrow if any thing else missing01:40
brinzhang_gmann: thanks01:40
lbragstadgmann sounds good - they all pass for me locally, i'll follow up tomorrow01:40
gmannbrinzhang_: key thing is to pass the existing tests and Tempest without any change means you are not breaking anything :)01:40
gmannlbragstad: great.01:40
gmannbrinzhang_: if you have to change anything then we are introducing some backward incompatible change01:41
*** dcapone2004 has quit IRC01:41
brinzhang_gmann: yeah, the tempest change will change as your suggestion, and it you can review tomorrow01:41
gmannbrinzhang_: +101:42
brinzhang_yes, we cannt break the old test in tempest01:42
gmannI can help on Tempest change tomorrow if anything you miss of need help with.01:42
gmann*or need01:42
*** lbragstad has quit IRC01:46
gmannbrinzhang_: going away. will check the series tomorrow. happy Friday to you :)01:47
brinzhang_gmann: thanks, good night^^01:47
*** hemanth_n has joined #openstack-nova02:05
*** hamalq has quit IRC02:24
*** brinzhang_ has quit IRC02:44
*** brinzhang_ has joined #openstack-nova02:45
*** dklyle has quit IRC02:54
*** david-lyle has joined #openstack-nova02:54
brinzhang_gmann: hi are you around?03:04
brinzhang_nova/tests/functional/api_sample_tests/api_samples/os-simple-project-usage/simple-tenant-usage-get.json.tpl like this file, the sample file do need keep ing the "os-simple-tenant-usage" dir?03:05
brinzhang_s/ing/under03:05
*** rcernin has quit IRC03:08
*** ianw is now known as ianw_pto03:12
*** macz_ has joined #openstack-nova03:17
*** macz_ has quit IRC03:21
*** lbragstad has joined #openstack-nova03:25
*** rcernin has joined #openstack-nova03:33
*** gyee has quit IRC03:36
*** macz_ has joined #openstack-nova03:38
*** tbachman has quit IRC03:38
*** david-lyle has quit IRC03:38
*** zenkuro has quit IRC03:40
*** tbachman has joined #openstack-nova03:43
*** macz_ has quit IRC03:43
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for resource providers  https://review.opendev.org/c/openstack/placement/+/76024003:44
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for aggregates  https://review.opendev.org/c/openstack/placement/+/76023503:44
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for allocations  https://review.opendev.org/c/openstack/placement/+/76023603:44
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for allocation candidates  https://review.opendev.org/c/openstack/placement/+/76023703:44
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for inventories  https://review.opendev.org/c/openstack/placement/+/76023803:45
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for resource classes  https://review.opendev.org/c/openstack/placement/+/76023903:45
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for traits  https://review.opendev.org/c/openstack/placement/+/76024103:45
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for usage  https://review.opendev.org/c/openstack/placement/+/76024203:45
*** links has joined #openstack-nova03:47
*** rcernin has quit IRC03:48
*** rcernin has joined #openstack-nova04:04
*** rcernin has quit IRC04:05
*** rcernin has joined #openstack-nova04:05
*** k_mouza has joined #openstack-nova04:18
*** mkrai has joined #openstack-nova04:20
*** k_mouza has quit IRC04:22
*** rouk has joined #openstack-nova04:30
*** vishalmanchanda has joined #openstack-nova04:47
*** manuvakery1 has joined #openstack-nova04:55
*** lbragstad has quit IRC05:11
*** mkrai has quit IRC05:22
*** mkrai_ has joined #openstack-nova05:22
*** psachin has joined #openstack-nova05:22
*** LinPeiWen has joined #openstack-nova05:43
*** ratailor has joined #openstack-nova05:44
*** mkrai_ has quit IRC05:51
*** mkrai has joined #openstack-nova05:52
*** k_mouza has joined #openstack-nova06:19
*** k_mouza has quit IRC06:23
*** sapd1 has quit IRC07:07
*** rcernin has quit IRC07:25
*** rpittau|afk is now known as rpittau07:34
*** rcernin has joined #openstack-nova07:37
*** rcernin has quit IRC07:42
*** ralonsoh has joined #openstack-nova07:48
*** rcernin has joined #openstack-nova07:50
*** slaweq has joined #openstack-nova07:54
*** rcernin has quit IRC07:55
*** luksky has joined #openstack-nova08:03
*** rcernin has joined #openstack-nova08:08
*** mgoddard has quit IRC08:11
*** mgoddard has joined #openstack-nova08:11
*** andrewbonney has joined #openstack-nova08:13
*** rcernin has quit IRC08:13
*** rcernin has joined #openstack-nova08:14
*** tesseract has joined #openstack-nova08:16
*** rcernin has quit IRC08:19
gibisean-k-mooney: hi! I don't find the bp https://blueprints.launchpad.net/nova/+spec/port-scoped-sriov-numa-affinity behind https://review.opendev.org/c/openstack/nova-specs/+/765901 spec08:20
*** rcernin has joined #openstack-nova08:26
*** rcernin has quit IRC08:31
*** xek_ has joined #openstack-nova08:34
*** rcernin has joined #openstack-nova08:44
*** tosky has joined #openstack-nova08:44
*** mkrai has quit IRC08:45
*** rcernin has quit IRC08:49
*** jamesdenton has quit IRC08:49
*** jamesdenton has joined #openstack-nova08:50
*** nightmare_unreal has joined #openstack-nova08:58
*** ociuhandu has joined #openstack-nova09:02
*** rcernin has joined #openstack-nova09:08
nightmare_unreali have submitted a bug report for placement on launchpad but I found there is a storyboard for placement, should I mark it as invalid and submit again on storyboard ?09:12
*** rcernin has quit IRC09:13
*** f0o has quit IRC09:14
*** janno has quit IRC09:14
*** PrinzElvis has quit IRC09:14
*** masayukig has quit IRC09:14
*** zigo has quit IRC09:14
*** sorrison has quit IRC09:14
*** f0o has joined #openstack-nova09:15
*** janno has joined #openstack-nova09:15
*** PrinzElvis has joined #openstack-nova09:15
*** masayukig has joined #openstack-nova09:15
*** zigo has joined #openstack-nova09:15
*** sorrison has joined #openstack-nova09:15
*** masterpe has quit IRC09:17
*** rouk has quit IRC09:21
fricklernightmare_unreal: interesting question. since placement has moved back to nova governance, I'd say it should also revert to using LP, but that's mainly my aversion against sb speaking09:22
nightmare_unrealokay frickler , i will wait for some time . Someone might look at it09:23
nightmare_unrealand will let me know if it should be moved or not09:24
fricklergibi: ^^ ptl topic probably ;)09:24
*** rcernin has joined #openstack-nova09:25
*** psachin has quit IRC09:26
gibinightmare_unreal, frickler: interesting question, I would not start a move back to launchpad just because we have a single core team09:27
gibibut there could be other legitimate reasons09:27
nightmare_unrealgot it09:27
gibinightmare_unreal: link me to the bug I please09:28
nightmare_unrealgibi:  https://bugs.launchpad.net/nova/+bug/191274909:29
openstackLaunchpad bug 1912749 in OpenStack Compute (nova) "clouds.yaml does not use placement API version variable" [Undecided,New]09:29
gibithnaks09:29
*** rcernin has quit IRC09:30
*** rcernin has joined #openstack-nova09:32
fricklergibi: I think the main question would be how to integrate storyboard into your bug tracking workflows, assuring that issues created there aren't just getting ignored09:40
*** derekh has joined #openstack-nova09:48
*** k_mouza has joined #openstack-nova09:59
openstackgerritLee Yarwood proposed openstack/nova master: docs: Add reference docs for internal block device structures  https://review.opendev.org/c/openstack/nova/+/77171109:59
*** dtantsur|afk is now known as dtantsur09:59
brinzhang_gibi: good moning^10:01
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_VIR_ERR_DEVICE_MISSING  https://review.opendev.org/c/openstack/nova/+/75470210:03
*** k_mouza_ has joined #openstack-nova10:06
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_VIR_ERR_DEVICE_MISSING  https://review.opendev.org/c/openstack/nova/+/75470210:07
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Drop support for UML  https://review.opendev.org/c/openstack/nova/+/74323010:07
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Drop support for Xen  https://review.opendev.org/c/openstack/nova/+/74323110:07
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION  https://review.opendev.org/c/openstack/nova/+/75470010:07
lyarwood^ had to rebase these for the ceph gate fix and to resolve a conflict in the UML and Xen removal changes10:07
*** masterpe has joined #openstack-nova10:08
brinzhang_lyarwood: morning^10:08
*** k_mouza has quit IRC10:09
*** hemanth_n has quit IRC10:13
*** hemanth_n has joined #openstack-nova10:13
lyarwoodbrinzhang_: morning, what's up?10:14
gibifrickler: regardless of the bug tracking tool, we are a bit behind triaging placement bugs due to multiple reasons10:16
openstackgerritLee Yarwood proposed openstack/nova master: zuul: Allow encryption tests in nova-ceph-multistore again  https://review.opendev.org/c/openstack/nova/+/77183710:16
gibilyarwood: plugged back my +2s on the libvirt bump series10:20
nightmare_unrealgibi:  did you saw the bug , does it fit on placement project ?10:22
openstackgerritWenping Song proposed openstack/nova master: Replace os-simple-tenant-usage with os-simple-project-usage  https://review.opendev.org/c/openstack/nova/+/76885210:22
brinzhang_lyarwood: this morning, gmann's give me some suggestion to update he tenant-usages patch, but while I do it, I found in one file inherit the orignal class, the function in the class with @wsgi.Controller.api_version limit the request version10:22
*** jangutter_ has quit IRC10:22
gibinightmare_unreal: I opened in a browser tab, but haven't spent time on it yet10:23
nightmare_unrealgibi: ACK :)10:23
gibinightmare_unreal: I will try to get to it10:23
nightmare_unrealthanks10:23
brinzhang_but when I test the case, I cannot reqire the 2.90 index or show api in https://review.opendev.org/c/openstack/nova/+/768852/8/nova/api/openstack/compute/simple_project_usage.py#40810:24
brinzhang_lyarwood: the request always request the api in 2.40~2.89, where is wrong?10:25
brinzhang_lyarwood: that case in https://review.opendev.org/c/openstack/nova/+/768852/8/nova/tests/unit/api/openstack/compute/test_simple_project_usage.py#58610:27
brinzhang_lyarwood: I am not sure what I missed, maybe the inherit function is not right10:28
lyarwoodbrinzhang_: so if a 2.90 request isn't being made then that's an issue with the test not the api code10:29
*** yonglihe has quit IRC10:29
lyarwoodbrinzhang_: and looking at the test I can't see anything that should stop it, I've never really touched this test code however so it might be something weird like an extra var required somewhere10:29
brinzhang_lyarwood: I am not found these test cases where need to change, I have spent almost 3 hours on this :(10:31
* lyarwood pulls down the change10:32
*** k_mouza has joined #openstack-nova10:40
*** k_mouza_ has quit IRC10:44
lyarwoodbrinzhang_: yeah I'm not sure, I thought you had to add a reference to the controller in the test but there's more missing10:51
lyarwoodbrinzhang_: I'll try to play around with it later if you're still stuck10:51
* lyarwood switches context to some downstream stuff10:51
brinzhang_lyarwood: thanks, I would like gmann can have a check today10:52
brinzhang_after dinner I will try to trace it10:53
brinzhang_gmann: update the project usage patach follow your suggestion, pls help me check the question10:54
brinzhang_gmann: the case test run failed https://review.opendev.org/c/openstack/nova/+/768852/8/nova/tests/unit/api/openstack/compute/test_simple_project_usage.py#58610:54
*** hemanth_n has quit IRC10:57
*** hemanth_n has joined #openstack-nova10:58
*** hoonetorg has quit IRC11:10
*** mgoddard has quit IRC11:30
*** rcernin has quit IRC11:31
openstackgerritStephen Finucane proposed openstack/placement master: WIP: Allow project readers to see project-specific usages  https://review.opendev.org/c/openstack/placement/+/77196411:36
*** rcernin has joined #openstack-nova11:38
*** zenkuro has joined #openstack-nova11:54
*** rcernin has quit IRC12:00
stephenfinlyarwood: Are you planning to review https://review.opendev.org/c/openstack/nova/+/743231/10 and https://review.opendev.org/c/openstack/nova/+/743230 or do you need to ask someone else to do it?12:01
stephenfinI just reviewed them myself, but while it's been a while I still wrote the bulk of it so probably shouldn't give the sign off :-\12:02
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Remove 'hypervisor_version' from 'libvirt_info'  https://review.opendev.org/c/openstack/nova/+/74419912:02
lyarwoodstephenfin: I've made changes so it might be better to get someone else to approve, I'll review anyway and +112:03
stephenfinMakes sense. Maybe bauzas? ^12:03
* stephenfin is just lining up tedious reviews for bauzas this week :)12:04
lyarwoodas a service12:04
lyarwood:D12:04
bauzasstephenfin: at lunch :)12:09
bauzasstephenfin: ping me at 1.30pm your time and I'll look at them ;)12:10
nightmare_unrealquestion : where does osc reads clouds.yaml file, can't seem to figure out :/ .12:21
*** ratailor has quit IRC12:27
*** rcernin has joined #openstack-nova12:27
*** iurygregory_ has joined #openstack-nova12:28
*** iurygregory has quit IRC12:28
*** iurygregory_ is now known as iurygregory12:29
*** k_mouza has quit IRC12:43
*** k_mouza has joined #openstack-nova12:44
*** rcernin has quit IRC12:48
*** k_mouza has quit IRC12:54
*** k_mouza_ has joined #openstack-nova12:54
*** k_mouza has joined #openstack-nova12:55
*** ociuhandu has quit IRC12:55
*** ociuhandu has joined #openstack-nova12:56
*** ociuhandu has quit IRC12:58
*** k_mouza_ has quit IRC12:58
*** ociuhandu has joined #openstack-nova12:59
*** k_mouza has quit IRC13:00
*** k_mouza has joined #openstack-nova13:02
*** ociuhandu has quit IRC13:02
*** ociuhandu has joined #openstack-nova13:02
*** ociuhandu has quit IRC13:02
*** k_mouza has quit IRC13:03
*** ociuhandu has joined #openstack-nova13:03
openstackgerritElod Illes proposed openstack/nova stable/queens: libvirt: Do not reference VIR_ERR_DEVICE_MISSING when libvirt is < v4.1.0  https://review.opendev.org/c/openstack/nova/+/74736213:03
*** k_mouza has joined #openstack-nova13:05
openstackgerritElod Illes proposed openstack/nova stable/queens: libvirt: Do not reference VIR_ERR_DEVICE_MISSING when libvirt is < v4.1.0  https://review.opendev.org/c/openstack/nova/+/74736213:06
*** k_mouza_ has joined #openstack-nova13:07
*** k_mouza has quit IRC13:11
*** hemanth_n has quit IRC13:11
*** ociuhandu has quit IRC13:15
*** tkajinam_ has quit IRC13:16
sean-k-mooneygmann: gibi we might want to increase the test timeout on tempest-integrated-compute form 2 hours to 313:29
sean-k-mooneyit timed out on https://review.opendev.org/c/openstack/nova/+/757614 during the senario tests but everything else ran fine13:29
sean-k-mooneyit does look like a potentally slow node but we should proably keep an eye on it13:29
sean-k-mooneythats the only time out there https://zuul.opendev.org/t/openstack/builds?job_name=tempest-integrated-compute&project=openstack/nova13:31
gibisean-k-mooney: ack. looking at the run historyt https://zuul.opendev.org/t/openstack/builds?job_name=tempest-integrated-compute&project=openstack/nova most of the runs are areound 1.5 hours13:31
sean-k-mooneyand it normally complete in aboiut 1:2013:31
sean-k-mooneyyep13:31
gibiso let's keep an eye on it13:31
gibibut right now I would not increase yet13:31
sean-k-mooneyi think in general the current time out looks right13:31
*** mgoddard has joined #openstack-nova13:32
sean-k-mooneythe fact we have a normal run lenght in the next patch in the chain would seam to indicate that the time it ran fro is unrealted to the patch too13:33
gibiyes, I think this was an extra slow node13:34
sean-k-mooneycollections.defaultdict(lambda: 0) huh thats a new one13:34
sean-k-mooneyso  ret: ty.DefaultDict[str, int] = collections.defaultdict(lambda: 0) is default initallising a dictionary where the values will be 013:36
*** hoonetorg has joined #openstack-nova13:37
gibidefaultdict takes a callable to produce the default value13:37
gibisometimes we use int instead of lamba: 013:38
gibiboth produce 013:38
gibias int() == 0 as well as lambda: 0() == 013:38
sean-k-mooneyya its a cleaver solution but also suprisign if you have not seen it before13:39
sean-k-mooneyi would have expect to just pass 0 honestly13:39
*** hemna has quit IRC13:39
*** ociuhandu has joined #openstack-nova13:39
sean-k-mooneybut i can see why that would not be a good patern in general13:40
gibiyou cannot pass 0 as that is not a callable13:40
sean-k-mooneyif it was a complex object instead of a literall it would be a problem to allow passing the object directly13:40
sean-k-mooneysince it would be shared not cloned13:40
sean-k-mooneyhence a callable to produce the defualt value13:40
gibiyepp13:40
gibiso we have to pass something that returns 013:41
gibiboth int, and lamba:0 is good for that, but a 0 literal is not13:41
sean-k-mooneyyep13:41
*** lbragstad has joined #openstack-nova13:42
sean-k-mooneyits more a deficite of the api of the datastucture then anything else. its more flexable however then the claical default approch13:42
sean-k-mooneyits just different form how get and set_default work13:43
sean-k-mooneybut inline with how sorted or filter works where you provide a key funcion as a callable13:43
*** hemna has joined #openstack-nova13:44
*** ociuhandu has quit IRC13:49
*** ociuhandu has joined #openstack-nova13:50
nightmare_unrealhow can i access clouds.yaml file once it's loaded. Suppose i want to read some values from it . For context i am trying to read placement_api_version13:52
sean-k-mooneythere is a lib for it but OSC will automaticlaly do it for vars in the right format13:53
sean-k-mooneyits just a yaml file however so in your own code you can jsut read it with the stardard lib13:54
sean-k-mooneyi think its in osc-lib13:55
nightmare_unreali am working on a bug : osc client does not read placement API version variable . variable name placement_api_version13:56
nightmare_unrealso i am trying to figure out where in osc it reads the yaml file13:56
sean-k-mooneyits case sensitive and has to be uppercase just an fyi13:57
sean-k-mooneyi think13:57
openstackgerritLee Yarwood proposed openstack/nova master: docs: Move the LibvirtDistroSupportMatrix wiki page into our docs  https://review.opendev.org/c/openstack/nova/+/77198113:57
sean-k-mooneythat would be in osc-placement13:57
nightmare_unrealobvious solution that will not be good is to read the config file in osc_placement plugin and look for that value if it's set then good .13:57
lyarwoodstephenfin / kashyap ; ^ would you mind reviewing this, still need to add some versions and work out where it should live (if not in our reference docs)13:57
sean-k-mooneynightmare_unreal: https://github.com/openstack/osc-placement/blob/master/osc_placement/plugin.py#L5313:58
*** mlavalle has joined #openstack-nova13:58
sean-k-mooneyit uses https://github.com/openstack/osc-lib/blob/ec1b7dda0e6dc1ec614a92e24267d058783ad27f/osc_lib/utils/__init__.py#L134-L14413:59
sean-k-mooneyto get the values for the env13:59
nightmare_unrealin clouds.yaml it should be defined like that ? OS_PLACEMENT_API_VERSION but i think there is a file that automatically does that13:59
sean-k-mooneybut i think cloud.yaml is expeorted to the evn13:59
nightmare_unrealI see13:59
nightmare_unrealin my head the env is when in cli we explicitly do export OS_BLAH_BLAH13:59
sean-k-mooneyyes it is14:00
nightmare_unrealbut if it takes clouds.yaml as env14:00
nightmare_unrealthen i have to think14:00
sean-k-mooneybut i think we extract some paramaters and add them to it14:00
sean-k-mooneyi havent looked at this closely but was how i understood it to work14:00
gmannbrinzhang_: ack, I will check those in details and see were it is wrong. basically having separate APi controller class should not cause any issue14:01
nightmare_unrealI see. yeah trying to figure it out myself  :)14:02
sean-k-mooneyso i think this is wher we read the cloud.yaml https://github.com/openstack/osc-lib/blob/ec1b7dda0e6dc1ec614a92e24267d058783ad27f/osc_lib/shell.py#L408-L41314:03
gmannsean-k-mooney: gibi I thikn 2 hr is even more time for that job disbale swift and does not run swift/keystone/slow tests14:03
sean-k-mooneygmann: ya i mentioned the timeout befor ei check the build page14:03
kashyaplyarwood: Hey, will check; sigh, missed the notif here14:03
gmannohk14:03
sean-k-mooneygmann: it shoudl be fine just a slow node14:03
gmannsean-k-mooney:  yeah.14:04
gmannexcept tripleo jobs almost all our gate are with max 2 hrs  job run14:04
sean-k-mooneyyep14:04
kashyaplyarwood: Thanks for moving; I think I have a half-done patch for it somewhere; but didn't update it14:05
nightmare_unrealthanks sean-k-mooney  , looking :D14:06
sean-k-mooneythat delegating to the sdk now14:06
sean-k-mooneynightmare_unreal: this is what does the actul loading i think https://github.com/openstack/openstacksdk/blob/4ce235d36dfa3420534ad1b145cc7c5f7438e8e0/openstack/config/loader.py#L14014:09
nightmare_unrealyeah i think so too, looking14:10
*** hemna has quit IRC14:12
sean-k-mooneynightmare_unreal: regardelss of how its parsed i think this is what you are really looking for https://github.com/openstack/openstacksdk/blob/4ce235d36dfa3420534ad1b145cc7c5f7438e8e0/openstack/config/loader.py#L105714:14
sean-k-mooneyto lookup a value in the config14:15
nightmare_unrealyes exactly , and also to check if the defined key:value in clouds.yaml is read or not14:15
nightmare_unrealthanks14:15
kashyaplyarwood: Do you havea  rendered version of it?14:16
kashyaplyarwood: sean-k-mooney: For testing w/ Secure Boot in the Gate ... none of the cloud images are built with EFI partition ... I'm wondering if are allowed to upload new images for testing?14:18
lyarwoodkashyap: http://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8b1/771981/1/check/openstack-tox-docs/8b1d7c4/docs/reference/libvirt-distro-support-matrix.html14:19
kashyapThe short context is: you'd need a disk image with EFI partition in it14:19
kashyapI'm thinking of making a custom template image for testing; is it allowed to upload somewhere via the infra and make use of it?14:19
* kashyap explores other, less invasive ways meanwhile14:19
lyarwoodkashyap: we shouldn't need to upload an image into CI, we can just host and reference one that does in a test job14:19
lyarwoodkashyap: assuming you're talking about the test instances and not the test hosts14:20
kashyaplyarwood: Ah; okay.  I'll find a suitable location to host it, once I prepare it14:20
*** ociuhandu has quit IRC14:20
*** jamesdenton has quit IRC14:20
kashyaplyarwood: Test instance, indeed.  I'm talking about a guest14:20
sean-k-mooneykashyap: do you mean the guest image e.g. cirros14:20
lyarwoodkashyap: right then there's likely best practice our where we can host it, if it needs to be cached etc but we should be able to drop in a suitable image14:21
lyarwoodpractice around*14:21
sean-k-mooneywe are not ment to pull arbiary blobs down in the jobs14:21
kashyaplyarwood: E.g. this is what a guest image that can do EFI/SB should have, to start with: http://paste.openstack.org/show/801871/14:21
kashyapsean-k-mooney: Yes, guest images indeed.14:21
sean-k-mooneyso if we need a special test image we need to have that hostsed/cached by infra14:21
kashyapsean-k-mooney: Okay; I'll check w/ the infra folks.14:21
sean-k-mooneykashyap: are there any test image packaged in distors14:21
sean-k-mooneykashyap: i know we sometimes have isos deistibted by rpms14:22
kashyapNo, none do it; most do it via install trees14:22
kashyapsean-k-mooney: For example, see how I test it here: https://kashyapc.fedorapeople.org/Create-a-SecureBoot-enabled-VM.bash14:22
sean-k-mooneywhat we would praobly want to do is convert that into a diskimage builder element14:23
sean-k-mooneythen have nodepool build it for us14:23
sean-k-mooneyand cache it on the provieders14:23
*** hemna has joined #openstack-nova14:24
lyarwoodcan the instances pull images from glance in these envs?14:24
lyarwoodor swift14:24
lyarwoodI guess so14:24
sean-k-mooneyi dont think so . i mean they could but dont have logisn to do so14:24
kashyapYeah, that's the question.  The simplest way to get this going is to have a pre-made template (which I'll make) image that Just Works14:25
sean-k-mooneybut ya having nodepool upload it as a glance image to the base cloud was one of the thigns i was thinking of14:25
sean-k-mooneybut infra also publish the images too on a webshare14:25
sean-k-mooneyhere https://nb03.opendev.org/14:27
* kashyap clicks14:27
sean-k-mooneyactully no that the lgos there is a place14:27
openstackgerritLee Yarwood proposed openstack/nova stable/rocky: Remove allocations before setting vm_status to SHELVED_OFFLOADED  https://review.opendev.org/c/openstack/nova/+/77198514:32
openstackgerritLee Yarwood proposed openstack/nova stable/queens: Remove allocations before setting vm_status to SHELVED_OFFLOADED  https://review.opendev.org/c/openstack/nova/+/77198614:32
*** jamesdenton has joined #openstack-nova14:45
*** ociuhandu has joined #openstack-nova14:49
*** rpittau is now known as rpittau|afk14:58
openstackgerritStephen Finucane proposed openstack/nova-specs master: Add spec for configurable-instance-hostnames  https://review.opendev.org/c/openstack/nova-specs/+/77084214:58
stephenfingibi: Address your comments ^14:58
stephenfinlyarwood: Fancy looking at that simple spec today? ^14:59
lyarwoodstephenfin: yup14:59
lyarwoodstephenfin: wasn't the deadline EOD yesterday ;)15:00
gibithe deadline was EOD yesterday :/15:01
*** tesseract has quit IRC15:02
*** tesseract has joined #openstack-nova15:02
gibistephenfin: do you want to persuade me to allow a small deadline extension for that? ;)15:02
kashyapgibi: If it's quick, and stephenfin is on fire, any problem in merging it?   I'm asking from a purely bystander PoV :-)  I don't need that feature myself15:02
* kashyap --> call; back later15:02
gibikashyap: no real problem, the worst thing is that we will not have time to implement it or review the implementation of it15:03
melwittelod: re: https://review.opendev.org/c/openstack/nova/+/747362 I had thought the cherry pick check would catch wrong hashes? :/15:05
kashyapgibi: (Nod)15:05
gibisean-k-mooney: hi! I don't find the bp https://blueprints.launchpad.net/nova/+spec/port-scoped-sean-k-mooney: I don't find the bp https://blueprints.launchpad.net/nova/+spec/port-scoped-sriov-numa-affinity behind https://review.opendev.org/c/openstack/nova-specs/+/765901 spec15:07
sean-k-mooneyoh i might not have filed it ill check just on a call15:07
* gibi almost managed to copy paste that message from the morning ^^15:07
gibisean-k-mooney: cool, thanks15:08
gibijust link me to the bp and I will approve it as the spec was mereged15:08
elodmelwitt: it did catch :)15:09
*** mugsie has joined #openstack-nova15:09
melwittelod: oh, good. ah, it hadn't had its zuul result yet at the time I reviewed. ok, all makes sense now :)15:10
stephenfingibi: I have the code done and the change is small, but it's also probably not important enough to warrant a spec freeze exception15:10
*** spatel has joined #openstack-nova15:10
gibistephenfin: let's merge it today15:10
gibilyarwood: ^^15:10
spatelsean-k-mooney: thank for you comments - https://bugs.launchpad.net/nova/+bug/191227315:11
openstackLaunchpad bug 1912273 in OpenStack Compute (nova) "SRIOV instance Error: Exception during message handling: KeyError: 'pci_slot'" [Undecided,Incomplete]15:11
lyarwoodgibi: ack, just on a call now, I'll review once this is over15:12
gibithanks15:12
elodmelwitt: the 'zuul summary' maybe a bit misleading as it shows the last runs, but does not show that a new test is running15:12
spatelThis is what i have noticed, when my vm get oom and stuck/hung then openstack destroying its all configuration from box (is that normal behavior?)15:12
melwittelod: yeah, I'm not sure what I did to miss it. might have just been pebkac :P15:13
elod:)15:14
elodanyway, it's fixed now, so we'll see :)15:14
melwitt++15:15
*** vishalmanchanda has quit IRC15:15
*** mugsie has quit IRC15:18
openstackgerritMerged openstack/nova master: libvirt: Stop NVMe and ScaleIO unit tests from calling os-brick  https://review.opendev.org/c/openstack/nova/+/77180615:20
*** links has quit IRC15:24
*** tesseract has quit IRC15:28
*** tesseract has joined #openstack-nova15:32
openstackgerritMerged openstack/nova master: libvirt: Load and cache volume drivers on-demand  https://review.opendev.org/c/openstack/nova/+/74154515:34
openstackgerritMerged openstack/nova stable/queens: sync_guest_time: use the proper errno  https://review.opendev.org/c/openstack/nova/+/74242315:36
openstackgerritMerged openstack/nova stable/queens: libvirt: Remove reference to transient domain when detaching devices  https://review.opendev.org/c/openstack/nova/+/60818615:36
openstackgerritMerged openstack/nova stable/rocky: [stable-only] Cap bandit and make lower-constraints job non-voting  https://review.opendev.org/c/openstack/nova/+/76649215:37
*** nweinber has joined #openstack-nova15:43
*** dklyle has joined #openstack-nova15:44
zigoIn Debian Unstable/Testing, I'm getting this for absolutely everything that connects to rabbit: http://paste.openstack.org/show/801876/15:45
zigoDoes anyone know what's going on ?15:46
*** mugsie has joined #openstack-nova15:51
*** nweinber has quit IRC15:52
*** nweinber has joined #openstack-nova15:53
fricklerzigo: looks like this one to me https://bugs.launchpad.net/oslo.messaging/+bug/190269615:56
openstackLaunchpad bug 1902696 in oslo.messaging "nova-compute fails with Unhandled error: TypeError: _wrap_socket_sni() got an unexpected keyword argument 'ca_certs'" [Undecided,New]15:56
zigofrickler: Thanks !15:57
* bauzas goes off (b/c of our curfew) \o15:58
lyarwoodstephenfin: looking at that spec now btw15:58
sean-k-mooneymnaser: hi o/ do you have time for a qq15:59
sean-k-mooneymnaser: we are talking about nova db archives downstream16:00
sean-k-mooneymnaser: we were just wondering how often you run them at vexhost and with what parmaters16:00
sean-k-mooneymnaser: also if you have the openstck ci cluster tuned differently the the rest of the public cloud16:01
lyarwoodartom / bauzas: https://review.opendev.org/c/openstack/nova-specs/+/770842 - you're listed on this review, do you want to give it a look before we +W it?16:08
lyarwoodstephenfin: ^ LGTM btw, but I'll give others a chance to also look before we merge16:09
*** lpetrut has joined #openstack-nova16:10
openstackgerritMerged openstack/nova stable/queens: libvirt: Handle VIR_ERR_DEVICE_MISSING when detaching devices  https://review.opendev.org/c/openstack/nova/+/74242416:11
*** ganso has quit IRC16:12
artomlyarwood, stephenfin, as I've just posted in the spec, we should also make OS-EXT-SRV-ATTR:hostname not admin-only by default then, right?16:13
artomSince users can now set it?16:13
artomNot sure I want to -1 over that this close to the deadline...16:13
stephenfinFair point. I can do a follow-up16:13
*** lpetrut has quit IRC16:14
lyarwoodooooff nice catch yeah16:14
lyarwoodthat's a new microversion at that point right?16:15
stephenfinthis is already a new microversion16:15
* lyarwood should make that into a t-shirt for our next ptg16:15
artomYeah, it already is, just add the visibility thing to it16:15
lyarwoodstephenfin: ah true sorry16:15
*** ganso has joined #openstack-nova16:15
lyarwoodbrb kid just got home16:16
artomBy herself? o_O16:16
*** ociuhandu has quit IRC16:18
*** ociuhandu has joined #openstack-nova16:19
*** raildo has joined #openstack-nova16:20
gibizigo, frickler: I think this is related https://review.opendev.org/c/openstack/requirements/+/76119416:23
zigogibi: I've bumped to 5.0.3, this fixes my issues, but now I'm getting:16:24
zigoAttributeError: 'Semaphore' object has no attribute '_at_fork_reinit'16:24
zigowhich looks like a problem with Eventlet + Python 3.9... :(16:24
gibi:/16:24
*** ociuhandu has quit IRC16:24
zigohttps://github.com/eventlet/eventlet/issues/64616:24
gibizigo: does eventlet officially states that it supports py3.9?16:25
lyarwoodartom: yeah we send them down the mines at ~6 months old around here /s16:26
zigogibi: I don't know, but Bullseye will be released with it (no py3.8 at all).16:26
zigoThis looks like fixing it, I'll try: https://github.com/eventlet/eventlet/pull/664/commits/087f2246ef5380cc5d519f614be7e928ab5ea02516:26
zigoDebian has already 18 patches in the Eventlet package.16:27
zigoHalf a decade ago, there was some discussions to get rid of Eventlet in OpenStack. I'd love to have this effort started some day !16:27
zigoEventlet is a Bi... (pardon my language).16:27
openstackgerritMerged openstack/nova-specs master: Add spec for configurable-instance-hostnames  https://review.opendev.org/c/openstack/nova-specs/+/77084216:28
zigoOh, we got that one already ... :/16:28
gibistephenfin: https://review.opendev.org/c/openstack/nova-specs/+/770842 will you push a followup fixing artom's comments?16:29
stephenfinon it16:29
gibistephenfin: also I dont find the related bp in launchpad16:30
openstackgerritLance Bragstad proposed openstack/placement master: Refactor usage tests to properly test project personas  https://review.opendev.org/c/openstack/placement/+/77206116:30
lbragstadstephenfin ^ that fixes the tests you modified to expose usage in placement to project users16:30
*** macz_ has joined #openstack-nova16:30
stephenfinlbragstad: oh, the project_id was different /o\16:33
stephenfinI even wrote a quick unit test that proved it worked, but I figured the unit test was wrong because the functional test was obviously correct /o\16:33
lbragstadstephenfin right - so the tenancy check was doing what it should16:33
lbragstadbut i found something else that's concerning and i'm not sure how it's working now16:34
lbragstadstephenfin https://review.opendev.org/c/openstack/placement/+/772061/1/placement/tests/functional/gabbits/usage-secure-rbac.yaml16:34
lbragstadi added negative tests to ensure project users from anther project can't fetch usage information for projects they don't have authorization on16:35
*** macz_ has quit IRC16:35
stephenfinOh, I noticed that when I was debugging the rule16:35
lbragstadand the project admin persona test fails consistently because the rule:admin_api rule is appended to the new default, regardless of what we're setting up in fixture16:35
lbragstadi went splunking through the enforce_new_default configuration behavior and it appears to be working as expected16:36
stephenfinHmm, I'm guessing misconfiguration _somewhere_. We have a lot of examples of project reader policies in nova and I'm pretty sure we have unit tests for them all that prove that other project admins can't access $RESOURCE16:37
lbragstadbut that deprecated rule check get appended to the default rule magically16:37
lbragstadyeah... i started looking at the placement fixture structure to see if it was doing something unexpected (not cleaning things up properly)?16:38
*** ociuhandu has joined #openstack-nova16:38
stephenfingibi: There's the BP https://blueprints.launchpad.net/nova/+spec/configurable-instance-hostnames16:38
gibistephenfin: on it16:38
lbragstadbut i couldn't find anything that stuck out - and i'm not that knowledgeable about gabbi/placement tests16:38
lbragstadso - i pushed what i have based on your patch... but if you pull that down you should be able to recreate the issue, it'll fail the check gate16:39
stephenfinNeither am I. They're tough to debug. I can't figure out how to even get logs from the placement server16:39
lbragstaddowngrade gabbi16:40
stephenfinI'll have to poke cdent or efried when they're about, to see if they have any suggestions16:40
efriedHowdy. TLDR or should I read scrollback?16:41
lbragstadif you're using gabbi > 2.0.0 the output logging is broken16:41
lbragstadhttps://github.com/cdent/gabbi/issues/28716:41
stephenfinlbragstad++ Well that makes my life much easier16:41
zigogibi: I just looked, Eventlet claims compat with ... python 3.7 ! :/16:42
lbragstadstephenfin downgrading to 1.49.0 works for me16:42
lbragstadin the sense that you can actually capture stdout in tests16:42
*** ociuhandu has quit IRC16:42
gibizigo: based on that we cannot even release OpenStack :/16:42
zigoYeah...16:43
zigoOr I stay forever on Buster...16:43
stephenfinefried: We're seeing some unusual policy behavior in placement as part of the RBAC work, and I was looking for a way to get more info from the placement server to debug. Sounds like it's just a downgrade of gabbi that's needed ^16:43
* zigo tries the latest version 0.30.0 to see what happens.16:43
efriedcool16:43
openstackgerritStephen Finucane proposed openstack/nova-specs master: Update spec for configurable-instance-hostnames  https://review.opendev.org/c/openstack/nova-specs/+/77206516:45
stephenfingibi: And there's the spec amendment ^16:46
stephenfinbauzas, lyarwood, artom also ^16:46
gibistephenfin: thanks16:47
*** ociuhandu has joined #openstack-nova16:47
zigoGreat, tests.patcher_test.test_fork_after_monkey_patch fails in Py 3.9 ... :/16:50
*** k_mouza_ has quit IRC16:52
*** jmlowe has quit IRC16:52
melwittlyarwood: I wanted to ask you about a failure in test_volume_swap I saw yesterday in the gate, have you seen a thing where it doesn't finish the copy and emits "COPY block job progress, current cursor: 1073741823 final cursor: 1073741824" a lot of times, showing that the cursor is only 1 from the end?16:53
melwitthttps://zuul.opendev.org/t/openstack/build/a078a17aa9924517b329cafc3f54fed4/log/controller/logs/screen-n-cpu.txt#1111516:54
lyarwoodmelwitt: I've not seen it 1 block (!?) away and not finish no16:57
lyarwoodmelwitt: it's typically much greater than that, did it get there pretty quickly and then stall?16:58
melwittack16:58
melwitterm.. let me check.16:58
melwittlyarwood: yeah looks like it actually. got there and then stuck on the last block for 4 minutes16:59
melwittdo you suppose this could be similar to problems with live migration where we've needed post copy/auto converge?17:00
melwittseems weird17:00
lyarwoodyeah it might be but I can't think that the cirros image would be writing that much to the volume if at all17:00
lyarwoodI think we write timestamps during the test and that's it17:00
melwitthm ok17:01
*** jmlowe has joined #openstack-nova17:02
lyarwoodmelwitt: we could write this up as a Ubuntu QEMU bug again and see if upstream can help debug this further17:06
lyarwoodmelwitt: assuming there's something we can log that would help them17:06
melwittlyarwood: good idea, let me go through and collect more data (if there is more) and I'll open one if I can find more to go on17:08
kashyapmelwitt: So ... just reading the scrollback; that "current" and "final" cursors differing means: the copy (i.e. migration w/ storage) hasn't finished succesfully17:08
kashyapIn the past we've hit that, and debugged on list; and I recall filing a libvirt RFE to fix that ... let me check17:08
*** k_mouza has joined #openstack-nova17:09
melwittkashyap: yeah, I think I understood that but the weird thing is it quickly gets to the last block and then stays stuck there for 4 minutes until the test wait in tempest times out and kills it17:09
kashyapHmm17:09
kashyapmelwitt: Yeah, this definitely looks ome something new17:09
kashyapBecause in that old behaviour, libvirt was just making "educated guess" when the syncing has finished17:10
melwittI see17:10
kashyapEspecially the "current cursor" terminology looks new to me.  I haven't seen the word "cursor" in this error's context before17:10
lyarwoodthis is blockCopy and not blockRebase btw kashyap17:11
kashyapmelwitt: I don't want to bore you with a long bug, but for the record, here it is: https://bugzilla.redhat.com/show_bug.cgi?id=138216517:11
openstackbugzilla.redhat.com bug 1382165 in libvirt "virDomainGetBlockJobInfo: Adjust job reporting based on QEMU stats & the "ready" field of `query-block-jobs`" [Unspecified,Closed: nextrelease] - Assigned to pkrempa17:11
kashyaplyarwood: I see; nod17:11
lyarwoodI think we dropped that workaround a while ago17:11
kashyapYep17:11
kashyapBut just for clarity, blockCopy() is a _superset_ of blockRebase()17:11
melwittkashyap: any data/background helps :)17:11
kashyap(So whatever worked, or failed w/ Rebase(), will also fails equal w/ Copy())17:12
*** zzzeek has quit IRC17:12
lyarwoodyeah https://review.opendev.org/c/openstack/nova/+/729596 removed it17:12
kashyapmelwitt: But yeah; filing an upstream Ubuntu QEMU bug would be cool to start with17:12
melwittk, I'll collect data points and write something up. thanks for the hints both17:13
kashyaplyarwood: Yep, recall it as much, it just reminded me of it.17:13
*** zzzeek has joined #openstack-nova17:13
* lyarwood goes back to ethics training17:14
lyarwoodif anyone has anything they'd like me review please let me know17:14
kashyapmelwitt: The libvirtd log might have some interesting stuff in there.  /me tries to fish it out17:14
lyarwoodPLEASE :|17:14
kashyapLOL17:15
melwittI feel for you. clicky clicky clicky17:15
melwittclick all the things17:15
kashyaplyarwood: I finished it the next day it came; I'm a free man!17:15
stephenfinkashyap: no one likes a show off17:15
* kashyap stops thumping chest over utterly trivial actions17:15
kashyapstephenfin: LOL17:15
lyarwood./ban kashyap17:15
lyarwood^_^17:15
kashyapstephenfin: I know; the secret pleasure of doing the right thing goes away if one brags17:16
kashyapmelwitt: I'm looking at the log file here: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a07/771749/1/check/nova-next/a078a17/controller/logs/libvirt/index.html17:18
kashyapHope that's the correct log tree17:18
melwittkashyap: yes that's right17:20
kashyapmelwitt: Unrelated: I know that the "controller" directory has the compute logs, but why is it named "controller"?17:21
kashyap(Because there's also the "compute" top-level directory, which has its own QEMU/libvirt logs.  It's always tripping me up17:21
melwittkashyap: in the case of multi-node job, to identify the "main" node vs one of the additional compute nodes17:21
kashyapI see17:21
melwittin this case you're in the right place bc the swap volume got stuck on the main node17:22
*** ociuhandu_ has joined #openstack-nova17:22
melwittif it had happened on the other compute node, you'd want to look in the compute1 dir17:22
kashyapmelwitt: It looks like so; because I don't see the failure in this libvirtd log17:22
kashyapAnd these copy failures are logged to libvirtd.  So let me look in the other one17:22
sean-k-mooneylyarwood: you should review gibis qos attach17:23
sean-k-mooneyseries17:23
melwittkashyap: would there be a failure though? there was no error. it's just that the cursor never moved from the last block17:23
kashyapmelwitt: Not a failure per-se; but in Gate, we have libvirt log filters enabled17:23
melwittand the test timed out at the tempest level as a result of never seeing the 'available' status for the volume17:23
sean-k-mooneylyarwood: im basically done with it and it looks ready to merge IMO17:23
kashyapmelwitt: ... so we should see more fine-grained info.  And I was hoping to hurl a useful bit at one of the US-based libvirt developers :D17:24
sean-k-mooneylyarwood: but it need a second core to review17:24
melwittkashyap: ack17:24
kashyapmelwitt: Oh, wait ... I first didn't see this error is _actually_ libvirt error.  Looks like it's not17:25
* kashyap looks closer17:25
sean-k-mooneylyarwood: specifclly i was refering to https://review.opendev.org/q/topic:%22bp%252Fsupport-interface-attach-with-qos-ports%22+(status:open%20OR%20status:merged) if you are still lookign for somethign easy to review17:25
*** ociuhandu has quit IRC17:26
melwittkashyap: yeah it's not emitting an error afaik. it's just that we keep checking the job status and it shows it one block from the end and it stays that way for about 4 minutes and then we give up17:26
*** ociuhandu_ has quit IRC17:27
kashyapmelwitt: Have you got the instance name, please?17:28
kashyapFound it17:31
melwittkashyap: uuid is 185d6573-58ee-47bc-8063-b447673db266, are you wanting the instanceNNNNNN id?17:32
melwittsorry, I was double checking whether I got the right one17:32
kashyapmelwitt: No-no; that's fine :-)17:32
melwittmy eyes are like @_@17:32
kashyapmelwitt: Heh; urgh; my FF browser is hung17:33
kashyapAlso there's an unpleasant easter egg in there (I knew it before too): ".txt" files are actually *gzip* files that will devour your RAM, if you try open them in your browser17:33
kashyapAlright it came back.  And it's this: instance-00000013 (185d6573-58ee-47bc-8063-b447673db266).  Let's see if there's anything useful17:35
sean-k-mooneykashyap: that depend on which of the ci provider the logs are uploaded too17:37
sean-k-mooneythe rackspace cloud for reason has there swift cluster configured in such a way that it always compresses them but does not send the correct headers17:37
sean-k-mooneyfor things like curl to work17:37
kashyapI see17:42
kashyapmelwitt: So ... something interesting (at least to me) :-) --17:42
kashyapI just extracted the exchange between libvirt and QEMU from the giant log here: https://kashyapc.fedorapeople.org/QMP_exchange_libvirtd_22Jan2021_NovaCI.txt17:43
kashyapSo ... the copy has actually *succeeded*.  Here's the evidence:17:43
kashyap- You `grep` for "blockdev-mirror" in there (at 2021-01-21 05:29:32.630+0000) -- that's the start of the copy job17:43
kashyap- Then there's constant polling going on -- you'll see a lot of "io-status": "ok" going on there17:44
kashyap- Finally, at 2021-01-21 05:34:24.571+000017:44
kashyapWe get the "ready" flag (which means copy has completed: notice the values of 'len' and 'offset' are equal:17:45
kashyap{"return": [{"auto-finalize": true, "io-status": "ok", "device": "copy-vdb-libvirt-5-format", "auto-dismiss": false, "busy": true, "len": 1073741824, "offset": 1073741824, "status": "running", "paused": false, "speed": 0, "ready": false, "type": "mirror"}], "id": "libvirt-843"}17:45
kashyapmelwitt: So, all I can say for now is the copy has succeeded.  So I'm not sure why it's saying the copy is not progressing17:46
*** ociuhandu has joined #openstack-nova17:47
kashyapmelwitt: I'm out of neurons, and need to make some food ... we can continue the chat in the (future) bug17:47
melwittthanks kashyap, that's helpful. have a nice weekend o/17:48
kashyap(I think it can't progress because it has reached the end of the copy -- the value in the error matches what I posted above: 1073741824.  Hm, needs further digging.)17:51
*** gyee has joined #openstack-nova17:51
kashyapHave a good weekend, you too!17:51
*** ralonsoh has quit IRC17:51
melwittkashyap: yeah it seems the job_status api is returning the wrong thing then, or such17:51
melwittget_job_info https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L863-L87317:52
*** ociuhandu has quit IRC17:53
lyarwoodis it just a race?17:58
* lyarwood hasn't checked the timestamps17:58
lyarwoodbut did libvirtd log the job as finished after n-cpu had already given up?17:58
*** tesseract has quit IRC17:59
*** derekh has quit IRC18:01
*** luksky has quit IRC18:01
*** luksky has joined #openstack-nova18:03
*** manuvakery1 has quit IRC18:03
openstackgerritBalazs Gibizer proposed openstack/nova master: Replace blind retry with libvirt event waiting in detach  https://review.opendev.org/c/openstack/nova/+/77024618:10
melwittlyarwood: I will check18:11
*** slaweq has quit IRC18:12
gibihave a nice weekend folks! o/18:15
lyarwoodand you gibi \o18:16
* lyarwood also drops18:16
*** lbragstad has quit IRC18:16
lyarwoodo/ all18:16
sean-k-mooneyo/18:16
melwittlyarwood: interesting, it looks like in libvirtd log it shows current == end but what we get in nova when we query is not, it's current == end - 118:17
melwittlyarwood: and also o/18:17
*** dtantsur is now known as dtantsur|afk18:17
sean-k-mooneygibi: finished reviewing you qos series, +1 on most of it but left two -1s nothing major just look to confirm detach will work too and a minor issue with comments18:18
sean-k-mooneygibi: but ya enjoy your weekend18:19
*** lbragstad has joined #openstack-nova18:22
*** nightmare_unreal has quit IRC18:25
gmannlbragstad: 1 comment on this https://review.opendev.org/c/openstack/placement/+/760240/15/placement/tests/functional/gabbits/resource-provider-legacy-rbac.yaml#2018:26
gmannlbragstad: we should check all persona for legacy policy also.18:27
gmannbasically checking all those tokens with enforce_scope false and true18:27
sean-k-mooneyok im going to finish there too o/18:27
lbragstadgmann ok - i was thinking you only wanted testing for the default personas that were supported prior to secure RBAC?18:30
lbragstadgmann does nova test system personas and reader personas with enforce_scope=False and enforce_new_defaults=False?18:30
openstackgerritMerged openstack/nova-specs master: Update spec for configurable-instance-hostnames  https://review.opendev.org/c/openstack/nova-specs/+/77206518:30
gmannlbragstad: yeah we test all those with all those combination of config18:31
*** macz_ has joined #openstack-nova18:31
gmannlbragstad: is there any admin-or-owner rule?18:31
lbragstadfor placement?18:31
gmannyeah18:32
gmannfor nova I added all persona to check owner thing basically18:32
lbragstadno - all of placement was protected with rule:admin_api which just checks role:admin essentially18:32
gmannok, in that case system-admin will also be able to access if enforce_scope is false18:32
gmannas no project-id is checked there18:33
lbragstadcorrect - system-admin will be able to access placement APIs18:33
gmannadmin is ok but reader we can make sure if new reader (project or system) cannot access in case of enforce_scope=false because we care changing the check_str to system_reader18:34
gmanndoes it make sense?18:35
sean-k-mooneywe generally dont want project admins to be able to access placment by the way18:35
sean-k-mooneyand im not sure domain admins should eb able to either18:35
*** macz_ has quit IRC18:36
sean-k-mooneyplamcnet is one of those things that really only system admins should have access too18:36
gmannsean-k-mooney: with new policy rigtht?18:36
sean-k-mooneywell system member would be ok too18:36
sean-k-mooneygmann: yep18:36
gmannyeah. that is lbragstad change does18:36
gmannlegacy Admin to SYSTEM_ADMIN18:37
sean-k-mooneybasically we need to enforce system scope18:37
sean-k-mooneyya18:37
gmannyeah18:37
lbragstadwell - there was a note in placement about exposing the usage API to project users18:37
sean-k-mooneythere may be uses for system reader or system member/user at somepoint18:38
lbragstadfrom when mriedem refactored placement's policy enforcement and moved default policies into code18:38
lbragstadstephenfin thought we should address that comment while we're in there mucking with the secure RBAC changes...18:38
sean-k-mooneylbragstad: well currenlty plamcnet is not recommended to be exposed to endusers at all18:38
sean-k-mooneyat least in a public cloud case its provide a view into your infrastucre that you dont want normal tenants to have18:39
lbragstadok - in that case we should let stephenfin know? i'm assume i'm the person with the least amount of placement context18:39
lbragstadhttps://review.opendev.org/c/openstack/placement/+/771964/118:39
sean-k-mooneythere is perhaps a usces with unified limits18:40
sean-k-mooneywhere  if we are counting quots with lmits some endpoint might be ok18:40
sean-k-mooneythe RP endpoint and allocation candiates endpoint no18:40
sean-k-mooneybut looking an allocations for your instance might be ok18:41
gmannbut does not it give usages for requested project18:41
sean-k-mooneyeach resouce provider has the hostname as its name so we cant allow project user to do an RP show for example18:41
sean-k-mooneyi though the usage endpoint was teh usage of an invetory18:42
sean-k-mooneyok https://docs.openstack.org/api-ref/placement/?expanded=list-usages-detail#list-usages18:43
sean-k-mooneywould be ok for project users18:43
sean-k-mooneythat is the quota sutff18:44
sean-k-mooneylbragstad: so https://review.opendev.org/c/openstack/placement/+/771964/1/placement/tests/functional/gabbits/usage-secure-rbac.yaml is fine18:44
gmannyeah it make sense to have project scope there18:45
sean-k-mooneyi think /resource_providers /reshaper /allocation_candidates need to be system scope at lest by default18:45
sean-k-mooneyim not sure about /allocations/{consumer_uuid}18:46
lbragstadyeah - stephenfin's change was specific to the usage API i believe18:47
sean-k-mooneyi think that would actully be ok as it is today for projects too https://docs.openstack.org/api-ref/placement/?expanded=list-allocations-detail#list-allocations18:47
lbragstadbecause of https://review.opendev.org/c/openstack/placement/+/771964/1/placement/policies/usage.py#5218:47
sean-k-mooneyya ok i think stephenfin's patch is going in the right direction18:49
sean-k-mooneyi dont see anything that is definetly incorrect e.g. too borad18:49
sean-k-mooneyill start it for monday18:50
lbragstadok - i'd appreciate it, i think there are some things in how placement sets up fixtures that need to be investigated, too https://review.opendev.org/c/openstack/placement/+/772061/1/placement/tests/functional/gabbits/usage-secure-rbac.yaml18:51
*** k_mouza has quit IRC18:56
gmannlbragstad: commented in this ^^. project_id is not passed for policy check18:56
*** andrewbonney has quit IRC18:56
gmannif we pass req.project_id in context.can() then it can fail if it is different from context.project_id18:57
gmannI have not tested that just checked it in code. I faced many such isuse in nova APIs18:57
lbragstadgmann i think we already fixed that in the previous patch18:58
gmannoh, did not see that18:58
*** jamesdenton has quit IRC19:43
*** jamesdenton has joined #openstack-nova19:43
*** slaweq has joined #openstack-nova19:46
*** slaweq has quit IRC20:01
*** bbowen has quit IRC20:33
*** nweinber has quit IRC20:34
*** viks____ has quit IRC20:50
*** READ10 has joined #openstack-nova20:54
*** k_mouza has joined #openstack-nova20:56
*** rcernin has joined #openstack-nova20:58
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for resource providers  https://review.opendev.org/c/openstack/placement/+/76024021:01
*** k_mouza has quit IRC21:01
*** rcernin has quit IRC21:02
*** rcernin has joined #openstack-nova21:03
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for resource providers  https://review.opendev.org/c/openstack/placement/+/76024021:05
*** k_mouza has joined #openstack-nova21:15
*** k_mouza has quit IRC21:20
*** rcernin has quit IRC21:28
*** READ10 has quit IRC21:41
*** xek_ has quit IRC21:48
*** macz_ has joined #openstack-nova22:33
*** bbowen has joined #openstack-nova22:37
*** macz_ has quit IRC22:37
*** rcernin has joined #openstack-nova22:46
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for resource providers  https://review.opendev.org/c/openstack/placement/+/76024022:47
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for aggregates  https://review.opendev.org/c/openstack/placement/+/76023522:47
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for allocations  https://review.opendev.org/c/openstack/placement/+/76023622:47
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for allocation candidates  https://review.opendev.org/c/openstack/placement/+/76023722:47
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for inventories  https://review.opendev.org/c/openstack/placement/+/76023822:47
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for resource classes  https://review.opendev.org/c/openstack/placement/+/76023922:47
openstackgerritLance Bragstad proposed openstack/placement master: Implement secure RBAC for traits  https://review.opendev.org/c/openstack/placement/+/76024122:47
*** spatel has quit IRC22:50
*** zzzeek has quit IRC22:56
*** zzzeek has joined #openstack-nova22:57
*** ociuhandu has joined #openstack-nova22:59
*** ociuhandu has quit IRC23:03
*** jamesdenton has quit IRC23:40
*** jamesdenton has joined #openstack-nova23:41

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!