| *** mlavalle has quit IRC | 00:15 | |
| *** tetsuro has joined #openstack-nova | 00:16 | |
| *** gyee has quit IRC | 00:31 | |
| *** swp20 has joined #openstack-nova | 00:41 | |
| *** gregwork has quit IRC | 00:42 | |
| openstackgerrit | xuyuanhao proposed openstack/nova master: Optimization the soft-delete logical https://review.opendev.org/724260 | 00:43 |
|---|---|---|
| *** songwenping_ has joined #openstack-nova | 00:52 | |
| *** swp20 has quit IRC | 00:56 | |
| *** Liang__ has joined #openstack-nova | 01:09 | |
| openstackgerrit | xuyuanhao proposed openstack/nova master: Optimization the soft-delete logical https://review.opendev.org/724260 | 01:25 |
| *** tkajinam has quit IRC | 01:29 | |
| *** tkajinam has joined #openstack-nova | 01:29 | |
| *** xiaolin has joined #openstack-nova | 01:43 | |
| openstackgerrit | Wenping Song proposed openstack/nova master: error may occur when filter scheduler with accelerator https://review.opendev.org/722651 | 02:03 |
| *** hongbin has joined #openstack-nova | 03:00 | |
| openstackgerrit | xuyuanhao proposed openstack/nova master: Optimization the soft-delete logical https://review.opendev.org/724260 | 03:16 |
| *** tetsuro has quit IRC | 03:24 | |
| *** tetsuro has joined #openstack-nova | 03:35 | |
| *** psachin has joined #openstack-nova | 03:39 | |
| *** eharney has quit IRC | 03:48 | |
| *** lbragstad has quit IRC | 03:51 | |
| *** ratailor has joined #openstack-nova | 04:06 | |
| *** songwenping_ has quit IRC | 04:07 | |
| *** songwenping_ has joined #openstack-nova | 04:07 | |
| openstackgerrit | Wenping Song proposed openstack/nova master: error may occur when filter scheduler with accelerator https://review.opendev.org/722651 | 04:29 |
| *** evrardjp has quit IRC | 04:36 | |
| *** evrardjp has joined #openstack-nova | 04:36 | |
| *** psachin has quit IRC | 05:03 | |
| *** hongbin has quit IRC | 05:04 | |
| *** songwenping_ has quit IRC | 05:05 | |
| *** links has joined #openstack-nova | 05:22 | |
| *** songwenping_ has joined #openstack-nova | 05:33 | |
| *** udesale has joined #openstack-nova | 05:35 | |
| openstackgerrit | Harshavardhan Metla proposed openstack/nova master: [Nova] Add reference to Placement installation guide https://review.opendev.org/726936 | 05:44 |
| *** dpawlik has joined #openstack-nova | 06:05 | |
| *** brinzhang has quit IRC | 06:11 | |
| *** brinzhang has joined #openstack-nova | 06:12 | |
| *** nightmare_unreal has joined #openstack-nova | 06:14 | |
| *** brinzhang has quit IRC | 06:14 | |
| *** brinzhang has joined #openstack-nova | 06:15 | |
| *** tkajinam has quit IRC | 06:35 | |
| *** dklyle has quit IRC | 06:35 | |
| *** tkajinam has joined #openstack-nova | 06:36 | |
| gibi | good morning Nova | 06:38 |
| *** ttsiouts has joined #openstack-nova | 06:39 | |
| *** ikla has joined #openstack-nova | 06:47 | |
| ikla | how do I passthrough multiple pci devices in nova? | 06:47 |
| gibi | ikla: https://docs.openstack.org/nova/pike/admin/pci-passthrough.html#configure-a-flavor-controller | 06:53 |
| *** ccamacho has joined #openstack-nova | 06:56 | |
| *** ccamacho has quit IRC | 06:56 | |
| *** belmoreira has joined #openstack-nova | 06:57 | |
| *** maciejjozefczyk has joined #openstack-nova | 06:59 | |
| sean-k-mooney | ikla: you just create a flavor with multiple alia's referenced | 07:00 |
| *** tony_ has joined #openstack-nova | 07:02 | |
| *** tesseract has joined #openstack-nova | 07:04 | |
| ikla | I have rocky | 07:09 |
| jkulik | hi, is there something like sub-flavors somewhere on nova's roadmap? Having a flavor that maps to different specific flavors per hardware generation would be great. | 07:10 |
| *** slaweq has joined #openstack-nova | 07:11 | |
| *** ccamacho has joined #openstack-nova | 07:13 | |
| ikla | do I need to define the devices in the nova controller? | 07:16 |
| gibi | ikla: the alias needs to be defined both on the controller and on the computes, the passthrough_whitelist needs to be defined on each compute | 07:18 |
| gibi | jkulik: I don't know about such feature on the roadmap. I think you can implement that top of nova | 07:19 |
| ikla | do I have multiple lines with passthrough_whitelist ? or can I do it all on one line? | 07:20 |
| bauzas | good morning nov | 07:21 |
| bauzas | err, nova even | 07:21 |
| gibi | ikla: you can have a full json dict, see the config doc https://docs.openstack.org/nova/rocky/configuration/config.html#pci.passthrough_whitelist | 07:22 |
| gibi | ikla: or even a list of such dicts | 07:22 |
| gibi | bauzas: o/ | 07:22 |
| *** awalende has joined #openstack-nova | 07:23 | |
| ikla | same for alias? | 07:25 |
| gibi | ikla: I think alias is a single dict, so you have to have multiple line of alias = {...} in your config to have multiple aliases | 07:26 |
| ikla | thanks | 07:28 |
| ikla | passthrough cannot have multiple lines? | 07:28 |
| jkulik | gibi, by on top of nova, do you mean on the client? Even if we would control the client, which we don't necessarily, we would have to decide on the precise flavor before going into scheduling afaics. So if a user doesn't care about the hardware version, we would have to make a decision anyways instead of taking "what's free" automatically. | 07:29 |
| *** huaqiang has joined #openstack-nova | 07:31 | |
| ikla | nvrmind :) | 07:32 |
| *** ralonsoh has joined #openstack-nova | 07:33 | |
| gibi | ikla: I think passthrough_whitelist also can be used multiple times | 07:34 |
| *** tosky has joined #openstack-nova | 07:35 | |
| ikla | does passthrough_whitelist work on rocky or does it need to be prefixed with pci_ ? | 07:36 |
| gibi | ikla: in the [pci] section you can use passthrough_whitelist config | 07:37 |
| *** xek has joined #openstack-nova | 07:38 | |
| *** sapd1_x has joined #openstack-nova | 07:45 | |
| openstackgerrit | Harshavardhan Metla proposed openstack/nova master: [Nova] Add reference to Placement installation guide https://review.opendev.org/726936 | 07:50 |
| *** lpetrut has joined #openstack-nova | 07:52 | |
| *** dtantsur|afk is now known as dtantsur | 07:54 | |
| *** rpittau|afk is now known as rpittau | 07:58 | |
| openstackgerrit | James Page proposed openstack/nova stable/queens: hardware: fix memory check usage for small/large pages https://review.opendev.org/726867 | 07:59 |
| openstackgerrit | James Page proposed openstack/nova stable/queens: Fix overcommit for NUMA-based instances https://review.opendev.org/726868 | 07:59 |
| *** tony_ has quit IRC | 08:17 | |
| openstackgerrit | sean mooney proposed openstack/nova-specs master: move implemented specs in ussuri https://review.opendev.org/721278 | 08:19 |
| sean-k-mooney | gibi: sorry ment to do that last week ^ if you want any changes let me know and ill respin the patch | 08:19 |
| *** tony- has joined #openstack-nova | 08:23 | |
| huaqiang | stephenfin: There are ~20 patches in bp/use-pcpu-and-vcpu-in-one instance, for roles of py-checks, realtime policy, and fixes for hardware.py | 08:29 |
| huaqiang | these are not depending on each other so much | 08:29 |
| huaqiang | how about let me re-arrange the order | 08:29 |
| huaqiang | and resolving some dependencies with slight change to make it possible to be reviewed parallelly | 08:30 |
| *** ttsiouts has quit IRC | 08:32 | |
| *** ttsiouts_ has joined #openstack-nova | 08:32 | |
| openstackgerrit | Nalini Varshney proposed openstack/nova master: Add migration to make key field type VARBINARY in aggregate_metadata table, https://review.opendev.org/725522 | 08:32 |
| *** awalende_ has joined #openstack-nova | 08:32 | |
| *** salmankhan has joined #openstack-nova | 08:33 | |
| bauzas | gibi: impressive count on open bugs https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New | 08:35 |
| *** awalende has quit IRC | 08:35 | |
| bauzas | gibi: how have you triaged them ? | 08:35 |
| *** ttsiouts has joined #openstack-nova | 08:36 | |
| *** ttsiouts_ has quit IRC | 08:38 | |
| *** ttsiouts has quit IRC | 08:39 | |
| *** lpetrut_ has joined #openstack-nova | 08:39 | |
| *** ttsiouts has joined #openstack-nova | 08:39 | |
| *** lpetrut has quit IRC | 08:42 | |
| *** derekh has joined #openstack-nova | 08:43 | |
| gibi | bauzas: most of the did not have enough information | 08:48 |
| gibi | bauzas: and during the last two weeks we get only about 5 new bug reports so the low inflow helped a lot | 08:48 |
| *** tony- is now known as tony_su | 08:48 | |
| gibi | sean-k-mooney: thanks, I will check it | 08:49 |
| *** martinkennelly has joined #openstack-nova | 08:50 | |
| gibi | jkulik: what you want to achive? Do you want to move users from old HW gen to new HW gen over time? | 08:50 |
| stephenfin | huaqiang: Sure. The only issue with that is that future rebases are harder. If you can keep the existing +2s though, go for it | 08:51 |
| jkulik | gibi, we have different hardware versions in place at the same time, where, because of NUMA and whatnot, flavors differ slightly. We want a user to be able to deploy a "2 TB instance" flavor, even if that means 1.8 on one node and 2.1 on the other. | 08:52 |
| sean-k-mooney | jkulik: that is how it works by default | 08:53 |
| sean-k-mooney | unless you meen 1.8 TB on one and 2.1 TB on the other? in not sure what those number refer too | 08:54 |
| jkulik | sean-k-mooney, the flavor has to map to 1.8 TB on one hardware version and 2.1 TB on the other | 08:54 |
| sean-k-mooney | jkulik: ya that is not allowed | 08:55 |
| jkulik | sean-k-mooney, we just want our users to be able to specify "give me that 2 TB thingy" | 08:55 |
| sean-k-mooney | right but its a 1.8TB thing and a 2.1TB thing | 08:55 |
| sean-k-mooney | its not a 2TB thing in either case | 08:55 |
| jkulik | because it's hard for us to keep hypervisors empty enough to support that size, the user can't really know up-front what hardware is currently free | 08:55 |
| sean-k-mooney | you could use boot form volume and not provide any default root disk | 08:55 |
| jkulik | we're talking RAM here :/ | 08:56 |
| sean-k-mooney | oh i taught you ment disk | 08:56 |
| sean-k-mooney | and the user should know know up front how much ram is free | 08:56 |
| sean-k-mooney | *should not | 08:56 |
| sean-k-mooney | jkulik: it sound like you want to reconfigure the scheduler to pack | 08:57 |
| openstackgerrit | Tony Su proposed openstack/nova-specs master: Re-propose provider-config-file spec for Victoria https://review.opendev.org/725788 | 08:57 |
| jkulik | I don't see how we could tell the user what flavors could currently get deployed via nova's api | 08:57 |
| sean-k-mooney | if you configure the schduler to pack based on ram usage it will try to keep hosts free if it will fit on othet hosts | 08:57 |
| sean-k-mooney | jkulik: well you cant and that is kind fo intentional | 08:57 |
| jkulik | sean-k-mooney, the problem is even bigger for us: we use vmware .. so we have a cluster in the compute node and have to do some magic behind the scenes to convince vmware to spawn this monster | 08:57 |
| sean-k-mooney | it breaks the cloud model if the tenant has a view of the aviaable capacity | 08:58 |
| jkulik | yeah, that's why I wanted a flavor, that can match any of those free hosts, by having sub-flavors ;) | 08:58 |
| sean-k-mooney | jkulik: yep which is not going to happen | 08:59 |
| sean-k-mooney | we kind of had something like that at one point | 08:59 |
| sean-k-mooney | for ironic | 08:59 |
| sean-k-mooney | but that has been gone for a long time | 08:59 |
| jkulik | we'll have to work something out downstream then, thanks for clarifying :) | 08:59 |
| sean-k-mooney | im trying to think if we could use placmenet and custom resouces classes for this | 09:00 |
| sean-k-mooney | but what you are trying to do is going against the grain of how nova is intended to be used ot a degree | 09:00 |
| bauzas | gibi: ack thanks | 09:00 |
| *** priteau has joined #openstack-nova | 09:02 | |
| sean-k-mooney | jkulik: im not sure how configurable the vmware dirver is | 09:03 |
| sean-k-mooney | but do you want to reserve space for this giant vms or no? | 09:03 |
| bauzas | gibi: agreed with Wontfix https://bugs.launchpad.net/nova/+bug/1838309 ? | 09:04 |
| openstack | Launchpad bug 1838309 in OpenStack Compute (nova) "Live migration might fail when run after revert of previous live migration" [Undecided,Won't fix] - Assigned to Vladyslav Drok (vdrok) | 09:04 |
| sean-k-mooney | what you could do woudl be to reserve ram on the specifci hosts where you intend to spawn these isntances. then create a custom resouce class for them and a flavor that request 2TB or ram but set resocues:memory_mb=0 resouces:CUSTOM_GIANT_VM=1 | 09:04 |
| jkulik | sean-k-mooney, not in general. we currently run something, that frees up enough space on one note - so we can spawn one. | 09:05 |
| sean-k-mooney | ok so that wont work if you want it to be dynamic then | 09:05 |
| jkulik | yeah, we have something like that. but it has to fit into normal quota for billing | 09:05 |
| tony_su | gibi: This is Tony Su and I am doing provider-config-file re-propose thing. Could you kindly spare some time to review its spec (only minor changes in History and Assignee sections vs. Ussuri version). https://review.opendev.org/#/c/725788/ | 09:05 |
| *** grandchild has joined #openstack-nova | 09:05 | |
| sean-k-mooney | jkulik: well the flavor.ram value would still be 2TB | 09:06 |
| jkulik | sean-k-mooney, we dynamically create a sharing child-resource-provider for a node, that has the CUSTOM_GIANT_VM resource | 09:06 |
| *** salmankhan has quit IRC | 09:06 | |
| *** salmankhan has joined #openstack-nova | 09:07 | |
| *** ociuhandu has joined #openstack-nova | 09:07 | |
| *** ttsiouts_ has joined #openstack-nova | 09:07 | |
| sean-k-mooney | ok but the issue is you still need to aling the flavor.ram to what that means for that host | 09:07 |
| jkulik | sean-k-mooney, so currently, it works for us. but the flavor's RAM doesn't align with NUMA on all nodes. meaning, we would have to create more flavors. | 09:07 |
| sean-k-mooney | based on the hardware version | 09:07 |
| jkulik | yes | 09:07 |
| *** ociuhandu has quit IRC | 09:08 | |
| sean-k-mooney | jkulik: may i ask why you actully use VM instead of ironic nodes at that scale | 09:08 |
| jkulik | but given that we only free up one node, that could fit 2 TB either because it's 2 TB in size or because it's bigger, the 2 TB are more a rounded value the custerom might request | 09:08 |
| *** ociuhandu has joined #openstack-nova | 09:08 | |
| tony_su | gibi: Really appreciate your support and help. | 09:08 |
| jkulik | sean-k-mooney, sure. we were on ironic, but our customers would like to have the automatic failover when a node goes down, that vmware provides. also TCO. | 09:08 |
| *** ttsiouts has quit IRC | 09:09 | |
| *** ociuhandu_ has quit IRC | 09:10 | |
| sean-k-mooney | i see so they are relying on the plathform for failover rahter then using an orchestration layer like k8s to mage there application | 09:10 |
| *** mgariepy has quit IRC | 09:10 | |
| ikla | the controller and node doesn't need to have the same device in in for passthrough? | 09:10 |
| sean-k-mooney | so this is very much a pets not cattel usecase | 09:10 |
| jkulik | since the customer can't find out, which precise flavor she can currently deploy via API, it would have been nice to have specific sub-flavors for ~ 2 TB | 09:10 |
| gibi | bauzas, stephenfin: easy +2 https://review.opendev.org/#/c/721278 (ussuri spec move) | 09:10 |
| ikla | my flavor keeps failing when trying to turn on a instance | 09:10 |
| sean-k-mooney | ikla: the alias need to be on the computes and contoler | 09:11 |
| jkulik | sean-k-mooney, yes. definitely pets | 09:11 |
| sean-k-mooney | ikla: the pci whitelist is only needed on the compute | 09:11 |
| ikla | alias yes, device only on compute.. | 09:11 |
| ikla | k | 09:11 |
| sean-k-mooney | ikla: so the contole just uses it to translate the value in the flavor into pci request for schduling | 09:11 |
| gibi | bauzas: I agree with WontFix https://bugs.launchpad.net/nova/+bug/1838309 | 09:12 |
| openstack | Launchpad bug 1838309 in OpenStack Compute (nova) "Live migration might fail when run after revert of previous live migration" [Undecided,Won't fix] - Assigned to Vladyslav Drok (vdrok) | 09:12 |
| bauzas | cool, moving on | 09:12 |
| gibi | tony_su: thanks for taking that feature over. I will look at the spec soon | 09:13 |
| *** ociuhandu_ has joined #openstack-nova | 09:13 | |
| *** ociuhand_ has joined #openstack-nova | 09:14 | |
| *** mgariepy has joined #openstack-nova | 09:15 | |
| gibi | bauzas: thanks for looking at the new bugs. You have the bug lock, I have yet another downstream originated issue to look at https://bugs.launchpad.net/nova/+bug/1878024 | 09:15 |
| openstack | Launchpad bug 1878024 in OpenStack Compute (nova) "disk usage of the nova image cache is not counted as used disk space" [Undecided,Confirmed] | 09:15 |
| sean-k-mooney | gibi: thats a tricky one | 09:16 |
| sean-k-mooney | in that really we should never fail to boot a vm becasue of a cached image taking up space | 09:16 |
| *** ociuhandu has quit IRC | 09:17 | |
| sean-k-mooney | i.e. i would expect as an operator for the cache to be perurged first | 09:17 |
| sean-k-mooney | so im not sure i would want it to be counted as used | 09:17 |
| sean-k-mooney | that siad we dont purge the cache today as far as i know at least not while the image is used on the host | 09:17 |
| sean-k-mooney | so i can see this causeing issues too | 09:17 |
| sean-k-mooney | could you just workaround this using the host reserved disk paramater | 09:18 |
| *** ociuhandu has joined #openstack-nova | 09:18 | |
| *** ttsiouts_ has quit IRC | 09:18 | |
| *** ociuhand_ has quit IRC | 09:18 | |
| *** ociuhandu_ has quit IRC | 09:18 | |
| sean-k-mooney | or perhaps we need a config option to limit the size of the cache | 09:18 |
| *** ttsiouts has joined #openstack-nova | 09:19 | |
| gibi | sean-k-mooney: the cache is used as a backing file for qcow root fs so it cannot be really purged | 09:19 |
| sean-k-mooney | the easy option is just to add the size of the cache to the reserved value in placment but that will reduce the amount of vms that can be spawned | 09:19 |
| sean-k-mooney | gibi: i tought we had two copies | 09:19 |
| sean-k-mooney | on in the cache and a second for the backing file | 09:20 |
| sean-k-mooney | isnt the cach module indpentend of the image backend | 09:20 |
| gibi | sean-k-mooney: reserved_host_disk_mb would only be useful if we could limit the size of the cache | 09:20 |
| sean-k-mooney | yep the two line "fix" is to jsut count the cache size and added it to the reserved value in placment | 09:21 |
| sean-k-mooney | but i dont think that is a resonable long term fix | 09:21 |
| gibi | simply counting it as reserved does not help when a VM is booted with a new image to a compute. There we would need to make sure that both the new cached image and the VM root fs fits the compute | 09:22 |
| gibi | we discussed these options with dansmith yesterday | 09:22 |
| gibi | see the summary in the bug and the link to the IRC discussion | 09:23 |
| openstackgerrit | Merged openstack/nova-specs master: move implemented specs in ussuri https://review.opendev.org/721278 | 09:24 |
| sean-k-mooney | gibi: are we sure we use the cache image directly for the backing file by the way | 09:25 |
| sean-k-mooney | gibi: i tought that was how it worked but i remeber someone telling me its not how it worked and that we chage two copies one in the caceh and one for the backing file | 09:26 |
| gibi | let me double check | 09:26 |
| ikla | any info on gpu passthrough w/ rtx 8000's ? | 09:27 |
| huaqiang | stephenfin: I am not sure how to keep exisiting +2, will that be kept if I do not change the | 09:27 |
| sean-k-mooney | ikla: are you trying to do full passthough or are you trying to use its vgpu capablity | 09:27 |
| aarents | I think it is still the same directory | 09:27 |
| huaqiang | will the +2 be kepet if I donot change the 'Change-ID'? | 09:28 |
| gibi | sean-k-mooney: http://paste.openstack.org/show/793424/ | 09:28 |
| gibi | sean-k-mooney: based on this the image in the _base dir is the backing store for the instance's image | 09:29 |
| gibi | I have two instances from the same image, and have one backing image in the _base dir | 09:29 |
| sean-k-mooney | gibi: yep but is that the cached image or is it a second copy | 09:30 |
| sean-k-mooney | depenidn gon how we set the force cow and force raw values this might change | 09:30 |
| sean-k-mooney | in that case we are using a raw backing file | 09:31 |
| sean-k-mooney | but waht formate was the image originally? | 09:31 |
| sean-k-mooney | i assume qcow? | 09:31 |
| *** martinkennelly has quit IRC | 09:31 | |
| gibi | sean-k-mooney: a) it doesn't matter. I can reword the bug that the bacing file size is not counted as used b) I think this is the cache as if I delete the instances it does not deleted imediately | 09:31 |
| *** jhesketh has joined #openstack-nova | 09:31 | |
| bauzas | ikla: not sure I understand your question about gpu passthrough | 09:31 |
| gibi | sean-k-mooney: I will try to look into the case when the VM uses a raw file to see if then the image in _base is deleted, or if I can make a small change to be deleted | 09:33 |
| sean-k-mooney | gibi: there is a periodic that deletes the image when no vms is using it on the host as far as i am aware | 09:34 |
| gibi | sean-k-mooney: yes, that is the cache manager :) | 09:35 |
| sean-k-mooney | so until that runs the cached copy will be there | 09:35 |
| gibi | sean-k-mooney: anyhow the core of the problem is that nova uses more disk that it counts as used and today there is no way to avoid that except putting the _base dir on a separate partition | 09:35 |
| gibi | when we had the DiskFilter it had the disk_available_least information from the compute to prevent overallocating the disks but placement does not have such info | 09:36 |
| sean-k-mooney | well not entirely | 09:37 |
| ikla | grid vgpu | 09:37 |
| sean-k-mooney | the disk avaiable least behavior was still affected by the disk allocation ratio | 09:37 |
| bauzas | ikla: gpu passthrough != grid vgpu , you know ? | 09:38 |
| sean-k-mooney | gibi: i still go back to https://gist.github.com/JCallicoat/43505cab0535057ca4fb every time i want to figure that out | 09:38 |
| bauzas | in one case, you're litterally giving up the gpu device to the guest | 09:39 |
| ikla | sorry I didn't explain that correctly | 09:39 |
| ikla | you could do a full passthrough | 09:39 |
| bauzas | in the other case, you're asking the nvidia driver to slice your gpu into pieces that can be provided to the guests | 09:39 |
| ikla | or do vgpu | 09:39 |
| ikla | correct | 09:39 |
| sean-k-mooney | ikla: yes both are supportred | 09:39 |
| bauzas | so, again, what's your question ? | 09:39 |
| ikla | is there docs on it | 09:40 |
| sean-k-mooney | you can use mdev based vgpus or you can do direct passhtough of the gpu | 09:40 |
| bauzas | ikla: indeed | 09:40 |
| bauzas | https://docs.openstack.org/nova/latest/admin/virtual-gpu.html | 09:40 |
| ikla | I just found it | 09:40 |
| ikla | oh... thanks. :) | 09:40 |
| bauzas | https://docs.openstack.org/nova/latest/admin/pci-passthrough.html | 09:40 |
| bauzas | we don't provide special bits of nvidia installing and OS preparation | 09:40 |
| bauzas | but you can get'em from the nvidia grid docs | 09:41 |
| bauzas | eg. https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#red-hat-el-kvm-install-configure-vgpu | 09:42 |
| gibi | sean-k-mooney: I can reword. Assuming disk_allocation_ratio = 1.0. DiskFilter detected if you overallocated the disk and nova does that with the image cache. placement does not know if you overallocate the disk | 09:42 |
| sean-k-mooney | gibi: placement has the allication raitio in the inventory recored too fyi | 09:43 |
| sean-k-mooney | gibi: im not that worried about the wording to be honest | 09:43 |
| sean-k-mooney | gibi: im more interested in how you plan to adress it | 09:43 |
| gibi | sean-k-mooney: this is not the intentional disk allocaiton ratio see my assumption above. It is the unintentional disk overallocation due to image cache | 09:43 |
| gibi | sean-k-mooney: have you seen the options in the bug comments? | 09:44 |
| sean-k-mooney | yes | 09:44 |
| sean-k-mooney | i have | 09:44 |
| sean-k-mooney | options A and b we have said wont work for different reasons | 09:45 |
| *** Liang__ has quit IRC | 09:45 | |
| sean-k-mooney | so the only viable one would be disabel cache or one of the workarounds | 09:45 |
| sean-k-mooney | i guess B could work if instead of purgin the image when the cache is full we just dont cache new iamges | 09:45 |
| sean-k-mooney | A wont work becasue we could race with concurrent boot requests | 09:46 |
| sean-k-mooney | gibi: i have an option d | 09:46 |
| gibi | shoot | 09:46 |
| sean-k-mooney | gibi: if we have consumer types we could create allocations for cached images against the RP | 09:47 |
| sean-k-mooney | using a nova consumer type | 09:47 |
| sean-k-mooney | if we cant create the allcoation because there is not enough sapce then we dont cache it | 09:47 |
| sean-k-mooney | if we can the it will prevent the issue as all usage will be tracked in placment | 09:48 |
| gibi | we migt not need consumer types, we can simple create an allocation where the consumer_id is not an instance or a migration but the cache itself. | 09:49 |
| sean-k-mooney | ya i was thinking we could use the image uuid as the consumer uuid and a cache consumer type | 09:49 |
| sean-k-mooney | that way we would know which image are cached on each node | 09:50 |
| gibi | in the allocation candidate query we have to either include 1x disk space if the host already caching the image or 2x disk space if the host will cache the image due to the current request | 09:50 |
| sean-k-mooney | gibi: not if we change the logic so that the cacheing is optional | 09:51 |
| *** ociuhandu has quit IRC | 09:51 | |
| sean-k-mooney | e.g. have the compute node make the allcoation after the fact when teh node is about to be spawned | 09:51 |
| sean-k-mooney | so if the image is not already cached, chech can you create an allcoation for it, if so cache it and proceed with the boot if not dont cache it and just create a copy | 09:52 |
| ikla | can I request I set multiple names in pci_passthrough for the flavor? | 09:52 |
| sean-k-mooney | for the vm | 09:52 |
| sean-k-mooney | gibi: the only time that would not cache is when the disk is almost full | 09:53 |
| ikla | what would be the syntax or is it multiple pci_passthrough lines for the flavor? | 09:53 |
| gibi | sean-k-mooney: do you see the complexity of making the cache optional in nova code? yesterday we conculded with dansmith that it could be pretty hairy change | 09:54 |
| gibi | but I'm not an expert on the image backends | 09:54 |
| sean-k-mooney | gibi: i know that part of the code makes heavy use of functools.partil and is really hard to follow | 09:55 |
| sean-k-mooney | so i would guess it would be non trivial | 09:55 |
| sean-k-mooney | but mainly due to techdebt | 09:55 |
| openstackgerrit | Jiri Suchomel proposed openstack/nova-specs master: Add spec for downloading images via RBD https://review.opendev.org/572805 | 09:55 |
| ikla | something like: {"pci_passthrough:alias"="name1:1, name2:1, name3:1"} | 09:56 |
| *** ociuhandu has joined #openstack-nova | 09:57 | |
| sean-k-mooney | ikla: yes its a comma seperate list | 09:57 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/pci/request.py#L234-L237 | 09:57 |
| gibi | sean-k-mooney: thanks. I will look into this direction as well | 09:58 |
| sean-k-mooney | so openstack flavor set --property "pci_passthrough:alias"="name1:1,name2:1,name3:1" my-flavor | 09:58 |
| ikla | Insufficient compute resources: Claim pci failed. | 10:01 |
| ikla | hmm | 10:01 |
| sean-k-mooney | ikla: you might need to weaken the default numa affintiy policy | 10:02 |
| sean-k-mooney | e.g. if you dont have all the device on the same numa node | 10:02 |
| *** tesseract-RH has joined #openstack-nova | 10:02 | |
| sean-k-mooney | ikla: you can set numa_policy=perfer in the alisa | 10:03 |
| sean-k-mooney | i just need to check that value is correct but i think its prefer or prefered | 10:03 |
| *** tesseract has quit IRC | 10:04 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Address issues with flake8 3.8.0 https://review.opendev.org/727140 | 10:05 |
| sean-k-mooney | ikla: its "preferred" | 10:05 |
| stephenfin | gibi, bauzas: if that isn't failing our gate yet, it's going to start soon ^ | 10:05 |
| sean-k-mooney | ikla: https://github.com/openstack/nova/blob/master/nova/objects/fields.py#L734 | 10:05 |
| *** tetsuro has quit IRC | 10:05 | |
| sean-k-mooney | ikla: it could have failed for other reasons too | 10:05 |
| stephenfin | https://review.opendev.org/727133 will fix it but I don't know how long until that's released | 10:05 |
| stephenfin | *how long it will be | 10:05 |
| bauzas | wait | 10:06 |
| bauzas | stephenfin: doesn't that provide a new exception ? | 10:06 |
| bauzas | stephenfin: do you have more context ? | 10:07 |
| stephenfin | yes, E741 and F522 seem to be new | 10:07 |
| bauzas | so we could ignore them first ? | 10:07 |
| stephenfin | but...why? | 10:07 |
| ikla | set in flavor? | 10:07 |
| ikla | oh, in the alias? | 10:08 |
| ikla | I get the same issue with one device | 10:08 |
| stephenfin | the correct fix is for hacking to limit flake8 to a given minor version instead of the major version range it's using, but we need a new release of hacking for that | 10:08 |
| *** kashyap has joined #openstack-nova | 10:09 | |
| sean-k-mooney | ikla: are you testing with a gpu | 10:09 |
| sean-k-mooney | ikla: or do you have any specific errors int eh nova compute agent log | 10:10 |
| ikla | no, these are network cards | 10:11 |
| ikla | nothing in logs | 10:12 |
| sean-k-mooney | ok then likely you if its not the numa issue the next thing to check woudl be the pci device type | 10:12 |
| sean-k-mooney | do the nics support sriov? | 10:12 |
| ikla | yes | 10:13 |
| ikla | they are setup and I can see them in the pci list with lspci -nnn | 10:13 |
| sean-k-mooney | if they dont have the capablity of SRIOV then the type will be type-PCI | 10:13 |
| ikla | Virtual x 4 | 10:13 |
| sean-k-mooney | if they do then the PF will be type-PF and the VFs will be type-VF | 10:13 |
| sean-k-mooney | you need to match the alis to the type | 10:13 |
| ikla | thats what I did | 10:14 |
| sean-k-mooney | ikla: by the way you only use that pci aliase for nics if you dont want them to be used with neutron | 10:14 |
| ikla | yup | 10:15 |
| *** martinkennelly has joined #openstack-nova | 10:15 | |
| *** udesale_ has joined #openstack-nova | 10:18 | |
| sean-k-mooney | ok so your alias is something like this | 10:18 |
| sean-k-mooney | | [pci] | 10:19 |
| sean-k-mooney | | alias = '{ | 10:19 |
| sean-k-mooney | | "name": "QuickAssist", | 10:19 |
| sean-k-mooney | | "product_id": "0443", | 10:19 |
| sean-k-mooney | | "vendor_id": "8086", | 10:19 |
| sean-k-mooney | | "device_type": "type-vf", | 10:19 |
| sean-k-mooney | | "numa_policy": "preferred" | 10:19 |
| sean-k-mooney | | }' | 10:19 |
| *** rpittau is now known as rpittau|bbl | 10:19 | |
| *** ttsiouts_ has joined #openstack-nova | 10:19 | |
| *** ttsiouts has quit IRC | 10:19 | |
| sean-k-mooney | actully it shoudl be type-VF | 10:20 |
| gibi | tony_su: could you make a small fix in https://review.opendev.org/#/c/725788 then I will +2 it | 10:20 |
| *** dpawlik has quit IRC | 10:21 | |
| *** songwenping_ has quit IRC | 10:21 | |
| *** udesale has quit IRC | 10:21 | |
| bauzas | stephenfin: I mean, your change is a bit unclear | 10:21 |
| *** dpawlik has joined #openstack-nova | 10:21 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/stein: stable-only: skip volume backup tests in cellsv1 job https://review.opendev.org/727147 | 10:22 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/rocky: stable-only: skip volume backup tests in cellsv1 job https://review.opendev.org/727148 | 10:22 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: stable-only: skip volume backup tests in cellsv1 job https://review.opendev.org/727150 | 10:23 |
| bauzas | stephenfin: so, IMO, we should add a new change ignoring the new errors, then having your own change modifyng what what needed and then also removing the error ignored line | 10:24 |
| *** songwenping_ has joined #openstack-nova | 10:24 | |
| bauzas | stephenfin: so in case a new patch would be merged, we would still make sure we would ignore the new errors until you are sure that all of them are fixed | 10:25 |
| sean-k-mooney | do we even need to do that | 10:30 |
| sean-k-mooney | lets just cap it | 10:30 |
| sean-k-mooney | then props a patch to uncap it and a patch that depends on that in nova | 10:30 |
| sean-k-mooney | it should fail and we can iterate on that patch until it passes | 10:31 |
| stephenfin | or, you know, fix the five things that have changed and move on with our lives | 10:31 |
| stephenfin | these are the only changes necessary. I checked. I'll update the commit message shortly | 10:31 |
| sean-k-mooney | stephenfin: you didnt explain why any of the five things needed to be fixed | 10:31 |
| stephenfin | I'll update the commit message shortly | 10:31 |
| sean-k-mooney | and several fo them had #noqa on them | 10:31 |
| sean-k-mooney | so they should have been ignofred | 10:31 |
| stephenfin | they already had noqa | 10:31 |
| sean-k-mooney | yep | 10:31 |
| stephenfin | check my replies | 10:31 |
| sean-k-mooney | so it should not have been checking them | 10:31 |
| sean-k-mooney | stephenfin: https://www.flake8rules.com/rules/E741.html does not feel like we should have it on by default | 10:33 |
| *** ttsiouts_ has quit IRC | 10:34 | |
| *** ttsiouts has joined #openstack-nova | 10:34 | |
| * bauzas goes on lunch | 10:34 | |
| stephenfin | sean-k-mooney: I'd really rather avoid that argument because those tend to be ratholes in nova. I'd much, much rather we just took what flake8 and hacking gave us and dealt with it. | 10:35 |
| sean-k-mooney | arbitary rules like that one make code worse and less readable | 10:36 |
| stephenfin | arbitrary rules like you're *never* allowed exceed 80 characters? | 10:36 |
| sean-k-mooney | i agree it can be consuing a time but its not any worse then using any 1 lettter valiable | 10:37 |
| stephenfin | one man's arbitrary rule is another's good idea | 10:37 |
| sean-k-mooney | stephenfin: yes that has been demonstrated to make code less readable | 10:37 |
| sean-k-mooney | and pep8 enforece 79 | 10:37 |
| sean-k-mooney | not 80 | 10:37 |
| * gibi left comment in the flake8 patch and goes for lunch | 10:37 | |
| stephenfin | there is evidence to suggest otherwise https://www.youtube.com/watch?v=wf-BqAjZb8M&t=260 | 10:37 |
| stephenfin | https://black.readthedocs.io/en/stable/the_black_code_style.html#line-length would be an informative read | 10:38 |
| stephenfin | but this is exactly where I don't want to end up :D damn it | 10:38 |
| stephenfin | gibi: ta | 10:38 |
| sean-k-mooney | stephenfin: i have read the black style guide it argues against the 80 column limit | 10:39 |
| sean-k-mooney | stephenfin its why it used a 80ish limit rahter then a fixed value. | 10:40 |
| stephenfin | yup | 10:40 |
| stephenfin | to be clear, my argument is against the strict limit | 10:41 |
| stephenfin | 80ish is fine | 10:41 |
| stephenfin | hence the emphasis on *never* above | 10:41 |
| openstackgerrit | Brin Zhang proposed openstack/nova master: Optimize _create_and_bind_arqs logic in conducor https://review.opendev.org/726564 | 10:45 |
| sean-k-mooney | stephenfin: i left some more comments. some of the noqas canbe remvoed with light tweeks or we can need to document why. | 10:47 |
| * sean-k-mooney my typing is worse then usual today | 10:49 | |
| *** songwenping__ has joined #openstack-nova | 10:50 | |
| *** ttsiouts has quit IRC | 10:51 | |
| *** songwenping_ has quit IRC | 10:53 | |
| *** ttsiouts has joined #openstack-nova | 11:00 | |
| *** ttsiouts_ has joined #openstack-nova | 11:03 | |
| *** ttsiouts has quit IRC | 11:04 | |
| openstackgerrit | Qiu Fossen proposed openstack/nova-specs master: specify mac for creating instance https://review.opendev.org/700429 | 11:19 |
| *** samueldmq_ has joined #openstack-nova | 11:28 | |
| *** samueldmq_ has quit IRC | 11:34 | |
| *** samueldmq_ has joined #openstack-nova | 11:36 | |
| *** jsuchome has joined #openstack-nova | 11:37 | |
| *** samueldmq_ is now known as samueldmq | 11:40 | |
| openstackgerrit | Nalini Varshney proposed openstack/nova master: Add migration to make key field type VARBINARY in aggregate_metadata table, https://review.opendev.org/725522 | 11:40 |
| *** raildo has joined #openstack-nova | 11:44 | |
| openstackgerrit | sean mooney proposed openstack/nova master: add workaround to disable multiple port bindings https://review.opendev.org/724386 | 11:45 |
| openstackgerrit | sean mooney proposed openstack/nova master: [DNM] testing with force_legacy_port_binding workaround https://review.opendev.org/724387 | 11:45 |
| *** nweinber has joined #openstack-nova | 11:52 | |
| openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.reraise https://review.opendev.org/726898 | 11:57 |
| huaqiang | stephenfin: Maybe I should drop my thought of reordering your bp/use-pcpu-and-vpuc-in-one-instance patches. My original thought was the followings patches could be acceleated since my patches donot depend on all the features introduced in the proceeding patches. | 12:06 |
| huaqiang | But the reorder will bring a lot of extra work | 12:06 |
| huaqiang | I'll rebase my pathces by following your patches. | 12:07 |
| *** ratailor has quit IRC | 12:07 | |
| *** vishalmanchanda has quit IRC | 12:07 | |
| *** sapd1_x has quit IRC | 12:08 | |
| openstackgerrit | Takashi Natsume proposed openstack/os-vif master: Remove egg_info in setup.cfg https://review.opendev.org/727173 | 12:10 |
| *** ttsiouts_ has quit IRC | 12:20 | |
| *** ratailor has joined #openstack-nova | 12:20 | |
| *** ttsiouts has joined #openstack-nova | 12:21 | |
| *** rpittau|bbl is now known as rpittau | 12:21 | |
| *** adrianc has quit IRC | 12:24 | |
| *** ttsiouts_ has joined #openstack-nova | 12:31 | |
| *** ttsiouts has quit IRC | 12:31 | |
| *** ratailor has quit IRC | 12:33 | |
| *** happyhemant has joined #openstack-nova | 12:36 | |
| *** ttsiouts_ has quit IRC | 12:38 | |
| *** ttsiouts has joined #openstack-nova | 12:38 | |
| *** belmoreira has quit IRC | 12:41 | |
| *** vishalmanchanda has joined #openstack-nova | 12:41 | |
| *** ttsiouts has quit IRC | 12:42 | |
| *** elod_pto is now known as elod | 12:42 | |
| *** belmoreira has joined #openstack-nova | 12:44 | |
| *** adrianc has joined #openstack-nova | 12:45 | |
| *** adrianc has quit IRC | 12:50 | |
| *** songwenping_ has joined #openstack-nova | 12:51 | |
| *** adrianc has joined #openstack-nova | 12:53 | |
| *** songwenping__ has quit IRC | 12:54 | |
| *** lbragstad has joined #openstack-nova | 12:56 | |
| *** dklyle has joined #openstack-nova | 13:00 | |
| openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.reraise https://review.opendev.org/726898 | 13:05 |
| *** ttsiouts has joined #openstack-nova | 13:06 | |
| *** ttsiouts has quit IRC | 13:21 | |
| *** ttsiouts has joined #openstack-nova | 13:21 | |
| *** avolkov has joined #openstack-nova | 13:25 | |
| *** ttsiouts has quit IRC | 13:25 | |
| *** ttsiouts has joined #openstack-nova | 13:26 | |
| *** spotz has joined #openstack-nova | 13:27 | |
| *** lbragstad_ has joined #openstack-nova | 13:30 | |
| *** lbragstad has quit IRC | 13:33 | |
| *** redrobot has quit IRC | 13:39 | |
| *** songwenping_ has quit IRC | 13:46 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/stein: stable-only: skip volume backup tests in cellsv1 job https://review.opendev.org/727147 | 13:46 |
| *** songwenping_ has joined #openstack-nova | 13:47 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/rocky: stable-only: skip volume backup tests in cellsv1 job https://review.opendev.org/727148 | 13:47 |
| dansmith | jsuchome: hey, I +2d that spec and then realized you forgot to address one critical piece of feedback from the last version | 13:48 |
| dansmith | jsuchome: *other* than that, I was going to be happy with it :) | 13:48 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: stable-only: skip volume backup tests in cellsv1 job https://review.opendev.org/727150 | 13:48 |
| *** openstackstatus has quit IRC | 13:53 | |
| *** openstackstatus has joined #openstack-nova | 13:54 | |
| *** ChanServ sets mode: +v openstackstatus | 13:54 | |
| jsuchome | dansmith: I see, I messed up the test part. And the other part I did not address intentionally, I'm not sure it deserves detailed explanation in the spec | 13:59 |
| jsuchome | dansmith: the 'significant changes' in glance.py are really only about executing extra download method under given conditions. And I think this is already covered | 13:59 |
| *** mriedem has joined #openstack-nova | 13:59 | |
| *** brinzhang_ has joined #openstack-nova | 14:01 | |
| *** songwenping__ has joined #openstack-nova | 14:01 | |
| dansmith | jsuchome: you're changing the base existing glance code more than just calling the rbd function and returning the way the plug point used to work, so I think it's pretty relevant | 14:01 |
| dansmith | jsuchome: there's also no documentation about it in the code change, so I don't even know what the goal of the change is | 14:02 |
| dansmith | jsuchome: it does't need to be detailed, but it should be something, IMHO. | 14:03 |
| jsuchome | I tried to explain it already but now I see I forgot to post my changes ... | 14:04 |
| *** songwenping_ has quit IRC | 14:04 | |
| *** brinzhang has quit IRC | 14:04 | |
| jsuchome | dansmith: I actually added some comment to that part you are (probably?) mentioning, I assume you mean "Load chunks from the downloaded image file... " bit | 14:05 |
| jsuchome | I've added some comments to the spec | 14:05 |
| dansmith | yeah, so re-reading the code this morning, it looks like we used to not do the image signature verification for things downloaded with the per-scheme handler, | 14:05 |
| jsuchome | (I mean: I've added some comments 1. to the code and 2. now I've also commented the spec) | 14:06 |
| dansmith | and this change is lining it up so we do right? | 14:06 |
| *** brinzhang_ has quit IRC | 14:06 | |
| jsuchome | yeah, in previous version, this signature verification was just skipped if there was anything downloaded by the download handler | 14:06 |
| jsuchome | So you think it still should be mentioned in the specs? | 14:07 |
| jsuchome | Maybe I could just mention it in the commit message of the code change | 14:08 |
| dansmith | yeah, so just a line in the spec under proposed change like this is fine: "The glance module also never used to perform image signature verification when the per-scheme module was used. Since we are moving this into core code, we will also fix this so that per-scheme images are verified like all the rest." | 14:08 |
| dansmith | jsuchome: please just add that one line (or something like it) to the spec when you fix the test thing and we can move on | 14:08 |
| dansmith | it should also go into the code change commit message, btw | 14:09 |
| jsuchome | OK, both places then. In a minute | 14:09 |
| dansmith | honestly, if I was doing it, I would break the code into two pieces, one for that fix and one for the rbd module being added | 14:10 |
| dansmith | but, it's already done, so. | 14:10 |
| dansmith | (biab) | 14:12 |
| jsuchome | yeah, it's not exactly part of the feature. Seems like the original author realized it during testing, it appeared in some later PS | 14:12 |
| *** sapd1_x has joined #openstack-nova | 14:12 | |
| *** adrianc has quit IRC | 14:15 | |
| *** ttsiouts has quit IRC | 14:16 | |
| *** adrianc has joined #openstack-nova | 14:17 | |
| openstackgerrit | Jiri Suchomel proposed openstack/nova-specs master: Add spec for downloading images via RBD https://review.opendev.org/572805 | 14:18 |
| openstackgerrit | Jiri Suchomel proposed openstack/nova master: Add ability to download Glance images into the libvirt image cache via RBD https://review.opendev.org/574301 | 14:18 |
| *** ttsiouts has joined #openstack-nova | 14:22 | |
| *** mgariepy has quit IRC | 14:31 | |
| *** ttsiouts_ has joined #openstack-nova | 14:31 | |
| dansmith | jsuchome: +2 on the spec, thanks | 14:32 |
| jsuchome | cool | 14:32 |
| *** ttsiouts has quit IRC | 14:35 | |
| *** tkajinam has quit IRC | 14:37 | |
| *** ralonsoh has quit IRC | 14:40 | |
| *** ralonsoh has joined #openstack-nova | 14:40 | |
| *** ttsiouts_ has quit IRC | 14:41 | |
| *** ttsiouts has joined #openstack-nova | 14:41 | |
| *** mgariepy has joined #openstack-nova | 14:44 | |
| *** ralonsoh has quit IRC | 14:44 | |
| *** ttsiouts has quit IRC | 14:46 | |
| jsuchome | dansmith: two questions: 1. should I add that admin guide change into the same PS as the one with code or rather a new one? 2. where is that ceph CI job I should try to copy& adapt? (I've never done this part before) | 14:48 |
| *** rcernin has quit IRC | 14:48 | |
| dansmith | jsuchome: no, do it in a separate patch please | 14:48 |
| dansmith | jsuchome: I'm not super up on the state of the job (i.e. whether it's a legacy or converted job), nor where those bits live depending | 14:49 |
| dansmith | but I bet sean-k-mooney knows | 14:49 |
| sean-k-mooney | which job | 14:50 |
| sean-k-mooney | ceph | 14:50 |
| sean-k-mooney | i think its converted but ill check | 14:50 |
| openstackgerrit | Ghanshyam Mann proposed openstack/python-novaclient master: Bump hacking min version to 3.0.1 https://review.opendev.org/727214 | 14:51 |
| sean-k-mooney | devstack-plugin-ceph-tempest-py3 i think is a zullv3 job. we are not defining any legacy playbooks for ti but it comes form devstack not zuul config | 14:52 |
| dansmith | sean-k-mooney: jsuchome needs to take that job, tweak the config on the compute node slightly, and get a one-off run of it at least | 14:52 |
| sean-k-mooney | dansmith: ah ok ill jsut triple check that its zuulv3 if so that is simpel to do | 14:52 |
| gmann | that is zuulv3, derived from tempest-full-py3. | 14:54 |
| lyarwood | https://review.opendev.org/#/c/708038/ is an example of me messing around with the ceph job recently | 14:54 |
| sean-k-mooney | yep its zuul v3 https://github.com/openstack/devstack-plugin-ceph/blob/master/.zuul.yaml#L57-L130 | 14:54 |
| *** links has quit IRC | 14:55 | |
| sean-k-mooney | jsuchome: what sepcfically do you need to add | 14:55 |
| dansmith | lyarwood: ah nice | 14:55 |
| dansmith | might need a DNM change against devstack to hack the half behavior into place and then control it with something like lyarwood's example | 14:56 |
| sean-k-mooney | dansmith: you should not need too you can create a job that uses it as a parrent and then add your changes | 14:56 |
| dansmith | sean-k-mooney: he needs to set up all the ceph stuff, but configure the compute to *not* use rbd backend, and have a new conf option set to enable direct-from-ceph download | 14:56 |
| sean-k-mooney | so jsuchome just need to override the imagebackend in the nova.conf to be qcow2 | 14:57 |
| dansmith | sean-k-mooney: depends on how the ceph bit works in devstack right? in lyarwood's example above, he had https://review.opendev.org/#/c/708035/ for that reason I think | 14:57 |
| sean-k-mooney | e.g. lev devstack and the plugin do its thing but just tell nova not to use it | 14:58 |
| dansmith | sean-k-mooney: probably enough, as long as that sticks and devstack or something downstream doesn't override | 14:58 |
| sean-k-mooney | ceph is set up by https://github.com/openstack/devstack-plugin-ceph | 14:58 |
| sean-k-mooney | but if jsuchome does something like https://review.opendev.org/#/c/724387/3/.zuul.yaml | 14:58 |
| sean-k-mooney | which is using the local.conf [[post-config:/etc/nova/nova.conf]] mechanism to set config options that will run after the plugin | 14:59 |
| dansmith | sean-k-mooney: will that override what the plugin does? | 14:59 |
| sean-k-mooney | ya i belive the order is intree modules then plugins then post-config form local.conf | 15:00 |
| lyarwood | wait, to disable ceph on Nova that plugin has a few variables you can set in the job | 15:00 |
| lyarwood | ENABLE_CEPH_NOVA=false etc iirc | 15:00 |
| sean-k-mooney | that would deploy it for cinder only then? | 15:00 |
| lyarwood | glance etc | 15:00 |
| sean-k-mooney | or glance i guess | 15:00 |
| lyarwood | I think that's the point of this test | 15:00 |
| dansmith | needs to be set for glance, yeah | 15:01 |
| lyarwood | but that might be missing the required creds to download over RBD that I assume is the point here | 15:01 |
| sean-k-mooney | ya so that sound like it would work | 15:01 |
| dansmith | right | 15:01 |
| dansmith | he still needs all the regular rbd config, just not the imagebackend part | 15:01 |
| sean-k-mooney | ya so basicaly what lyarwood example is doing is exactly what needs to be done. | 15:02 |
| dansmith | yup | 15:02 |
| sean-k-mooney | is jsuchome about | 15:02 |
| sean-k-mooney | if not i can submit a ptach that does that but i dont know what to put it on top | 15:02 |
| dansmith | hang on | 15:02 |
| jsuchome | sean-k-mooney: tweak few options, mostly for nova and one for glance | 15:03 |
| dansmith | sean-k-mooney: https://review.opendev.org/#/c/574301/ | 15:03 |
| jsuchome | (sorry I wasn't following for a while) | 15:03 |
| sean-k-mooney | clip notes the devstack-ceph plugin has an env varible that you can set to disable just the nova change it does | 15:04 |
| sean-k-mooney | so you can have it deploy cpeh for just glance and cinder | 15:04 |
| sean-k-mooney | so that should be easy to do im not sure what else you need to set but there is a simple way to set config options we can use | 15:05 |
| sean-k-mooney | if you need to do something more involved then you need to use a pre playbook to configure the jobs properly or a local.sh script | 15:05 |
| jsuchome | I think I need to test if nova correctly downloads image and spawns VM, and I need ceph as glance backend, and few options for glance and nova set for that | 15:06 |
| dansmith | jsuchome: what I'm asking for is to just configure fordirect download, make sure tempest runs normally, and ten we can look at the logs to convince ourselves that it's working | 15:07 |
| sean-k-mooney | ok ill put up a DNM patch that disablel the unnned jobs and just disables using ceph for nova | 15:07 |
| sean-k-mooney | then we can tweak that | 15:07 |
| dansmith | jsuchome: not a direct test that confirms it was somehow downloaded direct -- that would be very hard | 15:07 |
| jsuchome | true, tempest run should indeed be enough | 15:08 |
| *** dtantsur is now known as dtantsur|afk | 15:08 | |
| dansmith | (and logs) | 15:08 |
| dansmith | jsuchome: if you think that you need to add some logs to be able to validate it from the outside, you should go ahead and do that | 15:08 |
| dansmith | jsuchome: perhaps a LOG.debug('Downloading %(image)s from per-scheme handler %(handler)s') that we can look for | 15:09 |
| jsuchome | There's already "Successfully transferred using rbd" log line | 15:11 |
| dansmith | okay cool | 15:11 |
| jsuchome | LOG.info I beleive | 15:11 |
| dansmith | jsuchome: ah you mean the "attempting to export" one | 15:12 |
| dansmith | oh, no, | 15:12 |
| dansmith | I see that one too | 15:12 |
| dansmith | okay cool, should be covered and easy to validate | 15:12 |
| *** dasp_ has quit IRC | 15:14 | |
| *** dasp has joined #openstack-nova | 15:16 | |
| openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.moves https://review.opendev.org/727224 | 15:17 |
| lyarwood | jsuchome / sean-k-mooney ; https://review.opendev.org/727225 might work | 15:18 |
| sean-k-mooney | dansmith: so before i push this | 15:18 |
| lyarwood | ah sorry you're also working on it | 15:18 |
| lyarwood | I didn't think you were | 15:18 |
| sean-k-mooney | that is updating the plugin | 15:18 |
| sean-k-mooney | im working on a patch to do it for nova | 15:18 |
| lyarwood | it's the same either way | 15:19 |
| sean-k-mooney | did you need to make those nova con changes too | 15:19 |
| lyarwood | well no, you'd just use the job in Nova | 15:19 |
| lyarwood | this change already pulls in the Nova changes | 15:19 |
| dansmith | jsuchome: I dunno where you got so much karma, but enjoy this overly exuberant clamor to help while you can :D | 15:19 |
| lyarwood | I'm just trying to hack around with zuul as much as I can at the moment | 15:20 |
| lyarwood | I've still got the live migrastion job migration to zuulv3 in my backlog | 15:20 |
| lyarwood | migration* | 15:20 |
| tosky | lyarwood: is there something specific in that job compared to a "usual" devstack job? Maybe I can help | 15:20 |
| lyarwood | tosky: the LM jobs? | 15:21 |
| tosky | lyarwood: that one (and in general any job that requires to move from legacy to native zuulv3) | 15:21 |
| *** tesseract-RH has quit IRC | 15:22 | |
| *** belmoreira has quit IRC | 15:22 | |
| lyarwood | tosky: http://lists.openstack.org/pipermail/openstack-discuss/2020-March/013207.html covers some of it, it's currently three jobs in one | 15:23 |
| *** tesseract has joined #openstack-nova | 15:23 | |
| lyarwood | tosky: I ran out of time in U to break things up but feel free to poke the changes early in V if you have time before I get around to it | 15:23 |
| tosky | lyarwood: sure - I will ping everyone with pending legacy jobs as part of the work on the community goal | 15:24 |
| openstackgerrit | sean mooney proposed openstack/nova master: [DNM] ceph direct download testing https://review.opendev.org/727228 | 15:25 |
| sean-k-mooney | ok well that is just push to have it there. it will do the excat same thing as lyarwood version | 15:25 |
| sean-k-mooney | its just in nova rather then the devstack plugin | 15:25 |
| lyarwood | tosky: ack understood thanks :) | 15:26 |
| *** mlavalle has joined #openstack-nova | 15:27 | |
| jsuchome | lyarwood: sean-k-mooney: thanks a lot ... so, where exactly do I set those nova+glance config values? under $NOVA_CPU_CONF for nova? And the one for glance? | 15:31 |
| openstackgerrit | Jiri Suchomel proposed openstack/nova-specs master: Add spec for downloading images via RBD https://review.opendev.org/572805 | 15:32 |
| lyarwood | jsuchome: glance should be configured automatically, we only had to add the two specific configurables your change is using in Nova as we disabled the configuration of Nova by the ceph devstack plugin. | 15:35 |
| lyarwood | just corrected a mistake in my change, I'll check back in on it later. | 15:36 |
| *** Guest49293 has joined #openstack-nova | 15:37 | |
| jsuchome | lyarwood: for glance I need DEFAULT.show_image_direct_url=true, are you telling me this is set by default? | 15:38 |
| *** Guest49293 is now known as redrobot | 15:38 | |
| sean-k-mooney | im not sure its in the base job | 15:38 |
| sean-k-mooney | so we might need to set that | 15:38 |
| jsuchome | that's what I would expect | 15:38 |
| dansmith | jsuchome: I think it must be, otherwise nova wouldn't be able to tell that the image is in the same rbd it is configured for, with the rbd backend | 15:38 |
| jsuchome | I see, so you mean it already _is_ default for ceph jobs? | 15:39 |
| dansmith | i.e. we already look at the locations field of the image, that's what you need right? | 15:39 |
| dansmith | jsuchome: I expect | 15:39 |
| sean-k-mooney | dansmith: we only have one cluser set up by the plugin so that would not be an issue we would see in teh gate | 15:39 |
| dansmith | sean-k-mooney: if we look at the locations and require one that matches, it would | 15:39 |
| jsuchome | yes, that's for location | 15:39 |
| sean-k-mooney | i.e. it will always be the same | 15:39 |
| dansmith | but I'm looking | 15:40 |
| sean-k-mooney | i mean its not hard to add but ill chekc the parent jobs and see if we are doing it or not | 15:40 |
| sean-k-mooney | or actully the plugin is likely where that would be set | 15:41 |
| dansmith | https://zuul.opendev.org/t/openstack/build/fbd122f546684ffebdca5f2f73b6167c/log/controller/logs/etc/glance/glance-api_conf.txt | 15:41 |
| jsuchome | lyarwood: sean-k-mooney: also this for nova conf: glance.allowed_direct_url_schemes = rbd ... as this is (was! and should not be anymore) deprecated option, I would not expect it to be set | 15:41 |
| *** dasp has quit IRC | 15:41 | |
| dansmith | show_multiple_locations = True | 15:41 |
| sean-k-mooney | https://github.com/openstack/devstack-plugin-ceph/commit/62ea04c8d180c5419300ddc7784c5c46f9fcbdad | 15:41 |
| dansmith | I think ^ is what we need for the check we do | 15:42 |
| dansmith | not sure the difference between that and the show-direct-url one | 15:42 |
| dansmith | because, AFAIK, the locations are the direct url | 15:42 |
| *** dasp has joined #openstack-nova | 15:42 | |
| dansmith | ah, | 15:43 |
| jsuchome | so show_multiple_locations also implies the location info is present? | 15:43 |
| dansmith | that commit implies that one impliesthe other | 15:43 |
| dansmith | yeah | 15:43 |
| sean-k-mooney | lyarwood's job is curerntly running https://zuul.openstack.org/stream/9b459dcfc0cd46d0a13ce5a5a1be2afe?logfile=console.log so we will know one way or another in about an hour | 15:45 |
| sean-k-mooney | maybe too | 15:45 |
| sean-k-mooney | *two | 15:46 |
| sean-k-mooney | but if we need to add anything else we can do that quickly once its done | 15:46 |
| jsuchome | well that allowed_direct_url_schemes is a must, we only fire the new download handler if it is set | 15:47 |
| sean-k-mooney | so we need that in nova i can see if that is set by the plugin | 15:47 |
| dansmith | it won't be | 15:48 |
| dansmith | I thought lyarwood did it in his change | 15:48 |
| lyarwood | sorry I missed that | 15:48 |
| lyarwood | [glance]/allowed_direct_url_schemes=['rbd']? | 15:48 |
| sean-k-mooney | ya i was about to ask the same https://opendev.org/openstack/nova/src/branch/master/nova/conf/glance.py#L64 | 15:49 |
| dansmith | lyarwood: not sure what group, but yes, hang on | 15:49 |
| sean-k-mooney | its in the glance group | 15:49 |
| jsuchome | yep | 15:49 |
| dansmith | if 'rbd' in CONF.glance.allowed_direct_url_schemes: | 15:50 |
| sean-k-mooney | do we plan to turn this on by default | 15:50 |
| dansmith | no | 15:50 |
| jsuchome | it should be documented | 15:50 |
| sean-k-mooney | ok we proably should remove the deprecation of that option in jsuchome patch | 15:50 |
| sean-k-mooney | i assume it already does that | 15:51 |
| dansmith | the spec says we will undeprecate it yes | 15:51 |
| sean-k-mooney | k | 15:51 |
| *** lpetrut_ has quit IRC | 15:51 | |
| jsuchome | does it need a release note? | 15:51 |
| dansmith | yes | 15:51 |
| sean-k-mooney | the feature would even without the undeprecation | 15:51 |
| sean-k-mooney | the same one can cover both | 15:52 |
| jsuchome | ok, than it's another change for 574301 | 15:52 |
| dansmith | jsuchome: I probably wouldn't pile that in there personally | 15:53 |
| *** maciejjozefczyk has quit IRC | 15:53 | |
| *** gyee has joined #openstack-nova | 15:56 | |
| jsuchome | OK, another patch, no problem | 15:56 |
| dansmith | jsuchome: at least for the moment, we can always squash | 15:57 |
| dansmith | I think that by the time you get all the test stuff in this patch it will be plenty meaty | 15:57 |
| dansmith | one could argue that it could go in my first patch to remove the plug point, but that just means it has two semi-related atomic changes | 15:57 |
| dansmith | patches are cheap | 15:57 |
| jsuchome | yeah, and for tommorrow I got to work on the tests, I can see they are not enough | 15:58 |
| dansmith | cool | 15:58 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: WIP: allow disabling image cache for raw images https://review.opendev.org/727261 | 16:09 |
| gibi | dansmith, sean-k-mooney: I looked at how easy is to disable the image cache and this is my first stab on it (seem to work in devstack) ^^ | 16:09 |
| gibi | I will have to disappear now but feedback is appreciated | 16:10 |
| *** rpittau is now known as rpittau|afk | 16:10 | |
| dansmith | gibi: I don't understand that | 16:11 |
| *** tesseract has quit IRC | 16:12 | |
| sean-k-mooney | dansmith: i assumed it was just me :) | 16:13 |
| sean-k-mooney | also i expected to not underdstand that when looking at that code | 16:14 |
| dansmith | yeah | 16:18 |
| dansmith | I'll have to get all dug-in to that code again to really be able to speak intelligently about it, | 16:19 |
| dansmith | but that surely seems to be basically doing the same thing as above at first glance | 16:19 |
| dansmith | sean-k-mooney: ah, maybe because this is the Flat implementation | 16:22 |
| dansmith | but I really thought that it still cached even though it flattened the image before giving it to the instance, which this doesn't seem to change | 16:22 |
| sean-k-mooney | dansmith: it is cached | 16:23 |
| sean-k-mooney | we copy it | 16:23 |
| sean-k-mooney | on line 598 | 16:23 |
| sean-k-mooney | https://review.opendev.org/#/c/727261/1/nova/virt/libvirt/imagebackend.py@598 | 16:24 |
| dansmith | yeah | 16:24 |
| dansmith | I wonder if gibi tested this and we're missing something, or he's assuming something else | 16:24 |
| sean-k-mooney | well this would also have to be done for qcow right | 16:24 |
| dansmith | sean-k-mooney: we only copy if it doesn't exist, and he's passing self.path as the target | 16:24 |
| dansmith | but, so does the "if generating" case above | 16:24 |
| sean-k-mooney | right so if self.path whcih shoudl be the imnstance disk path does not exist | 16:25 |
| dansmith | so I wonder if we're normally in Flat to grab the base image, and this makes for another case where we just download the image to our target when not caching and that somehow bypasses, | 16:25 |
| *** songwenping_ has joined #openstack-nova | 16:25 | |
| sean-k-mooney | before we would take the else path and create a copy form the base path | 16:25 |
| dansmith | but I think the call path to the caching is too loopy to tell that | 16:25 |
| sean-k-mooney | i think this will actully work but only for the flat backend | 16:26 |
| dansmith | disabling the image cache entirely is also somewhat of a way-too-big hammer to solve this problem | 16:26 |
| dansmith | it's a workaround maybe, but it's really a terrible one | 16:26 |
| sean-k-mooney | ya | 16:26 |
| sean-k-mooney | did you see my converstaion with gibi this morining | 16:26 |
| dansmith | I did | 16:26 |
| dansmith | well, I skimmed it | 16:26 |
| sean-k-mooney | cool what do you think of the idea of creating allocation for the cached images inplacment | 16:27 |
| dansmith | sean-k-mooney: I suggested that yesterday | 16:27 |
| sean-k-mooney | so we can track how much is being used | 16:27 |
| sean-k-mooney | ok cool | 16:27 |
| dansmith | we have to be able to do it separately if the cache is on a different filesystem though, | 16:27 |
| dansmith | which will get messy, especially if you move it later | 16:27 |
| sean-k-mooney | ya | 16:27 |
| dansmith | and we'll need new healing stuff to make sure we can recover from getting out of sync | 16:27 |
| sean-k-mooney | and if people put the cache on nfs or somehting it will be even worse | 16:28 |
| dansmith | yes | 16:28 |
| *** songwenping__ has quit IRC | 16:28 | |
| dansmith | so yeah, it's a thing, but it's not trivial for sure | 16:28 |
| sean-k-mooney | when you say self heling are you thinking nova audit or a periodic taks in the agent | 16:28 |
| dansmith | yeah, just anything that causes us to leak some allocations for images that are no longer cached, | 16:29 |
| dansmith | and also notice that new images are on disk that aren't allocated | 16:29 |
| dansmith | because we already tell customers they can sideload images into the disk cache and nova will (rightly) use them propery | 16:29 |
| dansmith | but wouldn't have alocations for them | 16:29 |
| sean-k-mooney | dansmith: ideally i was think as well that it would make the caching best efffort. e.g. it would check if it was in the cache. if not check if it can create an allcotion for the image and cache it if it can and dont cache if it cant | 16:29 |
| dansmith | yes, but if it can't then it has to delete that allocation of course, | 16:30 |
| dansmith | and if everything goes nuts during that, we have to be able to heal away those stale ones when we reboot | 16:30 |
| sean-k-mooney | dansmith: do we actully support the sideloading or just it should work | 16:30 |
| dansmith | and if images get sideloaded, we have to notice, and if images get locally deleted, we have to notice | 16:30 |
| dansmith | sean-k-mooney: it does work and we (redhat) have prescribed it in a few cases :) | 16:30 |
| sean-k-mooney | .... im glad you have added the image cacahing feature in teh api to have an alternitive noew | 16:31 |
| dansmith | and I know people have in the past manually purged images from that cache before the timer fires | 16:31 |
| *** ccamacho has quit IRC | 16:31 | |
| dansmith | sean-k-mooney: yep, but that only works (by design) per aggregate, and doesn't let you purge, | 16:31 |
| sean-k-mooney | ya | 16:31 |
| sean-k-mooney | sitll an improvement | 16:32 |
| dansmith | so I think it would be foolish and fragile to not be able to reconcile the state of the disk with the other system | 16:32 |
| dansmith | for sure | 16:32 |
| sean-k-mooney | would you be ok with the compute agent calling placment to make those allocaitons and clean them up. i think it should be fine since its already updating placment in the update_avaiable_resouces funtion just not sure if i missed anything | 16:34 |
| sean-k-mooney | the compute node already needs to be able to reach the placmenet api so it really should not be much of a change in that regard | 16:35 |
| *** ttsiouts has joined #openstack-nova | 16:35 | |
| *** evrardjp has quit IRC | 16:36 | |
| *** evrardjp has joined #openstack-nova | 16:36 | |
| dansmith | the compute node the _only_ thing that _could_ do it... | 16:37 |
| sean-k-mooney | well normally we do the allocation candiate request in the conductor right and we claim the allocation before we get to the compute | 16:39 |
| sean-k-mooney | so the second allcotion for the cache image would have to be done on the ocmpute node and the claim | 16:40 |
| *** dpawlik has quit IRC | 16:40 | |
| sean-k-mooney | but ya the compute node is really the only thing that could keep them in sync | 16:40 |
| dansmith | but nothing outside the compute node knows about the state of the cache | 16:41 |
| *** udesale_ has quit IRC | 16:41 | |
| dansmith | so nothing else would have any idea if an image is cached or could be cached | 16:41 |
| sean-k-mooney | yep | 16:41 |
| sean-k-mooney | and even if they did it would be racey | 16:41 |
| dansmith | you mean "even if they tried to guess" :) | 16:41 |
| sean-k-mooney | so that means the healing task could not be part of nova audit | 16:41 |
| dansmith | no | 16:42 |
| sean-k-mooney | it would have to be in the compute manager which is fine | 16:42 |
| dansmith | image cache management is kinda weirdly split between the compute manager and virt driver | 16:43 |
| dansmith | which means we probably need some extra stuff between them I think | 16:43 |
| dansmith | I'd have to go look | 16:43 |
| dansmith | this *would* be a pretty heavyweight new addition, to be clear | 16:43 |
| dansmith | it's unfortunate that we'd need to do this, IMHO, given the complexity required | 16:44 |
| sean-k-mooney | ya | 16:44 |
| sean-k-mooney | do you see another path forward beyound just disabling hte cache | 16:44 |
| dansmith | so we need to make sure we think this is really worth all of that, other than kinda talking our way out of it | 16:44 |
| sean-k-mooney | or puting a size limit on the cache and contionally disabling | 16:45 |
| dansmith | I don't really have any better ideas, no, I just don't like this one enough to be excited about it | 16:45 |
| dansmith | we should consider some other ideas before we pull the trigger on this I mean | 16:46 |
| dansmith | like, we *could* look for a-cs that have $imgsize+$flavor.root available disk | 16:47 |
| dansmith | which may generate some operator confusion, and will definitely avoid being able to schedule the last byte of disk space | 16:47 |
| dansmith | but also makes some sense if you explain it to someone: there has to be enough disk for the image and the root, even if the image might be cached | 16:48 |
| dansmith | and that could be a behavior you enable with a pre-filter, which tries to avoid situations like bfv | 16:49 |
| sean-k-mooney | dansmith: we have 5 slightly different impleentation fo caching in that module | 16:49 |
| sean-k-mooney | each image backend is slightly different but mostly the same | 16:49 |
| sean-k-mooney | dansmith: could we do that and then srink the allocation | 16:50 |
| *** ttsiouts has quit IRC | 16:50 | |
| *** ttsiouts has joined #openstack-nova | 16:51 | |
| dansmith | sean-k-mooney: right, we'd not allocate that much, just look for hosts with enough to cover it | 16:51 |
| sean-k-mooney | so do $imgsize+$flavor.root then both the instance and shirnk the disk_gb ot $flavor.root | 16:52 |
| dansmith | except in the most pathological cases, we'd be fine.. you could come up with a race scenario, but it'd be very very targeted | 16:52 |
| dansmith | it's also something we could try and roll back without having to change or migrate data, | 16:52 |
| sean-k-mooney | ya its worth a try | 16:53 |
| dansmith | whereas the new allocations-per-image thing would be something we have to live with and migrate for a while if it doesn't pan out | 16:53 |
| sean-k-mooney | ir at least consiering | 16:53 |
| sean-k-mooney | most of the the time $imgsize+$flavor.root is not going to cause boot failure either | 16:54 |
| sean-k-mooney | as it will only be an issue if the cloud is very full | 16:54 |
| dansmith | right, and if it was, it's because you're trying to schedule the last byte of disk, which is something we say is not in our project scope | 16:54 |
| sean-k-mooney | if its a configurable prefilter then those that want every last gb could opt out | 16:54 |
| sean-k-mooney | well ya that too | 16:54 |
| *** ttsiouts has quit IRC | 16:55 | |
| sean-k-mooney | well it sound liek we have too light weight things. a disabel cache, b look for $imgsize+$flavor.root in the placment query and srhink to $flavor.root | 16:56 |
| dansmith | yeah, so maybe if gibi really has a do-not-cache fix here, and we provide that prefilter, maybe that's good enough for the moment | 16:56 |
| dansmith | yep :) | 16:56 |
| sean-k-mooney | and then we could look at the allcotion per image later if we needed too | 16:56 |
| sean-k-mooney | ya i think he need to copy paste it to the 4 other location i commented on but it might work | 16:57 |
| dansmith | yes, I'd be much happier punting that out to a last-resort type of situation | 16:57 |
| *** vishalmanchanda has quit IRC | 16:57 | |
| sean-k-mooney | this is a part of the code i would normally ping mdbooth or lyarwood to look at | 16:57 |
| sean-k-mooney | ok well im going to grab food o/ | 16:58 |
| dansmith | aye dee ohs | 16:59 |
| * lyarwood catches the hospital pass from sean-k-mooney | 16:59 | |
| lyarwood | oh joy the cache manager | 16:59 |
| sean-k-mooney | lyarwood: the intent of that patch is to have a config optioon to trun it off | 17:00 |
| sean-k-mooney | well to workaround the larger bug | 17:01 |
| sean-k-mooney | i think i commented on the other placces where it also need to be done but i proably missed one and i also dont realy understand the calling code paths so its just a guess | 17:01 |
| dansmith | honestly, as much as I trust you guys, I'd have to test it myself before I believed you | 17:03 |
| dansmith | just because I know how loopy all that code is | 17:03 |
| *** derekh has quit IRC | 17:03 | |
| dansmith | it's partial'd up the butthole | 17:03 |
| *** alistarle has joined #openstack-nova | 17:04 | |
| lyarwood | yup same I'd have to play around with this, at first glance I'd be worried about this racing with multiple requests to spawn from the same image tbh | 17:05 |
| lyarwood | but I'm likely missing locking somewhere in the imagebackend or driver that stops this | 17:05 |
| dansmith | well, this is trying to avoid ever downloading the image to the base location, | 17:06 |
| dansmith | so it really shouldn't be able to race because only one instance is booting per instance uuid at a time obviously | 17:06 |
| dansmith | but the call path to know what self.path is here is the critical bit, | 17:06 |
| lyarwood | ah so it's not caching and then copying? | 17:06 |
| lyarwood | my bad | 17:06 |
| dansmith | because normally that is the base image path | 17:06 |
| lyarwood | right understood | 17:07 |
| dansmith | lyarwood: that's the assertion, I just don't know how it got to that point here | 17:07 |
| *** salmankhan has quit IRC | 17:17 | |
| *** priteau has quit IRC | 17:17 | |
| *** ralonsoh has joined #openstack-nova | 17:19 | |
| *** songwenping__ has joined #openstack-nova | 17:20 | |
| *** songwenping_ has quit IRC | 17:24 | |
| *** iurygregory has quit IRC | 17:27 | |
| *** ralonsoh has quit IRC | 17:28 | |
| *** ociuhandu has quit IRC | 17:35 | |
| *** ociuhandu has joined #openstack-nova | 17:37 | |
| *** ociuhandu has quit IRC | 17:37 | |
| *** ociuhandu has joined #openstack-nova | 17:37 | |
| *** ociuhandu has quit IRC | 17:48 | |
| *** ralonsoh has joined #openstack-nova | 17:48 | |
| *** ociuhandu has joined #openstack-nova | 17:51 | |
| *** nightmare_unreal has quit IRC | 17:54 | |
| *** ociuhandu has quit IRC | 17:55 | |
| *** sapd1_x has quit IRC | 18:03 | |
| *** vishalmanchanda has joined #openstack-nova | 18:04 | |
| *** dustinc has joined #openstack-nova | 18:05 | |
| *** JamesBen_ has quit IRC | 18:06 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Bump hacking min version to 3.0.1 https://review.opendev.org/727347 | 18:08 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: Add functional test for bug 1550919 https://review.opendev.org/631294 | 18:15 |
| openstack | bug 1550919 in OpenStack Compute (nova) "[Libvirt]Evacuate fail may cause disk image be deleted" [Medium,In progress] https://launchpad.net/bugs/1550919 - Assigned to Matthew Booth (mbooth-9) | 18:15 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Don't delete disks on shared storage during evacuate https://review.opendev.org/578846 | 18:15 |
| *** happyhemant has quit IRC | 18:15 | |
| *** alistarle has quit IRC | 18:16 | |
| *** ralonsoh has quit IRC | 18:17 | |
| *** ociuhandu has joined #openstack-nova | 18:18 | |
| *** ralonsoh has joined #openstack-nova | 18:22 | |
| *** jamesden_ has joined #openstack-nova | 18:26 | |
| *** jamesdenton has quit IRC | 18:27 | |
| *** jamesden_ is now known as jamesdenton | 18:28 | |
| *** iurygregory has joined #openstack-nova | 18:31 | |
| *** ralonsoh has quit IRC | 18:43 | |
| melwitt | gmann: I've got a queens backport that consistently fails the releasenotes job bc of a job timeout. we are wondering if, with the addition of ussuri releasenotes, if it's just too much for the job to get done in a limited time? https://review.opendev.org/722822 | 18:52 |
| gmann | melwitt: it takes lot of time for all reno to build but should not be timeout. it pass on master. | 18:54 |
| gmann | let me check job timeout for that | 18:54 |
| melwitt | yeah... I just don't get how my backport could cause this and only cause it on queens? confused | 18:54 |
| melwitt | it got through rocky, stein, train without issue | 18:55 |
| elod | melwitt: when those got merged there weren't stable/ussuri yet, were there? | 19:00 |
| elod | this shows some master fails too: https://zuul.opendev.org/t/openstack/builds?job_name=build-openstack-releasenotes&project=openstack/nova | 19:00 |
| melwitt | elod: no there wasn't, was merged before branching | 19:01 |
| melwitt | oh you mean the stable merges. let me check | 19:01 |
| melwitt | train change merged on Apr 18 and I assume stable/ussuri was cut on Apr 24 | 19:03 |
| gmann | reno job timeout is 60 min which should be well enough https://github.com/openstack/openstack-zuul-jobs/blob/13ef0adb415e6296fe5c73d9ff9d1ca557843c54/zuul.d/jobs.yaml#L638 | 19:03 |
| melwitt | stein change merged on Apr 24 | 19:03 |
| melwitt | rocky change merged on May 11 | 19:03 |
| melwitt | the rocky change build does _not_ show processing of stable/ussuri notes https://zuul.opendev.org/t/openstack/build/bd1684e2f6d4460a985d822dfe773b81/log/job-output.txt#580-583 | 19:06 |
| melwitt | whereas the failing queens change build _does_ show processing of stable/ussuri notes https://zuul.opendev.org/t/openstack/build/e63085cb0ebf4e0ea8d91e07c94557ae/log/job-output.txt#536-542 so I think elod is right | 19:06 |
| *** ociuhandu has quit IRC | 19:12 | |
| *** ociuhandu has joined #openstack-nova | 19:12 | |
| *** ociuhandu has quit IRC | 19:13 | |
| *** ociuhandu has joined #openstack-nova | 19:13 | |
| gmann | melwitt: elod it seems job most of time run on the edge time when it pass. >50 for master reno also and 59m 52s in case of https://review.opendev.org/#/c/725146/ | 19:15 |
| melwitt | gmann: livin on the edge. I'm wondering the obvious question, is there a nice way we could run the job for only the relevant branches to the branch it's running on? eventually we're gonna have too many reno branches and job will take longer and longer | 19:16 |
| gmann | melwitt: you mean only build the reno for proposed branch ? | 19:17 |
| melwitt | gmann: something like that yeah. or some other way to avoid wasting resource on building irrelevant renos | 19:17 |
| melwitt | just thinking out loud | 19:18 |
| melwitt | we could increase the job timeout obvs but aside from that I'm wondering about future when we have more and more reno branches | 19:19 |
| *** ociuhandu has quit IRC | 19:23 | |
| *** ociuhandu has joined #openstack-nova | 19:24 | |
| gmann | melwitt: yeah, i remember we were facing same issue on tempest which has 10-15 releases notes bt might be less than nova. it was long back ago so cannot remember is indexing or per release directory fixed that | 19:25 |
| melwitt | ah... looks like nova has "only" 11, liberty-ussuri + unreleased | 19:27 |
| gmann | may be doug can help with some trick, he is not here but we can find him on openstack-dev | 19:29 |
| *** ociuhandu has quit IRC | 19:29 | |
| melwitt | ok, I just sent him a question over there | 19:38 |
| melwitt | thanks for suggesting | 19:38 |
| *** slaweq has quit IRC | 19:57 | |
| *** slaweq has joined #openstack-nova | 20:01 | |
| gmann | melwitt: along with the parallel run option, i think we can remove all these < newton-eol reno? anyways those will be present in older tag- https://github.com/openstack/nova/tree/newton-eol/releasenotes/notes | 20:02 |
| melwitt | gmann: oh you mean treat them as static? yeah I would think so. but keep in mind I don't understand this super well so I might be misunderstanding what you're saying :P | 20:06 |
| *** slaweq has quit IRC | 20:06 | |
| gmann | melwitt: i was thinking to remove as not sure how to make them static. but removing will remoev those from this site too which is issue or might be ok - https://docs.openstack.org/releasenotes/nova | 20:08 |
| melwitt | oh, yeah I wouldn't want to remove them from the site | 20:08 |
| melwitt | personally I'm leaning toward the parallel idea because that would be easier. but yes it's unreleased. I wonder if we could use LIBS_FROM_GIT to do it in the meantime | 20:10 |
| melwitt | and also note that sean said he tried making old stuff static but it didn't help bc most of the time looked to be spent in a rst => html conversion | 20:11 |
| melwitt | (I had to re-read it to pick that out) | 20:11 |
| gmann | melwitt: LIBS_FROM_GIT or required_project need job update but this should also work, as devstack should checkout the depends-on Depends-On: https://review.opendev.org/#/c/724666/ | 20:12 |
| *** nweinber has quit IRC | 20:14 | |
| melwitt | gmann: for a temporary test yes but not for mergeable fix right? | 20:14 |
| smcginnis | The suggestion for making those older branches static would be to call reno to emit the generated straight rst and use that to replace the current page that has the reno sphinx directive. That way it only needs to convert rst to HTML, and not have to generate the rst via the reno directive first. | 20:16 |
| smcginnis | That should save a little time, even if there were not a lot of release notes in those older series. | 20:17 |
| *** dpawlik has joined #openstack-nova | 20:17 | |
| gmann | melwitt: yeah, for mergeable fix if that improves we can ask for new reno release. if we do LIBS_FROM_GIT it might need more testing on reno master gate. | 20:17 |
| melwitt | gmann: yeah good point | 20:17 |
| openstackgerrit | melanie witt proposed openstack/nova master: DNM Try out running sphinx-build in parallel for releasenotes https://review.opendev.org/727429 | 20:21 |
| *** slaweq has joined #openstack-nova | 20:35 | |
| *** vishalmanchanda has quit IRC | 20:37 | |
| *** awalende_ has quit IRC | 20:40 | |
| *** ociuhandu has joined #openstack-nova | 20:40 | |
| *** jsuchome has quit IRC | 20:41 | |
| *** awalende has joined #openstack-nova | 20:42 | |
| *** ociuhandu has quit IRC | 20:53 | |
| *** ociuhandu has joined #openstack-nova | 20:54 | |
| *** ociuhandu has quit IRC | 20:54 | |
| *** ociuhandu has joined #openstack-nova | 20:54 | |
| *** jamesdenton has quit IRC | 20:59 | |
| melwitt | guh, 205 changes in the gate ... | 21:02 |
| *** jamesdenton has joined #openstack-nova | 21:03 | |
| gmann | yeah, pep8 fixing-try also stuck for long time | 21:04 |
| *** slaweq has quit IRC | 21:13 | |
| *** slaweq has joined #openstack-nova | 21:25 | |
| *** slaweq has quit IRC | 21:30 | |
| *** awalende has quit IRC | 21:36 | |
| *** slaweq has joined #openstack-nova | 21:40 | |
| *** ociuhandu has quit IRC | 21:41 | |
| *** ociuhandu has joined #openstack-nova | 21:42 | |
| *** ociuhandu has quit IRC | 21:42 | |
| *** ociuhandu has joined #openstack-nova | 21:43 | |
| *** slaweq has quit IRC | 21:44 | |
| *** ociuhandu has quit IRC | 21:52 | |
| openstackgerrit | melanie witt proposed openstack/nova master: DNM Try out running sphinx-build in parallel for releasenotes https://review.opendev.org/727429 | 21:53 |
| *** ociuhandu has joined #openstack-nova | 21:54 | |
| *** ociuhandu has quit IRC | 21:54 | |
| *** ociuhandu has joined #openstack-nova | 21:54 | |
| *** ociuhandu has quit IRC | 22:05 | |
| *** ociuhandu has joined #openstack-nova | 22:05 | |
| *** ociuhandu has quit IRC | 22:06 | |
| *** ociuhandu has joined #openstack-nova | 22:06 | |
| *** ociuhandu has quit IRC | 22:16 | |
| *** ociuhandu has joined #openstack-nova | 22:17 | |
| *** ociuhandu has quit IRC | 22:17 | |
| *** ociuhandu has joined #openstack-nova | 22:18 | |
| *** ociuhandu has quit IRC | 22:28 | |
| *** ociuhandu has joined #openstack-nova | 22:29 | |
| *** ociuhandu has quit IRC | 22:29 | |
| *** ociuhandu has joined #openstack-nova | 22:29 | |
| *** avolkov has quit IRC | 22:29 | |
| *** martinkennelly has quit IRC | 22:37 | |
| *** ociuhandu has quit IRC | 22:39 | |
| *** mriedem has left #openstack-nova | 22:39 | |
| *** ociuhandu has joined #openstack-nova | 22:41 | |
| *** ociuhandu has quit IRC | 22:51 | |
| *** ociuhandu has joined #openstack-nova | 22:52 | |
| *** ociuhandu has quit IRC | 22:52 | |
| *** ociuhandu has joined #openstack-nova | 22:53 | |
| *** tkajinam has joined #openstack-nova | 22:54 | |
| *** raildo_ has joined #openstack-nova | 22:57 | |
| *** raildo has quit IRC | 23:00 | |
| *** ociuhandu has quit IRC | 23:03 | |
| *** ociuhandu has joined #openstack-nova | 23:04 | |
| openstackgerrit | Takashi Natsume proposed openstack/nova master: Remove six.moves https://review.opendev.org/727224 | 23:13 |
| *** tosky has quit IRC | 23:14 | |
| *** ociuhandu has quit IRC | 23:15 | |
| *** ociuhandu has joined #openstack-nova | 23:16 | |
| *** rcernin has joined #openstack-nova | 23:19 | |
| *** ociuhandu has quit IRC | 23:21 | |
| *** raildo_ has quit IRC | 23:33 | |
| *** awalende has joined #openstack-nova | 23:36 | |
| *** awalende has quit IRC | 23:41 | |
| *** mlavalle has quit IRC | 23:53 | |
| openstackgerrit | Tony Su proposed openstack/nova-specs master: Re-propose provider-config-file spec for Victoria https://review.opendev.org/725788 | 23:56 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!