| *** slaweq_ has joined #openstack-nova | 00:10 | |
| *** slaweq has quit IRC | 00:12 | |
| *** gyee has quit IRC | 00:26 | |
| *** tbachman has quit IRC | 00:38 | |
| *** tbachman_ has joined #openstack-nova | 00:38 | |
| *** tbachman has joined #openstack-nova | 00:42 | |
| *** tbachman_ has quit IRC | 00:43 | |
| *** takamatsu has quit IRC | 00:45 | |
| *** spatel has quit IRC | 00:51 | |
| *** takamatsu has joined #openstack-nova | 00:51 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Migrate default policy file from JSON to YAML https://review.opendev.org/748059 | 00:54 |
|---|---|---|
| *** JamesBenson has joined #openstack-nova | 01:38 | |
| *** Liang__ has joined #openstack-nova | 01:41 | |
| *** JamesBenson has quit IRC | 01:42 | |
| *** Yumeng has joined #openstack-nova | 02:15 | |
| *** boxiang_ has quit IRC | 02:30 | |
| *** boxiang_ has joined #openstack-nova | 02:30 | |
| *** Liang__ has quit IRC | 02:36 | |
| *** Liang__ has joined #openstack-nova | 02:37 | |
| *** dklyle has quit IRC | 02:40 | |
| *** david-lyle has joined #openstack-nova | 02:40 | |
| openstackgerrit | Yingji Sun proposed openstack/nova master: Set different VirtualDevice.key https://review.opendev.org/713565 | 02:51 |
| *** chenhaw has quit IRC | 02:51 | |
| *** chenhaw has joined #openstack-nova | 02:52 | |
| openstackgerrit | Merged openstack/nova stable/stein: Should not skip volume_size check for bdm.image_id == image_ref case https://review.opendev.org/744553 | 02:55 |
| *** sapd1_x has joined #openstack-nova | 03:17 | |
| *** rcernin_ has joined #openstack-nova | 03:25 | |
| *** rcernin has quit IRC | 03:25 | |
| openstackgerrit | Merged openstack/nova stable/ussuri: compute: Do not allow rescue attempts using volume snapshot images https://review.opendev.org/742706 | 03:38 |
| *** JamesBenson has joined #openstack-nova | 03:39 | |
| *** JamesBenson has quit IRC | 03:44 | |
| *** links has joined #openstack-nova | 03:54 | |
| *** Liang__ has quit IRC | 03:57 | |
| *** psachin has joined #openstack-nova | 03:57 | |
| *** chenhaw has quit IRC | 04:00 | |
| *** chenhaw has joined #openstack-nova | 04:00 | |
| *** mkrai has joined #openstack-nova | 04:05 | |
| *** dave-mccowan has quit IRC | 04:07 | |
| openstackgerrit | Merged openstack/nova stable/rocky: compute: Allow snapshots to be created from PAUSED volume backed instances https://review.opendev.org/729177 | 04:14 |
| *** jmlowe has quit IRC | 04:18 | |
| openstackgerrit | Elancheran S proposed openstack/nova stable/ussuri: Removes the delta file once image is extracted https://review.opendev.org/748034 | 04:25 |
| *** jmlowe has joined #openstack-nova | 04:29 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-nova | 04:33 | |
| openstackgerrit | Tony Su proposed openstack/nova master: Provider Config File: Functions to merge provider configs to provider tree https://review.opendev.org/676522 | 04:51 |
| *** ratailor has joined #openstack-nova | 05:15 | |
| *** sridharg has joined #openstack-nova | 05:21 | |
| *** sridharg has quit IRC | 05:22 | |
| *** sridharg has joined #openstack-nova | 05:23 | |
| *** JamesBenson has joined #openstack-nova | 05:45 | |
| *** belmoreira has joined #openstack-nova | 05:46 | |
| *** belmoreira has quit IRC | 05:47 | |
| *** JamesBenson has quit IRC | 05:50 | |
| *** jsuchome has joined #openstack-nova | 05:52 | |
| *** redrobot has quit IRC | 05:53 | |
| *** mvorwerk_ has joined #openstack-nova | 06:11 | |
| *** mvorwerk has quit IRC | 06:13 | |
| *** slaweq_ has quit IRC | 06:14 | |
| *** david-lyle has quit IRC | 06:21 | |
| *** slaweq_ has joined #openstack-nova | 06:35 | |
| *** ralonsoh has joined #openstack-nova | 06:42 | |
| *** mkrai_ has joined #openstack-nova | 06:45 | |
| *** slaweq_ is now known as slaweq | 06:48 | |
| *** mkrai has quit IRC | 06:48 | |
| *** vishalmanchanda has joined #openstack-nova | 06:49 | |
| *** tesseract has joined #openstack-nova | 06:57 | |
| *** brinzhang_ has quit IRC | 06:59 | |
| *** brinzhang has joined #openstack-nova | 07:03 | |
| *** belmoreira has joined #openstack-nova | 07:12 | |
| openstackgerrit | Keigo Noha proposed openstack/nova master: Change default num_retries for glance to 3 https://review.opendev.org/740389 | 07:18 |
| *** yonglihe has joined #openstack-nova | 07:21 | |
| gibi | gmann: cool. I marked the policy by as implemented | 07:35 |
| brinzhang | gibi: hi, good morning | 07:39 |
| *** tosky has joined #openstack-nova | 07:40 | |
| gibi | brinzhang: hi | 07:40 |
| brinzhang | gibi: https://review.opendev.org/#/c/715326/19/nova/compute/manager.py@3654 this you mean, we should add the kwargs['accel_uuids'] to the rebuild() and _rebuild_default_impl() interfact, right? | 07:40 |
| brinzhang | that we should change the virt/driver.py and virt/ironic/driver.py rebuild() functions? | 07:41 |
| gibi | brinzhang: yes, I mean that this parameter should be part of the https://github.com/openstack/nova/blob/master/nova/virt/driver.py#L285 interface | 07:42 |
| brinzhang | gibi: does it need to change the irnic driver? https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L1669 | 07:43 |
| brinzhang | s/irnic/ironic/ | 07:43 |
| gibi | yes | 07:43 |
| gibi | it will be a parameter that the ironic driver gets but ignores at the moment | 07:44 |
| brinzhang | yeah, we will add this parameter, and send a mail to the ML later ^ | 07:46 |
| brinzhang | gibi: thanks | 07:46 |
| *** Luzi has joined #openstack-nova | 07:55 | |
| gibi | brinzhang: thanks | 07:57 |
| openstackgerrit | Alexandre Arents proposed openstack/nova master: Add a lock to prevent race during detach/attach of interface https://review.opendev.org/747957 | 08:01 |
| *** mgoddard has quit IRC | 08:10 | |
| *** mgoddard has joined #openstack-nova | 08:11 | |
| bauzas | gibi: fwiw, I'm starting to sheperd the rest of the provider config series | 08:17 |
| *** dosaboy has quit IRC | 08:17 | |
| bauzas | shepherd* even | 08:17 |
| gibi | bauzas: thanks. | 08:17 |
| *** dosaboy has joined #openstack-nova | 08:17 | |
| bauzas | it needed me to look back at the spec, but thanks for having started to review the series | 08:18 |
| gibi | unfortunately the implementation changed ownwer couple of times so there are thing that needs (needed) detective work to understand the context | 08:21 |
| *** rmart04 has joined #openstack-nova | 08:23 | |
| rmart04 | Hey All, Apologies if this isn't the right place to ask, but are there any obvious memory limitations gotchas when using dedicated cpu policy? I'm seeing OOM killing off my instances when the host has plenty of memory left :/ (Stein / CentOS7 Packages / KVM / Kolla-Ansible) | 08:26 |
| openstackgerrit | Elod Illes proposed openstack/nova stable/queens: compute: Allow snapshots to be created from PAUSED volume backed instances https://review.opendev.org/729178 | 08:27 |
| openstackgerrit | Brin Zhang proposed openstack/nova master: Cyborg evacuate support https://review.opendev.org/715326 | 08:30 |
| *** tosky has quit IRC | 08:32 | |
| *** tosky has joined #openstack-nova | 08:33 | |
| *** yonglihe has quit IRC | 08:33 | |
| *** arne_wiebalck has quit IRC | 08:33 | |
| *** luyao has quit IRC | 08:33 | |
| *** luyao has joined #openstack-nova | 08:34 | |
| *** arne_wiebalck has joined #openstack-nova | 08:35 | |
| *** Yumeng has quit IRC | 08:36 | |
| *** derekh has joined #openstack-nova | 08:37 | |
| *** yonglihe has joined #openstack-nova | 08:40 | |
| stephenfin | lyarwood, gibi: So I've got three series on the go rn. What would I need to pay you to get at least one of them moving? :) | 08:42 |
| gibi | stephenfin: the vpmem is on my menu for today | 08:43 |
| gibi | is that one of the three | 08:43 |
| gibi | ? | 08:43 |
| stephenfin | unless you meant vTPM, no :( | 08:43 |
| gibi | vtpm | 08:43 |
| stephenfin | then yes \o/ | 08:43 |
| gibi | cool | 08:43 |
| stephenfin | hurrah | 08:43 |
| gibi | then you can expect review on that in the coming hours | 08:43 |
| stephenfin | awesome. Thanks, gibi :) | 08:44 |
| gibi | no problemo, it is in the runway slot so it in focus :) | 08:44 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add type hints to 'nova.compute.manager' https://review.opendev.org/742863 | 08:46 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Don't unset Instance.old_flavor, new_flavor until necessary https://review.opendev.org/741995 | 08:46 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: privsep: Add support for recursive chown, move_tree operations https://review.opendev.org/742864 | 08:46 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add type hints to 'nova.virt.libvirt.utils' https://review.opendev.org/742865 | 08:46 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support for resize and cold migration of emulated TPM files https://review.opendev.org/639934 | 08:46 |
| *** adriant has quit IRC | 08:48 | |
| *** adriant has joined #openstack-nova | 08:48 | |
| stephenfin | alex_xu: Can you look at https://review.opendev.org/#/c/744021/ and its predecessor today? You probably have appropriate context | 08:54 |
| stephenfin | alex_xu: I suspect it impacts the mixed CPU work too | 08:54 |
| openstackgerrit | Brin Zhang proposed openstack/nova master: Cyborg evacuate support https://review.opendev.org/715326 | 08:58 |
| stephenfin | rmart04: Using the dedicated CPU policy implies a single-node NUMA topology by default, and using a NUMA topology implies consuming memory from a single host NUMA node for each guest NUMA node | 09:01 |
| stephenfin | rmart04: That's a roundabout way of saying that I suspect you haven't reserved memory on per-node basis and may have allocated something like hugepages unevenly across NUMA nodes | 09:02 |
| stephenfin | rmart04: nova won't let you overcommit against a host NUMA node using nova instances, but it can't account for host processes or huge page allocations consuming memory on that node. You've to do that manually | 09:05 |
| *** sapd1_x has quit IRC | 09:05 | |
| stephenfin | rmart04: There are two config options of value here: 'reserved_host_memory_mb', which is a total amount of memory to reserve across all NUMA nodes, and 'reserved_huge_pages' which is a per-node reservation that applies when you're using explicit pagesizes | 09:07 |
| bauzas | stephenfin: gibi: fwiw, after the provider series, I was heading to review your vTPM series | 09:07 |
| stephenfin | rmart04: I suggest whacking up the first ('reserved_host_memory_mb'). It would be good to inspect idle memory consumption without instances on the host and use that value with some buffer. | 09:07 |
| stephenfin | rmart04: The second is rarely used. Unless you're using explicit page sizes for all instances, I wouldn't worry about it for now | 09:08 |
| stephenfin | rmart04: Hope that helps | 09:08 |
| stephenfin | bauzas: awesome. Thanks :) | 09:08 |
| *** dtantsur|afk is now known as dtantsur | 09:09 | |
| lyarwood | stephenfin: just catching up with some downstream stuff but I'll get into your reviews after that | 09:13 |
| bauzas | alex_xu: gibi: I have a logging concern with https://review.opendev.org/#/c/693460/36 | 09:16 |
| bauzas | alex_xu: gibi: or we will get a lot of logs if operators don't use this new feature :) | 09:17 |
| gibi | bauzas: you mean https://review.opendev.org/#/c/693460/36/nova/compute/resource_tracker.py@120 ? | 09:18 |
| gibi | bauzas: I see one extra log per compute node at compute service startup | 09:18 |
| gibi | also that config has a default value so it is set even if not specified in the config file | 09:19 |
| bauzas | gibi: correct, this won't be very talkative, but we then try to merge inventories even if the operator doesn't use it | 09:19 |
| bauzas | gibi: https://review.opendev.org/#/c/693460/36/nova/compute/resource_tracker.py@1165 | 09:19 |
| bauzas | and https://review.opendev.org/#/c/676029/51/nova/compute/provider_config.py@391 | 09:20 |
| bauzas | all of this seems not needed | 09:20 |
| bauzas | gibi: I would have preferred a way to avoid calling all of this if no YAML files were there | 09:22 |
| gibi | Can you check from oslo config if the config value is set explicitly by the user or just the default value is applied by oslo? | 09:23 |
| gibi | because current calling down to get_provider_configs is the only way to detect if there are config files | 09:24 |
| *** spatel has joined #openstack-nova | 09:26 | |
| *** k_mouza has joined #openstack-nova | 09:26 | |
| gibi | regarding the actual logging: I can see a reasoning to log what provider config files are loaded if any instead of logging if nothing is loaded | 09:28 |
| *** spatel has quit IRC | 09:30 | |
| stephenfin | real trivial patch for the VMWare driver here https://review.opendev.org/#/c/713565/ | 09:39 |
| *** rcernin_ has quit IRC | 09:39 | |
| *** k_mouza has quit IRC | 09:40 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Move revert resize under semaphore https://review.opendev.org/747746 | 09:41 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Cleanup 'drop_move_claim' and '_drop_move_claim' https://review.opendev.org/747747 | 09:41 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: manager: Address TODO https://review.opendev.org/747748 | 09:41 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: manager: Move context manager up one level https://review.opendev.org/745674 | 09:41 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: compute: Add type hints for resize functions https://review.opendev.org/745341 | 09:41 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: compute: Add more type hints for resize functions https://review.opendev.org/745675 | 09:41 |
| stephenfin | lyarwood: are you respinning that nova-image-download-via-rbd series again today to add the missing '__init__.py' file etc.? | 09:42 |
| openstackgerrit | Merged openstack/nova stable/ussuri: libvirt: Provide VIR_MIGRATE_PARAM_PERSIST_XML during live migration https://review.opendev.org/747972 | 09:45 |
| gibi | stephenfin: I left a question in https://review.opendev.org/#/c/741995/10/nova/compute/manager.py@a4932 | 09:46 |
| lyarwood | stephenfin: yeah | 09:52 |
| *** sridharg has quit IRC | 09:54 | |
| stephenfin | gibi: replied | 09:54 |
| gibi | stephenfin: thanks it is make sense now. | 09:55 |
| *** k_mouza has joined #openstack-nova | 09:57 | |
| bauzas | gibi: sorry, missed your ping | 09:57 |
| stephenfin | gibi: fwiw, that patch is identical to this bug fix for a ResourceTracker race. I just didn't want to make the series depend on each other https://review.opendev.org/#/c/744958/ | 09:57 |
| stephenfin | gibi: Have been bugging melwitt and dansmith for reviews on that though so you're off the hook for reviews there :P | 09:57 |
| *** gryf has quit IRC | 09:58 | |
| gibi | stephenfin: if it is identical then I can add my +2 there :) | 09:58 |
| *** mkrai_ has quit IRC | 09:58 | |
| *** mkrai__ has joined #openstack-nova | 09:58 | |
| stephenfin | yup, iirc cherry-picked and change-id modified to mark them as independent | 09:59 |
| *** mkrai has joined #openstack-nova | 10:02 | |
| *** mkrai__ has quit IRC | 10:02 | |
| bauzas | stephenfin: just saw https://review.opendev.org/#/c/747744/ | 10:03 |
| bauzas | will try to look at it too | 10:03 |
| stephenfin | nice one | 10:04 |
| bauzas | stephenfin: starting to look at the vTPM series but... wow, I need a beer :p | 10:06 |
| gibi | isn't it a bit too early for a beer? ;) | 10:07 |
| lyarwood | blasphemy | 10:08 |
| alex_xu | stephenfin: checking now | 10:08 |
| alex_xu | bauzas: yea, as gibi said, it is just one log at startup, should be ok | 10:08 |
| bauzas | alex_xu: I changed my thoughts | 10:08 |
| bauzas | I don't wanna hold for a nit | 10:09 |
| bauzas | but... this change needs a rebase either way :) | 10:09 |
| bauzas | gibi: technically, it's beer o'time | 10:10 |
| bauzas | ten past nonn | 10:10 |
| bauzas | noon | 10:10 |
| bauzas | if that's too early, then I don't know when | 10:11 |
| alex_xu | bauzas: yea, that is due to tony_su rebase the previous patch, actually it needn't, just just gate failed, he should recheck instead of rebase. | 10:12 |
| bauzas | alex_xu: the branch I see from gerrit pulls the old revision | 10:13 |
| bauzas | alex_xu: what he can do tho is wait for the dependent change to be merged and then indeed a recheck should be fine | 10:14 |
| *** cgoncalves has quit IRC | 10:14 | |
| gibi | bauzas: I rest my case, it is really afternoon now | 10:16 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Provider Config File: Enable loading and merging of provider configs https://review.opendev.org/693460 | 10:18 |
| alex_xu | bauzas: yea, that is my thought also | 10:18 |
| stephenfin | alex_xu, bauzas: There you go | 10:18 |
| elod | lyarwood: the persist_xml patch has merged in ussuri. what do you think about triggering a release in ussuri? do you mind if i create a release patch? | 10:18 |
| alex_xu | stephenfin: hah, thanks :) | 10:19 |
| bauzas | stephenfin: thanks | 10:19 |
| * bauzas goes away for a barbecue grilling, and then I'm back in 1 hour and half | 10:20 | |
| elod | lyarwood: or should we wait until it gets merged all the way till stein and release ussuri / train / stein together? | 10:20 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Deprecate filters that have been replaced by placement filters https://review.opendev.org/745605 | 10:22 |
| gibi | bauzas: enjoy | 10:23 |
| lyarwood | elod: if you have time feel free to create the release change for ussuri now | 10:23 |
| lyarwood | elod: I was going to wait but there's nothing stopping us from releasing as it lands in each branch tbh | 10:24 |
| elod | lyarwood: ok, thanks, doing it now | 10:25 |
| lyarwood | awesome thanks :) | 10:26 |
| *** sridharg has joined #openstack-nova | 10:26 | |
| *** cgoncalves has joined #openstack-nova | 10:31 | |
| elod | lyarwood , gibi : https://review.opendev.org/#/c/748156/ | 10:35 |
| elod | seems minor version bump required anyway, but if you see other patch that should listed also that requires minor bump, too, just let me know and I'll update the commit message | 10:37 |
| gibi | elod: +1 | 10:39 |
| *** Luzi has quit IRC | 10:42 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Deprecate filters that have been replaced by placement filters https://review.opendev.org/745605 | 10:47 |
| *** sridharg has quit IRC | 10:53 | |
| *** mkrai has quit IRC | 10:55 | |
| rmart04 | Hi Stephenfin, thanks for this info. I'm just digesting what you've written. I am currently not using huge pages, but I think what you are saying is that one of the numa nodes ran out of memory, not the entire host. I'm not sure how reserved_host_memory_mb will help when the host has lots of memory available (250GB), i'm guessing we just ran out of memory on one particuar numa node? I should mention, we have | 10:58 |
| rmart04 | one large VM on this host with numa_nodes:2. | 10:58 |
| stephenfin | rmart04: Yes, exactly. We lock memory to a single NUMA node so it the instance tries to allocate memory and there's none on _that node_, you're flat out of luck | 10:59 |
| *** jangutter has joined #openstack-nova | 11:00 | |
| stephenfin | Does the per-node memory allocation of that very large VM bump up against the per-host node total? Are your DIMMs evenly split between host NUMA nodes? | 11:00 |
| stephenfin | i.e. you've 250GB available in total, but perhaps only a small bit on one particular NUMA node | 11:01 |
| sean-k-mooney | the OOM reaper in the kernel runs per numa node so if a numa node runs out of memory it will start killing process if the kerenl needs to allcoate even if the other node is entirely empty | 11:01 |
| rmart04 | So Host has 768GB split evenly across numa nodes. There is one instance, 750GB in size, and has seen more than half its memory used successfully in the past. I'm not sure how to see how much memory a particular numa node has available? Is that easy to see? | 11:02 |
| sean-k-mooney | if you are using numa but not pinning or hugepages | 11:02 |
| alex_xu | stephenfin: are we ensure the operator finish the pin_cpu_set to dedicated_cpu_set/sahred_cpu_set change before update to V release? | 11:02 |
| sean-k-mooney | then we will place all the vms on numa node 0 and never use the second numa node | 11:02 |
| sean-k-mooney | im working on fixing that this morning | 11:02 |
| rmart04 | cpu_policy is dedicated, so I believed its pinned | 11:03 |
| alex_xu | stephenfin: or the operator can do that change when upgrade U to V? | 11:03 |
| *** jangutter_ has quit IRC | 11:03 | |
| sean-k-mooney | ah in that case we will stack all vms on the first numa node untill we run our of pinned cpus then use the next one | 11:03 |
| stephenfin | alex_xu: A user can continue to use the legacy vcpu_pin_set in U. I'm hoping to drop support for it in V, but I have too many series in flight at the moment so it might slip to W | 11:04 |
| stephenfin | alex_xu: If that's what you mean? | 11:04 |
| sean-k-mooney | alex_xu: i think the plan is to remove vcpu_pin_set in v so they have to upgrade in U | 11:04 |
| sean-k-mooney | ideally they would do it in train | 11:04 |
| sean-k-mooney | but they can do it in U | 11:04 |
| sean-k-mooney | rmart04: you can see it in /sys | 11:05 |
| sean-k-mooney | ill get the path on sec | 11:05 |
| alex_xu | stephenfin: sean-k-mooney ,I'm thinking of 744021 may stop the upgrade from U to V if the operate transft from legacy to new config in that time | 11:06 |
| sean-k-mooney | rmart04: cat /sys/bus/node/devices/node*/meminfo | 11:06 |
| alex_xu | if we force that legacy config change finished in U, then it should be fine | 11:07 |
| sean-k-mooney | alex_xu: its there because we had customer upgraind into an invlaid config | 11:08 |
| stephenfin | alex_xu: I don't understand the issue | 11:08 |
| sean-k-mooney | they had vms with the old configuration where they were using isolate but the new config options | 11:08 |
| stephenfin | A user should never get into a situation where they have new-style config but a pinned instance is consuming VCPU instead of PCPU | 11:08 |
| sean-k-mooney | so they were using vcpus instead of pcpus | 11:08 |
| sean-k-mooney | even thogh the host reported pcpus | 11:09 |
| stephenfin | This patch should be backported to Ussuri and Train | 11:09 |
| sean-k-mooney | yes | 11:09 |
| stephenfin | Is there something upgrade specific I'm missing, perhaps? | 11:10 |
| sean-k-mooney | alex_xu: it was an edgecase we forgot to block, we taught it would be blocked by the numa toplogy filter but the check was missing | 11:10 |
| alex_xu | if the host with legacy config is running isolated thread instance, but the 744021 stop the isolated thread instance? (I'm still try to ramp up my mind) | 11:10 |
| sean-k-mooney | alex_xu: if its using legacy configs it will allow it to boot | 11:10 |
| sean-k-mooney | alex_xu: it only changes the behavior for new configs | 11:11 |
| alex_xu | oh, right | 11:11 |
| sean-k-mooney | alex_xu: we dont allow the shared and dedicated set to overlap so the only time they contian the same values is if we are using the legacy config options | 11:12 |
| sean-k-mooney | so and host_cell.pcpuset != host_cell.cpuset | 11:12 |
| sean-k-mooney | is an indirect check for is the host using new config options | 11:13 |
| lyarwood | elod: sorry missed that https://review.opendev.org/#/c/747357/ wasn't included | 11:13 |
| sean-k-mooney | that will also work in the schduler | 11:13 |
| lyarwood | elod: can we wait for that and then release? | 11:13 |
| sean-k-mooney | since we cant actully check the config directly there | 11:13 |
| jsuchome | lyarwood: gibi stephenfin https://review.opendev.org/#/c/574301/ finally green again, any further changes necessary? | 11:14 |
| *** jangutter has quit IRC | 11:14 | |
| *** gryf has joined #openstack-nova | 11:15 | |
| *** jangutter has joined #openstack-nova | 11:15 | |
| elod | lyarwood: oh, of course, sorry, I forgot to check what else on the queue :S | 11:15 |
| openstackgerrit | Merged openstack/nova master: Provider Config File: Functions to merge provider configs to provider tree https://review.opendev.org/676522 | 11:16 |
| lyarwood | jsuchome: https://review.opendev.org/#/c/746904/ - I just need to respin this after lunch and I think we should be good to go | 11:17 |
| alex_xu | sean-k-mooney: but with new config, why we meet the case fallback to query VCPU? | 11:17 |
| alex_xu | fallback query should find the host with legacy config | 11:18 |
| * bauzas is back | 11:19 | |
| sean-k-mooney | alex_xu: not if all host are using the new configs and you forget to trun off the fallback | 11:19 |
| sean-k-mooney | the fallback can select the vcpu on host with the new config options | 11:20 |
| sean-k-mooney | there is noting in the fallback query to prevent that | 11:20 |
| sean-k-mooney | we were relying on the numa toplogy filter to handel that case and this check was missing | 11:20 |
| sean-k-mooney | alex_xu: for what its worth i was 99% certin the numa toplogy filter would block this untill i repoduced the issue locally | 11:21 |
| alex_xu | sean-k-mooney: if forget turn the fallback off, the scheduler will chocie the host with shared_cpu_set? | 11:23 |
| alex_xu | oh, it is not, since host.pcpu is empty | 11:24 |
| sean-k-mooney | so with the fall back enabled on a cloudl with all new config options | 11:28 |
| sean-k-mooney | and smt enabled | 11:28 |
| sean-k-mooney | when we do the initall query the hyperthreading forbiden trait will cause the PCPU queury to retrun no hosts since they all have smt enabled | 11:29 |
| alex_xu | sean-k-mooney: it is new deployed host, with dedicated and shared cpu set, that allows the mixed instance. But the fallback still enabled, then in the case of there is no more pCPU, then scheduler fallback to VCPU. Then host with shared cpu set allow that scheduling? | 11:29 |
| sean-k-mooney | then the fallback with vcpus will return allocation agains tthe vcpu inventories | 11:29 |
| sean-k-mooney | alex_xu: yes basicly | 11:30 |
| sean-k-mooney | so what happens is we end up claiming vCPUs in placement and then pin the vm to the cpu_dedicated_set cpus | 11:31 |
| alex_xu | if that is the case, a shared thread instance also can be scheduled to that host, right? | 11:31 |
| sean-k-mooney | yes | 11:31 |
| alex_xu | emm...that is still wrong | 11:32 |
| sean-k-mooney | well shared instance and pinned can mix on the same host as of train | 11:32 |
| sean-k-mooney | that is ok | 11:32 |
| sean-k-mooney | but the allocation are wrong | 11:32 |
| alex_xu | so we should ask the user turn off the fallback before upgrading to V? | 11:32 |
| sean-k-mooney | we shoudl remove it when we removed vcpu_pin_set | 11:33 |
| *** JamesBenson has joined #openstack-nova | 11:33 | |
| sean-k-mooney | with https://review.opendev.org/#/c/744021/3 its safe to leave it on | 11:33 |
| sean-k-mooney | as placement will still do the wrong thing but the numa toplogy filter or the hardware module on teh compute host will block it | 11:33 |
| alex_xu | but it doesn't fix the case for shared thread instance, right? | 11:33 |
| sean-k-mooney | that is not broken | 11:34 |
| sean-k-mooney | if you have cpu_policy=shared then it will request vcpus | 11:34 |
| noonedeadpunk | hey everyone! I was wondering if it's possible to prohibit users to use ephemeral storage? I was trying to look through the policies but the only thing I found and was related is to allow use zero disk flavors. While it allows to use cinder volumes, it doesn't prohibits ephemeral ones... | 11:34 |
| sean-k-mooney | the fallback only applies to cpu_policy=dedicated | 11:34 |
| sean-k-mooney | and cpu_thread_policy only applies to cpu_polcy=dedicated | 11:35 |
| alex_xu | yes | 11:35 |
| noonedeadpunk | Eventually while I was looking I saw other ppl was also wondering about that opportuninty | 11:35 |
| sean-k-mooney | noonedeadpunk: yes you can | 11:35 |
| sean-k-mooney | via config | 11:35 |
| alex_xu | I mean the case cpu_thread_policy is shared and cpu_policy is dedicated | 11:35 |
| sean-k-mooney | oh in that case we dont claim vcpus at all | 11:35 |
| sean-k-mooney | well we will | 11:36 |
| sean-k-mooney | because of the bug | 11:36 |
| sean-k-mooney | alex_xu: the vm is still being pinned to the cpu_dedicated_set | 11:36 |
| sean-k-mooney | and its emulator treads will stll be pinned to the cpu_shared_set | 11:37 |
| sean-k-mooney | we just wont claim PCPUs in plament | 11:37 |
| sean-k-mooney | noonedeadpunk: https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.max_local_block_devices | 11:37 |
| sean-k-mooney | noonedeadpunk: you can set that to 0 | 11:37 |
| alex_xu | wait, the filter will stop that, since there is no enough pcpu in hostnuma.pcpu | 11:37 |
| sean-k-mooney | alex_xu: no it wont | 11:38 |
| noonedeadpunk | sean-k-mooney: oh, thanks! I was thinking about https://docs.openstack.org/nova/ussuri/configuration/config.html#libvirt.images_type setting to noop or smth but saw that's not an option | 11:38 |
| alex_xu | oh...I see now, due to hyper thread | 11:38 |
| sean-k-mooney | alex_xu: it will only stop it if thos pcpus are claimed | 11:38 |
| sean-k-mooney | so if the host is more or less empty it will spawn if its getting full then yes it can reject it | 11:39 |
| sean-k-mooney | but its not always going to | 11:39 |
| sean-k-mooney | noonedeadpunk: are you trybing to force all vms to be boot from volume | 11:40 |
| noonedeadpunk | yep | 11:40 |
| *** JamesBenson has quit IRC | 11:40 | |
| sean-k-mooney | ah i see ya so that config option is the way to go | 11:40 |
| *** JamesBenson has joined #openstack-nova | 11:40 | |
| noonedeadpunk | tbh ideal scenario would be to force all except some selected group | 11:40 |
| sean-k-mooney | but just be aware we consider swap to be a local disk too | 11:40 |
| sean-k-mooney | noonedeadpunk: this is a per host option by the way | 11:41 |
| sean-k-mooney | so you can set it on a subset of hosts | 11:41 |
| sean-k-mooney | and ideally use an aggreate to group them for scheduling | 11:41 |
| noonedeadpunk | Oh, so it's effective on compute, not on api/scheduler hosts? | 11:41 |
| sean-k-mooney | noonedeadpunk: correct | 11:41 |
| noonedeadpunk | Ok, good to know! | 11:41 |
| noonedeadpunk | thanks for the help and sorry for disturbing:) | 11:41 |
| noonedeadpunk | really wasn't able to find this specific option with code words I was thinking about) | 11:42 |
| sean-k-mooney | noonedeadpunk: what release are you on | 11:42 |
| alex_xu | sean-k-mooney: ah, i see now. The host has enough pcpu for that instance, but the instance forbidden the hyperthread, that is why the placement reject, then fallback to VCPU. And then filter begin to filter based pcpu again, then the scheduling success | 11:42 |
| sean-k-mooney | alex_xu: exactly | 11:42 |
| alex_xu | finally i see that | 11:42 |
| alex_xu | sean-k-mooney: thanks | 11:42 |
| sean-k-mooney | its an edgecase to and edgecase but customers hit it imeditely | 11:42 |
| noonedeadpunk | sean-k-mooney: I think train mostly | 11:43 |
| noonedeadpunk | (maybe some U) | 11:43 |
| sean-k-mooney | ok so i think trian has isolated aggreates support one sec whil i get the link | 11:43 |
| sean-k-mooney | https://docs.openstack.org/nova/latest/reference/isolate-aggregates.html | 11:43 |
| sean-k-mooney | what you can do is add a custom CUSTOM_BFV_ONLY trait to a subset of host and add it as a required trait in your BFV flavors | 11:44 |
| *** raildo has joined #openstack-nova | 11:44 | |
| sean-k-mooney | if you configure the aggreate to requrie it then it will block non BFV flavor from landing on those hosts | 11:45 |
| noonedeadpunk | is scheduler.enable_isolated_aggregate_filtering option also compute specific? | 11:45 |
| sean-k-mooney | then on those host you can also set the config option if you want too | 11:45 |
| noonedeadpunk | (I think not) | 11:45 |
| sean-k-mooney | noonedeadpunk: that is cloud wide | 11:45 |
| sean-k-mooney | but this allows you to use traits and host aggrate to dynamically force certen behavior | 11:46 |
| noonedeadpunk | yeah, that's pretty handy, thanks! | 11:46 |
| sean-k-mooney | in this case it will prevent vms without the trait form landing on host in a given aggreate | 11:46 |
| sean-k-mooney | noonedeadpunk: this would allow you to get teh same effect without needing to modify config options | 11:46 |
| noonedeadpunk | (I would probably still prefer this to be configurable by policy though tbh, but whatever) | 11:46 |
| sean-k-mooney | in what way | 11:47 |
| sean-k-mooney | have a specific bfv_only flag of some kind? per host or cloud wide? | 11:47 |
| sean-k-mooney | noonedeadpunk: im trying to understand if there is a gap that we might need to address beyond having a topic doc on how to use the exisitng feature to do this | 11:48 |
| noonedeadpunk | So yeah, we have `os_compute_api:servers:create:zero_disk_flavor` so maybe option like `os_compute_api:servers:create:local_drive` which by default allowed for everyone | 11:48 |
| noonedeadpunk | but this can be overriden and local_drive can be created only to specific groups of users | 11:49 |
| sean-k-mooney | ah to block flavor creation with root, ephmeral or swap | 11:49 |
| openstackgerrit | Merged openstack/nova stable/ussuri: libvirt: Do not reference VIR_ERR_DEVICE_MISSING when libvirt is < v4.1.0 https://review.opendev.org/747357 | 11:49 |
| sean-k-mooney | the only real issue i see with that is root ephmerla and swap might not be local | 11:50 |
| openstackgerrit | Merged openstack/nova master: Set different VirtualDevice.key https://review.opendev.org/713565 | 11:50 |
| sean-k-mooney | e.g. if you are using ceph for storage | 11:50 |
| noonedeadpunk | And eventually currently I am.... | 11:50 |
| sean-k-mooney | but we proably could add a policy flag for each filed gmann is that a reasonable thing to do? | 11:50 |
| sean-k-mooney | gmann: question i have is it resonable to add a policy flag to restirct creating flavor with non 0 root, swap or ephemeral storage | 11:51 |
| noonedeadpunk | so, if I have ceph backend configured, than max_local_block_devices is not really applicable? | 11:51 |
| sean-k-mooney | noonedeadpunk: more or less yes | 11:52 |
| sean-k-mooney | i think it techniall is still enforced but its not really local | 11:52 |
| noonedeadpunk | ah, ok, then this should work | 11:52 |
| sean-k-mooney | im not sure if the point where we check that option know what storage is configured | 11:53 |
| sean-k-mooney | we might have a skip for the rbd backend | 11:53 |
| sean-k-mooney | noonedeadpunk: storage is one area i understand but avoid so the details are not something i rember and have to lookup each time | 11:54 |
| * bauzas is backhttps://review.opendev.org/#/c/741500/10 ? | 11:55 | |
| bauzas | whoops | 11:55 |
| bauzas | gibi: any reason why you based https://review.opendev.org/#/c/742407/4 on top of https://review.opendev.org/#/c/741500/10 ? | 11:55 |
| sean-k-mooney | bauzas: a little over eger to be back :) | 11:55 |
| * bauzas can't translate :) | 11:56 | |
| gibi | bauzas: I have no ide how I made that | 11:56 |
| noonedeadpunk | Ok, so I have the following situation: some of the users, despite we don't have non-zero flavors in terms of disks, some of them still press "do not create volume" button in horizon, which creates ephemeral volume in ceph with size equall to the image size. And then they return crying that all data has been lost in case of some reboot/rebuilt/etc | 11:56 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Undeprecate the vmwareapi driver https://review.opendev.org/742407 | 11:56 |
| bauzas | gibi: cool | 11:56 |
| sean-k-mooney | its merged now so it does not really matter | 11:56 |
| sean-k-mooney | the vtpm patch | 11:56 |
| bauzas | yup | 11:56 |
| bauzas | but this was confusing me | 11:57 |
| bauzas | as I was looking at the hairy vtpm series | 11:57 |
| gibi | bauzas: thanks for noticing | 11:57 |
| sean-k-mooney | gibi: you proably were reviewing stephens patch before | 11:57 |
| bauzas | (and gosh, this spec is... terrible https://specs.openstack.org/openstack/nova-specs/specs/victoria/approved/add-emulated-virtual-tpm.html ) | 11:57 |
| gibi | sean-k-mooney: could be | 11:57 |
| sean-k-mooney | hehe its a good way to get extra review :P | 11:58 |
| sean-k-mooney | put random patches into a seriese in the runway | 11:58 |
| gibi | lol | 11:58 |
| gibi | I make a mental note to do it more often | 11:58 |
| gibi | :D | 11:58 |
| sean-k-mooney | with the amount of patches in stephens seriese im not even sure he would notice one more when he is rebaseing them all | 11:59 |
| gibi | :D | 11:59 |
| gibi | btw, stephenfin: I've finished reading and commenting the vtpm series | 11:59 |
| sean-k-mooney | it looks like they are all approved | 12:00 |
| sean-k-mooney | at least for spwan | 12:00 |
| sean-k-mooney | the move operation are still pending | 12:00 |
| *** bbowen has quit IRC | 12:01 | |
| gibi | yeah, spawn seems pretty solid to me | 12:01 |
| sean-k-mooney | this will make efried happy to see | 12:01 |
| bauzas | I'm just reviewing the rest of the series | 12:03 |
| bauzas | but honestly, mho is that we tried to be gentlemen with vtpm usage, by overcomplicating what we were providing as a feature :) | 12:03 |
| bauzas | trying to have vtpm instances be pets is nice | 12:03 |
| *** bbowen has joined #openstack-nova | 12:04 | |
| bauzas | but we're pulling swift and trying to do maths to get whether we can still migrate an emulated file device or not, which overcomplicates things | 12:04 |
| bauzas | but this is what it is, the spec is approved | 12:04 |
| sean-k-mooney | well we try to make it behave like a real server | 12:05 |
| bauzas | this is cloud | 12:05 |
| sean-k-mooney | yep | 12:06 |
| bauzas | most of the operations are admin-managed | 12:06 |
| sean-k-mooney | in a cloud that would be atypical | 12:06 |
| bauzas | so in this case, trying to solve all the problems is nice, but this comes with a cost | 12:06 |
| sean-k-mooney | well one of the proposal was no move operations | 12:06 |
| bauzas | but the ship has sailed. | 12:06 |
| sean-k-mooney | which was very non admin frendly | 12:07 |
| bauzas | I'm okay with creating a fresh new tpm device with move operations | 12:07 |
| bauzas | what I'm struggling with is trying to reuse the existing data | 12:07 |
| bauzas | this should be app-managed | 12:07 |
| sean-k-mooney | the check is pretty simple | 12:07 |
| sean-k-mooney | if anything changes version or type then data is not perserved | 12:07 |
| sean-k-mooney | if the version and type do not change we copy it | 12:08 |
| stephenfin | bauzas: well, don't worry about swift. None of that is done | 12:08 |
| sean-k-mooney | the special case is shelve | 12:08 |
| stephenfin | Is we might never do it. For now, shelve is simply blocked | 12:08 |
| sean-k-mooney | where we ahve to store it somewhere which is swift | 12:08 |
| sean-k-mooney | bauzas: the alternitive was to stick it in glance as a second image | 12:09 |
| sean-k-mooney | or not store it at all | 12:09 |
| bauzas | sean-k-mooney: yup, I saw in the series | 12:11 |
| bauzas | the shelve case is the most complicated | 12:11 |
| bauzas | I also have concerns with rebuild | 12:11 |
| bauzas | trying to reuse the existing backup file is nice, but as a user I'd expect a fresh vTPM device if I rebuild | 12:12 |
| sean-k-mooney | rebuild preseves the data if the version/type of tpm does not change | 12:12 |
| sean-k-mooney | bauzas: that fails the do what hardware does guideline | 12:12 |
| bauzas | this data is ephemeral, right? | 12:12 |
| sean-k-mooney | bauzas: if you reinstall the os on your laptop your tpm is not cleared | 12:12 |
| bauzas | ah, you're right, if you wanna mock the baremetal case | 12:13 |
| bauzas | but in this case, all moves are just data loss | 12:13 |
| bauzas | especially when I read https://specs.openstack.org/openstack/nova-specs/specs/victoria/approved/add-emulated-virtual-tpm.html#migrations-and-their-ilk | 12:13 |
| bauzas | brr... | 12:13 |
| sean-k-mooney | bauzas: one of the usecases for this is storing encyrption keys or certs for authentication | 12:13 |
| sean-k-mooney | so if you have data volumes or use the tpm as a cert store and use rebuild to upgrade your application you would not want to loose that data | 12:14 |
| bauzas | gibi: thoughts on keeping a list of CI maintainers somewhere in https://review.opendev.org/#/c/742407/5 | 12:14 |
| bauzas | gibi: if you have remembrance of anything about this... | 12:14 |
| openstackgerrit | Merged openstack/nova master: Fix indentation nits https://review.opendev.org/746765 | 12:14 |
| openstackgerrit | Merged openstack/nova master: Remove deprecated scheduler filters https://review.opendev.org/744800 | 12:14 |
| sean-k-mooney | bauzas: all move operation in the sepc are currently data perserving | 12:15 |
| bauzas | I do understand the usecase | 12:15 |
| sean-k-mooney | bauzas: we just copy the tpm data form one host to another | 12:15 |
| bauzas | I'm just saying we are adding more than necessary | 12:15 |
| bauzas | but again, the ship has sailed | 12:15 |
| sean-k-mooney | bauzas: well it was felt that without perserving the data on move ops we could not resonably support them | 12:16 |
| openstackgerrit | Merged openstack/nova master: tests: Add helpers for rebuild, cold migrate, and shelve/unshelve https://review.opendev.org/747792 | 12:16 |
| sean-k-mooney | bauzas: complexity is a valid concern but i think the current semantics are what i would expect | 12:16 |
| bauzas | sean-k-mooney: yup, and again, I won't bother | 12:19 |
| bauzas | we had a consensus, I do respect it. | 12:20 |
| efried | A lot of blood, sweat, and tears went into getting that spec to where it is. | 12:20 |
| efried | Which involved some very heavy tradeoffs and compromises along the way. | 12:21 |
| efried | There's some saying like, "You know it's a good compromise when nobody is happy." | 12:21 |
| *** _erlon_ has joined #openstack-nova | 12:22 | |
| efried | Specifically to address the notion of pets: if you want a TPM, your VM is already a pet. | 12:22 |
| gibi | efried: o/ | 12:23 |
| efried | I will say that I am happy to see things like vTPM and provider config merging now -- two releases after I stopped working on them, which was multiple releases after I (and others) started working on them. | 12:23 |
| elod | lyarwood: btw, should we wait this to merge too? >>> https://review.opendev.org/#/c/744550/ | 12:23 |
| lyarwood | elod: that's less important tbh | 12:24 |
| lyarwood | elod: just a nice to have, the other change fixes an issue introduced by an earlier change | 12:24 |
| sean-k-mooney | efried: yep it will be nice to see them in the victoria project update hopefully as highlights | 12:26 |
| elod | lyarwood: ok, then I've updated the release patch | 12:27 |
| elod | the other patch unfortunately had to recheck anyway, so it would mean at least 3-4 hrs to merge... | 12:28 |
| bauzas | gibi: lyarwood: stephenfin: -1 for relnote on https://review.opendev.org/#/c/631363/61 | 12:32 |
| bauzas | stephenfin: I'm currently digging the long list of https://review.opendev.org/#/q/topic:bp/add-emulated-virtual-tpm+status:merged but I can't find where we defined docs | 12:33 |
| gibi | ack, do we need to pull it out of the gate? | 12:33 |
| gibi | bauzas: doc is in the next patch | 12:33 |
| bauzas | ah right | 12:33 |
| gibi | https://review.opendev.org/#/c/739213/17 | 12:33 |
| bauzas | that's what I just saw | 12:34 |
| bauzas | okay, I think we can leave as it is since lyarwood already approved the above patch but this sucks :( | 12:34 |
| bauzas | and I personally feel we should rewrite the reno file to be more explicit by saying "sorry folks, only spawn as of now" | 12:34 |
| bauzas | given what efried said on the life of pets, once people see TPM devices around in nova, they gonna start using those without knowing they're stukc | 12:35 |
| bauzas | stephenfin: ^ | 12:35 |
| gibi | bauzas: at least the doc is explicit about what does not work with vtpm | 12:35 |
| bauzas | (provided we don't merge the move ops changes in the next 2 weeks obviously) | 12:35 |
| stephenfin | bauzas: I can do a follow-up | 12:35 |
| bauzas | stephenfin: wfm | 12:36 |
| stephenfin | cool. gimme a sec | 12:36 |
| bauzas | and yeah, docs are expicit | 12:36 |
| bauzas | explicit even | 12:36 |
| lyarwood | yeah I was under the impression that things were going to be enabled before GA anyway so it would be a waste of time detailing everything in the releasenote | 12:36 |
| efried | BTW, it'll be a good idea to take your cues from jroll and penick if you have questions about how these are going to be used irl. They're the primary consumer driving this work, at least initially. | 12:37 |
| lyarwood | as opposed to updating the docs | 12:37 |
| bauzas | stephenfin: cool, and then say we support resize, you could then amend your renofile to include "spawn [and resize]" | 12:37 |
| bauzas | lyarwood: I never predict on things being merged | 12:37 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: rbd: Move rbd_utils out of libvirt driver under nova.storage https://review.opendev.org/746904 | 12:37 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: Add ability to download Glance images into the libvirt image cache via RBD https://review.opendev.org/574301 | 12:37 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: Bring back allowed_direct_url_schemes in support of RBD image download https://review.opendev.org/728095 | 12:37 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Add devstack-plugin-ceph-compute-qcow2 to experimental queue https://review.opendev.org/743220 | 12:37 |
| bauzas | lyarwood: and I'm on the likes to consider master as something usable as of now | 12:38 |
| lyarwood | bauzas: right but my point is that it's easier to update the docs than a reno IMHO | 12:38 |
| bauzas | lyarwood: but I know this is an utopia | 12:38 |
| bauzas | lyarwood: well, it's just a file change, but either way | 12:38 |
| bauzas | docs are good for getting all the details | 12:38 |
| bauzas | release notes are good for knowing what's in the release | 12:38 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: releasenotes: Detail support for server ops with vTPM https://review.opendev.org/748215 | 12:40 |
| stephenfin | bauzas, gibi, lyarwood: nice and generic ^ | 12:40 |
| bauzas | ++ | 12:40 |
| sean-k-mooney | lyarwood: updating a reno is not that hard | 12:41 |
| sean-k-mooney | we can even backport reno updtes it does actully work | 12:41 |
| stephenfin | sean-k-mooney: it is tedious though | 12:41 |
| bauzas | tedious in what ? | 12:41 |
| stephenfin | especially when you've already got detailed info in docs | 12:41 |
| bauzas | stephenfin: well, docs were giving me a 404 | 12:42 |
| *** mkrai has joined #openstack-nova | 12:42 | |
| stephenfin | bauzas: to update incrementally as you tweak features | 12:42 |
| bauzas | hence the -1 | 12:42 |
| sean-k-mooney | yes as long as the reno is not wrong and directs you to the docs | 12:42 |
| sean-k-mooney | then keeping the docs up to date makes sense | 12:42 |
| lyarwood | bauzas: docs are the next change in the series | 12:42 |
| stephenfin | oh, I thought that was okay since it's fixed in the next change? | 12:42 |
| bauzas | I'm still on the side to have a digested list of features | 12:42 |
| bauzas | stephenfin: I'd say I passed over ti | 12:42 |
| bauzas | it | 12:42 |
| bauzas | but again, and I know I'm now maybe in the minority, I don't like merging broken code that will be fixed in the next change of the series | 12:43 |
| sean-k-mooney | stephenfin: i have done the update in the past its not that bad same as updating the docs | 12:43 |
| bauzas | even if the change in question is next in the queue | 12:43 |
| bauzas | that gives us bad habits | 12:43 |
| stephenfin | true. It's not broken code though, in fairness :) | 12:44 |
| lyarwood | are we classing releasenotes as code now? | 12:44 |
| bauzas | it's broken docs, this is worst, indeed :p | 12:44 |
| lyarwood | ops stephenfin beat me to it | 12:44 |
| stephenfin | lyarwood: coke, please | 12:44 |
| lyarwood | you're only getting warm tap water from me mate | 12:44 |
| bauzas | in our example, we could have left the change as it is, with a reno file just giving the exact things you provided in your FUP, without providing a broken link | 12:45 |
| bauzas | and in the next change, we could have provided the docs with an amended reno file that'd add the link | 12:45 |
| bauzas | that's what I just call good habits | 12:45 |
| stephenfin | yup, that's fair | 12:46 |
| stephenfin | In this instance though, are you okay to leave it in the gate queue or should I pull it out and rework it? | 12:46 |
| bauzas | stephenfin: on a fun side, should I *really* review https://review.opendev.org/#/c/742863/9 and all the type hints changes, given all the discussions we had in the past ? :D | 12:46 |
| bauzas | stephenfin: nah, I said I can assume this being fixed automatically | 12:47 |
| stephenfin | okay, cool | 12:47 |
| bauzas | stephenfin: so, tbh, I guess you're doing https://review.opendev.org/#/c/742863/9/nova/compute/manager.py unrelated to the vTPM series, right? | 12:48 |
| stephenfin | bauzas: Up to you :) It is really trivial, if that helps (not much hints being added, just getting mypy passing) | 12:48 |
| bauzas | ahah, I see | 12:48 |
| stephenfin | yeah, see my reply to lyarwood | 12:48 |
| stephenfin | I'm adding new type functions in the next change and I want to annotate them and validate those annotations. To enable validation, I need to fix a couple of issues with the file. That's what I'm doing here. | 12:48 |
| bauzas | you added type hints for the vtpm changes but this didn't work, gotcha | 12:48 |
| stephenfin | exactly | 12:48 |
| stephenfin | it felt wrong to add type hints but not validate them, and I couldn't enable validation without some additional work | 12:49 |
| stephenfin | so I did that work separately since it's not totally related | 12:49 |
| stephenfin | if that makes sense | 12:49 |
| bauzas | stephenfin: yeah | 12:49 |
| bauzas | that's not I wanna argue against type hints | 12:50 |
| bauzas | if you wanna use them, this works for me | 12:50 |
| bauzas | but let's not make it mandatory for all of us | 12:50 |
| stephenfin | yeah, to be clear, I'm not asking it of anyone who doesn't want to add them | 12:51 |
| *** smcginnis has joined #openstack-nova | 12:51 | |
| sean-k-mooney | once a file has first been enabeld for type hints you can add them inline | 12:51 |
| sean-k-mooney | instead of as a seperate patch | 12:51 |
| stephenfin | but I'd like to add them to my code. I find them helpful | 12:51 |
| *** ratailor has quit IRC | 12:51 | |
| sean-k-mooney | but when your ading a new file you often have to update unrelated code to get it to pass | 12:51 |
| sean-k-mooney | well semingly unrelated | 12:51 |
| bauzas | there are two ways of seeing type hints | 12:52 |
| stephenfin | sean-k-mooney: yeah, that's what I'm doing for new code at least | 12:53 |
| stephenfin | for expected backports, melwitt and others have asked I do them separately to keep changes to the backport to a minimum, which I can deal with :) | 12:53 |
| bauzas | either as a terrible waste of time for reviewers, or as an opportunity for code readability and static checks | 12:53 |
| sean-k-mooney | bauzas: yep i am stongly in teh latter camp | 12:53 |
| stephenfin | s/type hints/tests/ | 12:53 |
| bauzas | I'm not opiniated on any of both, I just try to be pragmatic | 12:53 |
| *** ratailor has joined #openstack-nova | 12:53 | |
| sean-k-mooney | lack of type hints is why i hated python for the first 2-3 years of working on openstack | 12:54 |
| stephenfin | same argument can be made, though the value of tests is definitely more significant | 12:54 |
| bauzas | my biggest concern being this a py3 feature, but given we now have a good consensus on things we can vs we can't do, I'm okay | 12:54 |
| sean-k-mooney | wehn we started adding the parms comment it helped a bit but not as much as type hints | 12:54 |
| stephenfin | but it's also dead code from the perspective of production | 12:54 |
| bauzas | adding only type hints on new code seems reasonable... but... | 12:54 |
| stephenfin | the current situation works for me (add them as you're going, and only if you want to) | 12:55 |
| stephenfin | bauzas: to be clear, new code or modified code | 12:55 |
| bauzas | there is a trade-off, the more we're using them, the higher chances we can have merge conflicts with backporting | 12:55 |
| sean-k-mooney | yes which i think is ok | 12:55 |
| sean-k-mooney | we will have merge conflitcs anyway | 12:55 |
| sean-k-mooney | due to python3 vs python 2 | 12:56 |
| lyarwood | ha if you think this is going to cause merge conflicts wait until you see stephenfin's func test series :D | 12:56 |
| sean-k-mooney | which one there is one every cycle :P | 12:56 |
| lyarwood | it's all worthwhile however :) | 12:56 |
| sean-k-mooney | yep stephenfin has been making the func tests more and more useful | 12:57 |
| bauzas | well, I don't know for you folks but I'm working in a company that promises 5-year support for OpenStack | 12:57 |
| stephenfin | as do I. I think we add a little _too_ much weight to preserving backportability, at the expense of making master worse | 12:57 |
| stephenfin | i.e. don't move these lines around because it'll make the backport harder, even though doing so would make things significantly easier to grok on master | 12:57 |
| stephenfin | gibi: good spot on the chown thing | 12:57 |
| lyarwood | bauzas: you missed a downstream discussion around stable/train while you were out FWIW | 12:57 |
| bauzas | damn, it's a cabal ! | 12:57 |
| stephenfin | gibi: Functional tests should have caught that. I don't know why it didn't for sure but I suspect the dumb mocking of that cleanup routine is the culprit though | 12:58 |
| *** ftarasenko has quit IRC | 12:58 | |
| sean-k-mooney | lyarwood: did i also miss that i dont rememebr it | 12:58 |
| * sean-k-mooney may have blocked out all knowladge of it as a defence mechanium not sure | 12:58 | |
| gibi | stephenfin: if you can catch those things in func test then I can ease on my tempest requirements | 12:58 |
| stephenfin | lyarwood: you mean the question of backporting functional test rework changes | 12:58 |
| bauzas | lyarwood: I feel like I was going to be caught in Varennes | 12:58 |
| bauzas | that's a coup ! | 12:58 |
| lyarwood | sean-k-mooney: I think you were there | 12:59 |
| lyarwood | stephenfin: yeah | 12:59 |
| sean-k-mooney | oh that | 12:59 |
| stephenfin | what does a Frenchman know of coups | 12:59 |
| sean-k-mooney | dont we always backport them | 12:59 |
| sean-k-mooney | to keep as close to master as possible | 12:59 |
| sean-k-mooney | to make future backports simpler | 12:59 |
| lyarwood | not always | 12:59 |
| *** bnemec has joined #openstack-nova | 12:59 | |
| stephenfin | sometimes yes, sometimes no | 12:59 |
| sean-k-mooney | ok i guess it depend on how invasive it is | 12:59 |
| stephenfin | lyarwood's suggesting we always do, even if there are quite a few, since it will ultimately make our lives easier | 13:00 |
| sean-k-mooney | just being pargmatic about it | 13:00 |
| stephenfin | and it's test only so impact on stability should be non-existent | 13:00 |
| lyarwood | yup | 13:00 |
| bauzas | I see | 13:00 |
| *** ftarasenko has joined #openstack-nova | 13:00 | |
| bauzas | well, for functional test only changes, this could work | 13:00 |
| sean-k-mooney | bauzas: i think it came up in the context of one of melwitt patches where she would have had to rewrite the tests if she did not backport a refactor patch | 13:01 |
| sean-k-mooney | and i suggested she shoudl backport the refactor | 13:01 |
| sean-k-mooney | to avoid that and keep the backport clean | 13:01 |
| stephenfin | gibi: Also, it's always been broken, i.e. blame efried, not me :P https://review.opendev.org/#/c/639934/30/nova/virt/libvirt/utils.py@691 | 13:01 |
| bauzas | sean-k-mooney: which makes sense from a pragmatic pov | 13:02 |
| *** mkrai has quit IRC | 13:02 | |
| * stephenfin stops shifting blame and starts fixing stuff | 13:02 | |
| efried | I blame reviewers | 13:02 |
| *** mkrai has joined #openstack-nova | 13:02 | |
| bauzas | but again, we all sign-off for maintaining code for half a decade, refrain your style changes | 13:02 |
| sean-k-mooney | bauzas: or backport them | 13:02 |
| bauzas | we all = those who wear some head coverage with a color (that is less red and more blue) | 13:03 |
| sean-k-mooney | the more consitent we are the eaiser it is to grock the code | 13:03 |
| efried | I don't actually recognize that code; I may get to blame mdbooth | 13:03 |
| bauzas | sean-k-mooney: I'd say that providing style changes in an implementation patch is terrible and we should avoid this | 13:03 |
| efried | or cfriesen | 13:04 |
| gibi | stephenfin, efried: I can take the blame found the issue :) | 13:04 |
| gibi | finding even | 13:04 |
| sean-k-mooney | right but type hints are not a style change | 13:04 |
| sean-k-mooney | they are a type of testing | 13:04 |
| lyarwood | have to say I disagree bauzas, we'd never be able to move the codebase forward otherwise | 13:04 |
| sean-k-mooney | its technically a liniting enhancement | 13:04 |
| stephenfin | as someone who's paid to work on the libvirt module, so do I | 13:05 |
| bauzas | gibi: the story of my life https://i.pinimg.com/564x/c5/de/f4/c5def4bd565b44f4d27e1b7478dfa528.jpg | 13:05 |
| stephenfin | so. much. cruft. | 13:05 |
| sean-k-mooney | bauzas: hehe | 13:05 |
| lyarwood | awww ./me hugs nova.virt.libvirt | 13:05 |
| stephenfin | heh | 13:05 |
| bauzas | lyarwood: stephenfin: I don't say we shouldn't modify code | 13:05 |
| sean-k-mooney | bauzas: worst part of git blame. who wrote this crap.. damb it was was me... | 13:06 |
| *** bhagyashris is now known as bhagyashris|away | 13:06 | |
| stephenfin | bauzas: well, take things like this https://review.opendev.org/#/c/631363/61/nova/tests/unit/virt/libvirt/fakelibvirt.py@135 | 13:06 |
| efried | Hah! Yes! I get to blame Chris! https://review.opendev.org/#/c/639934/7..8/nova/virt/libvirt/utils.py@630 | 13:06 |
| stephenfin | personally, I think that's a good thing | 13:06 |
| gibi | efried: good, he is not around to pass it forward :) | 13:07 |
| efried | Perfect | 13:07 |
| sean-k-mooney | oh recursive=true | 13:07 |
| stephenfin | I mean, I'm touching that code and the additional change makes the whole thing better | 13:08 |
| bauzas | stephenfin: again, no disagreement, just saying that the more we can decouple style changes from implementation changes, the better it would be as we could pull those style changes down to the stable branches without also pulling fancy new features | 13:08 |
| sean-k-mooney | stephenfin: did you see if this is need for livemiration by the way. i think we dont have too copy for live migration but we never validated that | 13:08 |
| sean-k-mooney | stephenfin: so currently we do just to be safe | 13:08 |
| stephenfin | sean-k-mooney: Currently I have live migration blocked because I haven't validated it | 13:09 |
| sean-k-mooney | ok well when you do can you try commenting out the tpm data copy | 13:09 |
| stephenfin | I do have a DevStack env on your cluster that I'll use to validate it once the rest has merged though | 13:09 |
| sean-k-mooney | and see if it works | 13:09 |
| bauzas | stephenfin: fortunately, git is smart enough to handle conflicts on blank lines | 13:09 |
| stephenfin | sean-k-mooney: If we're talking about the same thing, then that's for cold migration | 13:09 |
| bauzas | (or it should) | 13:09 |
| sean-k-mooney | https://review.opendev.org/#/c/639934/30/nova/virt/libvirt/utils.py@691 | 13:09 |
| sean-k-mooney | the save and restore | 13:09 |
| stephenfin | Yes, that is necessary. I checked the libvirt docs and included the relevant snippet in the commit message | 13:10 |
| sean-k-mooney | its needed for cold migration | 13:10 |
| stephenfin | Libvirt | 13:10 |
| stephenfin | won't do this automatically for us since cold migrations, or offline | 13:10 |
| stephenfin | migrations in libvirt lingo, do not currently support "copying | 13:10 |
| stephenfin | non-shared storage or other file based storages", which includes the | 13:10 |
| stephenfin | vTPM device [1]. | 13:10 |
| sean-k-mooney | stephenfin: but the qemu example seamed to imply it is not for live | 13:10 |
| sean-k-mooney | but there was not clear statement of that | 13:10 |
| stephenfin | ah, yes | 13:11 |
| stephenfin | well | 13:11 |
| stephenfin | <stephenfin> sean-k-mooney: Currently I have live migration blocked because I haven't validated it | 13:11 |
| stephenfin | so I can try when I do the validation, sure :) | 13:11 |
| sean-k-mooney | stephenfin: cool | 13:11 |
| sean-k-mooney | stephenfin: it was an open question in the spec to be figured out when we do the implemnation so jsut reminding you | 13:11 |
| stephenfin | bauzas: Fair. I guess it's just a matter of where to draw the line | 13:11 |
| stephenfin | I'd be less conservative about this stuff that you are, I suspect | 13:11 |
| bauzas | stephenfin: if any of us can just think every time we write on maintainability and upgrades, this would just be perfect. | 13:12 |
| bauzas | stephenfin: totally, and I recognize me being a lame coworker | 13:12 |
| bauzas | :p | 13:12 |
| stephenfin | yeah, stop being so lame! | 13:13 |
| stephenfin | ;) | 13:13 |
| sean-k-mooney | stephenfin: https://github.com/qemu/qemu/blob/master/docs/specs/tpm.rst#migration-with-the-tpm-emulator | 13:14 |
| sean-k-mooney | since there is no copy in those steps i was sepcualtign it was in the testvm.bin file | 13:15 |
| sean-k-mooney | but its also posible this was on the same host so -tpmstate dir=/tmp/mytpm1 could be acceable | 13:16 |
| *** dave-mccowan has joined #openstack-nova | 13:17 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/nova master: DO NOT REVIEW: Test OVN devstack module https://review.opendev.org/748226 | 13:18 |
| * bauzas bailing out for 1 hour | 13:19 | |
| *** sapd1_x has joined #openstack-nova | 13:22 | |
| *** nweinber has joined #openstack-nova | 13:23 | |
| *** brinzhang has quit IRC | 13:41 | |
| *** brinzhang has joined #openstack-nova | 13:41 | |
| *** ratailor has quit IRC | 13:46 | |
| *** brinzhang has quit IRC | 13:48 | |
| *** brinzhang has joined #openstack-nova | 13:48 | |
| *** Liang__ has joined #openstack-nova | 13:49 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: Ensure source compute is up when confirming a resize https://review.opendev.org/699291 | 13:50 |
| lyarwood | gibi: ^ would you be able to take a look at this when you have time? | 13:50 |
| *** Liang__ is now known as LiangFang | 13:50 | |
| gibi | lyarwood: sue | 13:50 |
| lyarwood | gibi: thanks | 13:50 |
| gibi | sure | 13:50 |
| lyarwood | stephenfin: the rebase to remove the merge conflict just removed your +2 if you can readd | 13:51 |
| *** mkrai has quit IRC | 13:52 | |
| stephenfin | will do | 13:54 |
| lyarwood | thanks | 13:54 |
| *** jangutter_ has joined #openstack-nova | 13:56 | |
| *** jangutter has quit IRC | 13:59 | |
| openstackgerrit | Merged openstack/nova master: libvirt: Add emulated TPM support to Nova https://review.opendev.org/631363 | 14:10 |
| openstackgerrit | Merged openstack/nova master: docs: Add docs for vTPM support https://review.opendev.org/739213 | 14:10 |
| lyarwood | bauzas / melwitt: https://review.opendev.org/#/c/747973/ - should be ready if either of you have time today. | 14:10 |
| openstackgerrit | Merged openstack/nova master: tests: Add reproducer for bug #1889633 https://review.opendev.org/744020 | 14:10 |
| openstack | bug 1889633 in OpenStack Compute (nova) "Pinned instance with thread policy can consume VCPU" [High,In progress] https://launchpad.net/bugs/1889633 - Assigned to Stephen Finucane (stephenfinucane) | 14:10 |
| openstackgerrit | Merged openstack/nova master: hardware: Reject requests for no hyperthreads on hosts with HT https://review.opendev.org/744021 | 14:11 |
| *** mkrai has joined #openstack-nova | 14:11 | |
| bauzas | lyarwood: ack, will look | 14:29 |
| openstackgerrit | Alexandre Arents proposed openstack/nova master: Add a lock to prevent race during detach/attach of interface https://review.opendev.org/747957 | 14:31 |
| *** derekh has quit IRC | 14:32 | |
| *** derekh has joined #openstack-nova | 14:32 | |
| lyarwood | bauzas: thanks :) | 14:36 |
| *** dustinc has joined #openstack-nova | 14:37 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add type hints to 'nova.virt.libvirt.utils' https://review.opendev.org/742865 | 14:39 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support for resize and cold migration of emulated TPM files https://review.opendev.org/639934 | 14:39 |
| stephenfin | gibi: Turns out a more granular mocking of migrate_disk_and_power_off is quite the bit of work. Have pushed up all the other fixes. Will keep working on that to get full coverage | 14:44 |
| *** lpetrut has joined #openstack-nova | 14:46 | |
| gmann | sean-k-mooney: noonedeadpunk: RE on policy for flavor ephemeral storage control. we can add policy rule for that and allow everyone by default so that we do not break existing usage but my concern is, if restricted by policy then it will control all request not specific to host or storage, is that fine? | 14:50 |
| bauzas | stephenfin: https://review.opendev.org/#/c/744958/7 and https://review.opendev.org/#/c/741995/10 look the same, I guess it's PEBKAC ? | 14:50 |
| sean-k-mooney | gmann: flavor creation is admin only by default | 14:50 |
| sean-k-mooney | gmann: so the new policy would also have to be the same | 14:50 |
| gmann | sean-k-mooney: ah yeah with current default. | 14:50 |
| openstackgerrit | Lucian Petrut proposed openstack/nova master: Avoid invalid file name, preventing git clone on win32 https://review.opendev.org/748250 | 14:51 |
| bauzas | stephenfin: wow https://review.opendev.org/#/q/owner:stephenfin%2540redhat.com+status:open | 14:51 |
| bauzas | do you sleep overnight? | 14:51 |
| sean-k-mooney | gmann: can we make default dynamic | 14:51 |
| gmann | sean-k-mooney: but i was reading chat, is noonedeadpunk use case for specific host to control such VM creation? | 14:51 |
| sean-k-mooney | gmann: e.g. default it to whatever teh current flavor create policy is | 14:51 |
| gmann | sean-k-mooney: dynamic? | 14:51 |
| gmann | sean-k-mooney: yeah, we can do that | 14:51 |
| sean-k-mooney | gmann: no i dont think so | 14:52 |
| sean-k-mooney | gmann: i think noonedeadpunk wanted to restrct who could create flavors with local storage | 14:52 |
| sean-k-mooney | there are ways to prevent host from booting gues with local sotrage alredy | 14:52 |
| lpetrut | hi, looks like we have an unusual issue on Windows: https://review.opendev.org/#/c/748250/ a release note filename containing pipes prevents Nova from being cloned on Windows :) | 14:52 |
| gmann | sean-k-mooney: i see | 14:52 |
| sean-k-mooney | gmann: either via config or using a trait and isolated aggreates feature | 14:53 |
| noonedeadpunk | to be exact, I wanted to restrict users who can create VMs with ephemeral storages | 14:53 |
| gmann | ok | 14:53 |
| sean-k-mooney | noonedeadpunk: that is slightly different then | 14:53 |
| noonedeadpunk | For example I generally want to prohibit usage of ephemeral/local storages,m but what 1 groups of users to be still able to do that | 14:53 |
| sean-k-mooney | one way to do that is to make the ephemeral flavors private | 14:53 |
| openstackgerrit | Alex Xu proposed openstack/nova master: Provider Config File: Enable loading and merging of provider configs https://review.opendev.org/693460 | 14:54 |
| sean-k-mooney | so make all you public flaovr bfv only | 14:54 |
| stephenfin | bauzas: They are identical, but I didn't want to make the two series depend on each other | 14:54 |
| noonedeadpunk | but even with lavor with 0 disk it's possible to create server without cinder volume | 14:54 |
| openstackgerrit | Stephen Finucane proposed openstack/nova stable/ussuri: tests: Add reproducer for bug #1889633 https://review.opendev.org/748251 | 14:54 |
| openstack | bug 1889633 in OpenStack Compute (nova) ussuri "Pinned instance with thread policy can consume VCPU" [High,Triaged] https://launchpad.net/bugs/1889633 | 14:54 |
| stephenfin | bauzas: I'll rebase once one of them lands | 14:54 |
| openstackgerrit | Stephen Finucane proposed openstack/nova stable/ussuri: hardware: Reject requests for no hyperthreads on hosts with HT https://review.opendev.org/748252 | 14:54 |
| noonedeadpunk | s/lavor/flavor/ | 14:54 |
| sean-k-mooney | then use flavor access to allow access to the other falvors to a specific tenant or set of tenants | 14:54 |
| gmann | sean-k-mooney: noonedeadpunk we can default it to 'os_compute_api:os-flavor-manage:create' rule itself so whatever create policy has in any deployment (default or overridden ) stays the same with new policy check. | 14:54 |
| alex_xu | stephenfin: gibi ^ rebased 693460, the conflict at admin doc | 14:54 |
| bauzas | stephenfin: I'd have appreciated a -2 somewhere :) | 14:54 |
| bauzas | or a DNM | 14:55 |
| sean-k-mooney | gmann: that would not actully address noonedeadpunk usecase | 14:55 |
| sean-k-mooney | it would do what i asked you | 14:55 |
| stephenfin | bauzas: well that would have defeated the purpose, wouldn't it? :P | 14:55 |
| sean-k-mooney | we would need a policy on spwan that blocked ephemeral flavors | 14:55 |
| lyarwood | elod: https://review.opendev.org/#/c/747358/ should also be ready now btw | 14:55 |
| bauzas | stephenfin: one is a bugfix, the other one is a feature that is proritized in a runwayu | 14:55 |
| lyarwood | elod: unless you want me to squash | 14:55 |
| sean-k-mooney | unless you were in a specific group | 14:55 |
| bauzas | stephenfin: so, honestly... :D | 14:56 |
| sean-k-mooney | but i think falvor access and tenants achive a similar result | 14:56 |
| noonedeadpunk | sean-k-mooney: I mean there's no way to create a flavor that prohibit usage of ephemeral/local volumes | 14:56 |
| gmann | sean-k-mooney: yeah | 14:56 |
| *** dklyle has joined #openstack-nova | 14:56 | |
| sean-k-mooney | noonedeadpunk: well the flavor specifcy the disk usage | 14:56 |
| sean-k-mooney | you cant consme disk based on image metadata | 14:57 |
| stephenfin | Yeah, I wasn't sure how else to do it without ending up with a 15+ patch series :-\ | 14:57 |
| sean-k-mooney | but you can restict who can use that flavor | 14:57 |
| noonedeadpunk | yes, but even with 0 disk and 0 swap - ephemeral will still be created if volume is not provided? | 14:57 |
| *** tkajinam has quit IRC | 14:57 | |
| stephenfin | especially given I couldn't place it at the bottom of either series | 14:57 |
| sean-k-mooney | noonedeadpunk: if you dont use boot form volume | 14:57 |
| sean-k-mooney | although i argued that its a bug that that is allows | 14:57 |
| noonedeadpunk | Yeah, so what I would be expecting is that nave printed that it's not allowed for instance | 14:58 |
| sean-k-mooney | i wanted 0 disk to mean bfv only | 14:58 |
| sean-k-mooney | other felt that would break some pepole | 14:58 |
| *** lpetrut has quit IRC | 14:58 | |
| sean-k-mooney | noonedeadpunk: we may or may not beable to contole that by policy dependingo n what info is currenlty passed to server create | 14:59 |
| noonedeadpunk | That why I was talking about policy, which can be configured if just reject users request when he tries to boot server without volume with 0 disk in flavor | 14:59 |
| gmann | flavor creation control is not needed really but booting VM control with new policy | 14:59 |
| sean-k-mooney | gmann: yes it would have to be a check on server create | 14:59 |
| sean-k-mooney | either an api check or policy | 14:59 |
| stephenfin | bauzas: if you can hit https://review.opendev.org/#/c/747744/ then I can drop one of those patches and rebase both series onto one | 15:00 |
| sean-k-mooney | oh donwstream call got to go | 15:00 |
| noonedeadpunk | like the point here is that ppl on public clouds usually not sure what they want, and just go the easiest path, which results in lost data for them... | 15:00 |
| bauzas | stephenfin: yeah, that's possible | 15:00 |
| *** hemna has quit IRC | 15:01 | |
| gmann | sean-k-mooney: noonedeadpunk: in that case we can default the new policy with this https://github.com/openstack/nova/blob/a7735d5e379c20c59cfb639f9f4d339bcffca2f9/nova/policies/servers.py#L167 | 15:01 |
| *** hemna has joined #openstack-nova | 15:02 | |
| *** LiangFang has quit IRC | 15:03 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: tests: Add reproducer for bug #1889633 https://review.opendev.org/748254 | 15:04 |
| openstack | bug 1889633 in OpenStack Compute (nova) ussuri "Pinned instance with thread policy can consume VCPU" [High,In progress] https://launchpad.net/bugs/1889633 - Assigned to Stephen Finucane (stephenfinucane) | 15:04 |
| openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: hardware: Reject requests for no hyperthreads on hosts with HT https://review.opendev.org/748255 | 15:04 |
| noonedeadpunk | I mean like `os_compute_api:servers:create:zero_disk_flavor` is already checking the flavor. Maybe somewhere near it's possible to check if flavor has zero size disk, but has no volume attached? | 15:05 |
| noonedeadpunk | I'm not really familiar with nova codebase as it's so huge... | 15:05 |
| sean-k-mooney | you proably need the block device mappings | 15:06 |
| * noonedeadpunk already got lost in suggestions....... | 15:07 | |
| sean-k-mooney | that said i think we have a bfv flag somewhere | 15:07 |
| noonedeadpunk | At least I hope I told userstory I'm trying to implement in an understandable way.... | 15:08 |
| sean-k-mooney | you did | 15:08 |
| sean-k-mooney | you want to restict who can boot a vm without bfv | 15:08 |
| sean-k-mooney | using policy and roles | 15:09 |
| sean-k-mooney | right | 15:09 |
| noonedeadpunk | ideally yes. How to restrict it on whole cloud I guess you've already answered and it's with max_local_block_devices | 15:10 |
| noonedeadpunk | eventually that will probably work for me as well | 15:10 |
| sean-k-mooney | yes the config option will also block it much latere on the compute host rather then api | 15:11 |
| noonedeadpunk | (in the worst case scenario would use isolate aggregates which will be also tricky because of the need to separate resources) | 15:12 |
| noonedeadpunk | but yeah, checking that on api level might be more efficient | 15:12 |
| noonedeadpunk | and produce more understandable output | 15:12 |
| sean-k-mooney | an api check woudl allow us to retrun a specific api respoce saying why | 15:14 |
| sean-k-mooney | currently it would jsut get a no valid host | 15:14 |
| noonedeadpunk | yeah | 15:14 |
| sean-k-mooney | which is not a great user experice | 15:14 |
| gmann | zero_disk_flavor policy already control the zero disk boot if it is not bfv | 15:14 |
| noonedeadpunk | zero_disk_flavor eventually can only prohibit spawning a VM when it's created with flavor with 0 disk | 15:15 |
| sean-k-mooney | oh so the edgecase it did not fix was allowing bfz when disk >0 | 15:15 |
| sean-k-mooney | there is an edgecase that it breaks | 15:15 |
| noonedeadpunk | even if its bfv | 15:15 |
| gmann | https://github.com/openstack/nova/blob/eef4b5435e7cdfe53ee9d9265d96c7dd278d9e93/nova/compute/api.py#L730 | 15:17 |
| sean-k-mooney | ya htat is blocking using 0 disk flavor without bfv | 15:17 |
| noonedeadpunk | I think it's pretty different | 15:17 |
| noonedeadpunk | yeah | 15:18 |
| sean-k-mooney | it does not prevent createing vms with non 0 disk flavors | 15:18 |
| noonedeadpunk | eventually... it blocks 0 disk flavor with bfv as well. At least blocked on train for me... | 15:18 |
| sean-k-mooney | it should not | 15:18 |
| gmann | yeah, it should not block that | 15:19 |
| sean-k-mooney | that check should only block non bfv vms with 0 disk flavors | 15:19 |
| noonedeadpunk | oh, yeah, maybe you're right | 15:19 |
| noonedeadpunk | sorry | 15:19 |
| *** dklyle has quit IRC | 15:19 | |
| noonedeadpunk | actually, even when instance has 0 disk flavor it can be still created without bfv | 15:20 |
| sean-k-mooney | not with that policy | 15:20 |
| noonedeadpunk | which also can't be covered with that policy | 15:20 |
| sean-k-mooney | that is what it block | 15:20 |
| sean-k-mooney | but on train it might now be enabeld by default | 15:20 |
| noonedeadpunk | oh, indeed, yeah, you're right | 15:20 |
| sean-k-mooney | its relitivly new | 15:21 |
| noonedeadpunk | so yeah, edgecase when flavor >0 | 15:21 |
| noonedeadpunk | I just got it overriden after upgrade to train, so mixing a bit, sorry | 15:21 |
| gmann | it is there before train and with admin only. before stein it was default with admin-or-owner | 15:21 |
| noonedeadpunk | so like another option is to drop all non-0 flavors and use that policy | 15:22 |
| gmann | ok so the case left here is to allow flavor.disk>0 in bfv case ? | 15:22 |
| sean-k-mooney | noonedeadpunk: yes | 15:23 |
| sean-k-mooney | noonedeadpunk: and create private flavor with disk if you need that and only give access to sepcific tenants | 15:23 |
| gmann | or instead of drop, control the flavor access | 15:23 |
| sean-k-mooney | the flavor access i tenatnt based rather then role based is the only wrinkel with that | 15:24 |
| sean-k-mooney | but it would work | 15:24 |
| noonedeadpunk | ok, thank you very much for your time. You gave me 2 options which is more than enough at this time for me:) | 15:25 |
| *** sapd1_x has quit IRC | 15:25 | |
| *** gyee has joined #openstack-nova | 15:26 | |
| *** dklyle has joined #openstack-nova | 15:26 | |
| noonedeadpunk | I just really thought that I had issues with creating even with volumes and 0 disk flavors when had os_compute_api:servers:create:zero_disk_flavor locked for admin | 15:27 |
| sean-k-mooney | if you can repoduce it simply then file a bug if you hit it | 15:27 |
| sean-k-mooney | as that should work | 15:27 |
| noonedeadpunk | yeah, sure! thanks again! | 15:30 |
| *** belmoreira has quit IRC | 15:36 | |
| rmart04 | thanks @sean-k-mooney, graphing this in grafana now, still unsure what the best approach is here though to stop OOMs. Was StephenFin saying that huge pages may help me over come this? | 15:53 |
| openstackgerrit | Vlad Gusev proposed openstack/nova stable/ussuri: replace the "hide_hypervisor_id" to "hw:hide_hypervisor_id" https://review.opendev.org/747189 | 15:58 |
| *** xek has quit IRC | 16:01 | |
| bauzas | stephenfin: gave you a terrible -1 https://review.opendev.org/#/c/747744/2 | 16:05 |
| * stephenfin shakes fist and clicks | 16:05 | |
| * bauzas hides | 16:05 | |
| stephenfin | rmart04: I was suggesting explicit page size allocations would help | 16:05 |
| bauzas | stephenfin: I know I'm a mean person | 16:06 |
| stephenfin | because then you can do per-node reserved memory amounts | 16:06 |
| stephenfin | e.g. 'hw:mem_page_size=small' | 16:06 |
| *** dklyle has quit IRC | 16:06 | |
| *** dklyle has joined #openstack-nova | 16:07 | |
| sean-k-mooney | that will force a singel numa node unless you also use hw:numa_nodes=X | 16:07 |
| sean-k-mooney | rmart04: do you allow ram oversubscription | 16:08 |
| sean-k-mooney | rmart04: and are you using train or newer? | 16:08 |
| rmart04 | We're using Stein and no RAM oversubscription | 16:09 |
| sean-k-mooney | you are using pinning yes | 16:09 |
| rmart04 | Yes | 16:09 |
| sean-k-mooney | ok so in that case you should use hugepages | 16:09 |
| sean-k-mooney | using explcit small pages could help but there hugepages will improve performance | 16:09 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Prevent starting services with older than N-1 computes https://review.opendev.org/738482 | 16:09 |
| sean-k-mooney | you are already using pinning so you cannot live migrete | 16:10 |
| sean-k-mooney | adding hugepages wont make that worse in this case | 16:10 |
| rmart04 | ok JohnGarbut also suggested this to me recently, i'll look into it and see how i get on. Does the host need any explicit configuration or is it just flavor metadata specs? | 16:10 |
| *** artom has quit IRC | 16:10 | |
| sean-k-mooney | rmart04: the host needs to have memroy preaccloated as hugepages | 16:10 |
| sean-k-mooney | you can do tha that on the kernel commandline | 16:11 |
| sean-k-mooney | you have 768 GB and are running a 750GB guest right | 16:11 |
| sean-k-mooney | so you would allocate 750GB of 1GB hugepages ideally | 16:12 |
| sean-k-mooney | then set hw:mem_page_size=large in the flavor | 16:12 |
| sean-k-mooney | default_hugepagesz=1G hugepagesz=1G hugepages=750 | 16:12 |
| sean-k-mooney | that is what you need to add to the kernel command line in grub | 16:13 |
| rmart04 | ah OK amazing, thank you | 16:13 |
| sean-k-mooney | you have only 1 numa node on the host currently right? | 16:13 |
| rmart04 | No two | 16:13 |
| sean-k-mooney | and the vm has how many numa nodes? 2 | 16:14 |
| rmart04 | yes | 16:14 |
| sean-k-mooney | ok then that makes sense | 16:14 |
| sean-k-mooney | otherwise i was wondering how the vm booted :) | 16:14 |
| rmart04 | I managed to get over that hurdle :P | 16:15 |
| sean-k-mooney | rmart04: you should see >20% incresse form hugepages in most workloads | 16:15 |
| *** mvorwerk_ has quit IRC | 16:15 | |
| sean-k-mooney | more if its memory or io bound | 16:15 |
| sean-k-mooney | less if it compute bound | 16:15 |
| sean-k-mooney | or disk bound | 16:15 |
| rmart04 | ok great, thanks for all your help | 16:17 |
| *** tesseract has quit IRC | 16:18 | |
| *** mvorwerk has joined #openstack-nova | 16:18 | |
| sean-k-mooney | by the way the reason i ask about oversubsrcitip is hugepage cant be swapped out ot disk but they also will never be reap as a result of OOM | 16:18 |
| sean-k-mooney | so it shoudl solve your issue although there is a very small change that the qemu process can still be selected | 16:18 |
| *** dosaboy has quit IRC | 16:19 | |
| *** links has quit IRC | 16:19 | |
| *** dosaboy has joined #openstack-nova | 16:19 | |
| sean-k-mooney | qemu will be using relitvly little non hugepage memeory in this config so it will have a low memery pressur score vs other process and memeory will be balance evenly across both host numa nodes for the vm | 16:20 |
| * bauzas goes off for the day | 16:20 | |
| yoctozepto | morning nova folks - today's question is when nova compute host name can get different from nova hypervisor host name and how | 16:20 |
| * yoctozepto too busy to read the code... | 16:21 | |
| *** psachin has quit IRC | 16:21 | |
| sean-k-mooney | yoctozepto: it will be different for ironic | 16:21 |
| yoctozepto | sean-k-mooney: true, let's focus on nova+kvm :-) | 16:22 |
| sean-k-mooney | out side of that it shoudl be the same unless you set teh HOST config element or you add a fqdn to you /etc/hostname | 16:22 |
| yoctozepto | actually evaluating noonedeadpunk's https://review.opendev.org/728629 (for masakari) | 16:22 |
| * noonedeadpunk patching | 16:22 | |
| stephenfin | yoctozepto: You can manually override '[DEFAULT] host' | 16:23 |
| sean-k-mooney | yoctozepto: this is how its set https://github.com/openstack/nova/blob/master/nova/conf/netconf.py#L50-L70 | 16:23 |
| stephenfin | it defaults to socket.gethostname() | 16:23 |
| stephenfin | what sean-k-mooney just linked to | 16:23 |
| sean-k-mooney | yep | 16:23 |
| sean-k-mooney | so they should be the same unless you change that config option | 16:23 |
| sean-k-mooney | if you use socket.fqdn() in your code it may not agreee | 16:24 |
| sean-k-mooney | i fixed this in cyborg | 16:24 |
| yoctozepto | then I would be changing the thing for nova-compute but not for the hypervisor, right? | 16:24 |
| sean-k-mooney | yes | 16:24 |
| sean-k-mooney | if you set [default]/host to something | 16:24 |
| yoctozepto | ok, so where does the name for the hypervisor spawn and how's it kept linked? :D | 16:25 |
| sean-k-mooney | hyperviour_hostname and host will not be the same for libvirt | 16:25 |
| sean-k-mooney | or other virt dirvers | 16:25 |
| sean-k-mooney | host is the compute service host and is set by that config and hypervisor_hostname is set by the virt driver | 16:25 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: functional: Don't inherit from 'ProviderUsageBaseTestCase' https://review.opendev.org/748271 | 16:26 |
| sean-k-mooney | the libvirt driver get it form libvirt | 16:26 |
| sean-k-mooney | and it calls socket.gethostname internally in c | 16:26 |
| stephenfin | bauzas: https://review.opendev.org/748271 | 16:26 |
| sean-k-mooney | socket.gethostname is a fucntion provided by libc and pythons socket module jsut call the libc version | 16:26 |
| sean-k-mooney | socket.gethostname() reads /etc/hostname if it exits on linux | 16:27 |
| yoctozepto | sean-k-mooney: ah, so it's consulted each time with libvirt? I guess that's why I had a hard time figuring this one out | 16:27 |
| sean-k-mooney | yes | 16:27 |
| yoctozepto | yeah, the rest makes sense | 16:27 |
| stephenfin | sean-k-mooney: https://review.opendev.org/#/c/748250/ | 16:27 |
| yoctozepto | I do wonder how noonedeadpunk (and some others) ended up desynchronizing them | 16:27 |
| bauzas | stephenfin: I don't get your comment, you're saying you can't just use the InstanceMixin ? | 16:28 |
| noonedeadpunk | I have situnation where socket.gethostname and socket.getfqdn are different things | 16:28 |
| bauzas | stephenfin: directly I mean ? | 16:28 |
| yoctozepto | noonedeadpunk: hmm, ah | 16:28 |
| yoctozepto | but then sean-k-mooney was only about gethostname | 16:28 |
| yoctozepto | you mean libvirt is actually doing getfqdn? | 16:28 |
| bauzas | stephenfin: of course, you'd have to specify the latest microversion but is that terrible to do ? | 16:28 |
| stephenfin | bauzas: Not really. I'd have to copy-paste all of this https://github.com/openstack/nova/blob/master/nova/tests/functional/integrated_helpers.py#L1107-L1132 | 16:29 |
| sean-k-mooney | yoctozepto: no its getting the hostname | 16:29 |
| stephenfin | bauzas: Which seems worse than having functions available that I'm not using | 16:29 |
| noonedeadpunk | sean-k-mooney: but I think that libvirt checks for the socket.getfqdn... | 16:29 |
| sean-k-mooney | stephenfin: lol ok | 16:29 |
| stephenfin | '_IntegratedTestBase' is basically what you're asking for but with all that boilerplate inline | 16:29 |
| yoctozepto | then there is a conflict between your testimonies, noonedeadpunk and sean-k-mooney :-) | 16:29 |
| sean-k-mooney | noonedeadpunk: it does not | 16:30 |
| yoctozepto | investigation ongoing! | 16:30 |
| yoctozepto | :D | 16:30 |
| sean-k-mooney | noonedeadpunk: unless they changed it wich would be a breaking change | 16:30 |
| stephenfin | there are some differences around policy but I'm resolving those in separate patches | 16:30 |
| noonedeadpunk | sean-k-mooney: http://paste.openstack.org/show/797181/ | 16:30 |
| yoctozepto | https://github.com/libvirt/libvirt/blob/7ba4838a18afb609d900190c57b0cbfb842ccbdf/src/util/virutil.c#L470 | 16:31 |
| noonedeadpunk | I can judge only by this... | 16:31 |
| stephenfin | melwitt: want to unbork cloning on Windows? https://review.opendev.org/#/c/748250/ | 16:31 |
| yoctozepto | it seems they are training to canonicalize on master | 16:31 |
| yoctozepto | that's how they ended up with an fqdn | 16:31 |
| bauzas | stephenfin: technically, you'd need to do things like https://github.com/openstack/nova/blob/master/nova/tests/functional/regressions/test_bug_1849409.py | 16:31 |
| sean-k-mooney | yoctozepto: noonedeadpunk we cant change this behavior | 16:32 |
| yoctozepto | so yeah, they b0rked | 16:32 |
| sean-k-mooney | its used in plamcnet | 16:32 |
| bauzas | stephenfin: but that should be it | 16:32 |
| yoctozepto | sean-k-mooney: what's plamcnet? ;d | 16:32 |
| noonedeadpunk | sean-k-mooney: yeah, that's fine) just sharing feedback about fqdn vs gethostname - fighting this 3 years I guess... | 16:32 |
| sean-k-mooney | the placement service | 16:32 |
| yoctozepto | noonedeadpunk: but now we know | 16:32 |
| yoctozepto | sean-k-mooney: ack | 16:32 |
| elod | lyarwood: no need to squash, it's good as it is. will review now | 16:33 |
| sean-k-mooney | noonedeadpunk: it is technially allowed to return an fqdn if you put an fqdn in /etc/hostname which you should not do but some people do | 16:33 |
| stephenfin | bauzas: Yes, but those are nearly identical. The only difference is the use of the CinderFixture and fake image service (essentially GlanceFixture) | 16:33 |
| yoctozepto | ok, this was a very worthwhile chat, thanks sean-k-mooney, stephenfin and noonedeadpunk | 16:33 |
| yoctozepto | sean-k-mooney: you can also get trapped by dns | 16:33 |
| stephenfin | bauzas: oh, and the cast as call | 16:33 |
| sean-k-mooney | the masikari patch is not correct by the way | 16:34 |
| melwitt | stephenfin: sure, but curious what does your comment mean about sphinx not finding the reno? does that mean the release note will disappear from the release notes? I had thought it would still be included even if the name is changed? | 16:34 |
| yoctozepto | sean-k-mooney: any reviews appreciated! | 16:34 |
| yoctozepto | any and all* I mean | 16:34 |
| stephenfin | melwitt: it'll find the reno with the new name but complain about a file matching the old name being missing | 16:34 |
| bauzas | stephenfin: yup, we would probably just need another base testclass which wouldn't pull all the placementesque methods | 16:34 |
| sean-k-mooney | yoctozepto: the seach is using hyperviour_hostname in the host filed | 16:34 |
| noonedeadpunk | sean-k-mooney: it's more complicated than that.. In /etc/hostname I have uacloud-nova03. But the thing is that python gets it not from /etc/hostname, but from /etc/hosts | 16:34 |
| stephenfin | i.e. "this existed in the past but I can't find it now; skipping" | 16:34 |
| sean-k-mooney | that only works for libvirt | 16:35 |
| melwitt | stephenfin: ah ok. thanks | 16:35 |
| bauzas | stephenfin: but in the meantime, instancemixin seems better to use IMHO + those extra lines | 16:35 |
| yoctozepto | sean-k-mooney: nah, that's bad naming | 16:35 |
| stephenfin | bauzas: We have it. It's call '_IntegratedTestBase' | 16:35 |
| yoctozepto | actually masakari uses this-called-string for corosync and nova-compute | 16:35 |
| yoctozepto | and asssumes that they are the same | 16:35 |
| stephenfin | bauzas: look at the superclasses for that https://github.com/openstack/nova/blob/master/nova/tests/functional/integrated_helpers.py#L1008 | 16:35 |
| yoctozepto | somehow they thought it always equals hypervisors... | 16:35 |
| yoctozepto | well, I would have as well probably | 16:35 |
| stephenfin | *It's called | 16:36 |
| bauzas | stephenfin: I do remember the pain writing functional tests that were requiring duplicate code | 16:36 |
| sean-k-mooney | yoctozepto: maybe its not clear if https://docs.openstack.org/api-ref/compute/?expanded=list-compute-services-detail#list-compute-services is the hyperviour_hostname or the host value form teh cofnig | 16:37 |
| sean-k-mooney | i suppect its the host value form the config | 16:37 |
| yoctozepto | sean-k-mooney: it is, it's just that noonedeadpunk kept the old name, I commented on that too | 16:38 |
| sean-k-mooney | ok so its the host value | 16:38 |
| sean-k-mooney | not the hypervisor_hostname | 16:38 |
| noonedeadpunk | At fisrt I was trying to keep changes minimal) | 16:38 |
| *** mkrai has quit IRC | 16:38 | |
| yoctozepto | noonedeadpunk: all in all, this patch makes masakari a bit better but still kinda delays the issue that one could actually want to monitor the hypervisors, and have their names agree with those in corosync cluster | 16:38 |
| bauzas | either way, it's becoming late for me, \o | 16:39 |
| sean-k-mooney | noonedeadpunk: thats an good goal generaly but but names existingon the compute node recored in teh db | 16:39 |
| sean-k-mooney | so we should try to use hypervisor_hostname and host name differently depending on which we mean | 16:39 |
| noonedeadpunk | Yeah, I see it now. Just wanted to made it as bug fix to get backported at first :p | 16:40 |
| noonedeadpunk | but you're totally right | 16:40 |
| *** k_mouza has quit IRC | 16:41 | |
| yoctozepto | noonedeadpunk: if we straighten the logic right, then you know who's the core who will accept those backports :-) | 16:41 |
| sean-k-mooney | noonedeadpunk: these are teh rules libvirt uses by the way | 16:42 |
| sean-k-mooney | https://github.com/libvirt/libvirt/blob/7ba4838a18afb609d900190c57b0cbfb842ccbdf/src/util/virutil.c#L470-L489 | 16:42 |
| *** k_mouza has joined #openstack-nova | 16:44 | |
| sean-k-mooney | the did modify this 8 months ago https://github.com/libvirt/libvirt/commit/26d9748ff114a060ee751959d108d062f737f5d9 | 16:45 |
| noonedeadpunk | sean-k-mooney: I think that C way of gethostname may differ from python implementation.... | 16:46 |
| noonedeadpunk | And I'm not C guy unfortunatelly... | 16:46 |
| sean-k-mooney | they did not in the past but now they are calling a new fucntion | 16:46 |
| sean-k-mooney | the python way called the c function | 16:47 |
| yoctozepto | sean-k-mooney: I linked to that libvirt code just above :-) | 16:47 |
| sean-k-mooney | libvirt is now using g_get_host_name(); | 16:47 |
| sean-k-mooney | instead of gethostname | 16:47 |
| *** k_mouza has quit IRC | 16:48 | |
| *** lpetrut has joined #openstack-nova | 16:49 | |
| *** lpetrut has quit IRC | 16:49 | |
| noonedeadpunk | Yeah, so things may broke for me with some libvirt update... Anyway that would mean time to work on naming convention:) Like I have only 1 such region - the first one I've built... | 16:49 |
| noonedeadpunk | but it created corner case.... | 16:49 |
| *** dklyle has quit IRC | 16:50 | |
| *** ralonsoh has quit IRC | 16:51 | |
| sean-k-mooney | 6.0 is the default in ubuntu and centos 8 | 16:52 |
| sean-k-mooney | i know ooo hardcodes the fqdn in /etc/hostname | 16:52 |
| yoctozepto | noonedeadpunk: masakari needs fixing nonetheless ;p | 16:52 |
| sean-k-mooney | and i think it also manually sets the [default]/host value | 16:52 |
| sean-k-mooney | nova depends on teh actull hostname not changing | 16:53 |
| *** mkrai has joined #openstack-nova | 16:53 | |
| *** mkrai has quit IRC | 16:55 | |
| *** artom has joined #openstack-nova | 16:56 | |
| *** dtantsur is now known as dtantsur|afk | 16:57 | |
| *** mvorwerk has quit IRC | 16:58 | |
| sean-k-mooney | yoctozepto: noonedeadpunk this is where we call libvirt if you are interested https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L976 | 16:59 |
| sean-k-mooney | and that is used here https://github.com/openstack/nova/blob/a7735d5e379c20c59cfb639f9f4d339bcffca2f9/nova/virt/libvirt/driver.py#L8418 | 17:00 |
| *** mvorwerk has joined #openstack-nova | 17:00 | |
| yoctozepto | sean-k-mooney: thanks, I just did not realize it's libvirt telling us its name, but thanks to you it's clear for me now | 17:00 |
| sean-k-mooney | ya genericly its the hypervior e.g vmware or hyperv or libvirt that provides this | 17:02 |
| *** mkrai has joined #openstack-nova | 17:02 | |
| openstackgerrit | Merged openstack/nova stable/train: libvirt: Provide VIR_MIGRATE_PARAM_PERSIST_XML during live migration https://review.opendev.org/747973 | 17:02 |
| openstackgerrit | Merged openstack/nova master: releasenotes: Detail support for server ops with vTPM https://review.opendev.org/748215 | 17:03 |
| noonedeadpunk | sean-k-mooney: just one more stupid question... trying to find host-evacuate api call in https://docs.openstack.org/api-ref/compute/ but don't see for some reason (only server evacuate) | 17:04 |
| *** sapd1_x has joined #openstack-nova | 17:04 | |
| yoctozepto | sean-k-mooney: https://docs.openstack.org/api-ref/compute/?expanded=evacuate-server-evacuate-action-detail#evacuate-server-evacuate-action ? | 17:05 |
| yoctozepto | noonedeadpunk: ^ | 17:05 |
| yoctozepto | it's server (instance/vm) that's getting evacuated | 17:05 |
| noonedeadpunk | and how does https://docs.openstack.org/nova/rocky/admin/evacuate.html#evacuate-all-instances work? It gets list of instances and evacuate one by one ? | 17:06 |
| *** mkrai has quit IRC | 17:07 | |
| *** derekh has quit IRC | 17:08 | |
| yoctozepto | noonedeadpunk: that's what I would assume, could use checking the code | 17:09 |
| sean-k-mooney | noonedeadpunk: host-evacuate is not an api action | 17:09 |
| sean-k-mooney | noonedeadpunk: its a client command | 17:09 |
| noonedeadpunk | ok, got it | 17:09 |
| noonedeadpunk | yeah | 17:10 |
| sean-k-mooney | noonedeadpunk: http://www.danplanet.com/blog/2016/03/03/evacuate-in-nova-one-command-to-confuse-us-all/ | 17:10 |
| sean-k-mooney | dansmith's blog that is basically mandatory reading on host-evacuate and how it related to every thing else | 17:11 |
| noonedeadpunk | oh, ok, now I know under what conditions our client loose all of their data from ephemeral drives... | 17:11 |
| sean-k-mooney | yoctozepto: host-evacuate does not call teh evacuate api | 17:11 |
| noonedeadpunk | when masakari calls instance evacuate | 17:11 |
| sean-k-mooney | yoctozepto: it does cold migrate | 17:11 |
| sean-k-mooney | actully no it does evacuate | 17:12 |
| sean-k-mooney | i should read the blog more often | 17:12 |
| noonedeadpunk | "The core of the evacuate process in nova is actually rebuild, which in many cases is a destructive operation" | 17:13 |
| noonedeadpunk | so it's not evacuate and you;re right | 17:13 |
| sean-k-mooney | from the blog | 17:13 |
| sean-k-mooney | "The nova host-evacuate command does not translate directly to a server-side operation, but is more of a client-side macro or “meta operation.” When you call this command, you provide a hypervisor hostname, which the client uses to list and trigger evacuate operations on each instance running on that hypervisor. You would use this command post-failure (just like the single-instance evacuate | 17:13 |
| sean-k-mooney | command) to trigger evacuations of all the instances on a failed compute host." | 17:13 |
| sean-k-mooney | noonedeadpunk: no it is evacuate | 17:14 |
| sean-k-mooney | noonedeadpunk: but evacuate does not mean what you think it does | 17:14 |
| noonedeadpunk | yeah | 17:14 |
| noonedeadpunk | so when masakari calls instance evacuate when this instance is not volume based - it get's wiped out | 17:15 |
| sean-k-mooney | evacuate only preseved data if you use bfv or are on shared storage | 17:15 |
| sean-k-mooney | noonedeadpunk: unless nova is using ceph via the rbd imageages type or /var/lib/nova is on nfs | 17:15 |
| noonedeadpunk | and I guess even with rbd drive rebuild is destructove for non-bfv? | 17:15 |
| noonedeadpunk | hm... | 17:16 |
| sean-k-mooney | noonedeadpunk: no its not with ceph we detech its on shared storage | 17:16 |
| noonedeadpunk | hm..... | 17:16 |
| * dansmith is enjoying the heck out of this | 17:16 | |
| sean-k-mooney | dansmith: its proably in your blog post | 17:17 |
| noonedeadpunk | then I still don't get why I got situations when ephemeral got lost or wiped out during node crush.... | 17:17 |
| noonedeadpunk | but whatever) | 17:17 |
| noonedeadpunk | I know how to disable this :p | 17:17 |
| sean-k-mooney | dansmith: i keep it bookmarked as a reference document | 17:18 |
| *** bnemec has quit IRC | 17:18 | |
| dansmith | yeah, I should charge admission | 17:18 |
| sean-k-mooney | noonedeadpunk: ootnote: In the case of volume-backed instances, the root disk of the instance is usually in a common location such as on a SAN device. In this case, the root disk is not destroyed, but any other instance state is recreated (which includes memory, ephemeral disk, swap disk, etc). | 17:19 |
| noonedeadpunk | yeah, so it's re-created from the image? | 17:19 |
| noonedeadpunk | as rebuild do | 17:19 |
| noonedeadpunk | which is destructive and all changes made on vm are gone which is eventually as intended? | 17:20 |
| sean-k-mooney | the root disk shoudl not be but if you have extra ephermeral disk they proably are i dont know how addtional ephemeral disks work with ceph. i now swap is store as a ceph volume | 17:20 |
| yoctozepto | sean-k-mooney, noonedeadpunk: I guess masakari should include a warning in big, red font about the necessity to have HA storage first before running masakari | 17:20 |
| noonedeadpunk | I guess so... | 17:21 |
| yoctozepto | it's obvious to us (well, me at least) but years of practice has proven it's not entirely that obvious to all of users | 17:21 |
| sean-k-mooney | yoctozepto: if masikari automate evacuate yes | 17:21 |
| *** ganso has quit IRC | 17:21 | |
| sean-k-mooney | or keep all your data on a cinder data volume | 17:21 |
| yoctozepto | yeah, that's what I meant | 17:21 |
| noonedeadpunk | I mean I have ceph for everything.... But still non bfv instances get's wiped out during evacuations... | 17:21 |
| noonedeadpunk | ok, whatever) | 17:21 |
| sean-k-mooney | noonedeadpunk: ill check the ceph code quickly but i tought we dedect ceph and preserved it. | 17:22 |
| *** k_mouza has joined #openstack-nova | 17:22 | |
| noonedeadpunk | maybe it's some recent change? Ie T or U? | 17:22 |
| sean-k-mooney | we pass an on_shared_storage flag internllay which should be set to true for ceph and nfs | 17:23 |
| sean-k-mooney | thats set here https://github.com/openstack/nova/blob/20459e3e88cb8382d450c7fdb042e2016d5560c5/nova/api/openstack/compute/evacuate.py#L97 | 17:24 |
| sean-k-mooney | oh you have to pass it as an option | 17:24 |
| noonedeadpunk | sean-k-mooney: yoctozepto seems like another patch to masakari?:) | 17:25 |
| sean-k-mooney | noonedeadpunk: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id12 | 17:25 |
| sean-k-mooney | so before 2.14 you had to pas it | 17:25 |
| sean-k-mooney | after that its automatic | 17:25 |
| noonedeadpunk | 2.12 was soooooo long ago.... | 17:26 |
| *** k_mouza has quit IRC | 17:26 | |
| *** ganso has joined #openstack-nova | 17:27 | |
| sean-k-mooney | is that what massikari is using | 17:27 |
| sean-k-mooney | https://github.com/openstack/nova/blob/20459e3e88cb8382d450c7fdb042e2016d5560c5/nova/compute/manager.py#L3504 we now ask the dirver if the insance is on shared storage | 17:27 |
| noonedeadpunk | hm. I think I'm reading it wrong.. But it returns None for 2.14+ ? https://github.com/openstack/nova/blob/20459e3e88cb8382d450c7fdb042e2016d5560c5/nova/api/openstack/compute/evacuate.py#L45 | 17:28 |
| sean-k-mooney | form 2.14 on | 17:28 |
| sean-k-mooney | noonedeadpunk: yes 2.14 on it returns none and we ask the driver | 17:28 |
| noonedeadpunk | ah | 17:28 |
| sean-k-mooney | which does self.image_backend.backend(). | 17:29 |
| sean-k-mooney | is_shared_block_storage() | 17:29 |
| sean-k-mooney | so unless you have differen image_backend on different hosts that should be correct | 17:29 |
| sean-k-mooney | rbd returns true https://github.com/openstack/nova/blob/20459e3e88cb8382d450c7fdb042e2016d5560c5/nova/virt/libvirt/imagebackend.py#L958 | 17:30 |
| sean-k-mooney | so you should see this log message when you evacuate https://github.com/openstack/nova/blob/20459e3e88cb8382d450c7fdb042e2016d5560c5/nova/compute/manager.py#L3514-L3516 | 17:30 |
| sean-k-mooney | and we should keep the disk content | 17:30 |
| noonedeadpunk | looks like this, yes | 17:31 |
| * noonedeadpunk goes to find node to drop | 17:31 | |
| sean-k-mooney | what api microversion is masikari using | 17:31 |
| noonedeadpunk | 2.53 for master... | 17:33 |
| sean-k-mooney | it looks like it does not set one https://github.com/openstack/masakari/blob/cd95f6660c14f506603f0864ca24dd25c278a6a8/masakari/engine/drivers/taskflow/host_failure.py#L262-L263 | 17:33 |
| sean-k-mooney | so nova client will default to the latest | 17:34 |
| noonedeadpunk | ok, on rocky it was 2.14 | 17:34 |
| * noonedeadpunk upgraded from rocky month ago | 17:34 | |
| *** rmart04 has quit IRC | 17:34 | |
| yoctozepto | we care about stein+ atm | 17:35 |
| noonedeadpunk | ++ | 17:35 |
| sean-k-mooney | 2.14 was mitaka | 17:35 |
| sean-k-mooney | so you should be good | 17:35 |
| sean-k-mooney | we have been preseviing shared storage vms for a very long time | 17:36 |
| *** mvorwerk has quit IRC | 17:36 | |
| *** mvorwerk has joined #openstack-nova | 17:38 | |
| *** mvorwerk has quit IRC | 17:45 | |
| *** artom has quit IRC | 17:45 | |
| *** mvorwerk has joined #openstack-nova | 17:46 | |
| *** sapd1_x has quit IRC | 17:47 | |
| *** k_mouza has joined #openstack-nova | 17:56 | |
| *** belmoreira has joined #openstack-nova | 17:57 | |
| noonedeadpunk | sean-k-mooney: just tested and during evacuate data is really preserved | 18:00 |
| sean-k-mooney | good that would have been a nasty bug if that was regressed | 18:00 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Change default num_retries for glance to 3 https://review.opendev.org/740389 | 18:00 |
| *** k_mouza has quit IRC | 18:01 | |
| *** mvorwerk has quit IRC | 18:02 | |
| noonedeadpunk | TIL | 18:02 |
| openstackgerrit | Merged openstack/nova stable/ussuri: compute: Validate a BDMs disk_bus when provided https://review.opendev.org/744550 | 18:02 |
| sean-k-mooney | noonedeadpunk: the api does not guarentee that data is preserved unless you are using bfv | 18:03 |
| sean-k-mooney | noonedeadpunk: so unless you deployed the cloud and know the vm is on shared storage when you dont use bfv dont rely on that behavior | 18:03 |
| *** mvorwerk has joined #openstack-nova | 18:04 | |
| noonedeadpunk | I see, yeah, I know that bfv is really preferable. But it's hard to explain in terms of public clouds, when even with no non-0 disk flavors ppl still do use ephemeral. But good part of them, is that they can handle ISO's while cinder wasn't able to boot ISO image properly last time I've checked... | 18:06 |
| openstackgerrit | Merged openstack/nova master: rbd: Move rbd_utils out of libvirt driver under nova.storage https://review.opendev.org/746904 | 18:06 |
| sean-k-mooney | noonedeadpunk: well i have no problem iwht non bfv instance | 18:06 |
| sean-k-mooney | i like the ceph backend for example | 18:07 |
| noonedeadpunk | yes, sure ceph backend is everywhere for me:) | 18:07 |
| sean-k-mooney | but its jsut driver devied if evauate preserves data or not | 18:07 |
| sean-k-mooney | for the libvirt dirver it does preseve data it the sorate is shared | 18:07 |
| sean-k-mooney | for other drivers that may not be the case | 18:07 |
| noonedeadpunk | dunno, I need better investigation on my side, it was just good lead, but no.. And I wasn't able to reproduce by far so yeah... | 18:08 |
| sean-k-mooney | ironic for example i would expect to always recreate the server altouhg if its boot form volue it might persve it | 18:08 |
| *** mvorwerk has quit IRC | 18:12 | |
| *** mvorwerk has joined #openstack-nova | 18:14 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Use absolute path during qemu img rebase https://review.opendev.org/739246 | 18:21 |
| *** brinzhang_ has joined #openstack-nova | 18:21 | |
| *** brinzhang has quit IRC | 18:25 | |
| *** belmoreira has quit IRC | 18:30 | |
| *** belmoreira has joined #openstack-nova | 18:32 | |
| sean-k-mooney | stephenfin: by the way for my libvirt/os-vif patch i have another pathc im addedin beneath it | 18:36 |
| sean-k-mooney | but ill adress your commend when i push both | 18:36 |
| *** mvorwerk has quit IRC | 18:36 | |
| stephenfin | sean-k-mooney: Okay. You've got reviews on https://review.opendev.org/#/c/745605/ too so if you get a chance to address that, I can re +2 tomorrow | 18:37 |
| * stephenfin had a Python Ireland meeting and is now going for food o/ | 18:37 | |
| sean-k-mooney | ya ill take a look at it | 18:38 |
| *** mvorwerk has joined #openstack-nova | 18:38 | |
| *** mvorwerk has quit IRC | 18:58 | |
| *** mvorwerk has joined #openstack-nova | 18:58 | |
| *** hamalq has joined #openstack-nova | 19:02 | |
| openstackgerrit | Jiri Suchomel proposed openstack/nova master: Add ability to download Glance images into the libvirt image cache via RBD https://review.opendev.org/574301 | 19:04 |
| *** zzzeek has quit IRC | 19:07 | |
| *** mvorwerk has quit IRC | 19:08 | |
| *** zzzeek has joined #openstack-nova | 19:09 | |
| *** mvorwerk has joined #openstack-nova | 19:10 | |
| *** belmoreira has quit IRC | 19:12 | |
| openstackgerrit | Merged openstack/nova master: Add type hints to 'nova.compute.manager' https://review.opendev.org/742863 | 19:22 |
| openstackgerrit | Merged openstack/nova master: Avoid invalid file name, preventing git clone on win32 https://review.opendev.org/748250 | 19:22 |
| *** mvorwerk has quit IRC | 19:24 | |
| *** zzzeek has quit IRC | 19:25 | |
| *** mvorwerk has joined #openstack-nova | 19:26 | |
| *** zzzeek has joined #openstack-nova | 19:28 | |
| *** zzzeek has quit IRC | 19:35 | |
| *** zzzeek has joined #openstack-nova | 19:36 | |
| *** tosky has quit IRC | 19:38 | |
| *** jsuchome has quit IRC | 19:46 | |
| *** mvorwerk_ has joined #openstack-nova | 19:47 | |
| *** mvorwerk has quit IRC | 19:48 | |
| *** artom has joined #openstack-nova | 19:55 | |
| *** k_mouza has joined #openstack-nova | 19:57 | |
| openstackgerrit | sean mooney proposed openstack/nova master: libvirt: delegate ovs plug to os-vif https://review.opendev.org/602432 | 20:00 |
| openstackgerrit | sean mooney proposed openstack/nova master: pass vifs_already_plugged when reverting a migration https://review.opendev.org/748296 | 20:00 |
| mnaser | does anyone know why regex based filtering is no longer working in the api with ussuri? | 20:00 |
| mnaser | i cant track any commits that might have caused it | 20:01 |
| *** nweinber has quit IRC | 20:01 | |
| mnaser | openstack --debug server list --all-projects --name 'foo' returns two servers which are 'foo1.bar' and 'foo2.bar' | 20:01 |
| mnaser | but `openstack --debug server list --all-projects --name 'foo\d'` returns nothing | 20:01 |
| *** k_mouza has quit IRC | 20:01 | |
| sean-k-mooney | did you check the client | 20:02 |
| sean-k-mooney | im not sure if this is api side | 20:02 |
| sean-k-mooney | it might be | 20:02 |
| sean-k-mooney | well i think it is but maybe there were some clinet changes that broke it | 20:02 |
| mnaser | sean-k-mooney: its not, i looked into the api-ref | 20:03 |
| mnaser | it says you can send actual regex | 20:03 |
| sean-k-mooney | string | 20:03 |
| sean-k-mooney | 20:03 | |
| sean-k-mooney | Filters the response by a server name, as a string. You can use regular expressions in the query. For example, the ?name=bob regular expression returns both bob and bobb. If you must match on only bob, you can use a regular expression that matches the syntax of the underlying database server that is implemented for Compute, such as MySQL or PostgreSQL. | 20:03 |
| mnaser | _regex_instance_filter() in https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py hasn't been touched for 5-8 years | 20:03 |
| mnaser | according to blame | 20:04 |
| sean-k-mooney | so if this behavior changed its likel a db change | 20:04 |
| mnaser | i mean, it went from myslq to mysql | 20:04 |
| mnaser | let me check if maybe its an option ;\ | 20:04 |
| sean-k-mooney | this was the last change to that code https://github.com/openstack/nova/commit/117fad897d5310d66cc2e690f3cd32e72614d8fd | 20:06 |
| mnaser | so it looks like it should be sending a REGEXP query | 20:07 |
| mnaser | let me try it out against the actual db server | 20:08 |
| mnaser | indeed, the regexp doesnt work | 20:09 |
| mnaser | `SELECT * FROM instances WHERE display_name REGEXP 'foo\d';` returns an empty set | 20:09 |
| sean-k-mooney | try 'foo\d.*' | 20:11 |
| sean-k-mooney | foo\d would not match foo1.bar | 20:11 |
| sean-k-mooney | https://regex101.com/r/uDuQBs/1/ | 20:14 |
| mnaser | sean-k-mooney: ok so it turns out that mysql seemingly uses a differnt regex lib than mariadb | 20:14 |
| mnaser | https://dev.mysql.com/doc/refman/5.7/en/regexp.html#regexp-operators | 20:14 |
| sean-k-mooney | your current regex is not correct however | 20:15 |
| sean-k-mooney | foo\d will not match foo1.bar | 20:15 |
| sean-k-mooney | it will match foo1 | 20:15 |
| sean-k-mooney | foo\d is the same as foo[0-9] | 20:16 |
| mnaser | sean-k-mooney: right, i agree with you on that, but the regex impl using in mysql does not | 20:16 |
| mnaser | SELECT display_name FROM instances WHERE display_name REGEXP 'foo[[:digit:]].'; works | 20:16 |
| sean-k-mooney | it looks like they want you to double escape | 20:17 |
| sean-k-mooney | so you could try foo\\d | 20:17 |
| mnaser | sean-k-mooney: nope | 20:18 |
| mnaser | see doc above, [[:digit:]] is what they use instead of \d according to that | 20:19 |
| sean-k-mooney | ok will this is not part of the nova api defintion | 20:19 |
| mnaser | i guess mariadb and mysql are different when it comes to this | 20:19 |
| sean-k-mooney | maybe maybe not | 20:19 |
| sean-k-mooney | \d works in pyton | 20:19 |
| sean-k-mooney | but it might not work in sql | 20:19 |
| mnaser | yeah well the fact we're relying on the db to do the filtering makes it very unpredictable | 20:19 |
| mnaser | because the regex can be different dependign on the backend | 20:19 |
| sean-k-mooney | ya well this is not a portable part of teh api | 20:21 |
| sean-k-mooney | we dont actully give any guarenttes this will work | 20:21 |
| sean-k-mooney | i know we looked at removing it at one point | 20:21 |
| sean-k-mooney | it is called out in the api ref at least | 20:22 |
| sean-k-mooney | If you must match on only bob, you can use a regular expression that matches the syntax of the underlying database server that is implemented for Compute, such as MySQL or PostgreSQL. | 20:22 |
| *** bbowen has quit IRC | 20:22 | |
| *** bbowen has joined #openstack-nova | 20:22 | |
| sean-k-mooney | mnaser: for what its worth that regex support is from the nova v1 api | 20:27 |
| sean-k-mooney | it predates v2 or microverions | 20:27 |
| sean-k-mooney | i belive it even predates novas use of specs | 20:28 |
| sean-k-mooney | so the fact it works at al is somewhat surprising | 20:28 |
| melwitt | agreed, but apparently other people have known this works, someone fixed a thing related to it 3 years ago https://review.opendev.org/506760 | 20:31 |
| sean-k-mooney | oh im sure people still use it | 20:33 |
| sean-k-mooney | just its not portable and it proably not well tested | 20:34 |
| *** lbragstad has quit IRC | 20:35 | |
| *** lbragstad has joined #openstack-nova | 20:35 | |
| *** vishalmanchanda has quit IRC | 20:37 | |
| melwitt | yeah | 20:37 |
| sean-k-mooney | apparently sqlalcamey support regex filters | 20:37 |
| sean-k-mooney | http://xion.io/post/code/sqlalchemy-regex-filters.html | 20:37 |
| sean-k-mooney | nick_regexp = '^' + re.escape(nick) + r'\d+$' | 20:37 |
| sean-k-mooney | return session.query(Person).filter(Person.nick.regexp(nick_regexp)).all() | 20:37 |
| sean-k-mooney | melwitt: mnaser so if we wanted to make it portable in the futrue we could use those to do so | 20:38 |
| sean-k-mooney | that would standardise on python regex syntax | 20:38 |
| mnaser | oooh | 20:38 |
| mnaser | yes i like that approach | 20:38 |
| melwitt | project idea for mnaser xD | 20:38 |
| sean-k-mooney | with a microversion since those are a thing now | 20:38 |
| sean-k-mooney | so you can got old and busted or new an shiny | 20:39 |
| mnaser | melwitt, sean-k-mooney: doesn't sound too harsh, probably once we're done upgrading everything to ussuri and this k8s operator for openstack project | 20:40 |
| *** bnemec has joined #openstack-nova | 20:40 | |
| sean-k-mooney | are you collaberating with redhat on the k8s operator | 20:40 |
| mnaser | i dont think redhat is building a k8s operator for openstack, we're building https://opendev.org/vexxhost/openstack-operator | 20:41 |
| mnaser | all work is in gerrit / testing using tempest so nothing exotic in that sense | 20:41 |
| sean-k-mooney | https://github.com/openstack-k8s-operators | 20:41 |
| sean-k-mooney | mnaser: redhat is | 20:41 |
| mnaser | lol, welp, TIL | 20:41 |
| mnaser | this is very new | 20:42 |
| sean-k-mooney | mnaser: and we will be supporting it in the next major verions | 20:42 |
| mnaser | gosh, this is exactly what we're doing | 20:42 |
| mnaser | lol | 20:42 |
| sean-k-mooney | altough proably as tech preview | 20:42 |
| mnaser | sean-k-mooney: who should i reach out to lol | 20:42 |
| sean-k-mooney | that is a good question i can follow up internally tommorow and ask | 20:43 |
| mnaser | sean-k-mooney: let me know because it pretty much already just works here... so we're really doing the same thing essentially :) | 20:44 |
| sean-k-mooney | mdbooth and mschuppert are involed with the nova part | 20:44 |
| mnaser | i.e. the keystone todo is already done in our case, heh | 20:44 |
| sean-k-mooney | do you have a fully contiarerised nova contol plane | 20:45 |
| sean-k-mooney | oh you are actully doing it in opendev | 20:46 |
| mnaser | sean-k-mooney: yes :) | 20:46 |
| sean-k-mooney | nice | 20:47 |
| mnaser | and comptues too, the only thing that runs on the physical nodes is openvswitch and libvirt | 20:47 |
| mnaser | the plan is to move openvswitch to containers too because we rely on the kernel data path so that should be ok | 20:47 |
| mnaser | and libvirt, planning to move towards the split daemon model and run only the kvm virt driver on the host (and i think even then that has a way of being containerized but not being killed) | 20:47 |
| sean-k-mooney | kolla uses ovs and libvirt in continers | 20:47 |
| mnaser | right but they have the advantage of managing a docker container and access to host | 20:48 |
| mnaser | i only can do things via k8s api | 20:48 |
| sean-k-mooney | right but you can run it with pid=host and as a deamonset | 20:48 |
| *** damien_r has quit IRC | 20:48 | |
| sean-k-mooney | that way the qemu instance are not tied to the lifetime of the libvirt pod | 20:48 |
| mnaser | well you could but there's like some tricky things to do with that in making sure the process doesnt disappear/etc/etc -- but yeah, not impossible! | 20:49 |
| sean-k-mooney | they are doint this in the nova operator i think | 20:49 |
| sean-k-mooney | they started with runnign the contoles in vms and only contianerisn the comptue nodes | 20:49 |
| sean-k-mooney | they are currently working on the contol plane | 20:50 |
| mnaser | yeah we just haven't gotten around that bit yet, but yeah, totally possible, we run nova-compute in containers too | 20:50 |
| sean-k-mooney | nova-compute is trivial | 20:50 |
| sean-k-mooney | libvirt is harder | 20:50 |
| mnaser | yep | 20:51 |
| mnaser | anyways, yeah, we should totally talk because we're probably doing the same thing.. | 20:51 |
| sean-k-mooney | yep | 20:51 |
| sean-k-mooney | and as i said this will be supported as a deployment model going forward | 20:51 |
| sean-k-mooney | in our downstream product | 20:52 |
| sean-k-mooney | it was anouched at the redhat summit a few months ago | 20:52 |
| mnaser | it's honestly the way to go :> | 20:52 |
| *** mvorwerk_ has quit IRC | 20:52 | |
| openstackgerrit | Merged openstack/nova master: Ensure source compute is up when confirming a resize https://review.opendev.org/699291 | 21:03 |
| *** slaweq has quit IRC | 21:18 | |
| *** k_mouza has joined #openstack-nova | 21:33 | |
| *** ganso has quit IRC | 21:40 | |
| *** bnemec has quit IRC | 21:42 | |
| *** ganso has joined #openstack-nova | 21:43 | |
| *** zzzeek has quit IRC | 21:45 | |
| *** zzzeek has joined #openstack-nova | 21:48 | |
| *** ganso has quit IRC | 21:52 | |
| *** bnemec has joined #openstack-nova | 21:52 | |
| *** ganso has joined #openstack-nova | 21:53 | |
| *** brinzhang0 has joined #openstack-nova | 22:00 | |
| *** raildo_ has joined #openstack-nova | 22:01 | |
| *** raildo has quit IRC | 22:04 | |
| *** brinzhang_ has quit IRC | 22:04 | |
| *** k_mouza has quit IRC | 22:06 | |
| *** bnemec has quit IRC | 22:09 | |
| *** ftarasenko has quit IRC | 22:17 | |
| *** bnemec has joined #openstack-nova | 22:21 | |
| *** rcernin has joined #openstack-nova | 22:40 | |
| *** bnemec has quit IRC | 22:44 | |
| *** bnemec has joined #openstack-nova | 22:48 | |
| *** dklyle has joined #openstack-nova | 22:50 | |
| *** JamesBenson has quit IRC | 22:50 | |
| *** bnemec has quit IRC | 22:54 | |
| *** tkajinam has joined #openstack-nova | 23:01 | |
| *** zzzeek has quit IRC | 23:10 | |
| *** zzzeek has joined #openstack-nova | 23:11 | |
| *** bnemec has joined #openstack-nova | 23:21 | |
| *** brinzhang0 has quit IRC | 23:22 | |
| *** brinzhang0 has joined #openstack-nova | 23:23 | |
| *** bnemec has quit IRC | 23:28 | |
| *** hamalq has quit IRC | 23:35 | |
| *** jamesdenton has quit IRC | 23:36 | |
| *** bnemec has joined #openstack-nova | 23:50 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
