| *** gyee has quit IRC | 00:21 | |
| *** altlogbot_1 has quit IRC | 00:23 | |
| *** irclogbot_3 has quit IRC | 00:23 | |
| *** edleafe has quit IRC | 00:24 | |
| *** takashin has joined #openstack-nova | 00:26 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | 00:27 |
|---|---|---|
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | 00:27 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | 00:27 |
| *** threestrands has quit IRC | 00:32 | |
| *** adriant7 has quit IRC | 00:32 | |
| *** adriant7 has joined #openstack-nova | 00:32 | |
| *** adriant7 has quit IRC | 00:32 | |
| *** markvoelker has joined #openstack-nova | 00:55 | |
| *** Sundar has joined #openstack-nova | 00:56 | |
| *** nicolasbock has quit IRC | 00:57 | |
| openstackgerrit | Merged openstack/nova master: re-calculate provider mapping during migration https://review.opendev.org/655112 | 01:00 |
| *** markvoelker has quit IRC | 01:01 | |
| *** Roamer` has quit IRC | 01:09 | |
| *** igordc has quit IRC | 01:16 | |
| *** BjoernT has joined #openstack-nova | 01:19 | |
| *** adriant has joined #openstack-nova | 01:21 | |
| *** irclogbot_3 has joined #openstack-nova | 01:34 | |
| *** irclogbot_3 has quit IRC | 01:39 | |
| *** irclogbot_2 has joined #openstack-nova | 01:41 | |
| *** irclogbot_2 has quit IRC | 01:43 | |
| *** BjoernT_ has joined #openstack-nova | 01:54 | |
| *** BjoernT has quit IRC | 01:55 | |
| *** tetsuro has joined #openstack-nova | 01:56 | |
| *** gbarros has joined #openstack-nova | 02:29 | |
| *** gbarros has quit IRC | 02:33 | |
| *** gbarros has joined #openstack-nova | 02:35 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | 02:39 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | 02:39 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | 02:39 |
| *** spsurya has joined #openstack-nova | 02:44 | |
| *** irclogbot_0 has joined #openstack-nova | 02:54 | |
| *** BjoernT_ has quit IRC | 02:56 | |
| *** irclogbot_0 has quit IRC | 02:59 | |
| *** markvoelker has joined #openstack-nova | 03:01 | |
| *** markvoelker has quit IRC | 03:05 | |
| *** tetsuro has quit IRC | 03:15 | |
| *** irclogbot_2 has joined #openstack-nova | 03:34 | |
| *** irclogbot_2 has quit IRC | 03:39 | |
| *** mkrai has joined #openstack-nova | 03:49 | |
| *** larainema has joined #openstack-nova | 03:56 | |
| *** mvkr has joined #openstack-nova | 04:02 | |
| *** irclogbot_1 has joined #openstack-nova | 04:02 | |
| *** tetsuro has joined #openstack-nova | 04:04 | |
| *** irclogbot_1 has quit IRC | 04:07 | |
| *** ricolin_ has joined #openstack-nova | 04:08 | |
| *** tetsuro has quit IRC | 04:08 | |
| *** ricolin_ has quit IRC | 04:08 | |
| *** ricolin has joined #openstack-nova | 04:10 | |
| openstackgerrit | Brin Zhang proposed openstack/nova master: Add delete_on_termination to volume-attach API https://review.opendev.org/673133 | 04:15 |
| *** gbarros has quit IRC | 04:18 | |
| *** udesale has joined #openstack-nova | 04:19 | |
| *** irclogbot_2 has joined #openstack-nova | 04:22 | |
| *** irclogbot_2 has quit IRC | 04:33 | |
| *** cfriesen has quit IRC | 04:35 | |
| *** bhagyashris has quit IRC | 04:37 | |
| *** Luzi has joined #openstack-nova | 05:00 | |
| *** cervigni has joined #openstack-nova | 05:10 | |
| cervigni | Hello, I have a dual socket server. Each cpu with two numa nodes. I would like to have the host to have a cpu overallocation of 2.0. At the same time I would like each flavor to stay bounded to its own numa node | 05:10 |
| cervigni | Is there a way to achieve that? This would mean 2x instances on each numa nodes | 05:10 |
| cervigni | I did not find a way to do that | 05:10 |
| cervigni | This is because each of the CPU is connected on the pci-e bus on a GPU | 05:10 |
| cervigni | and i need to respect the numa nodes configuration | 05:11 |
| *** Sundar has quit IRC | 05:20 | |
| *** irclogbot_2 has joined #openstack-nova | 05:21 | |
| *** Sundar has joined #openstack-nova | 05:21 | |
| *** irclogbot_2 has quit IRC | 05:25 | |
| *** rcernin_ has joined #openstack-nova | 05:31 | |
| *** rcernin has quit IRC | 05:33 | |
| openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider Config File: YAML file loading and schema validation https://review.opendev.org/673341 | 05:39 |
| openstackgerrit | Dustin Cowles proposed openstack/nova master: WIP: Provider Config File: Public method to retrieve custom resources https://review.opendev.org/676029 | 05:39 |
| openstackgerrit | Dustin Cowles proposed openstack/nova master: WIP: Provider Config File: Update provider tree with new custom resources https://review.opendev.org/676522 | 05:39 |
| *** ivve has quit IRC | 05:46 | |
| *** ratailor has joined #openstack-nova | 06:07 | |
| *** ratailor has quit IRC | 06:07 | |
| *** ratailor has joined #openstack-nova | 06:08 | |
| *** ricolin_ has joined #openstack-nova | 06:15 | |
| *** udesale has quit IRC | 06:15 | |
| *** irclogbot_3 has joined #openstack-nova | 06:17 | |
| *** ricolin has quit IRC | 06:17 | |
| *** irclogbot_3 has quit IRC | 06:21 | |
| *** udesale has joined #openstack-nova | 06:22 | |
| *** ratailor has quit IRC | 06:24 | |
| *** irclogbot_0 has joined #openstack-nova | 06:25 | |
| *** irclogbot_0 has quit IRC | 06:29 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove unused methods https://review.opendev.org/679918 | 06:30 |
| *** markvoelker has joined #openstack-nova | 06:34 | |
| *** markvoelker has quit IRC | 06:39 | |
| openstackgerrit | Yongli He proposed openstack/python-novaclient master: Microversion 2.78 - show server topology https://review.opendev.org/670790 | 06:49 |
| *** slaweq has joined #openstack-nova | 06:51 | |
| *** maciejjozefczyk has joined #openstack-nova | 06:54 | |
| *** markvoelker has joined #openstack-nova | 06:56 | |
| cervigni | guys, could you help me find out if this is a bug before I submit something stupid on Launchpad? | 06:58 |
| cervigni | https://pastebin.com/SfD63LgJ | 06:58 |
| *** zbr has quit IRC | 06:58 | |
| cervigni | it seems that a flavour configured as in the logs, continusly allocate cores only on the first numa node, instead of checking the actual cpus already allocated in that node | 06:59 |
| *** markvoelker has quit IRC | 07:00 | |
| *** rcernin_ has quit IRC | 07:01 | |
| gibi | cervigni: do you want dedicated physical cpu cores for your servers? | 07:02 |
| *** bbowen_ has joined #openstack-nova | 07:02 | |
| cervigni | I need to do an overallocation of 2x but that does not go out of its own numa node | 07:03 |
| cervigni | the virsh vcpuinfo for first instance is correct, for the second one is correct, but from the third one should start allocating on another numa node, since there is not available RAM left on numa node 1 | 07:03 |
| cervigni | I don't know how it can allocate further vcpus on that numa node since there is no ram available | 07:04 |
| *** bbowen has quit IRC | 07:04 | |
| * gibi looking at the ram side of the log | 07:04 | |
| cervigni | can I give you any more logs? | 07:05 |
| gibi | cervigni: so you don't have memory overallocation configured | 07:05 |
| cervigni | no | 07:05 |
| cervigni | memory allocation is 1.0 | 07:05 |
| gibi | cervigni: and you have the NUMATopologyFilter configured for the scheduler | 07:07 |
| cervigni | /etc/nova/nova.conf:enabled_filters=RetryFilter,AvailabilityZoneFilter,CoreFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,DifferentHostFilter,NUMATopologyFilter,AggregateInstanceExtraSpecsFilter,PciPassthroughFilter,AggregateMultiTenancyIsolation | 07:07 |
| cervigni | yes | 07:07 |
| gibi | then this feels like a bug, yes. | 07:08 |
| cervigni | I am using rocky | 07:08 |
| cervigni | oh dear :( | 07:08 |
| cervigni | should the scheduler fill a first round of the four numa nodes before overallocating? | 07:08 |
| *** ivve has joined #openstack-nova | 07:08 | |
| gibi | cervigni: I don't think so. If there is overallocation then the scheduler can overallocate before it fills the other numa nodes. The only rule is I know of that a server cannot overallocate against itself | 07:09 |
| *** zbr has joined #openstack-nova | 07:10 | |
| *** ratailor has joined #openstack-nova | 07:10 | |
| cervigni | ok | 07:10 |
| cervigni | so I will report it as a bug | 07:10 |
| cervigni | Thanks gibi | 07:10 |
| cervigni | I don't know if it relates of the VGPU side | 07:10 |
| cervigni | I hope not | 07:10 |
| gibi | cervigni: could you reporduce the problem without the VGPU resource in the flavor? | 07:11 |
| gibi | cervigni: thanks for reporting the bug | 07:11 |
| cervigni | I'll do it now | 07:11 |
| *** toabctl has quit IRC | 07:11 | |
| cervigni | gibi exactly same issue without VGPU | 07:13 |
| cervigni | any idea if there is a workaround I can use while the problem is fixed? | 07:13 |
| *** toabctl has joined #openstack-nova | 07:16 | |
| *** mdbooth_ has quit IRC | 07:19 | |
| gibi | unfortunately you cannot control the numa allocation directly from the outside | 07:22 |
| cervigni | https://bugs.launchpad.net/nova/+bug/1842616 | 07:22 |
| openstack | Launchpad bug 1842616 in OpenStack Compute (nova) "NUMA vcpus not correctly allocated against numa regions" [Undecided,New] | 07:22 |
| cervigni | I put you in the watchlist as well gibi | 07:23 |
| cervigni | Ok, for the moment I will leave no NUMA affinity, but with vgpus will be very messy | 07:23 |
| cervigni | Guess Stein will have the same issue? | 07:23 |
| *** luksky has joined #openstack-nova | 07:26 | |
| gibi | unfortunately I cannot try to reproduce it as I don't have numa + VGPU capable machine. I might get hold of a 2 numa nodes machine ... | 07:26 |
| *** tetsuro has joined #openstack-nova | 07:26 | |
| gibi | anyhow thanks for the report | 07:26 |
| cervigni | VGPU is irrelevant it seems | 07:26 |
| *** ratailor has quit IRC | 07:30 | |
| *** irclogbot_1 has joined #openstack-nova | 07:31 | |
| *** brinzhang has quit IRC | 07:32 | |
| *** brinzhang has joined #openstack-nova | 07:32 | |
| *** irclogbot_1 has quit IRC | 07:35 | |
| *** tetsuro has quit IRC | 07:38 | |
| *** Sundar has quit IRC | 07:38 | |
| *** brault has joined #openstack-nova | 07:41 | |
| *** pcaruana has joined #openstack-nova | 07:42 | |
| *** udesale has quit IRC | 07:42 | |
| *** udesale has joined #openstack-nova | 07:43 | |
| *** tetsuro has joined #openstack-nova | 07:46 | |
| *** irclogbot_1 has joined #openstack-nova | 07:51 | |
| *** irclogbot_1 has quit IRC | 07:53 | |
| *** ralonsoh has joined #openstack-nova | 07:55 | |
| *** ociuhandu has joined #openstack-nova | 07:57 | |
| *** ociuhandu has quit IRC | 07:59 | |
| *** ociuhandu has joined #openstack-nova | 08:00 | |
| kashyap | aspiers: Catching up with the relevant conversation w/ Eric in the scrollback. | 08:01 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: update allocation in binding profile during migrate https://review.opendev.org/656422 | 08:04 |
| *** yan0s has joined #openstack-nova | 08:04 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: prepare func test env for moving servers with bandwidth https://review.opendev.org/655109 | 08:06 |
| *** tkajinam has quit IRC | 08:07 | |
| *** damien_r has joined #openstack-nova | 08:09 | |
| *** damien_r has quit IRC | 08:09 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Func test for migrate server with ports having resource request https://review.opendev.org/655113 | 08:09 |
| *** damien_r has joined #openstack-nova | 08:10 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Make _rever_allocation nested allocation aware https://review.opendev.org/676138 | 08:12 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support reverting migration / resize with bandwidth https://review.opendev.org/676140 | 08:15 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Func test for migrate re-schedule with bandwidth https://review.opendev.org/676972 | 08:18 |
| *** Sundar has joined #openstack-nova | 08:18 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support migrating SRIOV port with bandwidth https://review.opendev.org/676980 | 08:20 |
| *** irclogbot_2 has joined #openstack-nova | 08:23 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Allow migrating server with port resource request https://review.opendev.org/671497 | 08:23 |
| *** ricolin_ is now known as ricolin | 08:25 | |
| *** irclogbot_2 has quit IRC | 08:25 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Do not query allocations twice in finish_revert_resize https://review.opendev.org/678827 | 08:26 |
| Sundar | Hi gibi, hoaw are you doing? | 08:27 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Allow resizing server with port resource request https://review.opendev.org/679019 | 08:29 |
| *** takashin has quit IRC | 08:31 | |
| openstackgerrit | Adrian Chiris proposed openstack/nova master: Revert "vif: Resolve a TODO and update another" https://review.opendev.org/675776 | 08:34 |
| *** markvoelker has joined #openstack-nova | 08:36 | |
| *** derekh has joined #openstack-nova | 08:36 | |
| *** ratailor has joined #openstack-nova | 08:39 | |
| *** mkrai has quit IRC | 08:39 | |
| *** ratailor has quit IRC | 08:39 | |
| *** markvoelker has quit IRC | 08:40 | |
| Sundar | Re. Cyborg notifications, I noticed that, once Nova kicks off the binding of ARQs, the CYborg event comes almost immediately if no programming is involved. The event gets lost because there are no waiters: I am doing the wait in the virt driver during spawn, The logs are here: http://paste.openstack.org/show/770470/ | 08:40 |
| *** ratailor has joined #openstack-nova | 08:40 | |
| openstackgerrit | Liang Fang proposed openstack/nova master: [WIP] Leverage openCAS cache framework for VM disks https://review.opendev.org/663542 | 08:42 |
| Sundar | gibi: ^ I am proposing to move the create/bind ARQs from build_and_run_instance (https://review.opendev.org/#/c/631244/37/nova/compute/manager.py@1940) to _build_resources (https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2392). | 08:43 |
| Sundar | gibi: This has 2 benefits: (a) The instance has th right host and hostname after resource claim, and (b) if VM spawn fails, there is a good place to delete the ARQs in flight. | 08:44 |
| *** priteau has joined #openstack-nova | 08:45 | |
| Sundar | The problem I am facing is this: can the logic be like http://paste.openstack.org/show/770481/? If non-ARQ-related stuff is not under the with, then the with statement is going to just kick off the binding and then wait till the resolution happens. Nothing else is going on during the wait. | 08:47 |
| *** tetsuro has quit IRC | 08:53 | |
| *** ociuhandu has quit IRC | 08:55 | |
| *** ratailor has quit IRC | 08:55 | |
| *** ociuhandu has joined #openstack-nova | 08:56 | |
| *** brault has quit IRC | 08:56 | |
| *** mdbooth has joined #openstack-nova | 08:58 | |
| *** ociuhandu has quit IRC | 09:00 | |
| *** trident has quit IRC | 09:01 | |
| *** shilpasd has joined #openstack-nova | 09:03 | |
| gibi | Sundar: on a call, I have to get back to you in 15 minutes | 09:04 |
| *** irclogbot_2 has joined #openstack-nova | 09:09 | |
| *** trident has joined #openstack-nova | 09:10 | |
| openstackgerrit | Brin Zhang proposed openstack/nova master: Add delete_on_termination to volume-attach API https://review.opendev.org/673133 | 09:11 |
| *** irclogbot_2 has quit IRC | 09:11 | |
| *** Roamer` has joined #openstack-nova | 09:16 | |
| *** slaweq has quit IRC | 09:16 | |
| gibi | Sundar: there is already a wait_for_instance_events call in nova.virt.libvirt.driver.LibvirtDriver._create_domain_and_network (called from spawn()) | 09:18 |
| gibi | Sundar: or you only want to wait for the cyborg specific event in _build_resources ? | 09:20 |
| *** dtantsur|afk is now known as dtantsur | 09:22 | |
| gibi | Sundar: but yeah, the goal is that you kick of the binding and while you are waiting for the response let other things progress for the build up until the point where you really need the ARQs to be bound | 09:27 |
| *** amrith has joined #openstack-nova | 09:27 | |
| *** amrith has left #openstack-nova | 09:28 | |
| gibi | Sundar: I might try to add the with wait_for_instance_events under with _build_resources call in https://github.com/openstack/nova/blob/cbaea3bd690c683e83a063acfa69919668eaa123/nova/compute/manager.py#L2233 | 09:30 |
| gibi | Sundar: I'm not sure how the the two context manager would combine if you put the wait_for_instance call inside the _build_resources without trying | 09:32 |
| *** slaweq has joined #openstack-nova | 09:33 | |
| *** ricolin has quit IRC | 09:34 | |
| *** ricolin has joined #openstack-nova | 09:35 | |
| kashyap | Huh, this commit added the CPU instruction: https://opendev.org/openstack/nova/commit/866ca1468dfa35f319f520d0f7b93b2d0fe32f62 | 09:36 |
| kashyap | (trait, i.e.) | 09:36 |
| kashyap | But it does _not_ exist in os-traits yet :-( | 09:37 |
| kashyap | There's no such trait as HW_CPU_X86_AVX512VNNI in 'os-traits' | 09:37 |
| kashyap | Hmm, never mind ... there _is_ the trait, and it is merged: https://review.opendev.org/#/c/672888/1/os_traits/hw/cpu/x86/__init__.py | 09:39 |
| *** cdent has joined #openstack-nova | 09:44 | |
| aspiers | sometimes I have to pip install the latest os-traits in my tox envs | 09:45 |
| aspiers | it's quicker than a --recreate | 09:45 |
| gibi | I guess tox did not pick up the os-traits >= 0.16.0 change in the requirements.txt | 09:46 |
| gibi | the nova master has it, maybe you need to rebase your patch | 09:46 |
| *** kaliya has joined #openstack-nova | 09:47 | |
| aspiers | tox doesn't seem good at noticing when to update anyway | 09:47 |
| aspiers | IME at least | 09:47 |
| *** shilpasd has quit IRC | 09:47 | |
| *** kaliya has quit IRC | 09:51 | |
| *** shilpasd has joined #openstack-nova | 10:07 | |
| openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Microversion 2.78: Add delete_on_termination to volume-attach API https://review.opendev.org/673485 | 10:09 |
| kashyap | aspiers: Yeah, I see what you mean; what I found was that in my venv: | 10:14 |
| kashyap | >>> import os_traits | 10:14 |
| kashyap | >>> print os_traits.HW_CPU_X86_AVX512VNNI | 10:14 |
| kashyap | HW_CPU_X86_AVX512VNNI | 10:14 |
| kashyap | (I _think_ I manually 'pip'-installed os-traits in there, I guess. I already forget) | 10:14 |
| *** irclogbot_0 has joined #openstack-nova | 10:29 | |
| *** shilpasd has quit IRC | 10:30 | |
| brinzhang | While run functional test cases, either execute tox or python -m testtools.run command, it will be raise "AssertionError: There is no script for 401 version", details http://paste.openstack.org/show/770633/ | 10:32 |
| *** udesale has quit IRC | 10:32 | |
| brinzhang | How can I avoid this issue? | 10:32 |
| *** udesale has joined #openstack-nova | 10:32 | |
| *** irclogbot_0 has quit IRC | 10:33 | |
| Sundar | gibi, Yes, we want to maximize the overlap by kicking off the binding as early as possible and wait for the ARQs as late as possible, but that is not easy without major surgery. So, I'll be content with dong it inside or after _build_resources. However, if we do: "with .... : foo(); bar()", then only foo() and bar() are done concurrently while wai | 10:35 |
| Sundar | ting for the event, right? Also, foo and bar must both be CYborg-related. | 10:35 |
| *** Sundar has quit IRC | 10:36 | |
| sean-k-mooney | brinzhang: "tox -e functional-py36 -- nova.tests.functional.api_sample_tests.test_quota_sets.NoopQuotaSetsSampleJsonTests.test_show_quotas" this works for me | 10:36 |
| sean-k-mooney | brinzhang: you might need to add -r to recreate the tox env. | 10:36 |
| gibi | Sundar: yes, only foo and bar will be parallel because when the with wait context manager exits it waits for the events | 10:36 |
| brinzhang | sean-k-mooney: I will have try, thanks | 10:36 |
| artom | brinzhang, did you play with database migrate versions? Because it looks like you bumped the version but didn't provide a migration file | 10:37 |
| brinzhang | artom: I am not update the db | 10:37 |
| gibi | Sundar: I don't see why foo and bar needs to be Cyborg related | 10:39 |
| artom | brinzhang, well, that's where the problem seems to be based on the paste you linked | 10:39 |
| artom | brinzhang, the stuff in nova/db/sqlalchemy/migrate_repo/versions/ | 10:40 |
| brinzhang | artom: yeah, that why I am confusing | 10:40 |
| *** markvoelker has joined #openstack-nova | 10:41 | |
| *** irclogbot_2 has joined #openstack-nova | 10:43 | |
| *** markvoelker has quit IRC | 10:45 | |
| *** irclogbot_2 has quit IRC | 10:45 | |
| brinzhang | sean-k-monney: http://paste.openstack.org/show/770638/ | 10:46 |
| sean-k-mooney | brinzhang: do you have local changes? | 10:47 |
| sean-k-mooney | i ran those tests against master | 10:47 |
| *** dave-mccowan has joined #openstack-nova | 10:47 | |
| brinzhang | Emm.. I am in https://review.opendev.org/#/c/673133/14 this patch | 10:48 |
| brinzhang | I will test in master now | 10:48 |
| sean-k-mooney | if you have a typo like incorrect indentaion or other syntax errors test discovery will fail | 10:49 |
| brinzhang | http://paste.openstack.org/show/770638/ it's a wrong director | 10:49 |
| sean-k-mooney | it will also fail if you if you have tempory filtes create by some edirtor like emacs where it adds a # to the filename | 10:50 |
| *** Sundar has joined #openstack-nova | 10:50 | |
| Sundar | gibi, Yes, we want to maximize the overlap by kicking off the binding as early as possible and wait for the ARQs as late as possible, but that is not easy without major surgery. So, I'll be content with dong it inside or after _build_resources. However, if we do: "with .... : foo(); bar()", then only foo() and bar() are done concurrently while wait | 10:51 |
| Sundar | ing for the event, right? Also, foo and bar must both be Cyborg-related. | 10:51 |
| gibi | Sundar: yes, only foo and bar will be parallel because when the with wait context manager exits it waits for the events | 10:51 |
| gibi | Sundar: I don't see why foo and bar needs to be Cyborg related | 10:51 |
| brinzhang | sean-k-mooney: I will delete the edit patch https://review.opendev.org/#/c/673133/14, and then re-fetch it. | 10:52 |
| Sundar | gibi: The timeout in the with will cover both foo and bar. Is it reasonable to impose that Cyborg ARQ bind timeout on non-Cyborg activities? it is 300 seconds by default. If it is ok, it will get us more concurrency. | 10:53 |
| *** irclogbot_3 has joined #openstack-nova | 10:59 | |
| gibi | Sundar: the timeout is not for foo and bar it is for waiting for the events | 11:00 |
| gibi | Sundar: also if the boot process cannot procede without such events then it is OK to wait for the events | 11:01 |
| *** ociuhandu has joined #openstack-nova | 11:01 | |
| *** irclogbot_3 has quit IRC | 11:02 | |
| Sundar | gibi: What do you think of waiting for the events in _build_resources, but actually querying Cyborg for ARQs only in the virt driver's spawn? If we do the query when the event comes, we have to store the ARQs in the instance object (or elsewhere), which will require changes to objects and db. | 11:03 |
| *** nicolasbock has joined #openstack-nova | 11:04 | |
| *** trident has quit IRC | 11:04 | |
| gibi | Sundar: I think it is better to wait earlier than potenitally loosing notifications as we start waiting too late | 11:05 |
| gibi | Sundar: I think you should also ask others, like dansmith about this problem | 11:05 |
| gibi | Sundar: if you start waiting in _build_resources then I would do the query in the virt driver instead of passing the ARQ down | 11:07 |
| gibi | Sundar: to limit the impact | 11:07 |
| Sundar | gibi: Great. That's what I had in mind. :) | 11:07 |
| Sundar | Thank you very much, gibi. | 11:08 |
| Sundar | Have a good day. | 11:08 |
| gibi | Sundar: same to you | 11:08 |
| *** Sundar has quit IRC | 11:10 | |
| *** shilpasd has joined #openstack-nova | 11:11 | |
| *** udesale has quit IRC | 11:12 | |
| *** trident has joined #openstack-nova | 11:13 | |
| sean-k-mooney | gibi: would you have time to review https://review.opendev.org/#/q/topic:bp/image-metadata-prefiltering+(status:open+OR+status:merged) today? im aiming to get that merged and free up a runway slot by the end of the week | 11:14 |
| sean-k-mooney | the 3 remaining pataches are relitivly short | 11:15 |
| gibi | sean-k-mooney: I will try... | 11:15 |
| sean-k-mooney | gibi: thanks if you dont get to it its ok i have others to bug too like bauzas | 11:16 |
| *** mkrai has joined #openstack-nova | 11:17 | |
| sean-k-mooney | bauzas: feel like re reviewing https://review.opendev.org/#/q/topic:bp/image-metadata-prefiltering+(status:open) | 11:17 |
| kashyap | aspiers: gibi: Commented here: https://review.opendev.org/#/c/644565/49/nova/virt/libvirt/utils.py@546 | 11:20 |
| sean-k-mooney | artom: i see you currently dont have any ci jobs running using the multi numa job so im going to swap it over too the new lables for FN | 11:21 |
| artom | sean-k-mooney, sure | 11:21 |
| sean-k-mooney | im also going to reduce the concurance to 1 | 11:21 |
| artom | sean-k-mooney, unless I'm missing something, the ball's in the reviewers's court | 11:22 |
| sean-k-mooney | assuming you are happy with them for testingin im going to then swap to testing the cpu work stephen has been doinging | 11:22 |
| gibi | kashyap: thanks! | 11:22 |
| *** zbr is now known as zbr|swimming | 11:23 | |
| artom | Though I need to play with the func test some more - the RPC pinning tests didn't pick up a thing they should have picked up | 11:23 |
| artom | sean-k-mooney, yep, that's fair, again much thanks for all your help | 11:23 |
| sean-k-mooney | thats the last patch in the seriese right e.g. it wont block other patches landing | 11:24 |
| artom | Func tests? Don't think so | 11:24 |
| artom | We can still land tests after FF, right? | 11:24 |
| sean-k-mooney | yes | 11:24 |
| artom | Even assuming the worst and don't get them sorted before then | 11:24 |
| sean-k-mooney | although its better to do it before | 11:24 |
| *** tbachman has quit IRC | 11:24 | |
| artom | That's the plan :) | 11:24 |
| brinzhang | sean-k-mooney, artom: I remember the unit test or functional test can run any linux env, depend on the devstack or openstack env is not necessary, is that? | 11:32 |
| sean-k-mooney | brinzhang: correct | 11:32 |
| sean-k-mooney | you do not need openstack or devstack to run them | 11:33 |
| brinzhang | sean-k-mooney, artom: I checked to the master and run the test, also report the error | 11:33 |
| sean-k-mooney | what os are you running on? | 11:33 |
| brinzhang | Centos7 | 11:33 |
| sean-k-mooney | brinzhang: you might want to clone a clean copy of nova and try it again | 11:33 |
| sean-k-mooney | brinzhang: ok i tested on popos and it worked fine | 11:34 |
| sean-k-mooney | i can check on rhel8 | 11:34 |
| brinzhang | On Monday, It runs ok, but today I git pull | 11:34 |
| sean-k-mooney | i know some test fail in rhel8 but i think that is due to the version of python it uses | 11:35 |
| brinzhang | Emm, in my env, I was installed py27 and py37,the default is py27 | 11:35 |
| sean-k-mooney | no i mean that using py36 on rhel8 some things fail that pass on py36 on pop_os/ubunutu | 11:37 |
| brinzhang | Clone the clean code is so slowly, sometimes maybe disconnect, I don’t want to do that. | 11:37 |
| sean-k-mooney | brinzhang: well i was only suggesting that since it might be related to unclean files in your working directory | 11:37 |
| sean-k-mooney | have you checked there are no tempory files created by your ide/editor | 11:38 |
| sean-k-mooney | im checking master on rhel8 now | 11:38 |
| sean-k-mooney | i guess i can check with a centos container too | 11:38 |
| brinzhang | I will check again in my working directory too, and try to clone it in another directory | 11:39 |
| sean-k-mooney | you can do a local clone by the way | 11:40 |
| sean-k-mooney | e.g. mkdir temp; cd temp; git clone ~/repos/nova; cd nova; tox -e ... | 11:40 |
| sean-k-mooney | it passed for me on rhel8 | 11:41 |
| sean-k-mooney | with no failurs at all | 11:41 |
| sean-k-mooney | which is better then it normally does | 11:42 |
| aspiers | sean-k-mooney: the issue with stestr and temporary editor files (e.g. from emacs) is fixed | 11:42 |
| sean-k-mooney | aspiers: as of when | 11:43 |
| aspiers | sean-k-mooney: https://github.com/mtreinish/stestr/issues/238 | 11:43 |
| *** irclogbot_0 has joined #openstack-nova | 11:43 | |
| aspiers | 2 weeks ago | 11:43 |
| sean-k-mooney | aspiers: i have hit its since march | 11:43 |
| aspiers | or 3 | 11:43 |
| aspiers | yeah you need to upgrade your stestr | 11:43 |
| aspiers | 2.5.0 | 11:43 |
| sean-k-mooney | maybe im just using whicever version tox installs | 11:44 |
| aspiers | Turns out that reporting bugs sometimes helps get them fixed ;-) | 11:44 |
| sean-k-mooney | although i have not recreate all my envs since then | 11:44 |
| aspiers | stestr>=1.0.0 # Apache-2.0 | 11:44 |
| aspiers | from test-requirements | 11:44 |
| aspiers | we should bump that | 11:44 |
| brinzhang | sean-k-mooney: yeah, cloning, but so slowly... | 11:44 |
| sean-k-mooney | brinzhang: if your doing it locally it should be fast | 11:44 |
| sean-k-mooney | e.g. the file system clone i suggested above | 11:45 |
| *** xek has joined #openstack-nova | 11:45 | |
| brinzhang | git clone ~/repos/nova what do you mean? isnot use git clone https://review.opendev.org/openstack/nova clone to another directory? | 11:46 |
| brinzhang | this is the clone speed "Receiving objects: 3% (21352/550825), 6.94 MiB | 27.00 KiB/s" | 11:47 |
| *** irclogbot_0 has quit IRC | 11:47 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: Bump min for oslo.service & .privsep to fix SIGHUP https://review.opendev.org/679974 | 11:50 |
| openstackgerrit | Silvan Kaiser proposed openstack/nova stable/stein: Exec systemd-run without --user flag in Quobyte driver https://review.opendev.org/660705 | 11:53 |
| *** tesseract has joined #openstack-nova | 12:03 | |
| *** larainema has quit IRC | 12:09 | |
| *** tbachman has joined #openstack-nova | 12:11 | |
| *** markvoelker has joined #openstack-nova | 12:13 | |
| *** xek has quit IRC | 12:21 | |
| sean-k-mooney | brinzhang: git can use filesystem urls | 12:26 |
| *** tbachman has quit IRC | 12:26 | |
| sean-k-mooney | brinzhang: so if you have a copy of a git repo you can clone a second copy using the file sytem path | 12:26 |
| sean-k-mooney | so for example i keep my dev repos under ~/repos | 12:27 |
| sean-k-mooney | if i do "mkdir ~/temp; cd temp" | 12:27 |
| efried | alex_xu, luyao: Are you around? | 12:27 |
| sean-k-mooney | then do "git clone ~/repos/openstack/nova" | 12:28 |
| sean-k-mooney | it give me a clean copy of the nova git tree in ~/temp/nova | 12:29 |
| brinzhang | sean-k-moony: Clone completed in another directory, and run tox -e ok, you are right, maybe the tempory file cause this issue. | 12:29 |
| luyao | efried: I'm here | 12:30 |
| brinzhang | sean-k-moony: I know your mean now, and I do it like you were said :P | 12:30 |
| efried | luyao: Do you need any help to make progress on the vpmem series at this point? Do you understand what dansmith is asking for on the bottom two patches? | 12:35 |
| luyao | efried: I do have some questions on the patch anout DB change. | 12:38 |
| efried | luyao: Okay. I don't have a good understanding of such things, but dansmith usually arrives on the scene in about 20 minutes. I know it's late for you... | 12:39 |
| luyao | efried: Ii'll wait for him, and I was going to wait. haha :) | 12:41 |
| efried | okay | 12:41 |
| *** zbr|swimming is now known as zbr | 12:42 | |
| *** etp has joined #openstack-nova | 12:44 | |
| *** eharney has quit IRC | 12:46 | |
| *** udesale has joined #openstack-nova | 12:48 | |
| sean-k-mooney | brinzhang: for what it is worth. whenever i hit test issue like that i first create a second local clone and test on the clean repo with master and with my patch branch if i still hit the issue i then check in a conainter. | 12:49 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: db: Add resources column in instance_extra table https://review.opendev.org/678447 | 12:49 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: object: Introduce Resource and ResouceList objs https://review.opendev.org/678448 | 12:49 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Add resources dict into _Provider https://review.opendev.org/678449 | 12:49 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Retrieve the allocations early https://review.opendev.org/678450 | 12:49 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Claim resources in resource tracker https://review.opendev.org/678452 | 12:49 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver discovering PMEM namespaces https://review.opendev.org/678453 | 12:49 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: report VPMEM resources by provider tree https://review.opendev.org/678454 | 12:49 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Support VM creation with vpmems and vpmems cleanup https://review.opendev.org/678455 | 12:49 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456 | 12:49 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/679640 | 12:49 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Add functional tests for virtual persistent memory https://review.opendev.org/678470 | 12:50 |
| *** ociuhandu has quit IRC | 12:50 | |
| brinzhang | sean-k-mooney: Good experience, got it. | 12:51 |
| *** luksky has quit IRC | 12:53 | |
| *** pas-ha has joined #openstack-nova | 12:54 | |
| *** markvoelker has quit IRC | 12:54 | |
| *** ociuhandu has joined #openstack-nova | 12:55 | |
| *** mkrai has quit IRC | 12:56 | |
| *** nweinber has joined #openstack-nova | 12:57 | |
| *** markvoelker has joined #openstack-nova | 13:00 | |
| *** tbachman has joined #openstack-nova | 13:03 | |
| *** mriedem has joined #openstack-nova | 13:04 | |
| *** markvoelker has quit IRC | 13:05 | |
| mriedem | bauzas: can you push this simple refactor through? https://review.opendev.org/#/c/679651/ | 13:09 |
| bauzas | mriedem: sure thing | 13:10 |
| luyao | dansmith: Are you around? | 13:10 |
| *** spatel has joined #openstack-nova | 13:11 | |
| bauzas | mriedem: gibi: I was paying attention to the qos-bandwith series but I'm a bit lost with it | 13:17 |
| bauzas | gibi: what's the top change ? | 13:17 |
| bauzas | https://review.opendev.org/#/c/679653/ ? | 13:17 |
| bauzas | because I don't see yet changes for cold migrate | 13:17 |
| openstackgerrit | sean mooney proposed openstack/nova master: multi numa nfv testing job https://review.opendev.org/679656 | 13:17 |
| openstackgerrit | sean mooney proposed openstack/nova master: [DNM] test migration with pinning https://review.opendev.org/679754 | 13:17 |
| mriedem | gibi: replied in https://review.opendev.org/#/c/656422/18/nova/compute/manager.py@2122 | 13:18 |
| bauzas | nah nevermind https://review.opendev.org/#/c/679019/4 | 13:18 |
| bauzas | gerrit fucks up with the series | 13:18 |
| bauzas | there are some branches | 13:18 |
| gibi | bauzas: this is the top https://review.opendev.org/#/c/679019/4 | 13:18 |
| gibi | bauzas: but I will respin the whole soonish to fix mriedem's comments | 13:19 |
| gibi | mriedem: looking | 13:19 |
| bauzas | gibi: okay i can wait then | 13:19 |
| mriedem | gibi: tl;dr i think you need the service version check in conductor after scheduler picks a dest and also a check in compute in case the compute rpc api versions are pinned in config | 13:19 |
| mriedem | i.e. the compute services can be running train but configured to only send stein level messages during an upgrade | 13:20 |
| mriedem | artom ran into this yesterday | 13:20 |
| mriedem | both are temporary and can be removed in U | 13:20 |
| brinzhang | mriedem: https://review.opendev.org/#/c/673133/14/nova/compute/api.py@4028, I was updated that comment, found the volume_bdm.save() report an "ObjectActionError: Object action obj_load_attr failed because: attribute id not lazy-loadable" | 13:21 |
| brinzhang | mriedem: That why? | 13:21 |
| gibi | mriedem: ack. I will put the checks in the conductor. Good point about that the service version is no enough | 13:21 |
| mriedem | brinzhang: in unit tests or a tempest run? | 13:21 |
| artom | *me meerkats up* | 13:22 |
| mriedem | brinzhang: if unit tests, you need to mock the BlockDeviceMapping.save call | 13:22 |
| brinzhang | mriedem: functioanl test failed | 13:22 |
| openstackgerrit | sean mooney proposed openstack/nova master: multi numa nfv testing job https://review.opendev.org/679656 | 13:22 |
| openstackgerrit | sean mooney proposed openstack/nova master: [DNM] test migration with pinning https://review.opendev.org/679754 | 13:22 |
| *** mkrai has joined #openstack-nova | 13:22 | |
| mriedem | brinzhang: ok i can look in a bit | 13:22 |
| gmann | brinzhang: mriedem just commented, it should be passed in via reserve_block_device_name() | 13:23 |
| artom | mriedem, I don't think they can be removed in U - dansmith can explain in more detail when he comes online because I'm not sure I understand it fully, but you can't remove a check until the next major version bump | 13:23 |
| artom | Or maybe I'm misunderstanding it | 13:24 |
| *** gbarros has joined #openstack-nova | 13:24 | |
| mriedem | gmann: no it shouldn't | 13:24 |
| gmann | we can directly save ? | 13:24 |
| mriedem | gmann: see my earlier comments on that very thing - i specifically don't want to do that b/c it adds a lot of unnecessary complexity | 13:24 |
| mriedem | i have to take my kid to the bus stop, will look into this later | 13:24 |
| gmann | ohk, did not see the previous comment | 13:25 |
| brinzhang | gmann: that will be change the rpc version, in PS13 do that | 13:25 |
| gibi | bauzas: It might take more time to update the bandwidth series than I first aniticipated. I will ping you when the new version is up | 13:26 |
| gmann | i did not see that, checking.. | 13:26 |
| *** markvoelker has joined #openstack-nova | 13:27 | |
| bauzas | gibi: no worries, I'm still working on updating the placement audit command | 13:28 |
| *** lpetrut has joined #openstack-nova | 13:31 | |
| *** Luzi has quit IRC | 13:32 | |
| dansmith | artom: to what are you referring? | 13:35 |
| artom | dansmith, all those checks I have about either making sure we got a param that appears in RPC 5.3, or that we can send RPC 5.3, those can only be removed in the next major version bump, right? | 13:36 |
| artom | dansmith, if so, mind explaining why, exactly? It's still not clear in my head why they can't just go away in U | 13:36 |
| dansmith | well, it's because there are two sets of "rules" at play here: | 13:37 |
| dansmith | The first is that we don't allow RPC upgrades from N-2 to N in terms of releases | 13:37 |
| dansmith | we do that not just because of strict rpc compatibility, but because of stuff buried deeper that expects fields to be filled out two releases from now, data migrations to have run, etc | 13:38 |
| artom | dansmith, this means that we don't support Train anything talking to Rocky anything, right? Only Stein | 13:38 |
| dansmith | the second is that the RPC versioning rules say that within a major version, you must be backwards compatible, able to accept anything back to 5.0 and able to send anything back to 5.0 if asked (by the version pin). If we always bumped versions on every release, then you could always remove stuff two releases from when you added it (since each release would have to bridge between two majors, you could never remove it in the next one) | 13:39 |
| *** udesale has quit IRC | 13:39 | |
| *** irclogbot_1 has joined #openstack-nova | 13:39 | |
| *** udesale has joined #openstack-nova | 13:39 | |
| dansmith | artom: that's what it means, but since the rpc version is all we have, it's really a soft statement of support that we can't validate or quantify, | 13:39 |
| dansmith | especially since people backport stuff | 13:40 |
| artom | People backport RPC stuff? | 13:40 |
| dansmith | the RPC versioning scheme that we have is why, in many cases, you actually can run N-2 talking to N when things haven't shifted much, and what would save our bacon if we had to start supporting such a thing | 13:40 |
| dansmith | upstream doesn't, but distros have in the past for sure | 13:41 |
| dansmith | however, | 13:41 |
| dansmith | it's easy to backport something that doesn't technically affect the rpc interface directly, but definitely changes behavior or semantics | 13:41 |
| dansmith | like something that causes a field to be filled out that wouldn't have been if the client was really as old as it is, etc | 13:41 |
| dansmith | like some distro that wanted to backport an api feature that now sends something newer, but doesn't technically need an rpc change to make it work | 13:42 |
| *** irclogbot_1 has quit IRC | 13:42 | |
| artom | So the reason the first rule doesn't invalidate the second rule is that even if we only allow a mix of N and N-1, there's nothing stopping an older RPC pin from being set? | 13:42 |
| dansmith | yes, but let me state it slightly differently: | 13:42 |
| dansmith | the first rule is our support policy and the second is the actual mechanical way we make sure our services can sanely talk to each other across versions | 13:43 |
| artom | Ah, I see | 13:43 |
| dansmith | so theoretically the mechanical bit should be able to support a wider envelope (rule 2) than we officially say we support (rule 1) | 13:43 |
| artom | IOW, we don't have mechanics in place to enforce our support policy, short of bumping major on every release | 13:44 |
| dansmith | because the N->N+2 thing has lots of other implications as well, like database migrations, things that need to check service versions to enable api features, etc, etc | 13:44 |
| dansmith | that's a true statement | 13:44 |
| artom | Aha, so the support policy is not just for RPC, it's for other things as well (that list you just said) | 13:45 |
| dansmith | yup | 13:45 |
| dansmith | because the other things are buried deep and don't necessarily even have a direct impact on rpc, but can definitely have an indirect one | 13:45 |
| dansmith | alright, so I've passed the torch of knowledge, now you can be the enforcer right? | 13:46 |
| artom | dansmith, not sure about that, but I'll write it up in https://docs.openstack.org/nova/stein/reference/index.html | 13:46 |
| mriedem | it's already written up in dan's blog... | 13:46 |
| mriedem | if you wanted to condense that, sure | 13:47 |
| dansmith | but writing it up in our docs would be good | 13:47 |
| mriedem | what i was thinking about was https://review.opendev.org/#/c/634606/73/nova/compute/manager.py@6574 specifically is only needed for train, | 13:47 |
| mriedem | since we don't support Stein -> U | 13:47 |
| dansmith | if my blog is part of the official nova docs, I'm going to throw some ads up there :) | 13:47 |
| artom | dansmith, for AWS :D | 13:47 |
| dansmith | mriedem: that's only the case because of the rule 1 above | 13:48 |
| mriedem | sure, | 13:48 |
| dansmith | mriedem: they can still set the pin lower than train in U and would fail to make that call.. it'd be an unsupported situation, but it's probably good if we just leave it in place | 13:48 |
| mriedem | and law of nova #35 says we'll forget to remove the compat code for 3 releases anyway | 13:48 |
| dansmith | heh | 13:48 |
| * artom doesn't want to know what rule 34 of nova is | 13:48 | |
| dansmith | I can tell you what rule #1 is | 13:49 |
| mriedem | there is a rule somewhere that you must have at least 3 different types of BlockDevice(Mapping) objects | 13:49 |
| dansmith | #17 iirc | 13:49 |
| artom | dansmith, you left us hanging on #1 there | 13:49 |
| dansmith | artom: don't talk about nova, obvi | 13:50 |
| * artom shuts up | 13:50 | |
| *** etp has quit IRC | 13:50 | |
| artom | OK, I lied. | 13:50 |
| alex_xu | dansmith: let me know I understand this rule correct or not https://review.opendev.org/#/c/678447/7/nova/db/sqlalchemy/migrate_repo/versions/401_add_resources.py@32 :) | 13:50 |
| artom | dansmith, mriedem, just so I'm clear, are we waiting on anything from me and/or CI for the NUMA LM series? | 13:50 |
| dansmith | artom: nope, I had it up to be looking at when I saw your ping | 13:51 |
| mriedem | i've been online for ~45 minutes and already have a full tab queue, so no | 13:51 |
| luyao | dansmith: Hi, I have a question about https://review.opendev.org/#/c/678447/8 . Do you mean the script can run out of the nova code, the script will just update DB, then db column is deleted while the updates on nova code have not taken effect.I don't understand how I should do. | 13:51 |
| alex_xu | luyao: i just asked :) | 13:52 |
| artom | mriedem, yep, fully away of your overcommitment, as just making sure I understood where we stood :) | 13:52 |
| artom | *was | 13:52 |
| dansmith | alex_xu: yes, scheduled_at is the only field we've removed in a long time since we started doing the model/schema sync, so follow that pattern | 13:52 |
| *** BjoernT has joined #openstack-nova | 13:52 | |
| alex_xu | dansmith: ah, got it, leave comment on the model obj, and rename the exisiting field to resources, then remove it next release | 13:53 |
| alex_xu | dansmith: thanks! | 13:53 |
| dansmith | alex_xu: rename the field on the model, but you have to add the column to the whitelist so it will be ignored, but yeah. | 13:53 |
| luyao | dansmith: I got this fail on test_models_sync , AssertionError: Models and migration scripts aren't in sync: | 13:53 |
| mriedem | "add the column to the whitelist so it will be ignored" | 13:53 |
| alex_xu | dansmith: what the whitelist mean? | 13:54 |
| *** luksky has joined #openstack-nova | 13:54 | |
| dansmith | alex_xu: look in those scheduled_at patches.. there's a whitelist for the test that validates that the model and schema match | 13:54 |
| *** amrith has joined #openstack-nova | 13:54 | |
| dansmith | artom: looks like the ci job on top never really completes because sean-k-mooney is working on it.. got a link to a recent run? | 13:55 |
| *** mlavalle has joined #openstack-nova | 13:55 | |
| artom | dansmith, so I left a comment on PS7 of https://review.opendev.org/#/c/679656/ linking to various signs of success | 13:56 |
| sean-k-mooney | dansmith: well i just started updateing them again today since there were no running jobs | 13:56 |
| artom | lemme see if I can find something more recent | 13:56 |
| dansmith | artom: yeah that was before the hugepage work right? | 13:56 |
| artom | dansmith, I believe that was after | 13:56 |
| sean-k-mooney | this was after https://46b5dc7879590352c3ef-b484a2b89d5d2c358f068133dfb2fa14.ssl.cf5.rackcdn.com/679805/2/experimental/nova-nfv-multi-numa-multinode/b040d28/testr_results.html.gz | 13:56 |
| artom | dansmith, 13:12 my time, I was talking to danpb that morning about hugepages | 13:57 |
| sean-k-mooney | this is from the run where i forced the vcpu_pin_set and cpu_shared_set to be reversed on each host | 13:58 |
| sean-k-mooney | https://46b5dc7879590352c3ef-b484a2b89d5d2c358f068133dfb2fa14.ssl.cf5.rackcdn.com/679805/2/experimental/nova-nfv-multi-numa-multinode/b040d28/compute/logs/etc/nova/nova_conf.txt.gz | 13:58 |
| luyao | dansmith, alex_xu : I found the whitelist. Thanks! | 13:59 |
| alex_xu | luyao: really, I'm not...still looking | 13:59 |
| *** irclogbot_1 has joined #openstack-nova | 13:59 | |
| alex_xu | ah, I found it | 14:00 |
| *** eharney has joined #openstack-nova | 14:00 | |
| luyao | alex_xu: it's in nova/tests/unit/db/test_migrations.py, def filter_metadata_diff | 14:00 |
| alex_xu | luyao: you win | 14:00 |
| alex_xu | dansmith: thanks again | 14:00 |
| *** irclogbot_1 has quit IRC | 14:02 | |
| yonglihe | alex_xu: Good evening. | 14:02 |
| alex_xu | yonglihe: good evening too :) | 14:02 |
| *** mkrai has quit IRC | 14:02 | |
| *** mkrai_ has joined #openstack-nova | 14:03 | |
| *** mkrai_ has quit IRC | 14:05 | |
| *** irclogbot_2 has joined #openstack-nova | 14:05 | |
| *** irclogbot_2 has quit IRC | 14:08 | |
| *** mkrai has joined #openstack-nova | 14:08 | |
| dansmith | artom: sean-k-mooney: What're ya'll doing to test the rollback case IRL? | 14:09 |
| artom | dansmith, IRL? nothing. Trying to understand why the func test wasn't picking it up | 14:10 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove Rocky compute compat checks for live migration with port bindings https://review.opendev.org/680005 | 14:10 |
| dansmith | artom: okay, I would expect you'd set up something to fail the migration to make sure the rollback actually happens properly | 14:10 |
| sean-k-mooney | e.g. a failure to migrate, i have inetvertenly tested it by forgettting to exchanged ssh keys and haveing the migrate fail and rollback | 14:10 |
| sean-k-mooney | that is the only testing of it i did | 14:10 |
| dansmith | sean-k-mooney: is it getting far enough that it actually calls the new method? | 14:10 |
| sean-k-mooney | yes | 14:10 |
| dansmith | okay | 14:10 |
| sean-k-mooney | i have not done that on the latest version | 14:11 |
| sean-k-mooney | want me to do that | 14:11 |
| sean-k-mooney | i just need to drop the ssh key form autherised keys | 14:11 |
| *** irclogbot_3 has joined #openstack-nova | 14:11 | |
| dansmith | sean-k-mooney: can you also do something else for us? pin the compute version on one machine to 5.2 and migrate both directions to make sure it still works | 14:11 |
| dansmith | because we really *really* don't want to break migrations for upgrades | 14:11 |
| artom | dansmith, I suppose I can setup stein/master and try it | 14:11 |
| sean-k-mooney | sure. i can do that in a ci test too | 14:11 |
| dansmith | if we find out late that we did and need to tweak rpc stuff to make it work, life is gonna suck | 14:12 |
| artom | dansmith, oh, just pinning. yeah :) | 14:12 |
| sean-k-mooney | but ill test both | 14:12 |
| dansmith | sean-k-mooney: okay thanks | 14:12 |
| sean-k-mooney | dansmith: i can also give you acess once i have it set up if you want to poke around | 14:12 |
| dansmith | I see the logs show it actually doing the new stuff so that's cool | 14:12 |
| dansmith | sean-k-mooney: that's okay, I' | 14:12 |
| artom | (Wouldn't func be sufficient - like, I get wanting to see real libvirt with the updated XML, and thank god because it caught bugs) | 14:12 |
| dansmith | sean-k-mooney: I'll trust some pastebin'd logs | 14:12 |
| artom | (But rollback is all compute) | 14:13 |
| dansmith | artom: func is great, but it's not the real thing | 14:13 |
| dansmith | artom: trust me, you don't want to be on a sev1 where you realize you're going to need to add an RPC .z version to fix this | 14:13 |
| dansmith | we did that in icehouse for a customer and it was not cool | 14:13 |
| artom | Fully agree there | 14:13 |
| *** irclogbot_3 has quit IRC | 14:14 | |
| artom | sean-k-mooney, how feasible would it be to modify the grenade job to run with a NUMA flavor? | 14:14 |
| artom | On FN hardwa | 14:15 |
| artom | *hardware | 14:15 |
| sean-k-mooney | so i am less familar with legacy jobs but it could be done | 14:15 |
| sean-k-mooney | the nodeset is not hard but forcing it to use nested virt is the tricky bit | 14:15 |
| sean-k-mooney | devstack gate is non tivial | 14:16 |
| *** tbachman has quit IRC | 14:16 | |
| sean-k-mooney | adding grendate on top is harder | 14:16 |
| sean-k-mooney | if we need to do that we could however | 14:16 |
| artom | sean-k-mooney, well, dansmith said he's satisfied with pastebin, so maybe more long term | 14:16 |
| sean-k-mooney | i would like to port greadne to a non legacy job at some point | 14:16 |
| sean-k-mooney | well well i have not updated https://review.opendev.org/#/c/679805/ to pick up the other changes i did this morning | 14:17 |
| sean-k-mooney | i can updated it and add the different compute versions in that | 14:17 |
| sean-k-mooney | ill try and test it locally first however | 14:17 |
| *** lpetrut has quit IRC | 14:19 | |
| mriedem | speaking of converting a job to zuulv3, https://review.opendev.org/#/c/674345/ converts the nova-lvm job to zuulv3 and has a +2 | 14:19 |
| sean-k-mooney | dansmith: do you know how to pin the rpc version of the top of your head. ill look it up in teh config ref if not | 14:19 |
| mriedem | sean-k-mooney: [upgrade_levels]/compute=stein | 14:20 |
| dansmith | yeah that | 14:20 |
| sean-k-mooney | thanks | 14:20 |
| mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#upgrade_levels.compute | 14:20 |
| sean-k-mooney | does it matter if i do this on the compute or the contoler? e.g. we want the conductor ect to not be pinned right | 14:21 |
| mriedem | once that nova-lvm change is merged the only legacy jobs we have are the live migration and grenade multinode ones, which i've probably put off b/c of the external script they run, but nova-next was converted and runs an external script so i can probably do similar | 14:21 |
| sean-k-mooney | just the source node? | 14:21 |
| mriedem | sean-k-mooney: if you pin in nova-cpu.conf i don't think it matters | 14:21 |
| mriedem | conductor is using a different config | 14:21 |
| sean-k-mooney | ya just want to make sure i dont mess up my local.conf and have to redeploy | 14:22 |
| sean-k-mooney | ill pin it on the dedicated compute node | 14:22 |
| sean-k-mooney | nova-cpu.conf would also work | 14:22 |
| * mriedem thinks he just said that | 14:22 | |
| sean-k-mooney | yes you did i was agreeing with you | 14:23 |
| mriedem | i can tinker later this afternoon with converting that nova-grenade-multinode job to zuulv3 | 14:24 |
| mriedem | those have been on my todo list for awhile | 14:24 |
| sean-k-mooney | i can also try if you run into issue but i have never actully run greade and i dont know if we have the ansibel playbooks to run it so im not sure how easy that will be | 14:25 |
| sean-k-mooney | is greade hard to run or is basically the same as devstack | 14:26 |
| *** edleafe has joined #openstack-nova | 14:27 | |
| *** tesseract has quit IRC | 14:30 | |
| mriedem | it's just a set of bash scripts that orchestrates setting up n-1 devstack, creating some resources, shutting down and then running N version devstack and making sure those resources are still around | 14:32 |
| mriedem | i've never run it locally manually, only in CI | 14:32 |
| gibi | mriedem: looking at the MigrationTask. When the dest is selected by the scheduler I can check right away if the dest has a new enough compute service version. But if not, then what? simulate a re-schedule locally in the conductor? https://github.com/openstack/nova/blob/cbaea3bd690c683e83a063acfa69919668eaa123/nova/conductor/tasks/migrate.py#L248 | 14:32 |
| mriedem | gibi: you'd have to iterate alternates | 14:33 |
| sean-k-mooney | mriedem: looking at the zuul config they dont have a non legacy way to run it | 14:33 |
| sean-k-mooney | https://opendev.org/openstack/grenade/src/branch/master/.zuul.yaml | 14:33 |
| mriedem | it's times like this that i've thought about having a compute service version filter in the scheduler | 14:33 |
| mriedem | so we can say internally "only give me computes with service version >= x for this feature" | 14:33 |
| sean-k-mooney | mriedem: well if that feature was a compute capablity tratis we could use placemtn for that | 14:34 |
| gibi | mriedem: Yeah that was what I mean by simualting a re-schedule locally , as I have to take care of num_retries and the allocations too | 14:34 |
| mriedem | sean-k-mooney: https://review.opendev.org/#/c/548936/ | 14:34 |
| sean-k-mooney | like your multi attach stuff | 14:34 |
| openstackgerrit | Merged openstack/nova master: Make _get_cpu_feature_traits() always return a dict https://review.opendev.org/679568 | 14:34 |
| openstackgerrit | Merged openstack/nova master: libvirt: Make scheduler filters customizable https://review.opendev.org/679745 | 14:34 |
| sean-k-mooney | mriedem: ya i saw that mentioned on the infra irc yesterday | 14:34 |
| mriedem | gibi: num_retries, you mean max_attempts? | 14:35 |
| sean-k-mooney | i was not sure how far along it was | 14:35 |
| mriedem | gibi: if so, that's built into the number of alternates you get back from the scheduler | 14:35 |
| gibi | mriedem: ohh, good point. If I run out of alternates then raise MaxRetriesExceeded | 14:35 |
| *** BjoernT_ has joined #openstack-nova | 14:35 | |
| mriedem | gibi: yeah, and write it in a separate method so we can drop that in U or V | 14:36 |
| gibi | mriedem: sure | 14:36 |
| sean-k-mooney | mriedem: ok it looks like that is passing. i wonder is there a reason it is not merged yet. i should go ask them | 14:37 |
| *** BjoernT has quit IRC | 14:37 | |
| gibi | mriedem: also I think I have to check the version of the source compute as well becuase request spec travels like cond -> dest -> source -> dest so if the source is old then the request_spec will not reach the dest at the end | 14:37 |
| mriedem | sean-k-mooney: the scheduler will already filter dest computes that have the required nested inventory for qos bw providers, but we do'nt know if that dest compute is stein or train to support cold migratoin with those things, and i don't think we want to get into a habit of adding a compute capability for every "supports_fancy_unicorn" feature just for rolling upgrade support | 14:37 |
| *** irclogbot_2 has joined #openstack-nova | 14:37 | |
| mriedem | sean-k-mooney: probably b/c grenade cores (of which i'm one) haven't looked at it | 14:37 |
| mriedem | gibi: correct, like the port binding stuff in live migration, you have to check source and dest | 14:38 |
| gibi | mriedem: OK. thanks for the help. | 14:38 |
| mriedem | gibi: the source you could check in the API i guess | 14:38 |
| mriedem | and 409 if it's old | 14:38 |
| gibi | mriedem: yes, I can do that | 14:38 |
| * gibi goes back to it's IDE | 14:39 | |
| *** tesseract has joined #openstack-nova | 14:39 | |
| mriedem | his? | 14:40 |
| *** irclogbot_2 has quit IRC | 14:40 | |
| * mriedem notes gibi just exposed that he's truly a cyborg | 14:40 | |
| *** markvoelker has quit IRC | 14:41 | |
| *** tbachman has joined #openstack-nova | 14:41 | |
| * gibi hates English for the 3 different grammatical genders | 14:42 | |
| *** tesseract has quit IRC | 14:44 | |
| gibi | (as a note, in hungarian are no gender specific words for he, she, it, just a single 'Å‘' ) | 14:46 |
| *** luksky has quit IRC | 14:48 | |
| *** shilpasd has quit IRC | 14:49 | |
| *** damien_r has quit IRC | 14:51 | |
| *** irclogbot_1 has joined #openstack-nova | 14:52 | |
| mriedem | could be worse, could be french, right? isn't everything gender specific in french? | 14:52 |
| *** damien_r has joined #openstack-nova | 14:52 | |
| gibi | I know german a bit, and that already worse :) | 14:54 |
| *** irclogbot_1 has quit IRC | 14:54 | |
| *** eharney has quit IRC | 14:54 | |
| gibi | and I can imagine hungarian being worse on many different level compared to english | 14:54 |
| *** damien_r has quit IRC | 14:55 | |
| sean-k-mooney | gibi: you can almost always use the gender nutral form in english if you prefer although singular they or it can annoy some | 14:55 |
| mriedem | brinzhang: the reason for the functional test failures is inline https://review.opendev.org/#/c/673133/ | 14:56 |
| stephenfin | mriedem, gibi: I need a way to prevent people only setting '[compute] cpu_dedicated_set' on hosts that have non-pinned instances (to prevent https://review.opendev.org/#/c/674895/22/nova/virt/libvirt/driver.py@7358) | 14:56 |
| sean-k-mooney | for some reason people dont like to be refered to as "it" | 14:56 |
| mriedem | it's because of stubs in the api samples tests | 14:56 |
| gibi | sean-k-mooney: thank. I try to remember to that | 14:56 |
| mriedem | if we were using real compute services and the CinderFixture we wouldn't have that problem, but re-writing those tests is beyond this patch | 14:56 |
| stephenfin | I'm thinking iterating through every single instance on the host in 'LibvirtDriver.init_host' is too expensive | 14:56 |
| gibi | stephenfin: don't we already iterating every instance in init_host? | 14:57 |
| sean-k-mooney | we do in update_resouces or whatevre its called | 14:57 |
| sean-k-mooney | i thikn we call that as part of init | 14:57 |
| stephenfin | gibi: If we do, I don't see it | 14:57 |
| sean-k-mooney | we iterate over all instance to plug there interfaces | 14:57 |
| sean-k-mooney | at the very least | 14:57 |
| stephenfin | It all seems to be basic config opt checks | 14:58 |
| stephenfin | Hmm | 14:58 |
| mriedem | gibi might be thinking of ComputeManager.init_host | 14:58 |
| dansmith | artom: check me here: https://review.opendev.org/#/c/634606/73 | 14:58 |
| gibi | yeah, I think I remember ComputeManager.init_host | 14:58 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: db: Add resources column in instance_extra table https://review.opendev.org/678447 | 14:58 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: object: Introduce Resource and ResouceList objs https://review.opendev.org/678448 | 14:58 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Add resources dict into _Provider https://review.opendev.org/678449 | 14:58 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Retrieve the allocations early https://review.opendev.org/678450 | 14:58 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Claim resources in resource tracker https://review.opendev.org/678452 | 14:58 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver discovering PMEM namespaces https://review.opendev.org/678453 | 14:58 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: report VPMEM resources by provider tree https://review.opendev.org/678454 | 14:58 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Support VM creation with vpmems and vpmems cleanup https://review.opendev.org/678455 | 14:58 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456 | 14:58 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/679640 | 14:58 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Add functional tests for virtual persistent memory https://review.opendev.org/678470 | 14:58 |
| dansmith | artom: that's a big nasty patch, so it's a good sign that I only had a couple minor things | 14:58 |
| mriedem | ComputeManager.init_host gets all instances https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1256 | 14:59 |
| stephenfin | Oh, ComputeManager.init_host | 14:59 |
| stephenfin | I was thinking this would have to live in 'LibvirtDriver.init_host' | 14:59 |
| artom | dansmith, witness me! | 14:59 |
| mriedem | stephenfin: is cpu_dedicated_set a new config option in that series? | 14:59 |
| stephenfin | yup | 15:00 |
| mriedem | how do you determine if an instance is pinned or not? | 15:00 |
| *** shilpasd has joined #openstack-nova | 15:00 | |
| mriedem | looking at the instance.numa_topology? | 15:00 |
| cdent | try to lift it up | 15:01 |
| cdent | if it doesn't move... | 15:01 |
| stephenfin | mriedem: Instance.numa_topology.cpu_pinning_requested | 15:01 |
| * mriedem cues rimshot | 15:01 | |
| mriedem | stephenfin: so none of that is libvirt specific yeah? | 15:01 |
| stephenfin | technically no. It just won't do anything without libvirt | 15:01 |
| mriedem | for now | 15:02 |
| stephenfin | That 'cpu_pinning_requested' property would never be set on non-libvirt driver | 15:02 |
| stephenfin | Correct | 15:02 |
| mriedem | so...add a compute driver capability so a nova-compute running vsphere doesn't have to look at that for 1K+ instances managed by that host | 15:02 |
| *** amrith has quit IRC | 15:02 | |
| mriedem | if self.driver.capabilities.get('supports_some_hw_magic', False): | 15:03 |
| mriedem | # do expensive thing | 15:03 |
| *** cfriesen has joined #openstack-nova | 15:03 | |
| stephenfin | so not "if CONF.compute_driver == 'libvirt'" ? | 15:04 |
| mriedem | alternatively we change the ComputeDriver.init_host interface to pass the list of instances on that host into the virt driver | 15:04 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: object: Introduce Resource and ResourceList objs https://review.opendev.org/678448 | 15:04 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Add resources dict into _Provider https://review.opendev.org/678449 | 15:04 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Retrieve the allocations early https://review.opendev.org/678450 | 15:04 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Claim resources in resource tracker https://review.opendev.org/678452 | 15:04 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver discovering PMEM namespaces https://review.opendev.org/678453 | 15:04 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: report VPMEM resources by provider tree https://review.opendev.org/678454 | 15:04 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Support VM creation with vpmems and vpmems cleanup https://review.opendev.org/678455 | 15:04 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456 | 15:04 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/679640 | 15:04 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Add functional tests for virtual persistent memory https://review.opendev.org/678470 | 15:04 |
| artom | stephenfin, eww | 15:04 |
| mriedem | stephenfin: right that ^ | 15:04 |
| artom | stephenfin, I have no context, but errr | 15:04 |
| artom | *eww | 15:04 |
| mriedem | we generally don't ever want to look at that config to make decisions | 15:04 |
| stephenfin | fair fair | 15:04 |
| mriedem | *something something factory pattern* | 15:04 |
| mriedem | anyway, maybe people would prefer to just swap those 3 lines of code in ComputeManager.init_host to pass the InstanceList to the driver and let the driver sort it out | 15:05 |
| mriedem | either way you're going to explode and kill the start of the service if it's misconfigured correct? | 15:05 |
| stephenfin | yup | 15:07 |
| luyao | dansmith: comments addressed. https://review.opendev.org/#/c/678447/9 and https://review.opendev.org/#/c/678448/12 | 15:07 |
| *** tbachman has quit IRC | 15:08 | |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: db: Add resources column in instance_extra table https://review.opendev.org/678447 | 15:12 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: object: Introduce Resource and ResourceList objs https://review.opendev.org/678448 | 15:12 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Add resources dict into _Provider https://review.opendev.org/678449 | 15:12 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Retrieve the allocations early https://review.opendev.org/678450 | 15:12 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Claim resources in resource tracker https://review.opendev.org/678452 | 15:12 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver discovering PMEM namespaces https://review.opendev.org/678453 | 15:12 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: report VPMEM resources by provider tree https://review.opendev.org/678454 | 15:12 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Support VM creation with vpmems and vpmems cleanup https://review.opendev.org/678455 | 15:12 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456 | 15:12 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/679640 | 15:12 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Add functional tests for virtual persistent memory https://review.opendev.org/678470 | 15:12 |
| *** tesseract has joined #openstack-nova | 15:14 | |
| *** damien_r has joined #openstack-nova | 15:16 | |
| *** damien_r has quit IRC | 15:16 | |
| *** yan0s has quit IRC | 15:17 | |
| *** mkrai has quit IRC | 15:17 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: Bump min for oslo.service & .privsep to fix SIGHUP https://review.opendev.org/679974 | 15:21 |
| openstackgerrit | Adam Spiers proposed openstack/nova master: Ensure q35 machine type is used when booting with SEV https://review.opendev.org/680065 | 15:22 |
| *** damien_r has joined #openstack-nova | 15:22 | |
| kashyap | cfriesen: Hi, you or someone you know picking this up? -- https://review.opendev.org/#/c/621646/ ("Preserve UEFI NVRAM variable store") | 15:23 |
| kashyap | Oh, actually zhu.boxiang has picked it up lately | 15:24 |
| cfriesen | kashyap: yeah, was just going to say. :) | 15:25 |
| *** zbr is now known as zbr|out | 15:25 | |
| sean-k-mooney | dansmith: i think the cross rpc migration is working. i have not looked at the logs to confim but it succeed via horizon | 15:30 |
| dansmith | sean-k-mooney: okay, with a numa flavor or non-numa flavor? | 15:30 |
| sean-k-mooney | with cpu pinning enabled | 15:31 |
| sean-k-mooney | so numa | 15:31 |
| dansmith | okay, so the behavior we expect is for it to work, not claim, and just translate the topo directly over to the other side, for better or worse, correct? | 15:31 |
| dansmith | artom: ^ | 15:31 |
| openstackgerrit | Adam Spiers proposed openstack/nova master: Ensure q35 machine type is used when booting with SEV https://review.opendev.org/680065 | 15:31 |
| aspiers | efried, kashyap, sean-k-mooney: ^^^ this implements what we agreed yesterday | 15:31 |
| artom | dansmith, yep, aka, what we have now | 15:32 |
| dansmith | yeah, okay | 15:32 |
| kashyap | aspiers: Already reading the code for a couple of mins (while on a call) | 15:32 |
| aspiers | kashyap: awesome, I just uploaded PS2 though | 15:32 |
| aspiers | realised I missed a few little bits out | 15:32 |
| sean-k-mooney | dansmith: do i need to test the sucess case or was it the revert case that you were worried about | 15:32 |
| dansmith | sean-k-mooney: so, one more test you could do is pop all the patches off from one node and just validate that you can still migrate between them (with the version pinned) | 15:32 |
| artom | dansmith, because as soon as we start claiming, it's all or nothing, we need to handle applying the migration context, dropping the claim on rollback, etc | 15:32 |
| dansmith | sean-k-mooney: yeah, that'd be good too | 15:33 |
| kashyap | aspiers: I see that you've put the check in hardware.py | 15:33 |
| aspiers | kashyap: yes. This new approach is actually quite a lot cleaner | 15:33 |
| *** irclogbot_2 has joined #openstack-nova | 15:33 | |
| aspiers | so I'm glad we had the discussions | 15:33 |
| *** ociuhandu has quit IRC | 15:34 | |
| kashyap | aspiers: Okay, then. Because there's a prereq on hardware feature, hardware.py? | 15:34 |
| aspiers | kashyap: it's just adding another check alongside the existing 2 extra specs / image props checks | 15:35 |
| openstackgerrit | Adam Spiers proposed openstack/nova master: Apply SEV-specific guest config when SEV is required https://review.opendev.org/644565 | 15:35 |
| stephenfin | mriedem: What exception type should I be raising in ComputeManager.init_host if my checks fail. I'm thinking 'InternalError' since it's an operator misconfiguration | 15:35 |
| sean-k-mooney | dansmith: ok so before i check artom code vs master or the revert case. was there somehting in partcal you wanted me to grab form the logs or db | 15:35 |
| kashyap | aspiers: Right; so I can probably put my _guest_machine_type_for_secure_boot() still in utils.py. | 15:35 |
| dansmith | sean-k-mooney: just want to make sure it's not doing numa migration I guess.. there are a couple logs that indicate it's doing the new stuff | 15:36 |
| *** irclogbot_2 has quit IRC | 15:36 | |
| aspiers | kashyap: now I realise why you were asking that | 15:36 |
| kashyap | aspiers: :-) | 15:36 |
| dansmith | sean-k-mooney: you don't really have to pastebin, I trust you, just check to make sure you don't see those on either side | 15:36 |
| artom | sean-k-mooney, dansmith, presence of LibvirtLiveMigrateNUMAInfo is a big sign | 15:36 |
| dansmith | artom: like this one: https://review.opendev.org/#/c/634606/73/nova/virt/libvirt/driver.py@7465 | 15:36 |
| dansmith | artom: yep | 15:36 |
| aspiers | kashyap: yeah I guess yours could still go in utils.py | 15:36 |
| sean-k-mooney | artom: are you expecting it to regenerat the xml when the RPC version is pinned? | 15:37 |
| artom | We log the migrate_data all over the place, so if that's not in it, we haven't NUMA live-migrated | 15:37 |
| artom | sean-k-mooney, no | 15:37 |
| sean-k-mooney | ok ill check that so | 15:37 |
| *** irclogbot_2 has joined #openstack-nova | 15:37 | |
| mriedem | stephenfin: i see the libvirt driver init_host raises Invalid, but we already have a InvalidConfiguration so that seems more appropriate | 15:37 |
| artom | Any RPC pin on either source, dest or cond should cause the current half-broken behaviour | 15:37 |
| mriedem | it doesn't really matter though | 15:37 |
| kashyap | aspiers: It's a similar method. But in your case, I see you're clubbing it all together with the relevant mem encryption- / SEV-related checks | 15:37 |
| stephenfin | ack | 15:37 |
| sean-k-mooney | the config i have would require the xml to be update to be correct so i shoudl see the xml updated in only 1 direction | 15:37 |
| sean-k-mooney | or not at all | 15:38 |
| aspiers | kashyap: right | 15:38 |
| *** gbarros has quit IRC | 15:38 | |
| kashyap | aspiers: I'll just Depends-On yours for the exception class | 15:39 |
| aspiers | kashyap: or just make it the child commit | 15:39 |
| aspiers | kashyap: isn't Depends-On more for cross-repo stuff? | 15:39 |
| *** altlogbot_1 has joined #openstack-nova | 15:39 | |
| sean-k-mooney | artom: dansmith: pinned source => unpinned dest no xml updates on migration | 15:39 |
| kashyap | aspiers: True, Depends-On is indeed normally used for cross-repo things | 15:40 |
| *** macz has joined #openstack-nova | 15:41 | |
| sean-k-mooney | artom: and i get erros in the compute log because the vm xml is invalid for the new host vcpu_pin_set so that is working as we expect | 15:41 |
| artom | "and i get erros in the comput" *panic* "so that is working as we expect" *relief | 15:41 |
| artom | u kill me man | 15:42 |
| sean-k-mooney | its compliing the vm cpu pinning is not valid for the host in the periodici task that caulates resouce usage | 15:42 |
| artom | Which is a known bug I think | 15:42 |
| sean-k-mooney | its compliing the vm cpu pinning is not valid for the host in the periodici task that caulates resouce usage | 15:42 |
| sean-k-mooney | wrong terminal | 15:43 |
| sean-k-mooney | well its not really a bug so much as use say hay you vm pinning is invalid | 15:43 |
| *** tbachman has joined #openstack-nova | 15:44 | |
| *** markvoelker has joined #openstack-nova | 15:45 | |
| artom | I mean there's a launchpad bug for it somewhere | 15:45 |
| sean-k-mooney | ya there is a patch that sugges we skip updating the resouse if its invalid | 15:46 |
| sean-k-mooney | which i think we both nack'd | 15:46 |
| sean-k-mooney | artom: dansmith same behavior for new source to old dest. no xml update | 15:47 |
| artom | sean-k-mooney, yep | 15:47 |
| dansmith | cool | 15:47 |
| artom | dansmith, well, figured out why my func tests weren't doing the expected thing | 15:47 |
| artom | I was stubbing stuff on the ClientRouter, which is a global singleton | 15:47 |
| * dansmith waits for it | 15:47 | |
| artom | So *all* 3 services got capped | 15:48 |
| dansmith | ah yeah | 15:48 |
| artom | I shouldn't hit the same problem if I replace the actual compute_rpcapi objects, though, right? | 15:49 |
| dansmith | you can get access to the managers in the services you start right? | 15:49 |
| dansmith | so manager.rpcapi = $newthing | 15:49 |
| artom | I couldn't figure out how to either cap the objects directly, or create new ones and replace. Looks like I have no choice to do that now. | 15:49 |
| dansmith | I think | 15:49 |
| artom | Yeah, it's "$newthing" that needs thinking about | 15:50 |
| artom | There's no __init__, so how do I cap it? | 15:50 |
| mriedem | compute = self.start_service('compute', host='host1') | 15:50 |
| mriedem | compute.manager.compute_rpcapi = my_stub | 15:50 |
| mriedem | or something like that | 15:51 |
| mriedem | stub the compute_rcpapi on one of the services in the func test | 15:51 |
| dansmith | that ^ | 15:51 |
| dansmith | I think you could also not stub the full rpcapi, and just stub out can_send_version on the one it sets up for you | 15:52 |
| openstackgerrit | Adam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption https://review.opendev.org/666616 | 15:53 |
| artom | can_send_version is on the client though | 15:53 |
| sean-k-mooney | so ... there is one issue. if we live migrate the server to a host where the pinning is invalide. | 15:53 |
| sean-k-mooney | then we cold migrate it | 15:53 |
| sean-k-mooney | the cold migrtion fails in resize confim | 15:53 |
| artom | sean-k-mooney, is pinning involved, or master to master? | 15:53 |
| sean-k-mooney | yes | 15:54 |
| artom | Yes to...? | 15:54 |
| sean-k-mooney | nova.exception.CP│ | 15:54 |
| sean-k-mooney | UUnpinningUnknown: CPU set to unpin [12, 15, 4, 7] must be a subset of known CPU set [] | 15:54 |
| artom | Wouldn't that be a latent thing? | 15:54 |
| sean-k-mooney | im going to triple check but i think cold migration of a pinned instnace is fine | 15:54 |
| sean-k-mooney | jsut not one that has been live migrtation to a host with out regenrting the xml if the xml is invalid for that host | 15:55 |
| *** priteau has quit IRC | 15:55 | |
| artom | That's still a different bug though, and why we moved to block numa live migration by default? | 15:55 |
| sean-k-mooney | i can check but i think this would happen on master too | 15:55 |
| *** dtantsur is now known as dtantsur|afk | 15:55 | |
| sean-k-mooney | by the way i was kind of expecting that might fail im just reporing what im finding as i try to break it | 15:57 |
| artom | Totally, this sort of testing is great | 15:57 |
| openstackgerrit | Adam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption https://review.opendev.org/666616 | 15:57 |
| artom | Wait, would live migrating to an invalid pinning situation go through? | 15:58 |
| openstackgerrit | Merged openstack/nova master: Refactor MigrationTask._execute https://review.opendev.org/679651 | 15:58 |
| openstackgerrit | Merged openstack/nova master: unit test: do not fill rp mapping for failed re-schedule https://review.opendev.org/679653 | 15:58 |
| sean-k-mooney | ok cold migration of a new instance works fine | 15:58 |
| artom | I guess as long as the host CPUs are online, we don't check whether they're part of vcpu_pin_set during the LM | 15:58 |
| sean-k-mooney | ill check reverting to master on the compute node then master only | 15:58 |
| artom | So it'd make sense for that to fail in a subsequent operation | 15:59 |
| *** gbarros has joined #openstack-nova | 15:59 | |
| *** gyee has joined #openstack-nova | 15:59 | |
| sean-k-mooney | artom: right libvirt checks they are are a thing but libvirt is not aware of the vcpu_pin_set | 15:59 |
| sean-k-mooney | and i have intentionally configred theyse hsot so the vcpu_pin_sets on each host does not overlap | 16:00 |
| artom | Right, to make sure the old broken behaviour is happening | 16:00 |
| sean-k-mooney | yes | 16:01 |
| sean-k-mooney | and to give a really hard edgecase | 16:01 |
| aspiers | kashyap: I've just built an image with an experimental kernel and grub secure boot fails due to "invalid signature" | 16:02 |
| aspiers | kashyap: any idea how to tell it the signature is trustworthy? | 16:02 |
| sean-k-mooney | one server is set up to only allow resouce on numa node 0 the other for node 1 only and it have the cpu_shared_set configure to use the opistie numa node to the vcpu_pin_set | 16:02 |
| kashyap | aspiers: Yeah, "experimental kernal" is the keyword :-) | 16:03 |
| sean-k-mooney | aspiers: it has not been sgined with a key that is in the secure boot keystore would be my guess | 16:03 |
| kashyap | So, there is a manual way to make it "trustworthy" | 16:04 |
| kashyap | aspiers: But I don't see a document for it. Last I checked with an admin who does it, said it's quite tedious | 16:04 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add delete_on_termination to volume-attach API https://review.opendev.org/673133 | 16:05 |
| sean-k-mooney | dansmith: artom os i have a proablem with 1 node on master and the other node on artoms code | 16:05 |
| sean-k-mooney | on startup the agent on master code is raising "ServiceTooOld: This service is older (v39) than the minimum (v40) version of the rest of the deployment." | 16:05 |
| donnyd | Is there a way to ask nova for a specific trait on an instance without configuring a flavor first? | 16:06 |
| openstackgerrit | Adam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption https://review.opendev.org/666616 | 16:06 |
| artom | sean-k-mooney, update the database I think | 16:06 |
| kashyap | aspiers: If Secure Boot is not a reqiurement for your test, then I'd suggest to disable it | 16:06 |
| sean-k-mooney | i guess i need to pin artoms code to the older version? | 16:06 |
| *** ociuhandu has joined #openstack-nova | 16:06 | |
| aspiers | kashyap: how? | 16:06 |
| mriedem | donnyd: required traits are defined on the flavor or image | 16:06 |
| artom | sean-k-mooney, I hit that a few times when I git pulled a new service version, but the version in the database for the service was old | 16:06 |
| sean-k-mooney | artom: well i didnt use the head of master. i used the version of master your code was bassed on | 16:06 |
| kashyap | aspiers: You mean, "disable how"? | 16:06 |
| aspiers | kashyap: yes, how do I disable it | 16:07 |
| sean-k-mooney | e.g. this is the frist commit before your code | 16:07 |
| dansmith | sean-k-mooney: you'll have to delete some service records | 16:07 |
| sean-k-mooney | ah right | 16:07 |
| dansmith | sean-k-mooney: that message is coming because you've got entries in the db that say everything should be upgraded already | 16:07 |
| sean-k-mooney | if i just stop the agent and do a compute servcie delete that should fix it right | 16:08 |
| sean-k-mooney | alternitivel i can restack that node | 16:08 |
| donnyd | Is there any way to place a custom request in via the sdk? | 16:08 |
| donnyd | The scenario is I use some sort of discovery to populate placement with traits, and I don't want to configure a flavor for each specific type | 16:08 |
| kashyap | aspiers: Don't know top off my head; but let me check with a dev who does this more often | 16:08 |
| mriedem | stephenfin: looking at https://review.opendev.org/#/c/621476/ alex is +2 and you asked for docs fixes in PS61 which are done now, so you might want to hit that again soonish, | 16:09 |
| mriedem | stephenfin: especially b/c i'm +2 on https://review.opendev.org/#/c/673133/15 which is competing for the same microversion | 16:09 |
| kashyap | aspiers: In the BIOS menu, it depends on your vendor | 16:09 |
| aspiers | kashyap: we are talking about a VM in nova here, remember | 16:10 |
| stephenfin | mriedem: Sure, I can do that before I leave | 16:10 |
| aspiers | it's some virtual BIOS thing | 16:10 |
| *** ociuhandu has quit IRC | 16:11 | |
| kashyap | aspiers: Right, it's a VM; a quick check -- you're using one of those enrolled "VARS" files shipped by your distro I guess? | 16:11 |
| kashyap | aspiers: If so, the quickest way to disable it is to just start your guest with a non-enrolled (i.e. not signed by MS) "VARS" file. | 16:12 |
| aspiers | kashyap: right, as per my comment on https://review.opendev.org/#/c/348394/ | 16:12 |
| sean-k-mooney | we need to update osc to better support service deleteion | 16:13 |
| aspiers | kashyap: in that final comment there is a list of all the "code" files | 16:13 |
| aspiers | kashyap: which have corresponding "vars" files | 16:13 |
| aspiers | kashyap: which one looks most promising? | 16:13 |
| kashyap | Looking | 16:13 |
| sean-k-mooney | osc only support the id not the service uuid | 16:13 |
| aspiers | kashyap: currently I'm using ovmf-x86_64-suse-code.bin and ovmf-x86_64-suse-vars.bin | 16:13 |
| mriedem | sean-k-mooney: that's not accurate | 16:13 |
| mriedem | https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/compute-service.html#compute-service-delete | 16:14 |
| aspiers | kashyap: I have no idea what "4m" means | 16:14 |
| kashyap | aspiers: I once did research on that SUSE naming, but totally forgot | 16:14 |
| kashyap | aspiers: Hmm, so you _are_ using a non-MS signed one | 16:14 |
| sean-k-mooney | well the help text in the version i have installed says it thakes the id and the uuid did not work | 16:14 |
| mriedem | sean-k-mooney: you have to specify the microversion with osc | 16:14 |
| mriedem | it defaults to 2.1 | 16:14 |
| mriedem | so: openstack --os-compute-api-version 2.53 compute service delete <service uuid> | 16:15 |
| sean-k-mooney | oh ya that is a thing | 16:15 |
| kashyap | aspiers: Hint: that "4m" will be described in your RPM spec file; but that's not useful for your immediate prob | 16:15 |
| sean-k-mooney | the help text does not methion that | 16:15 |
| *** damien_r has quit IRC | 16:15 | |
| kashyap | aspiers: Try this one: ovmf-x86_64.bin (and the corresponding VARS file) | 16:15 |
| mriedem | sean-k-mooney: b/c you have old osc | 16:15 |
| mriedem | i added that recently | 16:16 |
| sean-k-mooney | ah ok im using 3.19.0 | 16:16 |
| sean-k-mooney | its not a big deal anyway | 16:16 |
| *** ivve has quit IRC | 16:17 | |
| aspiers | kashyap: OK but as per my comment on https://review.opendev.org/#/c/348394/ ovmf-x86_64-code.bin did not work, so not confident that ovmf-x86_64.bin will work either | 16:17 |
| kashyap | aspiers: Reading that; I was relying on the file name suggested by Dirk correct :-( | 16:17 |
| kashyap | aspiers: Also, I find it crazy that the SUSE RPM has _so_ many binaries; you need a simple combination: | 16:17 |
| kashyap | aspiers: (a) a non-SB binary and a non-enrolled VARS file; (b) an SB-binary and an enrolled (MS-signed) VARS file | 16:18 |
| aspiers | yeah I have no clue why it's like that | 16:18 |
| kashyap | aspiers: Can you get a link to the RPM spec file, please? That will clarify what they are all. | 16:19 |
| *** zbr|out is now known as zbr | 16:19 | |
| kashyap | aspiers: Incidentally, the "light at the end of the tunnel" is it's going to be much easier, once SUSE ships: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=932269 | 16:20 |
| openstack | Debian bug 932269 in ovmf "Ship the firmware "descriptor files" as part of the 'ovmf' package" [Normal,Fixed] | 16:20 |
| kashyap | (Debian and Ubuntu now do. As does Fedora, for a while) | 16:21 |
| sean-k-mooney | hum so i still get the same error after deleting the compute service | 16:21 |
| *** jaosorior has joined #openstack-nova | 16:23 | |
| aspiers | kashyap: https://build.opensuse.org/package/view_file/Virtualization/ovmf/ovmf.spec?expand=1 | 16:23 |
| dansmith | sean-k-mooney: you have to stop them all, delete them all, then start up the old one first | 16:23 |
| sean-k-mooney | oh i have stop and delete the compute service on the "new" node too | 16:24 |
| sean-k-mooney | not just the one im puting ot master | 16:24 |
| sean-k-mooney | ok | 16:24 |
| donnyd | mriedem: is there a way to craft custom requests via the sdk? | 16:24 |
| sean-k-mooney | donnyd: if you mean raw http requests yes | 16:25 |
| aspiers | kashyap: do you really think the exception message should refer to "Q35" when the way to fix the exception is to set hw_machine_type=q35 (lowercase) ? | 16:25 |
| sean-k-mooney | donnyd: you can use the sdk to jsut get a rest client and then perform raw GET|POST|PUT|DELETE|... request with the client with whatever body you want | 16:26 |
| donnyd | So is it possible to craft a raw request that would allow me to ask placement for a node that meets X trait and then tell nova to provision that resource | 16:26 |
| kashyap | aspiers: Thanks for the link; that "4m" corresponds to a 4MB flash image. | 16:26 |
| aspiers | ah interesting | 16:26 |
| kashyap | aspiers: On the exception, ah, lowercase is indeed fine. (Sorry) | 16:26 |
| aspiers | kashyap: np, thanks | 16:27 |
| sean-k-mooney | the first half yes the scond no not direcltly | 16:27 |
| *** eharney has joined #openstack-nova | 16:27 | |
| sean-k-mooney | if you want nova to schduler a vm to a host with a give trait you can add it as a required trait in etiher the flavor or image | 16:27 |
| sean-k-mooney | and then nova will include it in the request to placment | 16:27 |
| sean-k-mooney | you cant however ask placmenet for an allocation caniate for a vm and tell nova to use it | 16:28 |
| donnyd | sean-k-mooney: maybe a little more context would help, this is for baremetal | 16:28 |
| openstackgerrit | Adam Spiers proposed openstack/nova master: Ensure q35 machine type is used when booting with SEV https://review.opendev.org/680065 | 16:28 |
| kashyap | aspiers: Also, can you please confirm from the SUSE OVMF pkg maintainer -- *which* of your biaries are built with SECURE_BOOT_ENABLE flag? | 16:28 |
| sean-k-mooney | you are trying to land a nova instance on a specific ironic host? | 16:28 |
| kashyap | aspiers: Need to dial into a call; back later. | 16:29 |
| donnyd | yes | 16:29 |
| donnyd | without having to create custom flavors... because it would result in a large # of flavors to manage and maintain | 16:29 |
| sean-k-mooney | do you want to be able to do it or do you want non admins to be able to do it | 16:30 |
| donnyd | CI jobs, so non admin | 16:30 |
| donnyd | Maybe have to create some custom policy or something like that | 16:31 |
| sean-k-mooney | am do you need to jsut deferenciate between ironic nodes and vms or do you need to select a specific host | 16:31 |
| donnyd | I need to be able to tell nova to select a specific metal host ( no vm's yet) | 16:32 |
| sean-k-mooney | ok the hacky way to do that is to abuse the availablty zone. but there are other ways | 16:32 |
| openstackgerrit | Adam Spiers proposed openstack/nova master: Ensure q35 machine type is used when booting with SEV https://review.opendev.org/680065 | 16:34 |
| donnyd | so like nova boot --availability-zone ZONE:HOST:NODE | 16:34 |
| sean-k-mooney | yes | 16:35 |
| donnyd | efried: shared that with me a while back | 16:35 |
| sean-k-mooney | im not sure if nodepool supports that | 16:35 |
| sean-k-mooney | i think it might | 16:35 |
| openstackgerrit | Adam Spiers proposed openstack/nova master: Apply SEV-specific guest config when SEV is required https://review.opendev.org/644565 | 16:35 |
| sean-k-mooney | the other way to do it would be a schduler hint | 16:35 |
| openstackgerrit | Adam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption https://review.opendev.org/666616 | 16:36 |
| sean-k-mooney | i think you could use the json filter to do this | 16:36 |
| donnyd | how do you do it with the scheduler hint? | 16:36 |
| sean-k-mooney | so the hoststate object contains the hostname uuid and nodename | 16:37 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L109-L111 | 16:37 |
| sean-k-mooney | the json filter can match on any field in the host state object | 16:37 |
| donnyd | ok cool. Thanks for entertaining my questions :) | 16:38 |
| aspiers | stephenfin, efried: is check_can_live_migrate_destination() the right place to prevent live migration in the SEV case? | 16:39 |
| sean-k-mooney | i think you do openstack server create --hint query="$host==ironic_host_name" | 16:40 |
| sean-k-mooney | again im not sure if the openstack nodepool driver suppots that. or not | 16:41 |
| sean-k-mooney | donnyd: the json filter is also off by defualt as normally we dont want people to be able to pick specific hosts | 16:41 |
| *** luksky has joined #openstack-nova | 16:42 | |
| sean-k-mooney | but both the availablity zone and json filters can technically do it | 16:42 |
| aspiers | efried, kashyap: nits with q35 check addressed | 16:42 |
| donnyd | With the json filter, will it fail if it cannot find the specific host? | 16:42 |
| sean-k-mooney | yes | 16:42 |
| donnyd | ok, I think both of those two options will work for this case | 16:43 |
| donnyd | much appreciated sean-k-mooney | 16:43 |
| sean-k-mooney | donnyd: the json filter basicaly allows you to match on anything in this object https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L109-L159 so you can be very specific | 16:44 |
| sean-k-mooney | donnyd: you might be able to use https://zuul-ci.org/docs/nodepool/configuration.html#attr-providers.[openstack].pools.labels.instance-properties | 16:46 |
| sean-k-mooney | but you defintely can use https://zuul-ci.org/docs/nodepool/configuration.html#attr-providers.[openstack].pools.availability-zones | 16:46 |
| sean-k-mooney | well i have never tired it | 16:47 |
| *** mdbooth has quit IRC | 16:50 | |
| *** cdent has quit IRC | 16:51 | |
| -openstackstatus- NOTICE: Zuul job logs stored in OVH may fail. We have updated the base job to remove OVH from our storage location. If you have POST_FAILURES a recheck should fix them at this point. | 16:52 | |
| mriedem | ewww jsonfilter | 16:52 |
| mriedem | i'm not sure how that helps with linking hosts with a given trait | 16:53 |
| sean-k-mooney | mriedem: it does not. donnyd wanted a way per vm to selct a given host | 16:53 |
| mriedem | donnyd: you can use the placement CLI to filter providers with required traits as well https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-list | 16:53 |
| donnyd | ask placement for host that matches x trait and then place request in for Host that is returned | 16:53 |
| mriedem | donnyd: which version of nova? | 16:54 |
| donnyd | .latest atm | 16:54 |
| *** mdbooth has joined #openstack-nova | 16:54 | |
| sean-k-mooney | latest as in stien not master | 16:54 |
| donnyd | This isn't for FN, its for a different CI I am working on | 16:54 |
| mriedem | so stein or master? | 16:54 |
| mriedem | b/c train has https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id66 | 16:54 |
| donnyd | Probably going to be train | 16:54 |
| mriedem | which would be your easiest thing | 16:54 |
| mriedem | in train the server create api takes a requested host | 16:55 |
| mriedem | and runs it through the scheduler | 16:55 |
| mriedem | otherwise you can use the JsonFilter like sean-k-mooney said or force the host of your choosing with admin creds | 16:55 |
| sean-k-mooney | ya that is the best approch if you can use train. although nodepool wont supprot that | 16:55 |
| mriedem | https://docs.openstack.org/nova/latest/admin/availability-zones.html | 16:55 |
| mriedem | ^ forced host | 16:55 |
| donnyd | sean-k-mooney: Not sure it will be using nodepool | 16:56 |
| mriedem | heed the JsonFilter warnings https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#jsonfilter | 16:56 |
| mriedem | JsonFilter is pretty common for putting a baremetal instance on a specific baremetal node | 16:56 |
| mriedem | 2.74 avoids the need for the JsonFilter for that though, which is nice | 16:57 |
| sean-k-mooney | yep | 16:57 |
| donnyd | mriedem: that is quite precisely the use case. Trying to pick a specific metal host | 16:57 |
| donnyd | in train will that work for metal and VM? | 16:58 |
| mriedem | yes | 16:58 |
| donnyd | nice | 16:58 |
| donnyd | well then it will be worth the wait | 16:58 |
| mriedem | host = nova-compute service hostname | 16:58 |
| donnyd | :) | 16:58 |
| mriedem | hypervisor_hostname = node name = ironic uuid | 16:59 |
| mriedem | for kvm the host and hypervisor_hostname are the same thing | 16:59 |
| sean-k-mooney | donnyd: by way do you openstack to reinstall the os on the ironic node | 16:59 |
| *** derekh has quit IRC | 17:00 | |
| sean-k-mooney | if you dont you can also use the static driver if you end up using node pool https://zuul-ci.org/docs/nodepool/configuration.html#static-driver | 17:00 |
| sean-k-mooney | adding support to the new feature to the openstack driver however could be useful | 17:01 |
| donnyd | sean-k-mooney: I openstack for everything in FN | 17:02 |
| *** shilpasd has quit IRC | 17:06 | |
| *** udesale has quit IRC | 17:07 | |
| *** kaisers has quit IRC | 17:09 | |
| *** ociuhandu has joined #openstack-nova | 17:22 | |
| * aspiers is lost in a twisty maze of API files, all alike | 17:22 | |
| *** ociuhandu has quit IRC | 17:26 | |
| *** kaisers has joined #openstack-nova | 17:28 | |
| *** maciejjozefczyk has quit IRC | 17:32 | |
| sean-k-mooney | artom: dansmit: sorry have been distracted by a headache that developed over the lst hour or two. | 17:37 |
| sean-k-mooney | i finally gets master compute + artom code on contoler | 17:38 |
| sean-k-mooney | the migration was the same as the pinned rpc. | 17:38 |
| sean-k-mooney | actully its stil pinned ill unpin and check | 17:38 |
| *** ociuhandu has joined #openstack-nova | 17:38 | |
| aspiers | sean-k-mooney: can you quickly advise me on where to add a check to refuse API requests to migrate/suspend SEV guests? is nova/compute/api.py the right place? | 17:38 |
| aspiers | it's libvirt-specific so kind of feels like it belongs in the driver | 17:38 |
| mriedem | aspiers: isn't it a generic trait in the flavor though? | 17:39 |
| mriedem | failing in the driver kind of sucks if you flat out don't support migrating that type of instance | 17:39 |
| sean-k-mooney | mriedem: well there is a generic resouce class | 17:39 |
| aspiers | mriedem: no, typically the trait won't be used until Intel MKTME comes in | 17:39 |
| mriedem | meaning you'll reschedule to another host that will also fail | 17:40 |
| aspiers | mriedem: right, that's what I was worried about | 17:40 |
| mriedem | what makes a sev instance a sev instance? | 17:40 |
| mriedem | something in the flavor or image right? | 17:40 |
| aspiers | yes | 17:40 |
| aspiers | hw:mem_encryption | 17:40 |
| sean-k-mooney | you could block it either in the api by checking for the flavor extra spec | 17:40 |
| sean-k-mooney | or in a prifilter i guess | 17:40 |
| aspiers | sean-k-mooney: that's what I want to do, yep | 17:40 |
| aspiers | I'm just trying to find the right bit of code | 17:40 |
| mriedem | pre-filter doesn't work since it doesn't have the context on you doing a move operatoin vs a server create | 17:41 |
| sean-k-mooney | its the validate_flavor_image_nostatus funciton | 17:41 |
| mriedem | so just failing fast in the api or conductor would be sufficient | 17:41 |
| sean-k-mooney | or one that it calls | 17:41 |
| aspiers | nova.compute.api.API.live_migrate()? | 17:41 |
| mriedem | what about cold migrate? does that work? | 17:42 |
| aspiers | no | 17:42 |
| sean-k-mooney | aspiers: you can do it the way we do for sriov migration i guess. | 17:42 |
| aspiers | need to also block suspend | 17:42 |
| sean-k-mooney | aspiers: no? | 17:42 |
| sean-k-mooney | why would cold migration not work | 17:42 |
| sean-k-mooney | suspenmd i get | 17:42 |
| aspiers | oh sorry | 17:42 |
| mriedem | heh, you might as well just write a decorator then | 17:42 |
| sean-k-mooney | its doing a managed save | 17:42 |
| aspiers | yeah cold migration probably OK | 17:42 |
| * aspiers rereads his own spec | 17:42 | |
| aspiers | http://specs.openstack.org/openstack/nova-specs/specs/train/approved/amd-sev-libvirt-support.html#limitations | 17:42 |
| *** ociuhandu has quit IRC | 17:42 | |
| aspiers | yes, cold migration/shelve is OK | 17:43 |
| aspiers | That's why I explicitly wrote that in the spec - cos I knew I'd forget later ;-) | 17:43 |
| mriedem | def block_for_sev(func, context, instance, *args, **kwargs): if 'hw:mem_encryption' in instance.flavor.extra_specs: raise OperationNotSupportedForSev() | 17:44 |
| sean-k-mooney | aspiers: if you do it in the conductor do it here. but if you eant to do it in the api. yuou need to do in a different location | 17:44 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L318 | 17:44 |
| mriedem | then just decorate the methods that don't support sev instances | 17:44 |
| sean-k-mooney | ill see if i can find it | 17:44 |
| mriedem | it would be nice to not duplicate the same checks all over the api and conductor... | 17:44 |
| aspiers | agreed | 17:44 |
| sean-k-mooney | ya | 17:44 |
| sean-k-mooney | sicne we know this just wont work for sev instace right now i think the api is better | 17:45 |
| aspiers | presumably better UX if it's done in the API? | 17:45 |
| sean-k-mooney | just flat out block it | 17:45 |
| sean-k-mooney | yes we will end up doing less work too | 17:45 |
| aspiers | right, so is e.g. nova.compute.api.API.live_migrate() the right place? | 17:45 |
| aspiers | using a decorator like mriedem suggested | 17:46 |
| sean-k-mooney | the problem with doing it in the api is when another dirver starts support sev then we need to remove it and move it to the condocor or somethwer else that can be aware of the hypervior type | 17:46 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start reporting PCPU inventory to placement https://review.opendev.org/671793 | 17:46 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: '_get_(v|p)cpu_total' to '_get_(v|p)cpu_available' https://review.opendev.org/672693 | 17:46 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: hardware: Differentiate between shared and dedicated CPUs https://review.opendev.org/671800 | 17:46 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start reporting 'HW_CPU_HYPERTHREADING' trait https://review.opendev.org/675571 | 17:46 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta https://review.opendev.org/671801 | 17:46 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add reshaper for PCPU https://review.opendev.org/674895 | 17:46 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Add 'InstanceNUMATopology.cpu_pinning' property https://review.opendev.org/680106 | 17:46 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Validate CPU config options against running instances https://review.opendev.org/680107 | 17:46 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Add 'NUMACell.pcpuset' field https://review.opendev.org/680108 | 17:46 |
| aspiers | sean-k-mooney: what other driver would start to support SEV? | 17:47 |
| stephenfin | mriedem: If you're interested, there's the startup check I was talking about earlier https://review.opendev.org/680107 | 17:47 |
| sean-k-mooney | hyperv maybe? vmware | 17:47 |
| sean-k-mooney | i mean its not a kvm tech it hsould be usable on other hyperviors | 17:47 |
| stephenfin | aspiers: Might be of interest to you, actually ^ | 17:47 |
| stephenfin | It's a similar thing to what you need, I suspect | 17:47 |
| stephenfin | sean-k-mooney can confirm | 17:47 |
| sean-k-mooney | aspiers: what i would do is write a check in hardware.py | 17:47 |
| stephenfin | though /me has to go | 17:47 |
| sean-k-mooney | then call that form the api | 17:47 |
| sean-k-mooney | and or conductor | 17:48 |
| aspiers | sean-k-mooney: there is already a check in hardware.py | 17:48 |
| sean-k-mooney | aspiers: well we use check in hardware.py from the api for other things | 17:48 |
| mriedem | fwiw i don't see any other virt driver supporting sev anytime soon | 17:48 |
| aspiers | mriedem: agreed | 17:49 |
| mriedem | the vmware driver is basically maintenance mode | 17:49 |
| mriedem | xen is on the way out | 17:49 |
| aspiers | sean-k-mooney: https://opendev.org/openstack/nova/src/branch/master/nova/virt/hardware.py#L1140 | 17:49 |
| mriedem | oh that reminds me, | 17:49 |
| mriedem | efried: should we formally deprecate the xen driver before releasing train? | 17:49 |
| dansmith | wow | 17:49 |
| mriedem | right now you just get a quality warning in the logs on startup | 17:49 |
| mriedem | dansmith: we talked about that at the ptg | 17:50 |
| mriedem | hasn't had CI in probably a year | 17:50 |
| dansmith | I know | 17:50 |
| sean-k-mooney | mriedem: didnt we say we would at the PTG | 17:50 |
| dansmith | it's just...a big and sad deal | 17:50 |
| sean-k-mooney | mriedem: well if nothing impove in train | 17:50 |
| mriedem | deprecating doesn't mean we remove in U | 17:50 |
| mriedem | just that we have the option | 17:50 |
| dansmith | yup | 17:50 |
| sean-k-mooney | mriedem: i dont think testing has impvoed so we proably sue | 17:51 |
| mriedem | we still need to nix nova-net | 17:51 |
| sean-k-mooney | shoudl | 17:51 |
| dansmith | deprecation should be a shot across the bow.. if nobody steps up at that point, then.. the signal is clear | 17:51 |
| mriedem | i'll put it on the meeting agenda for tomorrow so it's in the meeting logs | 17:51 |
| mriedem | and such | 17:51 |
| mriedem | such and such and so and so | 17:51 |
| sean-k-mooney | aspiers: ya so i would just call get_mem_encryption_constraint form the api and reject the live migration if its true | 17:52 |
| donnyd | hasn't nova-net been on the chopping block for the better half of a decade? | 17:52 |
| aspiers | sean-k-mooney: right, so is nova.compute.api.API.live_migrate() the right method to do the rejection? | 17:52 |
| mriedem | donnyd: deprecated in newton | 17:52 |
| aspiers | sean-k-mooney: and nova.compute.api.API.suspend()? | 17:52 |
| mriedem | donnyd: nova-net gets hard to remove when it's tendrils are in everything | 17:52 |
| mriedem | *its | 17:52 |
| donnyd | makes sense | 17:53 |
| melwitt | donnyd: did you ever figure out your slow image download problem? I had talked to penick about it and he said with the hardware specs you gave, download should be fast (like you were also thinking) | 17:53 |
| donnyd | no | 17:53 |
| donnyd | its still slow | 17:53 |
| melwitt | have you tried a simple scp from machine to machine to see if that's similarly slow? he suggested that | 17:53 |
| melwitt | to narrow down whether it's related to nova or not | 17:54 |
| *** ociuhandu has joined #openstack-nova | 17:54 | |
| dansmith | I haven't followed this issue, but an scp can be cpu bound and not max out a fast connection | 17:55 |
| dansmith | if it's really bad, then it going faster will tell you something, | 17:55 |
| dansmith | but if it's 10gE or something, generally iperf or straight netcat will help you go faste | 17:55 |
| dansmith | *faster | 17:56 |
| melwitt | oh, ok didn't know that | 17:56 |
| sean-k-mooney | aspiers: you could add a decorator and ecorate those metods but im not sure what the normal procedure would be | 17:56 |
| aspiers | sean-k-mooney: yes I'm writing a decorator as per mriedem's suggestion but I want to make sure I'm decorating the right methods :) | 17:56 |
| sean-k-mooney | for cyborge i think we chose not to do this | 17:56 |
| aspiers | sean-k-mooney: in the right file | 17:56 |
| mriedem | sean-k-mooney: there isn't really a normal procedure, | 17:57 |
| sean-k-mooney | we chose to jsut document what ws not supprotted | 17:57 |
| donnyd | scp openstack:/var/lib/glance/images/8dd685fd-277d-46fb-8d1c-b05ff7c5ecaa | 17:57 |
| donnyd | 8dd685fd-277d-46fb-8d1c-b05ff7c5ecaa 100% 6212MB 209.1MB/s 00:29 | 17:57 |
| mriedem | most times when we know shit doesn't work we just don't fix it, don't fail fast detect it, or document it | 17:57 |
| donnyd | so 2x faster than glance | 17:57 |
| mriedem | over time people eventually get tired of that so they put fail fast checks in place, like the things that don't work in the api with volume-backed servers | 17:57 |
| donnyd | but dansmith is correct. CPU bound on the controller side | 17:57 |
| sean-k-mooney | mriedem: yes so aspiers is breakign the mold by puting the check in place with the code | 17:58 |
| dansmith | donnyd: scp'ing from and to the same disks on each side presumably? | 17:58 |
| mriedem | sean-k-mooney: not really, like i said, we have "this doesn't work with this type of instance" checks in the API for certain things | 17:58 |
| *** ociuhandu has quit IRC | 17:58 | |
| mriedem | he's just doing it a bit differently with a decorator, it's the same idea though | 17:58 |
| sean-k-mooney | ya | 17:59 |
| sean-k-mooney | i understand that and that we do have check like this | 17:59 |
| sean-k-mooney | aspiers: so what are you going to return to the use | 17:59 |
| sean-k-mooney | a 4** | 17:59 |
| mriedem | the bw provider qos stuff fails fast in the api for move ops today too | 17:59 |
| sean-k-mooney | yes | 18:00 |
| sean-k-mooney | i was looking ot see if i could find that | 18:00 |
| donnyd | https://www.irccloud.com/pastebin/wQsy8Dyc/ | 18:00 |
| donnyd | not likely to be a network issue | 18:00 |
| mriedem | sean-k-mooney: for the bw provider case (supports_port_resource_request_during_move) we return a 400 | 18:00 |
| sean-k-mooney | for consitnece we proablyt want to fail the migration with the same respoce code | 18:00 |
| dansmith | donnyd: there are various chunk sizes for the glance backends, which may be too small by default for a fast network | 18:01 |
| sean-k-mooney | ok 400 bad request makes sense i guess | 18:01 |
| mriedem | i'm not sure that 400 was really correct there, 409 might have been better, but ... | 18:01 |
| *** tesseract has quit IRC | 18:01 | |
| aspiers | sean-k-mooney: I can't even find the place where migration chooses which error codes to return. There's no mention of migration in nova.api.openstack.compute.servers | 18:02 |
| sean-k-mooney | i was just looking at 409 | 18:02 |
| sean-k-mooney | https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/409 | 18:02 |
| donnyd | dansmith: using file backend because it performs the fastest | 18:02 |
| sean-k-mooney | i think we had this conversation for bandwidth too | 18:02 |
| aspiers | oh found it | 18:03 |
| dansmith | donnyd: yeah, there are chunk size knobs for several of them, filesystem included | 18:03 |
| aspiers | nova.api.openstack.compute.migrate_servers | 18:03 |
| dansmith | donnyd: 64k by default, apparently, which is pretty dang small | 18:03 |
| donnyd | Yea that is a little on the small side | 18:03 |
| dansmith | for 1gE that's probably okay, but not 10gE I'd think.. | 18:04 |
| sean-k-mooney | aspiers: right here https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/migrate_server.py#L61-L68 | 18:04 |
| donnyd | the controller side is 40G | 18:04 |
| aspiers | OK so I want the check to go in nova.api.openstack.compute.migrate_servers not nova.compute.api | 18:04 |
| donnyd | and each compute is 10 | 18:04 |
| dansmith | donnyd: the rbd default chunk is 8MiB, fwiw | 18:04 |
| sean-k-mooney | aspiers: or in your case you want the live_migrate check below it | 18:05 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/migrate_server.py#L127-L137 | 18:05 |
| dansmith | although it's not clear if that chunk size affects streaming like the filesystem one does | 18:05 |
| sean-k-mooney | aspiers: so that his proably the method you want to extend to call into hardware.py and check if sev is requested | 18:05 |
| mriedem | aspiers: i don't know that it matters really; i liked the decorator b/c you don't have to copy/paste the validation logic | 18:06 |
| mriedem | if you do it in nova.compute.api.API methods you'll have to handle whatever exception you raise so that the api doesn't return a 500 | 18:06 |
| aspiers | mriedem: got it | 18:06 |
| sean-k-mooney | mriedem: well you chould still use a decorator on thos funcitons | 18:06 |
| mriedem | sean-k-mooney: i'd rather not use a decorator on those | 18:07 |
| aspiers | yes, so decorate nova.compute.api.API and handle the new exception in nova.api.openstack.compute.migrate_server | 18:07 |
| mriedem | the decorator for wsgi stuff is really about wsgi | 18:07 |
| sean-k-mooney | ok | 18:07 |
| mriedem | the compute API methods already have decorators that do sniff tests | 18:07 |
| mriedem | like task_state, is locked, etc | 18:07 |
| aspiers | yeah I was copying those | 18:07 |
| aspiers | now we just have to spend 3 hours bikeshedding the new exception name | 18:07 |
| sean-k-mooney | aspiers: or just add a method to hardware.py that raise the correct excption | 18:08 |
| sean-k-mooney | then jsut call it in all the palces you need too | 18:08 |
| aspiers | hardware.py doesn't know what operation is being done | 18:08 |
| sean-k-mooney | most of the exception in hardware.py inherit form the http ones | 18:08 |
| sean-k-mooney | yes but you can call it form the api and pass in the oppertion if its needed | 18:09 |
| aspiers | that feels like mixing up layers to me | 18:09 |
| aspiers | I think I have enough hints to get it done now, thanks guys | 18:09 |
| sean-k-mooney | not sure i helped much | 18:09 |
| mriedem | stephenfin: ok, -1 :) | 18:10 |
| aspiers | sean-k-mooney: ;-) | 18:10 |
| *** markvoelker has quit IRC | 18:12 | |
| *** markvoelker has joined #openstack-nova | 18:12 | |
| openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider Config File: YAML file loading and schema validation https://review.opendev.org/673341 | 18:13 |
| openstackgerrit | Dustin Cowles proposed openstack/nova master: WIP: Provider Config File: Public method to retrieve custom resources https://review.opendev.org/676029 | 18:13 |
| openstackgerrit | Dustin Cowles proposed openstack/nova master: WIP: Provider Config File: Update provider tree with new custom resources https://review.opendev.org/676522 | 18:13 |
| sean-k-mooney | dansmith: artom: so live migration form (master) src node to (artoms code) dest node with unpinned or pinned rpc | 18:13 |
| sean-k-mooney | appears to work the same as if we pin to the old rpc version | 18:14 |
| sean-k-mooney | e.g. no xml updates | 18:14 |
| sean-k-mooney | and you can live migrate it back too | 18:15 |
| mriedem | here it comes | 18:16 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_dest compute method https://review.opendev.org/633293 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtDestTask https://review.opendev.org/627890 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_source compute method https://review.opendev.org/634832 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtSourceTask https://review.opendev.org/627891 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add finish_snapshot_based_resize_at_dest compute method https://review.opendev.org/635080 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add FinishResizeAtDestTask https://review.opendev.org/635646 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Execute CrossCellMigrationTask from MigrationTask https://review.opendev.org/635668 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Plumb allow_cross_cell_resize into compute API resize() https://review.opendev.org/635684 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Filter duplicates from compute API get_migrations_sorted() https://review.opendev.org/636224 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Start functional testing for cross-cell resize https://review.opendev.org/636253 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle target host cross-cell cold migration in conductor https://review.opendev.org/642591 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Validate image/create during cross-cell resize functional testing https://review.opendev.org/642592 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add zones wrinkle to TestMultiCellMigrate https://review.opendev.org/643450 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add negative test for cross-cell finish_resize failing https://review.opendev.org/643451 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Refresh instance in MigrationTask.execute Exception handler https://review.opendev.org/669012 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add negative test for prep_snapshot_based_resize_at_source failing https://review.opendev.org/669013 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add confirm_snapshot_based_resize_at_source compute method https://review.opendev.org/637058 | 18:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add ConfirmResizeTask https://review.opendev.org/637070 | 18:17 |
| sean-k-mooney | while its on the dest node with the invalid pinning we get the message we expect in the update resouce providers http://paste.openstack.org/show/770652/ whic is the same behavior as master today | 18:19 |
| sean-k-mooney | dansmith: artom im going to lie down and take some panadol and hopefully my headache will go away. let me know if i need to test anything else and i will do it tomorrow | 18:21 |
| dansmith | sean-k-mooney: cool, thanks for these confirmations... | 18:22 |
| donnyd | sean-k-mooney: using 2x the ceph defaults to start | 18:22 |
| donnyd | we will see how it goes | 18:22 |
| sean-k-mooney | 2x? | 18:22 |
| dansmith | was that intended for me? | 18:23 |
| melwitt | probably | 18:23 |
| *** mvkr has quit IRC | 18:24 | |
| sean-k-mooney | oh the rbd defualt chunk size | 18:24 |
| sean-k-mooney | ya that was for dan | 18:24 |
| mriedem | artom: can this be abandoned now? https://review.opendev.org/#/c/634605/ | 18:25 |
| mriedem | your series is kind of hard to follow in gerrit | 18:25 |
| mriedem | especially when the topic branch no longer matches the bp name | 18:25 |
| artom | mriedem, done | 18:25 |
| sean-k-mooney | i think this is the new topic branch https://review.opendev.org/#/q/topic:bp/numa-aware-live-migration-patch27+(status:open+OR+status:merged) | 18:25 |
| artom | mriedem, wtf, where did '-patch27' come from? | 18:25 |
| artom | sean-k-mooney, for your tests? | 18:26 |
| mriedem | artom: guessing you were doing some weird local feature branch rebasing | 18:26 |
| sean-k-mooney | no | 18:26 |
| artom | I didn't do it intentionally | 18:26 |
| sean-k-mooney | i mean that sames to be what most of the new ones are tagged with | 18:26 |
| mriedem | is anything using this? https://review.opendev.org/#/c/679319/ | 18:26 |
| sean-k-mooney | this is the last patch https://review.opendev.org/#/c/672595/ | 18:27 |
| artom | mriedem, nope, abandon away | 18:27 |
| sean-k-mooney | so that is the 7 pathces theat matter | 18:27 |
| sean-k-mooney | i guess 6 pathces. | 18:27 |
| mriedem | and https://review.opendev.org/#/c/679656/ is the thing testing it? | 18:28 |
| * sean-k-mooney really goes to lie down now | 18:28 | |
| melwitt | I've been wondering what happened to the automatic topic naming in gerrit. it was nice when it used to do that | 18:28 |
| mriedem | which is...on a different topic branch | 18:28 |
| mriedem | so hard to find... | 18:28 |
| sean-k-mooney | mriedem: feel free to change it | 18:28 |
| dansmith | melwitt: that is a function of git-review I think | 18:28 |
| mriedem | https://review.opendev.org/#/q/topic:bp/numa-aware-live-migration+(status:open+OR+status:merged)+project:openstack/nova | 18:29 |
| melwitt | hm, I didn't think I had changed git-review versions. I will check that | 18:29 |
| mriedem | looking better | 18:29 |
| *** brault has joined #openstack-nova | 18:29 | |
| dansmith | melwitt: see the -t section, which says "if not specified, a bug number from the commit summary will be used" https://linux.die.net/man/1/git-review | 18:30 |
| melwitt | thank you | 18:30 |
| mriedem | is that only if you're on the master branch? | 18:31 |
| mriedem | i'm always in a topic branch | 18:31 |
| artom | mriedem, cleaned it up a bit, sorry for the mess | 18:31 |
| dansmith | mriedem: no | 18:31 |
| dansmith | mriedem: bug takes precedence, otherwise local branch name | 18:31 |
| melwitt | I'm not sure what happened but in the past, I never had to worry about my random topic branch name getting pushed to gerrit (as long as I have a bug or bp) but lately I noticed it's not adjusting it anymore | 18:31 |
| dansmith | I never submit from the master branch locally.. | 18:31 |
| mriedem | same, ok i guess i just didn't notice since i'm always using bug/xxx or bp/xxx | 18:32 |
| sean-k-mooney | it will still take the topic if you do git checkout -b whatever | 18:33 |
| dansmith | sean-k-mooney: only if there is no other tag in the commit message | 18:33 |
| dansmith | sean-k-mooney: go lie down :) | 18:33 |
| sean-k-mooney | a recent version of git-review removed some of the auto topic stuff | 18:33 |
| sean-k-mooney | ya i will but stephenfin change how this works recently | 18:34 |
| *** brault has quit IRC | 18:34 | |
| melwitt | aw man | 18:34 |
| sean-k-mooney | i think they weanted to remvoe the auto topic form the commit message | 18:34 |
| melwitt | I need to downgrade then probably | 18:34 |
| sean-k-mooney | but leave the auto topic form branch name | 18:34 |
| mriedem | i thought sean-k-mooney was going to lie down? | 18:34 |
| melwitt | git-review version 1.28.0 | 18:34 |
| sean-k-mooney | ok o/ | 18:34 |
| mriedem | now that he's gone, | 18:35 |
| dansmith | https://opendev.org/opendev/git-review/commit/af955c932eb97433199781b5d4f7fb5fb5af8a6f | 18:35 |
| dansmith | https://opendev.org/opendev/git-review/commit/03768832c4a73c3079ce24d84fecae1da39b2d71 | 18:35 |
| * mriedem can't come up with good exotic hardware joke | 18:35 | |
| dansmith | "frequently does the wrong thing" ... not that I ever saw. | 18:35 |
| melwitt | yeah like, what? | 18:36 |
| dansmith | melwitt: that second commit is likely what you've noticed I guess | 18:36 |
| melwitt | stephenfin you're killin me | 18:36 |
| melwitt | and yeah, I think you're right dansmith | 18:36 |
| melwitt | I'm gonna downgrade | 18:37 |
| dansmith | the story links in that commit are broken, | 18:37 |
| dansmith | and no gerrit links so I dunno what kind of review that stuff gets | 18:37 |
| melwitt | fun | 18:37 |
| dansmith | I pretty much always name my local branches with bug/X and bp/Y anyway, so I probably wouldn't notice | 18:38 |
| dansmith | because I actually don't like that I end up with them different in gerrit vs. local | 18:38 |
| melwitt | I'm a renegade | 18:38 |
| melwitt | https://en.wikipedia.org/wiki/Renegade_(TV_series) | 18:39 |
| mriedem | was just thinking that | 18:39 |
| mriedem | i had a friend that couldn't leave the house all summer in middle school because his parents made him babysit his sister, | 18:39 |
| mriedem | and he'd be watching renegade every day | 18:39 |
| melwitt | lol, an excellent choice | 18:40 |
| mriedem | so we had to go to his house b/c he coudn't leave, but then we had to watch renegade | 18:40 |
| mriedem | who is lorenzo roundhouse kicking today? | 18:42 |
| melwitt | heh | 18:43 |
| *** BjoernT_ has quit IRC | 18:46 | |
| *** kaisers has quit IRC | 18:49 | |
| *** ricolin has quit IRC | 18:50 | |
| zbr | efried: does https://review.opendev.org/#/c/674057/ look ok now? | 18:54 |
| *** irclogbot_2 has quit IRC | 18:54 | |
| *** irclogbot_0 has joined #openstack-nova | 18:55 | |
| *** ralonsoh has quit IRC | 18:55 | |
| artom | I'm a dumbass | 18:59 |
| artom | I spent literally hours, at this point, trying to stub, mock, or hack __dict__ to pin a compute RPC object | 18:59 |
| artom | (Because def router is an @property that can't be set/stubbed, and it's using a global singleton) | 19:00 |
| *** amrith has joined #openstack-nova | 19:00 | |
| artom | When what I *should* have done is just inherited from ComputeAPI and overwrote def router >_< | 19:00 |
| *** irclogbot_0 has quit IRC | 19:02 | |
| *** kaliya has joined #openstack-nova | 19:02 | |
| dansmith | can't you just kill router on the instance that each one gets? | 19:02 |
| dansmith | service.manager.rpcapi.router = mock.Mock() ? | 19:03 |
| dansmith | or whatever | 19:03 |
| *** irclogbot_3 has joined #openstack-nova | 19:03 | |
| artom | @property makes it un-settable | 19:03 |
| dansmith | can you delattr it? | 19:04 |
| artom | Huh, didn't think to try that | 19:04 |
| dansmith | anyway, doesn't matter | 19:04 |
| *** amrith has quit IRC | 19:12 | |
| *** efried has quit IRC | 19:18 | |
| *** efried has joined #openstack-nova | 19:20 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | 19:30 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | 19:30 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | 19:30 |
| artom | dansmith, ^^ | 19:31 |
| artom | (FWIW, I had the -patch27 in my local branch name for some reason, fixed that) | 19:32 |
| * artom picks up daughter from school, back in a few | 19:32 | |
| dansmith | artom: one more thing I didn't catch last time, but it's easy | 19:36 |
| dansmith | artom: otherwise I think it looks good, probably time for mriedem to have a go at it | 19:36 |
| *** kaisers has joined #openstack-nova | 19:37 | |
| aspiers | can anyone point me at where the fake server instance object comes from in https://github.com/openstack/nova/blob/master/nova/tests/unit/api/openstack/compute/test_migrate_server.py#L579 ? | 19:41 |
| aspiers | I've trawled through the test setup code and can't find it | 19:41 |
| aspiers | ah | 19:43 |
| aspiers | admin_only_action_common.py | 19:43 |
| *** gbarros has quit IRC | 19:44 | |
| *** gbarros has joined #openstack-nova | 19:46 | |
| *** ociuhandu has joined #openstack-nova | 19:47 | |
| *** eharney has quit IRC | 19:51 | |
| aspiers | still don't get it | 19:53 |
| aspiers | if I use _stub_instance_get() then I get "Exception: This test uses methods that set internal oslo_db state, but it does not claim to use the database. This will conflict with the setup of tests that do use the database and cause failures later." | 19:53 |
| melwitt | aspiers: is there a link to a patch you're working on that will give more context? my first guess is maybe your new test is accessing an attribute on instance that isn't populated, so it's trying to lazy-load it from the database, which is flagging your test as USES_DB = True | 19:59 |
| openstackgerrit | Merged openstack/nova master: Remove unused methods https://review.opendev.org/679918 | 19:59 |
| *** BjoernT has joined #openstack-nova | 19:59 | |
| aspiers | melwitt: I'll upload | 20:00 |
| melwitt | that error comes from a test inheriting from NoDBTestCase and then accessing the database | 20:01 |
| aspiers | melwitt: does this test avoid it by patching out the meat of the code path (i.e. resize)? https://github.com/openstack/nova/blob/master/nova/tests/unit/api/openstack/compute/test_migrate_server.py#L124 | 20:05 |
| aspiers | so it's effectively only testing the HTTP stuff in the top layer? | 20:05 |
| melwitt | probably, that stub will fill in most of the instance objects attributes. so if anything in the code path accesses the attributes, they are already filled in, | 20:07 |
| melwitt | with objects, if an attribute is *not* filled in and it is accessed, the object will fall back on a read from the database to get the value | 20:07 |
| openstackgerrit | Merged openstack/nova master: Convert nova-lvm job to zuul v3 https://review.opendev.org/674345 | 20:08 |
| melwitt | so it will test the full code path, just not the database accesses (if there were going to be any) | 20:08 |
| efried | sean-k-mooney: in case this doesn't show up in your inbox/dashboard/whatever, would you please respond to this os-vif release patch: https://review.opendev.org/#/c/680094/ | 20:08 |
| aspiers | melwitt: ahhh maybe that's it | 20:09 |
| *** ociuhandu has quit IRC | 20:09 | |
| *** ociuhandu has joined #openstack-nova | 20:10 | |
| aspiers | melwitt: maybe it's missing the flavor or image_meta | 20:10 |
| melwitt | aspiers: so sometimes what happens is you're working on a new test whose code path will access an attribute that's not stubbed out/filled in, and then it falls back on a read from the db, and if you're inherited from NoDBTestCase, that error will raise | 20:10 |
| aspiers | no, just adding a call to self._stub_instance_get() is enough to trigger this db-related failure, *even* if I don't use the returned fake instance | 20:11 |
| aspiers | so it's not my test code which is triggering the error | 20:11 |
| melwitt | yeah if you look in https://github.com/openstack/nova/blob/master/nova/tests/unit/fake_instance.py#L45 you can see what's been filled in | 20:11 |
| aspiers | yeah I spotted that | 20:11 |
| melwitt | oh, hm | 20:12 |
| aspiers | but this makes no sense, because other tests happily call that same method | 20:12 |
| aspiers | melwitt: uploading now | 20:13 |
| openstackgerrit | Adam Spiers proposed openstack/nova master: Reject live migration and suspend on SEV guests https://review.opendev.org/680158 | 20:13 |
| aspiers | there ^^^ | 20:13 |
| aspiers | melwitt: https://review.opendev.org/#/c/680158/1/nova/tests/unit/api/openstack/compute/test_migrate_server.py@592 causes this db error | 20:14 |
| aspiers | why do other identical lines in the same file not cause it? | 20:14 |
| melwitt | at a glance, I notice the other tests calling _stub_instance_get are also mocking the live_migrate method in compute_api | 20:16 |
| aspiers | right, that's what I was saying earlier | 20:16 |
| aspiers | they mock the meat of the code path | 20:16 |
| *** aojea has joined #openstack-nova | 20:17 | |
| aspiers | but the call to _stub_instance_get() isn't inside the mocking?! | 20:17 |
| aspiers | so it can't be that | 20:17 |
| aspiers | super weird | 20:17 |
| openstackgerrit | Adam Spiers proposed openstack/nova master: Reject live migration and suspend on SEV guests https://review.opendev.org/680158 | 20:18 |
| melwitt | you're saying the _stub_instance_get call itself will not return and raises that error? | 20:18 |
| aspiers | yes | 20:18 |
| melwitt | O.o | 20:18 |
| melwitt | ok, that is unexpected | 20:18 |
| aspiers | oh no wait | 20:19 |
| aspiers | OK I get it now | 20:19 |
| aspiers | _stub_instance_get() has side effects of course | 20:19 |
| aspiers | it patches stuff | 20:19 |
| aspiers | so it doesn't matter if I don't use the return value | 20:19 |
| aspiers | yeah it's the lookup of instance.image_meta which fails | 20:20 |
| mriedem | dansmith: artom: ack, probably going to have to be in the morning at this point, just got off a call | 20:21 |
| *** markvoelker has quit IRC | 20:21 | |
| melwitt | aspiers: where's the lookup? I'm curious | 20:21 |
| aspiers | melwitt: https://review.opendev.org/#/c/680158/2/nova/compute/api.py@218 | 20:22 |
| aspiers | melwitt: I think my mocking of that hardware method is broken probably | 20:22 |
| aspiers | mocking in the wrong place is my guess | 20:22 |
| aspiers | need to mock in nova.compute.api | 20:22 |
| melwitt | ok, I thought you said _stub_instance_get was blowing up itself | 20:22 |
| aspiers | yeah I was wrong | 20:22 |
| melwitt | and I was curious where in there it accesses db | 20:23 |
| melwitt | ok | 20:23 |
| aspiers | ohh doh | 20:24 |
| aspiers | patching hardware.get_mem_encryption_constraint doesn't stop the calculation of the parameters being passed to it | 20:25 |
| aspiers | so I have to patch instance.image_meta too, or just set it | 20:25 |
| *** aojea has quit IRC | 20:25 | |
| *** markvoelker has joined #openstack-nova | 20:30 | |
| dansmith | mriedem: np | 20:31 |
| *** nweinber has quit IRC | 20:37 | |
| *** ociuhandu has quit IRC | 20:38 | |
| aspiers | melwitt: got it working finally | 20:41 |
| aspiers | melwitt: thanks for your help :) | 20:41 |
| melwitt | yay | 20:41 |
| openstackgerrit | Merged openstack/nova master: Modernize nova-lvm job https://review.opendev.org/674356 | 20:43 |
| *** dannins has quit IRC | 20:45 | |
| *** bbowen_ has quit IRC | 20:45 | |
| *** bbowen_ has joined #openstack-nova | 20:45 | |
| openstackgerrit | Adam Spiers proposed openstack/nova master: Reject live migration and suspend on SEV guests https://review.opendev.org/680158 | 20:45 |
| aspiers | melwitt: https://review.opendev.org/#/c/680158/2..3/nova/tests/unit/api/openstack/compute/test_suspend_server.py@47 | 20:46 |
| openstackgerrit | Adam Spiers proposed openstack/nova master: Enable booting of libvirt guests with AMD SEV memory encryption https://review.opendev.org/666616 | 20:47 |
| melwitt | aspiers: oh, so you didn't need to stub compute_api | 20:48 |
| aspiers | melwitt: right, it was just image_meta | 20:48 |
| aspiers | I didn't want to stub compute_api because that's where the desired exception gets thrown | 20:48 |
| melwitt | yeah, makes sense | 20:49 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: FUP for I66d8f06f19c5c631e33208580428aa843abb38d2 https://review.opendev.org/678951 | 20:55 |
| aspiers | oh crap, circular imports | 20:57 |
| *** munimeha1 has joined #openstack-nova | 21:00 | |
| aspiers | what's the ASCII smiley for a facepalm? | 21:01 |
| *** eharney has joined #openstack-nova | 21:07 | |
| *** slaweq has quit IRC | 21:08 | |
| aspiers | efried: make sense if I move sev_enabled() from libvirt.utils to virt.hardware to break the circular import? | 21:09 |
| efried | aspiers: without looking specifically at the patch, that sounds like a reasonable thing. | 21:10 |
| aspiers | I don't think it's possible to move get_mem_encryption_constraint() and its helpers in the other direction | 21:10 |
| aspiers | which is a shame, cause they're libvirt-specific | 21:10 |
| efried | except that I still resent that nova.virt.hardware is libvirt-specific | 21:10 |
| aspiers | right | 21:10 |
| efried | yeah | 21:10 |
| aspiers | thing is, they depend _get_flavor_image_meta which is shared | 21:10 |
| efried | that ship has sailed though. | 21:10 |
| aspiers | OK | 21:10 |
| aspiers | efried: I suggested to kashyap yesterday that we actually move the machine type stuff to a separate machine_types file | 21:11 |
| *** slaweq has joined #openstack-nova | 21:11 | |
| aspiers | libvirt.utils is a kinda dumb bag of random shit | 21:11 |
| efried | if you're just grasping for shit to do, you could factor the libvirt-specific stuff out of nova.virt.hardware and into a nova.virt.libvirt.hardware module, leaving anything that's actually generic (assuming there is anything) in nova.virt.hardware. | 21:11 |
| * aspiers LOLs at the thought of grasping for shit to do | 21:12 | |
| efried | but we probably have more important things to do | 21:12 |
| aspiers | but I dunno, maybe I can | 21:12 |
| aspiers | however I'm very conscious that once this runway window closes, that's my last chance to land SEV in Train gone | 21:12 |
| aspiers | anyway yet again it's 10pm and I haven't had dinner | 21:13 |
| aspiers | time to tear myself away | 21:13 |
| aspiers | efried: parting thought - once I fix the circular import, in *theory* the patch series is code complete (haha how many times have I said that before) | 21:14 |
| efried | aspiers: That refactor would have nothing to do with the SEV series, runways, or indeed probably Train, as we would likely prefer to wait until ussuri opens to land it. | 21:15 |
| aspiers | efried: cool | 21:15 |
| *** spatel has quit IRC | 21:15 | |
| aspiers | what do you think about the machine_types.py idea? | 21:15 |
| *** slaweq has quit IRC | 21:15 | |
| aspiers | nova.virt.machine_types | 21:16 |
| aspiers | I don't even know if it's libvirt-specific actually | 21:16 |
| aspiers | I think not | 21:16 |
| aspiers | but specific to QEMU so ... | 21:16 |
| aspiers | yeah, qemu-kvm -machine help lists them all | 21:17 |
| * aspiers goes | 21:18 | |
| *** psachin has joined #openstack-nova | 21:23 | |
| donnyd | melwitt: dansmith booya | 21:24 |
| donnyd | https://usercontent.irccloud-cdn.com/file/OdTlIVXX/image.png | 21:24 |
| dansmith | donnyd: chunk size or something else? | 21:24 |
| donnyd | chunk size numbers are looking much mo betta | 21:24 |
| * melwitt cues the hallelujah song | 21:24 | |
| dansmith | sweet | 21:24 |
| dansmith | donnyd: let me get you an address where you can send the check... | 21:24 |
| donnyd | LOL | 21:25 |
| donnyd | 5x faster | 21:26 |
| donnyd | that is a pretty big win | 21:26 |
| melwitt | yeah, that's awesome. I didn't know about glance chunk size until today | 21:27 |
| dansmith | donnyd: that was with 16MiB chunks? | 21:28 |
| donnyd | yea | 21:28 |
| dansmith | cool | 21:28 |
| donnyd | not sure if I should turn it up anymore or not | 21:28 |
| dansmith | higher will increase memory usage on the controllers of course | 21:29 |
| dansmith | by at least $chunk_size per parallel download thread I imagine | 21:29 |
| donnyd | I think its probably smart to left this settle in for a day or so | 21:30 |
| donnyd | I should know in the next 24 hours | 21:31 |
| dansmith | yeah | 21:31 |
| *** mdbooth has quit IRC | 21:39 | |
| *** takashin has joined #openstack-nova | 21:41 | |
| *** mdbooth has joined #openstack-nova | 21:46 | |
| *** markvoelker has quit IRC | 21:47 | |
| *** munimeha1 has quit IRC | 21:50 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Validate CPU config options against running instances https://review.opendev.org/680107 | 21:52 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Add 'NUMACell.pcpuset' field https://review.opendev.org/680108 | 21:52 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: hardware: Differentiate between shared and dedicated CPUs https://review.opendev.org/671800 | 21:52 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start reporting 'HW_CPU_HYPERTHREADING' trait https://review.opendev.org/675571 | 21:52 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta https://review.opendev.org/671801 | 21:52 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add reshaper for PCPU https://review.opendev.org/674895 | 21:52 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Use sane indent https://review.opendev.org/680229 | 21:52 |
| *** edleafe has left #openstack-nova | 21:55 | |
| *** eharney has quit IRC | 21:56 | |
| donnyd | dansmith: I couldn't resist | 22:05 |
| donnyd | 32m is too high | 22:05 |
| donnyd | 16m is the sweet spot, and if you are curious dansmith melwitt I can show the data | 22:05 |
| *** trident has quit IRC | 22:05 | |
| donnyd | https://grafana.fortnebula.com/d/9MMqh8HWk/openstack-utilization?orgId=2&refresh=30s&from=now-1h&to=now&fullscreen&panelId=14 | 22:06 |
| dansmith | donnyd: you went below 16m too I assume? | 22:06 |
| donnyd | No, I should probably test that direction as well | 22:06 |
| dansmith | yeah | 22:06 |
| melwitt | cool donnyd, that's good to know. I've already told penick about it and he's gonna check their settings too :) | 22:08 |
| donnyd | That link i just sent has the data to show the difference in both upload and download speeds at different chunk sizes | 22:09 |
| *** mriedem has quit IRC | 22:11 | |
| *** slaweq has joined #openstack-nova | 22:11 | |
| donnyd | maybe I should note that in the glance docs or something | 22:11 |
| *** trident has joined #openstack-nova | 22:14 | |
| donnyd | maximum d/l speed at 32m is 3Gs / 16m is 8G/s /8m 6.5G/s | 22:15 |
| donnyd | so it looks like 16m is the sweet spot for at least my infra | 22:15 |
| *** slaweq has quit IRC | 22:15 | |
| donnyd | Thanks again.. this was really the only thing i was unhappy with on FN... | 22:16 |
| donnyd | woot woot!!! | 22:16 |
| melwitt | yay | 22:16 |
| donnyd | https://grafana.fortnebula.com/d/9MMqh8HWk/openstack-utilization?orgId=2&from=1567633743742&to=1567635376855&fullscreen&panelId=14 | 22:16 |
| melwitt | I'm glad to know what the issue was. I've been wondering about it | 22:16 |
| donnyd | and here is 32-8 left to right | 22:16 |
| donnyd | me too | 22:17 |
| donnyd | LOL melwitt | 22:17 |
| melwitt | haha xD | 22:17 |
| donnyd | Its that dinner time around here | 22:17 |
| * donnyd goes to eat dinner with a giant grin | 22:17 | |
| melwitt | o/ | 22:17 |
| *** mlavalle has quit IRC | 22:21 | |
| *** jhesketh has quit IRC | 22:27 | |
| aspiers | efried: hmm, other option for breaking the cycle is to move get_machine_type() from utils to new machine_types.py | 22:30 |
| aspiers | maybe that's better | 22:31 |
| *** psachin has quit IRC | 22:31 | |
| aspiers | (than moving libvirt-specific sev_enabled() to virt.hardware) | 22:31 |
| *** jhesketh has joined #openstack-nova | 22:33 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (13) https://review.opendev.org/576020 | 22:35 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (14) https://review.opendev.org/576027 | 22:35 |
| *** markvoelker has joined #openstack-nova | 22:35 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (15) https://review.opendev.org/576031 | 22:37 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (16) https://review.opendev.org/576299 | 22:37 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (17) https://review.opendev.org/576344 | 22:37 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (18) https://review.opendev.org/576673 | 22:38 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (19) https://review.opendev.org/576676 | 22:38 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (20) https://review.opendev.org/576689 | 22:38 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (21) https://review.opendev.org/576709 | 22:38 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (22) https://review.opendev.org/576712 | 22:39 |
| *** markvoelker has quit IRC | 22:40 | |
| *** KeithMnemonic has quit IRC | 22:43 | |
| *** KeithMnemonic1 has joined #openstack-nova | 22:43 | |
| *** macz has quit IRC | 22:56 | |
| *** tkajinam has joined #openstack-nova | 22:58 | |
| *** rcernin has joined #openstack-nova | 23:04 | |
| *** luksky has quit IRC | 23:09 | |
| *** slaweq has joined #openstack-nova | 23:11 | |
| *** slaweq has quit IRC | 23:16 | |
| *** threestrands has joined #openstack-nova | 23:17 | |
| openstackgerrit | Nathan Kinder proposed openstack/nova master: Allow TLS ciphers/protocols to be configurable for console proxies https://review.opendev.org/679502 | 23:30 |
| *** macz has joined #openstack-nova | 23:32 | |
| *** sapd1_x has joined #openstack-nova | 23:34 | |
| *** ozzzo has joined #openstack-nova | 23:34 | |
| *** macz has quit IRC | 23:50 | |
| *** trident has quit IRC | 23:52 | |
| *** BjoernT has quit IRC | 23:53 | |
| *** BjoernT has joined #openstack-nova | 23:53 | |
| *** BjoernT has quit IRC | 23:54 | |
| *** BjoernT_ has joined #openstack-nova | 23:54 | |
| brinzhang | mriedem: pretty good, thanks | 23:54 |
| *** BjoernT_ has quit IRC | 23:54 | |
| *** BjoernT has joined #openstack-nova | 23:55 | |
| *** BjoernT has quit IRC | 23:55 | |
| *** BjoernT_ has joined #openstack-nova | 23:55 | |
| *** BjoernT_ has quit IRC | 23:56 | |
| *** BjoernT has joined #openstack-nova | 23:56 | |
| *** BjoernT has quit IRC | 23:57 | |
| *** BjoernT_ has joined #openstack-nova | 23:57 | |
| *** BjoernT_ has quit IRC | 23:58 | |
| *** BjoernT has joined #openstack-nova | 23:58 | |
| *** BjoernT has quit IRC | 23:58 | |
| *** BjoernT_ has joined #openstack-nova | 23:59 | |
| *** BjoernT_ has quit IRC | 23:59 | |
| *** BjoernT has joined #openstack-nova | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
