| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (22) https://review.opendev.org/576712 | 00:16 |
|---|---|---|
| *** markvoelker has joined #openstack-nova | 00:20 | |
| *** markvoelker has quit IRC | 00:25 | |
| *** itlinux_ has joined #openstack-nova | 00:46 | |
| *** itlinux has quit IRC | 00:49 | |
| *** hongbin has joined #openstack-nova | 00:51 | |
| *** yedongcan has joined #openstack-nova | 00:55 | |
| *** brinzhang has joined #openstack-nova | 00:57 | |
| *** BjoernT has joined #openstack-nova | 01:25 | |
| *** BjoernT has quit IRC | 01:37 | |
| *** sapd1 has joined #openstack-nova | 01:42 | |
| *** larainema has joined #openstack-nova | 01:49 | |
| *** gbarros has quit IRC | 01:59 | |
| openstackgerrit | zhufl proposed openstack/nova master: [Trivial]Remove unused helper get_vm_ref_from_name https://review.opendev.org/678444 | 02:00 |
| *** BjoernT has joined #openstack-nova | 02:03 | |
| openstackgerrit | zhufl proposed openstack/nova master: [Trivial]Remove unused helper _get_min_service_version https://review.opendev.org/678446 | 02:15 |
| *** zhubx has quit IRC | 02:17 | |
| *** boxiang has joined #openstack-nova | 02:18 | |
| *** boxiang has quit IRC | 02:22 | |
| *** boxiang has joined #openstack-nova | 02:22 | |
| openstackgerrit | ya.wang proposed openstack/nova master: vCPU model selection https://review.opendev.org/670298 | 02:38 |
| openstackgerrit | ya.wang proposed openstack/nova master: Add compatibility checks for CPU mode and CPU models and extra flags https://review.opendev.org/670299 | 02:38 |
| openstackgerrit | ya.wang proposed openstack/nova master: Support report multi CPU model traits https://review.opendev.org/670300 | 02:38 |
| *** zhubx has joined #openstack-nova | 02:38 | |
| *** boxiang has quit IRC | 02:40 | |
| *** masayukig has joined #openstack-nova | 02:41 | |
| *** sapd1 has quit IRC | 02:54 | |
| *** sapd1_x has joined #openstack-nova | 02:54 | |
| *** markvoelker has joined #openstack-nova | 02:55 | |
| openstackgerrit | Bhagyashri Shewale proposed openstack/nova master: tests: Split NUMA object tests https://review.opendev.org/672336 | 03:00 |
| openstackgerrit | Bhagyashri Shewale proposed openstack/nova master: scheduler: Flatten 'ResourceRequest.from_extra_specs', 'from_image_props' https://review.opendev.org/674894 | 03:00 |
| openstackgerrit | Bhagyashri Shewale proposed openstack/nova master: objects: Remove legacy '_from_dict' functions https://review.opendev.org/537414 | 03:00 |
| openstackgerrit | Bhagyashri Shewale proposed openstack/nova master: Remove 'hw:cpu_policy', 'hw:mem_page_size' extra specs from API samples https://review.opendev.org/675338 | 03:00 |
| openstackgerrit | Bhagyashri Shewale proposed openstack/nova master: libvirt: Start reporting PCPU inventory to placement https://review.opendev.org/671793 | 03:00 |
| openstackgerrit | Bhagyashri Shewale proposed openstack/nova master: libvirt: '_get_(v|p)cpu_total' to '_get_(v|p)cpu_available' https://review.opendev.org/672693 | 03:00 |
| openstackgerrit | Bhagyashri Shewale proposed openstack/nova master: trivial: Rewrap definitions of 'NUMACell' https://review.opendev.org/674395 | 03:00 |
| openstackgerrit | Bhagyashri Shewale proposed openstack/nova master: hardware: Differentiate between shared and dedicated CPUs https://review.opendev.org/671800 | 03:00 |
| openstackgerrit | Bhagyashri Shewale proposed openstack/nova master: objects: Rename 'fields' import to 'obj_fields' https://review.opendev.org/674103 | 03:00 |
| openstackgerrit | Bhagyashri Shewale proposed openstack/nova master: libvirt: Start reporting 'HW_CPU_HYPERTHREADING' trait https://review.opendev.org/675571 | 03:00 |
| openstackgerrit | Bhagyashri Shewale proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta https://review.opendev.org/671801 | 03:00 |
| openstackgerrit | Bhagyashri Shewale proposed openstack/nova master: libvirt: Fold in argument to '_update_provider_tree_for_vgpu' https://review.opendev.org/676729 | 03:00 |
| openstackgerrit | Bhagyashri Shewale proposed openstack/nova master: Add reshaper for PCPU https://review.opendev.org/674895 | 03:00 |
| *** markvoelker has quit IRC | 03:00 | |
| *** yaawang has quit IRC | 03:03 | |
| *** yaawang has joined #openstack-nova | 03:03 | |
| *** boxiang has joined #openstack-nova | 03:03 | |
| *** zhubx has quit IRC | 03:06 | |
| *** cervigni has joined #openstack-nova | 03:08 | |
| *** yedongcan has quit IRC | 03:08 | |
| cervigni | is there anyone online now that can help with VGPU scheduling? | 03:10 |
| *** rcernin_ has joined #openstack-nova | 03:15 | |
| *** rcernin has quit IRC | 03:15 | |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: db: Add resources column in instance_extra table https://review.opendev.org/678447 | 03:25 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: object: Introduce Resource and ResouceList objs https://review.opendev.org/678448 | 03:25 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Add resources dict into _Provider https://review.opendev.org/678449 | 03:25 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Retrive the allocations early https://review.opendev.org/678450 | 03:25 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Track orphan instances and error migrations in resource tracker https://review.opendev.org/678451 | 03:25 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Claim resources in resource tracker https://review.opendev.org/678452 | 03:25 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/678453 | 03:25 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: report VPMEM resources by provider tree https://review.opendev.org/678454 | 03:25 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Support VM creation with vpmems and vpmems cleanup https://review.opendev.org/678455 | 03:25 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456 | 03:25 |
| *** brinzhang has quit IRC | 03:26 | |
| *** brinzhang has joined #openstack-nova | 03:27 | |
| *** BjoernT has quit IRC | 03:39 | |
| *** ircuser-1 has quit IRC | 03:41 | |
| *** hamzy_ has joined #openstack-nova | 03:48 | |
| *** hamzy has quit IRC | 03:51 | |
| *** mkrai has joined #openstack-nova | 03:56 | |
| *** yedongcan has joined #openstack-nova | 04:01 | |
| *** hongbin has quit IRC | 04:02 | |
| *** udesale has joined #openstack-nova | 04:05 | |
| *** yedongcan has quit IRC | 04:13 | |
| *** yedongcan has joined #openstack-nova | 04:14 | |
| *** markvoelker has joined #openstack-nova | 04:20 | |
| *** markvoelker has quit IRC | 04:25 | |
| *** itlinux has joined #openstack-nova | 04:31 | |
| *** itlinux_ has quit IRC | 04:34 | |
| *** threestrands has joined #openstack-nova | 04:37 | |
| *** threestrands has quit IRC | 04:37 | |
| *** threestrands has joined #openstack-nova | 04:37 | |
| *** bhagyashris has joined #openstack-nova | 05:11 | |
| *** beekneemech has quit IRC | 05:16 | |
| *** bnemec has joined #openstack-nova | 05:20 | |
| openstackgerrit | zhufl proposed openstack/nova master: [Trivial]Remove unused helper _get_min_service_version https://review.opendev.org/678446 | 05:40 |
| *** Luzi has joined #openstack-nova | 05:43 | |
| openstackgerrit | zhufl proposed openstack/nova master: [Trivial]Remove unused helper get_vm_ref_from_name https://review.opendev.org/678444 | 05:49 |
| *** jaosorior has joined #openstack-nova | 05:51 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove descriptions of nonexistent hacking rules https://review.opendev.org/678462 | 05:56 |
| *** mkrai has quit IRC | 06:14 | |
| *** mkrai has joined #openstack-nova | 06:16 | |
| *** rcernin_ has quit IRC | 06:18 | |
| *** lpetrut has joined #openstack-nova | 06:18 | |
| *** dpawlik has joined #openstack-nova | 06:23 | |
| *** sridharg has joined #openstack-nova | 06:34 | |
| *** deepak_mourya_ has joined #openstack-nova | 06:34 | |
| *** takamatsu has joined #openstack-nova | 06:34 | |
| *** takamatsu has quit IRC | 06:42 | |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: db: Add resources column in instance_extra table https://review.opendev.org/678447 | 06:45 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: object: Introduce Resource and ResouceList objs https://review.opendev.org/678448 | 06:45 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Add resources dict into _Provider https://review.opendev.org/678449 | 06:45 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Retrive the allocations early https://review.opendev.org/678450 | 06:45 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Track orphan instances and error migrations in resource tracker https://review.opendev.org/678451 | 06:45 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Claim resources in resource tracker https://review.opendev.org/678452 | 06:45 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/678453 | 06:45 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: report VPMEM resources by provider tree https://review.opendev.org/678454 | 06:45 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Support VM creation with vpmems and vpmems cleanup https://review.opendev.org/678455 | 06:45 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456 | 06:45 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Add functional tests for virtual persistent memory https://review.opendev.org/678470 | 06:45 |
| *** bhagyashris has quit IRC | 06:47 | |
| *** aojea has joined #openstack-nova | 06:58 | |
| *** kashyap has joined #openstack-nova | 06:59 | |
| *** trident has quit IRC | 07:01 | |
| *** itlinux has quit IRC | 07:03 | |
| bauzas | good morning Nova | 07:04 |
| * bauzas is back after a long time | 07:04 | |
| gibi | good mornin bauzas | 07:04 |
| *** itlinux has joined #openstack-nova | 07:04 | |
| gibi | welcome back | 07:04 |
| alex_xu | good morning gibi bauzas | 07:04 |
| bauzas | thanks | 07:04 |
| gibi | hi alex_xu | 07:04 |
| *** shilpasd has joined #openstack-nova | 07:04 | |
| *** damien_r has joined #openstack-nova | 07:04 | |
| bauzas | gibi: good morning | 07:05 |
| bauzas | gibi: I'll slowly catch up this morning but in case any reviews for your series (which I have no crazy idea of the status yet), you can ping me | 07:05 |
| bauzas | I haven't forgotten train-3 is soon :) | 07:06 |
| * kashyap just back after a long time, too :D | 07:08 | |
| kashyap | Starts processing The Pile(tm). | 07:09 |
| *** trident has joined #openstack-nova | 07:10 | |
| gibi | bauzas: the bottom of the support-move-ops-with-qos-ports in here https://review.opendev.org/#/c/655110 mriedem has votes on the firt couple of patches. The current top of the series has the full support for cold migrate (and possibly resize) | 07:12 |
| gibi | bauzas: I appreciate any reviews :) | 07:12 |
| *** jawad_axd has joined #openstack-nova | 07:13 | |
| lpetrut | Hi, a quick question about failed live migrations: should the instances end up in error state even if the rollback is successful? this behavior was introduced by https://github.com/openstack/nova/blob/62f6a0a1bc6c4b24621e1c2e927177f99501bef3/nova/compute/manager.py#L6813-L6820 a couple of years ago. | 07:13 |
| lpetrut | looks like the libvirt driver won't raise an exception when the instance is recovered: https://github.com/openstack/nova/blob/62f6a0a1bc6c4b24621e1c2e927177f99501bef3/nova/virt/libvirt/driver.py#L8080-L8086 | 07:13 |
| lpetrut | I'm thinking about doing the same with the Hyper-V driver: avoid propagating migration exceptions if the instance is recovered. | 07:13 |
| openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Follow up for microversion 2.75 https://review.opendev.org/678473 | 07:20 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: update allocation in binding profile during migrate https://review.opendev.org/656422 | 07:20 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Extend NeutronFixture to handle migrations https://review.opendev.org/655114 | 07:20 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: prepare func test env for moving servers with bandwidth https://review.opendev.org/655109 | 07:20 |
| lpetrut | although the admin may just reset the state, not setting the instance to error state makes it clear that no further steps are required in order to fully recover the instance, reducing the amount of time required for debugging | 07:21 |
| *** threestrands has quit IRC | 07:24 | |
| *** xek has joined #openstack-nova | 07:25 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Func test for migrate server with ports having resource request https://review.opendev.org/655113 | 07:27 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Make _rever_allocation nested allocation aware https://review.opendev.org/676138 | 07:27 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support reverting migration / resize with bandwidth https://review.opendev.org/676140 | 07:32 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Func test for migrate re-schedule with bandwidth https://review.opendev.org/676972 | 07:34 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support migrating SRIOV port with bandwidth https://review.opendev.org/676980 | 07:37 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Allow migrating server with port resource request https://review.opendev.org/671497 | 07:37 |
| openstackgerrit | Akihiro Motoki proposed openstack/nova master: PDF documentation build https://review.opendev.org/676730 | 07:41 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Allow migrating server with port resource request https://review.opendev.org/671497 | 07:44 |
| *** phasespace has joined #openstack-nova | 07:46 | |
| openstackgerrit | Lucian Petrut proposed openstack/nova master: Avoid error state for recovered instances after failed migrations https://review.opendev.org/678481 | 07:52 |
| *** ivve has joined #openstack-nova | 07:54 | |
| openstackgerrit | Lucian Petrut proposed openstack/nova master: Avoid error state for recovered instances after failed migrations https://review.opendev.org/678481 | 07:55 |
| *** aojea has quit IRC | 07:57 | |
| *** markvoelker has joined #openstack-nova | 08:02 | |
| *** jangutter has joined #openstack-nova | 08:06 | |
| *** markvoelker has quit IRC | 08:07 | |
| *** derekh has joined #openstack-nova | 08:09 | |
| *** janki has joined #openstack-nova | 08:29 | |
| *** tkajinam has quit IRC | 08:30 | |
| *** janki has quit IRC | 08:32 | |
| *** janki has joined #openstack-nova | 08:34 | |
| *** avolkov has joined #openstack-nova | 08:35 | |
| openstackgerrit | Lucian Petrut proposed openstack/nova master: Avoid error state for recovered instances after failed migrations https://review.opendev.org/678481 | 08:37 |
| *** ralonsoh has joined #openstack-nova | 08:40 | |
| *** dtantsur|afk is now known as dtantsur | 08:46 | |
| *** rcernin_ has joined #openstack-nova | 09:14 | |
| *** damien_r has quit IRC | 09:16 | |
| stephenfin | sean-k-mooney: https://review.opendev.org/678497 | 09:16 |
| *** brinzhang_ has joined #openstack-nova | 09:17 | |
| *** brinzhang has quit IRC | 09:21 | |
| sean-k-mooney | im not sure how setup_develop works but will that pass mysql as an extra or just install an addtional package via pip | 09:21 |
| alex_xu | sean-k-mooney: stephenfin we can't revert a db migraton script, right? we should add new script to revert the upgrade for the old one, is that correct? | 09:27 |
| *** yedongcan has quit IRC | 09:27 | |
| sean-k-mooney | alex_xu: we should not do a revert in general | 09:27 |
| sean-k-mooney | are you asking for the pmem column | 09:28 |
| alex_xu | sean-k-mooney: yes | 09:28 |
| alex_xu | and we already have another upgrade script after vpmem | 09:28 |
| sean-k-mooney | just add a new migration that drops the column and adds the devices column | 09:28 |
| alex_xu | got it | 09:28 |
| sean-k-mooney | if we had made a release it would be a bit different as we would have had to do an online migration to move the data | 09:29 |
| alex_xu | luyao: ^, this doesn't sound right https://review.opendev.org/#/c/678447/2/nova/db/sqlalchemy/migrate_repo/versions/398_add_resources.py | 09:29 |
| sean-k-mooney | but since i doubt anywone has master deploy with vpmem in production it should be fine to drop and replace | 09:30 |
| alex_xu | yea, so good we aren't such late | 09:30 |
| alex_xu | yes, in the past, we should take care about that. but for now, just not ensure whether we should keep that policy | 09:31 |
| *** takashin has left #openstack-nova | 09:32 | |
| sean-k-mooney | i think the probme with just changing the new colum name woudl be it might mess up sqlachemys migration logic? or maybe im thinking about alembic. one of those compute a hash of the migration to generate a unic version number for the db | 09:33 |
| sean-k-mooney | so i think beyond the fact it would not fix any db that had run the old migration it could break other migrations. | 09:33 |
| alex_xu | yes, I think so | 09:34 |
| alex_xu | then you have to manually tune the migration version | 09:34 |
| sean-k-mooney | thinking about that a bit more i think that is only with alembic but still better to avoid it | 09:34 |
| kashyap | aspiers: Hey, just catching up with changes post PTO. Will respond once I finish processing The Pile. | 09:34 |
| kashyap | s/changes/patches/ | 09:35 |
| alex_xu | sean-k-mooney: yea, agree with you | 09:35 |
| sean-k-mooney | stephenfin: ah yes it is extras https://opendev.org/openstack/devstack/src/branch/master/inc/python#L419 | 09:41 |
| *** rcernin_ has quit IRC | 09:42 | |
| luyao | sean-k-mooney, alex_xu: Get it.! I will add a patch to remove vpmems column first. Thank you. | 09:43 |
| alex_xu | luyao: you needn't a separate patch remove vpmem column, you can have new script do the remove and add new resource column sametime | 09:44 |
| sean-k-mooney | luyao: basicaly dont update the migration script that added the vpmem column, just add another to remvoe it and add the resources column | 09:49 |
| *** bhagyashris__ has joined #openstack-nova | 09:57 | |
| openstackgerrit | Brin Zhang proposed openstack/nova master: Specify availability_zone to unshelve https://review.opendev.org/663851 | 09:57 |
| bhagyashris__ | stephenfin: Hi, I just address review comments that you have given on patch https://review.opendev.org/#/c/674895/17 related to functional test and uploaded it for community review | 09:59 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: setup.cfg: Cleanup https://review.opendev.org/677969 | 10:00 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: requirements: Move DB dependencies to 'extras' https://review.opendev.org/677475 | 10:00 |
| stephenfin | bhagyashris__: That's failing pretty hard at the moment, it seems? | 10:01 |
| stephenfin | Is there more patches expected? | 10:02 |
| stephenfin | *are | 10:02 |
| bhagyashris__ | stephenfin: I saw the functional test cases are failing because of change in dependent patches , I haven't yet checked where it's failing so are you going to check that point or should I? | 10:03 |
| stephenfin | It's late for you, right? I can do it if so | 10:03 |
| stephenfin | I need to fix the base patch anyway | 10:03 |
| bhagyashris__ | stephenfin: I just confirming because most of the time we reworked on same point | 10:04 |
| luyao | sean-k-mooney, alex_xu: 'migration script' you mean is the files under nova/db/sqlalchemy/migrate_repo/versions/ right? | 10:05 |
| bhagyashris__ | Yeah it's almost EOD here but I can do it by tomorrow | 10:05 |
| bhagyashris__ | stephenfin: if you are ok ? | 10:05 |
| stephenfin | Yup, I can do it | 10:05 |
| *** markvoelker has joined #openstack-nova | 10:05 | |
| stephenfin | Oh, sorry | 10:05 |
| stephenfin | Yeah, tomorrow is good | 10:05 |
| stephenfin | I'll work on fixing that other patch though, since I broke it | 10:06 |
| stephenfin | and leave the last one to you | 10:06 |
| bhagyashris__ | stephenfin: okay. Just reconfirming -> you will fix the main issue and after that I will continuously working on patch https://review.opendev.org/#/c/674895/17 to fix the functional test after your changes . | 10:08 |
| bhagyashris__ | stephenfin: right? | 10:08 |
| stephenfin | sure | 10:08 |
| bhagyashris__ | stephenfin: ok Thank you :) | 10:09 |
| *** bbowen_ has joined #openstack-nova | 10:10 | |
| *** markvoelker has quit IRC | 10:10 | |
| *** bbowen has quit IRC | 10:11 | |
| *** xek has quit IRC | 10:11 | |
| sean-k-mooney | stephenfin: so i just did a thing and it worked | 10:14 |
| *** bbowen_ has quit IRC | 10:15 | |
| *** bbowen has joined #openstack-nova | 10:16 | |
| sean-k-mooney | stephenfin: i finally tested my theory about nested pcie passthough. and yes you can do it if you add an iommu to the l1 guest and tweak the pcie layout so that the device is in a seperate iommu group | 10:16 |
| *** jaosorior has quit IRC | 10:26 | |
| *** xek has joined #openstack-nova | 10:26 | |
| *** markvoelker has joined #openstack-nova | 10:35 | |
| *** sapd1_x has quit IRC | 10:39 | |
| *** markvoelker has quit IRC | 10:40 | |
| bhagyashris__ | melwitt, mriedem, efrid, alex_xu: Requet you to review https://review.opendev.org/#/c/612626/ . | 10:45 |
| *** bhagyashris__ has quit IRC | 10:47 | |
| *** mkrai has quit IRC | 10:58 | |
| *** udesale has quit IRC | 11:04 | |
| *** francoisp has joined #openstack-nova | 11:07 | |
| *** tesseract has joined #openstack-nova | 11:12 | |
| *** ccamacho has joined #openstack-nova | 11:21 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta https://review.opendev.org/671801 | 11:24 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Fold in argument to '_update_provider_tree_for_vgpu' https://review.opendev.org/676729 | 11:24 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add reshaper for PCPU https://review.opendev.org/674895 | 11:24 |
| *** Luzi has quit IRC | 11:25 | |
| *** Luzi has joined #openstack-nova | 11:26 | |
| *** Garyx has quit IRC | 11:43 | |
| *** jaosorior has joined #openstack-nova | 11:44 | |
| *** jroll has quit IRC | 11:44 | |
| *** jroll has joined #openstack-nova | 11:45 | |
| *** rcernin_ has joined #openstack-nova | 11:53 | |
| shilpasd | stephenfin: need discussion related to python3-train patch for masakari-monitors https://review.opendev.org/#/c/669387/ for py27/py36 and py37, here observed 'import libvirt' is an issue | 11:59 |
| shilpasd | stephenfin: can you pl give me some pointers to resolve this | 11:59 |
| *** derekh has quit IRC | 12:00 | |
| *** markvoelker has joined #openstack-nova | 12:00 | |
| *** weshay_MOD is now known as weshay | 12:07 | |
| *** larainema has quit IRC | 12:08 | |
| *** artom has joined #openstack-nova | 12:09 | |
| *** rcernin_ has quit IRC | 12:32 | |
| *** nweinber has joined #openstack-nova | 12:36 | |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: db: Add resources column in instance_extra table https://review.opendev.org/678447 | 12:39 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: object: Introduce Resource and ResouceList objs https://review.opendev.org/678448 | 12:39 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Add resources dict into _Provider https://review.opendev.org/678449 | 12:39 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Retrive the allocations early https://review.opendev.org/678450 | 12:39 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Track orphan instances and error migrations in resource tracker https://review.opendev.org/678451 | 12:39 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Claim resources in resource tracker https://review.opendev.org/678452 | 12:39 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/678453 | 12:39 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: report VPMEM resources by provider tree https://review.opendev.org/678454 | 12:39 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Support VM creation with vpmems and vpmems cleanup https://review.opendev.org/678455 | 12:39 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456 | 12:39 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: Add functional tests for virtual persistent memory https://review.opendev.org/678470 | 12:39 |
| *** xek_ has joined #openstack-nova | 12:42 | |
| *** mgoddard has quit IRC | 12:47 | |
| *** xek_ has quit IRC | 12:47 | |
| *** hemna has quit IRC | 12:47 | |
| *** mgoddard has joined #openstack-nova | 12:47 | |
| *** martinkennelly has joined #openstack-nova | 12:51 | |
| *** gbarros has joined #openstack-nova | 12:53 | |
| *** jmlowe has quit IRC | 12:56 | |
| *** slaweq has joined #openstack-nova | 12:58 | |
| *** macz has joined #openstack-nova | 13:04 | |
| *** boxiang_ has joined #openstack-nova | 13:09 | |
| *** boxiang_ has left #openstack-nova | 13:09 | |
| *** eharney has quit IRC | 13:09 | |
| *** boxiang_ has joined #openstack-nova | 13:10 | |
| *** boxiang_ has left #openstack-nova | 13:10 | |
| *** boxiang_ has joined #openstack-nova | 13:10 | |
| *** boxiang_ has left #openstack-nova | 13:11 | |
| *** boxiang_ has joined #openstack-nova | 13:11 | |
| *** janki has quit IRC | 13:12 | |
| *** boxiang_ has quit IRC | 13:13 | |
| *** boxiang_ has joined #openstack-nova | 13:14 | |
| *** jmlowe has joined #openstack-nova | 13:16 | |
| *** gbarros has quit IRC | 13:16 | |
| *** boxiang_ has quit IRC | 13:18 | |
| *** KeithMnemonic has joined #openstack-nova | 13:19 | |
| *** dave-mccowan has joined #openstack-nova | 13:19 | |
| *** BjoernT has joined #openstack-nova | 13:23 | |
| *** gbarros has joined #openstack-nova | 13:27 | |
| *** tbachman has joined #openstack-nova | 13:29 | |
| *** slaweq has quit IRC | 13:36 | |
| *** boxiang_ has joined #openstack-nova | 13:37 | |
| *** Luzi has quit IRC | 13:44 | |
| *** hemna has joined #openstack-nova | 13:46 | |
| *** KeithMnemonic has quit IRC | 13:48 | |
| *** jawad_axd has quit IRC | 13:53 | |
| *** BjoernT_ has joined #openstack-nova | 13:56 | |
| *** mgoddard has quit IRC | 13:56 | |
| *** mgoddard has joined #openstack-nova | 13:58 | |
| *** BjoernT has quit IRC | 13:59 | |
| *** mriedem has joined #openstack-nova | 13:59 | |
| *** eharney has joined #openstack-nova | 14:01 | |
| *** boxiang has quit IRC | 14:18 | |
| *** boxiang has joined #openstack-nova | 14:18 | |
| efried | mriedem, melwitt, dansmith: Could I please get eyes on https://review.opendev.org/#/c/678237/ to unwedge u-c https://review.opendev.org/#/c/678207/ | 14:20 |
| *** eharney has quit IRC | 14:20 | |
| mriedem | " We get to stop doing this soon, I promise." fool me once, shame on you, fool me twice... | 14:21 |
| efried | but mordred *also* promises | 14:21 |
| sean-k-mooney | did we "fix" something like this recently | 14:22 |
| efried | sean-k-mooney: you mean https://review.opendev.org/#/c/676495/ ? | 14:22 |
| sean-k-mooney | yes | 14:22 |
| efried | You mean, every time we release a new sdk we have to re-fix our fixture that stubs out something private in sdk? Yes. | 14:22 |
| efried | the long term solution is to not stub out private sdk stuff. | 14:22 |
| mordred | srrsly | 14:23 |
| mordred | yes | 14:23 |
| efried | sdk is going to expose a fixture that we can simply import and use | 14:23 |
| sean-k-mooney | yep seams like what we should be doing | 14:23 |
| mordred | the sdk is going to give you the happy fixture | 14:23 |
| mordred | and you will use that | 14:23 |
| mordred | and it will be tested in the sdk | 14:23 |
| mordred | so we won't break you | 14:23 |
| efried | right, and then as they change internals, they'll change the fixture, and we'll automatically "keep up". | 14:23 |
| mriedem | i might as well just fast approve this | 14:23 |
| mriedem | it's test only and holding up the u-c change | 14:23 |
| *** eharney has joined #openstack-nova | 14:23 | |
| efried | yes, the u-c patch proves it works | 14:24 |
| sean-k-mooney | efried: we try to do that with plamcent but we have been broken with the placement fixture in the past | 14:24 |
| efried | and the nova patch proves it still works at 0.34.0 | 14:24 |
| efried | sure, but less likely | 14:24 |
| sean-k-mooney | we will need to similarly pull the sdk out of requirements and install it spereatly in tox right | 14:24 |
| sean-k-mooney | e.g. to consume it form master? | 14:24 |
| efried | eh? Why would we need to do that? | 14:24 |
| sean-k-mooney | the same reason we do it for the placement fixture | 14:25 |
| efried | oh, no, tracking against real releases should be fine. | 14:25 |
| sean-k-mooney | ok | 14:25 |
| efried | When we've needed to pre-test a feature, we twiddle the project list in .zuul.yaml | 14:25 |
| sean-k-mooney | i guess for placement we do it to get eraly acess to features | 14:25 |
| efried | yes, which is actually probably not necessary anymore | 14:25 |
| efried | actually probably not advisable. | 14:25 |
| efried | we should bring that up with cdent when he gets back. | 14:26 |
| sean-k-mooney | yes the second one | 14:26 |
| efried | (which I thought was today, but I guess not) | 14:26 |
| sean-k-mooney | i think it is still nessacary for some feature but we have a backlog of feature to catch up on | 14:26 |
| openstackgerrit | Merged openstack/nova master: Document archive_deleted_rows return codes https://review.opendev.org/677819 | 14:26 |
| efried | sean-k-mooney: I actually don't remember the last time we did something in nova that *needed* to track against unreleased placement. | 14:27 |
| efried | and iiuc having it set up this way allows us to do things like... merge a nova feature before placement is released accordingly, thereby breaking nova (cf. the aforementioned fixture issue). | 14:27 |
| sean-k-mooney | well all of the nested stuff would have needed it if we did that this cycle. | 14:27 |
| efried | yeah, but we can just release placement | 14:27 |
| efried | releases are pretty cheap. | 14:27 |
| sean-k-mooney | maybe. i guess depends on is really the way to test this stuff | 14:28 |
| efried | right, I'm saying, I'm not sure there's a good reason for us to have placement in required projects by default. | 14:29 |
| sean-k-mooney | i wasnt refering to required proejcts specifcally | 14:29 |
| efried | that's how we end up with nova running against master placement in the gate, nah? | 14:30 |
| mriedem | you need placement in required projects in the gate jobs b/c it's a service not a library | 14:30 |
| mriedem | if you want to freeze the fixture or something, yo'ud need to make it a library i think | 14:30 |
| openstackgerrit | Merged openstack/nova master: Change nova-manage unexpected error return code to 255 https://review.opendev.org/677832 | 14:31 |
| efried | do we have keystone/glance/ironic/neutron/cinder in required projects too? | 14:31 |
| mriedem | iow, if placement isn't in required-projects, gate jobs aren't going to run with placement from master | 14:31 |
| * efried looks | 14:31 | |
| mriedem | yes | 14:31 |
| openstackgerrit | Merged openstack/nova master: Document map_instances return codes in table format https://review.opendev.org/677835 | 14:31 |
| efried | okay, maybe I'm thinking of placement as more of a lib | 14:31 |
| dansmith | artom: you working on the test failures? I haven't looked to see what they are, but I assume they're relevant since all of them seem to be failing | 14:32 |
| mriedem | https://github.com/openstack/devstack/blob/master/.zuul.yaml#L368 | 14:32 |
| artom | dansmith, yeo | 14:32 |
| artom | *yep | 14:32 |
| artom | dansmith, wanted to ask you, we can't fully remove an RPC param until the next major version bump, right? | 14:33 |
| *** zigo has joined #openstack-nova | 14:33 | |
| artom | I wanted to remove destroy_disks from the rollback_live_migration call and have the method on the destination do all the deciding, since _live_migration_flags isn't host-dependant, it only cares about migrate_data. | 14:33 |
| artom | But we have to keep compatibility code until RPC 6.0, right? | 14:33 |
| dansmith | artom: fully or otherwise, yeah | 14:34 |
| *** phasespace has quit IRC | 14:35 | |
| artom | dansmith, wait, otherwise? So not at all? RPC params are only additive? | 14:35 |
| dansmith | artom: yeah, meaning if you supported the letter and intent of the law in the current version, you can't stop honoring either until the next bump | 14:36 |
| mriedem | what does destroy_disks have to do with numa? | 14:36 |
| dansmith | artom: in that, you can't just accept and ignore some flag that has some meaning | 14:36 |
| artom | mriedem, it's convoluted, but basically, we need to call rollback_live_migration_at_destination if we did a claim and we need to drop it | 14:37 |
| artom | Except now, it's called only the do_cleanup is True (as determined by _live_migration_flags) | 14:37 |
| mriedem | yeah i remmeber that | 14:38 |
| artom | There's a bunch of possibilities around that, but the end result it, everything becomes easier if we just always call rollback_live_migration_at_destination, and let it decide what cleanup needs to be done | 14:38 |
| mriedem | you mean rollback_live_migration_at_destination right? | 14:38 |
| artom | Instead of passing it booleans over RPC | 14:38 |
| artom | Except doing that would technically require removing the destroy_disks param | 14:38 |
| dansmith | artom: don't we pass that flag because there are cases where the other side can't decide what to do? | 14:39 |
| *** mlavalle has joined #openstack-nova | 14:40 | |
| artom | dansmith, might have been the case at one time, but doesn't look like it now. https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6946-L6979 doesn't depend on anything but migrate_data, which both sides have | 14:41 |
| *** markvoelker has quit IRC | 14:41 | |
| artom | Unless I'm really blind | 14:41 |
| artom | I guess it was done to save an RPC call if we didn't need it? | 14:41 |
| artom | sean-k-mooney, could you weigh in please: https://review.opendev.org/#/c/635669/39/nova/compute/resource_tracker.py@307 | 14:43 |
| dansmith | hmm, okay.. I really thought we had serious issues with that logic, | 14:44 |
| dansmith | but perhaps the passing of migrate data to more places made that better or something | 14:44 |
| dansmith | it's been a while | 14:44 |
| dansmith | mriedem probably remembers better | 14:44 |
| *** markvoelker has joined #openstack-nova | 14:44 | |
| mriedem | i would have to dig, _live_migration_cleanup_flags sucks | 14:44 |
| dansmith | yeah | 14:44 |
| mriedem | i had to workaround the same kind of thing where do_cleanup was False but i still needed to do network_api.setup_networks_on_host for the dest in the neutron case with port bindings | 14:45 |
| mriedem | https://review.opendev.org/#/q/I658e0a749e842163ed74f82c975bcaf19f9f7f07 | 14:46 |
| mriedem | artom: so you just need the source to make a call to the dest to rollback the claim on failure right? regardless of shared storage or not | 14:47 |
| artom | mriedem, yep | 14:47 |
| artom | I could always just add a new RPC method | 14:47 |
| mriedem | dansmith probably won't like what i'm about to suggest, but it might be better to just write a new rpc method, | 14:47 |
| artom | Seems weird to do that since we already have a thing called "rollback" | 14:47 |
| mriedem | rather than munge it into this steaming pile o shit | 14:47 |
| artom | You don't like wobbly shit castles? | 14:48 |
| artom | If dansmith's onboard I'm all for that | 14:48 |
| mriedem | we'd only make this new call if we know, from the source, that there would be something to cleanup, right? meaning the instance has numa claims in the migrate_data object or something? | 14:48 |
| dansmith | mriedem: I dunno why you'd say that... making another call that does what we want instead of calling this one and expecting very specific behavior is better, IMHO | 14:48 |
| mriedem | dansmith: b/c of our prep_resize hullabaloo a couple of weeks back | 14:49 |
| artom | 'tis settled. | 14:49 |
| * artom hax | 14:49 | |
| dansmith | mriedem: yeah, I mean, okay fair point, but that seemed a little more knife edge single-case to me | 14:49 |
| *** lpetrut has quit IRC | 14:55 | |
| *** mkrai has joined #openstack-nova | 14:57 | |
| *** belmoreira has joined #openstack-nova | 14:57 | |
| sean-k-mooney | artom: oh hi yes | 15:03 |
| mriedem | if i re-use the existing prep_resize, i will either have to pass it a new param to control the logic or change it to check if the migration.cross_cell_move flag is set and True (which won't work for old computes, but that could be filtered out in the api or conductor based on compute service version), but either way i'd need to change it to say (1) don't reschedule and (2) don't rpc cast to resize_instance on the source | 15:03 |
| sean-k-mooney | had neutron open for some reason ill take a look | 15:03 |
| *** boxiang_ has quit IRC | 15:03 | |
| *** boxiang_ has joined #openstack-nova | 15:03 | |
| *** boxiang_ has quit IRC | 15:04 | |
| *** boxiang_ has joined #openstack-nova | 15:05 | |
| *** tbachman has quit IRC | 15:05 | |
| *** boxiang_ has quit IRC | 15:06 | |
| *** boxiang_ has joined #openstack-nova | 15:07 | |
| *** boxiang_ has quit IRC | 15:08 | |
| *** boxiang_ has joined #openstack-nova | 15:08 | |
| dansmith | artom: the new call would be "unclaim $this on the dest" yeah? | 15:09 |
| sean-k-mooney | we used the migration_data for a few reason on of which was move claims were only used for cold migration and we did not want to have to depend on the for sriov migration. | 15:09 |
| artom | dansmith, yep | 15:09 |
| *** phillw has joined #openstack-nova | 15:10 | |
| sean-k-mooney | we already had the old vif bindings for the souce and the new vif bindigns for the dest in the migration data | 15:10 |
| dansmith | artom: and what if the source rebooted because of the failure and never calls that? | 15:10 |
| sean-k-mooney | so it was simple to extend that to also have the pci info | 15:10 |
| dansmith | artom: I don't think I was clear about where you're persisting anything from the claim you're doing, other than in memory | 15:10 |
| sean-k-mooney | the only thn we neede was the pci adress which is stored in the vifs port profile | 15:11 |
| artom | dansmith, claims aren't persisted anywhere. The instance has a migration context though | 15:11 |
| artom | dansmith, the update resources periodic would clean up if we don't drop the claim "manually", I believe | 15:11 |
| dansmith | artom: right, the claims themselves aren't, but | 15:12 |
| *** dr_gogeta86 has joined #openstack-nova | 15:12 | |
| dr_gogeta86 | hi | 15:12 |
| dansmith | you mean the periodic would find that the migration wasn't associated with us anymore and clean up the in-memory state on the dest | 15:12 |
| *** phillw has left #openstack-nova | 15:12 | |
| artom | dansmith, wait, what do you mean by in-memory state? | 15:12 |
| artom | dansmith, it would update the available resources in the database... | 15:13 |
| dansmith | artom: that's my point, I'm not sure where we're accouting for these resources in the database | 15:13 |
| sean-k-mooney | artom: in the host_cell object in the numa toplogy blob in the compute node | 15:13 |
| artom | dansmith, ^^ there you go :) | 15:14 |
| dansmith | where are we updating that? | 15:14 |
| *** rajinir has joined #openstack-nova | 15:14 | |
| artom | dansmith, lemme dig | 15:14 |
| dansmith | because I don't see it | 15:15 |
| artom | Knowing sean-k-mooney he might pull it out in seconds | 15:15 |
| sean-k-mooney | dansmith: when we claim the cpus and hugepages in pre live migereation at dest we shoudl update the db | 15:15 |
| *** factor has joined #openstack-nova | 15:15 | |
| sean-k-mooney | that is where we claim the pci deivces for sriov live migration | 15:15 |
| dansmith | sean-k-mooney: yeah, stop using words like "should" and handwaving with "the database".. because that doesn't help :) | 15:15 |
| dansmith | sure, pci devices are claimed with the manager, I get tat | 15:16 |
| dansmith | I don't think we ever claimed the basic resources in the db before we removed all that stuff | 15:16 |
| dansmith | so I'm not sure where this is happening | 15:16 |
| dansmith | mriedem: do you know? | 15:16 |
| artom | dansmith, we don't care about the basic resources anymore, do we? | 15:17 |
| artom | They're all in placement | 15:17 |
| dansmith | artom: no, that's my point exactly | 15:17 |
| artom | I was talking about specific CPUs and stuff | 15:17 |
| sean-k-mooney | well i know that is where we do it for sriov. i know artom is calling the hardware.py module to geenreate the dest numa toplogy and i belive he is claimign the resouces at that point. ill check his patches | 15:17 |
| artom | dansmith, ok, then I didn't understand your point | 15:17 |
| dansmith | sean-k-mooney: again, that's too vague | 15:17 |
| dansmith | artom: I'm just comparing what you're doing here to what we *used* to do with the basic resources, which is effectively what you're "claiming" that you're doing here | 15:18 |
| mriedem | dansmith: well, sort of - we'd update the usage values on the ComputeNode | 15:18 |
| dansmith | mriedem: usage for ram and cpu etc right? | 15:18 |
| mriedem | and disk yeah | 15:18 |
| artom | dansmith, I'm trying to find the specific call that mriedem'd talking about | 15:18 |
| mriedem | https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L1059 | 15:19 |
| artom | There's a lot of layers, so it's taking a while | 15:19 |
| dansmith | mriedem: did we use that for actual scheduling though? I didn't think we did, but even still, I'm not sure what usage info we have for a complex numa | 15:19 |
| mriedem | pretty sure the core/disk/ram filters looked at the _used values | 15:19 |
| artom | dansmith, so then I think https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L1123 would be the NUMA stuff | 15:19 |
| mriedem | https://github.com/openstack/nova/blob/stable/stein/nova/scheduler/filters/core_filter.py#L65 | 15:20 |
| artom | Though I haven't traced how that's called, exactly | 15:20 |
| dansmith | mriedem: okay I didn't think they did, but it's been a long time | 15:20 |
| dansmith | mriedem: okay, yeah, host state is coming back to me a bit | 15:20 |
| mriedem | right, HostState is a wrapper over ComputeNode | 15:20 |
| dansmith | artom: that's a blank line | 15:20 |
| mriedem | i recently found out one of the weighers looks at one of these values as well | 15:20 |
| artom | dansmith, dammit, I was basing it on my local code | 15:20 |
| artom | sec | 15:20 |
| mriedem | ah yes https://github.com/openstack/nova/blob/stable/stein/nova/scheduler/weights/cpu.py#L43 | 15:21 |
| dansmith | mriedem: okay I've dumped all of that out of my head as no longer necessary, thanks | 15:21 |
| dansmith | okay, so _update_usage() is writing a new numa topo for the host | 15:21 |
| artom | dansmith, https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L1095 | 15:22 |
| sean-k-mooney | yes the werigher do look at the numa info amoung other things | 15:22 |
| dansmith | artom: okay _get_usage_dict() is what I'm looking for I think | 15:23 |
| *** alex_xu has quit IRC | 15:23 | |
| dansmith | it looks like we write that compute node record a bunch of times? | 15:23 |
| dansmith | because we call it from update-from-migration, which generates usage and saves it, and would do that again for other migrations and other instances | 15:24 |
| artom | Yeah, I can't imagine we were too efficient with that | 15:24 |
| artom | That code is a whole bunch of _update_blah calling each other | 15:24 |
| artom | It works though | 15:24 |
| sean-k-mooney | so artom is afctully cretein the livemigration claim here https://review.opendev.org/#/c/634606/54/nova/compute/manager.py@6458 in heck_can_live_migrate_destination | 15:24 |
| *** alex_xu has joined #openstack-nova | 15:25 | |
| sean-k-mooney | *check_can_live_migrate_destination | 15:25 |
| artom | At the end of train, Windriver found that the update resources periodic task would remove the new instance's usage because IIRC instance.numa_topology wasn't being updated | 15:25 |
| mriedem | end of train? | 15:25 |
| mriedem | rocky? | 15:25 |
| artom | Err, stein | 15:25 |
| artom | I always mix those up | 15:25 |
| artom | Last cycle | 15:26 |
| dansmith | artom: so to summarize, this new call is a short circuit to the dest to remove the migration usage from the compute node record's topology, which would also happen if the periodic ran and noticed the migration was no longer relevant to it | 15:26 |
| artom | dansmith, yep | 15:27 |
| dansmith | I think part of my problem is, I've mixed up the new-world placement arrangement, with the counting quotas stuff, and then the few remaining things that still use this system | 15:27 |
| dansmith | heaven forbid we account for numbers in the same place | 15:27 |
| openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Clarify --migration-type migration value as cold migration https://review.opendev.org/678593 | 15:28 |
| sean-k-mooney | well this is really still used to account for assignment of indiviula resouce rather then count the number of resouce avialble | 15:28 |
| artom | dansmith, what sean-k-mooney said. We need to track specific resources like CPU cores, which placement will never do, so this will never fully go away | 15:29 |
| dansmith | in the pci and numa case you mean | 15:29 |
| mriedem | what is "this"? | 15:29 |
| dansmith | sean-k-mooney: ^ | 15:29 |
| *** macz has quit IRC | 15:29 | |
| artom | this == the resource tracker, and claims | 15:29 |
| sean-k-mooney | no we need to do it for pinned cpus too | 15:29 |
| mriedem | BBBZZZTTT | 15:29 |
| sean-k-mooney | hugepages can be totally done in placmenet however | 15:29 |
| artom | We need to say CPU 3 is used, not just "we have 1 CPU used out of 4" | 15:29 |
| sean-k-mooney | we just need to do the work to move it | 15:29 |
| mriedem | before removing the claims for them, the RT also "tracked" vcpu/ram/disk, just not at some individual device level | 15:29 |
| mriedem | so the RT did both before train | 15:29 |
| artom | mriedem, yep | 15:29 |
| mriedem | hence me and dansmith asking "what is this?" | 15:30 |
| *** hamzy_ is now known as hamzy | 15:30 | |
| sean-k-mooney | artom: there is a part that im not fully following in your code. i need to do a full review again but where are you calling into the hardware module to calulate the new numa toplogy/cpus/hugepages | 15:32 |
| *** mtanino has joined #openstack-nova | 15:32 | |
| artom | sean-k-mooney, not directly, it's hidden in the claim | 15:32 |
| sean-k-mooney | that code definetly exsited in the stien version but i have not traced it in the new version | 15:32 |
| sean-k-mooney | ok can i redeploy this again today and test it by the way | 15:33 |
| artom | sean-k-mooney, yeah, at fist I wanted to avoid claims entirely because their code is, umm, interesting | 15:33 |
| dansmith | artom: you're calling that in the claim to validate page_size, | 15:33 |
| artom | But if we don't want races we have to use them | 15:33 |
| dansmith | but nothing else right? | 15:33 |
| dansmith | from the claim I mean | 15:33 |
| artom | dansmith, the page size is special because the "normal" virt.hardware code would allow a page size change | 15:33 |
| dansmith | he probably wants _get_live_migrate_numa_info() in the libvirt patch | 15:33 |
| dansmith | https://review.opendev.org/#/c/634828/48/nova/virt/libvirt/driver.py | 15:34 |
| artom | Since it only goes by the flavor extra spec | 15:34 |
| artom | So we need extra logic to block it | 15:34 |
| dansmith | yeah, I know | 15:34 |
| dansmith | he's asking where the new guest numa topo gets created I think | 15:34 |
| sean-k-mooney | dansmith: artom shoudl be calling the hardware moduel/RT to caluate a new numa toplogy, validate page size and claim pCPUs on the destation | 15:34 |
| dansmith | sean-k-mooney: yeah, I'm not sure about that last bit | 15:34 |
| * artom searches | 15:34 | |
| dansmith | validate page size yes, in the claim | 15:34 |
| dansmith | calculate a new topology, in the above link | 15:35 |
| dansmith | but not sure I see where the dest claims that, tbh | 15:35 |
| sean-k-mooney | dansmith: it shoudl all be don in can_fit_numa_to_host or something like that let me see if i can find it | 15:35 |
| artom | sean-k-mooney, https://github.com/openstack/nova/blob/master/nova/compute/claims.py#L138 | 15:35 |
| sean-k-mooney | yes | 15:35 |
| *** brinzhang_ has quit IRC | 15:35 | |
| sean-k-mooney | hardware.numa_fit_instance_to_host | 15:35 |
| sean-k-mooney | is the entry point to all the nuam/hugepage/pinning code | 15:36 |
| *** brinzhang_ has joined #openstack-nova | 15:36 | |
| dansmith | oh, that's already there because of regular migrations right? | 15:36 |
| *** boxiang_ has quit IRC | 15:36 | |
| sean-k-mooney | it looks at all those requets and returns a numa toplogy object that fultiles all of the request or raise an excepiton | 15:36 |
| artom | dansmith, that's actually the non-move Claim, so even booting instances use it | 15:36 |
| dansmith | okay | 15:36 |
| sean-k-mooney | dansmith: its tere for cold migration | 15:36 |
| dansmith | yeah | 15:36 |
| dansmith | whatever | 15:37 |
| sean-k-mooney | if hardware.numa_fit_instance_to_host does not raise one of like50 exceptions then the host is valid and it returns the numa toplogy | 15:38 |
| artom | exception.TooHardIGiveUpDoItYourself | 15:39 |
| artom | exception.TheCloudWasAMistak | 15:39 |
| *** mtanino has quit IRC | 15:39 | |
| *** efried has quit IRC | 15:42 | |
| *** gyee has joined #openstack-nova | 15:45 | |
| *** boxiang has quit IRC | 15:48 | |
| *** tbachman has joined #openstack-nova | 15:49 | |
| *** xek has quit IRC | 15:51 | |
| *** efried has joined #openstack-nova | 15:51 | |
| *** boxiang has joined #openstack-nova | 15:51 | |
| *** boxiang has quit IRC | 15:52 | |
| *** boxiang has joined #openstack-nova | 15:52 | |
| *** factor has quit IRC | 15:53 | |
| *** factor has joined #openstack-nova | 15:54 | |
| *** slaweq has joined #openstack-nova | 15:56 | |
| *** mkrai has quit IRC | 15:56 | |
| sean-k-mooney | artom: :) that might be clearer then some of the messages we raise | 15:58 |
| *** ivve has quit IRC | 16:00 | |
| artom | sean-k-mooney, hah! exception.TurnBackWhileYouCan | 16:01 |
| *** belmoreira has quit IRC | 16:03 | |
| *** ircuser-1 has joined #openstack-nova | 16:24 | |
| *** gbarros has quit IRC | 16:27 | |
| *** gbarros has joined #openstack-nova | 16:28 | |
| stephenfin | bauzas: If you haven't already gone home, want to send this through? https://review.opendev.org/#/c/672336/ | 16:30 |
| bauzas | stephenfin: sure, that will help not going crazy with my visa application form | 16:31 |
| *** nicolasbock has joined #openstack-nova | 16:32 | |
| bauzas | stephenfin: ouch, a bit hard to review given we need to make sure we don't regress on the test coverage | 16:32 |
| bauzas | did efried checked it too ? | 16:32 |
| efried | yeah, I did. | 16:33 |
| stephenfin | Yeah, I haven't removed anything | 16:33 |
| stephenfin | It's just moving things around | 16:33 |
| stephenfin | I add a good few things to that later so I wanted it broken up | 16:34 |
| *** boxiang has quit IRC | 16:35 | |
| *** boxiang has joined #openstack-nova | 16:35 | |
| *** igordc has joined #openstack-nova | 16:37 | |
| *** factor has quit IRC | 16:39 | |
| bauzas | ok, let's make a trust bond | 16:43 |
| *** ociuhandu has joined #openstack-nova | 16:44 | |
| *** boxiang has quit IRC | 16:45 | |
| *** boxiang has joined #openstack-nova | 16:45 | |
| efried | fungi: stephenfin is quoting you here https://review.opendev.org/#/c/677969/ as agreeing we don't need eggs. If that's so, would you mind throwing a +1 on there for me? | 16:47 |
| fungi | efried: sure, just a sec | 16:47 |
| efried | fungi, stephenfin: okay, just as a sanity check, I codesearched: http://codesearch.openstack.org/?q=egg%3Dnova&i=nope&files=&repos= | 16:50 |
| *** ociuhandu has quit IRC | 16:51 | |
| fungi | as far as i know, that's merely how you tell pip what to name the package it's installing. in modern usage it's actually going to build a wheel from the code there | 16:51 |
| fungi | because it's not starting from an actual package, pip doesn't know what the package name for that repository should be | 16:52 |
| fungi | i believe the name of the option is a bit of unfortunate legacy naming for the sake of backwards compatibility | 16:52 |
| efried | o...kay | 16:52 |
| fungi | i'll double-check the reference materials to confirm | 16:53 |
| *** slaweq has quit IRC | 16:55 | |
| *** tesseract has quit IRC | 16:57 | |
| efried | dansmith: Are we allowed to rename a field like this https://review.opendev.org/#/c/678447/3/nova/db/sqlalchemy/models.py or do we have to burn it and just create a new one? | 17:00 |
| dansmith | efried: no you can't do that | 17:00 |
| *** eharney has quit IRC | 17:00 | |
| *** slaweq has joined #openstack-nova | 17:00 | |
| dansmith | efried: well, wait, nothing has ever been written there right? | 17:00 |
| efried | right | 17:01 |
| dansmith | that one specifically | 17:01 |
| efried | right | 17:01 |
| dansmith | technically you could although it's... icky | 17:01 |
| efried | never used for anything at all | 17:01 |
| dansmith | this is why, btw, we should never merge db changes until the patches above are at least close | 17:01 |
| efried | https://review.opendev.org/#/c/678447/3/nova/db/sqlalchemy/migrate_repo/versions/401_add_resources.py help? | 17:01 |
| sean-k-mooney | efried: i was discussing this with alex_xu this morning | 17:01 |
| efried | yeah, I still have the bruises from the original smackdown dansmith | 17:01 |
| sean-k-mooney | i was suggesting just ahveing anothe migration to nuke it and add the new field | 17:01 |
| sean-k-mooney | given that nothing has used it and we have not released with it | 17:02 |
| sean-k-mooney | the previous version did this which was worse https://review.opendev.org/#/c/678447/2/nova/db/sqlalchemy/migrate_repo/versions/398_add_resources.py | 17:03 |
| efried | dansmith: I'll -2 this for reasons stated if you want to review/approve it. | 17:03 |
| *** slaweq_ has joined #openstack-nova | 17:05 | |
| *** slaweq has quit IRC | 17:05 | |
| *** ociuhandu has joined #openstack-nova | 17:09 | |
| fungi | efried: stephenfin: https://pip.pypa.io/en/stable/user_guide/#requirements-files #4 has an example of that syntax, and the terminology is still semi-relevant since sdists include a packagename.egg-info directory for their metadata, but the egg_info block in setup.cfg looks like it's related to old d2to1 functionality related to some sphinx extension for reporting version information (which make sense | 17:12 |
| fungi | given the names of the options in it) | 17:12 |
| fungi | that has appeared in nova's setup.cfg since 2010 according to git, and seems to have been cargo-culted into numerous projects who copied their setup.cfg from nova's | 17:12 |
| *** ociuhandu has quit IRC | 17:13 | |
| efried | shocking | 17:13 |
| fungi | it seems to be entirely vestigial at this stage, and a lot of newer projects don't include it (those newer projects which do include it have, i suspect, just copied it from older projects not knowing any better) | 17:13 |
| efried | okay, well, I suppose it's probably an easy enough thing to revert & backport if we find out in four years that we broke somebody. | 17:14 |
| efried | Thanks fungi | 17:14 |
| fungi | yeah, if all the copying started from nova, then maybe the undoing also has to start there ;) | 17:14 |
| fungi | mordred and/or dhellmann may have sufficient context to explain what that block was actually for once upon a time | 17:15 |
| fungi | the fact that it references svn dates it nicely | 17:15 |
| *** shilpasd has quit IRC | 17:16 | |
| mordred | fungi: what did I do? | 17:18 |
| mordred | oh that. I'd love if it went away | 17:19 |
| fungi | mordred: discussing the necessity of the [egg_info] block in setup.cfg | 17:19 |
| fungi | (or lack of necessity) | 17:19 |
| mordred | oh - wait - I thought it was the boilerplate in setup.py for eventlet ... one sec | 17:19 |
| fungi | mordred: egg_info.{tag_build,tag_date,tag_svn_revision} was something to do with a sphinx extension? | 17:20 |
| fungi | i tried piecing together old commit messages and changelog entries from d2to1 to work out why it was there | 17:21 |
| mordred | ah. I believe that is some text that setuptools would write to setup.cfg if it wasn't there | 17:21 |
| mordred | so we added it to avoid it being added and being a diff when running jobs | 17:22 |
| *** martinkennelly has quit IRC | 17:22 | |
| dansmith | efried: did you mean +2? | 17:22 |
| mordred | I'm guessing setuptools has since stopped writing random crap to setup.cfg senselessly :) | 17:22 |
| fungi | fwiw i've not seen setuptools write anything into setup.cfg files in modern usage | 17:22 |
| mordred | yeah | 17:22 |
| mordred | it's possible it was a distutils thing even | 17:22 |
| efried | dansmith: I mean I'm putting a -2 to hold the series, so that you can review and +2 it without fear of it accidentally merging like last time. | 17:22 |
| mordred | like - that's some OLD cruft | 17:22 |
| dansmith | efried: ah, cool | 17:23 |
| fungi | thanks mordred! | 17:23 |
| *** dtantsur is now known as dtantsur|afk | 17:25 | |
| efried | mriedem: would I be correct in surmising that "cold migration" existed in the world before "live migration" | 17:26 |
| *** ralonsoh has quit IRC | 17:26 | |
| sean-k-mooney | efried: for most hyperviors yes | 17:26 |
| sean-k-mooney | when nova was recreate both where a thing | 17:27 |
| efried | I can't think of any other reason why "migration" means "cold migration" in the API. | 17:27 |
| sean-k-mooney | but i think live migration was added later | 17:27 |
| sean-k-mooney | cold migration works in more cases the live migration can | 17:27 |
| sean-k-mooney | for excampl you could cold migreate and ironic server or rsd system in principal | 17:28 |
| sean-k-mooney | live migrating either would be much harder even if it was possibel which it is not today | 17:28 |
| sean-k-mooney | so haveing cold migration which is a lower barrier to entry be the default makes sense | 17:28 |
| sean-k-mooney | efried: i dont have a linke but extra is inteded to optional dependcies or tools that are need for building but not using a project. we do not need postgress or mysql to run our tests we can use sqlite to execute all the db test | 17:31 |
| sean-k-mooney | we opertunistalcly try to use mysql or postgerss if they are installed but its not a requirement | 17:31 |
| sean-k-mooney | what stephens change would allow you to do in theroyr is git clone nova and use tox to run the test with out having to intall mysql or postgress client | 17:32 |
| sean-k-mooney | although i think https://review.opendev.org/#/c/677475/3/tox.ini might undo that last bit | 17:32 |
| sean-k-mooney | so i like the patch in general but if https://review.opendev.org/#/c/677475/3/tox.ini is till forcing them to be install i dont see any point in doing it | 17:33 |
| sean-k-mooney | ideally we would not force them to be installd and isnead modify the gate job to install them | 17:33 |
| mriedem | efried: i think you might be reading too much into the names of those Migration.migration_type values | 17:36 |
| mriedem | yay in the beginning there was migrate and migrateLive, and thou said to thee, | 17:37 |
| mriedem | let there be a type https://review.opendev.org/#/c/181110/6/nova/objects/migration.py | 17:37 |
| mriedem | and it was so | 17:37 |
| sean-k-mooney | is it bad that my mind goes to monty python first | 17:38 |
| sean-k-mooney | robustify-evacuate you say. how did that go | 17:40 |
| dansmith | sean-k-mooney: you should have seen it before robustification | 17:43 |
| *** macz has joined #openstack-nova | 17:44 | |
| sean-k-mooney | i unfortunetly did see some of it. but i was only a year or so into dealing with openstack at that point and was still mainly looking at neturon at that point | 17:45 |
| sean-k-mooney | we used to pass a log more dicts of random garbage around back then | 17:45 |
| mriedem | and lo before the times of the great robustification there was much deleting of servers and gnashing of teeth | 17:46 |
| sean-k-mooney | a lot of the legacy systems in nova are kind of like rabbitmq. sure they can cause you pain but they generally work well enough that its more pain to replace them with something else so we dont. better the dragons you know and all that | 17:48 |
| *** macz has quit IRC | 17:49 | |
| *** macz has joined #openstack-nova | 17:52 | |
| *** eharney has joined #openstack-nova | 18:03 | |
| *** boxiang has quit IRC | 18:06 | |
| *** boxiang has joined #openstack-nova | 18:06 | |
| *** boxiang has quit IRC | 18:09 | |
| *** boxiang has joined #openstack-nova | 18:09 | |
| *** tbachman has quit IRC | 18:09 | |
| *** boxiang has quit IRC | 18:10 | |
| *** boxiang has joined #openstack-nova | 18:10 | |
| *** zhubx has joined #openstack-nova | 18:22 | |
| *** boxiang has quit IRC | 18:25 | |
| artom | *sigh* | 18:32 |
| artom | new_ != _new | 18:32 |
| efried | artom: neither one is reserved or anything, right? | 18:39 |
| *** trident has quit IRC | 18:40 | |
| artom | efried, hah, no, this isn't Python, it's migration context prefix grumbling | 18:40 |
| *** trident has joined #openstack-nova | 18:40 | |
| *** gbarros has quit IRC | 18:41 | |
| dansmith | sean-k-mooney: you know we're looking for your input here right? https://review.opendev.org/#/c/635669/39/nova/compute/resource_tracker.py | 18:42 |
| efried | dansmith: In order to compare two OVOs, it is (apparently) necessary to overrid __eq__. Is nova.objects.numa.all_things_equal the accepted way to do this? | 18:44 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Introduce live_migration_claim() https://review.opendev.org/635669 | 18:44 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: New objects for NUMA live migration https://review.opendev.org/634827 | 18:44 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: add support for augmenting migrate_data with info from claims https://review.opendev.org/634828 | 18:44 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: add support for updating NUMA-related XML on the source https://review.opendev.org/635229 | 18:44 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | 18:44 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | 18:44 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional test for NUMA live migration https://review.opendev.org/672595 | 18:44 |
| dansmith | efried: well, I don't know without looking, but let me tell you why it's not built in: | 18:44 |
| dansmith | efried: since OVOs can have only part of the database state loaded, we don't provide a default equality operator because determining equality with missing things may be doable in some cases and not others | 18:45 |
| * artom writes a func test for rollback while CI runs on ^^^ | 18:45 | |
| dansmith | efried: and with a nested object the parent and child may have differing rules there | 18:45 |
| efried | dansmith: ack, thanks for the explanation. | 18:46 |
| mriedem | true story, i was writing some ovo compare stuff for cross cell resize and had masked bugs because something was passing b/c one of the objects didn't have a field loaded | 18:46 |
| dansmith | efried: so for an object with primitive field types, then yes that should work, but if it has any nested objects, it won't | 18:46 |
| openstackgerrit | Merged openstack/nova master: Update SDK fixture for openstacksdk 0.35.0 https://review.opendev.org/678237 | 18:46 |
| dansmith | efried: for some objects, "does the row id match" is good enough, and for others, you need to compare the contents in more depth | 18:47 |
| *** brault has joined #openstack-nova | 18:51 | |
| efried | dansmith: so for this Resource thing, we're going to be asking the virt driver to populate the (local) resources associated with a provider every time RT update runs (so on periodics, instance ops, etc). I would think we want to avoid writing the same records back to the database every iteration. | 18:53 |
| efried | Which means we need to be able to compare with existing | 18:53 |
| efried | what decides how much of the object gets loaded from the db? | 18:53 |
| dansmith | efried: the code that loads it from the db | 18:54 |
| dansmith | efried: this is a performance-hating blob-based object right? in that case, it always comes out whole | 18:54 |
| efried | The touchpoint with the db is that InstanceExtra gets a resources field which is of type ResourceList | 18:55 |
| *** brault has quit IRC | 18:56 | |
| efried | so ultimately I guess we need to be able to compare a ResourceList we get/build from the virt driver with whatever's in Instance.extras | 18:56 |
| *** markvoelker has quit IRC | 18:57 | |
| *** xek_ has joined #openstack-nova | 18:58 | |
| *** ivve has joined #openstack-nova | 18:59 | |
| sean-k-mooney | dansmith: yes although i was not sure if we already covered it on irc | 19:00 |
| sean-k-mooney | ill resond on the revew https://review.opendev.org/#/c/635669/39/nova/compute/resource_tracker.py | 19:01 |
| *** slaweq_ has quit IRC | 19:01 | |
| *** tbachman has joined #openstack-nova | 19:08 | |
| *** markvoelker has joined #openstack-nova | 19:11 | |
| artom | In an artificial func test scenario with 2 compute hosts, we boot 2 instances. The first one can only fit on one host, the second one on either. Can we be 100% it will be placed on the host that does *not* have the first instance? | 19:11 |
| artom | 100% sure? | 19:11 |
| artom | IOW, in that contrived scenario, how deterministic is scheduling? | 19:11 |
| dansmith | if it can't fit, it should be fully deterministic | 19:12 |
| artom | dansmith, the first instance, yeah. But the second instance can go on either host. | 19:12 |
| dansmith | there are a couple tests that add a weigher which always prefers the first host to help it be deterministic in the order it selects even the first one | 19:12 |
| dansmith | artom: so you're asking if the second one, which will *still* fit will go to one specifically? | 19:12 |
| *** tbachman has quit IRC | 19:13 | |
| sean-k-mooney | if you boot it in that order tehn its determistic | 19:13 |
| artom | dansmith, yeah. The scheduler's choice is "host A with another instance" or "host B with no instances" | 19:13 |
| dansmith | sean-k-mooney: not if host_subset_size != 1 | 19:13 |
| sean-k-mooney | if you wan to force it to the other host i thike we added --host option recently | 19:13 |
| dansmith | artom: I would rather you write the test to be more specific than that if possible | 19:13 |
| artom | dansmith, so force it with a weigher to a host? | 19:14 |
| artom | It doesn't change the mechanics of the test, just the numbers that we assert. | 19:14 |
| sean-k-mooney | is the scenarion that the ifrst instace can oly fit on the first host and the second can fit on either(but not after the first is booted) | 19:14 |
| artom | sean-k-mooney, no, the second can fit on either regardless of the first instance. | 19:14 |
| sean-k-mooney | oh | 19:14 |
| dansmith | artom: yeah, I don't really like altering the behavior in a functional test, but scheduling determinism bugs in functional tests are annoying, | 19:15 |
| dansmith | artom: especially if they work now, but start to fail the obscure assumptions made in a year | 19:15 |
| sean-k-mooney | then use https://github.com/openstack/nova-specs/blob/master/specs/train/approved/add-host-and-hypervisor-hostname-flag-to-create-server.rst | 19:15 |
| artom | dansmith, well, what I currently have is just an if | 19:15 |
| artom | if hostA; assert hostA things; else assert hostB things | 19:15 |
| artom | (really just the number of the pinned CPUs) | 19:15 |
| dansmith | artom: meaning the test tolerates it being either place? | 19:15 |
| sean-k-mooney | i think we merged the code for that a few weeks ago right | 19:15 |
| artom | dansmith, exactly | 19:15 |
| dansmith | artom: kinda seems like that's begging for testing both scenarios, no? | 19:16 |
| dansmith | artom: can you boot a third like the second that will have to go in the spot not taken by the second instance and then assert it al? | 19:16 |
| artom | dansmith, yeah, but why? | 19:16 |
| artom | This is all just a "forcing" function to make sure that when I live migrate it, the XML will have to get updates | 19:16 |
| dansmith | artom: well, if you put it on the second host, I'll ask if you're sure it would have worked if co-located | 19:17 |
| sean-k-mooney | we are litrally adding a feature this cycle to allow this. | 19:17 |
| dansmith | artom: and if it is co-located, I'll ask if it would have worked if by itself :D | 19:17 |
| dansmith | artom: ah, right, forgot about the migration | 19:17 |
| dansmith | artom: so you an migrate it to and fro and cover both cases right? | 19:17 |
| sean-k-mooney | and its marked as implemented https://blueprints.launchpad.net/nova/+spec/add-host-and-hypervisor-hostname-flag-to-create-server | 19:17 |
| dansmith | i.e. migrate twice instead of boot thrice | 19:17 |
| artom | It's about forcing the migration to move the server to a cell that as different CPU numbers | 19:18 |
| *** tbachman has joined #openstack-nova | 19:18 | |
| sean-k-mooney | you are trying to force the xml to be regenerated | 19:18 |
| artom | Yeah | 19:18 |
| *** damien_r has joined #openstack-nova | 19:18 | |
| dansmith | artom: right, so two migrations will cover the bases of migrating to an empty host which could take anything and to a host where it can't (as long as it didn't choose the same by luck) | 19:18 |
| artom | The idea is to have a server with 2 NUMA cells: 2 CPUs and 3 CPUs | 19:18 |
| artom | And another with just a single cell: 2 CPUs | 19:19 |
| sean-k-mooney | so why not use the host/hypervioer_host name parmaters on server create | 19:19 |
| artom | But such that the 2-CPU cells have different physical CPUs in them | 19:19 |
| sean-k-mooney | it will allow you slect the host and it will use the schdler to validate it is sutable | 19:19 |
| artom | So host2: [0, 1], host3: [0, 1, 2] [3, 4] | 19:19 |
| artom | "Fill" the 3-CPU cell with a first instance | 19:20 |
| artom | Then boot the 2-CPU instance | 19:20 |
| dansmith | artom: is this for a tempest test or a functional test? | 19:20 |
| artom | I don't care where it goes, as long as I live migrate it, its CPU pins have changed | 19:20 |
| artom | dansmith, functional | 19:20 |
| dansmith | okay, I think we have other tests that migrate twice just to make sure we hit both hosts, regardless of the ordering | 19:21 |
| dansmith | which I prefer to modifying the scheduler, if it will cover us | 19:21 |
| dansmith | targeting the build works too I guess, but it's also not the more realistic case you're trying to replicate | 19:21 |
| sean-k-mooney | we currently dont have any live migration test the suceed and use the fakelibvirt dirver in teh functional tests | 19:21 |
| artom | Yey, I'm first :( | 19:22 |
| sean-k-mooney | so ignrogn the lack of live migration tests i dont see what the big deal is | 19:22 |
| sean-k-mooney | you stare each compute node with a different config with non overlaping cpus | 19:22 |
| sean-k-mooney | in the dedicate_cpu_set | 19:23 |
| sean-k-mooney | then spawn a singel vm | 19:23 |
| sean-k-mooney | and migrated to the ohter node | 19:23 |
| sean-k-mooney | and your done | 19:23 |
| artom | Uhh, literally *ALL* tests are failing with "keystoneauth1.exceptions.auth_plugins.MissingAuthPlugin: An auth plugin is required to determine endpoint URL" | 19:23 |
| artom | I think devstack bork | 19:23 |
| *** sridharg has quit IRC | 19:23 | |
| sean-k-mooney | you need to rebase | 19:23 |
| sean-k-mooney | this is fixed on master | 19:23 |
| artom | ack | 19:24 |
| *** efried has quit IRC | 19:24 | |
| artom | When I push the new func test | 19:24 |
| sean-k-mooney | are you goint to do ^ | 19:24 |
| artom | I gotta run for a daughter school thing (yes, she's starting shool!) | 19:24 |
| artom | sean-k-mooney, can't have different configs in func tests | 19:25 |
| sean-k-mooney | yes you can | 19:25 |
| artom | Nope, CONF is global | 19:25 |
| sean-k-mooney | you need to chente the config flag and start the compute agent serially | 19:25 |
| sean-k-mooney | it does not have to be with mocks and we read most of the values on start up | 19:26 |
| dansmith | sean-k-mooney: that only works for a subset of cases | 19:26 |
| * artom -> away | 19:26 | |
| sean-k-mooney | dansmith: yes but i think it will work in this case. i do not belview we parse the config on ever iteration of the perodic tasks. | 19:30 |
| sean-k-mooney | if we do we should stop and cache it. we dont support change this config at runtime. | 19:31 |
| *** damien_r has quit IRC | 19:31 | |
| dansmith | it's not just the periodic that matters of course | 19:35 |
| dansmith | % fgrep 'CONF.' -r nova/virt/hardware.py | wc -l | 19:36 |
| dansmith | 9 | 19:36 |
| sean-k-mooney | true but i think it would work in this case. the simple way would to test this would be just try it | 19:36 |
| sean-k-mooney | at least 7 of those shoudl one be check at star up i think | 19:37 |
| dansmith | I'm just saying, this stuff is spread everywhere | 19:38 |
| sean-k-mooney | i think all of them are just used in init | 19:38 |
| sean-k-mooney | ya it is | 19:38 |
| sean-k-mooney | i just think/hope this is simpler to do then it first seams | 19:38 |
| sean-k-mooney | if we test this simple with conf setting that would be better then create a complicate test setup | 19:39 |
| sean-k-mooney | anyway i replied in https://review.opendev.org/#/c/635669/39/nova/compute/resource_tracker.py@304 not sure if that is what ye wanted. | 19:44 |
| sean-k-mooney | ill redeploy with artoms code shortly and start testing it again. | 19:45 |
| *** gbarros has joined #openstack-nova | 19:51 | |
| *** mlavalle has quit IRC | 19:54 | |
| *** xek_ has quit IRC | 20:00 | |
| *** efried has joined #openstack-nova | 20:02 | |
| TheJulia | mriedem: remember that seemingly weird rebalance race on stable/stein that we spotted with ironic last week? I just spotted it on master branch testing https://d01b2e57f0a56cb7edf0-b6bc206936c08bb07a5f77cfa916a2d4.ssl.cf5.rackcdn.com/678298/4/check/ironic-tempest-ipa-wholedisk-direct-tinyipa-multinode/92c65ac/compute1/logs/screen-n-cpu.txt.gz <-- nova 19.1.0 :\ | 20:02 |
| sean-k-mooney | TheJulia: implying that backport the other fix may not resolve the issue | 20:04 |
| sean-k-mooney | assuming on master it still had the db issue | 20:04 |
| TheJulia | indeed :( | 20:05 |
| sean-k-mooney | those errors seam to be indicating that the RP is not found | 20:06 |
| *** mlavalle has joined #openstack-nova | 20:06 | |
| sean-k-mooney | im not sing the db conclit on the compute node uuid | 20:06 |
| TheJulia | seeing? | 20:06 |
| *** efried has quit IRC | 20:07 | |
| sean-k-mooney | yes :) | 20:07 |
| sean-k-mooney | i dropped the ee it seams | 20:07 |
| TheJulia | no worries, my brain does similar things | 20:08 |
| TheJulia | so without the conflict.... hmmmm | 20:08 |
| * TheJulia wonders if now is a good time for scotch | 20:08 | |
| sean-k-mooney | but it might be that the other compute node is not recreating the rp | 20:09 |
| sean-k-mooney | or it is but we race | 20:10 |
| TheJulia | I'll go with it is but we race | 20:11 |
| * TheJulia will get something to sip on and continue digging shortly | 20:12 | |
| *** efried has joined #openstack-nova | 20:13 | |
| *** xek has joined #openstack-nova | 20:13 | |
| mriedem | umm yeah doesn't look like the same issue, no db conflicts on restart of the service that i see | 20:15 |
| mriedem | i see this: | 20:15 |
| mriedem | Aug 26 18:41:38.367990 ubuntu-bionic-rax-ord-0010443319 nova-compute[747]: INFO nova.compute.resource_tracker [None req-a894abee-a2f1-4423-8ede-2a1b9eef28a4 None None] ComputeNode eabf1567-cb9f-4a93-b77e-44a80ddd50b0 moving from ubuntu-bionic-rax-ord-0010443317 to ubuntu-bionic-rax-ord-0010443319 | 20:15 |
| mriedem | Aug 26 18:41:38.818412 ubuntu-bionic-rax-ord-0010443319 nova-compute[747]: INFO nova.compute.resource_tracker [None req-a894abee-a2f1-4423-8ede-2a1b9eef28a4 None None] ComputeNode 61dbc9c7-828b-4c42-b19c-a3716037965f moving from ubuntu-bionic-rax-ord-0010443317 to ubuntu-bionic-rax-ord-0010443319 | 20:16 |
| mriedem | and then a bunch of errors about resource provider not found | 20:16 |
| mriedem | Aug 26 18:41:38.992017 ubuntu-bionic-rax-ord-0010443319 nova-compute[747]: ERROR nova.scheduler.client.report [None req-a894abee-a2f1-4423-8ede-2a1b9eef28a4 None None] [req-208b7275-abc4-4445-9551-8edbce49e0e1] Failed to retrieve traits from placement API for resource provider with UUID 61dbc9c7-828b-4c42-b19c-a3716037965f. Got 404: {"errors": [{"status": 404, "request_id": "req-208b7275-abc4-4445-9551-8edbce49e0e1", "detail" | 20:17 |
| mriedem | he resource could not be found.\n\n No resource provider with uuid 61dbc9c7-828b-4c42-b19c-a3716037965f found: No resource provider with uuid 61dbc9c7-828b-4c42-b19c-a3716037965f found ", "title": "Not Found"}]}. | 20:17 |
| sean-k-mooney | it looks like the perodic taks the cleans up allocation ran before the compute node rp was recreated by the other compute service | 20:17 |
| mriedem | the traceback hits get_provider_tree_and_ensure_root which should ensure the provider exists, so i'm not sure why it wouldn't, | 20:18 |
| mriedem | unless it's only looking at it's local provider tree cache and that is stale or something | 20:18 |
| TheJulia | that does seem to possibly be the case. just glancing at records for eabf1567-cb9f-4a93-b77e-44a80ddd50b0, it is all over both compute logs | 20:23 |
| TheJulia | in the same time window it looks like... | 20:24 |
| *** david-lyle has quit IRC | 20:25 | |
| *** slaweq has joined #openstack-nova | 20:29 | |
| openstackgerrit | Merged openstack/nova master: Remove descriptions of nonexistent hacking rules https://review.opendev.org/678462 | 20:29 |
| TheJulia | hmmmmm.... I wonder | 20:36 |
| *** jmlowe has quit IRC | 20:38 | |
| *** dklyle has joined #openstack-nova | 20:39 | |
| openstackgerrit | Merged openstack/nova master: tests: Split NUMA object tests https://review.opendev.org/672336 | 20:41 |
| TheJulia | I'm starting to wonder, at least digging at it, if things did't moderately correct themselves at least resource tracker wise as time went on. Looks like everything changed to no longer reserved around 18:46, placement reports still one node reserved 3 minutes later | 20:47 |
| *** nweinber has quit IRC | 20:55 | |
| *** slaweq has quit IRC | 21:01 | |
| *** macz has quit IRC | 21:05 | |
| *** lpetrut has joined #openstack-nova | 21:07 | |
| *** CeeMac has quit IRC | 21:07 | |
| *** xek has quit IRC | 21:07 | |
| mriedem | i think this is bogus | 21:09 |
| mriedem | Aug 26 18:41:38.992017 ubuntu-bionic-rax-ord-0010443319 nova-compute[747]: ERROR nova.scheduler.client.report [None req-a894abee-a2f1-4423-8ede-2a1b9eef28a4 None None] [req-208b7275-abc4-4445-9551-8edbce49e0e1] Failed to retrieve traits from placement API for resource provider with UUID 61dbc9c7-828b-4c42-b19c-a3716037965f. Got 404: {"errors": [{"status": 404, "request_id": "req-208b7275-abc4-4445-9551-8edbce49e0e1", "detail" | 21:09 |
| mriedem | he resource could not be found.\n\n No resource provider with uuid 61dbc9c7-828b-4c42-b19c-a3716037965f found: No resource provider with uuid 61dbc9c7-828b-4c42-b19c-a3716037965f found ", "title": "Not Found"}]}. | 21:09 |
| mriedem | because right before that we get inventories and aggregates for the provider | 21:09 |
| mriedem | Aug 26 18:41:38.881026 ubuntu-bionic-rax-ord-0010443319 nova-compute[747]: DEBUG nova.scheduler.client.report [None req-a894abee-a2f1-4423-8ede-2a1b9eef28a4 None None] Refreshing inventories for resource provider 61dbc9c7-828b-4c42-b19c-a3716037965f {{(pid=747) _refresh_associations /opt/stack/nova/nova/scheduler/client/report.py:761}} | 21:09 |
| mriedem | Aug 26 18:41:38.953685 ubuntu-bionic-rax-ord-0010443319 nova-compute[747]: DEBUG nova.scheduler.client.report [None req-a894abee-a2f1-4423-8ede-2a1b9eef28a4 None None] Refreshing aggregate associations for resource provider 61dbc9c7-828b-4c42-b19c-a3716037965f, aggregates: None {{(pid=747) _refresh_associations /opt/stack/nova/nova/scheduler/client/report.py:770}} | 21:09 |
| mriedem | unless at the same time, the other host is deleting the resource provider | 21:12 |
| mriedem | because the driver no longer reports it there | 21:12 |
| sean-k-mooney | well that could happen. we are not syncronising any of this | 21:12 |
| mriedem | Aug 26 18:41:38.832749 ubuntu-bionic-rax-ord-0010443317 nova-compute[19290]: INFO nova.compute.manager [None req-d5a9c4b6-f197-4f6c-8b12-8f736bbdb11c None None] Deleting orphan compute node 6 hypervisor host is 61dbc9c7-828b-4c42-b19c-a3716037965f, nodes are set([u'1d23263a-31d4-49d9-ad68-be19219c3bae', u'be80f41d-73ed-46ad-b8e4-cefb0193de36', u'f3c6add0-3eda-47d9-9624-c1f73d488066', u'2c909342-b5dc-4203-b9cb-05a8f29c6c35', u | 21:12 |
| mriedem | 1f5d8-8b39-4d03-8423-8e8404128ece']) Aug 26 18:41:38.962237 ubuntu-bionic-rax-ord-0010443317 nova-compute[19290]: INFO nova.scheduler.client.report [None req-d5a9c4b6-f197-4f6c-8b12-8f736bbdb11c None None] Deleted resource provider 61dbc9c7-828b-4c42-b19c-a3716037965f | 21:13 |
| mriedem | ^ is the other host | 21:13 |
| mriedem | yup, 18:41:38 | 21:13 |
| mriedem | so the new host starts refreshing it's RT, the old host deletes the provider, then the new host blows up in the RT, but eventually corrects itself on the next run | 21:14 |
| *** trident has quit IRC | 21:14 | |
| mriedem | dId SoMeOnE sAy EtCd?!?! | 21:14 |
| sean-k-mooney | ... and follow cinders lead | 21:14 |
| mriedem | i'm joking | 21:15 |
| sean-k-mooney | i know | 21:15 |
| sean-k-mooney | a distribute lock could prevent this by deadlocking the sytem so we dont get there | 21:16 |
| sean-k-mooney | we proably need to cacht the error and if it happens remove the compute node form whatever data structure we were iterating over and move on | 21:17 |
| mriedem | i think the bug is that we stash the node on the new host in RT.compute_nodes, then hit the failure, and the next pass in the RT periodic task we see the node is already in the RT.compute_nodes and don't attempt to go through the _update and flush to placement flow, so the new host doesn't recreate the provider until inventory changes | 21:18 |
| mriedem | so find the thing on the new host and store it here https://github.com/openstack/nova/blob/71478c3eedd95e2eeb219f47460603221ee249b9/nova/compute/resource_tracker.py#L513 | 21:18 |
| mriedem | blow up here https://github.com/openstack/nova/blob/71478c3eedd95e2eeb219f47460603221ee249b9/nova/compute/resource_tracker.py#L516 | 21:19 |
| mriedem | and on the next pass find the node here https://github.com/openstack/nova/blob/71478c3eedd95e2eeb219f47460603221ee249b9/nova/compute/resource_tracker.py#L546 | 21:19 |
| mriedem | but don't call _update | 21:19 |
| mriedem | until inventory changes | 21:19 |
| mriedem | hence the RP not showing up again for awhile | 21:19 |
| mriedem | TheJulia: ^ | 21:19 |
| mriedem | a la https://review.opendev.org/#/q/I9fa1d509a3de405d6246fb8670612c65c10cc93b | 21:19 |
| *** trident has joined #openstack-nova | 21:20 | |
| *** markvoelker has quit IRC | 21:21 | |
| *** lpetrut has quit IRC | 21:22 | |
| mriedem | oh actually, | 21:22 |
| mriedem | because we put the provider back into the cache and then failed, we don't attempt to create it later in the new host either | 21:22 |
| mriedem | so the provider tree cache is hiding the fact the provider doesn't actually exist anymore | 21:23 |
| mriedem | efried: fun stuff ^ | 21:24 |
| mriedem | guess i'll just report a bug for now | 21:25 |
| *** trident has quit IRC | 21:25 | |
| *** igordc has quit IRC | 21:26 | |
| efried | mriedem: Under what circumstances do we put a provider into the cache and then fail? | 21:27 |
| efried | is this during update_from_provider_tree somehow? | 21:28 |
| mriedem | yeah, it will probably be more clear when i post the bug report | 21:28 |
| * efried awaits | 21:29 | |
| *** trident has joined #openstack-nova | 21:33 | |
| *** tonyb[m] has quit IRC | 21:35 | |
| *** dosaboy has joined #openstack-nova | 21:35 | |
| *** dosaboy has quit IRC | 21:39 | |
| *** dosaboy has joined #openstack-nova | 21:39 | |
| *** tonyb has quit IRC | 21:39 | |
| mriedem | efried: TheJulia: https://bugs.launchpad.net/nova/+bug/1841481 | 21:39 |
| openstack | Launchpad bug 1841481 in OpenStack Compute (nova) "Race during ironic re-balance corrupts local RT ProviderTree and compute_nodes cache" [Medium,Triaged] | 21:39 |
| * TheJulia reads | 21:39 | |
| *** dosaboy has quit IRC | 21:40 | |
| *** dosaboy has joined #openstack-nova | 21:41 | |
| TheJulia | funnn..... and by fun, I think I mean anti-fun | 21:42 |
| openstackgerrit | sean mooney proposed openstack/nova master: Libvirt: report storage bus traits https://review.opendev.org/666914 | 21:44 |
| openstackgerrit | sean mooney proposed openstack/nova master: libvirt: use domain capabilities to get supported device models https://review.opendev.org/666915 | 21:44 |
| openstackgerrit | sean mooney proposed openstack/nova master: Add transform_image_metadata request filter https://review.opendev.org/665775 | 21:44 |
| fungi | can anyone double-check for me that bug 1837252 was introduced in os-vif 1.12.0? sean-k-mooney said it "was introduced in stein" but there were multiple releases in master before it branched to stable/stein | 21:45 |
| openstack | bug 1837252 in os-vif stein "IFLA_BR_AGEING_TIME of 0 causes flooding across bridges" [High,In progress] https://launchpad.net/bugs/1837252 - Assigned to sean mooney (sean-k-mooney) | 21:45 |
| fungi | just want to be sure before i go sending this to mitre, so i don't have to update them with errata | 21:45 |
| sean-k-mooney | i think the exact commit is in the bug somewhere but ya ill check | 21:45 |
| sean-k-mooney | it was right at the end of the cycle | 21:45 |
| fungi | oh, sorry if i overlooked it, checking | 21:46 |
| fungi | 5027ce8 | 21:46 |
| sean-k-mooney | https://github.com/openstack/os-vif/commit/1f6fed6a69e9fd386e421f3cacae97c11cdd7c75 | 21:47 |
| sean-k-mooney | so 1.15.0 | 21:47 |
| fungi | aha, thanks | 21:47 |
| sean-k-mooney | thats the commit where i swaped the linux bridge plugin to use the common code intoduced in the previous commit | 21:48 |
| fungi | any chance you can skim the rest of the impact description in comment #17 there and see if it makes sense, aside from the affected versions obviously being an incorrect guess? | 21:48 |
| * sean-k-mooney clicks | 21:49 | |
| sean-k-mooney | im not actully sure if it disable mac learning or if mac learning is still enabled but macs are never unlearnt | 21:50 |
| sean-k-mooney | both would result in flooding in different situations | 21:50 |
| sean-k-mooney | form the inital bug i woudl have suspected that former | 21:52 |
| sean-k-mooney | sorry the latter | 21:52 |
| fungi | hrm, yeah at first i thought it simply filled the mac table, but logan- suggested it disabled learning | 21:53 |
| sean-k-mooney | the fdb does have macs | 21:53 |
| sean-k-mooney | maybe ip route has some info | 21:53 |
| sean-k-mooney | or ip tools | 21:53 |
| fungi | hence the diff between the impact descriptions in comment #15 and #17 | 21:53 |
| sean-k-mooney | brctl setageing <brname> <time> sets the ethernet (MAC) address ageing time, in seconds. After <time> seconds of not having seen a frame coming from a certain address, the bridge | 21:54 |
| sean-k-mooney | will time out (delete) that address from the Forwarding DataBase (fdb). | 21:54 |
| sean-k-mooney | so i would think it never delete the macs | 21:55 |
| sean-k-mooney | so it would only flood if the mac was learned on two ports | 21:55 |
| mriedem | melwitt: dansmith: if either of you are around can we move this stein backport series through please? https://review.opendev.org/#/q/topic:bug/1839560+branch:stable/stein | 21:55 |
| sean-k-mooney | that would only happen if the vm was migrated | 21:55 |
| fungi | sean-k-mooney: or if the table filled i guess? | 21:56 |
| sean-k-mooney | i dont think there is a limit on the table size | 21:56 |
| fungi | there is bound to be some practical limit on table size, whether it is reachable or not is another matter ;) | 21:56 |
| sean-k-mooney | maybe there is but there the docs on linux bridge are rather limited | 21:56 |
| dansmith | mriedem: hit me tomorrow and I will, I gotta run | 21:57 |
| fungi | not sure what sort of layer 2 filters are generally in place to prevent poisoning, but could an instance spoof frames from random macs to overrun the bridge table? or spoof specific macs for other victim instances so they show up on multiple ports? | 21:58 |
| dansmith | nm, I'll just do it now | 21:58 |
| * dansmith feels mriedem's guilty stare | 21:58 | |
| sean-k-mooney | fungi: neutron adds iptbales rules to prevent mac spoofing and nova used to also | 21:59 |
| melwitt | heh, dansmith wins | 21:59 |
| fungi | sean-k-mooney: yeah, and i think that would be a more general concern beyond the scope of this particular bug anyway | 21:59 |
| mriedem | dansmith: <3 | 22:00 |
| fungi | sean-k-mooney: so anyway, if the mac really does need to appear on multiple ports (e.g. because the instance was migrated) then that does significantly reduce the risk | 22:00 |
| * dansmith blows mriedem a kiss as he rides off into the sunset | 22:00 | |
| fungi | sean-k-mooney: i'll update the bug with a summary of the discussion here and see if anyone contests these assertions | 22:01 |
| sean-k-mooney | reduce yes but not eliminate so its still a concern | 22:02 |
| sean-k-mooney | cool | 22:02 |
| fungi | absolutely | 22:03 |
| sean-k-mooney | the primary concern is really the external netwrok / cross tenant shared networks | 22:03 |
| fungi | but good to capture the exact risk scenario in the impact description and advsory | 22:03 |
| fungi | thanks! | 22:03 |
| sean-k-mooney | security groups are still applied so within a tenatn network its less concerning but still not good | 22:03 |
| sean-k-mooney | no worries | 22:03 |
| *** markvoelker has joined #openstack-nova | 22:05 | |
| *** trident has quit IRC | 22:05 | |
| *** markvoelker has quit IRC | 22:10 | |
| melwitt | sean-k-mooney: I just read through your discussion about the os-vif bug, does the bug only affect VMs that are being migrated or can it affect more? sorry I didn't quite get that part | 22:10 |
| *** damien_r has joined #openstack-nova | 22:11 | |
| sean-k-mooney | melwitt: i originally thought i would only affect vms that were moved as my understand of setting ageing=0 was that all learned mac address would be permement and never age out | 22:12 |
| sean-k-mooney | although in the bug the assertion was made that ageing=0 disable mac learning entirly and casues all packets to flood | 22:13 |
| sean-k-mooney | i am not sure which is correct | 22:13 |
| sean-k-mooney | the scope of the bug is obviosly change signifcantly based on that | 22:13 |
| *** trident has joined #openstack-nova | 22:14 | |
| melwitt | hm, yeah. ok | 22:14 |
| fungi | sean-k-mooney: looking more closely at the example in the bug description, it looks like the macs there are only duplicated between master and vlan1 entries for the same port? | 22:15 |
| sean-k-mooney | that confused me a bit | 22:16 |
| fungi | i don't see the same macs appearing on different port numbers | 22:16 |
| *** mlavalle has quit IRC | 22:16 | |
| sean-k-mooney | for linxu bridge we shoudl not be adding vlans to the tap devices | 22:16 |
| fungi | so i think it's just vlan0/trunk and vlan1/tagged both being listed? | 22:17 |
| sean-k-mooney | my understnading was the the linux bridge driver applied the vlan to the veth pari that connted the tenant bridge to the br-int | 22:18 |
| sean-k-mooney | so the ports on the tenant network should be untagged | 22:18 |
| fungi | also it's unclear to me whether those entries are marked permanent because they were added through some other mechanism than learning, or whether they're showing up as permanent because their ageing time is 0 | 22:19 |
| sean-k-mooney | i can deploy with linux bridge and try to see what happens | 22:19 |
| sean-k-mooney | or perment because the port exist on the bridge | 22:20 |
| fungi | if you get a chance that would be stellar, but if not i'll dig in the docs when i get a moment | 22:20 |
| sean-k-mooney | it looks to me like the only macs in teh fdb are for the local ports | 22:21 |
| sean-k-mooney | implying that the macs are never learned | 22:21 |
| sean-k-mooney | for non local porst | 22:21 |
| sean-k-mooney | nomaly we would expec the eno1.102 port to havee the macs of vms on other host listed | 22:22 |
| sean-k-mooney | if that is the case the desciption in 17 is closer to being corerect as we would flood any packet to a non local port | 22:23 |
| sean-k-mooney | which would mean i need a 2 node deployment to test | 22:23 |
| fungi | aha, so it could be that the problem is traffic addressed to a non-local port is flooded to all ports including the local ports | 22:24 |
| sean-k-mooney | which i guess i can do i jsut need to set it up but i likely wont get to it until tomorow | 22:24 |
| sean-k-mooney | fungi: yep | 22:24 |
| sean-k-mooney | that what im suspecting | 22:24 |
| fungi | that would make more sense in light of the example bridge table and the other suggestions in the ensuing bug discussion | 22:25 |
| fungi | thanks! | 22:25 |
| *** eharney has quit IRC | 22:26 | |
| *** BjoernT_ has quit IRC | 22:30 | |
| sean-k-mooney | artom: do you mind if i rebase your series to fix the keystone_auth issues | 22:30 |
| sean-k-mooney | artom: just running the test locally to confirm | 22:30 |
| *** ivve has quit IRC | 22:32 | |
| sean-k-mooney | ill quickly deploy and test it locally then sign off for the night | 22:32 |
| *** mriedem has quit IRC | 22:36 | |
| openstackgerrit | sean mooney proposed openstack/nova master: Introduce live_migration_claim() https://review.opendev.org/635669 | 22:39 |
| openstackgerrit | sean mooney proposed openstack/nova master: New objects for NUMA live migration https://review.opendev.org/634827 | 22:39 |
| openstackgerrit | sean mooney proposed openstack/nova master: LM: add support for augmenting migrate_data with info from claims https://review.opendev.org/634828 | 22:39 |
| openstackgerrit | sean mooney proposed openstack/nova master: LM: add support for updating NUMA-related XML on the source https://review.opendev.org/635229 | 22:39 |
| openstackgerrit | sean mooney proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | 22:39 |
| openstackgerrit | sean mooney proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | 22:39 |
| openstackgerrit | sean mooney proposed openstack/nova master: Functional test for NUMA live migration https://review.opendev.org/672595 | 22:39 |
| *** dklyle has quit IRC | 22:40 | |
| *** jmlowe has joined #openstack-nova | 22:42 | |
| *** rcernin has joined #openstack-nova | 22:45 | |
| *** tbachman has quit IRC | 22:51 | |
| *** tkajinam has joined #openstack-nova | 23:02 | |
| *** tbachman has joined #openstack-nova | 23:04 | |
| *** slaweq has joined #openstack-nova | 23:11 | |
| *** dave-mccowan has quit IRC | 23:11 | |
| sean-k-mooney | artom: it looks like your getting closer. did you modify the hugepage code. | 23:16 |
| *** slaweq has quit IRC | 23:16 | |
| sean-k-mooney | i think migrtion with cpu pinning is now working but i think the hugepage migration is not | 23:16 |
| sean-k-mooney | i will test it more corretly tomorrwo by force the cpus set to not over lap on each host and actully allocating hugepages rther then forcing small pages with hw:mem_page_size=small | 23:17 |
| *** dklyle has joined #openstack-nova | 23:26 | |
| *** dave-mccowan has joined #openstack-nova | 23:33 | |
| *** tbachman has quit IRC | 23:35 | |
| openstackgerrit | Merged openstack/nova master: Add test for create server with integer AZ https://review.opendev.org/678328 | 23:36 |
| *** markvoelker has joined #openstack-nova | 23:41 | |
| *** avolkov has quit IRC | 23:45 | |
| *** markvoelker has quit IRC | 23:46 | |
| artom | sean-k-mooney, ugh, I had local changes I wanted to push | 23:51 |
| artom | You just rebased them? | 23:51 |
| artom | I guess I'll keep my local branch, rebase, then push | 23:52 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!