| *** mriedem_afk is now known as mriedem | 00:03 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add InstanceAction/Event create() method https://review.opendev.org/614036 | 00:14 |
|---|---|---|
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Instance.hidden field https://review.opendev.org/631123 | 00:14 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add TargetDBSetupTask https://review.opendev.org/627892 | 00:14 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add CrossCellMigrationTask https://review.opendev.org/631581 | 00:14 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Execute TargetDBSetupTask https://review.opendev.org/633853 | 00:14 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_dest compute method https://review.opendev.org/633293 | 00:14 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtDestTask https://review.opendev.org/627890 | 00:14 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_source compute method https://review.opendev.org/634832 | 00:14 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova.compute.utils.delete_image https://review.opendev.org/637605 | 00:14 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtSourceTask https://review.opendev.org/627891 | 00:14 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Refactor ComputeManager.remove_volume_connection https://review.opendev.org/642183 | 00:14 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add power_on kwarg to ComputeDriver.spawn() method https://review.opendev.org/642590 | 00:14 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add finish_snapshot_based_resize_at_dest compute method https://review.opendev.org/635080 | 00:14 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add FinishResizeAtDestTask https://review.opendev.org/635646 | 00:14 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Destination.allow_cross_cell_move field https://review.opendev.org/614035 | 00:14 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Execute CrossCellMigrationTask from MigrationTask https://review.opendev.org/635668 | 00:14 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Plumb allow_cross_cell_resize into compute API resize() https://review.opendev.org/635684 | 00:14 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Filter duplicates from compute API get_migrations_sorted() https://review.opendev.org/636224 | 00:14 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Support cross-cell moves in external_instance_event https://review.opendev.org/658478 | 00:14 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add cross-cell resize policy rule and enable in API https://review.opendev.org/638269 | 00:14 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Enable cross-cell resize in the nova-multi-cell job https://review.opendev.org/656656 | 00:14 |
| *** brault has quit IRC | 00:27 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add placement request filter for disabled computes https://review.opendev.org/654596 | 00:35 |
| *** hamzy has joined #openstack-nova | 00:38 | |
| *** tbachman has quit IRC | 00:48 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop pre-cinder 3.44 version compatibility https://review.opendev.org/621061 | 00:51 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Follow-up for I6a777b4b7a5729488f939df8c40e49bd40aec3dd https://review.opendev.org/664967 | 00:51 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove 'MultiattachSupportNotYetAvailable' exception https://review.opendev.org/651315 | 00:51 |
| openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Enable rebuild for instances in cell0 https://review.opendev.org/648686 | 01:06 |
| *** imacdonn has quit IRC | 01:09 | |
| *** imacdonn has joined #openstack-nova | 01:10 | |
| *** lbragstad has quit IRC | 01:11 | |
| *** bhagyashris has joined #openstack-nova | 01:15 | |
| *** tbachman has joined #openstack-nova | 01:16 | |
| bhagyashris | mriedem, Sean-k-mooney, alex_xu: Hi, I would like your insights on https://review.opendev.org/#/c/612626/ | 01:17 |
| openstackgerrit | Merged openstack/nova-specs master: Enable rebuild for instances in cell0 https://review.opendev.org/648686 | 01:26 |
| *** adriant has joined #openstack-nova | 01:40 | |
| adriant | Hello! Out of curiosity, was there ever a bug in Nova that stopped delete-on-terminate working specifically for volumes created from an instance snapshot? | 01:41 |
| adriant | ^ we have a case where a volume made from a volume snapshot, delete-on-terminate works. The volume just goes away with the instance | 01:42 |
| adriant | but a volume made from an instance snapshot, delete-on-terminate doesn't work. And the volume must be manually delete afterwards. | 01:42 |
| adriant | we are running an old version of nova, so just curious if this was a known bug, and if it ever got fixed, or if this is just some weird combination of failures somewhere on our deployment | 01:43 |
| openstackgerrit | Merged openstack/nova stable/stein: Noop CantStartEngineError in targets_cell if API DB not configured https://review.opendev.org/663030 | 01:44 |
| sean-k-mooney | not that i see in launchpad | 01:44 |
| openstackgerrit | Merged openstack/nova stable/stein: Fix python3 compatibility of rbd get_fsid https://review.opendev.org/664512 | 01:44 |
| adriant | yeah, I didn't find anything either | 01:45 |
| *** yaawang has quit IRC | 01:45 | |
| sean-k-mooney | to be clear your workflow is you boot a vm. create a snapshot then later boot another vm from the instnace shapshot with delete-on-terminate | 01:46 |
| adriant | yes | 01:46 |
| adriant | and if you instance make a volume snapshot of that instances root disk, and boot from that, it works as expected | 01:46 |
| adriant | your* | 01:46 |
| sean-k-mooney | that should be the same as a boot form a standard image really | 01:46 |
| adriant | I know :P | 01:47 |
| adriant | but somehow it isn't | 01:47 |
| *** gouthamr has quit IRC | 01:49 | |
| adriant | I'm just confirming that I can replicate it with a new instance | 01:50 |
| adriant | but I've seen a client do it | 01:50 |
| *** gouthamr has joined #openstack-nova | 01:52 | |
| *** mriedem has quit IRC | 01:54 | |
| *** rajinir has quit IRC | 01:55 | |
| yonglihe | mriedem, sean-k-mooney: what i got from discuss, 1) use UUID instead of name for querying orphans instances 2) checking metadata (which is already done per Sean's suggest), 3) clean up if possible , if db record gone, i don't how to do that. for reap/poweroff/delete, we already had configuration options. 4) re-use/extending current period task instead of create new one. | 01:58 |
| yonglihe | mriedem, sean-k-mooney: these changes seems quite a lot for this 2 patches. i want to make sure i'm on right track before i start it. | 01:59 |
| *** gouthamr has quit IRC | 02:02 | |
| *** gouthamr has joined #openstack-nova | 02:06 | |
| *** brinzhang has joined #openstack-nova | 02:07 | |
| *** gouthamr has quit IRC | 02:07 | |
| sean-k-mooney | yonglihe: you likely do not need to have 2 seperate patches. | 02:09 |
| sean-k-mooney | most of the chagnes in the first patch are not required. | 02:09 |
| sean-k-mooney | yonglihe: melwitt: was suggeting we might be able to jsut create a fake instance object and reuse the existing destroy method in the driver | 02:10 |
| sean-k-mooney | that occured to me als but i was not sure it that would be valid in all cases | 02:10 |
| sean-k-mooney | but yes 1.) we would prefer to use uuids instead of names, 2.) for the libvirt dirver check the metadata(which you have already done) 3.) determin if we can clean up local resoces like disk or ports on a vswitch. 4.) combine you chagnes with the existing periodic task as a new confg option reap_unknown | 02:12 |
| *** gouthamr has joined #openstack-nova | 02:14 | |
| *** gouthamr has quit IRC | 02:17 | |
| *** gouthamr has joined #openstack-nova | 02:19 | |
| *** ricolin has joined #openstack-nova | 02:20 | |
| *** threestrands has joined #openstack-nova | 02:23 | |
| yonglihe | sean-k-mooney: sure, seems we cloud move on, it's gonna drop a lot and come up with new one, though. about clean up local resources, i might need reach for help, might trouble you -:) | 02:27 |
| *** gouthamr has quit IRC | 02:29 | |
| sean-k-mooney | the concern is if we jsut delete the domain we would leave the root disk and vswitch ports behind. and in that case the operator would not know what instance they belonged or if it wa safe to remvoe them | 02:31 |
| sean-k-mooney | we might decide that is ok but if we can sanely clean up that would be nice to do | 02:32 |
| *** tbachman has quit IRC | 02:33 | |
| *** gouthamr has joined #openstack-nova | 02:35 | |
| *** tbachman has joined #openstack-nova | 02:36 | |
| yonglihe | we might need to add power_off_unknow in case admin want to keep them for investigating. | 02:38 |
| sean-k-mooney | yes i think that would be fine and in that case no cleanup is needed | 02:40 |
| *** gouthamr has quit IRC | 02:42 | |
| yonglihe | I gonna add them both. | 02:47 |
| *** gouthamr has joined #openstack-nova | 02:49 | |
| *** gouthamr has quit IRC | 02:54 | |
| *** tbachman has quit IRC | 02:55 | |
| *** gouthamr has joined #openstack-nova | 02:58 | |
| *** whoami-rajat has joined #openstack-nova | 03:05 | |
| *** gouthamr has quit IRC | 03:05 | |
| *** tonyb has quit IRC | 03:05 | |
| *** gouthamr has joined #openstack-nova | 03:10 | |
| *** cfriesen has quit IRC | 03:17 | |
| *** gouthamr has quit IRC | 03:27 | |
| *** gouthamr has joined #openstack-nova | 03:31 | |
| *** gouthamr has quit IRC | 03:33 | |
| *** tonyb has joined #openstack-nova | 03:36 | |
| *** gouthamr has joined #openstack-nova | 03:37 | |
| *** psachin has joined #openstack-nova | 03:40 | |
| openstackgerrit | sean mooney proposed openstack/nova master: [WIP] Libvirt: report storage bus traits https://review.opendev.org/666914 | 03:45 |
| openstackgerrit | sean mooney proposed openstack/nova master: [WIP] use domain capablites to get supported device models https://review.opendev.org/666915 | 03:45 |
| openstackgerrit | sean mooney proposed openstack/nova master: Add transform_image_metadata request filter https://review.opendev.org/665775 | 03:45 |
| *** gouthamr has quit IRC | 03:49 | |
| *** tbachman has joined #openstack-nova | 03:54 | |
| *** gouthamr has joined #openstack-nova | 03:58 | |
| *** gouthamr has quit IRC | 04:00 | |
| *** guozijn has joined #openstack-nova | 04:04 | |
| *** gouthamr has joined #openstack-nova | 04:06 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Fix test_flavors to run with correct microversion https://review.opendev.org/666536 | 04:09 |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add missing tests for flavor extra_specs mv 2.61 https://review.opendev.org/667600 | 04:10 |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Multiple API cleanup changes https://review.opendev.org/666889 | 04:10 |
| *** gouthamr has quit IRC | 04:10 | |
| *** gouthamr has joined #openstack-nova | 04:16 | |
| *** gouthamr has quit IRC | 04:17 | |
| *** gouthamr has joined #openstack-nova | 04:22 | |
| *** mkrai_ has joined #openstack-nova | 04:23 | |
| *** udesale has joined #openstack-nova | 04:24 | |
| *** _alastor_ has quit IRC | 04:25 | |
| *** gouthamr has quit IRC | 04:25 | |
| *** brault has joined #openstack-nova | 04:26 | |
| *** gouthamr has joined #openstack-nova | 04:30 | |
| *** mkrai_ has quit IRC | 04:33 | |
| *** gouthamr has quit IRC | 04:35 | |
| *** pcaruana has joined #openstack-nova | 04:36 | |
| *** mkrai_ has joined #openstack-nova | 04:39 | |
| *** guozijn has quit IRC | 04:41 | |
| *** gouthamr has joined #openstack-nova | 04:41 | |
| *** gouthamr has quit IRC | 04:50 | |
| *** gouthamr has joined #openstack-nova | 04:57 | |
| *** gouthamr has quit IRC | 05:06 | |
| *** gouthamr has joined #openstack-nova | 05:12 | |
| *** gouthamr has quit IRC | 05:19 | |
| *** gouthamr has joined #openstack-nova | 05:24 | |
| *** Luzi has joined #openstack-nova | 05:26 | |
| *** gouthamr has quit IRC | 05:31 | |
| *** gouthamr has joined #openstack-nova | 05:37 | |
| *** guozijn has joined #openstack-nova | 05:38 | |
| openstackgerrit | Merged openstack/nova master: Fix test_flavors to run with correct microversion https://review.opendev.org/666536 | 05:39 |
| *** bbowen__ has quit IRC | 05:42 | |
| *** ivve has quit IRC | 05:43 | |
| *** gouthamr has quit IRC | 05:44 | |
| *** luksky has joined #openstack-nova | 05:45 | |
| *** ratailor has joined #openstack-nova | 05:51 | |
| *** gouthamr has joined #openstack-nova | 05:51 | |
| *** gouthamr has quit IRC | 05:52 | |
| *** gouthamr has joined #openstack-nova | 05:58 | |
| *** slaweq has joined #openstack-nova | 06:00 | |
| *** guozijn has quit IRC | 06:01 | |
| *** guozijn has joined #openstack-nova | 06:02 | |
| *** gouthamr has quit IRC | 06:03 | |
| *** gouthamr has joined #openstack-nova | 06:08 | |
| *** lpetrut has joined #openstack-nova | 06:09 | |
| *** lpetrut has quit IRC | 06:09 | |
| *** lpetrut has joined #openstack-nova | 06:10 | |
| *** gouthamr has quit IRC | 06:11 | |
| *** gouthamr has joined #openstack-nova | 06:16 | |
| *** gouthamr has quit IRC | 06:16 | |
| *** dpawlik has joined #openstack-nova | 06:18 | |
| *** gouthamr has joined #openstack-nova | 06:22 | |
| *** ivve has joined #openstack-nova | 06:22 | |
| *** gouthamr has quit IRC | 06:29 | |
| *** threestrands has quit IRC | 06:34 | |
| *** gouthamr has joined #openstack-nova | 06:36 | |
| *** maciejjozefczyk has joined #openstack-nova | 06:41 | |
| *** gouthamr has quit IRC | 06:46 | |
| gmann | efried: can you re-approve this. it was rebased - https://review.opendev.org/#/c/667600/4 | 06:51 |
| *** mkrai has quit IRC | 06:52 | |
| *** ccamacho has joined #openstack-nova | 06:53 | |
| *** artom has joined #openstack-nova | 06:53 | |
| *** artom is now known as artom|gmtplus3 | 06:54 | |
| *** gouthamr has joined #openstack-nova | 06:58 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Multiple API cleanup changes https://review.opendev.org/666889 | 07:03 |
| *** gouthamr has quit IRC | 07:04 | |
| *** damien_r has joined #openstack-nova | 07:08 | |
| *** guozijn has quit IRC | 07:09 | |
| *** gouthamr has joined #openstack-nova | 07:12 | |
| *** guozijn has joined #openstack-nova | 07:13 | |
| *** gouthamr has quit IRC | 07:15 | |
| *** rcernin has quit IRC | 07:16 | |
| *** rdopiera has joined #openstack-nova | 07:17 | |
| *** tesseract has joined #openstack-nova | 07:17 | |
| *** guozijn has quit IRC | 07:18 | |
| *** gouthamr has joined #openstack-nova | 07:23 | |
| *** ttsiouts has joined #openstack-nova | 07:26 | |
| *** tesseract has quit IRC | 07:30 | |
| *** gouthamr has quit IRC | 07:30 | |
| *** tesseract has joined #openstack-nova | 07:32 | |
| *** itssurya has joined #openstack-nova | 07:34 | |
| *** gouthamr has joined #openstack-nova | 07:39 | |
| *** brault has quit IRC | 07:49 | |
| *** guozijn has joined #openstack-nova | 07:49 | |
| kashyap | lyarwood: Morning; stable/stein has merged, if you want to ACK its stable/rocky counterpart: https://review.opendev.org/#/c/667436/1 | 07:49 |
| *** gouthamr has quit IRC | 07:51 | |
| * kashyap goes to fiddle with Queens | 07:54 | |
| *** lpetrut has quit IRC | 07:56 | |
| *** gouthamr has joined #openstack-nova | 07:57 | |
| *** gouthamr has quit IRC | 07:59 | |
| *** brault has joined #openstack-nova | 07:59 | |
| lyarwood | kashyap: the same rules apply to stable as master fwiw so I can't just +2+W this, I'll recheck and leave it to others to +W. | 08:00 |
| *** ralonsoh has joined #openstack-nova | 08:01 | |
| kashyap | lyarwood: Okay, thought I was under the illusion that johnthetubaguy had also ACKed the stable/rocky one | 08:01 |
| lyarwood | kashyap: I think that was a different one no? | 08:03 |
| kashyap | lyarwood: Which one was it? Strangely I recall there were 2 ACKs from John on this same change | 08:03 |
| kashyap | Maybe he ACKed the Git/master one and I confused with it; /me goes to look | 08:03 |
| kashyap | (No, never mind; I'll chalk it up to poor memory) | 08:04 |
| *** brault has quit IRC | 08:04 | |
| *** gouthamr has joined #openstack-nova | 08:04 | |
| *** yonglihe has quit IRC | 08:05 | |
| kashyap | I find it dispiriting that even for multiple backports there's more burden of getting *two* "ACKs" from separate companies. | 08:06 |
| openstackgerrit | Kashyap Chamarthy proposed openstack/nova stable/queens: libvirt: Rework 'EBUSY' (SIGKILL) error handling code path https://review.opendev.org/668111 | 08:07 |
| *** psachin has quit IRC | 08:08 | |
| *** gouthamr has quit IRC | 08:09 | |
| *** gouthamr has joined #openstack-nova | 08:14 | |
| *** xek has joined #openstack-nova | 08:14 | |
| *** gouthamr has quit IRC | 08:17 | |
| *** gouthamr has joined #openstack-nova | 08:23 | |
| *** psachin has joined #openstack-nova | 08:25 | |
| *** gouthamr has quit IRC | 08:30 | |
| *** whoami-rajat has quit IRC | 08:34 | |
| *** gouthamr has joined #openstack-nova | 08:40 | |
| *** guozijn has quit IRC | 08:42 | |
| *** gouthamr has quit IRC | 08:47 | |
| *** gouthamr has joined #openstack-nova | 08:53 | |
| *** gouthamr has quit IRC | 08:56 | |
| *** jaosorior has quit IRC | 08:57 | |
| *** jaosorior has joined #openstack-nova | 09:00 | |
| *** gouthamr has joined #openstack-nova | 09:02 | |
| *** davidsha has joined #openstack-nova | 09:04 | |
| *** gouthamr has quit IRC | 09:05 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/stein: libvirt: flatten rbd images when unshelving an instance https://review.opendev.org/668118 | 09:07 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/rocky: libvirt: flatten rbd images when unshelving an instance https://review.opendev.org/668119 | 09:08 |
| *** gouthamr has joined #openstack-nova | 09:14 | |
| *** psachin has quit IRC | 09:16 | |
| *** gouthamr has quit IRC | 09:19 | |
| *** ricolin has quit IRC | 09:21 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: libvirt: flatten rbd images when unshelving an instance https://review.opendev.org/668123 | 09:22 |
| *** gouthamr has joined #openstack-nova | 09:23 | |
| *** gouthamr has quit IRC | 09:25 | |
| *** trident has quit IRC | 09:27 | |
| *** trident has joined #openstack-nova | 09:29 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/stein: DNM - Test rbd unshelve fix on stable stein https://review.opendev.org/668126 | 09:30 |
| *** gouthamr has joined #openstack-nova | 09:32 | |
| *** mkrai_ has quit IRC | 09:36 | |
| *** mkrai__ has joined #openstack-nova | 09:36 | |
| *** gouthamr has quit IRC | 09:37 | |
| *** gouthamr has joined #openstack-nova | 09:42 | |
| *** gouthamr has quit IRC | 09:43 | |
| openstackgerrit | zhufl proposed openstack/nova master: Fix invalid assertIsNone states https://review.opendev.org/668132 | 09:43 |
| *** mkrai__ has quit IRC | 09:44 | |
| *** jaosorior has quit IRC | 09:49 | |
| *** itssurya has quit IRC | 09:54 | |
| *** gouthamr has joined #openstack-nova | 09:54 | |
| *** ivve has quit IRC | 09:56 | |
| *** xek has quit IRC | 09:59 | |
| *** gouthamr has quit IRC | 10:00 | |
| *** gouthamr has joined #openstack-nova | 10:06 | |
| *** ociuhandu has joined #openstack-nova | 10:07 | |
| *** ociuhandu has quit IRC | 10:14 | |
| *** ociuhandu has joined #openstack-nova | 10:17 | |
| *** gouthamr has quit IRC | 10:18 | |
| *** gouthamr has joined #openstack-nova | 10:26 | |
| *** ccamacho has quit IRC | 10:30 | |
| *** ccamacho has joined #openstack-nova | 10:31 | |
| *** gouthamr has quit IRC | 10:35 | |
| openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Add flavor group https://review.opendev.org/663563 | 10:36 |
| openstackgerrit | Merged openstack/python-novaclient stable/rocky: Use SHA256 instead of MD5 in completion cache https://review.opendev.org/664025 | 10:38 |
| *** brinzhang has quit IRC | 10:38 | |
| *** shilpasd has joined #openstack-nova | 10:39 | |
| *** ivve has joined #openstack-nova | 10:40 | |
| *** gouthamr has joined #openstack-nova | 10:41 | |
| *** bbowen has joined #openstack-nova | 10:43 | |
| *** gouthamr has quit IRC | 10:43 | |
| openstackgerrit | Dakshina Ilangovan proposed openstack/nova-specs master: Spec: Provider config YAML file https://review.opendev.org/612497 | 10:45 |
| *** bhagyashris has quit IRC | 10:51 | |
| *** whoami-rajat has joined #openstack-nova | 10:51 | |
| *** gouthamr has joined #openstack-nova | 10:52 | |
| *** aloga has quit IRC | 10:53 | |
| *** mrch_ has joined #openstack-nova | 10:54 | |
| *** ttsiouts has quit IRC | 10:56 | |
| *** ttsiouts has joined #openstack-nova | 10:57 | |
| *** gouthamr has quit IRC | 10:57 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/rocky: DNM - Test rbd unshelve fix on stable rocky https://review.opendev.org/668141 | 11:03 |
| *** gouthamr has joined #openstack-nova | 11:06 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: DNM - Test rbd unshelve fix on stable queens https://review.opendev.org/668142 | 11:06 |
| *** ccamacho has quit IRC | 11:11 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: libvirt: Do not reraise DiskNotFound exceptions during resize https://review.opendev.org/660363 | 11:21 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: libvirt: flatten rbd images when unshelving an instance https://review.opendev.org/668123 | 11:21 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: DNM - Test rbd unshelve fix on stable queens https://review.opendev.org/668142 | 11:21 |
| *** tbachman has quit IRC | 11:29 | |
| *** yikun has quit IRC | 11:33 | |
| *** ratailor has quit IRC | 11:35 | |
| *** tbachman has joined #openstack-nova | 11:47 | |
| *** dpawlik has quit IRC | 12:15 | |
| efried | gmann: done | 12:17 |
| *** dpawlik has joined #openstack-nova | 12:20 | |
| *** udesale has quit IRC | 12:22 | |
| *** udesale has joined #openstack-nova | 12:23 | |
| *** mvkr has quit IRC | 12:29 | |
| *** lbragstad has joined #openstack-nova | 12:31 | |
| *** pcaruana has quit IRC | 12:40 | |
| *** pcaruana has joined #openstack-nova | 12:41 | |
| *** eharney has joined #openstack-nova | 12:53 | |
| openstackgerrit | Merged openstack/nova master: Fix invalid assertIsNone states https://review.opendev.org/668132 | 13:04 |
| *** ivve has quit IRC | 13:04 | |
| *** lbragstad has quit IRC | 13:08 | |
| *** lbragstad has joined #openstack-nova | 13:08 | |
| *** jdillaman has quit IRC | 13:13 | |
| *** tbachman has quit IRC | 13:15 | |
| *** KeithMnemonic has joined #openstack-nova | 13:21 | |
| *** lpetrut has joined #openstack-nova | 13:23 | |
| *** mriedem has joined #openstack-nova | 13:24 | |
| kashyap | johnthetubaguy: Can you please put this stable/rocky backport through? https://review.opendev.org/#/c/667436/ | 13:26 |
| kashyap | johnthetubaguy: You've ACKed the stable/stein one | 13:26 |
| kashyap | (And this, stable/rocky version, already has one +2) | 13:27 |
| kashyap | Or anyone else, if John is not around | 13:27 |
| *** jdillaman has joined #openstack-nova | 13:28 | |
| *** davidsha has quit IRC | 13:30 | |
| KeithMnemonic | is there anyone here that is maintaining or familiar with https://github.com/openstack/nova/blob/master/nova/virt/vmwareapi/ds_util.py | 13:32 |
| kashyap | KeithMnemonic: Your best bet is GKotton (who is not here on IRC) | 13:33 |
| efried | mriedem, dansmith: placement agg sync is automatic now, right? No need to run nova-manage placement sync-aggregates? https://review.opendev.org/#/c/667952/1/doc/source/reference/forbidden-aggregates.rst@44 | 13:34 |
| kashyap | KeithMnemonic: You might want to try his e-mail (gkotton@vmware.com). | 13:34 |
| dansmith | efried: yeah, the manage sync is for fixup and upgrades at this point I think | 13:34 |
| kashyap | [And of course, Cc the list, so others could learn, too.] | 13:34 |
| efried | thanks dansmith | 13:34 |
| KeithMnemonic | yes i was looking for him, we know each other so hopefully i can find him next week | 13:35 |
| KeithMnemonic | thanks kashyap | 13:35 |
| kashyap | KeithMnemonic: By "the list" I meant: openstack-discuss@lists.openstack.org | 13:36 |
| mriedem | efried: right what dan said, | 13:36 |
| mriedem | except if the api fails to remove a provider from an aggregate, sync_aggregates won't fix that | 13:36 |
| mriedem | sync_aggregates is only additive | 13:36 |
| KeithMnemonic | ok, i thought you meant wait for him to show up back here. i will send an email today | 13:36 |
| efried | also, mriedem, /me Remind / ping / harass re https://review.opendev.org/662881 (sdk spec) | 13:38 |
| mriedem | d'oh! | 13:39 |
| mriedem | can i hit snooze on that until after i take my kid to camp? | 13:39 |
| efried | of course. I can hit you up hourly | 13:39 |
| dansmith | cron | 13:40 |
| dansmith | it's they only way he'll learn. | 13:40 |
| *** BjoernT has joined #openstack-nova | 13:42 | |
| openstackgerrit | Merged openstack/nova master: Add missing tests for flavor extra_specs mv 2.61 https://review.opendev.org/667600 | 13:46 |
| *** mlavalle has joined #openstack-nova | 13:54 | |
| *** trident has quit IRC | 13:56 | |
| *** ratailor has joined #openstack-nova | 13:56 | |
| *** shilpasd has quit IRC | 13:57 | |
| *** trident has joined #openstack-nova | 13:58 | |
| *** Luzi has quit IRC | 14:02 | |
| *** davidsha has joined #openstack-nova | 14:08 | |
| efried | ugh, do we not have `openstack resource provider trait add` ? | 14:08 |
| *** liuyulong has joined #openstack-nova | 14:10 | |
| *** dpawlik has quit IRC | 14:10 | |
| *** lpetrut has quit IRC | 14:11 | |
| mriedem | efried: https://docs.openstack.org/osc-placement/latest/cli/index.html#trait-create | 14:24 |
| mriedem | osc verbs are create/set/unset/delete/list | 14:24 |
| mriedem | and /show | 14:24 |
| efried | Right, I mean "add this trait to resource provider X without fing with any of its existing traits" | 14:25 |
| efried | so I don't have to do trait list + add to that + trait set | 14:25 |
| mriedem | oh, yeah we have a few gaps in ux like that in osc-placement | 14:25 |
| mriedem | it's annoying, especially for things like adding inventory with a new resource class to a provider or allocations | 14:25 |
| efried | meanwhile, how tf do I get a compute node UUID? | 14:26 |
| mriedem | openstack --os-compute-api-version 2.53 hypervisor list | 14:26 |
| efried | by name? | 14:26 |
| efried | got it | 14:26 |
| efried | phew | 14:26 |
| mriedem | --matching <hostname> | 14:26 |
| mriedem | https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/hypervisor.html#hypervisor-list | 14:26 |
| * mriedem starts to charge a fee | 14:27 | |
| efried | openstack --os-compute-api-version 2.53 hypervisor show my-compute-name -f value -c id | 14:27 |
| efried | ? | 14:27 |
| mriedem | to get the id, probably -c ID | 14:28 |
| mriedem | node_uuid=$(openstack --os-compute-api-version 2.53 hypervisor show <hostname> -f value -c ID) | 14:28 |
| efried | `ID` wasn't working for me, but `id` does. | 14:29 |
| mriedem | i'm not totally sure osc is working with 2.53 everywhere yet | 14:29 |
| mriedem | ah ok | 14:29 |
| mriedem | i've noticed some inconsistencies with ID vs id in osc | 14:29 |
| efried | yeah, totally | 14:29 |
| mriedem | for image nad server it's ID i think | 14:29 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: WIP: Add rollback to heal port allocation https://review.opendev.org/668184 | 14:29 |
| mriedem | open a story bug | 14:29 |
| gibi | mriedem, efried: I hacked up the rollback code for heal port allocation. https://review.opendev.org/668184 Based on the code I feel I'm just pushing the human-interaction-needed problem one level deeper, when the rollback fails | 14:30 |
| mriedem | efried: there is a story for the trait append thing https://storyboard.openstack.org/#!/story/2005258 | 14:30 |
| mriedem | i knew it sounded familiar.... | 14:30 |
| efried | thanks | 14:30 |
| *** _alastor_ has joined #openstack-nova | 14:32 | |
| mriedem | gibi: i left some comments, | 14:33 |
| *** mrch_ has quit IRC | 14:34 | |
| mriedem | but i haven't fully thought through which is worse - the port with the binding:profile.allocation set to something when the allocation doesn't exist in neutron vs the allocation existing in neutron but the port binding profile not mapped to that provider | 14:34 |
| mriedem | *doesn't exist in placement | 14:34 |
| gibi | mriedem: if the rollback retry fails the is it OK to ask for the human to help? | 14:35 |
| gibi | I feel at the end we need the human anyhow | 14:35 |
| *** lbragstad has quit IRC | 14:36 | |
| *** artom|gmtplus3 has quit IRC | 14:37 | |
| gibi | if we set the allocation key in neutron without having the allocation placemen then we tell neutron to use a resource that is not really allocated. But the physical bandwidth anyhow was used even before we started to heal | 14:37 |
| mriedem | so the risk there is over-committing the resource right? | 14:38 |
| mriedem | b/c placement isn't tracking the allocation | 14:38 |
| gibi | yes, but the overcommit situation can already exists (hence the need of healing) | 14:38 |
| mriedem | then isn't that better than potentially having the allocations in placement w/o the neutron port binding profile tracking the allocation and if the admin screws up the manual steps, doubling the allocation by re-running the command? iow, it's no different than the situation they could already be in | 14:40 |
| mriedem | if you tried to run the command again we wouldn't heal that instance / port combo b/c the port would already say it's allocated when really it might not be | 14:40 |
| mriedem | i agree there is some amount of "we failed our main objective, and we failed to rollback, you need to step in now" if we get there | 14:40 |
| mriedem | but i would rather we at least *try* to rollback if possible | 14:41 |
| sean-k-mooney | i have not been following two closely but how do you determin currently a port needs healing? | 14:41 |
| mriedem | and it sounds like rolling back the allocation changes is harder since we merged the resources | 14:41 |
| mriedem | sean-k-mooney: it's a port with a resource_request and doesn't have an allocation set in the binding profile | 14:41 |
| sean-k-mooney | that could be a problem | 14:42 |
| mriedem | that makes me think, | 14:42 |
| mriedem | we should also be making sure the port is actually bound to a host right? | 14:42 |
| gibi | mriedem: rolling back the allocations can be done by saving what was the original allocation to restore | 14:42 |
| sean-k-mooney | what about cases where we set the qos policy on a network | 14:42 |
| mriedem | gibi: ...yeah but that could also get messy right b/c we could lose a race and our generation is off | 14:43 |
| sean-k-mooney | we only create the allocation if you pass in the port right | 14:43 |
| mriedem | then what do we do? | 14:43 |
| mriedem | rollling back the port binding profile allocation field seems easier to me | 14:43 |
| mriedem | sean-k-mooney: yes | 14:43 |
| gibi | mriedem: correct, if something else updates the allocation in between then we are rolling back to a wrong allocation | 14:43 |
| mriedem | we do'nt support creating ports on networks with a qos policy | 14:43 |
| gibi | mriedem: rolling back the neutron updated seems easy to me too | 14:44 |
| sean-k-mooney | at all? | 14:44 |
| gibi | easyier | 14:44 |
| sean-k-mooney | or we create the ports but dont request the allcotion | 14:44 |
| *** cfriesen has joined #openstack-nova | 14:44 | |
| sean-k-mooney | because we created the port in the compute node | 14:44 |
| mriedem | sean-k-mooney: this is the code that determines if we need to heal allocations for the port https://review.opendev.org/#/c/637955/28/nova/cmd/manage.py@1783 | 14:45 |
| mriedem | sean-k-mooney: we fail | 14:45 |
| mriedem | sean-k-mooney: https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L468 | 14:45 |
| *** lpetrut has joined #openstack-nova | 14:46 | |
| mriedem | gibi: so i think we agree that rolling back the port binding update is simpler than the allocation | 14:46 |
| gibi | mriedem: good point about port bound to a host. But can it be a port with device_id=instance_uuid that is not bound? | 14:46 |
| mriedem | and i'd prefer we include a rollback | 14:46 |
| * gibi is slow | 14:46 | |
| mriedem | gibi: "But can it be a port with device_id=instance_uuid that is not bound?" that i'm not sure about | 14:47 |
| mriedem | sean-k-mooney: ^ | 14:47 |
| sean-k-mooney | ... ok was an api breakage on upgrade but i understand why it was done | 14:47 |
| mriedem | sean-k-mooney: oh i think we can, | 14:47 |
| mriedem | because of shelve offload | 14:47 |
| *** ratailor has quit IRC | 14:47 | |
| mriedem | a shelved instance still has its ports and volumes | 14:47 |
| mriedem | but those ports and volumes aren't "bound" to a host | 14:47 |
| sean-k-mooney | yes shelve offloaded would still have the device id set | 14:47 |
| gibi | ack | 14:47 |
| gibi | then I have to check for boundness as well | 14:47 |
| mriedem | does heal_allocations filter out instances that aren't on a host.... | 14:48 |
| mriedem | it should implicitly, | 14:48 |
| mriedem | because it's using instance.node | 14:48 |
| *** xek has joined #openstack-nova | 14:48 | |
| *** panda has quit IRC | 14:48 | |
| sean-k-mooney | gibi: you can filter py ports with vif_type!=vif-unbound | 14:48 |
| gibi | yeah it checks instance.node then | 14:48 |
| gibi | so we don't need an extra vif_type!=vif-unbound check | 14:48 |
| *** panda has joined #openstack-nova | 14:48 | |
| sean-k-mooney | ok i guess that makes sense | 14:49 |
| gibi | if we don't know the where the instance runs then we don't know which RP tree need to be targeted with the healing | 14:49 |
| gibi | mriedem: I don't get your comment at https://review.opendev.org/#/c/668184/1/nova/cmd/manage.py@1891 | 14:50 |
| sean-k-mooney | yep. so we whould never need to heal offloaded instances | 14:50 |
| mriedem | now i'm not so sure where we unbind the port on shelve offload, | 14:50 |
| gibi | mriedem: is it about trying the rollback for each port even if one of it fails? | 14:50 |
| mriedem | because https://github.com/openstack/nova/blob/324da0532f3b59aa16233a93a260d289e55860fb/nova/compute/manager.py#L5168 is a noop for neutron | 14:50 |
| mriedem | gibi: yes | 14:51 |
| mriedem | right now you'd stop on the first port update that fails | 14:51 |
| mriedem | i'm saying we should try to clean all and fail at the end | 14:51 |
| gibi | mriedem: OK got it, that could be done | 14:51 |
| mriedem | https://github.com/openstack/nova/blob/324da0532f3b59aa16233a93a260d289e55860fb/nova/network/neutronv2/api.py#L3183 | 14:52 |
| mriedem | so are ports unbound when the instance is shelved offloaded.... | 14:52 |
| sean-k-mooney | maybe in driver.destroy | 14:53 |
| sean-k-mooney | ill go check | 14:53 |
| *** ttsiouts has quit IRC | 14:53 | |
| sean-k-mooney | but they might not be | 14:53 |
| sean-k-mooney | destroy should at least unplug them | 14:53 |
| *** ttsiouts has joined #openstack-nova | 14:54 | |
| mriedem | right | 14:54 |
| sean-k-mooney | acttuly it cant unbind | 14:54 |
| mriedem | driver.destroy just does the unplug, the virt drivers never mess with port bindings | 14:54 |
| sean-k-mooney | because we use destoy in reboot | 14:54 |
| *** ttsiouts has quit IRC | 14:54 | |
| sean-k-mooney | and we dont ubind there | 14:54 |
| mriedem | anyway for gibi's heal_allocation changes it doesn't really matter b/c we wouldn't get that far anyway for a shelved offloaded instance b/c we won't try to heal allocations on an instance that doesn't have a host est | 14:54 |
| mriedem | *set | 14:55 |
| gibi | ack | 14:55 |
| sean-k-mooney | right its just something to look into to see if we have a latent bug | 14:56 |
| sean-k-mooney | i would expect it to happen as part of moving it ot offloaded | 14:56 |
| *** Conqueror has left #openstack-nova | 14:57 | |
| *** mkrai_ has joined #openstack-nova | 14:57 | |
| sean-k-mooney | but i dont think it would break anything either | 14:57 |
| gibi | so in summary. I will add the rollback of neutron updates. But I only add a todo to possible rollback-retry | 14:57 |
| gibi | mriedem: is that works for you ? | 14:57 |
| gibi | mriedem: or you are a bit more on the paranoid side :) | 14:57 |
| mriedem | the backoff retry could be a follow up | 14:58 |
| gibi | mriedem: OK | 14:58 |
| mriedem | try to stagger these changes as much as possible | 14:59 |
| mriedem | since this is a large change | 14:59 |
| sean-k-mooney | oh other edgecase. do we currently allow attaching ports with resouce requests | 14:59 |
| mriedem | sean-k-mooney: no | 14:59 |
| sean-k-mooney | cool | 14:59 |
| gibi | :) | 14:59 |
| *** jangutter has quit IRC | 14:59 | |
| gibi | mriedem: I can split out the helper functions from the current commit if that helps | 14:59 |
| mriedem | gibi: sure | 15:00 |
| mriedem | sean-k-mooney: this is where that would fail https://github.com/openstack/nova/blob/324da0532f3b59aa16233a93a260d289e55860fb/nova/compute/api.py#L4408 | 15:00 |
| sean-k-mooney | cool. AttachInterfaceWithQoSPolicyNotSupported is slightly missleading but it makes sense | 15:01 |
| sean-k-mooney | we allow attaching ports with QoS polices jut not with min bandwith qos policies | 15:01 |
| sean-k-mooney | e.g. DSCP is fine as is max bandwith | 15:01 |
| sean-k-mooney | its really we dont allow attaching with resource requests. | 15:02 |
| gibi | sean-k-mooney: file a bug, I'm happy to amend the message of AttachInterfaceWithQoSPolicyNotSupported exception with a more specific message | 15:02 |
| mriedem | efried: the reason i didn't jump on https://storyboard.openstack.org/#!/story/2005258 was because of the osc semantic patterns to follow for other osc commands as mentioned in the comments - i gets a bit hairy | 15:04 |
| mriedem | rather than just like a simple --append or --remove | 15:04 |
| sean-k-mooney | gibi: sure i could but we will also be changing that at some point too right | 15:04 |
| sean-k-mooney | e.g. we will eventrually allow it if the host can fulfil the request but like not until U+ | 15:04 |
| gibi | sean-k-mooney: based on my current speed of progress with the server move operation patches it will be far in the future | 15:04 |
| gibi | sean-k-mooney: but yes, in theory I would like to get the support for attach interface if the request fitts on the current host | 15:05 |
| *** lpetrut has quit IRC | 15:05 | |
| sean-k-mooney | well i know that neutorn want to start modeing ip pools in placment and use resouce resquest in the future | 15:05 |
| sean-k-mooney | so i expect it to become more common | 15:06 |
| gibi | yeah | 15:06 |
| mriedem | are you talking about for routed networks? | 15:06 |
| mriedem | modeling routed networks | 15:06 |
| sean-k-mooney | not just routed netwrok but that is one usecase | 15:06 |
| mriedem | i.e. https://review.opendev.org/#/c/656885/ | 15:06 |
| mriedem | that's been kicked around for many years and is going to be complicated i imagine | 15:07 |
| mriedem | we don't even have integration testing for the routed networks support that is documented in the neutron docs | 15:07 |
| * gibi feels sometimes that bandwidth is already too complicated | 15:07 | |
| mriedem | https://docs.openstack.org/neutron/latest/admin/config-routed-networks.html | 15:08 |
| sean-k-mooney | i think they also wanted to supprot using sharing aggrates to model things like floating ips and what host can recive an ip form which subnet pool | 15:08 |
| mriedem | well the good news is pop up teams officially exist now so all of this work can get done! | 15:08 |
| sean-k-mooney | ya without a cross project effort this is hard to make progress on | 15:09 |
| mriedem | i was being a sarcastic jerk but yeah | 15:09 |
| sean-k-mooney | hehe well i have tried to do enough stuff that crosses the nova neutron boundry that i know without getting buyin form both core teams up front and finding people to care about it it will fail | 15:11 |
| *** whoami-rajat has quit IRC | 15:11 | |
| sean-k-mooney | anyway thats a tangent | 15:11 |
| mriedem | lyarwood: i guess you got busy https://review.opendev.org/#/q/topic:bug/1653953+(status:open+OR+status:merged) | 15:11 |
| *** rajinir has joined #openstack-nova | 15:12 | |
| lyarwood | mriedem: busy spamming yes, hopefully that's what you had in mind. | 15:13 |
| mriedem | it is | 15:13 |
| lyarwood | wonderful, there are also DNM changes actually testing things on each branch btw | 15:14 |
| lyarwood | I'll add them to the same topic now to make it clear | 15:14 |
| mriedem | oh i see | 15:14 |
| mriedem | nova DNM changes that depend on the related devstack-plugin-ceph change per branch that depeds on the backport per nova branch.... | 15:15 |
| lyarwood | and around and around we go | 15:15 |
| lyarwood | but that should allow the fix to land then the change to enable the test without everything blowing up like it did with my cinder migrate series. | 15:16 |
| mriedem | this is going to be a pain for cross-cell resize, i'm not sure how i'm going to handle that | 15:16 |
| *** mkrai_ has quit IRC | 15:16 | |
| *** igordc has joined #openstack-nova | 15:17 | |
| lyarwood | I really need to go over the spec and code for that, are you using shelve/unshelve there? | 15:17 |
| mriedem | no, | 15:17 |
| mriedem | but for non-volume-backed servers, like shelve i'm creating a snapshot to get the disk from the source cell to the target cell | 15:18 |
| mriedem | and then spawning from that snapshot in the target cell and then delete the temporary snapshot, like unshelve | 15:18 |
| mriedem | so the fix here about checking the vm_state wont' work for cross-cell resize | 15:18 |
| *** whoami-rajat has joined #openstack-nova | 15:19 | |
| lyarwood | yeah true | 15:19 |
| mriedem | we could check something like the task_state used during that part of the cross-cell resize spawn, but that's going to be used during a normal resize as well, and i'm not sure if flatten should be used there | 15:19 |
| mriedem | we could temporarily hack something into the image meta or the instance system_metadata to be read by the driver, but again that's super hacky | 15:19 |
| lyarwood | You could make it optional for resize and spawn tbh | 15:19 |
| mriedem | optional how? | 15:20 |
| *** jistr is now known as jistr|off | 15:20 | |
| lyarwood | with a configurable, but it would slightly slow down spawns and resizes etc in any env that used it | 15:20 |
| lyarwood | so maybe not | 15:20 |
| mriedem | i think i know a bit better solution, | 15:20 |
| mriedem | the instance.migration_context should be set during a cross-cell resize, and from that we can get the Migration record which will have a cross_cell_move boolean on it so i'd just check that | 15:21 |
| mriedem | from the driver | 15:21 |
| mriedem | it's not fun but it's better than driver interface changes or hacking metadata temporarily or checking vm/task state | 15:21 |
| lyarwood | ack that works | 15:22 |
| lyarwood | we don't have a migration record for shelve/unshelve right? | 15:22 |
| lyarwood | IOW I haven't missed an easier way of fixing my issue? | 15:22 |
| lyarwood | ignore me, of course we don't. | 15:23 |
| mriedem | nope | 15:23 |
| dansmith | that's interesting, | 15:24 |
| dansmith | I've never thought about that | 15:24 |
| dansmith | a migration record would let us provide some unshelve (whilst offloaded) host affinity | 15:25 |
| dansmith | not sure how important that is really, but... | 15:25 |
| efried | dansmith, mriedem: How do y'all feel about reorganizing how request filters are enabled via conf? | 15:25 |
| efried | As it stands, it looks like we're headed for random options scattered around various groups. | 15:25 |
| efried | Wondering if it would make sense e.g. to collect them into [request_filter] $name = {enabled|disabled} | 15:25 |
| efried | or [scheduler] request_filters_enabled = $name[,...] | 15:25 |
| efried | or ... | 15:25 |
| efried | My hesitation is that that's really more of a dev-oriented structure; I don't know that the filters are really similar enough in spirit that that would make sense to the operator. | 15:25 |
| efried | Thoughts? | 15:25 |
| mriedem | dansmith: what do you mean? unshelve back to the original host? | 15:26 |
| dansmith | efried: I think we should aim for enabling behaviors not filters | 15:26 |
| *** xek has quit IRC | 15:26 | |
| dansmith | efried: and we should aim for most of those to be always on by default | 15:26 |
| mriedem | i'd like to avoid a list like enabled_filters | 15:27 |
| dansmith | yeah | 15:27 |
| dansmith | people enable or disable filters based on name, without really knowing that their behavior can be composite of multiple things | 15:28 |
| efried | okay, that's kind of where my head was going, thanks for the validation. | 15:28 |
| dansmith | mriedem: yes, for unshelve affinity back to the original host, after offload | 15:30 |
| mriedem | dansmith: we save off the original host in system_metadata | 15:30 |
| dansmith | oh, do we? | 15:30 |
| mriedem | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L5112 | 15:30 |
| dansmith | okay | 15:31 |
| dansmith | a migration record would make that a little less special-to-shelve-y and more a record of the move | 15:31 |
| dansmith | but anyway, just hadn't thought about it before | 15:31 |
| mriedem | yeah i'm not sure how much it matters either, | 15:32 |
| mriedem | the original host could be gone by the time you unshelve | 15:32 |
| dansmith | sure, it'd just be a weight thing | 15:32 |
| mriedem | right, was just going to say that, | 15:32 |
| mriedem | pass a weight hint to the scheduler | 15:32 |
| mriedem | if you can, do else no biggy | 15:32 |
| dansmith | right | 15:33 |
| dansmith | in "edge" cases, | 15:33 |
| dansmith | where you have two boxes at a site that are largely for hosting specific types of things, | 15:33 |
| dansmith | you'd have better image affinity | 15:33 |
| *** _alastor_ has quit IRC | 15:33 | |
| dansmith | if you're enforcing it then it doesn't matter | 15:34 |
| dansmith | anyway, | 15:34 |
| dansmith | just a thought that shelve could be more of a move for harmony with the other ops | 15:34 |
| *** jistr|off is now known as jistr | 15:42 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: Support filtering of hosts by forbidden aggregates https://review.opendev.org/667952 | 15:44 |
| efried | donnyd: Yo Mr. Docs, would you mind giving this a scan and seeing if it makes sense to you with your operator hat on? https://review.opendev.org/#/c/667952/2/doc/source/reference/forbidden-aggregates.rst | 15:46 |
| efried | mriedem: btw, that's what all those handy dandy CLIs were for ^ | 15:47 |
| mriedem | jesus that is a large patch, | 15:50 |
| mriedem | can that be split up? | 15:50 |
| donnyd | It reads fine to me. I would just use the same terminology everywhere. we call the function host-aggregate, so when referring to aggregate i would sub for host-aggregate | 15:50 |
| donnyd | or whatever the proper term is | 15:50 |
| mriedem | maybe it's mostly testing and docs... | 15:52 |
| efried | mriedem: It could maybe be split up, but it would be kind of contrived. It's pretty easy to read imo | 15:52 |
| efried | yes. | 15:52 |
| donnyd | Yea, i had no issue at all understanding what is trying to be conveyed | 15:53 |
| efried | donnyd: Thanks for the look. I actually don't know what the accepted term is, I probbaly flubbed several of those. | 15:53 |
| efried | donnyd: (mriedem was talking about the patch as a whole, not specifically the doc) | 15:54 |
| efried | I'll just go on record as saying I think this is a pretty darn cool feature. | 15:54 |
| * mriedem makes a note for when all of the regressions and weird side issues show up | 15:55 | |
| efried | Considering how weird and complex the implementation is, the external-facing usage model is pretty slick. | 15:55 |
| efried | the most effed up part is how you have to collect the existing traits before you can add the one you want to the host :P | 15:56 |
| efried | ...which of course has nothing to do with the forbidden aggregates feature. | 15:56 |
| dansmith | is this already merged? | 15:57 |
| mriedem | ok ok you win, we can add the PUT /resource_providers/{uuid}/traits/{trait_name} you've always wanted | 15:57 |
| dansmith | I'm confused about how you get the exclusion just because one flavor is tagged | 15:58 |
| mriedem | just like this fine mess https://developer.openstack.org/api-ref/compute/#server-metadata-servers-metadata | 15:58 |
| dansmith | oh jeez, | 15:58 |
| efried | dansmith: not merged | 15:58 |
| dansmith | this is implementation and docs too | 15:58 |
| dansmith | thought this was just a doc patch | 15:58 |
| efried | mriedem: I'm not pulling for an *API* to add a trait, though that wouldn't be terrible. I don't mind if osc-placement does a get-add-replace. What I mind is that the CLI user has to do that. That sucks. | 16:00 |
| efried | but we've already talked about this, hence the story | 16:00 |
| *** damien_r has quit IRC | 16:01 | |
| mriedem | efried: i was joking about the api | 16:05 |
| mriedem | the cli sugar should happen, it just needs a warm body, | 16:06 |
| efried | yuh | 16:06 |
| mriedem | speaking of which, i remember someone telling me recently they wanted to work on some code... | 16:06 |
| efried | I think I have zero client patches to my name | 16:06 |
| efried | actually, I might have an ironicclient patch or two :P | 16:06 |
| mriedem | consider this an opportunity for growth | 16:07 |
| *** davidsha has quit IRC | 16:07 | |
| mriedem | efried: dustinc: lots of comments and such in https://review.opendev.org/#/c/662881/ but +2; i'm going to post to the ML that i think other cores should be on board with this or nack it before EOD next tuesday for the spec review sprint | 16:11 |
| efried | I count | 16:12 |
| efried | 3 python-ironicclient | 16:12 |
| efried | 1 python-glanceclient | 16:12 |
| efried | 1 python-cinderclient | 16:12 |
| efried | And pretty sure none of those were actually CLI. | 16:12 |
| efried | thanks mriedem | 16:12 |
| *** tbachman has joined #openstack-nova | 16:14 | |
| *** efried is now known as fried_rolls | 16:15 | |
| *** ociuhandu has quit IRC | 16:16 | |
| *** mriedem is now known as mriedem_away | 16:18 | |
| *** wwriverrat has joined #openstack-nova | 16:21 | |
| *** udesale has quit IRC | 16:26 | |
| *** rdopiera has quit IRC | 16:30 | |
| sean-k-mooney | i havent reviewed https://review.opendev.org/#/c/662881/8/specs/train/approved/openstacksdk-in-nova.rst | 16:36 |
| sean-k-mooney | but im in favor of using the sdk in general | 16:36 |
| sean-k-mooney | ill try to read it while i wait for dinner but ill proably be +1 on it | 16:37 |
| sean-k-mooney | thanks for mailing the ist as it was not on my radar | 16:37 |
| *** BjoernT has quit IRC | 16:46 | |
| *** kmalloc is now known as needscoffee | 16:47 | |
| *** ralonsoh has quit IRC | 17:18 | |
| *** ricolin has joined #openstack-nova | 17:19 | |
| *** mriedem_away is now known as mriedem | 17:27 | |
| *** bbowen has quit IRC | 17:27 | |
| *** ivve has joined #openstack-nova | 17:38 | |
| *** ricolin has quit IRC | 17:45 | |
| *** damien_r has joined #openstack-nova | 17:48 | |
| dustinc | thanks for reviewing everyone | 17:49 |
| *** mrch_ has joined #openstack-nova | 17:50 | |
| *** whoami-rajat has quit IRC | 17:51 | |
| *** damien_r has quit IRC | 17:52 | |
| *** mvkr has joined #openstack-nova | 17:52 | |
| *** tesseract has quit IRC | 18:13 | |
| openstackgerrit | Merged openstack/nova stable/stein: Fix type error on call to mount device https://review.opendev.org/667921 | 19:00 |
| *** Sundar has joined #openstack-nova | 19:01 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Remove needs:* todo from deprecated APIs api-ref https://review.opendev.org/668234 | 19:07 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add StopWatch profiling to SchedulerManager.select_destinations https://review.opendev.org/668235 | 19:08 |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Remove needs:* todo from deprecated APIs api-ref https://review.opendev.org/668234 | 19:09 |
| *** bbowen has joined #openstack-nova | 19:14 | |
| *** whoami-rajat has joined #openstack-nova | 19:14 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add StopWatch profiling to SchedulerManager.select_destinations https://review.opendev.org/668235 | 19:15 |
| openstackgerrit | Merged openstack/nova stable/stein: Change InstanceFaultRollback handling in _error_out_instance_on_exception https://review.opendev.org/666638 | 19:16 |
| openstackgerrit | Merged openstack/nova stable/stein: Raise InstanceFaultRollback for UnableToMigrateToSelf from _prep_resize https://review.opendev.org/666639 | 19:17 |
| openstackgerrit | Merged openstack/nova stable/stein: Avoid unnecessary joins in InstanceGroup.get_hosts https://review.opendev.org/664271 | 19:17 |
| openstackgerrit | Merged openstack/nova stable/stein: Handle PortLimitExceeded in POST /servers/{server_id}/os-interface https://review.opendev.org/665513 | 19:17 |
| openstackgerrit | Merged openstack/nova stable/rocky: Share snapshot image membership with instance owner https://review.opendev.org/643853 | 19:17 |
| *** tbachman has quit IRC | 19:20 | |
| mriedem | fried_rolls: gah -f value -c ID fails me again http://logs.openstack.org/94/667994/2/check/nova-next/1519821/logs/devstack-gate-post_test_hook.txt.gz#_2019-06-28_01_50_02_145 | 19:20 |
| *** ivve has quit IRC | 19:28 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add integration testing for heal_allocations https://review.opendev.org/667994 | 19:29 |
| *** Sundar has quit IRC | 19:29 | |
| *** dklyle has quit IRC | 19:36 | |
| *** maciejjozefczyk has quit IRC | 19:39 | |
| *** Sundar has joined #openstack-nova | 19:42 | |
| mriedem | fried_rolls: what are your thoughts on doing an os-traits release next week? trying to workaround not having https://review.opendev.org/#/c/667209/ released is becoming a pain. | 19:45 |
| *** fried_rolls is now known as efried | 19:49 | |
| efried | mriedem: There's a release proposed right now, which includes ^. smcginnis is waiting for cdent (PTL) ack, but we could probably convince him to push it now if it's going to help you. | 19:49 |
| efried | mriedem: https://review.opendev.org/667111 | 19:50 |
| smcginnis | If it's holding up something, we can push it through. Even though it's a Friday and we generally try to avoid that kind of thing right before the weekend. | 19:50 |
| smcginnis | Just depends how time critical it is. | 19:51 |
| efried | this one's pretty benign, release-risk-wise | 19:51 |
| efried | like, really really benign | 19:51 |
| efried | nearly betegn | 19:51 |
| smcginnis | Yeah, I figured something like that would be low risk. | 19:51 |
| mriedem | cool thanks | 19:52 |
| efried | mriedem: you're still a placement core. Add your +1 and let's rip it. cdent will forgive us. | 19:53 |
| mriedem | done | 19:54 |
| efried | nice | 19:54 |
| efried | smcginnis: ^ | 19:54 |
| smcginnis | OK, I'll send 'er through. And watch out for the wrath of cdent. :D | 19:54 |
| *** tbachman has joined #openstack-nova | 19:55 | |
| efried | thank you sir | 19:55 |
| efried | mriedem: You'll need requirements bump before you can use it anyway. | 19:56 |
| mriedem | oh i know | 19:58 |
| mriedem | but that's automatic | 19:58 |
| mriedem | the proposal anyway | 19:58 |
| *** BjoernT has joined #openstack-nova | 19:59 | |
| *** BjoernT has quit IRC | 20:04 | |
| *** BjoernT has joined #openstack-nova | 20:08 | |
| mriedem | AHA https://bugs.launchpad.net/nova/+bug/1834691 | 20:09 |
| openstack | Launchpad bug 1834691 in OpenStack Compute (nova) "AttributeError: 'HostState' object has no attribute 'failed_builds'" [Undecided,New] | 20:09 |
| *** dklyle has joined #openstack-nova | 20:18 | |
| efried | mriedem: takes a while though. | 20:19 |
| efried | (requirements bot) | 20:20 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Init HostState.failed_builds https://review.opendev.org/668243 | 20:21 |
| mriedem | what doesn't take awhile is +2ing this trivial bug fix ^ melwitt dansmith efried | 20:21 |
| *** slaweq has quit IRC | 20:23 | |
| efried | mriedem: done. tbc the other thing is a separate issue? | 20:38 |
| mriedem | yup | 20:38 |
| mriedem | patch forthcoming | 20:38 |
| efried | mriedem: requirements bump https://review.opendev.org/#/c/668245/ | 20:39 |
| efried | smcginnis: https://review.opendev.org/#/c/668245/ do you have the power? | 20:39 |
| *** mvkr has quit IRC | 20:40 | |
| smcginnis | efried: Just waiting for zuul to report results on that. | 20:40 |
| efried | slave to process you are, sheesh | 20:40 |
| smcginnis | I can +2, but don't want to ninja that one through. | 20:40 |
| smcginnis | :D | 20:40 |
| Sundar | efried, cdent: Is there a master topic/blueprint for all nested magic stuff? I am looking for better query than https://review.opendev.org/#/q/project:openstack/placement+nested | 20:42 |
| efried | Sundar: It's all in a single spec | 20:43 |
| efried | Sundar: https://docs.openstack.org/placement/latest/specs/train/approved/2005575-nested-magic-1.html | 20:43 |
| efried | unless you mean a doc for all currently implemented features. That would be https://docs.openstack.org/placement/latest/usage/provider-tree.html | 20:43 |
| efried | plus the api-reference https://developer.openstack.org/api-ref/placement/ | 20:44 |
| Sundar | We've also had: https://review.opendev.org/657419. Some of your specs, since abandoned (like https://review.opendev.org/658510) were also useful to read | 20:44 |
| Sundar | It would be useful to have a single topic or blueprint for all of these. What do you think? | 20:45 |
| efried | Sundar: That patch was for the arbitrary suffix feature, which is included in the above spec. | 20:46 |
| efried | If you want to track all the patches, you can look at the story | 20:46 |
| efried | https://storyboard.openstack.org/#!/story/2005575 | 20:46 |
| efried | agree that topic branch story/$story-$task isn't as useful as just story/$story for searching purposes. | 20:47 |
| efried | but too late to modify topic for merged patches, sorry. | 20:47 |
| Sundar | NP, got it. Good to know that the spec and storyboard is all there is. Is there a spec/patch on Nova side for how it all comes together -- from flavor to Placement query? | 20:49 |
| Sundar | i.e. 'all there is to read for somebody to understand the whole shebang | 20:49 |
| *** mvkr has joined #openstack-nova | 20:53 | |
| Sundar | efried: ^ | 20:55 |
| efried | Sundar: There's not likely to be something that high level. And there's as yet no blueprint/spec I'm aware of on the nova side for exploiting these features. | 21:02 |
| melwitt | mriedem: the failed builds thing, my fave | 21:02 |
| *** mvkr has quit IRC | 21:07 | |
| *** slaweq has joined #openstack-nova | 21:11 | |
| *** Sundar has quit IRC | 21:12 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Update HostState.*_allocation_ratio earlier https://review.opendev.org/668252 | 21:14 |
| *** slaweq has quit IRC | 21:15 | |
| *** pcaruana has quit IRC | 21:16 | |
| *** dpawlik has joined #openstack-nova | 21:20 | |
| *** mvkr has joined #openstack-nova | 21:21 | |
| *** dpawlik has quit IRC | 21:22 | |
| *** tbachman has quit IRC | 21:32 | |
| *** tbachman has joined #openstack-nova | 21:33 | |
| mriedem | wow some fun debugging here | 21:53 |
| mriedem | on initial start of a comptue service, we'll create the compute node, then later try to update it here https://github.com/openstack/nova/blob/324da0532f3b59aa16233a93a260d289e55860fb/nova/compute/resource_tracker.py#L1010 | 21:53 |
| mriedem | if that fails, we've changed old_resources and then a subsequent run will return false from _resource_change and we'll go on to create the resource provider in placement and report inventory | 21:53 |
| mriedem | but the compute node is botched | 21:53 |
| mriedem | and then things in the scheduler blow up if we try to use that node | 21:54 |
| *** tbachman_ has joined #openstack-nova | 21:59 | |
| *** tbachman has quit IRC | 21:59 | |
| *** tbachman_ is now known as tbachman | 21:59 | |
| openstackgerrit | Merged openstack/nova stable/queens: libvirt: Do not reraise DiskNotFound exceptions during resize https://review.opendev.org/660363 | 22:00 |
| *** slaweq has joined #openstack-nova | 22:11 | |
| *** slaweq has quit IRC | 22:16 | |
| mriedem | efried: melwitt: would you think that on start of the compute service, if we hit a DBConnectionError here https://github.com/openstack/nova/blob/324da0532f3b59aa16233a93a260d289e55860fb/nova/compute/manager.py#L8129 and startup=true, we should bail and not let the service start? | 22:20 |
| mriedem | i guess it might correct itself... | 22:21 |
| mriedem | maybe that's why we don't bail out | 22:21 |
| efried | mriedem: it might. But in the meantime, everything is effed. Should at least disable the service at that point. | 22:21 |
| mriedem | heh, if you can....need a db connection to do that :) | 22:21 |
| melwitt | I know that in tripleo-land they don't want it to bail out because of puppet starting up services in different orders etc | 22:22 |
| melwitt | they want it to keep retrying the db (for at least a limited time) before bailing out | 22:22 |
| mriedem | yeah, i wonder if https://docs.openstack.org/nova/latest/configuration/config.html#database.use_db_reconnect would help since it's disabled by default i've never tried it | 22:23 |
| melwitt | I've dealt multiple times with tickets getting opened for not services not retrying [temporarily] upon startup | 22:23 |
| melwitt | mriedem: are you saying it is or isn't bailing out today? sorry I don't remember how it behaves at this point. some of our services retry db connection underneath because oslo.service does it. but for some reason that I don't remember, it didn't work on compute | 22:24 |
| mriedem | use_db_reconnect wouldn't help nova since it only tells a decorator in oslo.db it's ok to retry (safe_for_db_retry) which we don't use | 22:25 |
| mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#database.max_retries indicates that we'd have retries by default on connection failures | 22:26 |
| mriedem | with 10 seconds between retries by default (according to retry_interval) | 22:26 |
| melwitt | yeah. last I knew, compute does not retry and I can't remember if I figured out why it doesn't. because from all evidence, it should, the way all our other services do | 22:26 |
| melwitt | let me dig up that old moldy BZ | 22:27 |
| mriedem | it'd be whatever conductor the compute is talking to | 22:27 |
| mriedem | anyway, the connectoin failure doesn't kill the compute on startup so we can get into a weird state with what nova has in the compute nodes table and what is in placement for the provider | 22:28 |
| mriedem | i've debugged one issue as a result https://bugs.launchpad.net/nova/+bug/1834712 | 22:28 |
| openstack | Launchpad bug 1834712 in OpenStack Compute (nova) "ResourceTracker._update should restore previous old_resources value if ComputeNode.save fails" [Medium,Triaged] | 22:28 |
| efried | we need some kind of decorator to... like... @safely_connect | 22:28 |
| mriedem | heh | 22:28 |
| mriedem | i mean, i have a very unrealistic overloaded host situation here | 22:29 |
| melwitt | that bug report makes my head hurt | 22:29 |
| efried | and with that lightning jolt of sarcasm, I'm outta here. See y'all Monday. | 22:29 |
| melwitt | I think that situation would be realistic for our RDO cloud. I've seen so much weird crap happen with it | 22:29 |
| mriedem | yeah this is a critical section it turns out https://github.com/openstack/nova/blob/324da0532f3b59aa16233a93a260d289e55860fb/nova/compute/resource_tracker.py#L1005-L1012 | 22:30 |
| melwitt | like orphaned build request records etc because of spurious db errors | 22:30 |
| melwitt | instance records that live in both cell0 and cell1 (I still don't get how on earth that is possible to happen) | 22:30 |
| mriedem | and _resource_change mutates a thing in the RT that is checked on a later periodic run | 22:30 |
| *** factor has joined #openstack-nova | 22:31 | |
| melwitt | yeah, I see | 22:31 |
| melwitt | I'd be in favor of your rollback idea, or any other way to keep it in sync without killing the service | 22:31 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Update HostState.*_allocation_ratio earlier https://review.opendev.org/668252 | 22:33 |
| mriedem | yeah killing the service was an idea but not a good one b/c of unknown side effects | 22:33 |
| melwitt | yeah | 22:34 |
| *** brinzhang1 has joined #openstack-nova | 22:38 | |
| *** rajinir has quit IRC | 22:42 | |
| *** luksky has quit IRC | 22:52 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Restore RT.old_resources if ComputeNode.save() fails https://review.opendev.org/668263 | 23:05 |
| *** whoami-rajat has quit IRC | 23:13 | |
| openstackgerrit | Merged openstack/nova stable/stein: libvirt: flatten rbd images when unshelving an instance https://review.opendev.org/668118 | 23:22 |
| *** spatel has joined #openstack-nova | 23:25 | |
| *** spatel has quit IRC | 23:29 | |
| *** _alastor_ has joined #openstack-nova | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!