| *** dpawlik has joined #openstack-nova | 00:02 | |
| *** dpawlik has quit IRC | 00:06 | |
| *** tetsuro has joined #openstack-nova | 00:10 | |
| *** slaweq has joined #openstack-nova | 00:11 | |
| *** slaweq has quit IRC | 00:15 | |
| *** mriedem_away has quit IRC | 00:18 | |
| *** brinzhang has joined #openstack-nova | 00:37 | |
| *** Dinesh_Bhor has joined #openstack-nova | 01:11 | |
| *** hongbin has joined #openstack-nova | 01:19 | |
| alex_xu | gmann: sorry, I didn't back at that time yesterday, I have late meeting at yesterday lunchtime | 01:25 |
|---|---|---|
| gmann | alex_xu: no prob. | 01:27 |
| gmann | cfriesen: did not get completely about query on microversion need. are you saying to return 400 if image properties and flavor extra-spec if in rebuild/resize/create request ? | 01:28 |
| *** Dinesh_Bhor has quit IRC | 01:31 | |
| *** liuyulong has joined #openstack-nova | 01:32 | |
| *** Dinesh_Bhor has joined #openstack-nova | 01:37 | |
| *** openstackgerrit has joined #openstack-nova | 01:52 | |
| openstackgerrit | zhaodan7597 proposed openstack/nova master: Unable to delete volume when a vmware instance bfv is failed. https://review.openstack.org/571112 | 01:52 |
| *** cfriesen has quit IRC | 01:53 | |
| *** mrsoul has joined #openstack-nova | 01:54 | |
| *** cfriesen has joined #openstack-nova | 02:01 | |
| cfriesen | gmann: we're planning on adding extra validation for flavor extra-specs and image properties for operations that could change either or both of them. if the combination of flavour extraspecs and image properties doesn't make sense, we will return an error (400 presumably) back to the user. | 02:03 |
| cfriesen | gmann: currently this request would be accepted but would fail later on down on the compute node, but that's an RPC cast and so the user wouldn't get an error message. | 02:04 |
| gmann | cfriesen: ok, what all APIs? i am wondering if that is taken care by additionalProperties or not. | 02:08 |
| *** Bhujay has joined #openstack-nova | 02:19 | |
| *** Bhujay has quit IRC | 02:19 | |
| *** Bhujay has joined #openstack-nova | 02:20 | |
| *** Dinesh_Bhor has quit IRC | 02:22 | |
| *** Dinesh_Bhor has joined #openstack-nova | 02:26 | |
| *** tbachman has quit IRC | 02:43 | |
| *** liuyulong has quit IRC | 02:48 | |
| *** psachin has joined #openstack-nova | 02:48 | |
| *** imacdonn has quit IRC | 02:50 | |
| *** markvoelker has joined #openstack-nova | 02:50 | |
| *** imacdonn has joined #openstack-nova | 02:51 | |
| *** Bhujay has quit IRC | 02:59 | |
| *** k_mouza has joined #openstack-nova | 03:00 | |
| *** k_mouza has quit IRC | 03:04 | |
| openstackgerrit | Brin Zhang proposed openstack/nova master: Add support volume_type in compute api https://review.openstack.org/605573 | 03:07 |
| openstackgerrit | Brin Zhang proposed openstack/nova master: Add support volume_type in compute api https://review.openstack.org/605573 | 03:16 |
| *** litao has quit IRC | 03:17 | |
| openstackgerrit | zhaodan7597 proposed openstack/nova master: Unable to delete volume when a vmware instance bfv is failed. https://review.openstack.org/571112 | 03:20 |
| openstackgerrit | zhaodan7597 proposed openstack/nova master: Unable to delete volume when a vmware instance bfv is failed. https://review.openstack.org/571112 | 03:21 |
| brinzhang | Kevin_zheng: Take a look at this patch https://review.openstack.org/#/c/605573/2/nova/compute/api.py | 03:26 |
| brinzhang | Kevin_zheng: VOLUME_TYPE_MIN_COMPUTE_VERSION = 52 this variable is not need, instead of it with CINDER_V3_VOLUME_TYPE_MIN_COMPUTE_VERSION = 35 | 03:27 |
| brinzhang | to check the volume type is supported in cinder min version. | 03:28 |
| Kevin_Zheng | you should re arrange your patchsets, it now seems very hard to follow | 03:29 |
| Kevin_Zheng | replied in your new patch | 03:33 |
| brinzhang | Yeah, updating | 03:40 |
| *** takashin has joined #openstack-nova | 03:42 | |
| *** jiapei has joined #openstack-nova | 03:42 | |
| *** rcernin_ has quit IRC | 03:42 | |
| *** rcernin has joined #openstack-nova | 03:43 | |
| *** dave-mccowan has quit IRC | 03:46 | |
| *** hongbin has quit IRC | 03:47 | |
| *** sapd1 has quit IRC | 03:55 | |
| *** sapd1 has joined #openstack-nova | 04:00 | |
| *** dpawlik has joined #openstack-nova | 04:02 | |
| *** tetsuro has quit IRC | 04:04 | |
| *** dpawlik has quit IRC | 04:07 | |
| openstackgerrit | Merged openstack/nova master: Revert "Make host_aggregate_map dictionary case-insensitive" https://review.openstack.org/604898 | 04:11 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (3) https://review.openstack.org/574104 | 04:12 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (4) https://review.openstack.org/574106 | 04:12 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (5) https://review.openstack.org/574110 | 04:13 |
| *** vivsoni_ has joined #openstack-nova | 04:13 | |
| *** vivsoni has quit IRC | 04:15 | |
| *** udesale has joined #openstack-nova | 04:18 | |
| *** rmulugu has joined #openstack-nova | 04:48 | |
| *** dave-mccowan has joined #openstack-nova | 04:50 | |
| *** macza has joined #openstack-nova | 05:03 | |
| *** tetsuro has joined #openstack-nova | 05:07 | |
| *** macza has quit IRC | 05:08 | |
| *** tbachman has joined #openstack-nova | 05:10 | |
| *** slaweq has joined #openstack-nova | 05:11 | |
| *** cfriesen has quit IRC | 05:12 | |
| *** tbachman has quit IRC | 05:15 | |
| *** slaweq has quit IRC | 05:15 | |
| *** Bhujay has joined #openstack-nova | 05:35 | |
| *** bnemec has quit IRC | 05:39 | |
| openstackgerrit | fupingxie proposed openstack/nova master: Don't recreate inst_base on source when using rbd backend in resize https://review.openstack.org/605590 | 05:42 |
| *** ratailor has joined #openstack-nova | 05:52 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (7) https://review.openstack.org/571992 | 06:05 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (8) https://review.openstack.org/571993 | 06:05 |
| *** hoangcx has quit IRC | 06:11 | |
| *** hoangcx has joined #openstack-nova | 06:12 | |
| openstackgerrit | Tao Li proposed openstack/nova master: Rollback instance vm_state to original where instance claims failed https://review.openstack.org/592252 | 06:13 |
| *** slaweq has joined #openstack-nova | 06:23 | |
| *** dpawlik has joined #openstack-nova | 06:23 | |
| *** Dinesh_Bhor has quit IRC | 06:24 | |
| *** pcaruana has joined #openstack-nova | 06:33 | |
| *** adrianc_ has joined #openstack-nova | 06:34 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/rocky: placement: Always reset conf.CONF when starting the wsgi app https://review.openstack.org/604694 | 06:35 |
| *** jamesdenton has quit IRC | 06:35 | |
| *** Dinesh_Bhor has joined #openstack-nova | 06:43 | |
| openstackgerrit | Bryan Song proposed openstack/nova master: Creation image for volume-backend instance should use volume size in image property 'min_disk' https://review.openstack.org/605596 | 06:45 |
| openstackgerrit | Tao Li proposed openstack/nova master: Don't persist retry information into database https://review.openstack.org/605011 | 07:03 |
| *** Luzi has joined #openstack-nova | 07:06 | |
| *** litao_ has joined #openstack-nova | 07:11 | |
| *** rcernin has quit IRC | 07:12 | |
| *** helenafm has joined #openstack-nova | 07:12 | |
| openstackgerrit | Merged openstack/nova master: consumer gen: move_allocations https://review.openstack.org/591810 | 07:20 |
| openstackgerrit | huanhongda proposed openstack/nova master: Allow to attach/detach port when vm_state is soft-delete https://review.openstack.org/605602 | 07:24 |
| *** mhen has joined #openstack-nova | 07:26 | |
| *** ralonsoh has joined #openstack-nova | 07:44 | |
| *** hshiina has quit IRC | 07:48 | |
| openstackgerrit | huanhongda proposed openstack/nova master: Allow to attach/detach port when vm_state is soft-delete https://review.openstack.org/605602 | 08:00 |
| *** jpena|off is now known as jpena | 08:01 | |
| *** tetsuro has quit IRC | 08:10 | |
| *** tetsuro_ has joined #openstack-nova | 08:10 | |
| *** Dinesh_Bhor has quit IRC | 08:16 | |
| *** k_mouza has joined #openstack-nova | 08:20 | |
| kashyap | gibi: Morning, want to put this through: https://review.openstack.org/#/c/605060/ | 08:21 |
| kashyap | gibi: We got confirmation from all the relevant distros | 08:21 |
| gibi | kashyap: good morning. looking | 08:21 |
| kashyap | Thank you! | 08:21 |
| *** alexchadin has joined #openstack-nova | 08:21 | |
| gibi | kashyap: there are couple of FIXMEs in https://wiki.openstack.org/wiki/LibvirtDistroSupportMatrix regarding minimum libvirt and qemu versions | 08:25 |
| gibi | kashyap: I guess if we get the relevant infos from the distros then we can fill those out now | 08:26 |
| kashyap | gibi: Yep, I just sent the two reminders to Iain from Oracle and Colleen from SUSE to fill the FIXME there | 08:26 |
| kashyap | gibi: I added the FIXMEs there :-) | 08:27 |
| gibi | kashyap: cool :) | 08:27 |
| *** janki has joined #openstack-nova | 08:27 | |
| gibi | kashyap: +2 | 08:27 |
| kashyap | Sweet, thank you! | 08:28 |
| gibi | kashyap: thank you for picking this work up | 08:28 |
| kashyap | gibi: No worries; I did that last cycle, and once or twice before too. Thought I'd "remove the bandage quickly" this time too :-) | 08:29 |
| *** jiapei has quit IRC | 08:31 | |
| *** lpetrut has joined #openstack-nova | 08:33 | |
| *** dtantsur|afk is now known as dtantsur | 08:36 | |
| stephenfin | kashyap: I assume you're going to follow that up with a patch to bump the current minimums? | 08:41 |
| kashyap | stephenfin: Yeah, indeed. | 08:41 |
| kashyap | stephenfin: Want to ACK the above, already got one from gibi | 08:41 |
| stephenfin | I can. Just reading through the notes on the Wiki first | 08:42 |
| kashyap | Sure | 08:42 |
| kashyap | stephenfin: In short: Oracle Linux already has the relevant versions we bumped to and SLES will have it (Colleen confirmed on the review). | 08:42 |
| kashyap | And the rest of the distributions already have it. | 08:43 |
| kashyap | That's the thread on the list: lists.openstack.org/pipermail/openstack-operators/2018-September/015929.html | 08:43 |
| *** derekh has joined #openstack-nova | 08:43 | |
| kashyap | Clicakable: http://lists.openstack.org/pipermail/openstack-operators/2018-September/015929.html | 08:43 |
| *** Sigyn has quit IRC | 08:44 | |
| *** Sigyn has joined #openstack-nova | 08:45 | |
| *** Sigyn has quit IRC | 08:45 | |
| *** Sigyn has joined #openstack-nova | 08:45 | |
| *** Sigyn has quit IRC | 08:45 | |
| *** ttsiouts has joined #openstack-nova | 08:49 | |
| *** helenafm has quit IRC | 08:49 | |
| *** skatsaounis has quit IRC | 08:50 | |
| *** Sigyn has joined #openstack-nova | 08:50 | |
| *** Sigyn has quit IRC | 08:51 | |
| *** mgoddard has joined #openstack-nova | 08:51 | |
| openstackgerrit | Radoslav Gerganov proposed openstack/nova master: VMware: Live migration of instances https://review.openstack.org/270116 | 08:53 |
| *** mgoddard has quit IRC | 08:53 | |
| *** mgoddard has joined #openstack-nova | 08:54 | |
| *** Dinesh_Bhor has joined #openstack-nova | 08:55 | |
| *** Sigyn has joined #openstack-nova | 08:56 | |
| *** skatsaounis has joined #openstack-nova | 08:56 | |
| *** ttsiouts has quit IRC | 08:58 | |
| *** ttsiouts has joined #openstack-nova | 09:00 | |
| *** panda|off is now known as panda | 09:00 | |
| *** a-pugachev has joined #openstack-nova | 09:01 | |
| *** tetsuro_ has quit IRC | 09:02 | |
| *** k_mouza has quit IRC | 09:02 | |
| *** mgoddard has quit IRC | 09:03 | |
| *** mgoddard has joined #openstack-nova | 09:04 | |
| *** tssurya has joined #openstack-nova | 09:11 | |
| *** janki has quit IRC | 09:13 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Add API ref guideline for body text https://review.openstack.org/605628 | 09:17 |
| *** Bhujay has quit IRC | 09:19 | |
| *** Bhujay has joined #openstack-nova | 09:25 | |
| *** k_mouza has joined #openstack-nova | 09:28 | |
| *** Dinesh_Bhor has quit IRC | 09:29 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Follow up for Ib6f95c22ffd3ea235b60db4da32094d49c2efa2a https://review.openstack.org/604743 | 09:34 |
| *** k_mouza has quit IRC | 09:35 | |
| *** k_mouza has joined #openstack-nova | 09:38 | |
| *** alexchadin has quit IRC | 09:42 | |
| *** cdent has joined #openstack-nova | 09:44 | |
| *** moshele has joined #openstack-nova | 09:52 | |
| openstackgerrit | Chen proposed openstack/nova master: remove commented-out code https://review.openstack.org/605635 | 10:00 |
| *** erlon has joined #openstack-nova | 10:00 | |
| *** Dinesh_Bhor has joined #openstack-nova | 10:04 | |
| *** purplerbot has joined #openstack-nova | 10:08 | |
| *** cdent has quit IRC | 10:08 | |
| *** helenafm has joined #openstack-nova | 10:15 | |
| openstackgerrit | Brin Zhang proposed openstack/nova master: Specifies the storage backend to boot instance https://review.openstack.org/579360 | 10:18 |
| *** brinzhang has quit IRC | 10:22 | |
| *** Dinesh_Bhor has quit IRC | 10:29 | |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Don't delete disks on shared storage during evacuate https://review.openstack.org/578846 | 10:35 |
| openstackgerrit | tianhui proposed openstack/nova master: Update doc: launch-instance-from-volume https://review.openstack.org/605640 | 10:35 |
| *** ttsiouts has quit IRC | 10:41 | |
| *** udesale has quit IRC | 10:54 | |
| *** cdent has joined #openstack-nova | 11:00 | |
| *** jpena is now known as jpena|lunch | 11:06 | |
| *** mgoddard has quit IRC | 11:09 | |
| *** macza has joined #openstack-nova | 11:11 | |
| *** macza has quit IRC | 11:15 | |
| openstackgerrit | Merged openstack/nova master: Pick next minimum libvirt / QEMU versions for "T" release https://review.openstack.org/605060 | 11:15 |
| *** alexchadin has joined #openstack-nova | 11:22 | |
| *** helenafm has quit IRC | 11:24 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Followup for Iba230201803ef3d33bccaaf83eb10453eea43f20 https://review.openstack.org/605653 | 11:27 |
| *** ratailor has quit IRC | 11:40 | |
| *** mgoddard has joined #openstack-nova | 11:45 | |
| *** ttsiouts has joined #openstack-nova | 11:45 | |
| *** pcaruana has quit IRC | 11:50 | |
| *** Bhujay has quit IRC | 11:51 | |
| *** Bhujay has joined #openstack-nova | 11:52 | |
| *** Bhujay has quit IRC | 11:53 | |
| *** Bhujay has joined #openstack-nova | 11:53 | |
| *** Bhujay has quit IRC | 11:54 | |
| *** rmulugu has quit IRC | 11:55 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Follow up for Iba230201803ef3d33bccaaf83eb10453eea43f20 https://review.openstack.org/605653 | 11:57 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Follow up for Ie991d4b53e9bb5e7ec26da99219178ab7695abf6 https://review.openstack.org/605658 | 11:57 |
| gibi | jaypipes: I've left answer to your question in https://review.openstack.org/#/c/591811/ | 12:01 |
| *** artom has quit IRC | 12:12 | |
| mdbooth | Hmm, I just discovered that initialisation by defaultdict isn't threadsafe | 12:13 |
| cdent | mdbooth: that statement has an mdbooth number of 7 | 12:13 |
| mdbooth | It means that if you've got: foo = defaultdict(threading.Lock) | 12:14 |
| mdbooth | and 2 threads concurrently do: foo['thing'] | 12:14 |
| mdbooth | They might get different threading.Lock objects | 12:14 |
| * mdbooth had always assumed that the library authors weren't actually evil | 12:15 | |
| *** jamesdenton has joined #openstack-nova | 12:16 | |
| cdent | there's some interesting but potentially old discussion at https://stackoverflow.com/questions/17682484/is-collections-defaultdict-thread-safe | 12:16 |
| mdbooth | cdent: Yep, that's where I read it :) | 12:17 |
| *** mriedem has joined #openstack-nova | 12:22 | |
| *** takashin has left #openstack-nova | 12:32 | |
| *** mriedem has quit IRC | 12:34 | |
| *** jpena|lunch is now known as jpena | 12:34 | |
| *** pcaruana has joined #openstack-nova | 12:39 | |
| openstackgerrit | huanhongda proposed openstack/nova master: Allow to attach/detach port when vm_state is soft-delete https://review.openstack.org/605602 | 12:43 |
| *** alexchadin has quit IRC | 12:43 | |
| openstackgerrit | Elod Illes proposed openstack/nova master: Reject interface attach with QoS aware port https://review.openstack.org/570078 | 12:48 |
| *** mriedem has joined #openstack-nova | 12:49 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Null out instance.availability_zone on shelve offload https://review.openstack.org/599087 | 12:50 |
| mriedem | johnthetubaguy: alex_xu: ^ was just a rebase due to a new notification test that needed a sample update | 12:50 |
| *** tbachman has joined #openstack-nova | 13:03 | |
| gibi | mriedem: is the skip_filters flag of the scheduler you mention in https://github.com/openstack/nova/blob/8c3d02ac3d890f414ce4e05c41d44dca3b385424/nova/conductor/tasks/live_migrate.py#L103-L108 a reality or just a dream? | 13:11 |
| gibi | mriedem: I'm trying to solve the nested force migration issues in https://review.openstack.org/#/c/604084/2/nova/tests/functional/test_servers.py@4897 and I think that skip_filters flag is the solution | 13:12 |
| *** liuyulong has joined #openstack-nova | 13:15 | |
| *** scroll has joined #openstack-nova | 13:16 | |
| mriedem | gibi: i'm not working on it if that's what you mean | 13:17 |
| mriedem | but i think the idea is still ok - pass a flag to the scheduler to not run filtered hosts, just do the claim on the forced host | 13:18 |
| mriedem | treats the scheduler more like a library | 13:18 |
| *** bnemec has joined #openstack-nova | 13:18 | |
| mriedem | the point of that todo was more about DRYing up the code | 13:18 |
| gibi | mriedem: would scheduler still call GET allocation_candidates? | 13:18 |
| *** ttsiouts has quit IRC | 13:19 | |
| mriedem | if it needs those to make the claim, then i guess? but that kind of goes against what forcing a host is all about, | 13:19 |
| mriedem | which is, claims be damned this is where i want the thing to go | 13:19 |
| mriedem | but having said that, | 13:19 |
| mriedem | we already broke that contract with https://github.com/openstack/nova/blob/8c3d02ac3d890f414ce4e05c41d44dca3b385424/nova/conductor/tasks/live_migrate.py#L103-L108 | 13:19 |
| mriedem | because we have to keep allocations straight in placement | 13:20 |
| mriedem | or that's what we said to justify that change at the time | 13:20 |
| mriedem | honestly force with bypassing the scheduler was always a terrible idea | 13:20 |
| gibi | mriedem: when we forcing a host we cannot blindly copy the source allocation to the dest host in case of nested | 13:20 |
| gibi | mriedem: as we only know the dest host rp_uuid but not the nested rps | 13:21 |
| mriedem | so we need a new set of allocation candidates | 13:21 |
| gibi | mriedem: one way to solve that is to call GET a_c but limit the search for the given dest host | 13:21 |
| mriedem | that seems reasonable | 13:21 |
| mriedem | essentially, | 13:21 |
| mriedem | in the case of nested RP allocations + force, we'd just go through the scheduler with the requested host but kind of ignore the force flag | 13:22 |
| mriedem | i.e. you can live migrate today by specifying a specific host w/o forcing it and we'll validate that requested host with the scheduler | 13:22 |
| mriedem | it sounds like you want to do the same | 13:22 |
| mriedem | s/want/need/ | 13:22 |
| gibi | mriedem: that seems to be the way forward. So now I go and dig that code path | 13:23 |
| gibi | mriedem: thanks | 13:23 |
| mriedem | in the case of an unforced live migration to a specific host, we just set the RequestSpec.requested_destination to the requested host/node and send that to the scheduler | 13:24 |
| mriedem | it sounds like you'd just need some logic up-front to determine, is this an instance that has allocations on nested RPs | 13:24 |
| mriedem | and if so, ignore the force flag | 13:24 |
| gibi | mriedem: yeah, forced live migration will be less forced as it can return NoValidHost after this change | 13:25 |
| mriedem | https://github.com/openstack/nova/blob/8c3d02ac3d890f414ce4e05c41d44dca3b385424/nova/compute/api.py#L4374 | 13:25 |
| mriedem | gibi: and i think that's ok - forced live migration could always fail even if you bypassed the scheduler b/c the conductor task still does some prechecks on the forced host | 13:25 |
| gibi | mriedem: which means if every instance will be on nested RP then force livemigration will be equal to non forced livemigration with provided host | 13:25 |
| mriedem | https://github.com/openstack/nova/blob/8c3d02ac3d890f414ce4e05c41d44dca3b385424/nova/conductor/tasks/live_migrate.py#L160 | 13:26 |
| mriedem | like, self._check_host_is_up(self.destination) is the ComputeFilter | 13:26 |
| mriedem | self._check_destination_has_enough_memory() is the RamFilter | 13:26 |
| mriedem | etc | 13:26 |
| mriedem | i'd really like to just get rid of that _check_requested_destination method | 13:26 |
| gibi | mriedem: OK, this seems to be a way forward to eventually get rid of the force live migration altogether | 13:27 |
| jaypipes | mriedem: "recheck slow node" <-- worst 90s band name EVAH. | 13:27 |
| mriedem | jaypipes: it's definitely not as good as butthole surfers | 13:27 |
| jaypipes | indeed. | 13:27 |
| jaypipes | true fact: my high school band in the early 90s was called "Slow Children at Play". | 13:28 |
| mriedem | highly offensive | 13:28 |
| jaypipes | embarrassing, I know. | 13:28 |
| mriedem | https://www.youtube.com/watch?v=SkjJLQUhxks | 13:29 |
| mriedem | now i know what needs to be playing this morning | 13:29 |
| mriedem | bauzas: depending on where you check if you should ignore the force flag it could get messy, | 13:32 |
| mriedem | https://github.com/openstack/nova/blob/8c3d02ac3d890f414ce4e05c41d44dca3b385424/nova/compute/api.py#L4359 | 13:32 |
| mriedem | because the api determines what gets passed to conductor | 13:32 |
| mriedem | it's all very tightly coupled | 13:32 |
| mriedem | i.e. if force is True, the requested host parameter isn't passed to conductor, but the request spec is set with the requested_destination | 13:33 |
| mriedem | but if force is True, the request spec is untouched and the requested host is passed to conductor | 13:33 |
| mriedem | and conductor has to know what that means | 13:33 |
| gibi | mriedem: I see | 13:34 |
| mriedem | i assume the logic to determine if the instance has allocations against nested RPs should probably live in conductor | 13:34 |
| mriedem | so we don't block the API response | 13:34 |
| mriedem | although having said that.... | 13:34 |
| mriedem | before some microversion, live migration is aysync rpc call from api-conductor-scheduler until we pick a host and cast to it | 13:34 |
| mriedem | https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id31 | 13:35 |
| mriedem | meaning it's pretty easy to timeout the API while picking hosts before 2.34 | 13:35 |
| *** ttsiouts has joined #openstack-nova | 13:35 | |
| mriedem | cold migrate is the same way - it's all synchronous until we cast to the chosen compute | 13:35 |
| mriedem | :( | 13:36 |
| *** psachin has quit IRC | 13:36 | |
| mriedem | anywho, i'd recommend doing your nested RP calculations in conductor rather than api | 13:36 |
| gibi | mriedem: ack | 13:37 |
| mriedem | maybe you don't need to look at placement? maybe you can just glean if it's got nested rps by looking at the flavor? | 13:37 |
| mriedem | i guess you care about allocations for ports | 13:37 |
| gibi | mriedem: even if flavor has granular groups I don't know if the host the instance is on are already reshaped to nested or not | 13:38 |
| mriedem | right yeah | 13:38 |
| gibi | mriedem: can we simply ignore the force flag for every instance not just for the nested ones? I mean the end user will not know if his instance is nested or not | 13:39 |
| mriedem | it's not really about the owner of the instance (the user), force is for the admin | 13:40 |
| gibi | mriedem: true, I missed that | 13:40 |
| mriedem | if we're really going down a path of blatantly ignoring the force parameter, we should probably consider just deprecating it in the api | 13:41 |
| mriedem | like i said, we kind of already ignore it today for ram/disk/vcpu | 13:41 |
| mriedem | plus some other sanity checks that conductor does | 13:41 |
| mriedem | caching scheduler is the only thing that would still truly force today since it doesn't create allocations | 13:41 |
| mriedem | but we can maybe remove the caching scheduler now | 13:41 |
| mriedem | mgagne: i've been meaning to follow up with you about the caching scheduler removal... | 13:42 |
| gibi | mriedem: does this ignoring behavior needs to be guarded with a new api microversion? I hope not as maintaining the old behavior is not easily possible for nested | 13:43 |
| mriedem | gibi: no i don't think so, | 13:44 |
| mriedem | we're already ignoring force for some filters like i said | 13:44 |
| mriedem | a microversion would be more of a signaling mechanism, | 13:45 |
| mriedem | plus things like nova CLI that do version discovery and pass the latest microversion by default would simply opt into the new microversion where force is never passed | 13:45 |
| mriedem | iow, force is legacy from when we just had claims in the compute and overcomitting was "simpler" | 13:46 |
| gibi | mriedem: so there would be a microversion to remove the force flag from the API but ignoring the force flag would be retroactive in every version | 13:46 |
| bauzas | mriedem: I'm in and out due to some customer issue, but I'll reply to you later on | 13:46 |
| mriedem | well, i don't know if we should blatantly ignore it if we can help it | 13:46 |
| mriedem | but i'm not sure how we determine if we should ignore it or not for nested RPs | 13:47 |
| gibi | mriedem: we cannot easily help it in case of nested, and sooner or later (after NUMA) every instance will be nested | 13:47 |
| mriedem | b/c like you said, the instance might have allocations on nested RPs on the source host but maybe not the forced dest? | 13:47 |
| mriedem | well, every instance with numa allocations | 13:47 |
| mriedem | not all instances are those kinds though | 13:47 |
| gibi | mriedem: if VCPU resources are moved to NUMA RP then the simple instance will become nested | 13:48 |
| mriedem | are we talking about doing that? | 13:53 |
| mriedem | i guess for PCPU/VCPU modeling? | 13:53 |
| mriedem | if so, shiiiiit that reshape is going to be big in a public cloud | 13:53 |
| gibi | mriedem: honestly I don't know what will be the numa model in plaacement | 13:53 |
| gibi | mriedem: https://review.openstack.org/#/c/555081/18/specs/stein/approved/cpu-resources.rst@261 | 13:55 |
| gibi | mriedem: in this spec VCPU is under NUMA | 13:55 |
| *** itlinux has quit IRC | 13:59 | |
| mriedem | gibi: ok left a comment / question in the upgrade impact section, | 14:00 |
| mriedem | we also said at the ptg that that spec wouldn't be a priority for stein | 14:00 |
| gibi | mriedem: correct | 14:00 |
| *** awaugama has joined #openstack-nova | 14:01 | |
| mriedem | just getting vgpu reshaping done is going to be a big hurdle | 14:01 |
| gibi | mriedem: yeah so vgpu and bandwidth using instance will be the first nested instances. So during this time it make sense to keep a separate code path for them | 14:01 |
| gibi | mriedem: I try to implement what you described. checking the nestedness in conductor and ignoring the force flag for nested | 14:02 |
| *** artom has joined #openstack-nova | 14:08 | |
| *** mlavalle has joined #openstack-nova | 14:13 | |
| *** ttsiouts has quit IRC | 14:15 | |
| openstackgerrit | Christoph Manns proposed openstack/nova master: Fix stacktraces with redis caching backend https://review.openstack.org/605748 | 14:24 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova-specs master: Re-propose numa-aware-live-migration spec https://review.openstack.org/599587 | 14:24 |
| *** ttsiouts has joined #openstack-nova | 14:30 | |
| *** jistr is now known as jistr|call | 14:31 | |
| openstackgerrit | Mark Goddard proposed openstack/nova master: Don't emit warning when ironic properties are zero https://review.openstack.org/605754 | 14:34 |
| *** lpetrut has quit IRC | 14:36 | |
| mnaser | filtering happens after placement, correct? | 14:36 |
| *** lpetrut has joined #openstack-nova | 14:36 | |
| mnaser | is there no warning message that says "i couldn't find anything?" | 14:37 |
| mnaser | i guess i can just rely on conductor's "Setting instance to ERROR state." | 14:37 |
| bauzas | mriedem: I'm still working on the vgpu reshape patch, and yes, it's a big hurdle :( | 14:37 |
| bauzas | gibi: FWIW, I need to split https://review.openstack.org/#/c/552924/ in two, one targeted for Stein with no NUMA affinity | 14:38 |
| openstackgerrit | Mark Goddard proposed openstack/nova master: Don't emit warning when ironic properties are zero https://review.openstack.org/605754 | 14:38 |
| bauzas | mnaser: you're right, filters are called after we found an allocation candidate | 14:38 |
| gibi | bauzas: ack | 14:38 |
| openstackgerrit | Merged openstack/nova master: consumer gen: more tests for delete allocation cases https://review.openstack.org/591811 | 14:38 |
| mnaser | thanks bauzas, im relying on this logstash query to monitor those: tags:nova AND message:"Setting instance to ERROR state." AND message:NoValidHost_Remote | 14:39 |
| bauzas | mnaser: but you should at least still have the filtering logs | 14:39 |
| mnaser | yeah the filter logs are there but if 0 computes match at the end, but i'm curious if there's any warning if no allocation candidates that come in the first place | 14:40 |
| bauzas | hah, yeah, but we provide an INFO log saying "heh, 0 hosts found" | 14:40 |
| mnaser | so i guess if its gets 0 allocation candidates, it'll just go through filters and end up with 0 anyways | 14:40 |
| bauzas | I also think we tell in the logs whether we found no candidates after placement | 14:40 |
| mnaser | let me verify | 14:40 |
| bauzas | mnaser: sec, checking the gate | 14:40 |
| bauzas | ok | 14:40 |
| mriedem | i don't think we do, | 14:40 |
| mriedem | we just pass an empty list to ComputeNodeList.get_by_uuids() which returns an empty list and passes that down to the filter scheduler driver which then raises NoValidHost | 14:41 |
| *** moshele has quit IRC | 14:41 | |
| mriedem | oh i guess we log something at debug, | 14:41 |
| mriedem | but that won't be indexed by logstash | 14:41 |
| mnaser | yeah we dont do that, too much data | 14:42 |
| mriedem | https://github.com/openstack/nova/blob/master/nova/scheduler/manager.py#L150 | 14:42 |
| bauzas | mnaser: yup, we do http://logs.openstack.org/72/585672/7/check/tempest-full-py3/f1d0b34/controller/logs/screen-n-sch.txt.gz#_Sep_26_19_02_54_285546 | 14:42 |
| mriedem | mnaser: do you have a failure log to check? | 14:42 |
| mriedem | placement should be logging some stuff now too | 14:43 |
| mnaser | mriedem: i might if it hasnt rotated out | 14:43 |
| bauzas | mriedem: see, we put an info log on how many hosts we got from placement ^ | 14:43 |
| mriedem | as to which "filters" in placement resulted in 0 allocation candidates | 14:43 |
| mriedem | Sep 26 19:02:54.285546 ubuntu-xenial-rax-ord-0002338068 nova-scheduler[18720]: DEBUG nova.filters [None req-8a6074ae-e62f-4ac7-a525-4c411c130c39 tempest-AutoAllocateNetworkTest-2112321998 tempest-AutoAllocateNetworkTest-2112321998] Starting with 1 host(s) {{(pid=20232) get_filtered_objects /opt/stack/nova/nova/filters.py:70}} | 14:43 |
| bauzas | oh shit, that's DEBUG | 14:43 |
| mnaser | mriedem: that is a LOG.debug() so a normal deployment wont see it | 14:43 |
| mnaser | yeah | 14:43 |
| mriedem | right | 14:43 |
| bauzas | my bad | 14:43 |
| mriedem | GAWD! | 14:43 |
| mnaser | i think it's useful to get that warning because a lot of times when placement isnt returning anything | 14:44 |
| bauzas | we info out when we have the filtering results | 14:44 |
| mnaser | i would get really confused | 14:44 |
| mriedem | logging https://github.com/openstack/nova/blob/master/nova/scheduler/manager.py#L150 at INFO might be ok | 14:44 |
| mnaser | yeah but we don't even pass things down to filter | 14:44 |
| bauzas | WTF | 14:44 |
| mnaser | if we get 0 allocation candidates | 14:44 |
| mnaser | if i understand what mriedem linked above | 14:44 |
| bauzas | that's correct | 14:44 |
| bauzas | we just said "meh, that's bad" | 14:44 |
| bauzas | my point was, if we end up with 0 hosts from filtering, some INFO log is done | 14:45 |
| *** evrardjp has quit IRC | 14:45 | |
| bauzas | so, having the pre-filtering result to be INFO seems consistent and valid to me | 14:45 |
| mnaser | yeah that scenario is taken care of i agree | 14:45 |
| bauzas | lemme dig the code | 14:45 |
| mnaser | i'd even go as far as say that's a warning | 14:45 |
| mnaser | https://github.com/openstack/nova/blob/master/nova/scheduler/manager.py#L150 -- just switch that to warning ? | 14:45 |
| bauzas | but I'm pretty sure we say it's INFO (and no ERROR or warning, because a capacity problem isn't a scheduling problem) | 14:46 |
| * mnaser is tempted to make his patch as spammy as possible | 14:46 | |
| mnaser | "change debug level for more info" | 14:46 |
| bauzas | mnaser: I'd advocate for INFO | 14:46 |
| bauzas | no WARN | 14:46 |
| mnaser | it would be consistent with the other stuff | 14:46 |
| bauzas | lemme find the existing log we raise post-filtering | 14:46 |
| mriedem | bauzas: unrelated, but is it just me or do we persist RequestSpec.requested_destination? | 14:46 |
| mriedem | and probably shouldn't... | 14:46 |
| mnaser | bauzas: its info, i have an entry here | 14:47 |
| bauzas | mriedem: wait, wait wait | 14:47 |
| mnaser | bauzas: 2018-09-27 12:37:00.467 394218 INFO nova.filters [<snip>] Filter ComputeFilter returned 0 hosts | 14:47 |
| mriedem | mnaser: i'd say info | 14:47 |
| bauzas | mriedem: probably yet another PEBKAC then | 14:47 |
| mriedem | it's not a warning if someone is trying to resize to a flavor that won't fit anywhere | 14:47 |
| *** evrardjp has joined #openstack-nova | 14:47 | |
| bauzas | (for the persisted field) | 14:47 |
| bauzas | mriedem: zactly | 14:47 |
| bauzas | (16:46:11) bauzas: but I'm pretty sure we say it's INFO (and no ERROR or warning, because a capacity problem isn't a scheduling problem) | 14:47 |
| bauzas | gosh, already 4:46pm here :( | 14:48 |
| mriedem | so if we do persist the request spec requested_destination, i'm just not sure how it doesn't cause problems | 14:48 |
| mriedem | maybe we just get lucky and don't call request_spec.save() on the dirty request spec? | 14:49 |
| mnaser | would we be able to backport that log level change? it's kinda useful. if we can i guess i'll file a bug? | 14:50 |
| mriedem | mnaser: sure | 14:50 |
| bauzas | mriedem: https://github.com/openstack/nova/blob/master/nova/objects/request_spec.py#L29 | 14:51 |
| mriedem | bauzas: that doesn't really tell me anything | 14:51 |
| bauzas | and shit, I threw my day on some internal bug and now I'm done, I have to go into a meeting | 14:51 |
| * bauzas has a very productive day | 14:51 | |
| mriedem | mnaser: this is your justification https://github.com/openstack/nova/blob/c6218428e9b29a2c52808ec7d27b4b21aadc0299/nova/filters.py#L130 | 14:52 |
| mriedem | b/c if we got allocation candidates, but the filters rejected all of them, we log something at INFO | 14:52 |
| mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Filtering%20removed%20all%20hosts%20for%20the%20request%20with%5C%22%20AND%20tags%3A%5C%22screen-n-sch.txt%5C%22&from=7d | 14:53 |
| bauzas | mriedem: I guess I have to doublecheck this spaghetti code | 14:54 |
| bauzas | mriedem: but since requested_destination is only set on a live migration or an evacuation, I just wonder whether we .save() this | 14:55 |
| mriedem | it's also set on resize | 14:55 |
| mriedem | b/c you can pass a host on resize since queens | 14:55 |
| mriedem | and on resize we persist the request spec with the new flavor before casting to compute | 14:56 |
| mriedem | i'm pretty sure i raised this with takashi when he was writing that | 14:56 |
| bauzas | ah | 14:56 |
| mriedem | ah this is how he dealt with that https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3505 | 14:56 |
| bauzas | ghood point | 14:56 |
| bauzas | anyway, I need to jump on a call | 14:57 |
| mriedem | but....that's likely not good enough if you resize to a specific host, and then live migrate without specifying a host... | 14:57 |
| bauzas | mriedem: live migrate has the same logic IIRC | 14:57 |
| bauzas | we null out the field | 14:58 |
| *** itlinux has joined #openstack-nova | 14:58 | |
| mriedem | i don't see that happening | 14:58 |
| mriedem | for live migrate | 14:58 |
| bauzas | oh shit no you're right | 14:58 |
| bauzas | bug bug bug | 14:58 |
| *** ttsiouts has quit IRC | 14:58 | |
| openstackgerrit | Mohammed Naser proposed openstack/nova master: Use INFO for logging no allocation candidates https://review.openstack.org/605765 | 14:58 |
| *** evrardjp has quit IRC | 14:58 | |
| mnaser | mriedem: bauzas ^ | 14:58 |
| mnaser | took me longer to come up with a decent commit message jeez | 14:59 |
| *** ttsiouts has joined #openstack-nova | 14:59 | |
| melwitt | . | 14:59 |
| *** Swami has joined #openstack-nova | 14:59 | |
| mriedem | bauzas: i'll give myself a todo to write a regression test for this | 15:00 |
| bauzas | mriedem: ack | 15:00 |
| *** hamzy_ has quit IRC | 15:00 | |
| mriedem | mnaser: +2 | 15:01 |
| openstackgerrit | Christoph Manns proposed openstack/nova master: Fix stacktraces with redis caching backend https://review.openstack.org/605748 | 15:01 |
| bauzas | mnaser: +Wipped | 15:01 |
| bauzas | mnaser: please make a cherry-pick for rocky | 15:02 |
| *** ttsiouts has quit IRC | 15:04 | |
| *** cfriesen has joined #openstack-nova | 15:06 | |
| openstackgerrit | Mohammed Naser proposed openstack/nova stable/rocky: Use INFO for logging no allocation candidates https://review.openstack.org/605771 | 15:06 |
| mnaser | bauzas: done | 15:07 |
| *** ratailor has joined #openstack-nova | 15:11 | |
| *** Luzi has quit IRC | 15:13 | |
| *** cfriesen has quit IRC | 15:14 | |
| *** cfriesen has joined #openstack-nova | 15:15 | |
| *** ttsiouts has joined #openstack-nova | 15:20 | |
| *** ttsiouts has quit IRC | 15:23 | |
| *** dave-mccowan has quit IRC | 15:23 | |
| *** hamzy_ has joined #openstack-nova | 15:26 | |
| mnaser | AggregateRamFilter is still relevant and working? i remember there was an aggregate filter that had a long ml discussion about how it wasnt really working? | 15:26 |
| *** k_mouza has quit IRC | 15:27 | |
| mriedem | mnaser: ask jaypipes re https://review.openstack.org/#/c/544683/ and https://review.openstack.org/#/c/552105/ | 15:28 |
| * mnaser reads | 15:28 | |
| *** hamzy_ has quit IRC | 15:30 | |
| *** artom has quit IRC | 15:31 | |
| melwitt | mnaser: this is the situation http://lists.openstack.org/pipermail/openstack-dev/2018-January/126283.html and it's still the case now. those two specs ^ are what's needed to restore the ability to set allocation ratios per aggregate | 15:38 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Ignore forcing of live migration for nested instance https://review.openstack.org/605785 | 15:41 |
| *** dave-mccowan has joined #openstack-nova | 15:41 | |
| gibi | mriedem: my first stab for ignoring the force flag ^^ | 15:41 |
| mgagne | @mriedem: what's up with caching scheduler? | 15:42 |
| mriedem | mgagne: are we ok to remove it now? | 15:42 |
| mriedem | in stein | 15:42 |
| mnaser | melwitt: i see, i think i might be looking at the wrong filter then | 15:43 |
| mriedem | i.e. is heal_allocation sufficient for you right now to get upgraded to a FilterScheduler world | 15:43 |
| mgagne | mriedem: I think we figured out it was ok after you wrote the allocation healing tool. | 15:43 |
| mgagne | ++ | 15:43 |
| mriedem | mgagne: ok | 15:43 |
| mriedem | thanks | 15:43 |
| mnaser | maybe it was a weigher that had the flip of a switch | 15:43 |
| mnaser | -1 or 1 to pack vs distribute | 15:43 |
| mriedem | host_subset_size? | 15:43 |
| mnaser | yeah it is gr | 15:43 |
| mnaser | https://github.com/openstack/nova/blob/master/nova/scheduler/weights/ram.py i guess no way of having that per host or per aggregate or anything | 15:44 |
| mgagne | mnaser: we have a private implementation of the RAMWeigher per aggregate | 15:44 |
| melwitt | mgagne: while you're here, don't forget to re-propose https://review.openstack.org/312626 for stein. I was +2 on the implementation but we were at feature freeze at the time | 15:44 |
| mgagne | melwitt: thanks for the follow up | 15:45 |
| mnaser | mgagne: that'd be nice to have upstream i guess, assuming that's a possiblity | 15:45 |
| *** k_mouza has joined #openstack-nova | 15:45 | |
| mgagne | mnaser: sure, unfortunately, I'm not sure it's gonna play well with placement API :D | 15:46 |
| mgagne | or maybe it's not related? | 15:46 |
| mnaser | weighers are after placement | 15:46 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Ignore forcing of live migration for nested instance https://review.openstack.org/605785 | 15:46 |
| mnaser | so it doesnt matter | 15:46 |
| mgagne | mnaser: awesome | 15:46 |
| mnaser | its just "which machine do i prefer" | 15:46 |
| mnaser | but im not a nova dev but thats as far as i understand it | 15:46 |
| mnaser | weighers run after placement allocations AND filters have ran | 15:46 |
| *** jistr|call is now known as jistr | 15:46 | |
| mgagne | so you just need a high host_subset_size for weigher to get some hosts to choose from | 15:46 |
| *** macza has joined #openstack-nova | 15:47 | |
| mnaser | yeah we have that bumped up. the silly idea is right now by default nova spreads vms which is ok but with large vms it becomes problematic | 15:47 |
| mnaser | because we have enough capacity for them but just not in aggregate if that makes sense | 15:47 |
| mgagne | hehe, I know =) | 15:47 |
| bauzas | mriedem: you're my rebuild specialist, so lemme bug you about some silly question | 15:48 |
| mnaser | but we had some really bad decision making with our first flavors so running it with -1 ram multiplier for * .. bad things happen | 15:48 |
| bauzas | mriedem: as of today, do we rebuild by calling the scheduler or have we stopped this ? /me is confused by the number of bugs we had about | 15:48 |
| bauzas | my brain sucks | 15:49 |
| openstackgerrit | Elod Illes proposed openstack/nova master: Reject networks with QoS policy https://review.openstack.org/570079 | 15:50 |
| mgagne | mnaser: based on Newton (sorry): https://gist.github.com/mgagne/142e20e32049abd0cdf5d2da7e048608 | 15:51 |
| melwitt | mnaser: hm, supposed to pack by default, I thought | 15:51 |
| mnaser | mnaser: by default the weighers are set to positive values so distribute | 15:52 |
| melwitt | ok. in the past the scheduler used to pack by default, so I'm not sure when/how that changed | 15:52 |
| mgagne | doesn't make much sense to pack since if you have lets say the openstack infra team spawning 120 VMs at the same time, you will overload the same hosts with requests. | 15:53 |
| melwitt | not since claims in the scheduler, but in the past yeah | 15:53 |
| mgagne | true | 15:53 |
| *** gyee has joined #openstack-nova | 15:53 | |
| *** rpittau has quit IRC | 15:54 | |
| *** gyee has quit IRC | 15:54 | |
| mgagne | but still, maybe you don't want 6 images being downloaded for the first time on the same host at the same time. or other similar expansive operations. | 15:54 |
| *** dpawlik has quit IRC | 15:54 | |
| mnaser | mgagne: thanks for that, ill have a look | 15:54 |
| melwitt | generally speaking, I think pack is the more desired behavior for efficient usage of compute hosts. the only reason people increased subset size, as I understand it, was to avoid the racing of parallel requests trying to claim the same nodes with the old way of claiming | 15:55 |
| mnaser | yeah that was what we had to do for a while | 15:55 |
| mriedem | bauzas: we call the scheduler if the server is being rebuilt with a new image | 15:55 |
| mriedem | b/c we need to validate the new image for the host that the instance is on | 15:55 |
| bauzas | ok, I didn't remember all the conditionals | 15:56 |
| melwitt | yeah, if it's a first time ever image. but when I worked at yahoo we used to warm the cache for images on the compute hosts before letting users at it | 15:56 |
| bauzas | mriedem: thanks | 15:56 |
| mnaser | with ceph it's not even an issue because cow | 15:56 |
| melwitt | true. we didn't use ceph | 15:56 |
| mriedem | mnaser: you're asking about having the RamWeigher applied to an aggregate? | 15:57 |
| bauzas | mriedem: that's from Queens, right? | 15:57 |
| bauzas | I remember the CVE | 15:57 |
| mriedem | bauzas: i think so, but it was backported so ... | 15:57 |
| *** gyee has joined #openstack-nova | 15:57 | |
| bauzas | mriedem: okay | 15:57 |
| *** dpawlik has joined #openstack-nova | 15:58 | |
| mriedem | mnaser: reminds me of Kevin_Zheng's spec https://review.openstack.org/#/c/599308/ | 15:58 |
| mriedem | trying to make the weight configuration not global | 15:58 |
| *** dpawlik has quit IRC | 15:59 | |
| *** dpawlik has joined #openstack-nova | 15:59 | |
| mgagne | mriedem: yes and I commented that I had a similar solution per aggregate, not per flavor. code posted above in a gist =) | 16:00 |
| *** liuyulong has quit IRC | 16:00 | |
| mriedem | mgagne: ok i remember reading your comment but totally missed the part about having the weight configuration per aggregate | 16:03 |
| mriedem | so Kevin_Zheng's spec is maybe way too extreme on the granular side, being per-flavor, | 16:03 |
| mriedem | but global weight configs is also pretty extreme, | 16:04 |
| mriedem | it seems per-aggregate weight configuration would be a nice compromise | 16:04 |
| *** ratailor has quit IRC | 16:04 | |
| bauzas | mriedem: mgagne: I think there was a consensus on that approach, even at the PTG | 16:05 |
| bauzas | putting on a flavor, I nacked, but I'm okay with it per aggregate | 16:05 |
| bauzas | (in the spec, I meant) | 16:05 |
| mriedem | yeah i totally didn't connect the dots on what the alternative was (weights per aggregate) | 16:06 |
| mriedem | i must have been thinking about just pinning flavors to aggregates or something, idk | 16:06 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Fix a race evacuating instances in an anti-affinity group https://review.openstack.org/605436 | 16:07 |
| bauzas | you can mix both indeed | 16:07 |
| bauzas | if that helps your case | 16:07 |
| bauzas | stick flavors to aggregates, the latter having specific weight policies | 16:07 |
| bauzas | that would fit Kevin_Zheng's concern | 16:07 |
| *** dave-mccowan has quit IRC | 16:08 | |
| bauzas | anyway, I need to disappear for a meetup, \o | 16:08 |
| *** efried has quit IRC | 16:10 | |
| *** efried has joined #openstack-nova | 16:10 | |
| nicolasbock | mriedem: Hi. I had asked you about "lost" servers a while back, i.e. servers that were migrated but nova's database was not updated. You had mentioned that resource provider allocation show will tell me about where placement thinks the server is running. | 16:14 |
| cfriesen | gmann: sorry, I didn't notice your question earlier. the validation of flavor extra-specs and image properties would be done on instance creation, instance resize, and instance rebuild. | 16:14 |
| nicolasbock | Unfortunately, in our deployment none of the hypervisors shows anything using this command | 16:14 |
| nicolasbock | Is there an issue with placement? Or are we missing some configuration? Sorry if I sound confused, but I am ;) | 16:15 |
| openstackgerrit | Chen proposed openstack/nova master: remove commented-out code https://review.openstack.org/605635 | 16:19 |
| melwitt | nicolasbock: for that command, you need to pass the instance uuid "the consumer". did you pass that or something else? https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-allocation-show | 16:21 |
| nicolasbock | melwitt: I ran 'resource provider list' first and took the UUIDs as argument for 'resource provider allocation show' | 16:22 |
| melwitt | nicolasbock: ok, those would be the compute host uuids, which is not what you need to pass. you need to pass the uuid of the instance/server, that is "lost" | 16:23 |
| melwitt | and then it will show you information about that instance's allocations and where they are, which resource provider aka which compute host | 16:24 |
| nicolasbock | Oh sorry, I totally misunderstood the command :( | 16:25 |
| nicolasbock | It's working much better now | 16:25 |
| nicolasbock | Thanks! | 16:25 |
| melwitt | no worries, I had thought the same thing the first time I learned about the command | 16:25 |
| *** cdent has quit IRC | 16:26 | |
| *** dpawlik has quit IRC | 16:26 | |
| *** dpawlik has joined #openstack-nova | 16:30 | |
| *** a-pugachev has quit IRC | 16:34 | |
| *** sapd1_ has joined #openstack-nova | 16:40 | |
| *** k_mouza has quit IRC | 16:41 | |
| *** k_mouza has joined #openstack-nova | 16:43 | |
| cfriesen | mdbooth: I think I found a flaw in your fail-fast algorithm for https://review.openstack.org/605436 | 16:43 |
| *** Swami has quit IRC | 16:46 | |
| cfriesen | We're proposing kind of a "big hammer" fix for a missing marker during online data migration. (https://review.openstack.org/#/c/605164/) Does anyone have a more elegant solution? | 16:46 |
| *** mdbooth has quit IRC | 16:49 | |
| *** jackding has joined #openstack-nova | 16:49 | |
| *** med_ has joined #openstack-nova | 16:53 | |
| *** k_mouza has quit IRC | 16:53 | |
| *** tbachman has quit IRC | 16:53 | |
| *** cdent has joined #openstack-nova | 16:53 | |
| *** dtantsur is now known as dtantsur|afk | 16:54 | |
| *** mgoddard has quit IRC | 16:54 | |
| mriedem | melwitt: nicolasbock: https://docs.openstack.org/osc-placement/latest/cli/index.html#cmdoption-openstack-resource-provider-allocation-show-arg-uuid describes the uuid but we could rename that metavar to be consumer_uuid so it's more obvious from the beginning | 16:56 |
| mriedem | consumers aren't a top-level resource in placement so that's probably why it's confusing | 16:57 |
| mriedem | unlike openstack resource provider show https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-show | 16:57 |
| melwitt | yeah. it does say "consumer". I think the confusion comes from the fact that it's in the resource provider command family | 16:57 |
| *** dpawlik has quit IRC | 16:58 | |
| melwitt | the first time I read it, I thought the documentation was a mistake. but edleafe confirmed that it is indeed supposed to be "consumer" uuid | 16:58 |
| mriedem | probably should have been "openstack resource allocation list <consumer_uuid>" | 16:58 |
| *** dpawlik has joined #openstack-nova | 16:59 | |
| mriedem | could still add that and deprecate the old command | 16:59 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Cross-cell resize https://review.openstack.org/603930 | 17:02 |
| *** med_ has quit IRC | 17:03 | |
| *** derekh has quit IRC | 17:04 | |
| *** tbachman has joined #openstack-nova | 17:05 | |
| *** artom has joined #openstack-nova | 17:06 | |
| *** jpena is now known as jpena|off | 17:11 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix stacktraces with redis caching backend https://review.openstack.org/605748 | 17:15 |
| *** cdent has quit IRC | 17:16 | |
| *** adrianc_ has quit IRC | 17:18 | |
| * mnaser takes deep breath | 17:22 | |
| mnaser | openstackclient doesnt let you do a live migration unless you specify a host? | 17:22 |
| mnaser | https://github.com/openstack/python-openstackclient/blob/c0567806916995698e94734d2b2c422a4bf5a1db/openstackclient/compute/v2/server.py#L1333-L1337 | 17:23 |
| nicolasbock | Thanks mriedem and melwitt . Yes, the wording could be clearer, but then again I probabaly could have read the help a little bit more carefully :) | 17:23 |
| nicolasbock | I think that's true mnaser | 17:23 |
| * melwitt runs away from mnaser | 17:24 | |
| mnaser | nova can do live migrations without specifying a host | 17:24 |
| mnaser | the novaclient lets you do it | 17:24 |
| mnaser | and i think there's been voices of "forcing a host in migrations is a bad idea™" | 17:24 |
| nicolasbock | But in 'nova live-migration' you also need to specify a host | 17:25 |
| nicolasbock | Same as in 'openstack server migrate' | 17:25 |
| cfriesen | mnaser: yeah, it's messed up | 17:25 |
| nicolasbock | sorry 'openstack server migrate --live' | 17:26 |
| mnaser | nova live-migration does not require a host | 17:26 |
| cfriesen | mnaser: migration and live migration in OSC need help | 17:26 |
| mnaser | it is optional | 17:26 |
| melwitt | yeah, there are unfortunate discrepancies between novaclient and openstackclient. we talked about it a bit at the PTG L721 https://etherpad.openstack.org/p/nova-ptg-stein | 17:26 |
| mnaser | i guess this is a lot harder than expected because we'd break "api" | 17:26 |
| mnaser | im not sure what'd be the ideal solution | 17:26 |
| melwitt | I was thinking we could just "fix" the openstackclient side to be able to do the same stuff as novaclient. just someone has to do it | 17:27 |
| mnaser | hmm im thinking add a positional argument [host] | 17:27 |
| melwitt | would have to talk to dtroyer about it more | 17:27 |
| mnaser | and make osc ignore the parameter provided to --live | 17:27 |
| mnaser | at least to let you do clean live migrations | 17:28 |
| cfriesen | mnaser: while you're in there you could also fix up all the *other* live-migration related stuff that OSC doesn't handle. | 17:28 |
| cfriesen | automatic detection of block/shared, for example | 17:29 |
| melwitt | oh yep, that's one that our customers have hit many times | 17:29 |
| artom | Careful about that though, they're moving to openstacksdk, so it might be more intelligent to just help with the move | 17:29 |
| artom | (Assuming openstacksdk doesn't have the same problems) | 17:29 |
| artom | (Can I just say that I feel like like the *clients, then openstackclient, then openstacksdk is a case of https://xkcd.com/927/ ) | 17:30 |
| mnaser | yep | 17:30 |
| melwitt | moving to openstacksdk is going to change the CLI arguments? I haven't read in detail about what that will involve | 17:30 |
| mnaser | openstacksdk has some 'intelligence' | 17:31 |
| mnaser | inherited from shade | 17:31 |
| mnaser | so it does a lot of figuring out of the right thing to do | 17:31 |
| mnaser | or the ideal set of defaults so to speak | 17:31 |
| melwitt | oh good, it can fix everything automatically then | 17:31 |
| mnaser | where a live migration python call in python-novaclient assumes nothing and sends rest, openstacksdk tries to workaround all the weird things we have and deliver a reasonable end result "a successful live migration" | 17:32 |
| mnaser | https://review.openstack.org/#/c/589012/ | 17:32 |
| mnaser | this is good though | 17:32 |
| melwitt | oh, nice patch | 17:33 |
| mnaser | it was in the etherpad, so it gets through that step at least | 17:33 |
| melwitt | ah, I'm blind | 17:33 |
| cfriesen | the thing I don't really like about openstackclient is that the help text isn't sensitive to API version | 17:35 |
| artom | Very few things are :/ | 17:35 |
| mnaser | im trying to force myself to use it | 17:35 |
| mnaser | so that people can use it | 17:35 |
| *** dpawlik has quit IRC | 17:36 | |
| mnaser | btw | 17:37 |
| mnaser | shall we update the topic? | 17:37 |
| melwitt | we should but dansmith is out until next week. I'm not sure who else can do it | 17:38 |
| mnaser | i can | 17:38 |
| *** ChanServ changes topic to "This channel is for Nova development. For support of Nova deployments, please use #openstack." | 17:39 | |
| *** dpawlik has joined #openstack-nova | 17:39 | |
| *** dpawlik has quit IRC | 17:39 | |
| mnaser | let me know if you want to switch it to something else or whatever :> | 17:39 |
| *** dpawlik has joined #openstack-nova | 17:40 | |
| *** dpawlik has quit IRC | 17:40 | |
| *** mvkr has quit IRC | 17:42 | |
| *** jamesdenton has quit IRC | 17:44 | |
| melwitt | oh, heh. ok, IIRC we had the current release schedule in it https://wiki.openstack.org/wiki/Nova/Stein_Release_Schedule and included the current runways, which would be use-nested-allocation-candidates right now | 17:45 |
| *** panda is now known as panda|off | 17:46 | |
| *** tssurya has quit IRC | 17:55 | |
| mnaser | melwitt: wanna give me something to copy pasta into the topic? | 18:00 |
| melwitt | yeah, lemme see. I can't remember what order it was in | 18:00 |
| mriedem | i've talked with dtroyer about the osc live migration support, i think he basically wants to just re-write the command on a major version | 18:01 |
| mriedem | mnaser: if you do specify a host with osc's live migration command, make sure you are using the microversion that doesn't bypass the scheduler | 18:02 |
| mriedem | https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id27 | 18:02 |
| mnaser | mriedem: i switched to using python-novaclient because i explicitly want the scheduler to decide for me | 18:02 |
| mriedem | tbc, the live migration API explicitly requires that the host param is sent, it's not optional, but the value in the REST API can be None (it's dumb) | 18:03 |
| mriedem | but ^ isn't possible in osc b/c you can't specify None on the command line | 18:03 |
| mriedem | you would need to make --host optional | 18:03 |
| mriedem | er --live <hostname> | 18:04 |
| mriedem | https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/server.html#server-migrate | 18:04 |
| mriedem | like, --live '' | 18:04 |
| mnaser | mriedem: hence https://review.openstack.org/#/c/589012/ | 18:04 |
| mriedem | yes, there is another one as well | 18:04 |
| mnaser | but yeah i dunno | 18:04 |
| mnaser | microversions for osc api? | 18:04 |
| mnaser | heck yeah | 18:04 |
| mriedem | https://review.openstack.org/#/c/460059/ | 18:05 |
| mnaser | (i'm kidding please don't end me) | 18:05 |
| mriedem | you can pass microversions through for osc | 18:05 |
| *** dims_ is now known as dims | 18:05 | |
| mriedem | openstack --os-compute-api-version 2.30 migrate --live | 18:05 |
| mriedem | or set an env var | 18:05 |
| mnaser | yeah no but i meant like | 18:05 |
| mriedem | osc doesn't do version negotiation to default to the latest like nova cli | 18:05 |
| mnaser | the actaul openstack client shell api or whatever | 18:05 |
| mnaser | but i guess we call those just *releases* of the client | 18:06 |
| mriedem | https://review.openstack.org/#/c/460059/7//COMMIT_MSG@15 | 18:06 |
| mriedem | has my suggestion in it | 18:06 |
| mriedem | you also can't bfv with osc | 18:08 |
| mriedem | unless you use an existing volume | 18:08 |
| mriedem | those are two pretty big gaps in functionality | 18:08 |
| mnaser | now on another note i think live migrations are scheduled when they are received, not when they start, right? | 18:08 |
| mriedem | correct, | 18:08 |
| mriedem | we have to pick the dest host | 18:08 |
| mriedem | so we can setup things like port bindings and volume attachments | 18:09 |
| mnaser | makes our hypervisor evacuations a bit more annoying in that we can kinda do one at a time | 18:09 |
| mriedem | can only do? | 18:09 |
| mnaser | well if we do host-evacuate-live on 3 nodes at once | 18:09 |
| mnaser | its possible that they schedule to each other | 18:09 |
| mnaser | (sometimes we actually do host-evacuate-live only for the purpose of having instances go in the right place after we make scheduler changes) | 18:10 |
| *** Swami has joined #openstack-nova | 18:10 | |
| mriedem | if you are evacuating the hosts, you could disable the compute service | 18:10 |
| mriedem | not evacuate like the evacuate API, i mean "get the vms off this host" | 18:10 |
| mnaser | yeah when we're shutting things down that's what we go for, but when shuffling things around | 18:10 |
| mnaser | you kinda just want instances to move into where they are supposed to go | 18:11 |
| mnaser | anyways | 18:11 |
| mnaser | very minor thing | 18:11 |
| *** ralonsoh has quit IRC | 18:11 | |
| mriedem | "you kinda just want instances to move into where they are supposed to go" | 18:11 |
| mriedem | ha | 18:11 |
| mriedem | of course! | 18:11 |
| mriedem | silly scheduler | 18:11 |
| mriedem | well as a tc big wig, | 18:12 |
| mriedem | you can influence the goal setting for T | 18:12 |
| mnaser | usually after we make scheduling tweaks | 18:12 |
| mnaser | we just do a rolling live migration | 18:12 |
| mriedem | so watcher but without watcher | 18:12 |
| *** sapd1_ has quit IRC | 18:12 | |
| mnaser | just a one time watcher | 18:13 |
| mnaser | i think watcher is super interesting but i think it depends on too many things | 18:13 |
| mriedem | well as a tc big wig, | 18:14 |
| mriedem | you can influence adoption of a new top-level project: watcher-lite | 18:15 |
| mriedem | watcher zero | 18:15 |
| mriedem | all of the flavor, none of the guilt | 18:15 |
| mnaser | lols | 18:17 |
| melwitt | mnaser: Current runways: use-nested-allocation-candidates -- This channel is for Nova development. For support of Nova deployments, please use #openstack. | 18:21 |
| *** lpetrut has quit IRC | 18:21 | |
| *** ChanServ changes topic to "Current runways: use-nested-allocation-candidates -- This channel is for Nova development. For support of Nova deployments, please use #openstack." | 18:22 | |
| mnaser | melwitt: voila i've made myself useful for today | 18:22 |
| melwitt | \o/ | 18:23 |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: nova-manage - fix online_data_migrations counts https://review.openstack.org/605828 | 18:23 |
| mriedem | imacdonn: +W on https://review.openstack.org/#/c/605329/ and i found an example of a migration that has total > 0 with completed == 0 | 18:26 |
| imacdonn | mriedem: ack. I was just about to ask you if you intentionally didn't +W with your +2 :) | 18:26 |
| mriedem | it was intentional, | 18:26 |
| mriedem | because i was going to backport to stable and get a grenade run where i knew we actually had things to migrate | 18:27 |
| mriedem | but i found one in stein too | 18:27 |
| imacdonn | got it | 18:27 |
| mriedem | http://logs.openstack.org/29/605329/2/check/neutron-grenade/2200365/logs/grenade.sh.txt.gz#_2018-09-27_11_17_32_536 | 18:27 |
| mriedem | 2 rows matched query migrate_instances_add_request_spec, 0 migrated | 18:27 |
| mriedem | ^ is with your change | 18:27 |
| mriedem | | populate_queued_for_delete | 2 | 2 | | 18:27 |
| mriedem | http://logs.openstack.org/88/605488/1/check/neutron-grenade/d64e316/logs/grenade.sh.txt.gz#_2018-09-27_01_15_47_182 is without | 18:27 |
| mriedem | | populate_queued_for_delete | 0 | 0 | | 18:27 |
| imacdonn | \o/ | 18:27 |
| mriedem | https://github.com/openstack/nova/blob/e658f41d686e4533640b101622f2342348c0316d/nova/objects/request_spec.py#L707 is the example where total can be >0 but we don't actually migrate anything | 18:28 |
| mriedem | so that with the explanation here https://github.com/openstack/nova/blob/e658f41d686e4533640b101622f2342348c0316d/nova/cmd/manage.py#L374 is confusing | 18:29 |
| imacdonn | that may be a bug | 18:29 |
| mriedem | it does say, "If found is nonzero and done is zero, some records are # not migratable, but all migrations that can complete have # finished." | 18:29 |
| mriedem | "not migrateable" should really be, "don't require migration" | 18:29 |
| imacdonn | per Dan's description, count_all should never be greater than max_count | 18:29 |
| mriedem | a lot of the migrations return found==done because the query to find the $found number is filtering on things that need to be migrated | 18:30 |
| mriedem | e.g. select * bdms where uuid is None; | 18:30 |
| mriedem | found == done ^ | 18:30 |
| mriedem | but that's not the same with the request spec migratoin | 18:30 |
| mriedem | *migration | 18:30 |
| mriedem | since we have to hit 2 different dbs | 18:30 |
| *** jamesdenton has joined #openstack-nova | 18:31 | |
| imacdonn | personally I think the batch mechanism is a bit broken, at least as it's described in the comments | 18:31 |
| mriedem | as i mentioned on your change, the 'Total Needed' column is misleading | 18:31 |
| mriedem | http://logs.openstack.org/29/605329/2/check/neutron-grenade/2200365/logs/grenade.sh.txt.gz#_2018-09-27_11_17_32_536 | 18:31 |
| imacdonn | but, as you said, we need Dan for that conversation | 18:31 |
| mnaser | so has anyone ever thought what happens when we hit instance-ffffffff | 18:41 |
| imacdonn | fsshhh ... that'll never happen | 18:43 |
| melwitt | in case anyone is wondering about the failing ceph job, I'm trying out a fix here https://review.openstack.org/605833 | 18:44 |
| * mordred waves to melwitt and mriedem and mnaser having seen conversation in the past about sdk and osc related things | 18:47 | |
| *** jistr has quit IRC | 18:47 | |
| mnaser | mordred: it's a bit of a difficult position but ideally figuring out what the best way to deal with cold/live migration and reworking it.. (openstack server migrate) | 18:48 |
| mordred | yah. as you know, the sdk code for that is ... fun :) | 18:48 |
| mnaser | mainly my issue was osc forces you to specify a host when its optional | 18:48 |
| mordred | fwiw - mriedem is right - osc doesn't currently do version negotiation. once we start migrating it to sdk though, it'll pick up that ability | 18:48 |
| * mnaser looks forward for that but knows that's quite the effort | 18:49 | |
| *** jistr has joined #openstack-nova | 18:49 | |
| mriedem | mnaser: easy: make --live just an option with no value, add --host (optional, takes a value), and add --cold | 18:49 |
| mnaser | mriedem: but the not breaking scripts part i guess | 18:49 |
| mriedem | or, let --live take a value for compat but proxy it to --host if --host isn't used | 18:50 |
| mriedem | i'm not sure how you could specify --live w/o a host though if --live can take a host | 18:50 |
| mriedem | gd CLIs | 18:50 |
| mriedem | empty string? | 18:50 |
| mriedem | pretty janky | 18:50 |
| mriedem | openstack server migrate --live-but-with-no-host-seriously my_server | 18:51 |
| mordred | mriedem: ++ | 18:51 |
| mordred | that's totally the right answer | 18:51 |
| mriedem | do i win something? | 18:51 |
| mordred | you win this bucket of parts I just found | 18:52 |
| *** tbachman has quit IRC | 18:54 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add more documentation for online_data_migrations CLI https://review.openstack.org/605836 | 18:54 |
| mriedem | imacdonn: efried: ^ does this make life better? | 18:54 |
| imacdonn | only slightly, IMO | 18:55 |
| mriedem | :( | 18:56 |
| imacdonn | if those two rows don't need migration, then they couldn't be included in something named "Total Needed"......... ? | 18:57 |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: nova-manage - fix online_data_migrations counts https://review.openstack.org/605839 | 18:57 |
| mriedem | like i said, total needed is a bad title, | 18:57 |
| imacdonn | if "Total Needed" means "Total Rows that exist that may or may not need it", we should see a lot less zeroes | 18:57 |
| mriedem | i'm not sure if renaming that to Total Found breaks any kind of compat, | 18:57 |
| mriedem | Total Candidates | 18:58 |
| mriedem | something like that | 18:58 |
| mriedem | no one should be parsing the output of this command for column headers and such anyway | 18:58 |
| efried | hah | 18:58 |
| imacdonn | I suspect that the existing migration methods may already interpret it inconsistently, but I haven't analysed it to confirm | 18:58 |
| efried | yeah, to me, seeing needed=2/completed=0 feels like it should be an error | 18:59 |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: nova-manage - fix online_data_migrations counts https://review.openstack.org/605840 | 19:00 |
| efried | I mean, this is better than it was before, because there's at least *some* attempt to explain wtf is going on. | 19:01 |
| efried | Is this dansmith's bailiwick btw? Something he would want to review? | 19:01 |
| *** tbachman has joined #openstack-nova | 19:05 | |
| mriedem | i assume he would yes | 19:05 |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: nova-manage - fix online_data_migrations counts https://review.openstack.org/605842 | 19:06 |
| *** jistr has quit IRC | 19:08 | |
| *** jistr has joined #openstack-nova | 19:08 | |
| *** itlinux has quit IRC | 19:09 | |
| mordred | mriedem, cfriesen: I just left a suggestion on the osc live migration patch about a way to make --live take an optional argument | 19:11 |
| cfriesen | mordred: sweet, I think that's probably the best way to handle backwards compatibility | 19:12 |
| mriedem | there are two, but i found it | 19:13 |
| *** tbachman has quit IRC | 19:14 | |
| mriedem | i like that idea yeah | 19:14 |
| mriedem | note you can't also cold migrate and specify a target host now... | 19:14 |
| mriedem | so i'm not sure how that would play with this too | 19:14 |
| mriedem | *can also | 19:15 |
| imacdonn | I always thought that was weird .. that you can't do that (but can for live) | 19:15 |
| imacdonn | (always => since icehouse days, at least) | 19:15 |
| cfriesen | imacdonn: artificial OSC limitation | 19:16 |
| cfriesen | imacdonn: the compute API lets you specify a host since 2.56 | 19:16 |
| imacdonn | hmm | 19:16 |
| cfriesen | (which is admittedly fairly new) | 19:17 |
| imacdonn | right .. I was just checking ;) | 19:17 |
| imacdonn | so "the struggle was real" when I last looked | 19:18 |
| artom | I'm guessing splitting live and cold migration into different subcommands is no longer an option at this point, right? | 19:18 |
| artom | Since they're, you know, fundamentally different operations? | 19:18 |
| cfriesen | artom: to the end user, they're very similar. | 19:19 |
| artom | cfriesen, you mean besides the fact that your workload goes down? | 19:19 |
| artom | ;) | 19:19 |
| artom | And that live migration is admin-only (by default)? | 19:20 |
| cfriesen | artom: cold is too, isn't it? | 19:21 |
| artom | cfriesen, doh, you're right | 19:22 |
| imacdonn | to a typical sysadmin (I'm thinking private cloud), "migrate" means "VM is running on node A, and I want it to be running <somewhere else>" .. if we can do it without shutting the VM OS down, that'll make my life better | 19:22 |
| artom | Wait no, I was looking at the wrong bit of api-ref | 19:22 |
| imacdonn | from that perspective, they're basically the same thing | 19:22 |
| artom | ... aaand no, still admin-only | 19:22 |
| *** jistr has quit IRC | 19:23 | |
| artom | I dunno, there's just a whole bunch of things that cold migration can do that live migration can't | 19:23 |
| artom | And anyways, I was being rhetorical, we're obviously not going to limit osc migrate to cold migration and add a new osc live-migrate at this point. | 19:24 |
| *** jistr has joined #openstack-nova | 19:26 | |
| mriedem | cold migrate is admin only, resize is non-admin | 19:27 |
| mriedem | w/ cold migrate under the covers | 19:27 |
| artom | Kinky. | 19:27 |
| mriedem | only on anniversaries dude | 19:28 |
| *** jistr has quit IRC | 19:28 | |
| mriedem | resize and cold migrate have been married awhile | 19:28 |
| artom | Yeah, I'm surprised there's still anything going on under the covers. | 19:28 |
| imacdonn | better than cold feet | 19:29 |
| *** jistr has joined #openstack-nova | 19:29 | |
| *** jistr has quit IRC | 19:34 | |
| *** jistr has joined #openstack-nova | 19:37 | |
| *** hamzy has joined #openstack-nova | 19:38 | |
| *** jistr has quit IRC | 19:39 | |
| *** hamzy has quit IRC | 19:44 | |
| *** jistr has joined #openstack-nova | 19:49 | |
| *** tbachman has joined #openstack-nova | 19:50 | |
| *** pcaruana has quit IRC | 19:54 | |
| *** Nel1x has joined #openstack-nova | 20:04 | |
| mriedem | melwitt: i think the vmware live migration change can go into a runway slot | 20:07 |
| mriedem | rgerganov updated it to get ci passing, | 20:07 |
| mriedem | i've done another pass, still -1 but it's closer | 20:07 |
| melwitt | ok, cool. I'll add it then. missed the +1 vote from the vmware CI earlier today after I rechecked it | 20:08 |
| *** hamzy has joined #openstack-nova | 20:09 | |
| *** med_ has joined #openstack-nova | 20:10 | |
| *** hamzy has quit IRC | 20:20 | |
| *** hamzy has joined #openstack-nova | 20:20 | |
| *** hamzy has quit IRC | 20:25 | |
| *** hamzy has joined #openstack-nova | 20:28 | |
| *** macza has quit IRC | 20:31 | |
| *** macza has joined #openstack-nova | 20:32 | |
| *** artom has quit IRC | 20:33 | |
| *** hamzy has quit IRC | 20:34 | |
| *** hamzy has joined #openstack-nova | 20:34 | |
| *** dpawlik has joined #openstack-nova | 20:42 | |
| *** hamzy has quit IRC | 20:43 | |
| *** dpawlik has quit IRC | 20:46 | |
| melwitt | nova meeting in 11 minutes | 20:49 |
| melwitt | 10 minutes | 20:50 |
| mriedem | melwitt: want to hit this backport? https://review.openstack.org/#/c/605260/ - would be good to get those reverts in stable merged/released since they regressed blazar | 20:52 |
| *** takashin has joined #openstack-nova | 20:52 | |
| mriedem | the fix to replace the original is stacked on top on master but the gate is....not cooperating | 20:52 |
| melwitt | can do. any idea why lee removed his vote? | 20:53 |
| mriedem | no idea | 20:57 |
| mriedem | classic lee | 20:57 |
| *** awaugama has quit IRC | 21:01 | |
| *** erlon has quit IRC | 21:04 | |
| *** jamesdenton has quit IRC | 21:15 | |
| *** mchlumsky has quit IRC | 21:37 | |
| *** mriedem has quit IRC | 21:55 | |
| *** burt has quit IRC | 21:59 | |
| *** mriedem has joined #openstack-nova | 22:07 | |
| *** scarab_ has joined #openstack-nova | 22:09 | |
| *** scarab_ has quit IRC | 22:11 | |
| *** mvkr has joined #openstack-nova | 22:13 | |
| *** rcernin has joined #openstack-nova | 22:29 | |
| *** takashin has left #openstack-nova | 22:31 | |
| *** macza has quit IRC | 22:38 | |
| *** macza has joined #openstack-nova | 22:38 | |
| *** dpawlik has joined #openstack-nova | 22:42 | |
| cfriesen | is there an equivalent of CONF.reserved_huge_pages but for regular memory? (per numa node though) | 22:45 |
| cfriesen | I can see CONF.reserved_host_memory_mb but that's not specifically 4k pages and is per compute node, not numa node. | 22:46 |
| *** dpawlik has quit IRC | 22:47 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Update RequestSpec.flavor on resize_revert https://review.openstack.org/605879 | 22:47 |
| *** macza has quit IRC | 22:51 | |
| *** macza_ has joined #openstack-nova | 22:51 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: Update RequestSpec.flavor on resize_revert https://review.openstack.org/605880 | 23:01 |
| mriedem | stable branch core review please https://review.openstack.org/#/c/600113/ | 23:03 |
| cfriesen | mriedem: what would you think of something like CONF.reserved_huge_pages but for regular memory? (ie, to reserve specific amounts of 4K memory on each host numa node) | 23:08 |
| cfriesen | or can CONF.reserved_huge_pages be used for 4k pages as well even though the name implies otherwise | 23:10 |
| mriedem | cfriesen: i don't think you realize that i don't know anything about that nova/virt/hardware stuff | 23:11 |
| cfriesen | heh | 23:11 |
| cfriesen | I'm off to go dig through code | 23:11 |
| mriedem | so not reserved_host_ram or whatever we have? | 23:11 |
| mriedem | that goes on the compute node? | 23:11 |
| cfriesen | yeah, "reserved_host_memory_mb" is per compute node | 23:12 |
| mriedem | reserved_host_memory_mb | 23:12 |
| mriedem | and you want something to reserve ram per numa node? | 23:12 |
| *** artom has joined #openstack-nova | 23:12 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: Fix instance evacuation with PCI devices https://review.openstack.org/605881 | 23:13 |
| mriedem | melwitt: fyi i'm trying to flush through my stable queens and pike changes which also apply to ocata, and then gonna probably send a thing to the ML to wrassle a stable branch review sprint for next week | 23:13 |
| mriedem | to flush all stable branches so we can get ocata released and tagged for EM | 23:13 |
| mriedem | it took an entire week to just get stuff merged for the last round of stable releases | 23:14 |
| mriedem | b/c of the gate | 23:14 |
| cfriesen | mriedem: after looking at the code, I think I could use CONF.reserved_huge_pages to reserve 4K memory per numa node, even though it's not actually huge pages. now to actually try it out | 23:16 |
| melwitt | mriedem: sounds good | 23:17 |
| *** macza_ has quit IRC | 23:18 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: Update nova network info when doing rebuild for evacuate operation https://review.openstack.org/605882 | 23:20 |
| *** dpawlik has joined #openstack-nova | 23:20 | |
| *** dpawlik has quit IRC | 23:25 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: unquiesce instance after quiesce failure https://review.openstack.org/605884 | 23:25 |
| *** mriedem has quit IRC | 23:26 | |
| *** Swami has quit IRC | 23:45 | |
| *** erlon has joined #openstack-nova | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!