| Jorhson | sean-k-mooney:do you help to review the patch 761264? | 02:13 |
|---|---|---|
| bauzas | good morning Nova | 08:44 |
| *** brinzhang_ is now known as brinzhang | 12:38 | |
| arne_wiebalck | Hi! In one of the conductor groups in our Nova/Ironic cell, some compute nodes have been soft-deleted (for not yet understood reasons). While hard-deleting from the compute_nodes table lets Nova re-insert it, it seems the allocations are not fixed by this. The result is that placement is advertising physical nodes which are active ... is there a straight-forward/automatic way to correct this? (This is on Nova Stein :) | 12:51 |
| opendevreview | Jorhson Deng proposed openstack/nova master: process instance in soft-deleting task state during _init_instance https://review.opendev.org/c/openstack/nova/+/761264 | 13:26 |
| opendevreview | Jorhson Deng proposed openstack/nova master: process instance in soft-deleting task state during _init_instance https://review.opendev.org/c/openstack/nova/+/761264 | 13:35 |
| opendevreview | Jonathan Race proposed openstack/nova master: Adds Emulation for aarch64, and ppc64 on x86_64 processors https://review.opendev.org/c/openstack/nova/+/822053 | 13:57 |
| bauzas | arne_wiebalck: hola, sorry wasn't looking at the IRC chan given it was quiet :) | 14:10 |
| bauzas | arne_wiebalck: so you mean you have persisted allocations for some not longer compute services, right? | 14:10 |
| arne_wiebalck | bauzas: heh, yeah, wasn't sure if anyone is still around :) | 14:10 |
| bauzas | until tonight :) | 14:10 |
| arne_wiebalck | bauzas: so, the compute are marked as deleted, but placement offers them, which makes me think the allocations have been deleted with the nodes ... but did not come back when the nodes were re-inserted | 14:11 |
| arne_wiebalck | bauzas: we had this in the past, and Belmiro did some magic, but he is not available at the moment | 14:12 |
| arne_wiebalck | bauzas: I will not risk too much either, just in case there is some sync I need to kick off, I may try it | 14:13 |
| arne_wiebalck | bauzas: my main worry is that placement offers physical nodes which are installed :-S | 14:13 |
| arne_wiebalck | bauzas: Ironic stops the installation, but free nodes cannot be installed either (since placement offers all nodes) | 14:14 |
| arne_wiebalck | bauzas: I wasn't sure either if it is safe to remove active nodes from the placement aggregrate (or even deleted the resource providers) | 14:15 |
| arne_wiebalck | bauzas: I have done this with available nodes, but not with active ones | 14:15 |
| bauzas | arne_wiebalck: tbc, do you have resource providers for the deleted compute services ? | 14:15 |
| arne_wiebalck | bauzas: I must have otherwise placement would not offer them, no? | 14:16 |
| arne_wiebalck | bauzas: but let me check for one ... | 14:16 |
| bauzas | arne_wiebalck: sorry I still don't understand | 14:16 |
| arne_wiebalck | bauzas: it is probable since I do not use the correct terms | 14:16 |
| bauzas | arne_wiebalck: so, which Resource Providers do you have ? | 14:17 |
| bauzas | arne_wiebalck: is the usage for every RP good or not ? | 14:17 |
| bauzas | arne_wiebalck: and do you have "orphaned" allocations ? | 14:17 |
| bauzas | an orphaned allocation is an allocation telling it's about some RP UUID but with the RP no longer here | 14:18 |
| bauzas | RP : Resource Provider | 14:18 |
| arne_wiebalck | bauzas: I have resource providers which placement offers me as allocation candidates, but the nodes are actually active in Ironic, so there is an instance on them | 14:19 |
| bauzas | arne_wiebalck: I see so you no longer have allocations for this RP | 14:19 |
| bauzas | they were deleted when the compute service was deleted | 14:20 |
| arne_wiebalck | bauzas: right, this is what I think | 14:20 |
| arne_wiebalck | bauzas: right, this is my understanding | 14:20 |
| arne_wiebalck | bauzas: and I was thinking the resource tracker would update this, but it does not | 14:21 |
| opendevreview | Jorhson Deng proposed openstack/nova master: recheck the attachment_id after the reschedule successful https://review.opendev.org/c/openstack/nova/+/796209 | 14:21 |
| bauzas | arne_wiebalck: just verifying, could you please ask : "openstack resource provider usage show <rp_uuid>" ? https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-usage-show | 14:21 |
| bauzas | if so, we have a nova-manage command helping for you | 14:22 |
| arne_wiebalck | usage 0 | 14:22 |
| bauzas | \o/ | 14:22 |
| arne_wiebalck | https://paste.opendev.org/show/811772/ | 14:22 |
| bauzas | okay, so please https://docs.openstack.org/nova/rocky/cli/nova-manage.html#placement | 14:23 |
| bauzas | shit | 14:23 |
| bauzas | nevermind, same for stein | 14:23 |
| bauzas | nova-manage placement heal_allocations [--max-count <max_count>] [--verbose] [--dry-run] [--instance <instance_uuid>] | 14:23 |
| bauzas | this should heal your allocations | 14:23 |
| bauzas | verify it with --dry-run firstr | 14:23 |
| arne_wiebalck | uh, that looks like what I was looking for, let me try it :) | 14:24 |
| arne_wiebalck | what can go wrong one day before the break :) | 14:24 |
| bauzas | https://docs.openstack.org/nova/latest/cli/nova-manage.html#placement-heal-allocations | 14:24 |
| bauzas | arne_wiebalck: that's why you should test it first :D | 14:25 |
| arne_wiebalck | bauzas: give me a sec, I need to find a good node ... | 14:29 |
| bauzas | sure | 14:29 |
| arne_wiebalck | bauzas: hmm, seems that our client neither has --dry-run, nor --instance ... at least the one I found so far | 14:36 |
| bauzas | looks like it was added in Train | 14:37 |
| bauzas | so maybe we added it in a backport that you don't have in your client | 14:37 |
| arne_wiebalck | yeah, totally possible | 14:37 |
| bauzas | the master doc is explaning it | 14:37 |
| bauzas | but I can doublecheck the stable/stein branch | 14:38 |
| arne_wiebalck | thanks, will do! | 14:38 |
| bauzas | YUUUP https://github.com/openstack/nova/commits/stable/stein/nova/cmd/manage.py | 14:38 |
| arne_wiebalck | ok, let me confirm on our side ... | 14:39 |
| bauzas | https://review.opendev.org/c/openstack/nova/+/693199 | 14:39 |
| bauzas | Branches: stable/stein Tags: 19.1.0 19.2.0 19.3.0 19.3.1 19.3.2 stein-em | 14:40 |
| bauzas | 19.0.0 doesn't have it | 14:40 |
| arne_wiebalck | seems like ours is based on 19.0.1 | 14:41 |
| arne_wiebalck | yeah, that fits | 14:41 |
| arne_wiebalck | without these two options I will rather wait until Belmiro is back :) | 14:42 |
| arne_wiebalck | sounds too risky to run it across the whole deployment | 14:42 |
| arne_wiebalck | is that a sychronous command, i.e. would it block on the CLI ? | 14:44 |
| arne_wiebalck | (or does it launch healing on the background?) | 14:45 |
| bauzas | arne_wiebalck: agreed on holding until belmiro is back | 14:51 |
| bauzas | arne_wiebalck: yes, it's synchronous, it won't fork threads | 14:51 |
| arne_wiebalck | bauzas: thanks a lot for your help! | 14:51 |
| gmann | bauzas: before you start you leave, can you please check these Yoga py3.6 testing updates https://review.opendev.org/c/openstack/nova/+/821636 https://review.opendev.org/c/openstack/os-traits/+/819205 https://review.opendev.org/c/openstack/placement/+/819206 | 15:29 |
| opendevreview | Ilya Popov proposed openstack/nova master: Fix to implement 'pack' or 'spread' VM's NUMA cells https://review.opendev.org/c/openstack/nova/+/805649 | 15:46 |
| bauzas | gmann: ack | 15:48 |
| bauzas | gmann: I just gave you a Christmas present for those 3 :) | 15:50 |
| gmann | bauzas: thanks :), happy Christmas in advance and enjoy your vacation. | 15:51 |
| bauzas | gmann: sure, will be on the ski slopes :) | 15:52 |
| gmann | :) | 15:52 |
| opendevreview | Ghanshyam proposed openstack/nova stable/train: DNM: test tempest train-last tag https://review.opendev.org/c/openstack/nova/+/816598 | 16:12 |
| bauzas | folks, calling it a year. Have good times and hopefully, 2022 will be a better year. | 17:01 |
| *** bauzas is now known as bauzas_pto_till_Jan3rd | 17:13 | |
| gmann | o/ | 17:29 |
| opendevreview | Jonathan Race proposed openstack/nova master: Adds Emulation for aarch64, and ppc64 on x86_64 processors https://review.opendev.org/c/openstack/nova/+/822053 | 17:51 |
| opendevreview | Merged openstack/nova master: Re-add python 3.6 functional testing https://review.opendev.org/c/openstack/nova/+/821636 | 19:32 |
| opendevreview | Merged openstack/placement master: Updating python testing as per Yoga testing runtime https://review.opendev.org/c/openstack/placement/+/819206 | 19:32 |
| opendevreview | Jonathan Race proposed openstack/nova master: Adds Emulation for aarch64, and ppc64 on x86_64 processors https://review.opendev.org/c/openstack/nova/+/822053 | 20:29 |
| opendevreview | Jonathan Race proposed openstack/nova master: Adds Emulation for aarch64, and ppc64 on x86_64 processors https://review.opendev.org/c/openstack/nova/+/822053 | 21:54 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!