| *** hemna5 is now known as hemna | 01:45 | |
| *** hemna5 is now known as hemna | 03:21 | |
| gibi | sean-k-mooney: thanks. enjoy PTO today | 07:51 |
|---|---|---|
| gibi | chateaulav: you have to look into the openstack service logs or 'spread' VM's NUMA cells | 07:52 |
| gibi | sorry | 07:53 |
| gibi | wrong copy paste buffer | 07:53 |
| gibi | https://zuul.opendev.org/t/openstack/build/39dcedc2915b4dce9bacce7d1c21f8fe/logs | 07:53 |
| gibi | so heer in the job result | 07:53 |
| gibi | under controller/logs and comupute1/logs you will find screen-n-cpu.txt with the nova-compute logs | 07:53 |
| opendevreview | Felix Huettner proposed openstack/nova stable/victoria: Gracefull recovery when attaching volume fails https://review.opendev.org/c/openstack/nova/+/829504 | 07:56 |
| brinzhang | bauzas, gibi, songwenping: vGPU support in Cyborg may need a slot of PTG, do you have some suggestion? | 07:58 |
| opendevreview | Felix Huettner proposed openstack/nova stable/train: Gracefull recovery when attaching volume fails https://review.opendev.org/c/openstack/nova/+/829507 | 08:00 |
| brinzhang | we would like to register 2:00UTC-3:UTC at April 5, is it ok? | 08:01 |
| opendevreview | Ghanshyam proposed openstack/nova master: Separate flavor extra specs policy for server APIs https://review.opendev.org/c/openstack/nova/+/829626 | 08:02 |
| gibi | brinzhang: I'm not sure we have a ptg etherpad yet. as far as I know RedHat folks including bauzas is PTO on today. | 08:04 |
| brinzhang | gibi: ack | 08:07 |
| gibi | so let's get back to this on Monday | 08:07 |
| brinzhang | https://etherpad.opendev.org/p/nova-zed-ptg I saw this link, it's nothing else | 08:07 |
| brinzhang | gibi: ok, we can discuss on monday ^^ | 08:07 |
| gibi | chateaulav: so for example the logs from your trial with my last suggestion visible here (filtered to error only) https://zuul.opendev.org/t/openstack/build/08789d9f0b0546cb9e6fb2b6f4f0231c/log/compute1/logs/screen-n-cpu.txt?severity=4 | 08:12 |
| opendevreview | Felix Huettner proposed openstack/nova stable/stein: Gracefull recovery when attaching volume fails https://review.opendev.org/c/openstack/nova/+/829859 | 08:44 |
| tobias-urdin | gibi: friendly request for backport review https://review.opendev.org/c/openstack/nova/+/828407 :) | 08:45 |
| opendevreview | Felix Huettner proposed openstack/nova stable/rocky: Gracefull recovery when attaching volume fails https://review.opendev.org/c/openstack/nova/+/829860 | 08:51 |
| opendevreview | Felix Huettner proposed openstack/nova stable/queens: Gracefull recovery when attaching volume fails https://review.opendev.org/c/openstack/nova/+/829861 | 08:55 |
| gibi | tobias-urdin: hi! I don't have +2 rights on stable branches :/ | 09:15 |
| tobias-urdin | gibi: oh sorry for the noice! | 09:17 |
| gibi | tobias-urdin: no worries | 09:17 |
| opendevreview | Ghanshyam proposed openstack/nova master: Complete phase-1 of RBAC community-wide goal https://review.opendev.org/c/openstack/nova/+/829866 | 09:29 |
| chateaulav | gibi: ok,I was trying a few things yesterday, because greens was still failing. I'll put the change back and then investigate that way. | 10:51 |
| chateaulav | Grenade | 10:51 |
| gibi | chateaulav: probably easier to create a unit test that calls obj_make_compatible on a populated ComputeNode object | 10:56 |
| gibi | with that you can troubleshoot locally | 10:56 |
| chateaulav | gibi: Yeah. Biggest thing was trying to have zuul pass which it did. I'll see about doing that, and can test to ensure the new values aren't processed | 10:59 |
| chateaulav | gibi: so then with the current patchset, grenade and everything is happy. | 13:22 |
| chateaulav | it did not like those exceptions or moving the backport for hvspec above super | 13:23 |
| gibi | chateaulav: but it did not like with a different reasons. | 13:23 |
| chateaulav | gotcha | 13:24 |
| gibi | chateaulav: I personally would like to keep rejecting new archs like here https://review.opendev.org/c/openstack/nova/+/828369/14/nova/objects/hv_spec.py | 13:24 |
| gibi | chateaulav: I know that you removed this to please grenade | 13:24 |
| gibi | but I think that just hides the problem | 13:24 |
| gibi | I have not time right now to proposa a patch top of yours showing how to make a unit test to show the same problem in a you local env | 13:25 |
| chateaulav | ok, and thats what im starting to see and understand. getting use to zuul and where to find tthings. | 13:25 |
| gibi | but I think that would be a way forward to troubleshoot | 13:25 |
| chateaulav | makes sense | 13:25 |
| gibi | I try to get to your problem before end of today | 13:26 |
| chateaulav | appreciate it, ill be working all day on it. gonna see about that unit test | 13:29 |
| opendevreview | Felix Huettner proposed openstack/nova stable/train: Gracefull recovery when attaching volume fails https://review.opendev.org/c/openstack/nova/+/829507 | 14:06 |
| jamespage | o/ - is there a good reference on how scheduling should behave when hypervisors have partial hugepage memory configuration (say 150GB of 512GB) | 14:51 |
| jamespage | ? | 14:51 |
| gibi | jamespage: I don't think we have. if your instance needs numa topology becasue of cpu pinning or huge pages then the NUMATopologyFilter is responsible to select the proper host | 15:05 |
| artom | jamespage, I don't think the "partial" matters. If the host has enough pages of the correct size to fit the instance, it should pass scheduling | 15:11 |
| jamespage | gibi, artom: interestingly instances with larger page size configuration schedule fine - the problem I'm looking at happens when an instances without large pages gets scheduled to the hypervisor | 15:13 |
| jamespage | deployment is using instances with large amounts of RAM - exceeding the diff between Total RAM - HugePages - Reserved by quite a bit | 15:13 |
| gibi | jamespage: does instances without huge pages are they have any numa related requirements, i.e. cpu pinning? | 15:13 |
| jamespage | gibi: nope no extra specs at all | 15:14 |
| artom | jamespage, wait, that just sounds like there's not enough RAM | 15:15 |
| jamespage | artom: I quite agree there is not enough - but I also expected the instance not to be scheduled to a hypervisor with this memory configuration | 15:16 |
| artom | jamespage, ah, so scheduling passes, but the instance doesn't boot because not enough RAM | 15:16 |
| jamespage | well it does - it then sucks all of the ram up and gets OOM'ed | 15:16 |
| artom | That's... I think that's a known issue? In the sense that Nova doesn't correctly track memory when it's a mix of hugepages and "normal" page size | 15:17 |
| gibi | artom: yeah I think if the small pages instance doesn't request any numa related things then the NUMATopologyFilter is not applied on that instance | 15:19 |
| gibi | but I'm sure sean-k-mooney know a lot more about this | 15:20 |
| gibi | but most of the RH folks on PTO today | 15:20 |
| sean-k-mooney | i just sat down to do something else | 15:20 |
| artom | gibi, it's not even that, I think we treat all memory as available when scheduling, but when the instance gets on the host the huge pages obviously cannot be allocated to it | 15:20 |
| sean-k-mooney | somethign about large guests? | 15:21 |
| * artom tried to avoid pinging Sean to let him enjoy his PTO :P | 15:21 | |
| gibi | sh*t, sorry | 15:21 |
| sean-k-mooney | if you dont use hw:mem_page_size=small or 4k then the numa code does not run | 15:21 |
| sean-k-mooney | its | 15:21 |
| sean-k-mooney | fine | 15:21 |
| artom | Well, it's on his shoulders as well, ignoring IRC is always an option :P | 15:22 |
| sean-k-mooney | i was just goign to check my home insurance renewal | 15:22 |
| gibi | sean-k-mooney: it is about mixing NUMA and non NUMA guests | 15:22 |
| sean-k-mooney | ya that is not supported today | 15:22 |
| gibi | jamespage: above sees memory overallocation issues | 15:22 |
| gibi | sean-k-mooney: so you say that if hw:mem_page_size=small is added then we allow mixing small and large instances? | 15:23 |
| sean-k-mooney | you must not under any cirucmstance mix numa and non numa guest or it will break all our memory tracking | 15:23 |
| sean-k-mooney | yes | 15:23 |
| sean-k-mooney | if you use hw:mem_page_size=small its fine | 15:23 |
| sean-k-mooney | brb | 15:23 |
| sean-k-mooney | tl;dr is using hw:mem_page_small will numa affine the guest and track the memory correctly | 15:25 |
| sean-k-mooney | if you dont it will float and cause random OOM events | 15:25 |
| gibi | thanks! | 15:26 |
| sean-k-mooney | placement is not enough to save us since the OOM reaper runs per numa node and also hugepage guest dont supprot oversubsricption | 15:26 |
| sean-k-mooney | so if if you have memeroy over subscrion it wont really work right | 15:26 |
| sean-k-mooney | we have something about this in our downstream docs but basically we say use hostaggreates to prevnt mixing numa and non numa guests | 15:27 |
| sean-k-mooney | jamespage: if you have enuch swap by the way those other instace coudl technicaly boot but the system would likely be unstable so its better to avoid that | 15:29 |
| jamespage | sorry - power outage just at the wrong moment | 15:29 |
| gibi | jamespage: https://meetings.opendev.org/irclogs/%23openstack-nova/latest.log.html here are the IRC logs if you are dropped | 15:30 |
| sean-k-mooney | jamespage: the reason it can be schduled to the host is in placemnt we report total ram and then only reserver what set in host_reserver_ram or what ever that option is called | 15:31 |
| * jamespage reads backscroll | 15:31 | |
| sean-k-mooney | jamespage: we dont currently automaticlly reserve the hugepage memrory because we dont currenlty track that as a seperate pool | 15:31 |
| sean-k-mooney | so the schduelr is only looking at total-reserved not total -reserved - hugepages | 15:32 |
| sean-k-mooney | actully if its a recent release we dont even have a ram filter anymore so its just placment that is checking | 15:33 |
| jamespage | sean-k-mooney: having read most of the code that was my hunch so thanks for confirming... | 15:33 |
| sean-k-mooney | so ya that is why we say you cant mix numa and non numa today | 15:34 |
| sean-k-mooney | if we ever get to tracking numa in placment then it will fix that | 15:34 |
| sean-k-mooney | sicne we will have a seperate pool per page size | 15:34 |
| sean-k-mooney | and or numa node | 15:34 |
| sean-k-mooney | we proably could paper over this temporally with a filter similar to the nuam one or even enhance the numa one to work with non numa instance but right now we just tell people not ot do it | 15:36 |
| artom | I guess another option would be to tell operators to include the amount of hugepages in reserved_host_ram? | 15:42 |
| artom | It's... weird, but should work? | 15:42 |
| sean-k-mooney | no | 15:42 |
| sean-k-mooney | it will break schduling | 15:43 |
| artom | Wouldn't it just prevent scheduling to a host if it doesn't have enough "normal" RAM? | 15:43 |
| sean-k-mooney | yes but you vould not use the hugepages and normal ram | 15:43 |
| artom | Oh, right | 15:44 |
| sean-k-mooney | you coul only use total - reserved | 15:44 |
| sean-k-mooney | so that is what you do if you are using the hugepage on the host | 15:44 |
| sean-k-mooney | + the huge page reserved option | 15:45 |
| sean-k-mooney | so we coudl set max allocation size in placement = to total -hugepages | 15:50 |
| sean-k-mooney | that would help | 15:51 |
| jamespage | side effect to that would be that you could only ever allocate total - hugepages on that hypervisor no? | 15:52 |
| sean-k-mooney | no | 15:52 |
| jamespage | oh that's the max allocation size - driving the maximum single instance footprint - I see | 15:53 |
| sean-k-mooney | https://docs.openstack.org/api-ref/placement/?expanded=update-resource-provider-inventory-detail#update-resource-provider-inventory | 15:53 |
| sean-k-mooney | yes | 15:53 |
| sean-k-mooney | max_unit | 15:53 |
| sean-k-mooney | sorry was just on the phone | 15:54 |
| sean-k-mooney | so ya total woudl be set to total memroy | 15:54 |
| sean-k-mooney | reserved = reserved form config | 15:54 |
| sean-k-mooney | but if we set max_unit to total-reserved-hugepages that should help | 15:55 |
| sean-k-mooney | although it would have to be the larger of | 15:55 |
| sean-k-mooney | total-reserved-hugepages and hugepages | 15:55 |
| sean-k-mooney | to not limit the size of hugepage guests | 15:56 |
| sean-k-mooney | jamespage: is it a resent release | 15:56 |
| sean-k-mooney | if so im not sure if you can use provider.yaml to twaak the make unit | 15:57 |
| sean-k-mooney | gibi: do you rememebr if that can tweak non custom resouce classes? | 15:57 |
| jamespage | ussuri for this particular deploy | 15:57 |
| gibi | Only CUSTOM_* resource classes and traits may be managed this way. | 15:58 |
| gibi | so no | 15:58 |
| gibi | https://docs.openstack.org/nova/latest/admin/managing-resource-providers.html | 15:59 |
| sean-k-mooney | ya ok | 15:59 |
| sean-k-mooney | it was afetr ussuri anyway | 15:59 |
| sean-k-mooney | jamespage: best we likely could do is a config option that could posible be backported to set or clamp the max unit | 15:59 |
| sean-k-mooney | or just not mix for now | 16:00 |
| gibi | we merged in victoria | 16:00 |
| sean-k-mooney | jamespage: sorry i cant really be more help there. you could write a custom filter to do the same thing out of tree and load that in the env since its plugable | 16:01 |
| jamespage | sean-k-mooney: no worries - you have been more that helpful in getting my knowledge up to speed in this area :) | 16:02 |
| sean-k-mooney | for existign deployment that is the only way to make mixing numa/non numa on the same host somewhat ok but it still risks OOM events | 16:02 |
| sean-k-mooney | ok i need to go do some other things and my home insure is not done so ill be afk now until monday | 16:03 |
| sean-k-mooney | enjoy your weekends folks o/ | 16:03 |
| gibi | o/ | 16:04 |
| gibi | chateaulav: sorry I run out of time today. I will try to look at the issue tomorrow | 16:51 |
| gibi | o/ | 16:51 |
| chateaulav | thats sounds good, i think im in the right place and have been playing around with the tests for compute node. | 16:52 |
| opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: WIP Improve remote-managed port test coverage https://review.opendev.org/c/openstack/nova/+/829974 | 18:33 |
| opendevreview | Rajat Dhasmana proposed openstack/nova master: WIP: Add support for volume backed server rebuild https://review.opendev.org/c/openstack/nova/+/820368 | 18:34 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: block_device_info: Add swap to inline https://review.opendev.org/c/openstack/nova/+/826523 | 21:03 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: libvirt: Improve creating images INFO log https://review.opendev.org/c/openstack/nova/+/826524 | 21:03 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: libvirt: Remove defunct comment https://review.opendev.org/c/openstack/nova/+/826525 | 21:03 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: imagebackend: default by_name image_type to config correctly https://review.opendev.org/c/openstack/nova/+/826526 | 21:03 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: image_meta: Add ephemeral encryption properties https://review.opendev.org/c/openstack/nova/+/760454 | 21:03 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: BlockDeviceMapping: Add encryption fields https://review.opendev.org/c/openstack/nova/+/760453 | 21:03 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: BlockDeviceMapping: Add is_local property https://review.opendev.org/c/openstack/nova/+/764485 | 21:03 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: compute: Update bdms with ephemeral encryption details when requested https://review.opendev.org/c/openstack/nova/+/764486 | 21:03 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: virt: Add ephemeral encryption flag https://review.opendev.org/c/openstack/nova/+/760455 | 21:03 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: scheduler: Add an ephemeral encryption pre filter https://review.opendev.org/c/openstack/nova/+/760456 | 21:03 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: block_device: Add DriverImageBlockDevice to block_device_info https://review.opendev.org/c/openstack/nova/+/826527 | 21:03 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: block_device: Add encryption attributes to image and ephemeral disks https://review.opendev.org/c/openstack/nova/+/826528 | 21:04 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: virt: Add block_device_info helper to find encrypted disks https://review.opendev.org/c/openstack/nova/+/826529 | 21:04 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: blockinfo: Add encryption details to the disk_info mappings when provided https://review.opendev.org/c/openstack/nova/+/772272 | 21:04 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: imagebackend: Add disk_info_mapping as an optional attribute of Image https://review.opendev.org/c/openstack/nova/+/826530 | 21:04 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: privsep: Move qemu-img create calls under nova.privsep.qemu https://review.opendev.org/c/openstack/nova/+/826750 | 21:04 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: privsep: Return QemuImgInfo objects from qemu-img info calls https://review.opendev.org/c/openstack/nova/+/826751 | 21:04 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: privsep: Add encryption support to qemu-img create command https://review.opendev.org/c/openstack/nova/+/826752 | 21:04 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: libvirt: Report ephemeral encryption traits based on imagebackend https://review.opendev.org/c/openstack/nova/+/826753 | 21:04 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: libvirt: Configure and teardown ephemeral encryption secrets https://review.opendev.org/c/openstack/nova/+/826754 | 21:04 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: imagebackend: Add support to libvirt_info for LUKS based encryption https://review.opendev.org/c/openstack/nova/+/826755 | 21:04 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: imagebackend: Cache the key manager when disk is encrypted https://review.opendev.org/c/openstack/nova/+/826756 | 21:04 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: libvirt: Introduce support for qcow2 with LUKS https://review.opendev.org/c/openstack/nova/+/772273 | 21:04 |
| opendevreview | Jonathan Race proposed openstack/nova master: object/notification for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828369 | 22:02 |
| opendevreview | Jonathan Race proposed openstack/nova master: driver/secheduler/docs for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/822053 | 22:02 |
| opendevreview | Jonathan Race proposed openstack/nova master: zuul-job for Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/828372 | 22:02 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!