| gokhan | hello folks, we are using nova with ceph backend. Sometimes we are getting "nova-compute: Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up" warnings. when I look at the resource provider inventory list, hosts DISK_GB value is not correct. Only one host shows corresct value as ceph pool capacity. https://paste.openstack.org/show/b9g8Js7V63bdf7rKuLqg/ | 07:37 |
|---|---|---|
| gokhan | . I want to be sure if this is expected behaviour with ceph. | 07:37 |
| frickler | gokhan: that does sound like a bug to me, what versions of nova and ceph? probably best create a bug report | 07:58 |
| gokhan | frickler, nova version is 29.2.0 and ceph is 18.2.2. But in my other deployments which has same version, DISK_GB on every hosts is correct. | 08:02 |
| gokhan | frickler, ohh sorry ceph versions sre different. İt is working with ceph 18.2.4 but with ceph 18.2.2 it is not working. may be there is a known issue | 08:06 |
| *** elodilles_pto is now known as elodilles | 08:19 | |
| *** ralonsoh_ is now known as ralonsoh | 09:49 | |
| ishanwar | Hi Everyone! | 10:50 |
| ishanwar | I am planning to take a shot at this LHF https://bugs.launchpad.net/nova/+bug/2027156 | 10:50 |
| ishanwar | Can someone help me understand why we invented to use to the neutron quota API here ? Please dont mind me question I am new to nova dev | 10:51 |
| ishanwar | intend | 10:51 |
| ishanwar | @Sean-k-mooney @bauzas | 10:52 |
| sean-k-mooney | ishanwar: am short version is today we are pulling the full port object with all the extentions and then counting them in python | 10:59 |
| sean-k-mooney | to see if we exceed the quota the project has for total ports | 11:00 |
| sean-k-mooney | but neutron has an api that just returns this as a number | 11:00 |
| sean-k-mooney | so its much faster to use the dedciated api and less work for both nova and neutron if we do | 11:00 |
| sean-k-mooney | so this is just a performance optimisation ot reduce the amount of data sent over the wire and reduce the database load on the neutron side | 11:01 |
| sean-k-mooney | ishanwar: so the idea is, 1 try and use the neutron api exteion for quota details, if that works great we can skip the extra work of counting the ports ourselves, if not fall back to what we do today | 11:02 |
| ishanwar | Hi Sean | 11:06 |
| ishanwar | Thanks for your response | 11:06 |
| ishanwar | The big mentions that the quota api seems to be controlled by Also, the extension seems to be controlled by the cfg.CONF.QUOTAS.quota_driver config option: https://github.com/openstack/neutron/blob/6fa9142ced0eed8de9a89042747438b93de7202d/neutron/extensions/quotasv2_detail.py#L37 | 11:06 |
| ishanwar | So all neutron servers may not have this, how can we check the availability of the api ? | 11:07 |
| sean-k-mooney | there are two way, we can assuem its there and just catch the excption when not and fall back | 11:11 |
| sean-k-mooney | or we can call the extention list api | 11:11 |
| sean-k-mooney | https://docs.openstack.org/api-ref/network/v2/#show-extension-details | 11:11 |
| sean-k-mooney | using the extension list api is probaly better. in either case we have a convention of caching that result in nova | 11:12 |
| sean-k-mooney | like this https://github.com/openstack/nova/blob/master/nova/network/neutron.py#L1435-L1455 | 11:13 |
| ishanwar | thanks let me try to gather more details before create a gerrit review request | 11:39 |
| opendevreview | mitya-eremeev-2 proposed openstack/nova master: Add more thorough handling of evacuated instances https://review.opendev.org/c/openstack/nova/+/944058 | 13:28 |
| dansmith | gibi: I don't understand your response to my comment | 13:39 |
| dansmith | _find_pool() is not where things get sorted into pools, right? It looks through the already-sorted pools and finds the one that _dev_pool (which I think is just the device, right?) matches? | 13:41 |
| opendevreview | mitya-eremeev-2 proposed openstack/nova master: Remove instance directory on live-migration https://review.opendev.org/c/openstack/nova/+/944058 | 13:41 |
| gibi | dansmith: yes, find_pool tries to find a matching pool, the callers of _find_pool creating a logic that if a pool is not found for a dev then a new pool is created for the dev | 13:47 |
| gibi | but as matching logic was broken in _find_pool the upper layer created new pool for the devs | 13:48 |
| gibi | during deallocation | 13:48 |
| dansmith | ohhh, I see okay, let me follow that thread | 13:48 |
| dansmith | since you're changing it and the allocation part I was mostly linking those in my head (and because _find_pool is almost like _filter_pools) | 13:49 |
| dansmith | so if I comment out the _find_pool() part, I fail this part of the test: | 13:50 |
| gibi | yeah the two changed code path are only linked from afar. the _find_pool runs during new device discovery and device deallocatin (putting back to pool). The _apply runs during scheduling of a VM, the consume codepath runs during the pci claim from the pci tracker in the compute | 13:50 |
| dansmith | # assert that the single pool is not broken into two during the de-allocation | 13:50 |
| dansmith | which makes sense based on what you said | 13:50 |
| gibi | yeas | 13:50 |
| gibi | and the _find_pool fix is the only thing that is necessary, the other code change is just safetynet | 13:51 |
| dansmith | gibi: okay and the place where we do that (add a pool if we don't find it) is in add_device().. | 13:54 |
| dansmith | it doesn't look like we run remove_device() when we consume it from the pool or anything, | 13:54 |
| dansmith | so I'm not sure why we're re-running add_device() there, except maybe just as idempotentcy/healing or something? | 13:55 |
| gibi | I think add_device is used both during new device discover at startup and during deallocation when we place the device back to the pool | 13:56 |
| dansmith | I see it used in consume_requests() but the only other place is update_device() which isn't called from here at all | 13:57 |
| dansmith | anyway, it doesn't matter that much because I see the problem now, it just seems confusing to me | 13:57 |
| dansmith | however, at least it's consistent with all the other confusing terminology in here (devs, pools, dev_pools, stats, all of which are kinda the same :) | 13:58 |
| gibi | _allocate_devs pops the device from the pool | 13:59 |
| dansmith | we also call add_device() from _free_device() in compute manager while we're deallocating | 13:59 |
| gibi | during consume_request | 13:59 |
| dansmith | oh, it doesn't call remove_device() but it removes it some other way? | 13:59 |
| gibi | yeah | 13:59 |
| gibi | :/ | 13:59 |
| sean-k-mooney | the way we test in teh schduler if a host passes is we call consume_requests on a copy of the pools and just dont save it from Supprots_requests() | 14:00 |
| gibi | literarily pci_dev = pool['devices'].pop() | 14:00 |
| dansmith | ack | 14:00 |
| dansmith | okay so the add_device is adding it back in after it's no longer used I guess instead of just being marked as in-use... | 14:00 |
| gibi | sean-k-mooney: nope, we call _apply, consume is for the pci tracker | 14:00 |
| gibi | dansmith: yes that is my understanding | 14:01 |
| sean-k-mooney | i may have mixed up the names but the point i wanted to make is, we test if we can allcoate the device by trying to do it on a copy of the data) | 14:01 |
| gibi | sort of yes | 14:01 |
| gibi | even in the scheduler (host manager) we do an allocation on the stats via apply to allow mutlicreate to see a proper resource availability | 14:03 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/pci/stats.py#L779-L821 | 14:03 |
| sean-k-mooney | yep | 14:04 |
| gibi | bah we have 3 entry point not just two: | 14:05 |
| gibi | 1. filters do supports_requests https://github.com/openstack/nova/blob/master/nova/scheduler/filters/pci_passthrough_filter.py#L62 | 14:06 |
| gibi | 2. host manager do apply_request https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L313 | 14:06 |
| gibi | 3. pci_tracker do cosume_request https://github.com/openstack/nova/blob/master/nova/pci/manager.py#L311 | 14:06 |
| gibi | but you are right 1 is basically 2 on a copy of the pools | 14:07 |
| sean-k-mooney | that not all of them | 14:09 |
| sean-k-mooney | it kind of is but | 14:09 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/scheduler/filters/numa_topology_filter.py#L105-L112 | 14:09 |
| sean-k-mooney | numa_fit_instance_to_host also takes into acocunt the pci requests | 14:10 |
| gibi | yeah that is 1 with extras, it calls support_request and look into the pools as well | 14:10 |
| sean-k-mooney | ya so supprots_request is the entry point from the filters effectivly | 14:11 |
| gibi | this is not for the faint-hearthed | 14:11 |
| * gibi hopes he used that idiom in the correct context | 14:12 | |
| dansmith | yup | 14:12 |
| sean-k-mooney | i could have sworn that we had factor the common logic out by the way | 14:14 |
| sean-k-mooney | i.e. that supprot called consume or that support and consume both called apply | 14:14 |
| sean-k-mooney | i dont recall why that does nto seam to be the case | 14:14 |
| gibi | it needs some divergence as consume needs to return a specific device while apply only decrase pool count | 14:15 |
| sean-k-mooney | oh right | 14:15 |
| sean-k-mooney | consume is what we run on the compute | 14:15 |
| gibi | yepp | 14:15 |
| sean-k-mooney | its actully operating not just on the ppoos but the pci_device records and selecting the device | 14:16 |
| dansmith | gibi: another thing that seems messy is that we take a thing, which is a dict'ing of an object, and pop out things that we ignore, and then see if "everything else is the same" right? | 14:16 |
| dansmith | seems like we should opt-in to the things that distinguish them so that new things that get added in later don't cause fracturing of the pools if you don't also update the exclusion list | 14:17 |
| dansmith | (or just do this differently) | 14:17 |
| sean-k-mooney | that would be more maintianable wehn this was first writtin jay was pretty again adding new tag in the future | 14:18 |
| sean-k-mooney | since then we have added several | 14:18 |
| sean-k-mooney | btu never really revisited that orgianl approch | 14:18 |
| gibi | dansmith: I think the concept is that a pool is a set of devs with a set of common field values. So it is the defintion of the interesting fields is the question here | 14:19 |
| sean-k-mooney | gibi: yep | 14:20 |
| gibi | and yes I share you feeling that ignoring flags is less maintainable than listing interesting flags | 14:20 |
| dansmith | right | 14:20 |
| gibi | I'm not motivated to change this logic in a bugfix though. I can queue up a refactor after it. | 14:21 |
| sean-k-mooney | and this is what i mean when i say tis placement like a pool is effectivly an rp with and inventory fo common "fungable" dvices | 14:21 |
| gibi | I would even wrap these dev and pool dicts in proper objects in memory (not ovos) for more readability (and mypy) | 14:22 |
| gibi | I'm bit affraid of what can of worms this opens up | 14:23 |
| gibi | and I have the ovo version bump preemptive unit test and refactor series to get back to as well | 14:24 |
| gibi | I need more time for actually writing code | 14:24 |
| dansmith | gibi: no I'm not suggesting you fix it in this bugfix or even later, I'm just lamenting the design | 14:24 |
| gibi | dansmith: OK | 14:24 |
| dansmith | _especially_ if it means more mypy in this code :) | 14:25 |
| gibi | sh*t I have this from the PCI in Placement work remaining as well https://review.opendev.org/q/topic:%22bp/pci-device-spec-cleanup%22 (danger, it contain mypy changes as well) | 14:26 |
| dansmith | going on three years old there... | 14:26 |
| gibi | yeah it is sad | 14:26 |
| sean-k-mooney | ya... | 14:27 |
| sean-k-mooney | this https://review.opendev.org/c/openstack/nova/+/844491 i think was one fo the more valuable changes | 14:27 |
| sean-k-mooney | although the seise of a whole impoved alot of things | 14:28 |
| gibi | maybe in Flamingo :) | 14:30 |
| gibi | less golang more python :) | 14:31 |
| sean-k-mooney | :) | 14:33 |
| gibi | I don't know what is the relationship of flamingos a pythons, who would eat who | 14:33 |
| dansmith | pythons would eat flamingos no doubt | 14:34 |
| gibi | cool then. (we have neither this side of the globe) | 14:35 |
| gibi | and gopher (the golang mascot) would definitely loose againts python | 14:36 |
| * gibi feels better now | 14:36 | |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Reproduce bug/2098496 https://review.opendev.org/c/openstack/nova/+/941673 | 16:03 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Ignore metadata tags in pci/stats _find_pool logic https://review.opendev.org/c/openstack/nova/+/944277 | 16:03 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Ignore metadata tags in pci/stats _find_pool logic https://review.opendev.org/c/openstack/nova/+/944277 | 16:09 |
| sean-k-mooney | werid i just got a destop notificaiton form a comment gibi made a few hours ago | 17:12 |
| sean-k-mooney | dansmith: not that im disagreeign but dont we test migration with postgres in the tox jobs | 17:13 |
| sean-k-mooney | i would not try and use it in production | 17:13 |
| sean-k-mooney | but we do install it with bindep in the tox env | 17:14 |
| sean-k-mooney | and the opertunistic database stuff should kick in but maybe that broke at some point | 17:14 |
| sean-k-mooney | we defintlly dont have stong testin gthere as we only really apply the migration ot an empty db | 17:14 |
| * sean-k-mooney looks if that even runs | 17:16 | |
| sean-k-mooney | ya nova.tests.unit.db.main.test_migrations.TestMigrationsWalkPostgreSQL.test_walk_versions is still running in tox | 17:18 |
| dansmith | (back in a sec) | 17:20 |
| sean-k-mooney | dansmith: its not urgent dont worry. teh prostgress test are still runing but obviously those migration tests dont test that much | 17:21 |
| sean-k-mooney | pointing to https://governance.openstack.org/tc/resolutions/20170613-postgresql-status.html is still the best responce | 17:21 |
| sean-k-mooney | i was just concerned that those opertuneistic tests broke at some point and we missed it but the pg and mysqlone are running in addtion to sqlite | 17:23 |
| dansmith | sean-k-mooney: we do, but not full of real world data | 17:26 |
| dansmith | so maybe I was less than accurate about 'we don't test it' but testing with an empty DB is not really testing IMHO | 17:27 |
| dansmith | it makes sure all the schema SQL is right, but not much else | 17:27 |
| sean-k-mooney | ya so i know we incorrectly set the coration type on at least one of the tables when movign to alembic | 17:27 |
| sean-k-mooney | so that might cuase some issues only when there is data | 17:27 |
| sean-k-mooney | or on some locales but not our default | 17:28 |
| sean-k-mooney | i think neutron had a postgress job up until about a year ago | 17:28 |
| dansmith | IMHO people that want to run postgres are more than welcome to test and fix the things | 17:30 |
| sean-k-mooney | ya i dont think its unreasonable | 17:31 |
| sean-k-mooney | i just know that it broke for them at some poitn and no one stepped up to fix it so they droped testing | 17:32 |
| sean-k-mooney | oh wow | 17:32 |
| sean-k-mooney | they actully dropped postgress code https://github.com/openstack/neutron/commit/16e4f0af89cc67b7a8901bfd357fd49b86de73c8 | 17:32 |
| sean-k-mooney | i pointed out to stephen that we techinially still supprot it | 17:33 |
| sean-k-mooney | we just dont test it | 17:33 |
| sean-k-mooney | dansmith: im reluctantly +2 but +2 none the same on the one time use device spec | 18:09 |
| dansmith | I will take a reluctant +2 over the alternative thanks :) | 18:09 |
| sean-k-mooney | i lost that tab like 4 times today.... | 18:09 |
| sean-k-mooney | dansmith: by the way you mentioned a crash when you used the qxl device to test this, you might have better luck with somethign like the random number generator or a virtio-blk device | 18:11 |
| sean-k-mooney | or igb vfs if your qemu supports it | 18:11 |
| sean-k-mooney | i have had better look with nic/block devices when palying with double passthough then graphics cards | 18:12 |
| sean-k-mooney | not sure why but just my personal experince | 18:12 |
| dansmith | yeah, no igb on this box's qemu but yeah I need to use a different device | 18:14 |
| dansmith | honestly it reboots in seconds and I always need to restart nova services after each fail, so it hasn't been so bad :P | 18:14 |
| sean-k-mooney | if you try e1000 instead i think that will work | 18:15 |
| sean-k-mooney | it diefenly works with a doubel passthough of a phsyical e1000 nic but i think the virutal ones work too | 18:15 |
| opendevreview | Dan Smith proposed openstack/nova master: Ignore metadata tags in pci/stats _find_pool logic https://review.opendev.org/c/openstack/nova/+/944277 | 19:13 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!