| *** amoralej|off is now known as amoralej | 06:34 | |
| elodilles | good morning o/ fyi, i see lot of tox-docs job failure in zuul (nova + in several other projects) which seems to be a result of the latest oslo.policy release from yesterday. I'll try to look into it, but first I've pinged the oslo team on #openstack-oslo | 06:42 |
|---|---|---|
| gibi | ^^ fix is up and I checked locally it works for the nova jobs https://review.opendev.org/c/openstack/oslo.policy/+/839711 | 09:51 |
| bauzas | thanks gibi | 10:02 |
| bauzas | (sorry was on and off) | 10:03 |
| sean-k-mooney | gibi: when you get a chance can you look at the comments i left on https://review.opendev.org/c/openstack/nova/+/838555 | 10:08 |
| gibi | sean-k-mooney: looking... | 10:41 |
| gibi | and pulling the patch down... | 10:46 |
| gibi | sean-k-mooney: hm, the test passing locally for me even after a rebase | 10:50 |
| opendevreview | Balazs Gibizer proposed openstack/nova stable/train: Remove unavailable but not reported PCI devices at startup https://review.opendev.org/c/openstack/nova/+/839717 | 10:50 |
| opendevreview | Balazs Gibizer proposed openstack/nova stable/train: Simulate bug 1969496 https://review.opendev.org/c/openstack/nova/+/839718 | 10:51 |
| opendevreview | Balazs Gibizer proposed openstack/nova stable/train: Allow claiming PCI PF if child VF is unavailable https://review.opendev.org/c/openstack/nova/+/839719 | 10:51 |
| gibi | let see if after the rebase it will be clean | 10:51 |
| gibi | if not then something is interfeering between test cases | 10:51 |
| gibi | hm | 10:56 |
| gibi | now gerrit says it is in merge conflict | 10:56 |
| gibi | bah I pushed it against stein :D | 10:57 |
| gibi | *train | 10:57 |
| sean-k-mooney | hehe | 10:57 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Remove unavailable but not reported PCI devices at startup https://review.opendev.org/c/openstack/nova/+/838553 | 10:58 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Simulate bug 1969496 https://review.opendev.org/c/openstack/nova/+/838554 | 10:58 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Allow claiming PCI PF if child VF is unavailable https://review.opendev.org/c/openstack/nova/+/838555 | 10:58 |
| gibi | I blame it for the lack of coffein | 10:58 |
| sean-k-mooney | gibi: sorry was pinned for something else runing it locally but ill readd my +2 and review the final patch when it completes | 11:39 |
| gibi | no worries, we have a block in the docs job until https://review.opendev.org/c/openstack/oslo.policy/+/839711 merged and released | 11:40 |
| * sean-k-mooney needs to install openssl dev packages | 11:40 | |
| sean-k-mooney | i breifly looked at that but did not get the full context | 11:41 |
| sean-k-mooney | so oslo-config-generator chagne the kw paramater to a flag? | 11:42 |
| sean-k-mooney | removing the boolean paramater | 11:42 |
| sean-k-mooney | and this is adapting to that backwards incompatible change correct | 11:42 |
| sean-k-mooney | that shoudl have been a major version bump | 11:44 |
| sean-k-mooney | as in oslo.policy=4.0.0 not 3.12.0 | 11:44 |
| sean-k-mooney | gibi: nova.tests.unit.pci.test_manager.PciDevTrackerTestCase.test_set_hvdevs_unavailable_vf_removed is failing for me locally | 11:45 |
| sean-k-mooney | File "/home/sean/repos/openstack/nova-3/nova/objects/pci_device.py", line 238, in _from_db_object | 11:45 |
| sean-k-mooney | setattr(pci_device, key, db_dev[key]) | 11:45 |
| sean-k-mooney | KeyError: 'id' | 11:45 |
| sean-k-mooney | its form the create call | 11:46 |
| sean-k-mooney | File "/home/sean/repos/openstack/nova-3/nova/tests/unit/pci/test_manager.py", line 413, in test_set_hvdevs_unavailable_vf_removed | 11:46 |
| sean-k-mooney | self._create_tracker([fake_db_dev_3, fake_db_dev_4, fake_db_dev_5]) | 11:46 |
| gibi | looking | 11:47 |
| gibi | strange, it is passing for me | 11:49 |
| gibi | do you have commit hash f395e71168 ? | 11:49 |
| sean-k-mooney | f395e71168d843f06c8de7b04874c29f1e10e5a8 | 11:58 |
| sean-k-mooney | ill run it again | 11:58 |
| sean-k-mooney | with -r | 11:58 |
| sean-k-mooney | i think it was a clean venv but we will see if it repoduces | 11:59 |
| sean-k-mooney | oh odd | 12:00 |
| sean-k-mooney | https://zuul.opendev.org/t/openstack/build/f877f8e9afa04b5bb993d6615b8bd558 | 12:00 |
| sean-k-mooney | it failed on the 3.8 arm job | 12:00 |
| sean-k-mooney | but passed on 3.9 | 12:00 |
| gibi | hm I use 3.9 locally | 12:00 |
| sean-k-mooney | let me check which version im using i have 3.8-3.10 locally | 12:01 |
| sean-k-mooney | im proably useing 3.8 | 12:01 |
| gibi | ok it fails iwith 3.8 locally for me too | 12:01 |
| sean-k-mooney | that is super weird | 12:01 |
| gibi | yes | 12:02 |
| sean-k-mooney | this is not in code you are changin i think it appear to be in the fixture/test setup code | 12:02 |
| sean-k-mooney | is it this https://review.opendev.org/c/openstack/nova/+/838553/3/nova/tests/unit/pci/test_manager.py#147 | 12:03 |
| gibi | yes but now I can trace and compare | 12:03 |
| sean-k-mooney | id is technically a reserved keywrod for the id function | 12:03 |
| sean-k-mooney | but when you asign to it as a kwarg it shoudl intoduce it also as a variable | 12:04 |
| sean-k-mooney | its disucuraged but legal | 12:04 |
| sean-k-mooney | hum | 12:05 |
| sean-k-mooney | i wonder if this is realted to not cloning the fake object or soemthing like that | 12:06 |
| sean-k-mooney | i didnt get the failure this time | 12:07 |
| sean-k-mooney | oh this is failing on the first patch | 12:09 |
| sean-k-mooney | oh is it | 12:09 |
| sean-k-mooney | no third i just have the wrong tab open | 12:09 |
| sean-k-mooney | the first patch also hass the same issue | 12:10 |
| gibi | hm, you have a point about cloning | 12:10 |
| gibi | if the id is dropped by our db code then that is now a global change on the db dict | 12:11 |
| sean-k-mooney | yep | 12:11 |
| sean-k-mooney | nova.tests.unit.pci.test_manager.PciDevTrackerTestCase.test_set_hvdevs_unavailable_pf_removed is failing in the first patch | 12:11 |
| sean-k-mooney | you need to do | 12:12 |
| sean-k-mooney | fake_pci_devs = [copy.deepcopy(fake_pci), copy.deepcopy(fake_pci_2), | 12:12 |
| sean-k-mooney | copy.deepcopy(fake_pci_3)] | 12:12 |
| gibi | but non of the test does it so probably the rest of test can be broken too | 12:14 |
| gibi | I can add the deepcopy to _fake_get_pci_devices to fix them all | 12:14 |
| sean-k-mooney | some do | 12:14 |
| sean-k-mooney | like test_set_hvdev_changed_stal | 12:14 |
| sean-k-mooney | many do all_devs = fake_db_devs_tree[:] | 12:15 |
| gibi | that is a shallow copy | 12:15 |
| sean-k-mooney | self._create_tracker(all_devs) | 12:15 |
| gibi | only duplicate the list | 12:15 |
| sean-k-mooney | it is yes | 12:15 |
| sean-k-mooney | but i guess they dont currently modify thigns i guess. | 12:16 |
| gibi | but that does not duplicat the dict having the id field | 12:16 |
| sean-k-mooney | we likely do need to fix other test but i think we have just got lucky | 12:16 |
| sean-k-mooney | thre are certenly sevel test that explcitly do a deepcopy | 12:17 |
| sean-k-mooney | i dont think it s the _create_tracker that copl;es the global state | 12:19 |
| sean-k-mooney | i think its self.tracker._set_hvdevs([fake_db_dev_3, fake_db_dev_4]) | 12:19 |
| sean-k-mooney | _create_tracker does not actully pass the fake devs to the tracker | 12:20 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/tests/unit/pci/test_manager.py#L144-L147= | 12:21 |
| gibi | yeah you are right | 12:22 |
| gibi | the set_hvdev does the coupling | 12:23 |
| sean-k-mooney | yep so you are not actully initalisign the tracker with the devices you created before using them | 12:23 |
| *** amoralej is now known as amoralej|lunch | 12:25 | |
| gibi | I will push a fix | 12:26 |
| gibi | for the new test cases | 12:26 |
| gibi | and a separate commit for the existing ones not having deepcopy before _set_hvdevs | 12:27 |
| sean-k-mooney | so i tihnk it was "working" becaue of this stub https://github.com/openstack/nova/blob/5f5551448dcfcde26095963e223f973b90e6f637/nova/tests/unit/pci/test_manager.py#L153-L154 | 12:27 |
| sean-k-mooney | i was just trying to fiture out how the the tracker had any devices when it was not called | 12:28 |
| sean-k-mooney | that just returns self.fake_devs https://github.com/openstack/nova/blob/5f5551448dcfcde26095963e223f973b90e6f637/nova/tests/unit/pci/test_manager.py#L153-L154 | 12:28 |
| sean-k-mooney | so ya anything not doign a deep copy was sharing state | 12:29 |
| trident | We have had a few policy overrides to allow regular users to create flavors. At some point they seem to have stopped working (probably upgrade to Victoria) - Neither a policy.json with the old format or a policy.yaml seem to help. | 12:32 |
| trident | Any known reasons why a policy override like "os_compute_api:os-flavor-manage:create": "rule:admin_or_owner" (and the same for update and delete) would not work nowadays? | 12:33 |
| trident | Version 22.3.2 | 12:34 |
| sean-k-mooney | it should be possibel to configure that today | 12:41 |
| sean-k-mooney | not advised but possible | 12:42 |
| trident | Weird. I still get a 403 Policy doesn't allow os_compute_api:os-flavor-manage:create to be performed back... | 12:45 |
| trident | sean-k-mooney: You say "not advised" - what would be the advised way to accomplish that? | 12:48 |
| sean-k-mooney | you said it was victoria | 12:49 |
| sean-k-mooney | 22.3.2 | 12:49 |
| sean-k-mooney | 22.3.2 does not exist in git | 12:49 |
| sean-k-mooney | on 22 | 12:49 |
| sean-k-mooney | aslo no does not exist | 12:50 |
| sean-k-mooney | https://github.com/openstack/nova/tree/22.3.2/nova | 12:50 |
| sean-k-mooney | https://github.com/openstack/nova/tree/22.3.0/nova is the most recent release i see | 12:51 |
| sean-k-mooney | same via opendev https://opendev.org/openstack/nova/src/tag/22.3.0 | 12:51 |
| sean-k-mooney | https://opendev.org/openstack/nova/src/tag/22.3.0/nova/policies/flavor_manage.py#L22-L59 | 12:52 |
| sean-k-mooney | this looks correct | 12:52 |
| sean-k-mooney | am you have not turned on scope enforement have you | 12:53 |
| sean-k-mooney | this requires a system scope token | 12:53 |
| sean-k-mooney | trident: you cannot use a project scope token if you have scope enfroment enabled | 12:53 |
| sean-k-mooney | so normal project_member tokens would not work even i fyou udated teh check_str unless you also added 'project' to the scope_types | 12:54 |
| sean-k-mooney | trident: by the way im not sure admin_or_ower shoudl work in this context | 12:56 |
| sean-k-mooney | trident: the resouce does not exsit yet so you cant be an ower of it | 12:56 |
| sean-k-mooney | you woudl want something likse we use for server create | 12:57 |
| sean-k-mooney | https://opendev.org/openstack/nova/src/tag/22.3.0/nova/policies/servers.py#L166-L175 | 12:57 |
| sean-k-mooney | which is just project_member | 12:57 |
| sean-k-mooney | actully no | 12:58 |
| sean-k-mooney | https://opendev.org/openstack/nova/src/tag/22.3.0/nova/policies/create_backup.py#L24-L36 | 12:58 |
| sean-k-mooney | creat backup is better | 12:58 |
| sean-k-mooney | trident: you woudl want PROJECT_MEMBER_OR_SYSTEM_ADMIN | 12:58 |
| sean-k-mooney | with both scope types | 12:58 |
| sean-k-mooney | again we dont really recommend doing this but if i was to do that personally i woudl create role and then in the custom polciy string allow peopel with admin or the "create_flavor" role to create flavors | 12:59 |
| sean-k-mooney | that way you can atleast limit it to a subset of peopel but admin_or_owner shoudl not work for create | 13:00 |
| trident | Hm, yeah, that makes sense - as you say, it doesn't exist, so owner doesn't make sense at the time of creation. I'm however pretty sure those rules have worked previously. Probably at least on train. | 13:01 |
| sean-k-mooney | admin_or_ower was defiend as 'is_admin:True or project_id:%(project_id)s' | 13:01 |
| sean-k-mooney | but flavors are not part of projects | 13:01 |
| sean-k-mooney | so that woudl not work in anycase | 13:01 |
| trident | Thanks for the advise! I'll do some testing and see what I end up with. :) | 13:02 |
| sean-k-mooney | on train it was more or less the same https://opendev.org/openstack/nova/src/tag/train-em/nova/policies/base.py#L27-L30 | 13:02 |
| *** dasm|off is now known as dasm | 13:10 | |
| *** amoralej|lunch is now known as amoralej | 13:11 | |
| gibi | sean-k-mooney: I can reproduce the keyerror on master too with a bit of change and I think I see the issue. _create_tracker should take dict that come from the db with id field but _set_hvdevs should take dict that come from the hypervisor (no id field). My new tests (and one existing test) passed db dict to _set_hvdevs causing the db dict to get corrupted | 13:13 |
| gibi | https://github.com/openstack/nova/blob/028b3bca16c750f6c7edf1b389ed6c79a2c9843d/nova/tests/unit/pci/test_manager.py#L361 | 13:13 |
| sean-k-mooney | ya so that can work if its a copy | 13:13 |
| sean-k-mooney | since it wont affect the gloabl sate but ya | 13:14 |
| gibi | so we need the copy in L361 as that already corrupts the global state | 13:14 |
| gibi | even on master | 13:14 |
| sean-k-mooney | yep | 13:14 |
| sean-k-mooney | well | 13:15 |
| sean-k-mooney | we shoudl do the copy on line 347 | 13:15 |
| sean-k-mooney | or we shoudl hvae _create_tracker do the copy internally | 13:15 |
| sean-k-mooney | and then work on self.fake devs if need | 13:15 |
| sean-k-mooney | but ya in anycase i wonder why we are not seeign this fail in general | 13:16 |
| sean-k-mooney | theses test are not flaky in my experince | 13:16 |
| gibi | we only have one test on master that does the mistake to pass a db dict with id to _sethvdevs. That corrupts the global state but no other test depends on that. Then I added another test that did this mistake and blow if the two test run in the same executor | 13:18 |
| gibi | you can reproduce the issue by duplicating test_set_hvdev_remove_tree_maintained_with_allocations on master | 13:18 |
| sean-k-mooney | ack ok | 13:18 |
| gibi | I will try to clean this up | 13:18 |
| sean-k-mooney | i wonder if we shoudl jsut have _create_tracker deep copy in genreal | 13:19 |
| sean-k-mooney | we can still explictly do it but that would remove the need to do it in the default case | 13:20 |
| sean-k-mooney | only if you call set_hvdevs | 13:20 |
| sean-k-mooney | gibi: glad we caught that before it was merged | 13:21 |
| sean-k-mooney | that kind of think is a pain to debug in the gate when it only failes ocationally | 13:21 |
| gibi | yes, it was a good catch | 13:21 |
| gibi | nobody likes interfeering test cases :) | 13:21 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Isolate PCI tracker unit tests https://review.opendev.org/c/openstack/nova/+/839766 | 13:56 |
| gibi | sean-k-mooney: ^^ I will base the current series on top of this | 13:56 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Isolate PCI tracker unit tests https://review.opendev.org/c/openstack/nova/+/839766 | 14:02 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Remove unavailable but not reported PCI devices at startup https://review.opendev.org/c/openstack/nova/+/838553 | 14:02 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Simulate bug 1969496 https://review.opendev.org/c/openstack/nova/+/838554 | 14:02 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Allow claiming PCI PF if child VF is unavailable https://review.opendev.org/c/openstack/nova/+/838555 | 14:02 |
| sean-k-mooney | gibi: ack ok that makes sesne | 14:13 |
| opendevreview | Elod Illes proposed openstack/nova master: [CI] Install dependencies for docs target https://review.opendev.org/c/openstack/nova/+/839781 | 15:21 |
| elodilles | gibi bauzas : another stable gate fix to review ^^^ when you have time o:) | 15:29 |
| elodilles | (today's broken oslo.policy release showed an error in our tox docs target as the job is failing for stable branches due to a release on zed) | 15:30 |
| *** amoralej is now known as amoralej|off | 15:36 | |
| clarkb | elodilles: gibi bauzas email was sent about that problem a few weeks ago http://lists.openstack.org/pipermail/openstack-discuss/2022-April/028016.html there is a good chance that much of openstack needs that sort of update | 15:46 |
| gibi | elodilles, clarkb: thanks I'm +2 on it | 15:47 |
| sean-k-mooney | when did that get remvoed | 15:49 |
| sean-k-mooney | we used to install requiremets.txt | 15:49 |
| clarkb | sean-k-mooney: a while back there was a big push to switch to trimming the doc requirements down so you didn't have to install everything. What that missed was that the doc builds depended on the projects to collect cli command output and such. Basically I think it was docs having their own requirements that introduced the bug | 15:50 |
| clarkb | the intent was good, but no one realized that this flaw existed | 15:50 |
| sean-k-mooney | ah so it was applied genericly | 15:51 |
| sean-k-mooney | i just did not recally this patch going in | 15:51 |
| sean-k-mooney | we might also need test-requiremetns in some cases but in generaly not | 15:51 |
| elodilles | clarkb: thanks, i'll try to check other projects as i've seen +24 broken stable-periodic tox-docs job today (neutron has already a similar patch on the gate right now) | 15:53 |
| elodilles | (this one: https://review.opendev.org/c/openstack/neutron/+/839777 ) | 15:54 |
| sean-k-mooney | clarkb: so it would b enice to have included the change id of the change that remvoed it but i dont think we shoudl hold this up for that so ill review it now | 15:59 |
| clarkb | I mean its not my change. I just helped debug a similar problem a few weeks ago and we told everyone about it hoping they would audit and fix their repos | 15:59 |
| clarkb | seems that didn't happen hence the current situation | 15:59 |
| sean-k-mooney | i never new this happend i must have missed the mail | 15:59 |
| clarkb | it was a huge cross openstack effort to change the doc build system | 16:02 |
| clarkb | it was a while ago so I don't remember the details just that it happened and a lot of stuff got updates | 16:02 |
| opendevreview | Merged openstack/nova master: [CI] Install dependencies for docs target https://review.opendev.org/c/openstack/nova/+/839781 | 17:57 |
| opendevreview | Balazs Gibizer proposed openstack/nova stable/ussuri: Reproduce bug 1953359 https://review.opendev.org/c/openstack/nova/+/822047 | 18:01 |
| opendevreview | Balazs Gibizer proposed openstack/nova stable/ussuri: Extend the reproducer for 1953359 and 1952915 https://review.opendev.org/c/openstack/nova/+/822048 | 18:01 |
| opendevreview | Balazs Gibizer proposed openstack/nova stable/ussuri: [rt] Apply migration context for incoming migrations https://review.opendev.org/c/openstack/nova/+/822050 | 18:01 |
| opendevreview | Elod Illes proposed openstack/nova stable/yoga: [CI] Install dependencies for docs target https://review.opendev.org/c/openstack/nova/+/839809 | 18:17 |
| opendevreview | Elod Illes proposed openstack/nova stable/xena: [CI] Install dependencies for docs target https://review.opendev.org/c/openstack/nova/+/839810 | 18:22 |
| opendevreview | Elod Illes proposed openstack/nova stable/wallaby: [CI] Install dependencies for docs target https://review.opendev.org/c/openstack/nova/+/839811 | 18:23 |
| opendevreview | Elod Illes proposed openstack/nova stable/victoria: [CI] Install dependencies for docs target https://review.opendev.org/c/openstack/nova/+/839812 | 18:25 |
| opendevreview | Elod Illes proposed openstack/nova stable/ussuri: [CI] Install dependencies for docs target https://review.opendev.org/c/openstack/nova/+/839813 | 18:26 |
| melwitt | dansmith: yoga fix for docs job is ready https://review.opendev.org/c/openstack/nova/+/839809 | 19:06 |
| dansmith | melwitt: I'm going to go out on a limb and say it'd be okay for you to slam those mofos in :) | 19:07 |
| dansmith | you know, IMHO :D | 19:07 |
| sean-k-mooney | i certenly would not object | 19:08 |
| melwitt | haha ok | 19:11 |
| *** artom__ is now known as artom | 19:32 | |
| *** dasm is now known as dasm|off | 20:32 | |
| opendevreview | Dan Smith proposed openstack/nova master: DNM: Run against performance.json patch https://review.opendev.org/c/openstack/nova/+/838934 | 21:37 |
| dansmith | clarkb: around? | 22:11 |
| clarkb | dansmith: hi | 22:15 |
| dansmith | not even a full devstack run and 57k queries to the keystone db.. seems high, no? | 22:15 |
| clarkb | dansmith: that does seem high. But openstackclient does have to get a new token for everything since there is no token caching | 22:16 |
| clarkb | perhaps related to that? | 22:16 |
| dansmith | still, 57k | 22:16 |
| dansmith | also, they're almost all select | 22:16 |
| dansmith | don't we have to insert when we create a token? | 22:16 |
| clarkb | yes I think so | 22:16 |
| dansmith | https://termbin.com/s2xj | 22:17 |
| clarkb | I wonder if we need to instrument keystoen directly to try and identify that? | 22:17 |
| dansmith | I think I'd like to know, cause that seems like some n^2 stuff to me | 22:17 |
| clarkb | all of the other services look pretty reasonable | 22:17 |
| dansmith | yes | 22:17 |
| dansmith | neutron is pretty high, | 22:17 |
| dansmith | and I see the number climb pretty fast when it's creating our network and subnet, which seems weird, | 22:18 |
| dansmith | but it's still not 57k-level concerning | 22:18 |
| clarkb | the token issuance would've been my first guess but I agree that those should be writes not reads | 22:18 |
| dansmith | I would think | 22:18 |
| clarkb | and even then we don't do 57k osc commands | 22:18 |
| clarkb | like maybe 1k | 22:18 |
| dansmith | right | 22:18 |
| clarkb | I guess every other api request wiht a keystone token may have to validate with keystone? | 22:19 |
| clarkb | but napkin math adding everything else together there doens't come close to 57k | 22:19 |
| dansmith | yes, I expect validates to turn into selects, but still seems crazy high | 22:20 |
| dansmith | 113 inserts, so maybe say 100 of those are tokens | 22:20 |
| dansmith | that is 570 validates for each one | 22:21 |
| dansmith | I guess catalog lookups maybe | 22:21 |
| dansmith | but still, production systems must be getting _hammered) | 22:21 |
| clarkb | it definitely seems like identifying the source of those and either reducingthem or making them more performant would be a worthwhile exercise | 22:21 |
| dansmith | yeah curious to see if the keystone people think that's crazy or not | 22:22 |
| dansmith | clarkb: check this: https://zuul.opendev.org/t/openstack/build/f31b8439b6dc47a19f9c99bbe3653d74/log/controller/logs/devstacklog.txt#20158 | 22:24 |
| dansmith | 82k by the end of the devstack run | 22:24 |
| dansmith | no wonder my 100k limit was rolling over on a full tempest run | 22:25 |
| clarkb | wow and that is before tempest runs | 22:25 |
| dansmith | yeah | 22:26 |
| dansmith | like maybe some ORM usage is causing a bunch of lookups each time we pull a token or something | 22:27 |
| dansmith | like 20 queries to pull the catalog entries or something | 22:27 |
| dansmith | dmendiza[m]: I dunno what tz you're in, but are you around? | 22:28 |
| clarkb | dansmith: if you want we can hold a node then you can check the count and check it again 5 minute slater and see if background tasks have a big impact. You can also do things like catalog list and check the delta etc | 22:28 |
| clarkb | though I seem to recall you are good about running local devstack too | 22:29 |
| dansmith | clarkb: it repros locally just fine, but thanks :) | 22:29 |
| dansmith | yeah | 22:29 |
| dansmith | my local run crashed in the middle, which is why I only got to 57k apparently | 22:30 |
| dansmith | it might be cool to have a way to run tempest single-threaded and capture the after-before numbers for each test to see which operations inflate numbers like these | 22:31 |
| dansmith | clarkb: one test into tempest locally and keystone selects increase by 2500 | 23:12 |
| clarkb | dansmith: once you track this down every database behind openstack will owe you beers :) | 23:14 |
| dansmith | I'm just surprised, this might be common knowledge, I dunno | 23:15 |
| dansmith | and I'm just telling you because, I dunno, I need a buddy to be surprised with | 23:15 |
| clarkb | I mean I'm surprised too | 23:17 |
| clarkb | that seems pretty excessive to do a lgocial unit of work in an openstack cloud | 23:17 |
| sean-k-mooney[m] | we are using fernet tokens now instead of uuid tokens right? | 23:17 |
| dansmith | sean-k-mooney[m]: are we? I thought that meant we didn't have to actually store them in order to validate | 23:18 |
| dansmith | provider = fernet | 23:18 |
| sean-k-mooney[m] | so i tought we made the switch by default a while ago | 23:18 |
| dansmith | yep ^ | 23:18 |
| sean-k-mooney[m] | have you tried deploying just keystone and then makeing a singel token issue request to see what that results in from a db point of view | 23:20 |
| dansmith | no, I'm just watching the numbers during runs and was surprised.. wasn't looking for anything, was just like "this number has too many zeroes" | 23:20 |
| clarkb | even then why do fernet tokens make you think thousnads of db queries per logical cloud action? | 23:21 |
| dansmith | neutron queries are super high too, | 23:21 |
| dansmith | I'm running tempest mostly doing compute tests right now and neutron and keystone are both ~25k selects | 23:21 |
| dansmith | and nova-api is ~2500 for reference | 23:22 |
| sean-k-mooney[m] | clarkb they done i was wonderign if we were sitll using uuid tokens by mistake or something like that | 23:22 |
| dansmith | sean-k-mooney[m]: ah, good thought then.. but config mentions fernet, so I assume.. | 23:22 |
| sean-k-mooney[m] | im kind of surpiseed how low placement its in those results | 23:24 |
| sean-k-mooney[m] | for a service that is basicaly a restapi bolted on top a db it does not do much in the db during that run | 23:25 |
| dansmith | sean-k-mooney[m]: that is before a tempest run, so not much for placement to have done yet | 23:25 |
| sean-k-mooney[m] | i assume that keystone is somewhat inflated as presumable devstack is not caching the tokens so every osc call is a new token ectra | 23:26 |
| dansmith | it is, but still, how can it be 80k? | 23:26 |
| sean-k-mooney[m] | it might be interseting to just add a count of osc calls to devstack but ya its very high | 23:27 |
| dansmith | so in my ongoing tempest run, we're testing compute, nova-api has made 3700 select calls, keystone has made 42k so far | 23:28 |
| dansmith | and tempest doesn't get a token for every operation, AFAIK | 23:28 |
| dansmith | (neutron has made 47k btw) | 23:29 |
| sean-k-mooney[m] | that is nuts considering that neutron with ovn which moves some of the state to the ovn db | 23:37 |
| sean-k-mooney[m] | granted ovn is mainly ment to reduce rpc load not db load | 23:37 |
| dansmith | ...and considering we're not testing neutron yet, just using it through nova | 23:37 |
| dansmith | I'm up to 52k for keystone, 80k for neutron | 23:37 |
| sean-k-mooney[m] | there are quite a few warning in the keystone log | 23:38 |
| sean-k-mooney[m] | https://zuul.opendev.org/t/openstack/build/f31b8439b6dc47a19f9c99bbe3653d74/log/controller/logs/screen-keystone.txt?severity=3 | 23:38 |
| dansmith | 6k for nova-api | 23:38 |
| sean-k-mooney[m] | looks the initalial account creation | 23:38 |
| sean-k-mooney[m] | but not sure why the default domain and roles cant be found | 23:39 |
| dansmith | before bootstrap is run maybe? | 23:39 |
| sean-k-mooney[m] | maybe but i would have expected us to bring up keystone pretty early in devstack | 23:40 |
| dansmith | it's mid-early | 23:40 |
| dansmith | or, early-mod | 23:40 |
| dansmith | mid | 23:40 |
| sean-k-mooney[m] | anyway my tablet is about to die so ill check back tomorrow | 23:41 |
| dansmith | o/ | 23:41 |
| dansmith | I'm about to break 100k on neutron, I should crack a beer | 23:41 |
| clarkb | its over 9 thouasand | 23:46 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!