Thursday, 2021-08-26

opendevreviewMerged openstack/nova stable/train: Raise InstanceMappingNotFound if StaleDataError is encountered  https://review.opendev.org/c/openstack/nova/+/77725400:35
jameshydeis this a known bug? every time after live migrating a VM from one hypervisor (source) to another hypervisor (target), nova-compute service is still up and running, but nova-conductor from controller keeps saying this: Failed to compute_task_migrate_server: Compute service of hostname_here is unavailable at this time.: nova.exception.ComputeServiceUnavailable: Compute service of hostname_here is 02:45
jameshydeit complains about nova compute service on source host02:46
jameshyderestarting nova-compute works around this issue, but this requiring restarting nova-compute every time after vm live migration away from source host. 02:47
*** abhishekk is now known as akekane|home03:55
*** akekane|home is now known as abhishekk03:55
*** akekane_ is now known as abhishekk06:01
*** iurygregory_ is now known as iurygregory06:28
opendevreviewJorhson Deng proposed openstack/nova master: recheck the attachment_id after the reschedule successful  https://review.opendev.org/c/openstack/nova/+/79620906:57
*** rpittau|afk is now known as rpittau07:24
kevinzsean-k-mooney: thanks for the update,  will ping them next week.07:28
lyarwoodjamesdenton:  That sounds like a bug if the source service is marked as up and active08:11
lyarwoodjameshyde sorry08:12
opendevreviewsean mooney proposed openstack/nova master: db: Handle parameters in DB strings  https://review.opendev.org/c/openstack/nova/+/80566309:59
sean-k-mooneylyarwood: it proably just the bug where rabbit mq loses i thin the topic on the compute recive queue10:00
opendevreviewStephen Finucane proposed openstack/nova master: api: Add support for 'hostname' parameter  https://review.opendev.org/c/openstack/nova/+/77855010:00
opendevreviewStephen Finucane proposed openstack/nova master: tests: Speed up 'servers' API tests  https://review.opendev.org/c/openstack/nova/+/77873210:00
opendevreviewStephen Finucane proposed openstack/nova master: policy: Deprecate field from 'os-extended-server-attributes' policy  https://review.opendev.org/c/openstack/nova/+/80613110:00
sean-k-mooneythe one where the compute agent can send rpcs like the heartbeat but not recive them until you recreate the queue10:00
sean-k-mooneyeight that or the hostname has change to the instace.host does not match the current host10:01
sean-k-mooneyboth would break the conductors ablity to manage vms on the host but the agent would be up10:01
lyarwoodI was assuming it came from here and the servicegroup API tbh10:02
lyarwoodhttps://github.com/openstack/nova/blob/2a78626a85954997d35f5fe62c50de297e2ca92d/nova/conductor/tasks/live_migrate.py#L284-L28810:02
lyarwoodso assuming that's using the db driver10:02
sean-k-mooney well the host in that case comes form instnace.host10:02
sean-k-mooneyso if the host name changed on the node that would not fine it properly10:03
lyarwoodyup right10:03
sean-k-mooneylyarwood: melwitt  by the way just updated the db url test assertin gthe content of the alembic conf and equating it to the the input string10:05
lyarwoodah awesome missed that above10:06
* lyarwood rechecks the tripleo test patch10:06
sean-k-mooneyjust did10:06
sean-k-mooneyaltought its complaing about failing to merge?10:06
sean-k-mooneyoh we both did10:06
sean-k-mooneyright pep8...10:07
sean-k-mooneyfor some reason i dont have pre-commmit in this once copy of nova10:08
lyarwoodfailed to merge was just because you pushed a new ps of the nova change10:08
sean-k-mooneyoh ok10:08
sean-k-mooneyoh melwitt  was right about the glbal needing to be reset10:11
sean-k-mooneydepending on order the test breaks ill fix that and push it again10:12
sean-k-mooneythis is infuriating10:58
sean-k-mooneymy test works on its own but the connection string is not being passed to alembic if i run  multiple db tests10:59
sean-k-mooneyreference = 'sqlite:///:memory:?read_default_group=data+with%2Fa+percent_%25-and+symbols%21'10:59
sean-k-mooneyactual    = 'sqlite://'10:59
sean-k-mooneyok i think i have it working11:02
sean-k-mooneybut its one of those thing where you need to run it many times to be sure11:03
opendevreviewsean mooney proposed openstack/nova master: db: Handle parameters in DB strings  https://review.opendev.org/c/openstack/nova/+/80566311:07
opendevreviewsean mooney proposed openstack/nova master: db: Handle parameters in DB strings  https://review.opendev.org/c/openstack/nova/+/80566311:14
sean-k-mooneylyarwood: ok unless there is review feedback ^ is the final version of that patch11:15
sean-k-mooneyi have also restarted the ooo dnm patch so we will see if it works for them in a while11:15
sean-k-mooneystephenfin: oh you reviewd at some point am i did not see that until i looked at my eamil11:16
lyarwoodmelwitt: https://review.opendev.org/c/openstack/nova/+/800634/12..13/nova/tests/unit/cmd/test_manage.py - would you mind taking another look at this today when you're online, I think I've finally addressed everything.11:23
opendevreviewsean mooney proposed openstack/nova master: [FUP] minor cleanup in db migration tests  https://review.opendev.org/c/openstack/nova/+/80614311:47
opendevreviewsean mooney proposed openstack/nova master: [FUP] minor cleanup in db URL tests  https://review.opendev.org/c/openstack/nova/+/80614311:48
artomHeh, why the fup? Original patch hasn't gotten a single +2 yet? :)12:10
artomAh, gate12:10
sean-k-mooneyya i want to see the results12:13
sean-k-mooneyill likely squash it12:13
gibisomething is happening with placement concurrent allocation update issue https://paste.opendev.org/show/808348/13:26
gibithe amount of job failure due to that getting significantly worse13:27
sean-k-mooneyhehe that one way to shorten a url13:27
gibiif you have a hammer ...13:27
sean-k-mooneyyes it hit melwitt patches yesterday i think13:27
sean-k-mooneyi this the same issue that is cause by a concurent delete request13:28
sean-k-mooneyif so melwitt  has patches for that as im sure your aware13:28
gibiprobably we always had this as in http://status.openstack.org/elastic-recheck/#183675413:28
gibibut now it become significant13:28
sean-k-mooneyhttps://review.opendev.org/c/openstack/nova/+/688802 this is what i was thinking of13:32
sean-k-mooneyalthough i think we shoudl jsut go back to calling delete on the allocation and ignoring the version personally instead of doing a put13:33
opendevreviewsean mooney proposed openstack/nova master: Add autopep8 to tox and pre-commit  https://review.opendev.org/c/openstack/nova/+/80618213:35
artomsean-k-mooney, we may need for melwitt to wake up for this one, but... does the database fixture actually mock out anything?13:41
artomI don't grok how it's actually used13:41
sean-k-mooneyits creating a real db using sqlight in memory database feature13:42
artomOK... So for instance if you pass it a connection URL, it'll create a new context and use its engine: https://github.com/openstack/nova/blob/master/nova/tests/fixtures/nova.py#L62513:43
sean-k-mooneyits an interesting question though if i need it  now that i have added the other fixture for get engine13:43
artomBut then the migration code will always just use the main_db_api.get_engine: https://github.com/openstack/nova/blob/master/nova/db/migration.py#L4713:43
sean-k-mooneyill try droping it now that i have added the addtional fixture and see if that is enough13:44
artomReason I'm asking is - I've tried using the connection=url kwarg to the Database fixture to avoid your mocking while still fixing the global engine issue13:44
sean-k-mooneyi am using this to run a db sync after all so i dont need the db fixture to set other things up for me13:44
sean-k-mooneyartom: ya i trided that too but it wont work for 2 reaons13:45
artomAnd they are definitely different engines - but the test uses the main_db_api engine, and not the fixture engine13:45
sean-k-mooneyartom: im mocking out get_enging in the migration 13:46
sean-k-mooneymodule13:46
sean-k-mooneyso it shoudl be useing the new fixtures  engine 13:46
artomRight, I get that. I'm trying to understand why it's even necessary with the database fixture13:46
sean-k-mooneybecause the engine.url is imutable13:47
sean-k-mooneyand its reused between multipel tests13:47
sean-k-mooneyso it end up with a different connection string then the one we pass13:47
artomRight - but we can't use different engines?13:47
sean-k-mooneyeven when we use the connection parmater in the db fixture13:47
artomPer test, for tests that need it?13:47
sean-k-mooneythat is what https://review.opendev.org/c/openstack/nova/+/805663/6/nova/tests/unit/db/test_migration.py#50 is doing13:48
sean-k-mooneyartom: the engine url used to be mutable im not sure our db fixtuer has worked with custom connections since that change in sqlalcahme about a year or 2 agao13:49
artomYeah... I just thought the fixture would handle that13:49
artom?13:49
artomEspecially with the enticingly named connection=url kwarg :)13:49
sean-k-mooneyi think it used too but it definetly does not now13:50
artomWeird.13:50
artomAnd it's only used in 1 spot, the kwarg13:50
sean-k-mooneyok so i dont need the db fixture just my one for 'nova.db.migration._get_engine'13:51
sean-k-mooneythe test will still work if i remove the db fixture13:51
sean-k-mooneygood catch ill update that when i squah things13:52
sean-k-mooneyooo is still runnign so i dont want to interupt that job13:53
sean-k-mooneyartom: you mentioned its used in only one place? 13:56
sean-k-mooneywhere is that13:56
sean-k-mooneymaybe i should remove the parmater in a follow up and fix that13:56
opendevreviewJorhson Deng proposed openstack/nova master: recheck the attachment_id after the reschedule successful  https://review.opendev.org/c/openstack/nova/+/79620913:56
sean-k-mooneyor well make the connnection parmater work by creating an new engin instnace13:57
artomsean-k-mooney, well, it appear to correctly create a new engine14:14
artom.. which then never gets used anywhere, AFAICT14:14
artomnova/tests/functional/db/test_connection_switch.py is where it gets used14:15
artom        self.useFixture(nova_fixtures.Database(connection=self.fake_conn))                    14:15
lyarwoodstupid question if anyone with vif/neutron background can help, I'm trying to work out how to find a default for hw_vif_model when it's unset14:18
lyarwoodI thought osinfo.HardwareProperties(instance.image_meta).network_model was what I wanted but I'm seeing it return None14:18
sean-k-mooneylyarwood: its virtio 14:18
lyarwoodjust hardcoded?14:18
sean-k-mooneyi think so i think i know where it is14:19
lyarwoodkk I've missed it then14:19
sean-k-mooneyits virt dirver dependent14:19
sean-k-mooneyvmware default to a differnt thing then libvirt14:19
opendevreviewStephan Pampel proposed openstack/nova master: docs: admin/networking rename neutron_tunneled to neutron_tunnel  https://review.opendev.org/c/openstack/nova/+/80619714:20
lyarwoodhttps://github.com/openstack/nova/blob/master/nova/virt/osinfo.py#L99-L112 was where I was looking FWIW14:20
lyarwoodjust without os_distro set this doesn't seem to do anything14:20
sean-k-mooneyhttps://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py#L211-L21514:21
sean-k-mooneysorry wrong link14:21
sean-k-mooneybut its not commig form osinfo its form vif.py14:21
lyarwoodkk14:21
sean-k-mooneyhttps://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py#L162-L17814:22
sean-k-mooneywe do check os_info but its more complicated then that14:22
lyarwoodah14:23
sean-k-mooneythis is where its then called in get_base_config https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py#L20514:24
sean-k-mooneylyarwood: out of interest why were you looking for that14:25
lyarwoodsean-k-mooney: for the device bus migration stuff, the initial part is to register the current values14:25
lyarwoodsean-k-mooney: so I was trying to see if I can pull the default for this easily like I can with hw_disk_bus/hw_cdrom_bus from disk_info14:26
sean-k-mooneyah right ya so you can jsut call that driver function to get the model14:26
lyarwoodsean-k-mooney: without inspecting the instance itself etc14:26
lyarwoodsean-k-mooney: any idea how I'd get an instance of that VIF class?14:26
sean-k-mooneywe construct it in the driver14:26
* lyarwood greps around14:26
lyarwoodkk14:26
lyarwoodah self.vif_driver, awesome14:27
sean-k-mooneyif you need too though you can just constuct it with no args14:27
sean-k-mooney get_vif_model does not rely on self.host but ya self.vif_driver14:28
sean-k-mooneythis used to be plugable 14:28
sean-k-mooneyi dont think it is anymore14:28
melwittgibi, sean-k-mooney: I support reverting the patch that changed from DELETE of allocations and I've been thinking about it more lately and think it's the better thing to do anyway14:59
gibimelwitt, sean-k-mooney: I have to load context for that. I only saw the placement conflict error appearnig more frequently in the last ~24 hours. 15:00
melwittgibi: yes, it's because of consumer types I'm afraid :(15:01
gibimelwitt: is there a connection?15:03
melwittgibi: I think there is but I want to look through to make sure it makes sense15:03
gibiOK, I can also spend some time on it tomorrow15:05
melwittlyarwood: ack will look again15:06
melwittsean-k-mooney: ack will look at the db url patch again15:07
sean-k-mooneymelwitt: i have one update to push for that which is the db fixture is not required 15:54
sean-k-mooneyand im going to suqah the follow up patch in to adress stephens nits but the main content of hte patch wont change15:54
sean-k-mooneymelwitt: actully the ooo job has now completed so ill push those change in the next 5 mins or so15:55
sean-k-mooneyok that looks like it fixes ooo https://review.opendev.org/c/openstack/tripleo-heat-templates/+/80610315:57
*** rpittau is now known as rpittau|afk16:02
opendevreviewsean mooney proposed openstack/nova master: db: Handle parameters in DB strings  https://review.opendev.org/c/openstack/nova/+/80566316:07
sean-k-mooneyartom: melwitt lyarwood ^ let me know if there is anything else needed im going to switch to something else for a bit16:08
*** akekane_ is now known as abhishekk16:18
sean-k-mooneyartom: thanks ill adress those comments later today16:52
opendevreviewRodrigo Barbieri proposed openstack/nova stable/victoria: Fix 1vcpu error with multiqueue and vif_type=tap  https://review.opendev.org/c/openstack/nova/+/80600417:01
sean-k-mooneymelwitt: actully in terms of gloabls i dont think we should be resetting them i relaly think we should be mocking them instead17:31
sean-k-mooneymelwitt: but i didnt really want to open that discussion in those patches17:31
sean-k-mooneyi dont think resetting them is safe from a concurancy point of view17:32
melwittok17:32
sean-k-mooneywhere asd the mock approch should be17:32
sean-k-mooneyi guess it might be ok since we will be runing the test in different python processes17:33
melwittyeah, that's why it's ok to do17:33
melwitt*ok to do in our case17:33
melwittit's about (random) test ordering and whether leaving a particular state will impact a test running after17:34
sean-k-mooneyright form a style point of view im not sure direct asignment is preferable vs the mock but ill pull this out into a follow up patch if that works for you and artom17:34
sean-k-mooneymelwitt: yep i have defiently seen the issues sharing state can cause17:35
melwittsean-k-mooney: I'm completely fine with waiting until it's a problem. it may be that we don't have any tests that would be impacted by leftover state17:35
artomDon't fix problems we don't have :)17:35
artomWe have enough problems that we, err, have :P17:35
sean-k-mooneyartom: well the concern here is this is really really hard to debug when it happens17:36
melwitt++17:36
artomFair point, too17:37
artomIt's just weird seeing code that basically goes "weeeell just in case"17:37
melwittfwiw I agree the mocks in the patch should be removed, if we reset the globals it should be for all tests, not only the one17:39
sean-k-mooneyi partly agree with taht but not convifced how we do it today is correct :) ill remove it for now17:40
melwittre: resetting the globals in nova/test.py like the rest of them, I think that's the right thing to do (it's the correct thing to do) but it's not directly related to the patch at hand. that could and probably should be a separate patch17:40
sean-k-mooneyartom: just to be clear what melwitt  was concerned about is https://github.com/openstack/nova/blob/e27a0135f8ec986d3583b11276715a5c0d0c302a/nova/db/migration.py#L75-L83 would retrun a cached copy of the config object and that would have state that would break us or cause inter test depency17:40
sean-k-mooneywhich is technially a valid concern the connection stirg will proably leak if i remove the current mock17:41
sean-k-mooneyit should not break anything but ya it will leak state17:42
artomWell, there's no proper way to fix that then, is there?17:42
sean-k-mooneyits what the mock does17:42
melwittit's just something I noticed and I have unfortunately broken the gate twice I think with forgetting to reset global vars I introduced, so I am particularly attuned to it 😂17:42
sean-k-mooneyit replace the dict its using as a cache 17:42
artomBecause while we can definitely reset state *after* our tests, what happens for concurrently running tests?17:42
sean-k-mooneyartom: they are in differnt python processes17:42
artomThey're not running 1 per process, are they?17:43
sean-k-mooneywe only ever run one test in each proces at a time17:43
melwittartom: we currently do them in test.py. here's some examples and there are more if you read further down in setUp() https://github.com/openstack/nova/blob/2a78626a85954997d35f5fe62c50de297e2ca92d/nova/test.py#L227-L23117:43
sean-k-mooneythe test a loadbalnce over a set of process by tox17:43
sean-k-mooneymelwitt: well that is not the only place17:43
sean-k-mooneywe have other local reset too17:44
* artom is leaning to melwitt's approach - if any global state needs to be reset between tests, base setUp() is where to do it...17:44
artomOtherwise it's whackamole, no?17:45
sean-k-mooneywell we would have to reset alto more state tehere then we curerntly do17:45
sean-k-mooneynormally we reset them in the test that tested the code tha tuse the global17:45
melwittI didn't come up with the approach but I have followed the pattern when I had patches that introduced new global vars17:45
sean-k-mooneymelwitt: i have always followed a differnt pattern like this https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/tests/unit/test_cinder.py#L87 https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/volume/cinder.py#L5517:46
melwittwell, for things like a CELL_CACHE *lots* of things use it, even if they're not specifically setting out to use it17:47
sean-k-mooneythis is the first time i have seen it centralised17:47
dansmithI dunno what patch ya'll are talking about,17:47
sean-k-mooneyi expect that modules have a reset funciton in them to rest the module sttate17:47
dansmithbut anything the test expects should be mocked, and only incidental globals i.e. CELL_CACHE should be reset in test.py each time, IMHO17:47
sean-k-mooneydansmith: https://review.opendev.org/c/openstack/nova/+/80566317:48
sean-k-mooneywe cache the alembic config in a global17:48
sean-k-mooneydansmith: so i was mocking the dict use for the cache so that we start form an empty dict and dont leak the state17:49
dansmithyeah, that's exactly the sort of incidental global we _should_ reset in test.py I think17:49
sean-k-mooneybut we dont do that for a lot of other cases17:49
dansmithpresumably that gets touched by some tests that don't even realize it right?17:49
artomsean-k-mooney, wait, shouldn't the clearing be happening *after* the test?17:49
artomOr before the other tests run, in setUp()?17:50
dansmithartom: in setUp() is conventional yeah17:50
sean-k-mooneyartom: no with the mock it happens before and it restores the previous state after17:50
dansmithartom: i.e. don't clean up from other tests, reset state to known before running next test17:50
artomOn right, the mock context manager17:50
artomBut it'd still restore any state that was there previously, if another test leaked it...17:50
sean-k-mooneyyes17:51
artomSo yeah, I vote in setUp() in a follow up patch :)17:51
artomAlso, the Database fixture connection= kwarg is weird and needs to go, it's misleading17:51
sean-k-mooneyi can put it in setup but i dont think it should be in test.py17:51
sean-k-mooneyhttps://github.com/openstack/nova/search?q=reset_globals17:51
sean-k-mooneyi thikn that patern is cleaner the reset_gloals function is called in setup of the test class that test that module17:52
dansmithsean-k-mooney: is it only ever set by this test, or can it be set by other tests that use the DB and don't really pay attention to the fact that it's set?17:52
artomNothing actually uses the engine that it creates with connection=url passed in17:52
sean-k-mooneydansmith: its only set by things that run the db migrations17:52
dansmithsean-k-mooney: well, everything runs the db migrations17:52
sean-k-mooneydansmith: they dont use creat_all?17:53
dansmithsean-k-mooney: do you mean only things that *test* the db migrations in specific ways?17:53
sean-k-mooneythat woul dbe much quicker17:53
sean-k-mooneydansmith: i think its only set by thign that call migration.db_sync17:53
dansmithsean-k-mooney: but that's a ton of things, right?17:53
dansmithincidentally17:53
dansmiththings that want a DB setup have that done for them in the background right?17:54
sean-k-mooneyi dont know sound like i should add a function that will reset the gloabls and then either call that in test.py or in setup in the current test class17:54
dansmith++ for test.py for db things17:54
dansmithDB stuff is so global and sticky IME that it belongs in the test.py, IMHO17:54
sean-k-mooneydansmith: there are 2 ways to create the db. either start with blank schema or tell the model to create_all17:55
sean-k-mooneyok ill do that so17:55
sean-k-mooneystill not conviced it the best way long term but ill do that for now17:55
melwittthis is where we db_sync for every use of the Database fixture https://github.com/openstack/nova/blob/master/nova/tests/fixtures/nova.py#L64317:56
dansmithI feel like melwitt and I get extra argument points for the scars we have from all the cellsv1 and v2 test setup experience17:56
melwittand the CellDatabases fixture https://github.com/openstack/nova/blob/master/nova/tests/fixtures/nova.py#L39717:56
melwitt😂17:56
dansmithby "argument points" i mean.. votes :)17:57
dansmithI didn't fully read the backscroll, but melwitt you're pro-test.py right?17:57
sean-k-mooneydansmith: well its more i have previosue used a different pattern for reseting global state mainly for cached stuff in the past17:57
sean-k-mooneydansmith: that was nto the db cofnig but its the first time i have seen a prefernce to do this in test.py centrally17:58
melwittdansmith: yes17:58
sean-k-mooneyrahter then in the test module that cares17:58
sean-k-mooneyok ill add a reset_globals function to the nova.db.migration module and then call that form test.py17:59
sean-k-mooneyill push that in a few minutes17:59
*** tbachman is now known as Guest550019:14
*** tbachman is now known as Guest550420:26
*** tbachman is now known as Guest550720:50
*** tbachman is now known as Guest551021:18
*** tbachman is now known as Guest551623:03

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!