Thursday, 2022-06-16

opendevreviewSteve Baker proposed openstack/nova master: Align ironic driver secure boot with other drivers  https://review.opendev.org/c/openstack/nova/+/84424302:08
opendevreviewnorman shen proposed openstack/nova master: Clear connection info if vol disconnected  https://review.opendev.org/c/openstack/nova/+/84599502:34
opendevreviewTakashi Kajinami proposed openstack/os-vif master: Drop lower-constraints.txt and its testing  https://review.opendev.org/c/openstack/os-vif/+/84002004:47
*** elodilles is now known as elodilles_pto07:04
gibigood morning08:14
Ugglagibi, good morning.08:35
* bauzas waves late here09:08
gibisean-k-mooney: another question, maybe you remember. Do we set the PciDevice.instance_uuid to the migration_uuid on the source host  when the VM using the device is being migrated?10:37
sean-k-mooneyi dont think so10:37
sean-k-mooneywe dont need too10:37
gibiOK, then I look up running migrations for reshape10:37
gibiand pair them to PciDevs10:38
gibias I have to heal the allocation for the running migrations too10:38
sean-k-mooneythe instance uuid should be set to the instance uuid on both hosts10:38
gibiack10:38
gibithat what I guessed10:38
sean-k-mooneyi think the corralation is done via the migrate_data object which is only in memory10:39
sean-k-mooneyalthough you can just look at the relevent host form the migration object10:39
sean-k-mooneyto know which one it belongs too10:39
sean-k-mooneye.g migration vs instance10:39
gibiyeah something like that10:39
sean-k-mooneythat reminds me we still do not use consumer types10:40
sean-k-mooneythat is out of scope but ya10:40
sean-k-mooneyare we going to have a reshape in the future that will move all migrations to the migration consumer type10:41
gibiwe should have add that yes10:44
sean-k-mooneythats what i assuemd but i dont recall if the placment spec actully spoke about how nova or other porjects would use them once the feature was added10:45
gibime neither10:45
sean-k-mooneyso i was not sure if we woudl do a reshape or just stat using the consumer type for new migrations10:45
gibiI think reshape would be better10:46
sean-k-mooneyand let it heal over time as old migration get compelte/aborted.10:46
sean-k-mooneyit would be cleaner for sure10:46
sean-k-mooneybut proably a nova manage command so that it does not have any impact on rolling upgrade10:46
sean-k-mooneyalthough old nova will proably ignore the type so i might be over thinking it10:47
sean-k-mooneyfuture us problems10:47
gibiyepp10:50
gibi\one more architectural difference for today: for pci it cannot be decided if reshape is needed without checking existing nova allocations of pci also exists in placement too. As during rolling upgrade new pci allocations can arrive to a compute without placmeent allocation10:53
gibiso for pci we will do periodic healing10:53
gibiinstead of a one time reshape10:53
sean-k-mooneyfun10:56
sean-k-mooneyis that not the case for cpus?10:56
sean-k-mooneythe reshape will be tied to the pci/repot_to_placement conf or whatever it was right10:57
sean-k-mooneyso this will be skiped if that is not set10:58
gibiit seems for cpus we did not needed this11:10
sean-k-mooneyim not really sure why it woudl be differnt11:11
gibiyes, the whole tracking and reshape code starts with an if not CONF.pci.report_in_placement: return11:11
gibisean-k-mooney: for cpus we do two a_c queries to enforce allocation if placement has inventory11:12
sean-k-mooneyoh the fallback query11:12
gibiand fall back to legacy query11:12
gibiyeah11:12
sean-k-mooneyi mean we can do that for pci11:12
sean-k-mooneyif we want too11:12
sean-k-mooneyi guess that does nto really work the same if we use a prefilter11:13
sean-k-mooneywe had to do some extra chagnes for the fallback11:13
gibiperiodic healing feels less complex to me code wise11:14
sean-k-mooneyack 11:14
gibiI also would like make the healing code accessible from nova-manage11:15
gibihealing / audit11:15
gibibasically the same code can tell if something is missing and also can fix it11:16
gibior delete if there are overallocations in placement11:16
sean-k-mooneyyep that makes sense too11:23
sean-k-mooneypersonally i think we should merge audit and heal by the way or at least move  the delete allcoation logic form audit to heal11:24
sean-k-mooneyaudit is very very badly named11:25
gibiyepp11:25
gibiI agree11:25
gibiwe need only one CLI to check and fix11:25
sean-k-mooneyyep basicaly one command with a dry run mode11:25
sean-k-mooneythat just prints the problems that would be fixed11:26
sean-k-mooneybut does not actully do them, if you are happy drop --dry-run or add --apply whatever way we want to set the default11:26
jkulikmay I ask, how do you run nova-manage? Do you have the config to access stuff "locally" or do you connect to some VM/container containing nova-api or something or do you have a special "manage" host/container for that?11:28
sean-k-mooneyyou normlaly ssh to one of your contolers11:28
sean-k-mooneyand run it form there11:28
sean-k-mooneybut you can run it locally with configs yes if you need too11:28
sean-k-mooneyif you are using something like kolla you would do an exec into the contaienr to run it yes11:29
jkulikok. thanks. always felt a little wrong to me. why does it have to be a CLI and not an admin api?11:29
sean-k-mooneythe api process does not have access to everything that is needed to run it11:30
sean-k-mooneyjkulik: im sure you dont really want to be sendign you db/rabit password over a rest api request in general too11:31
opendevreviewMerged openstack/os-vif master: Drop lower-constraints.txt and its testing  https://review.opendev.org/c/openstack/os-vif/+/84002011:31
sean-k-mooneyjkulik: there would be ways to have apis do similar things but you need to use nova-manage to bootstrap things like the db11:32
jkuliksean-k-mooney: you mean for setting up cells? otherwise, both are in the db11:32
sean-k-mooneydoing offline db migrations11:32
sean-k-mooneyopenstack and nova in this case genreal requried all contoler process to agre on the db schema11:33
jkulikhm ... yeah, didn't think about those. fair point, that there are definitely commmands that can't go via API11:33
sean-k-mooneythe cell info is generaly in the db yes but if you are using templated db/rabbit urls in the db the passwords come form the nova.config11:34
jkulikoh. I didn't know about that possibility11:34
sean-k-mooneyhttps://docs.openstack.org/nova/latest/admin/cells.html#template-urls-in-cell-mappings11:35
sean-k-mooneyjkulik: they are optional11:35
sean-k-mooneybut they were created to make doing password rotation simplere with config tools like ansible11:35
sean-k-mooneysince you just need to update the configs and the passward on the db11:36
sean-k-mooneythat and to not store passwords in plain text in the db11:36
jkulikyeah, immediately wondering if we want that, too11:36
jkulikdownstream I mean11:36
sean-k-mooneyin which downstream11:37
jkulikI mean in our installations of Nova. sorry for the confusing wording11:37
sean-k-mooneytripleo uses it by defualt i think11:37
jkulikwe have a helm-chart to deploy Nova11:37
sean-k-mooneyah ok11:37
sean-k-mooneyi dont know if openstack ansible or kolla ever addopted it11:38
sean-k-mooneybut ya its up to each installer11:38
sean-k-mooneygibi: just because i happen to be looking at the code do you knwo why we dont allwo you to plug in image_types backends using stevador extention points like we do for filteres?11:47
sean-k-mooneyi just noticed it was a dict https://github.com/openstack/nova/blob/93a65f06df67ce39d65827692150c78013c7f6d5/nova/virt/libvirt/imagebackend.py#L1286-L1294=11:47
gibiI think nobody asked for it11:48
gibialso we removed a bunch of extension points in the past11:48
gibiso we might not want to add more11:48
gibibut technically we could add one11:49
sean-k-mooneyya i know proably more then we should have but that would be one way to allow people to play with a bfv backend and protype it.11:49
sean-k-mooneythe image ABC is intneded to define the api https://github.com/openstack/nova/blob/93a65f06df67ce39d65827692150c78013c7f6d5/nova/virt/libvirt/imagebackend.py#L81=11:50
sean-k-mooneygibi: or for example add the ablity to use libvirt sotorage pools.11:51
sean-k-mooneyalthough im relaly not sure how solid that interface actully is11:51
gibibut why this needs to be an extension why cannot be just proposed to nova as a backend?11:52
sean-k-mooneybecause if its an exstion we can easilly move backend we dont like out of tree11:53
sean-k-mooneybut also because it seams liek a nice extion point ot have11:53
sean-k-mooneyi was just wondering11:53
jrosser_sean-k-mooney: OSA uses templated db url but there is a bug we hit https://github.com/openstack/openstack-ansible-os_nova/blob/master/tasks/nova_db_setup.yml#L58-L8711:56
gibiextension point has a cost though, we cannot change the API freely11:56
gibias that might break out of tree extensions11:57
sean-k-mooneyyep11:59
sean-k-mooneygibi: by the way its a supper low priortiy right now but if i every get around to updating my black patch to the currnent 79 charater line lenght are you still interested in reviewing it12:47
sean-k-mooneyi was just cleaning up some terminals and closed the one i was working on that in 12:48
sean-k-mooneywell one of them 12:48
sean-k-mooneyi still have it open in emacs12:48
gibiyes I stil think we should merge it on master with 79 lines12:49
* sean-k-mooney since i ran out of memory the other day i tought it might be a good time to clean up my open windows12:49
sean-k-mooneyoh thats where i put the default chagne stuff12:51
sean-k-mooneyi started that in vscode since i was using that to debug the repoducer12:51
sean-k-mooneyi spent like 30 mins looking for it in differnt emacs and terminal windows12:51
gibi:D12:52
sean-k-mooneyit also does not help that i have 4 copies of nova and i started workign on it in the one i reseve for quickly pulling down patches and testing them locally12:53
sean-k-mooneyaka ~/repos/openstack/nova-3 12:53
gibiI started using git worktree so I can track where I my nova checkouts :D12:53
gibis/I/are/12:54
sean-k-mooneywhat is git worktree12:54
gibiyou can have multiple working direktory for a single git repo12:54
gibieach directory can have a different version checked out12:54
gibibut all tight to the same local git repo12:54
jkulikloving git worktree for working on multiple things in parallel :)12:54
sean-k-mooneyi normally do must of my dev on vms or my server so the local checkouts on my laptop are usually only for pep8/unit tests fixing12:55
sean-k-mooneygibi: oh ok i shoudl look into that12:55
sean-k-mooneysimilar concept but differnt approch to git-nit12:55
sean-k-mooneycommit-ish12:56
sean-k-mooneyhehe i like that12:56
sean-k-mooneyso i coudl tie my nova-1 nova-2 and nova-3 repos togenter into one nova repo and use those folders as work dirs instead of full repos12:57
sean-k-mooneyif im in a git work tree folder and i do git checkout or rebase ectra does it work like a normal repo?12:58
sean-k-mooneybut only affecting that workign tree12:58
gibiyes12:59
gibiand you can aways say git worktree list to see where are your workdirs :D12:59
sean-k-mooneyok so i coudl have 2 rebases going on in differnt workign trees12:59
gibiI think so yes12:59
sean-k-mooneycool12:59
sean-k-mooneyi dont do that often but sometimes if im pinged to respine X while working on y i will13:00
opendevreviewribaudr proposed openstack/nova master: [WIP] Attach Manila shares via virtiofs (db)  https://review.opendev.org/c/openstack/nova/+/83119313:33
opendevreviewribaudr proposed openstack/nova master: [WIP] Attach Manila shares via virtiofs (objects)  https://review.opendev.org/c/openstack/nova/+/83940113:33
opendevreviewribaudr proposed openstack/nova master: [WIP] Attach Manila shares via virtiofs (drivers)  https://review.opendev.org/c/openstack/nova/+/83309013:33
opendevreviewribaudr proposed openstack/nova master: [WIP] Attach Manila shares via virtiofs (api)  https://review.opendev.org/c/openstack/nova/+/83683013:33
opendevreviewsean mooney proposed openstack/nova master: update default numa allocation strategy  https://review.opendev.org/c/openstack/nova/+/84616913:40
opendevreviewArtom Lifshitz proposed openstack/nova stable/wallaby: fake: Ensure need_legacy_block_device_info returns False  https://review.opendev.org/c/openstack/nova/+/84367814:28
opendevreviewArtom Lifshitz proposed openstack/nova stable/wallaby: Add a regression test for bug 1939545  https://review.opendev.org/c/openstack/nova/+/84370214:28
opendevreviewArtom Lifshitz proposed openstack/nova stable/wallaby: compute: Ensure updates to bdms during pre_live_migration are saved  https://review.opendev.org/c/openstack/nova/+/84368014:28
opendevreviewArtom Lifshitz proposed openstack/nova stable/wallaby: fup: Make connection_info returned by CinderFixture unique per attachment  https://review.opendev.org/c/openstack/nova/+/84459414:28
opendevreviewArtom Lifshitz proposed openstack/nova stable/wallaby: fup: Assert state of connection_info during LM rollback in func tests  https://review.opendev.org/c/openstack/nova/+/84459514:28
opendevreviewribaudr proposed openstack/nova master: [WIP] Attach Manila shares via virtiofs (objects)  https://review.opendev.org/c/openstack/nova/+/83940114:53
opendevreviewribaudr proposed openstack/nova master: [WIP] Attach Manila shares via virtiofs (manila abstraction)  https://review.opendev.org/c/openstack/nova/+/83119414:53
opendevreviewribaudr proposed openstack/nova master: [WIP] Attach Manila shares via virtiofs (drivers)  https://review.opendev.org/c/openstack/nova/+/83309014:53
opendevreviewribaudr proposed openstack/nova master: [WIP] Attach Manila shares via virtiofs (api)  https://review.opendev.org/c/openstack/nova/+/83683014:53
opendevreviewBalazs Gibizer proposed openstack/nova master: Report PCI inventory to placement basics  https://review.opendev.org/c/openstack/nova/+/84618715:27
opendevreviewBalazs Gibizer proposed openstack/nova master: Report PCI inventory to placement basics  https://review.opendev.org/c/openstack/nova/+/84618715:30
dansmithdoes anyone know of a recentish change to nova that might have inflated n-cpu memory footprint?15:51
sean-k-mooneynot that im aware of that has merged16:00
sean-k-mooneydansmith: gibi has a change that adds a trivial cache for speeding up numa toplogy computation16:01
sean-k-mooneybut i dont think that has merged16:01
sean-k-mooneydansmith: are you trying to track down a memory increase 16:02
dansmithmaybe yeah16:02
sean-k-mooneyhttps://review.opendev.org/c/openstack/nova/+/845896 but its not merged so its not that16:03
dansmithokay, so it's not a regression, or at least, not a recent one16:04
dansmithn-cpu memory usage after a single-node run is ~500m, but after a multinode one, it's 1.5G to 2.5G16:05
dansmithso I wonder if we're buffering something in memory due to live migration or something16:05
sean-k-mooneydo you have swap awailable16:05
dansmithwhat does it matter? it'16:05
dansmithit's not OOMing, we're just noticing it's much larger16:06
sean-k-mooneyjust tha ti saw high memory usage in the past when swap was disabled16:07
sean-k-mooneythat did not happen when it was available16:07
sean-k-mooneybut i guess you have the saem config in singel vs multinode16:07
dansmithyeah, these are CI workers16:07
sean-k-mooneyso is  it that  somehow adding a second compute node increase memory usage on the first16:07
sean-k-mooneylike if you enable a multi node install but idsable nova-cpu on the subnode16:08
sean-k-mooneydoes it still happen16:08
dansmithI was thinking more like in live migration we read in the configdrive to memory or something16:08
dansmithyou know how python increases the water mark but basically never decreases it16:08
sean-k-mooneyhum ok maybe16:09
sean-k-mooneyyou could confirm that by disabling move operations tests 16:09
sean-k-mooneythe config drive is pretty small but maybe16:09
dansmithjust an example16:09
sean-k-mooneyya im jsut trying to think what else is like that16:10
sean-k-mooneyis the memory usage the same on both hosts?16:10
dansmithI don't collect the other so dunno16:10
sean-k-mooneywould it be hard to add that16:13
sean-k-mooneydo you have the link to the ci job by the way16:14
dansmithit's all the jobs, there's no need to look at one specifically16:16
dansmithI thought it was recent, but I compared to some old snaps I had and they were high then too, but only for multinode16:16
gibisean-k-mooney, bauzas, artom: on master we still reject the evacuate request if the instance is in transient state (i.e. powering-off) https://paste.opendev.org/show/bQphEfOf8eLBnM6XmleQ/16:36
gibiso for me this seem like a valid request to evacuate the instance in this state16:37
sean-k-mooneyits not safe nessisarly to do by default16:38
gibieven if the compute is forced down?16:39
sean-k-mooneyif its force down its oke16:39
sean-k-mooneybut only because16:39
sean-k-mooneythe operator is16:39
sean-k-mooneypromising they have fenced it16:39
gibiyepp16:39
sean-k-mooneybut normlaly evacuate without force down would not be safe16:39
gibibtw, reset-state --active helps https://paste.opendev.org/show/bVI7D8H5g9Oqjjo4rKfk/16:39
sean-k-mooneywe cant tell that the vm is not running16:39
gibiyepp in generate evacuate without fencing is unsafe in non transient states too16:40
gibis/generate/general/16:40
sean-k-mooneyso ya if we check both then we can ignore the task state16:40
sean-k-mooney*both-> force-down16:40
sean-k-mooneyso question16:40
sean-k-mooneydoes the decorator support ignoring task state16:40
gibiso on master I'm OK to ignore the task state 16:42
gibidownstream I would suggest the customer to use the reset-state, so no code changes is needed there16:42
sean-k-mooneyreset_state only resets vm state not task state right16:43
sean-k-mooneyor dose it force task_state=None16:43
artomIIRC it can do both16:43
sean-k-mooneywell you cant reset taskstate with it16:43
sean-k-mooneyconfigurably16:43
sean-k-mooneyi.e. reset-state vm-sate=active task-state=none16:44
sean-k-mooneybut i dont know if task-state=none16:44
sean-k-mooneyis hardcoded internally16:44
gibisean-k-mooney: reset-state resets the task state too16:45
sean-k-mooneythen ya it should be good16:45
sean-k-mooneyit would be nice if we could add task-state to the api in the futrue16:46
sean-k-mooneyhttps://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/api/openstack/compute/schemas/reset_server_state.py#L1516:46
sean-k-mooneybut for this case we shoudl be good16:46
sean-k-mooneyhttps://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/api/openstack/compute/admin_actions.py#L77-L78=16:47
gibiall good16:48
gibi:)\16:48
sean-k-mooneygibi: are you going to file an upstream bug for the transient state16:48
gibisure I can I have the repro ready. But I won't assign it to myself to fix it :)16:49
gibiat least not right now16:49
gibisean-k-mooney, artom, bauzas https://bugs.launchpad.net/nova/+bug/197898316:58
gibisean-k-mooney: btw, I added the pci inventory reporting to the resource tracker level https://review.opendev.org/c/openstack/nova/+/846187 it does not need to be in the virt layer so I did not force to it17:00
artomNice17:00
opendevreviewArtom Lifshitz proposed openstack/nova stable/ussuri: fake: Ensure need_legacy_block_device_info returns False  https://review.opendev.org/c/openstack/nova/+/84395017:00
opendevreviewArtom Lifshitz proposed openstack/nova stable/ussuri: Add a regression test for bug 1939545  https://review.opendev.org/c/openstack/nova/+/84395117:00
opendevreviewArtom Lifshitz proposed openstack/nova stable/ussuri: compute: Ensure updates to bdms during pre_live_migration are saved  https://review.opendev.org/c/openstack/nova/+/84395217:00
opendevreviewArtom Lifshitz proposed openstack/nova stable/ussuri: fup: Make connection_info returned by CinderFixture unique per attachment  https://review.opendev.org/c/openstack/nova/+/84460617:00
opendevreviewArtom Lifshitz proposed openstack/nova stable/ussuri: fup: Assert state of connection_info during LM rollback in func tests  https://review.opendev.org/c/openstack/nova/+/84460717:00
opendevreviewArtom Lifshitz proposed openstack/nova stable/ussuri: functional: Use tempdir for CONF.instances_path  https://review.opendev.org/c/openstack/nova/+/84620417:00
sean-k-mooneygibi: cool just triaged it as medium since you have repoduced it and there is a workaround17:03
gibisean-k-mooney: thanks17:03
sean-k-mooneyby the way by doing it at the compute manager level that shoudl work for all virt drivers which is nice17:03
gibiyepp that also a +17:04
sean-k-mooneywe will likely only test with libvirt but if someone was to test with powervm and it worked they could update the docs17:04
gibiyes17:04
opendevreviewGorka Eguileor proposed openstack/nova master: DNM: Testing memory usage of n-cpu (1/3)  https://review.opendev.org/c/openstack/nova/+/84620717:11
opendevreviewGorka Eguileor proposed openstack/nova master: DNM: Testing memory usage of n-cpu (2/3)  https://review.opendev.org/c/openstack/nova/+/84620817:11
opendevreviewGorka Eguileor proposed openstack/nova master: DNM: Testing memory usage of n-cpu (3/3)  https://review.opendev.org/c/openstack/nova/+/84620917:11
geguileodansmith: sean-k-mooney ^ Those are the test patches to see if we can figure something out...17:12
geguileofist is doing nothing, next is trimming malloc memory, last is trimming and setting malloc arenas17:12
* geguileo facepalms hard, as I seem to not know how to change .zuul17:13
opendevreviewGorka Eguileor proposed openstack/nova master: DNM: Testing memory usage of n-cpu (1/3)  https://review.opendev.org/c/openstack/nova/+/84620717:14
opendevreviewGorka Eguileor proposed openstack/nova master: DNM: Testing memory usage of n-cpu (2/3)  https://review.opendev.org/c/openstack/nova/+/84620817:14
opendevreviewGorka Eguileor proposed openstack/nova master: DNM: Testing memory usage of n-cpu (3/3)  https://review.opendev.org/c/openstack/nova/+/84620917:14
dansmithgeguileo: I believe no human has ever gotten a zuul config change correct on the first try17:16
sean-k-mooneyso you are directly calling into mallac to free memory17:20
sean-k-mooneythats interesting17:20
sean-k-mooneysory forgot the ...17:20
opendevreviewArtom Lifshitz proposed openstack/nova stable/ussuri: functional: Use tempdir for CONF.instances_path  https://review.opendev.org/c/openstack/nova/+/84620417:56
opendevreviewArtom Lifshitz proposed openstack/nova stable/ussuri: Add a regression test for bug 1939545  https://review.opendev.org/c/openstack/nova/+/84395117:56
opendevreviewArtom Lifshitz proposed openstack/nova stable/ussuri: compute: Ensure updates to bdms during pre_live_migration are saved  https://review.opendev.org/c/openstack/nova/+/84395217:56
opendevreviewArtom Lifshitz proposed openstack/nova stable/ussuri: fup: Make connection_info returned by CinderFixture unique per attachment  https://review.opendev.org/c/openstack/nova/+/84460617:56
opendevreviewArtom Lifshitz proposed openstack/nova stable/ussuri: fup: Assert state of connection_info during LM rollback in func tests  https://review.opendev.org/c/openstack/nova/+/84460717:56
opendevreviewBalazs Gibizer proposed openstack/nova master: Report PCI inventory to placement basics  https://review.opendev.org/c/openstack/nova/+/84618718:29
opendevreviewBalazs Gibizer proposed openstack/nova master: Extend device_spec with resource_class and traits  https://review.opendev.org/c/openstack/nova/+/84621818:29
opendevreviewBalazs Gibizer proposed openstack/nova master: Ignore PCI devs with physical_network tag  https://review.opendev.org/c/openstack/nova/+/84621918:29

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!