Tuesday, 2019-08-27

« Monday, 2019-08-26 Index Wednesday, 2019-08-28 »
*** jmlowe has quit IRC00:00
*** markvoelker has joined #openstack-nova00:10
*** markvoelker has quit IRC00:15
sean-k-mooneyit was an automatic rebase against master00:16
sean-k-mooneyno conflicts00:16
sean-k-mooneyso just commit and rebase against master and push00:16
sean-k-mooneyartom: ^00:16
artomsean-k-mooney, ack, thanks00:16
sean-k-mooneyalso something is broken with hw:mem_page_size=small. i get an error about pagesize changeing when i try to migrate when i set that.00:18
*** gyee has quit IRC00:18
sean-k-mooneynot sure if that is related to your changes or not00:18
*** threestrands has joined #openstack-nova00:24
*** hongbin has joined #openstack-nova00:37
*** tbachman has joined #openstack-nova00:38
*** brault has joined #openstack-nova00:52
artomsean-k-mooney, do the 2 hosts have different page sizes?00:52
artom(Or it could be a legit bug with that code, as it's new)00:52
*** ociuhandu has joined #openstack-nova00:53
*** brault has quit IRC00:57
*** ociuhandu has quit IRC00:57
artomOh snap, rollback works00:59
artom*actually* works00:59
artomIf I don't drop the move claim on the dest (comment that bit out), the test fails with the expected pinning assertion01:00
openstackgerritArtom Lifshitz proposed openstack/nova master: Introduce live_migration_claim()  https://review.opendev.org/63566901:01
openstackgerritArtom Lifshitz proposed openstack/nova master: New objects for NUMA live migration  https://review.opendev.org/63482701:01
openstackgerritArtom Lifshitz proposed openstack/nova master: LM: add support for augmenting migrate_data with info from claims  https://review.opendev.org/63482801:01
openstackgerritArtom Lifshitz proposed openstack/nova master: LM: add support for updating NUMA-related XML on the source  https://review.opendev.org/63522901:01
openstackgerritArtom Lifshitz proposed openstack/nova master: NUMA live migration support  https://review.opendev.org/63460601:01
openstackgerritArtom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration  https://review.opendev.org/64002101:01
openstackgerritArtom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration  https://review.opendev.org/67259501:01
* artom -> gym01:01
*** slaweq has joined #openstack-nova01:11
*** rcernin has quit IRC01:13
*** slaweq has quit IRC01:16
*** factor has joined #openstack-nova01:33
openstackgerritMerged openstack/nova master: [Trivial]Remove unused helper get_vif_devname_with_prefix  https://review.opendev.org/67813601:35
*** zhubx has quit IRC01:57
*** zhubx has joined #openstack-nova01:57
*** zhubx has quit IRC02:08
*** slaweq has joined #openstack-nova02:11
*** rcernin has joined #openstack-nova02:13
*** slaweq has quit IRC02:16
*** larainema has joined #openstack-nova02:23
openstackgerritMerged openstack/nova stable/stein: rt: only map compute node if we created it  https://review.opendev.org/67627802:30
openstackgerritMerged openstack/nova stable/stein: Add functional regression recreate test for bug 1839560  https://review.opendev.org/67650702:30
openstackbug 1839560 in OpenStack Compute (nova) stein "ironic: moving node to maintenance makes it unusable afterwards" [High,In progress] https://launchpad.net/bugs/1839560 - Assigned to Matt Riedemann (mriedem)02:30
openstackgerritMerged openstack/nova stable/stein: Restore soft-deleted compute node with same uuid  https://review.opendev.org/67650902:30
*** dannins has joined #openstack-nova02:38
*** gbarros has quit IRC03:04
*** markvoelker has joined #openstack-nova03:10
*** markvoelker has quit IRC03:15
*** nicolasbock has quit IRC03:25
*** psachin has joined #openstack-nova03:33
openstackgerritArtom Lifshitz proposed openstack/nova master: NUMA live migration support  https://review.opendev.org/63460603:47
openstackgerritArtom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration  https://review.opendev.org/64002103:47
openstackgerritArtom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration  https://review.opendev.org/67259503:47
*** udesale has joined #openstack-nova04:06
*** markvoelker has joined #openstack-nova04:10
*** slaweq has joined #openstack-nova04:11
*** hongbin has quit IRC04:13
*** markvoelker has quit IRC04:15
*** slaweq has quit IRC04:16
*** ratailor has joined #openstack-nova04:20
*** mkrai has joined #openstack-nova04:31
*** shilpasd has joined #openstack-nova04:33
openstackgerritMerged openstack/nova master: [Trivial]Remove unused helper _get_min_service_version  https://review.opendev.org/67844604:35
*** dave-mccowan has quit IRC04:36
*** ociuhandu has joined #openstack-nova04:53
*** ociuhandu has quit IRC04:57
*** macz has joined #openstack-nova04:58
*** shilpasd has quit IRC05:02
*** macz has quit IRC05:03
*** sridharg has joined #openstack-nova05:10
*** sridharg has quit IRC05:10
*** markvoelker has joined #openstack-nova05:10
openstackgerritDustin Cowles proposed openstack/nova master: WIP: Load the custom resource providers to resource tracker  https://review.opendev.org/67652205:10
*** sridharg has joined #openstack-nova05:11
*** slaweq has joined #openstack-nova05:11
*** dustinc has joined #openstack-nova05:11
*** Sundar has joined #openstack-nova05:12
*** markvoelker has quit IRC05:15
*** slaweq has quit IRC05:16
*** Sundar has quit IRC05:18
*** boxiang has joined #openstack-nova05:23
*** factor has quit IRC05:41
*** Luzi has joined #openstack-nova05:43
*** damien_r has quit IRC05:47
*** ccamacho has quit IRC05:52
*** hamzy has quit IRC05:59
*** lpetrut has joined #openstack-nova06:02
*** macz has joined #openstack-nova06:11
*** zhubx has joined #openstack-nova06:11
*** slaweq has joined #openstack-nova06:11
*** boxiang has quit IRC06:11
*** macz has quit IRC06:15
*** slaweq has quit IRC06:15
*** prometheanfire has quit IRC06:18
openstackgerritmelanie witt proposed openstack/nova master: nova-manage db archive_deleted_rows is not multi-cell aware  https://review.opendev.org/50748606:22
openstackgerritmelanie witt proposed openstack/nova master: Verify archive_deleted_rows --all-cells in post test hook  https://review.opendev.org/67284006:22
*** prometheanfire has joined #openstack-nova06:22
*** zhubx has quit IRC06:24
*** zhubx has joined #openstack-nova06:24
*** zhubx has quit IRC06:26
*** lee1 has joined #openstack-nova06:32
*** brault has joined #openstack-nova06:38
*** macz has joined #openstack-nova06:39
*** epoojad has joined #openstack-nova06:40
*** macz has quit IRC06:44
*** trident has quit IRC07:00
*** mkrai_ has joined #openstack-nova07:02
*** mkrai__ has joined #openstack-nova07:05
*** mkrai has quit IRC07:05
*** markvoelker has joined #openstack-nova07:06
*** mkrai_ has quit IRC07:08
*** trident has joined #openstack-nova07:10
*** slaweq has joined #openstack-nova07:11
*** markvoelker has quit IRC07:15
*** slaweq has quit IRC07:15
*** ivve has joined #openstack-nova07:17
*** macz has joined #openstack-nova07:17
*** macz has quit IRC07:22
*** sapd1_x has joined #openstack-nova07:25
*** xek has joined #openstack-nova07:30
*** threestrands has quit IRC07:32
*** markvoelker has joined #openstack-nova07:35
*** dtantsur|afk is now known as dtantsur07:37
*** damien_r has joined #openstack-nova07:39
*** rcernin has quit IRC07:40
*** markvoelker has quit IRC07:40
*** lee1 is now known as lyarwood07:46
*** slaweq has joined #openstack-nova07:52
*** ralonsoh has joined #openstack-nova07:52
*** panda has quit IRC08:02
*** panda has joined #openstack-nova08:02
*** tkajinam has quit IRC08:07
*** epoojad has quit IRC08:12
*** mkrai__ has quit IRC08:15
*** dougsz has joined #openstack-nova08:17
*** tetsuro has joined #openstack-nova08:21
*** bhagyashris has joined #openstack-nova08:26
aspierskashyap: welcome back08:29
* kashyap waves hi; thanks08:29
aspierskashyap: hope you had a good time!08:36
*** jangutter has quit IRC08:37
kashyapPartly, yes.  And partly it was a "responsibility trip" :-)08:37
*** markvoelker has joined #openstack-nova08:40
openstackgerritzhangyujun proposed openstack/nova master: Should not raise when restore power on failed  https://review.opendev.org/62485408:41
*** mkrai__ has joined #openstack-nova08:44
*** markvoelker has quit IRC08:45
*** avolkov has joined #openstack-nova08:46
*** ccamacho has joined #openstack-nova08:47
*** shilpasd has joined #openstack-nova08:47
*** priteau has joined #openstack-nova08:50
*** derekh has joined #openstack-nova08:50
*** slaweq has quit IRC08:53
*** macz has joined #openstack-nova08:53
*** mkrai__ has quit IRC08:54
*** macz has quit IRC08:58
*** jangutter has joined #openstack-nova09:01
*** cdent has joined #openstack-nova09:02
*** cdent has quit IRC09:04
*** licanwei has joined #openstack-nova09:05
licanwei#join #airshipit09:05
*** cdent has joined #openstack-nova09:06
*** slaweq has joined #openstack-nova09:10
*** mkrai__ has joined #openstack-nova09:10
*** CeeMac has joined #openstack-nova09:12
*** zbr has joined #openstack-nova09:13
*** slaweq has quit IRC09:14
*** slaweq has joined #openstack-nova09:20
*** slaweq has quit IRC09:25
*** dtantsur is now known as dtantsur|bbl09:31
*** brinzhang_ has quit IRC09:32
*** brinzhang_ has joined #openstack-nova09:32
aspierssean-k-mooney: let me know if I can help with bp/image-metadata-prefiltering09:34
aspierssean-k-mooney: that's not me being generous, you're just ahead of me on the runway ;-)09:34
aspierslooks like there are some CI failures, maybe I could fix those09:35
*** psachin has quit IRC09:37
*** psachin has joined #openstack-nova09:41
*** jaosorior has quit IRC09:43
openstackgerritLuyao Zhong proposed openstack/nova master: db: Add resources column in instance_extra table  https://review.opendev.org/67844709:44
openstackgerritLuyao Zhong proposed openstack/nova master: object: Introduce Resource and ResouceList objs  https://review.opendev.org/67844809:44
openstackgerritLuyao Zhong proposed openstack/nova master: Add resources dict into _Provider  https://review.opendev.org/67844909:44
openstackgerritLuyao Zhong proposed openstack/nova master: Retrive the allocations early  https://review.opendev.org/67845009:44
openstackgerritLuyao Zhong proposed openstack/nova master: Track orphan instances and error migrations in resource tracker  https://review.opendev.org/67845109:44
openstackgerritLuyao Zhong proposed openstack/nova master: Claim resources in resource tracker  https://review.opendev.org/67845209:44
openstackgerritLuyao Zhong proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces  https://review.opendev.org/67845309:44
openstackgerritLuyao Zhong proposed openstack/nova master: libvirt: report VPMEM resources by provider tree  https://review.opendev.org/67845409:44
openstackgerritLuyao Zhong proposed openstack/nova master: libvirt: Support VM creation with vpmems and vpmems cleanup  https://review.opendev.org/67845509:44
openstackgerritLuyao Zhong proposed openstack/nova master: Parse vpmem related flavor extra spec  https://review.opendev.org/67845609:44
openstackgerritLuyao Zhong proposed openstack/nova master: Add functional tests for virtual persistent memory  https://review.opendev.org/67847009:44
*** psachin has quit IRC09:48
*** brinzhang_ has quit IRC09:49
*** brinzhang_ has joined #openstack-nova09:50
*** sapd1_x has quit IRC09:59
openstackgerritmathieu bultel proposed openstack/nova master: Return security groups by name  https://review.opendev.org/67877610:02
*** bbobrov has quit IRC10:04
*** dpawlik has quit IRC10:04
*** bbobrov has joined #openstack-nova10:04
*** cgoncalves has quit IRC10:04
*** cgoncalves has joined #openstack-nova10:04
*** bhagyashris has quit IRC10:11
aspiersbauzas: I was just looking at fixing some of the issues in sean-k-mooney's patch you just reviewed https://review.opendev.org/#/c/666914/10:18
bauzascool10:18
aspiersalthough I'm not sure if I'd be treading on his toes10:18
bauzaswell, sean is there :)10:19
aspiershttps://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_14/666914/11/check/openstack-tox-py27/ee133dd/testr_results.html.gz shows an ordering error which is probably the dict.values issue you mention10:19
aspiersonly on py27 not py310:19
aspiersI'm not sure if he's around today10:19
*** macz has joined #openstack-nova10:19
sean-k-mooneybauzas: yep i know values change between py27 and py3X10:20
sean-k-mooneyi normally would use six.iteritems10:20
sean-k-mooneybut aspiers and stephenfin  prefer that i dont10:21
openstackgerritGorka Eguileor proposed openstack/nova master: Use os-brick locking for volume attach and detach  https://review.opendev.org/61419010:21
aspierssean-k-mooney: I don't personally care but it seems that everyone else does10:21
bauzaswhy so ?10:21
sean-k-mooneyaspiers: i dont like use writing slow code by default10:21
bauzaswait, six.itervalues() isn't exactly slow10:21
sean-k-mooneyno thats what i prefer to use10:22
sean-k-mooneyfor values10:22
sean-k-mooneyor iteritems for items10:22
stephenfinI only care insofar as it generally doesn't make any difference and I'd rather one less 'six' thing to clean up when we move to Python 3 only10:22
sean-k-mooneystephenfin: in aggreate it does10:22
bauzashttps://github.com/benjaminp/six/blob/master/six.py#L58410:22
stephenfinon python 2, yes10:23
bauzasthat's what does six.itervalues10:23
aspiersI'm with stephenfin on this. The majority of cases we are talking about involve like 3 elements10:23
bauzasstephenfin: of course10:23
stephenfinbut OSP 16 will be deployed with Python 3 (RHEL 8)10:23
aspiersso the cost of cleaning up six stuff is higher than the performance hit10:23
stephenfinI suspect SUSE and Canonical will be doing something similar10:23
stephenfinYeah, that ^10:23
sean-k-mooneyaspiers: yes but if we ever packport the code we pay that cost for years10:23
bauzastechnically we support py2 for a while10:23
sean-k-mooneyhell if we dont we still do10:23
bauzasand the "we" is about the upstream code10:24
aspiersif there are exceptions with like 10000 element dicts then fine we can make an exception for those10:24
sean-k-mooneywe have people only finally moving to queens10:24
stephenfinbauzas: Correct. But no distro is packaging it that I'm aware of10:24
sean-k-mooneywe will have people on py27 downstream until 202310:24
stephenfinsean-k-mooney: There's no performance impact either, right? It's a memory impact10:24
stephenfinNot for new code10:24
bauzasstephenfin: sure but then that's not a good reason for not supporting py27 :)10:24
*** macz has quit IRC10:24
sean-k-mooneystephenfin: no its slower to use .items or .values on py27 too10:24
bauzasanyway10:24
aspiersthis is a classic bikeshed discussion X-D10:25
stephenfinno, but it is a good reason to make an already mostly unnecessary thing even more unnecessary10:25
bauzasaspiers: this10:25
stephenfinI want blue.10:25
sean-k-mooneywe are making a copy of all the values and construction an new list10:25
bauzasstephenfin: i want snow10:25
aspiersstephenfin: you can have any blue, as long as it's black10:25
sean-k-mooneybauzas: but anyway do like the new version in general10:25
stephenfinbauzas: You can't handle the snow.10:25
sean-k-mooneybauzas: its closer to what i had originally planned for the funciton10:25
bauzastesttools.matchers._impl.MismatchError: [1, 2, 3, 4] != [1, 2, 4, 3]10:25
bauzasthis ^10:25
aspiersright10:25
sean-k-mooneyya i saw that10:25
sean-k-mooneyi was going to fix it10:25
cdentThese conversations make me laugh. We're talking about trying to maintain py2 code when real people won't see it for many months, they are all still long time behind now.10:25
sean-k-mooneyim relying on dict order10:26
sean-k-mooneywich is only a py36+ thing10:26
sean-k-mooneyill fix that10:26
aspierssean-k-mooney: you want me to submit my typo/pep8 fixes?10:26
stephenfinbauzas: Oh, here's another one you'll probably want to look at https://review.opendev.org/#/c/674894/10:26
sean-k-mooneyaspiers: you can if you like.10:26
cdentWe should be on at least py3.7 everywhere, because by the time the majority of people are using the code...10:26
stephenfinI think you know that code from the vGPU work10:26
stephenfincdent: only ~ two months to go10:27
stephenfintick-tock tick-tock10:27
cdent:fist bump:10:27
cdentor :something:10:27
bauzasstephenfin: looking10:27
stephenfinand when we do, I'm doing this _everywhere_ https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/integrate-mypy-type-checking10:27
stephenfinstarting with oslo10:28
sean-k-mooneybauzas: i can move the _project_traits function out of the class too.10:28
bauzassean-k-mooney: honestly, that really depends on what you want10:28
bauzassean-k-mooney: I'm not particularly opinionated10:28
sean-k-mooneyi want it to be a staticmethod10:28
bauzassean-k-mooney: but I feel we should just make it simplier10:28
bauzassean-k-mooney: what's fun is that you explain that this method is absolutely unrelated to the driver, but you call it by its libvirt class name :)10:29
sean-k-mooneyif we can make it simpler without reducing functionality the sure. have you looked at the folow up patches10:29
sean-k-mooneybauzas: no i want to call it with self10:29
*** mkrai__ has quit IRC10:30
bauzassean-k-mooney: no I haven't looked at othre patches yet10:30
openstackgerritAdam Spiers proposed openstack/nova master: Libvirt: report storage bus traits  https://review.opendev.org/66691410:30
aspierssean-k-mooney: ^^^10:30
sean-k-mooney@staticmothod is intened to allow you to associate free functions with a class so that they are callable like methods10:30
aspierssean-k-mooney: also, might be just me, but I would personally prefer test_flatten_iterable to be split into multiple smaller test cases10:31
sean-k-mooneyam i could to that10:31
aspierssean-k-mooney: currently when an assertion fails, you can't immediately see which one failed10:31
sean-k-mooneyi built it iteritivly so it was simpler to built up with more test cases10:31
aspiersalso giving each a separate method name would add some clarity to exactly what is being tested10:31
aspierssure10:32
sean-k-mooneyi kind of grouped them intentionally expceting that comment :)10:32
sean-k-mooneyso ill split it in the next version10:32
bauzassean-k-mooney: have you clicked on my link ? :)10:32
sean-k-mooneyyes10:32
sean-k-mooneythat is not my usecase and it is not a good argument10:33
sean-k-mooneyi want to be able to call it via self not via the class name10:33
sean-k-mooneyi can call it via the class name if im in a non instance/class method10:34
sean-k-mooneybut i an instnace method i wanted to call it via self10:34
*** tetsuro has quit IRC10:34
aspierssean-k-mooney: cool thanks10:35
bauzassean-k-mooney: just make it a classmethod the,n10:36
bauzasanyway, I need to eat10:36
sean-k-mooneyit does not need to use any class method or variables.10:36
sean-k-mooneyanyway i dont really want to argue about this. static method would be the correct decorator to use but maybe instead of sayign what i actully mean i should write the dumbest version that works...10:40
*** macz has joined #openstack-nova10:40
openstackgerritChris Dent proposed openstack/nova master: Add a "Caveats" section to the eventlet profiling docs  https://review.opendev.org/67667210:41
*** macz has quit IRC10:45
sean-k-mooneybauzas: by the way do you think flatten_iterable shoudl yield the values as it does now or the item tuples. i started with the tuples but it was making the code more complex. i feel like a sperate flatten_mapping would be better if we need it in the future10:49
openstackgerritBalazs Gibizer proposed openstack/nova master: allow getting resource request of every bound ports of an instance  https://review.opendev.org/65511010:49
openstackgerritBalazs Gibizer proposed openstack/nova master: Pass network API to the conducor's MigrationTask  https://review.opendev.org/65511110:51
openstackgerritBalazs Gibizer proposed openstack/nova master: Add request_spec to server move RPC calls  https://review.opendev.org/65572110:54
*** slaweq has joined #openstack-nova10:55
openstackgerritBalazs Gibizer proposed openstack/nova master: re-calculate provider mapping during migration  https://review.opendev.org/65511210:56
openstackgerritBalazs Gibizer proposed openstack/nova master: update allocation in binding profile during migrate  https://review.opendev.org/65642210:58
openstackgerritBalazs Gibizer proposed openstack/nova master: Extend NeutronFixture to handle migrations  https://review.opendev.org/65511411:01
*** macz has joined #openstack-nova11:01
*** nicolasbock has joined #openstack-nova11:02
*** udesale has quit IRC11:02
openstackgerritBalazs Gibizer proposed openstack/nova master: prepare func test env for moving servers with bandwidth  https://review.opendev.org/65510911:03
openstackgerritBalazs Gibizer proposed openstack/nova master: Func test for migrate server with ports having resource request  https://review.opendev.org/65511311:05
*** macz has quit IRC11:06
openstackgerritBalazs Gibizer proposed openstack/nova master: Make _rever_allocation nested allocation aware  https://review.opendev.org/67613811:08
*** brinzhang_ has quit IRC11:09
*** brinzhang_ has joined #openstack-nova11:09
openstackgerritBalazs Gibizer proposed openstack/nova master: Support reverting migration / resize with bandwidth  https://review.opendev.org/67614011:10
openstackgerritMerged openstack/nova master: [Trivial]Remove unused helper get_vm_ref_from_name  https://review.opendev.org/67844411:10
*** tesseract has joined #openstack-nova11:11
openstackgerritBalazs Gibizer proposed openstack/nova master: Func test for migrate re-schedule with bandwidth  https://review.opendev.org/67697211:12
openstackgerritBalazs Gibizer proposed openstack/nova master: Support migrating SRIOV port with bandwidth  https://review.opendev.org/67698011:15
openstackgerritBalazs Gibizer proposed openstack/nova master: Allow migrating server with port resource request  https://review.opendev.org/67149711:17
sean-k-mooneygibi: does ^ work with sriov livemigrtaion or just cold migration11:17
*** hamzy has joined #openstack-nova11:18
*** dpawlik has joined #openstack-nova11:18
gibisean-k-mooney: just cold so far11:18
gibisean-k-mooney: I still have to work on live migrate, evacuate, and shelve offload11:18
sean-k-mooneyok do you have live migration support ingeneral11:18
sean-k-mooneyah ok11:18
sean-k-mooneyi dont think you will have to do anything spcial for sriov in that regard11:19
gibisean-k-mooney: I expect resize to work out of the box but I have to add functinal coverage for it11:19
sean-k-mooneyif you get generic live migration working then it should work for sriov too11:19
gibisean-k-mooney: the selection of pci devices needs to be driven the same way for live migration as it is now done for cold migration11:20
*** jaosorior has joined #openstack-nova11:20
gibisean-k-mooney: but besides that I don't expect any sriov specific thing11:20
sean-k-mooneywe handel that differently for sriov11:20
sean-k-mooneywe do not use move claims11:20
sean-k-mooneyso i hope you are not assuming we do11:21
sean-k-mooneyill take a look at the patch in either case11:21
gibisean-k-mooney: how does live migration selects the pci device on the target host?11:22
*** macz has joined #openstack-nova11:22
gibisean-k-mooney: if it uses the InstancePCIRequest then I think my strategy will work11:23
sean-k-mooneywe claim the in the pci resouce tracker in check_can_live_migrate_at_dest11:23
sean-k-mooneywe create new instancePCIRequest object i think11:23
gibisean-k-mooney: OK, then I need to patch those requests with https://review.opendev.org/#/c/676980/6/nova/compute/manager.py@214111:24
sean-k-mooneythen we store the pci addresse of the new vif objects in the migrate_data11:25
sean-k-mooneyoh the provider_mappings11:25
sean-k-mooneyya proably11:25
sean-k-mooneydid you see my comment by the way regarding the lenght of that parmater/variable11:25
gibisean-k-mooney: I'm going through the series today so I will11:26
sean-k-mooneybasically  request_group_resource_providers_mapping -> provider_mappings11:27
*** macz has quit IRC11:27
aspierssean-k-mooney: shall I rebase the other 2 patches on top so that they have a chance of passing pep8 at least?11:27
sean-k-mooneyif we need to explain it is a request froup resouce provder mapping we can state that in the doc string11:28
aspiershmm, I guess they will still fail on test_utils11:28
sean-k-mooneyaspiers: no need ill do that later11:28
gibisean-k-mooney: ack11:28
sean-k-mooneywas getting through my email11:28
*** slaweq has quit IRC11:31
sean-k-mooneyaspiers: grabing tea but ill rebase the follow up patch and fix the test to work on py27 shortly11:32
*** slaweq has joined #openstack-nova11:33
*** dave-mccowan has joined #openstack-nova11:34
*** tbachman has quit IRC11:36
*** slaweq has quit IRC11:41
*** slaweq has joined #openstack-nova11:52
*** slaweq has quit IRC11:57
*** tbachman has joined #openstack-nova11:59
aspierssean-k-mooney: cool12:00
*** bjolo has joined #openstack-nova12:00
openstackgerritBalazs Gibizer proposed openstack/nova master: allow getting resource request of every bound ports of an instance  https://review.opendev.org/65511012:04
openstackgerritBalazs Gibizer proposed openstack/nova master: Pass network API to the conducor's MigrationTask  https://review.opendev.org/65511112:04
openstackgerritBalazs Gibizer proposed openstack/nova master: Add request_spec to server move RPC calls  https://review.opendev.org/65572112:04
openstackgerritzhangyujun proposed openstack/nova master: Should not raise when restore power on failed  https://review.opendev.org/62485412:07
*** dtantsur|bbl is now known as dtantsur12:08
*** ratailor_ has joined #openstack-nova12:08
*** ratailor has quit IRC12:08
*** markvoelker has joined #openstack-nova12:10
openstackgerritBalazs Gibizer proposed openstack/nova master: re-calculate provider mapping during migration  https://review.opendev.org/65511212:11
*** slaweq has joined #openstack-nova12:12
openstackgerritBalazs Gibizer proposed openstack/nova master: update allocation in binding profile during migrate  https://review.opendev.org/65642212:13
openstackgerritBalazs Gibizer proposed openstack/nova master: Extend NeutronFixture to handle migrations  https://review.opendev.org/65511412:13
*** slaweq has quit IRC12:17
openstackgerritBalazs Gibizer proposed openstack/nova master: prepare func test env for moving servers with bandwidth  https://review.opendev.org/65510912:18
*** ratailor_ has quit IRC12:18
gibisean-k-mooney: addressed your comments ^^12:19
openstackgerritBalazs Gibizer proposed openstack/nova master: Func test for migrate server with ports having resource request  https://review.opendev.org/65511312:20
openstackgerritBalazs Gibizer proposed openstack/nova master: Make _rever_allocation nested allocation aware  https://review.opendev.org/67613812:22
sean-k-mooneycool ill try an emake it through the full seriese end to end this week12:24
sean-k-mooneyill also try and deploy it but my backlog of thig to deploy and test is kind of long at the moment12:25
openstackgerritBalazs Gibizer proposed openstack/nova master: Func test for migrate re-schedule with bandwidth  https://review.opendev.org/67697212:25
openstackgerritBalazs Gibizer proposed openstack/nova master: Support migrating SRIOV port with bandwidth  https://review.opendev.org/67698012:27
openstackgerritBalazs Gibizer proposed openstack/nova master: Allow migrating server with port resource request  https://review.opendev.org/67149712:27
openstackgerritPeter Penchev proposed openstack/nova master: libvirt: use native AIO mode for StorPool Cinder volumes.  https://review.opendev.org/67617212:27
*** larainema has quit IRC12:35
*** jaosorior has quit IRC12:39
*** jaosorior has joined #openstack-nova12:39
*** slaweq has joined #openstack-nova12:40
gibisean-k-mooney: tanks12:41
*** nweinber has joined #openstack-nova12:43
*** priteau has quit IRC12:44
*** dougsz has quit IRC12:46
*** gbarros has joined #openstack-nova12:48
openstackgerritArtom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration  https://review.opendev.org/67259512:53
*** dougsz has joined #openstack-nova12:57
*** slaweq has quit IRC13:04
*** jmlowe has joined #openstack-nova13:04
*** jaosorior has quit IRC13:09
*** macz has joined #openstack-nova13:10
*** brinzhang_ has quit IRC13:12
*** brinzhang has joined #openstack-nova13:14
*** KeithMnemonic has joined #openstack-nova13:14
*** macz has quit IRC13:15
openstackgerritBalazs Gibizer proposed openstack/nova master: Do not query allocations twice in finish_revert_resize  https://review.opendev.org/67882713:16
openstackgerritsean mooney proposed openstack/nova master: Libvirt: report storage bus traits  https://review.opendev.org/66691413:17
openstackgerritsean mooney proposed openstack/nova master: libvirt: use domain capabilities to get supported device models  https://review.opendev.org/66691513:17
openstackgerritsean mooney proposed openstack/nova master: Add transform_image_metadata request filter  https://review.opendev.org/66577513:17
*** mriedem has joined #openstack-nova13:22
*** eharney has joined #openstack-nova13:23
*** shilpasd has quit IRC13:25
mriedemlyarwood: if you're hitting stable reviews, https://review.opendev.org/#/c/678254/ is a simple docs fix that has confused many operators13:29
lyarwoodmriedem: yup trying to, one more downstream fire to go first...13:30
*** udesale has joined #openstack-nova13:32
*** jaosorior has joined #openstack-nova13:33
*** dklyle has quit IRC13:38
*** david-lyle has joined #openstack-nova13:38
openstackgerritBalazs Gibizer proposed openstack/nova master: Add resize tests to nova-grenade job  https://review.opendev.org/67884113:41
*** priteau has joined #openstack-nova13:44
*** macz has joined #openstack-nova13:48
stephenfingibi: I'm seeing the following in a functional test. Any idea why that might be? 'AllocationDeleteFailed: Failed to delete allocations for consumer [uuid]'13:48
stephenfinBeing raised by 'delete_allocation_for_instance' in 'nova/scheduler/client/report.py'13:48
stephenfinI suspect it's to do with cleanup, but I've no idea why this test is triggering it but no others13:49
gibistephenfin: I'm on a call, I will get back to you in an hour13:49
stephenfin(y)13:49
*** macz has quit IRC13:52
*** davee_ has joined #openstack-nova13:53
artomdansmith, did you notice the new TODO above https://review.opendev.org/#/c/634827/50/nova/objects/migrate_data.py? I thought it would answer your question.13:54
dansmithartom: yes13:54
artomdansmith, OK, then I don't understand your question :)13:55
dansmithartom: is it the current instance's numa toplology?13:55
artomYes13:55
*** mlavalle has joined #openstack-nova13:55
dansmithso you're passing a whole extra nested object, which is a thing that the other side already has, to functional as a sentinel?13:55
artomWell if you put it that way, or course it's going to sound dumb ;)13:56
artomSo... just a boolean flag then?13:56
dansmithwe have a couple other boolean fields above it that indicate dest or source support for a feature13:56
sean-k-mooneywe also need the data it contians for updating the xml right13:57
dansmiththat's why I'm asking, but I don't think so13:57
artomsean-k-mooney, the dest can pull it from the database13:57
artomI need to double check, but it's probably already loaded13:57
dansmithand/or has it on the instance which is sent via rpc anyway13:57
*** mkrai has joined #openstack-nova13:58
artomAnd we can always add it to expected_attrs in the initial REST API method13:58
dansmithyou claim you already do that13:58
dansmithalso,13:58
sean-k-mooneyby the way im not really following what you are refering to are ye talkign about the LibvirtLiveMigrateNUMAInfo object13:58
dansmithit should be lazy-loadable on the instance, and if not, make it13:59
dansmithsean-k-mooney: no13:59
sean-k-mooneyoh ok13:59
artomdansmith, am I? It's possible, it's been a while :)13:59
dansmithartom: I left a question about it a few minutes ago13:59
artomRight, but before that13:59
sean-k-mooneyoh the numa_toplogy field in https://review.opendev.org/#/c/634827/50/nova/objects/migrate_data.py@24713:59
dansmithartom: I think it's probably time for mriedem to take a run through these to see how he's feeling about some of the softer things I have complained about previously,14:00
dansmithlike the two nano patches in the middle, and some of the dead code stuff you're setting up14:00
dansmithI also need to look at the functional test patch yet14:00
artomdansmith, aha, it's loaded in the REST API layer since stephenfin's "disable LM for NUMA" patch14:01
dansmithremind me why we can't test this with multinode in the gate? We can't have arbitrarily simple numa topo on a flavor that will always work, but exercise some of this?14:01
sean-k-mooneydo we need it for revert? initally i did not think we should need to have the source node numa topology in the LibvirtLiveMigrateData14:01
dansmithartom: okay, well, I think that's a long walk for a short drink of water, and selective backports could break that assumption14:02
*** Luzi has quit IRC14:02
artomdansmith, you're saying we could backport something that would stop loading instance.numa_topology in _migrate_live()?14:03
artomThat would break a whole lot of things14:03
dansmithor backport this without that, or backport them both but at different times such that it's no longer a useful sentinel14:03
artomNUMA LM isn't backportable...14:03
sean-k-mooneywe cant backport object change though14:03
mriedemwhat are you talking about? why can't the API pre-load the instance.numa_topology field?14:04
dansmithI'm just saying, tie the flag to the actual support, not to some other side effect you think is recent enough14:04
*** mrjk has joined #openstack-nova14:04
artomdansmith, oh, sorry, we're talking about different things. I'll definitely replace numa_topology in migrate_data with a boolean sentinel14:04
mriedemis there just some random, "if 'numa_topology' in instance: <make numa lm work>" logic?14:04
sean-k-mooneymriedem: we are talking about is https://review.opendev.org/#/c/634827/50/nova/objects/migrate_data.py@247 requried or can we just make it a bool14:04
dansmithmriedem: artom is saying the api is pre-loading it, and then he's checking somewhere to see if it's pre-loaded and determining whether or not the source is new enough to do the numa lm14:04
stephenfinWow, 'NUMAServersWithNetworksTest.test_cold_migrate_with_physnet' takes 12 seconds to run. That's a long-ass functional test14:04
*** liuyulong has joined #openstack-nova14:04
artomI was saying the destination will have the instance object with numa_topology loaded already14:05
sean-k-mooneycant we just check the compute node verion in the conductor14:05
artomsean-k-mooney, dansmith was saying there's caching that makes this fragile, IIRC14:05
dansmithuh what?14:06
sean-k-mooneywe do it for sriov live migration14:06
artomdansmith, there's no way I'll find the exact gerrit review now, but I definitely remember you being against checking individual compute hosts's service versions for this14:06
dansmithartom: we do it for other things, but there was probably some other wrinkle14:07
dansmithbut14:07
dansmithI'm not arguing for that here14:07
dansmithI'm saying the thing you're using does not make any sense14:07
mriedempiling on14:07
artomdansmith, and I agree :) It's a good point, I'll change it to a boolean sentinel, and the dest can pull numa_topology from the instance object it already has14:08
mriedemcan't the dest check the source compute service version like we did for file-backed memory live migration? https://github.com/openstack/nova/blob/18.0.0/nova/virt/libvirt/driver.py#L661214:08
mriedemdo we need the boolean sentinel at all?14:08
sean-k-mooneyartom: we do this https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L35-L5814:08
dansmithmriedem: we have a boolean for file-backed in that same object14:08
artommriedem, we need *something* for both computes to agree that they can do NUMA-LM14:08
sean-k-mooneyi would emulate https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L203-L23714:09
mriedemso we do https://github.com/openstack/nova/blob/18.0.0/nova/objects/migrate_data.py#L23714:09
mriedemwhich is used when generating the xml on the source to pass to the dest https://github.com/openstack/nova/blob/46a3bcd80b41e99ec4923c7cf3d0f8dd8505e97c/nova/virt/libvirt/migration.py#L26914:10
bauzasjust catching the discussion, but why can't we assume checking the RPC versions ?14:12
artomThinking about it, I believe the service version check wasn't accepted because we still want to allow operators to shoot themselves in the foot, so to speak14:12
bauzaslike we do for a couple of RPC intercommunication ?14:12
dansmithbauzas: you can't see the client version from the server side in rpc14:13
sean-k-mooneyartom: we can still do it and but not make it fail14:13
artomSo we don't just block the LM entirely if one of the hosts is too old14:13
dansmiththere's still no reason to even be talking about this14:13
dansmithall we need is to pass a boolean and we're done14:13
*** kashyap has quit IRC14:13
artomYeah, I'm with dansmith14:13
artomSeems way more elegant14:13
bauzascool, we did that with scheduler, IIRC14:13
dansmithconductor will strip it out if you're talking to an old compute automatically14:13
bauzaswe had a sentinel14:13
sean-k-mooneywell we can sett the bool in the conductor14:14
dansmithno14:14
dansmithit's migrate_data, set it in the compute14:14
dansmithagain, the only reason we're even talking about this, is artom was sending a whole topology object instead of saying "yes".14:15
sean-k-mooneyyou could. we need to tell the dest if the souce supprot the feature14:15
sean-k-mooneydoes that line up wiht the calls14:15
dansmithhe's already sending it from the compute at the right time14:15
dansmithhe's just saying too much. much like this discussion.14:15
sean-k-mooneythen ok14:15
artomsean-k-mooney, I'll be blunt and pull rank (though not my own): in case two things work the same dansmith's opinion is worth more than yours :P And I say that with massive admiration for the amount of information you keep in your head14:15
sean-k-mooneyartom: fair its anorrying not to have these checks in the same place in the conductor with the other migration checks14:16
sean-k-mooneybut i agree we should just have a bool flag14:16
dansmithno, this isn't a rank thing, it's just a simplicity thing...14:16
sean-k-mooneyso just update your current code14:16
artomdansmith, a not very convincing thing, apparently :)14:17
dansmithmigratedata is created after we're done talking to conductor for the last time, AFAIK14:17
dansmithso I'm not sure how conductor would ever set it for us14:17
artomYeah, it's created by the source compute, IIRC14:17
artomThe source driver, in fact14:17
dansmithhence why this flag should be set by the computes14:17
dansmithit also depends on the config of each compute,14:18
dansmithwhich you can't inspect from conductor14:18
dansmithif the two computes involved are pinned to 5.0, then this can't happen even if they both support it14:18
artomhttps://github.com/openstack/nova/blob/18.0.0/nova/compute/manager.py#L6056-L608914:18
stephenfingibi: calling 'self._delete_server' at the end of the test did the trick. I don't know why. Missing mocks or something like that. Good enough for me though14:19
artomI need to get one kid to daycare, after that it's hax time14:19
artommriedem, you OK with squashing, as per dansmith's -1 here: https://review.opendev.org/#/c/635229/48?14:20
sean-k-mooney ok i was suggeting passing in the booll here https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L318-L322 to check_can_live_migrate_destination but im fine with what dan suggest too14:21
openstackgerritStephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta  https://review.opendev.org/67180114:23
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Fold in argument to '_update_provider_tree_for_vgpu'  https://review.opendev.org/67672914:23
openstackgerritStephen Finucane proposed openstack/nova master: Add reshaper for PCPU  https://review.opendev.org/67489514:23
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Simplify 'fakelibvirt.HostInfo' object  https://review.opendev.org/67886114:23
* stephenfin declares cpu-resources complete.14:23
mriedemartom: my head is buried in something else atm so you're going to have to wait14:23
artommriedem, understood14:24
stephenfindansmith: If you're bored, I went and wrote that online migration to allow us to drop the Liberty-era NUMA topology blobs https://review.opendev.org/#/c/537414 I don't want to brag, but I think it's perfect14:25
dansmithstephenfin: you know that if you say that, I double-fist red pens right? :D14:27
* sean-k-mooney clicks and prepares to run screaming out of the room14:27
stephenfindansmith: I was counting on it ;)14:27
dansmithheh14:27
gibistephenfin: I'm happy that you solved it. I have no idea what was the problem without digging14:29
sean-k-mooneystephenfin: pet peve but i find the use of ~ for negation in filters really hard to spot14:30
sean-k-mooneyit took me 3 times to notice it was not containts give how far apart they are14:30
dansmithartom: did you see my question about the multinode test?14:31
dansmithartom: earlier in here14:31
*** udesale has quit IRC14:32
artomdansmith, yeah... I'd have to think about it some more, but I don't think it's possible to test it in the current gate14:33
artomOr if it is, it'd be a very small part14:33
dansmithstephenfin: you just added an online migration to the patch with the removal as before right?14:33
dansmithartom: if I had a guest that just requested one numa node, wouldn't that boot in our current gate workers?14:33
artomI *think* I'd rather concentrate on getting whitebox under openstack-qa and getting multi-NUMA node flavors from Fort Nebula14:33
stephenfindansmith: Yeah. That's what I should do, right?14:33
artomdansmith, it might, but then what?14:34
dansmithstephenfin: it doesn't change the fact that you can't remove that stuff in the same release14:34
dansmithartom: live migrate such an instance between workers and that will exercise all of this code, even if it's a boring example right?14:34
*** spsurya has joined #openstack-nova14:34
stephenfinOh, because it would have to be a blocker migration to do that?14:35
dansmithstephenfin: online migrations run after you've already rolled out new code14:35
*** dpawlik has quit IRC14:35
dansmithstephenfin: you can't have a blocker and a removal in the same release either, but I was saying a blocker migration is likely too expensive here because of what it has to check, so we should do it with time and warnings14:35
dansmithstephenfin: i.e. land an online migration now, and a nova-status check, wait a release or two, then remove the compat code after everyone has had a chance to patch up all their data14:36
dansmithstephenfin: a blocker migration helps make sure they clean up before they move, but again, too expensive here I tink14:36
stephenfinDamn. So how long does that have to hang around for once I've the migration in-place?14:36
stephenfinmkay14:36
dansmithstephenfin: well, at least a cycle, but with no blocker, it'd be nice to leave it longer14:36
* stephenfin respins14:37
dansmithmaintaining compat sucks14:37
stephenfinI'll probably pull that out of the series too, if I can14:37
dansmithit's a good reason to try to get stuff right the first time14:37
dansmithyeah, good idea14:37
stephenfintrue dat14:37
artomdansmith, it'll exercise the code, but assuming the workers have identical NUMA topologies, we'll have no way of knowing14:38
artomBecause in those cases, current live migration "works"14:38
stephenfinsean-k-mooney: fyi, six doesn't have a wrapper for the collections -> collections.abc thing14:39
*** priteau has quit IRC14:39
stephenfinthere's an open bug but no patch for it14:39
artomBoth hosts being identical means we don't need to update the XML or claim any "new" resources14:39
sean-k-mooneystephenfin: right but it does for is this python 3 or not14:39
sean-k-mooneystephenfin: but ya i need to stil fix that actully so ill respin shortly14:40
stephenfinOh, right, gotcha14:40
stephenfinCould you spin out 'flatten_iterable' into a separate patch too, in that case?14:40
stephenfinIf you have uses for it elsewhere14:40
dansmithartom: no way of knowing what? if we see the debugs that show that you're doing the stuff, we'll know it's at least running your code, and not breaking something simple14:40
sean-k-mooneyi had thought you were using sum in a few places to flatten lists which was the other usecase i wanted to adress with it14:40
dansmithartom: even if it generates the same xml on the other side, it's still doing all that work, even if for just a boring topo14:41
stephenfinYeah, I still don't think it's necessary when the sum thing is a well known pattern, but if you're going that way then it's definitely a patch in its own right14:41
sean-k-mooneybut i cant find them anymore. did you get rid of them recently14:41
dansmithartom: seems like a worthwhile thing to have when a reviewer feels uneasy14:41
sean-k-mooneyand yes ill break it out14:41
bauzasstephenfin: are you about to respin ?14:41
stephenfinoh, no other uses?14:41
stephenfinHmm14:41
bauzasI was on https://review.opendev.org/#/c/674894/11//COMMIT_MSG14:41
dansmithartom: and I would think it would be pretty much just tweaking the flavor we create for a regular live migration job14:41
stephenfindo we _really_ need it then?14:41
sean-k-mooneyyes14:41
* stephenfin is amazed Python doesn't have something like this in stdlib14:41
sean-k-mooneythe sum pattern is not a common idiom14:42
sean-k-mooneyyou are the only person i know of that has suggested it14:42
sean-k-mooneyso i would like to have a fucntion that names that algoritium14:42
sean-k-mooneyso we can reuse it and not have to rely on trible knowladge14:42
stephenfincould we use itertools.chain?14:43
stephenfinthat does this same thing, right?14:43
sean-k-mooneyits close but it does not do the right things for maps14:43
stephenfinno?14:43
sean-k-mooneyit allos does not flatten a list14:43
sean-k-mooney*list of list14:44
sean-k-mooneyi could use it internally14:44
*** lpetrut has quit IRC14:44
stephenfinyou can use '*' for that14:44
stephenfinI mean, you've one caller14:44
stephenfinThis is so overengineered :D14:44
bauzassean-k-mooney: I honestly feel flattening an iterable of iterables is just a common pattern that doesn't really need to have a general method14:44
sean-k-mooneybut based on your feedback to not use the functional part of the standard libary like reduce i wrote it as genertor14:44
stephenfinYeah, I prefer reduce to this /o\14:45
bauzassean-k-mooney: it's more like, you have so many ways to do what you want, pick one wisely14:45
stephenfinI mean, I'm not in love with it, but 2 lines >> ~50 lines14:45
sean-k-mooneyok i am going to take a break form irc now before i say some i should not14:45
bauzassean-k-mooney: like, returning a generator if you pass a couple of lists isn't really helpful, righT?14:46
*** shilpasd has joined #openstack-nova14:47
mriedemi think at this point i might just take this over to move it along https://review.opendev.org/#/c/663851/4214:56
mriedemalex_xu: gmann: efried: ^ any problem with me doing that?14:56
efriedI haven't been following it14:57
mriedemit's close, in a runway for a couple more days, and there are at least 3 other changes open for review that are conflicting for the same microversion14:57
efriedyou planning to +2 it after you "move it along"?14:57
mriedemefried: i know, just saying from a PM standpoint14:57
mriedemthat's the question - my changes would be docs/test really14:57
mriedembut alex is the only other core that's really gone through it much14:58
mriedemso yeah i'd still want to be able to +214:58
efriedIf your changes are docs/test, then I'm good with it.14:58
*** david-lyle has quit IRC15:01
*** david-lyle has joined #openstack-nova15:01
*** sridharg has quit IRC15:02
*** mtanino has joined #openstack-nova15:10
*** tbachman has quit IRC15:12
*** jmlowe has quit IRC15:13
artomdansmith, true. I don't mind doing something like that as a follow-up, as I think func tests that test more scenarios are more important to get done first15:13
dansmithartom: they're important for sure, they just don't give me all that much confidence15:13
artomdansmith, honest question, why?15:13
artomWe have no way to be sure real libvirt would accept the XML we generate?15:14
*** jmlowe has joined #openstack-nova15:14
*** mkrai has quit IRC15:14
*** damien_r has quit IRC15:14
dansmithartom: because every other thing we've landed like this has been fine in the contrived tests, but not actually work in the real world15:15
artomdansmith, heh, fair15:15
dansmithlike because we generate bad libvirt xml, or were digesting not-real-world host xml, etc15:15
dansmithnuma itself was DoA, IIRC, and pci wasn't actually usable for a long time, etc15:15
openstackgerritBalazs Gibizer proposed openstack/nova master: Make _rever_allocation nested allocation aware  https://review.opendev.org/67613815:16
openstackgerritBalazs Gibizer proposed openstack/nova master: Support reverting migration / resize with bandwidth  https://review.opendev.org/67614015:16
artomdansmith, would the current combination of manual tests + func tests be enough initially?15:17
artom... I guess it doens't address your point, because all previous things you mention were exactly in that situation15:18
dansmithartom: you mean, similar to the manual + func tests we had for things we landed borken in the past?15:18
*** itlinux has quit IRC15:18
dansmithartom: it just seems like this should be relatively simple to try, by altering the flavor on a LM job, while you're waiting for review or something15:18
dansmithmriedem: how hard is it to float a new job or changed job which does one of our multinode LM tests, but with a different flavor that specifies a simple numa topo?15:18
mriedemdansmith: the hard part there is likely just hacking the flavor,15:20
mriedembut we could probably do that within the script that runs the live migration tests themselves after configuring tempest.conf to use the new flavor15:20
dansmithyeah, I was thinking that would be doable15:20
openstackgerritBalazs Gibizer proposed openstack/nova master: Func test for migrate re-schedule with bandwidth  https://review.opendev.org/67697215:20
mriedemhttps://github.com/openstack/nova/blob/master/nova/tests/live_migration/hooks/run_tests.sh15:20
mriedemdansmith: in there ^15:21
dansmithnot necessarily landable, but just hacked into place so we can see it run15:21
dansmithartom: ^15:21
mriedemwe should be able to create a new flavor and then re-configure tempest (on the controller host) to use that for the flavor before running the tests15:21
artomdansmith, yeah, I was looking at that as well15:21
mriedemyou can see we already configure tempest from the script using ansible15:22
mriedem$ANSIBLE primary --become -f 5 -i "$WORKSPACE/inventory" -m ini_file -a "dest=$BASE/new/tempest/etc/tempest.conf15:22
mriedemso yeah, just create a new flavor and re-configure tempest with the id15:22
mriedemthis one https://github.com/openstack/tempest/blob/master/tempest/config.py#L28515:23
openstackgerritBalazs Gibizer proposed openstack/nova master: Support migrating SRIOV port with bandwidth  https://review.opendev.org/67698015:23
*** ccamacho has quit IRC15:23
openstackgerritBalazs Gibizer proposed openstack/nova master: Allow migrating server with port resource request  https://review.opendev.org/67149715:25
openstackgerritBalazs Gibizer proposed openstack/nova master: Do not query allocations twice in finish_revert_resize  https://review.opendev.org/67882715:25
artommriedem, ack, appreciated :)15:26
openstackgerritStephen Finucane proposed openstack/nova master: objects: Add online migration for legacy NUMA objects  https://review.opendev.org/53741415:27
dansmithartom: sean-k-mooney wrote these functional tests didn't he?15:27
artomdansmith, no... You're implying there's loads of typos?15:27
dansmithhehe, yes15:27
dansmithyou must be hanging around him too much then15:28
sean-k-mooney:)15:28
artomI suck up to the best :D15:29
sean-k-mooneyeventrully my influnce will spread to all in nova15:29
*** dpawlik has joined #openstack-nova15:29
dansmithover my dead body15:29
artomThat can be arranged15:29
dansmithbelieve me, I know15:31
artomo_O15:31
artomDid you upset the mafia or something15:31
sean-k-mooneybauzas: stephenfin im just running the test locally but i have droped the flatten_iterable function and replaced it with itertools.chain(*model.values()) as stephen suggested it does more or less the same thing so it will do form my usecase15:31
dansmithartom: I assume you have russian mob connections15:31
artomdansmith, I like to pretend I do :)15:32
bauzassean-k-mooney: honestly, I don't want to overthink this :p15:32
bauzasitertools.chain() is cool with me15:32
sean-k-mooneyneither do i but i want something that will merge without me haveing to keep rewriting it15:32
bauzassean-k-mooney: because of the unpredictability of the dict ordering ?15:33
sean-k-mooneyok15:33
stephenfinbauzas: RE: https://review.opendev.org/#/c/674894/11/, will I just generate a stub 'RequestSpec' for the 'resources_from_flavor' caller?15:33
*** dpawlik has quit IRC15:33
stephenfinas in to address https://review.opendev.org/#/c/674894/11/nova/scheduler/utils.py@39315:33
bauzassean-k-mooney: there are ways to predict a dict ordering but whatever15:33
sean-k-mooneyno this is the 4 way i have done this15:33
sean-k-mooneyin in those pathces15:33
bauzasstephenfin: yeah, I think it would be fine15:33
stephenfin(y)15:34
* stephenfin respins15:34
sean-k-mooneybauzas: it is noting to do with dict ordering15:34
sean-k-mooneythat was just a side effect of how i wrote the tests15:34
bauzasstephenfin: I was cool with (flavor, image) as parameters15:34
bauzasstephenfin: but is_bfv really did hurt my guts :p15:34
bauzassean-k-mooney: ok15:34
stephenfinunderstandable :)15:35
openstackgerritMerged openstack/nova stable/stein: lxc: make use of filter python3 compatible  https://review.opendev.org/67649615:37
openstackgerritMerged openstack/nova stable/stein: Add an issue releasenote for placement eventlet stall  https://review.opendev.org/67697315:37
*** gyee has joined #openstack-nova15:37
*** tbachman has joined #openstack-nova15:38
*** mkrai has joined #openstack-nova15:42
*** macz has joined #openstack-nova15:42
openstackgerritArtom Lifshitz proposed openstack/nova master: DNM: Run LM integration tests with NUMA flavor  https://review.opendev.org/67888715:45
artomI'm pretty sure ^^ is entirely broken, but it's a first step15:46
*** jangutter has quit IRC15:46
* artom goes back to Nova code15:46
dansmithartom: seems plausible15:48
sean-k-mooneyartom: that will fail because we dont have nested verit or a new enough qemu to enable the mttcg backend15:50
artomsean-k-mooney, I thought vexxhost enabled that?15:50
sean-k-mooneythey do15:50
sean-k-mooneydid you change the nodeset to the vexxhost one15:50
*** brault has quit IRC15:50
artom... no :(15:51
sean-k-mooneyit could land on vexhost still15:51
sean-k-mooneyand pass15:51
sean-k-mooneyactuly it will still fail15:51
sean-k-mooneywe disable kvm in the gate too15:51
sean-k-mooneyyou would have to change the node set and then set the virt type to kvm15:51
sean-k-mooneythen it should work15:51
*** brinzhang has quit IRC15:52
*** brinzhang has joined #openstack-nova15:52
dansmithsean-k-mooney: why will it fail without kvm? is it because the host will have no numa info so we'll fail to schedule anything?15:53
*** gbarros has quit IRC15:53
sean-k-mooneyno it fails becasue when we set hw:numa_node=1 we pin the guest cpus to float over just 1 host numa node15:54
sean-k-mooneywe do that using the vcpupin element instad of the cpuset attibute on the vcpu element15:54
sean-k-mooneythe vcpupin element which does per core pinning is only supported with kvm or the mttcg backend15:55
dansmithah okay15:55
sean-k-mooneythe normal tsg backend that qemu ueses only support pinning via the <vcpu cpuset=""> method15:56
sean-k-mooneyso in pricaipal there is no reason it cant work just the way we generate the xml breaks it15:56
sean-k-mooneyor use a newer qemu/libvirt15:57
*** shilpasd has quit IRC15:57
sean-k-mooneyi belive the one in fedroa 31 will be new enough by defualt15:57
sean-k-mooneyf29 and i think f30 still need the virt preview repo enabeld15:57
dansmithsean-k-mooney: maybe you could fix up that patch for artom to do whatever it is that needs doing?15:58
sean-k-mooneyi can try yes15:59
sean-k-mooneyi also need to fix https://review.opendev.org/#/c/652197/15:59
sean-k-mooneythat is my upstream nfv job15:59
sean-k-mooneyit uses the really new qemu form the virt preview repo but devstack/zuul jobs move a file or change the permisions16:00
sean-k-mooneyso its currenly broken16:00
sean-k-mooneyyou can see i was testing hugepage + numa + pinning in the gate16:00
sean-k-mooneyhttps://review.opendev.org/#/c/652197/18/playbooks/nfv/nfv.yaml16:00
sean-k-mooneyif i can get that working again we can test artoms live migration stuff in the gate and stephenfin  cpu pinning spec16:01
*** mtanino has quit IRC16:02
sean-k-mooneyThe destination directory (/opt/stack/devstack) is not writable by the current user. Error was: [Errno 13] Permission denied16:03
sean-k-mooneyartom: is it /opt/stack/new/devstack?16:03
artomsean-k-mooney, at this point, whatever's quicker/more demonstrative16:05
sean-k-mooneyi think if i jsut add "become: yes" to my patch it might work16:05
openstackgerritStephen Finucane proposed openstack/nova master: scheduler: Flatten 'ResourceRequest.from_extra_specs', 'from_image_props'  https://review.opendev.org/67489416:05
openstackgerritStephen Finucane proposed openstack/nova master: Remove 'hw:cpu_policy', 'hw:mem_page_size' extra specs from API samples  https://review.opendev.org/67533816:05
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Start reporting PCPU inventory to placement  https://review.opendev.org/67179316:05
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: '_get_(v|p)cpu_total' to '_get_(v|p)cpu_available'  https://review.opendev.org/67269316:05
openstackgerritStephen Finucane proposed openstack/nova master: trivial: Rewrap definitions of 'NUMACell'  https://review.opendev.org/67439516:05
openstackgerritStephen Finucane proposed openstack/nova master: hardware: Differentiate between shared and dedicated CPUs  https://review.opendev.org/67180016:05
openstackgerritStephen Finucane proposed openstack/nova master: objects: Rename 'fields' import to 'obj_fields'  https://review.opendev.org/67410316:05
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Start reporting 'HW_CPU_HYPERTHREADING' trait  https://review.opendev.org/67557116:05
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Simplify 'fakelibvirt.HostInfo' object  https://review.opendev.org/67886116:05
openstackgerritStephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta  https://review.opendev.org/67180116:05
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Fold in argument to '_update_provider_tree_for_vgpu'  https://review.opendev.org/67672916:05
openstackgerritStephen Finucane proposed openstack/nova master: Add reshaper for PCPU  https://review.opendev.org/67489516:05
sean-k-mooneyill do that and then look at yours16:05
openstackgerritArtom Lifshitz proposed openstack/nova master: DNM: Run LM integration tests with NUMA flavor  https://review.opendev.org/67888716:06
openstackgerritsean mooney proposed openstack/nova master: Libvirt: report storage bus traits  https://review.opendev.org/66691416:08
openstackgerritsean mooney proposed openstack/nova master: libvirt: use domain capabilities to get supported device models  https://review.opendev.org/66691516:08
openstackgerritsean mooney proposed openstack/nova master: Add transform_image_metadata request filter  https://review.opendev.org/66577516:08
mriedemsean-k-mooney: we can override the virt_type in local.conf for the job to use kvm16:09
mriedemsince this is a hack test patch anyway16:10
sean-k-mooneyyes we can16:10
*** jawad_axd has joined #openstack-nova16:10
sean-k-mooneyartom: i think the nodeset you want is ubuntu-bionic-vexxhost but i need to double check16:11
artomsean-k-mooney, yeah, I'm going in completely blind and relying on the gate to tell me what's wrong16:11
artomSo that's entirely possible16:11
openstackgerritMatt Riedemann proposed openstack/nova master: DNM: Run LM integration tests with NUMA flavor  https://review.opendev.org/67888716:11
sean-k-mooneyyou use it like this16:11
sean-k-mooneyhttps://opendev.org/openstack/magnum/src/branch/master/.zuul.yaml#L152-L15816:11
artommriedem, *tips fedora* your help is appreciated16:12
sean-k-mooneydo you want to test master our your change?16:13
openstackgerritsean mooney proposed openstack/nova master: Libvirt: add nfv job  https://review.opendev.org/65219716:17
sean-k-mooneyi think ^ will fix my nfv job16:17
*** mkrai has quit IRC16:19
sean-k-mooneythe multi node version of that enable live migration alther i might need to add the flag to allow that16:19
*** ivve has quit IRC16:22
*** cdent has quit IRC16:22
*** david-lyle is now known as dklyle16:23
openstackgerritsean mooney proposed openstack/nova master: Libvirt: add nfv job  https://review.opendev.org/65219716:25
*** gbarros has joined #openstack-nova16:32
*** slaweq has joined #openstack-nova16:33
*** liuyulong has quit IRC16:33
openstackgerritStephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta  https://review.opendev.org/67180116:36
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Fold in argument to '_update_provider_tree_for_vgpu'  https://review.opendev.org/67672916:36
openstackgerritStephen Finucane proposed openstack/nova master: Add reshaper for PCPU  https://review.opendev.org/67489516:36
openstackgerritStephen Finucane proposed openstack/nova master: libvirt: Start checking compute usage in functional tests  https://review.opendev.org/67890216:36
stephenfinbauzas: fwiw, I think I've everything in https://review.opendev.org/674894 resolved if you wanted to take another look16:37
stephenfinI, however, am out of here o/16:37
openstackgerritsean mooney proposed openstack/nova master: DNM: Run LM integration tests with NUMA flavor  https://review.opendev.org/67888716:39
sean-k-mooneyartom: mriedem dansmith i think ^ will fix it16:39
*** dtantsur is now known as dtantsur|afk16:42
*** amotoki is now known as amotoki_16:42
efriedstephenfin: you still around?16:45
*** mdbooth_ has quit IRC16:46
efriedI've been in meetings for 2h wanting to talk to you about this, but afraid I may have missed my window :(16:46
artomsean-k-mooney, ack, much thanks!16:50
sean-k-mooneywe should have 3 jobs trying to test numa with the gate as we speak16:50
sean-k-mooneythe singel and multinode jobs i wrote and running and yours16:51
sean-k-mooneyi assume i should be able to go test the latest version of your code again manually?16:51
*** psachin has joined #openstack-nova16:53
sean-k-mooneyoh i just rediscovered somthing quite useful16:56
sean-k-mooneythe experimenatl queue is way fater the check. which i kind of knew other pipleine are faster but i forgot16:57
openstackgerritMatt Riedemann proposed openstack/nova master: Specify availability_zone to unshelve  https://review.opendev.org/66385116:57
*** dougsz has quit IRC16:59
*** derekh has quit IRC17:02
mriedemalex_xu: for your tomorrow, please take another pass on ^17:03
mriedemso we can get 2.77 in and move on17:03
*** jawad_axd has quit IRC17:03
dansmithartom: it might be easier to discuss https://review.opendev.org/#/c/635669/39/nova/compute/resource_tracker.py here17:06
artomdansmith, shoot :)17:06
dansmithartom: my point is that you're assuming that if we're live migrating an instance, you can pass an empty PCIRequests object into the migration claim, because a live-migrating instance couldn't possibly have pci devices, right?17:07
artomdansmith, no, because we don't want the claim to think it has any PCI devices17:07
dansmithum17:08
dansmithartom: why is that?17:08
artomBecause for a subset of PCI devices (SRIOV Neutron ports), that's handled buy Sean's code17:08
artomSo we don't want to refuse based something that should otherwise work17:09
artomThe missing piece here is that all other PCI devices *should* get refused17:09
dansmithshould get refused somewhere earlier in the stack right?17:09
artomYeah17:09
dansmiththat's the part I don't like17:09
artomI agree - it's also current/latent17:10
artomIIRC we just blow up if we try to migrate an instance with PCI alias-based passthrough17:10
sean-k-mooneywe check and fail the migration here https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L216-L22117:11
dansmithartom: it's not latent because it's the code you're adding17:11
sean-k-mooneyif the instance has a pcirequest that is not a neutron port we never call artoms code17:11
dansmithsean-k-mooney: currently.17:11
artomsean-k-mooney, aha, thanks for that - that confirms my idea that we should do claims *after* your MigrationPreCheck17:11
artomSo I'll have to change that17:12
sean-k-mooneyartom: https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L317-L32617:12
sean-k-mooneywe do the pci check before calling check_can_live_migrate_destiation17:12
sean-k-mooneyso we should do the claims in check_can_live_migrate_destination17:13
artomAh, sorry, missed the wider context in your link. OK, then we're good.17:13
dansmithartom: so my concern is that you say "we should never get here" and throw up an empty PCIRequests object, which I think goes into the migration context, and eventually gets applied to the instance17:13
artomdansmith, ah, indeed17:13
sean-k-mooneydansmith: we get there for cold migration17:13
dansmithartom: so if the conductor stuff changes in the future and we *do* get here, I worry that we could dump the PCIRequests from the instance when we apply17:13
dansmithsean-k-mooney: I know, but cold migration is still handled.. that's not my problem.. my problem is that live migration is *begging* to become a silent data corruption problem in the future I think17:14
artomdansmith, so for live migration _test_pci in the claim would have to be skipped entirely17:14
sean-k-mooneywe put the check in specifcally to prevent issue related to pci devices17:14
sean-k-mooneywe would only remove it if we added support for it17:15
artom... while still keeping pci_requests?17:15
dansmithartom: it looks to me like for cold, we get the requests and filter them based on their handled-by-neutron-ness or something17:15
sean-k-mooneywhich i dont think we ever will as we cant safely hot unplug generic pci devices17:15
* artom double-checks how the migration context is applied/dropped for pci devices17:15
dansmithartom: blindly without checking17:16
sean-k-mooneyok so why don t we jsut do the same check again17:16
dansmithyou guys are saying it's fine to make dangerous assumptions because "nobody will ever ask for X" but that's *always* a bad assumption17:16
*** jaosorior has quit IRC17:16
dansmithsean-k-mooney: that's my point, assert that the thing is the way you think it is, instead of just assuming you can ignore all of it because something upstream of you did the check17:16
artomdansmith, so if we keep pci_requests, but skip _test_pci...17:16
dansmithbecause in the future when that check is gone, different, mixed-versions, etc...17:16
sean-k-mooneydansmith: no im not im saying we specificaly thought about this edgecase in the spec and code and added a check to prevet the dangours edge case for the sriov mirgation17:17
dansmithartom: would it make sense to just make _test_pci check the neutron-ness (or whatever) and not freak out over those things?17:17
sean-k-mooneybut im totally fine with adding another chacke for numa path17:17
sean-k-mooneyjust copy past17:17
sean-k-mooneyfor pci_request in self.instance.pci_requests.requests:17:17
sean-k-mooney            if pci_request.source != objects.InstancePCIRequest.NEUTRON_PORT:17:17
artomdansmith, so move the check that sean-k-mooney linked into the claim?17:17
sean-k-mooney                # allow only VIF related PCI requests in live migration.17:17
sean-k-mooney                raise exception.MigrationPreCheckError(17:17
sean-k-mooney                    reason= "non-VIF related PCI requests for instance "17:17
sean-k-mooney"are not allowed for live migration.")17:17
sean-k-mooneyno17:18
sean-k-mooneywe dont want to fail late17:18
sean-k-mooneybu just check again in the compute node17:18
dansmithcan we not do the same check in _test_pci, and ignore any that are vif-related, and fail if we find non-vif-related ones/17:18
*** jungleboyj has joined #openstack-nova17:18
artomdansmith, conversely, make apply_migration_context() smarter, and only set the new_ fields on the instance if they're set on the migration_context?17:19
dansmithartom: no17:19
dansmithartom: because that is deeply-buried silent ignoring of data you're trying to save, just because you only think that will happen from the migration path17:19
dansmiththis is all about making claims work for pci devices, so make it work, don't just "fix" the problem with lots of side effects everywhere else17:19
artomdansmith, claims *already* work for PCI devices17:20
dansmithsorry, making claims work during live migration17:20
dansmithmind-o there17:20
artomThey're used when booting and cold migrating17:20
dansmithyes yes I know17:20
artomThis is about having PCI live migration in different places17:20
artom*PCI live migration logic in  different17:21
artomdansmith, so correct me if I'm wrong, but looking at https://github.com/openstack/nova/blob/master/nova/objects/instance.py#L1004-L100717:21
artomSeems apply_migration_context is already smart enough to only look at fields in migration_context that are actually set17:21
dansmithwhat's your point?17:22
artomSo what's so bad about removing pci_requests from live migration claims entirely?17:22
artomWell, I'm assuming that'll also remove them from the migration context17:22
dansmithbecause that's obscure behavior to solve the actual problem, which is that you need to keep pci_requests, but only allow the claim to work if they're of the right type17:23
*** brault has joined #openstack-nova17:23
dansmithwhy just ignore them entirely and keep them out of the context as a special case when you can keep all the other code the same, and just not proceed if they're not valid for live migration?17:23
artomI'd dispute that - pci_requests is handled outside the claim, so why does the claim need to work with them?17:23
sean-k-mooneywhy do we not jsut check the migration type here https://review.opendev.org/#/c/635669/39/nova/compute/resource_tracker.py@307 and raise an excepion if any pci device are requested that are not related to neutron17:23
sean-k-mooneydoes ^ that not solve the issue17:24
dansmithsean-k-mooney: YES17:24
dansmithit's what I'm asking for17:24
sean-k-mooneyit should never raise but f it does it means we fucked up and remove the conductor check by mistake17:24
dansmithor something changed17:24
sean-k-mooneyya17:25
dansmithif that were to happen with the current code, we would just blow away pci_requests on the instance17:25
sean-k-mooneybut it would catch such a change17:25
*** ivve has joined #openstack-nova17:25
dansmithright17:25
artomsean-k-mooney, will the claim pass though?17:25
dansmithwe're already examining the requests for other migration types and doing things like this, we should do the same for live migration, but whatever different behavior we need17:25
artomI'm worried about double-claiming17:25
sean-k-mooneywe should not try to claim them if it is a live migration17:26
sean-k-mooneyif it not we should17:26
sean-k-mooneybut we should check instead of just nulling it out17:26
artomsean-k-mooney, exactly, if this is a live migration, the claim cannot have any pci_requests, because your code handles all that17:26
*** ivve has quit IRC17:26
sean-k-mooneyyes.17:26
*** slaweq has quit IRC17:27
sean-k-mooneywe could make the other code use the move claims17:27
artomYou said you don't want to fail late17:27
sean-k-mooneybut we didnt wnat to do that since that code make so many assumtion about it not being a live migration17:27
*** ivve has joined #openstack-nova17:27
sean-k-mooneyi dont17:27
sean-k-mooneyi still want to keep the check we have in the conductor17:27
dansmithme too17:27
sean-k-mooneyim just saying we check twice and dont care that we already checked17:28
*** slaweq has joined #openstack-nova17:28
artomOK, so say we keep that check, and add a the same kind of check at https://review.opendev.org/#/c/635669/39/nova/compute/resource_tracker.py@30717:28
artomAnd we live migrate an instance with NUMA and a Neutron SRIOV port17:28
artomBoth of those checks will pass, right?17:28
sean-k-mooneyyes17:28
sean-k-mooneyand in that case you can set pci_resst=[] safely or we can rewrite the sriov stuff to get teh device form the move claim17:29
*** psachin has quit IRC17:29
*** dpawlik has joined #openstack-nova17:29
*** tesseract has quit IRC17:30
artomWell we can't just set pci_requests=[], because as dansmith pointed out, when we apply that claim, we'll clobber the instance's pci requests with the empty one from the claim17:30
artomSo we have to keep the current logic17:30
artomExcept17:30
artomYour code is claiming PCI devices here: https://review.opendev.org/#/c/634606/58/nova/compute/manager.py@646517:31
artomAnd mine will run the MoveClaim either immediately before or after yours17:31
sean-k-mooneywe would only do it in the neutron case and we create new request in that case in my code anyway17:31
dansmithwhy is it that this needs to be different at all between live and cold migration?17:31
artomNow this may be ignorance on my part, but to me it would mean one of those claims will fail because the previous one will have used up the device17:31
artomOr can fail, at least17:32
sean-k-mooneymainly when we reviewd the sriov spec it was decied that it did nto make sense to make move claims work for live migration but we decided to use the for numa17:32
* dansmith is confused and frustrated17:33
*** slaweq has quit IRC17:33
*** psachin has joined #openstack-nova17:33
artomdansmith, I get ya17:33
*** dpawlik has quit IRC17:34
artomI'm proposing the "workaround" of unsetting pci_requests from live migration claims to avoid any clobbering, and we can consolidate SRIOV and NUMA claims in a follow up?17:34
dansmith-2 on that17:34
artomSo... straight up consolidation right away?17:35
dansmithanswer my question above about why this needs to be different17:35
sean-k-mooneyif we most but i really dont like move claimes17:35
dansmithnot sure if sean-k-mooney answered it or not, but I didn't understand the words17:36
sean-k-mooneybut it should not be too hard to make it work17:36
artomdansmith, why live migration claims need to be different with respect to PCI devices?17:36
dansmithsean-k-mooney: isn't the whole point of this work to integrate the move claims with this stuff so that it works?17:36
dansmithartom: yes17:36
sean-k-mooneywe thought that was too much work to justify doing that for sriov17:36
sean-k-mooneyalso if i recall jay was not a fan of that approch17:37
*** bbowen has quit IRC17:37
dansmithso the plan was just to re-use most of the same code paths but special-case out the live migration?17:37
sean-k-mooneywhich is why we built on the multiple port binding feature and do not use port bindings17:37
artomdansmith, because for the subset of pci_requests that's supported for live migration, claiming resources on the destination is already being handled in https://review.opendev.org/#/c/634606/58/nova/compute/manager.py@646517:38
sean-k-mooneyno sriov migration jsut does not use move claims because live migration dose not use them17:38
artomsean-k-mooney, so you left me to do the gruntwork l)17:38
*** ralonsoh has quit IRC17:38
artom;)17:38
sean-k-mooneyartom: honestly i was hoping you would not use move claims either but i kind of gave up on that17:39
artomdansmith, and that code predates using claims for live migration at all17:39
*** bbowen has joined #openstack-nova17:39
dansmithI honestly don't understand this17:39
artomdansmith, can you expand on that "this" is :)17:39
dansmithyou want to use move claims for numa but not pci and have a separate code path just for pci live migration?17:39
artomdansmith, PCI predates NUMA17:40
dansmithand?17:40
artomAnd they decided not use to claims, presumably because the PCI tracker makes it easier to claim resources without going through actual MoveClaim objects17:40
sean-k-mooneyand both spec propsoed different approche to the same thing because different peopel reveies and had differen preferences17:40
*** itlinux has joined #openstack-nova17:40
dansmithyou seem to be saying "I can make the mess bigger because it's already a mess" and "even though we're rounding out live support for a thing that already works cold, I'm going to do it a different way because there are already lots of fragments"17:40
dansmithneither of those really satisfy me17:41
artomFor NUMA LM, MoveClaim was a handy way to get both claiming of resources and the new instance NUMA topology in a "single operation"17:41
sean-k-mooneymove claims are not stictly requried for numa migration either17:41
sean-k-mooneybut it would require a lttile more work17:42
artomNo, but they do make things less racy and simplify getting the new instance numa topology17:42
sean-k-mooneyartom was trying to reuse the existing cold migraiton code to reduce the code change17:42
sean-k-mooneyartom: i dont think they do17:42
sean-k-mooneywe both claim in the same place. more or less17:42
*** gbarros has quit IRC17:42
artomsean-k-mooney, yeah, maybe there was a way to do it less racy-ly even without claims17:44
dansmithtbh, I don't really care whether this uses claims or not,17:44
dansmithbut if you're going to,17:44
dansmithI think that just "if live migration: do different thing" in a bunch of random places is not moving us forward17:44
*** psachin has quit IRC17:45
dansmithespecially when it comes to things that may silently break data17:45
mriedem"if pci: something something dragons that no one but sean understands"17:45
artomdansmith, so that would mean folding SRIOV live migration into the claims17:45
mriedem^ since juno17:45
dansmithartom: btw, you've done all this local testing of this.. have you included sriov migration and not seen it clobber pci_requests?17:45
sean-k-mooneywe currently do the claim here for sriov https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6428-L646717:45
artomdansmith, SRIOV with this is completely untested :(17:45
sean-k-mooneyright after we call check_can_live_migrate_source in check_can_live_migrate_destination17:46
dansmithartom: so it's very likely that you're blowing those away with this yeah?17:46
artomAlthough sean-k-mooney's saying apparently all recent-ish NICs can do SRIOV, so maybe I *do* have the hardware?17:46
sean-k-mooneyartom: i have hardware and i set up port forwading so you can ssh in.17:46
sean-k-mooneybut im going to check both again after dinner17:47
*** hemna has quit IRC17:47
*** itlinux has quit IRC17:47
sean-k-mooneyim going to grab dinner but ill kick of a devstack run before i go and setup the  test enviromint17:49
artomdansmith, seems likely, yeah17:49
*** itlinux has joined #openstack-nova17:50
sean-k-mooneywe dont use the pci request spec object from the instance sriov migration by the way17:51
sean-k-mooneyhttps://github.com/openstack/nova/blob/master/nova/compute/manager.py#L6428-L899317:51
dansmithsean-k-mooney: because why? we store details in neutron about it?17:51
artomHrmm17:52
artomSo actually17:52
sean-k-mooneythe only info we need to pass back as the pci address of the new device17:52
artomAFAICT17:52
artomThe PCI stuff in the claim doesn't actually *claim* any resources17:52
sean-k-mooneyso we store that in the vif port profile which is wher ewe pass it to neutron17:52
artomJust tests that it's supported17:52
dansmithsean-k-mooney: okay, but nova's own structure would still have them in the pci_requests right?17:53
sean-k-mooneyyes17:53
dansmithso what about like rebuild or something?17:53
artomAh, no, it does the _decrease_pool_count() thing, which "claims" the resource17:53
*** eharney has quit IRC17:54
sean-k-mooneyhow does it interact with this?17:54
sean-k-mooneythe claims have 3 states17:54
sean-k-mooneyfree,cliamed and allocated17:54
sean-k-mooneysomething like that17:54
sean-k-mooneyone of them meens its reserved for an instace the the otehr is its in use17:55
dansmithI'm saying I would expect we use pci_requests if we're doing a rebuild on the instance17:55
dansmithor other operations17:55
*** itlinux has quit IRC17:55
dansmithpoint is just that corrupting our own accounting because sriov is tracked in neutron is not okay I don't think17:55
sean-k-mooneyso we are claiming the ot reserve them and later we update the instace with them weh we move it to the allocated state i think. its beed a whild and i dont really rememerb the details17:55
sean-k-mooneythis is the import cahgne https://review.opendev.org/#/c/620115/35/nova/compute/manager.py17:56
mriedemwe do'nt claim for rebuild17:56
*** itlinux has joined #openstack-nova17:56
mriedemit's a noop claim17:56
dansmithmriedem: sure, not related to claim17:56
dansmithcross-cell-migrate looks at pci_requests17:56
dansmithlooks like regular live migration also looks at them to determine if they're all neutron-related17:57
dansmithjust trying to confirm that blowing them away is not something we can just ignore :)17:58
*** jaosorior has joined #openstack-nova18:00
efrieddansmith: I'm looking at consecutive_build_service_disable_threshold and not seeing how it could be working18:01
dansmithefried: we removed the compute side of that, if that's what you're looking at18:01
efried"the compute side"18:01
efriedmeaning... the part that makes it behave in any way other than a bool?18:01
dansmiththe part where the compute node self-disables18:01
dansmiththere should be renos about this18:02
efriedack18:02
dansmithchange-consecutive-boot-failure-counter-to-weigher-428de7da0ed2033a.yaml18:02
efrieddansmith: okay18:03
efriedso18:03
efriedthis is now up to deployers installing their own weigher?18:04
dansmitheh?18:04
dansmithdid you read the reno?18:04
dansmithefried: https://review.opendev.org/#/c/572195/18:04
efriedack18:05
dansmithadded a weigher, changed the meaning of the threshold for compatibility reasons18:05
mriedemhttps://docs.openstack.org/nova/latest/user/filter-scheduler.html#weights18:05
mriedemBuildFailureWeigher18:05
dansmithefried: you hip to that jive?18:07
efriedStill trying to grok the implications.18:07
efriedSo in a biggish cloud, if I have a bunch of failures in a row on a particular host, even if I don't hit the default multiplier, it's still going to make scheduling to that node way less likely18:08
dansmithmriedem: beat you to the youtube ref: https://www.youtube.com/watch?v=sgW3RxKdN0Q18:08
*** dpawlik has joined #openstack-nova18:08
dansmithefried: hit the multiplier meaning "change" it?18:09
efriedsorry, no, I just mean...18:09
* mriedem feels like efried has been tricked into doing bug triage for the RH team18:09
efriedIIUC the default multiplier is really big so that by default you won't effectively-disable a compute until it's seen a really lot of build failures.18:09
dansmithefried: no18:10
dansmithefried: read the reno and commit for reasoning about the multiplier18:10
dansmithefried: it has to compete against disk weigher, which is like scored by mb free or something18:10
dansmithefried: a big multiplier makes it more likely to have an effect, not les18:11
dansmith*less18:11
efriedah18:11
dansmithand the threshold is now just "report failures if nonzero", so it's not a threshold18:11
dansmithif you want it off, set the "threshold" to zero and computes won't even report the number18:11
efriedack18:12
dansmithand if you do, then you tune the multiplier based on whatever else you have configured18:12
*** gbarros has joined #openstack-nova18:12
*** dpawlik has quit IRC18:13
efriedgot it. Thanks dansmith. (And no, mriedem, I'm helping donnyd figure out why his CI hosts get effectively-disabled-without-actually-being-disabled when they spend some time trying to boot with not-yet-ready images)18:15
mriedem:P18:16
mriedemo-)18:16
efrieddansmith: save me tracing the code, does it still have the "reset to zero" behavior as soon as we get one success?18:16
dansmithIIRC yes18:16
dansmithefried: https://review.opendev.org/#/c/572195/6/nova/compute/manager.py18:16
donnydthats pending it ever actually gets rescheduled18:16
donnydI have enough space that essentially it doesn't18:17
efriedyeah18:17
dansmithhttps://review.opendev.org/#/c/572195/6/nova/compute/stats.py18:17
efrieddansmith: is this one of those things that you should theoretically be able to reset via SIGHUP?18:17
dansmithsee my FIXME in there18:17
dansmithefried: for sure18:17
efriedk. btw, not sure if you saw the update, but between bnemec and me, SIGHUP is (soon to be) fixed.18:18
efriedsoon as we can get this code merged & released18:18
donnydWhat i really need is a faster way to download glance images so this   isn't an issue18:21
efriedjust walk a flash drive across the room18:24
donnydLOL efried18:24
donnydWell the glance image store is on an nvme drive that will move at the speed of the rest of the network... however glance doesn't feel that same way, and limits my download speeds to what one core can do... which is about 100M/s18:25
donnydI think it has something to do with requests if I am not mistaken18:25
sean-k-mooneydonnyd: are you useing a 1G network for your managment network18:33
donnyd1018:33
donnydand the controllers are all 4018:33
donnydSo on the control plane for the compute side 10, and controller side 4018:33
mriedemdonnyd: are you using ceph?18:33
donnydno18:33
donnydfilestore is the fastest i have measured thus far18:34
melwittmriedem: just fyi if you didn't see, I updated the multi-cell archive patches yesterday night18:34
mriedemmelwitt: i didn't, and was going to look about 30 minutes ago, but was distracted, but i'll look in a bit, thanks18:34
donnydi tried cinder and swift backends, and they only made it slower18:34
melwittmriedem: coolness, thanks18:35
mriedemdonnyd: have you tried pre-caching the images on the computes when you have a new image?18:35
donnydif I was on ceph, there would be no wait time at all18:35
melwittand sorry for the delay18:35
mriedemdonnyd: the only thing i'd have to check is if there is some config to keep the image cache manager periodic from deleting the images if there are no guests on the host using them18:35
donnydmriedem: I didn't know that was an option... but I am also pretty sure nodepool is all over it whenever there is a new image18:36
mriedemthe image cache stuff in nova isn't documented at all outside of config (i don't think anyway), so i wouldn't be surprised18:36
*** gbarros has quit IRC18:36
donnydmostly the issue is when new images are loaded, they can't be downloaded fast enough by compute18:36
mriedemi don't know it all that well either18:36
mriedemright, by pre-caching you wouldn't need compute to download them, the images would be there18:37
mriedembut that's something yo'ud have to orchestrate outside of nova18:37
donnydwell nodepool asks for them nearly the instant that they are active in glance18:37
donnydat least from what i can see18:37
mriedemok yeah so maybe wouldn't help18:38
donnydwhen you say pre-caching, i am thinking you mean have nova launch something based on the new image and then kill it immediately after its active.18:38
mriedemthat's one way18:39
*** jmlowe has quit IRC18:39
mriedemprobably the easiest18:39
donnydI don't know how to setup the http store for glance and have it download direct from say an apache server18:39
mriedemor you could have something push the image here https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.image_cache_subdirectory_name18:39
mriedemjroll: doesn't verizon do extensive image pre-caching?18:39
melwittpre-caching (warming the cache by booting instance with new image 1 per compute host) is the only other way I know to deal with avoiding download speed issues, other than using ceph. I know at least in the past they did the cache warming thing at yahoo18:40
mriedemok yeah that ^18:40
jrollif we do it idk about it18:40
melwittmriedem: haha, my slow message composition strikes again18:40
mriedemwas going to ask penick but he's not here18:40
jrollrax did some of that too18:40
donnydI thought about setting up a share from my block image server that would share _base across hypervisors18:40
donnydmaking it only have to download once18:40
donnydwhich is acceptable18:40
donnydbut i am not sure of the other performance implications of doing such a thing18:41
jrollmriedem: is there something I can help with around that?18:41
mriedemyeah, people do that https://bugs.launchpad.net/nova/+bug/180426218:41
openstackLaunchpad bug 1804262 in OpenStack Compute (nova) "ComputeManager._run_image_cache_manager_pass times out when running on NFS" [Medium,In progress] - Assigned to Matthew Booth (mbooth-9)18:41
mriedemjroll: not really18:41
jrollk :)18:41
mriedemi thought i saw something (blog/talk?) where penick was talking about doing this18:41
jrollpossibly18:41
mriedemi was probably surprised because it was some non-baremetal thing18:42
donnydI was only going to mount _base so the first hypervisor to download the image would speed it up for the rest18:42
jrolljust not sure if there's a question to be answered or if you're just curious18:42
jrollrax did bare metal image caching and it was awesomesauce18:42
mriedemdonnyd: yeah i'm pretty sure i was talking with someone in here awhile back that had the same idea18:42
melwittyeah sharing _base/ makes sense18:42
donnydjroll: Well when new images are loaded from nodepool, it takes a while for compute to download from glance X # of hypervisors18:43
donnydso some pre-caching or sharing of _base would likely speed things up18:43
donnydGlance is seemingly limited to one core's worth of power when downloading18:44
donnydwhich for my controllers is unfortunately not fast18:45
donnydmelwitt: my only concern in mounting _base on a shared drive is that what happens when multiple hypervisors are all trying to download the same image at the same time18:46
mriedemthere might be a lock in the code, but would need to verify18:47
melwitthm, yeah18:47
mriedemin fact, i think starlingx people added a configurable lock there18:47
jrolldonnyd: gotcha, thanks for context18:47
donnydwould they all download it anyways, or as mriedem just said would they see another hypervisor already has that in-flight18:47
artomsean-k-mooney, do you recall if the SRIOV live migration code ever updated instance.pci_requests?18:48
artomI don't see anything quickly glancing through your patches18:48
mriedemdonnyd: this is what i'm thinking of https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/imagebackend.py#L25718:48
sean-k-mooneywe do update them when we allocate the claimed pci deivce i belive18:49
mriedemhttps://docs.openstack.org/nova/latest/configuration/config.html#compute.max_concurrent_disk_ops was the thing starlingx added18:50
donnydwell I should give it a spin.. probably cut the "new image loaded so i will fail a bunch" down to almost nothing18:51
openstackgerritMerged openstack/python-novaclient master: Follow up for microversion 2.75  https://review.opendev.org/67847318:51
sean-k-mooneyartom: i think it get saved here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L7229-L725618:51
mriedemdonnyd: if you do let us know how it goes,18:51
sean-k-mooneyin post_live_migration_at_destination18:51
sean-k-mooneyso we only update it if the migration succeeds18:51
mriedemdonnyd: i also threw it in https://bugs.launchpad.net/nova/+bug/1838819/comments/7 so we don't forget docs about it18:51
openstackLaunchpad bug 1838819 in OpenStack Compute (nova) "Docs needed for tunables at large scale" [Undecided,Confirmed]18:51
sean-k-mooneyif it was reverted we do not modify the instace pci_requests18:52
mriedemimage cache docs in general would be great but stephenfin just refuses to support us documenting things18:52
mriedemo-)18:52
donnydLOL mriedem18:52
artomsean-k-mooney, AFAICT that calls down to https://github.com/openstack/nova/blob/master/nova/objects/pci_device.py#L37318:54
artomsean-k-mooney, which saves pci_devices, but not pci_requests18:54
artomDoes it even make sense for pci_requests to change for a live migration?18:56
sean-k-mooneywell the number and type of request wont change18:56
artomThe spec might?18:57
mriedemi would hope that pci requests don't change for a live migration18:58
mriedemotherwise we f'ed up somewhere in the data modeling18:58
sean-k-mooneywhat do you mean18:58
mriedempci devices are the per-compute/instance thing that rack inventory and what's allocated18:58
mriedemthe pci requests should just be the host agnostic request18:58
sean-k-mooneyya they more or less are18:58
sean-k-mooneythe can have an alisa18:58
sean-k-mooneyor my have tags like a neutorn physnet18:59
melwittdonnyd, mriedem: I just asked penick and he said they've only done image cache warming once or twice on an ad hoc basis, for example for a 40G windows image that was needed on certain hypervisors. he said even still, downloads don't take too long (couple minutes at most) bc they always have their glance in the same datacenter. he also said they disable image checksums to avoid that slowness18:59
sean-k-mooneyor device type e.g. type-VF18:59
artomsean-k-mooney, right, but that can't change during a live migration, right?18:59
sean-k-mooneybut the request does not have any info about the specifc device18:59
sean-k-mooneyartom: correct it cant18:59
*** eharney has joined #openstack-nova19:00
donnydmelwitt: My downloads are in the same rack on a 10G network (compute side) and they are painfully slow for the gear that underpins them19:00
melwittdonnyd: I didn't notice if you mentioned how long is a long time in your scenario. is it a few minutes or like 20 minutes19:00
artomSo we seem fairly confident pci_requests can't change during a live migration. So back to our original problem, it should be find for a Claim to write them back to the instance19:02
artomIn the sense that, the SRIOV live migration code won't have changed them19:03
donnydwell the drive glance is hosted on is an nvme device with 4G/s in read speed and the machine its on has 40G networking... so I expected to get somewhere around 1/4 of that or about 1G/s (ish) in download speed19:03
dansmithright, which was my original thing.. why not just keep them?19:03
donnydthe reality is 100M/s19:03
artomdansmith, I was worried about them changing under us19:03
artomSo by keeping them we'd end up clobbering the new ones set by the SRIOV live migration code19:04
donnydIf I could speed up the downloading part, i wouldn't even notice19:04
artomWhich is why I wanted to strongly to avoid touching them altogether19:04
donnydbecause a new image would be downloaded in 30-40 seconds19:04
dansmithartom: we stash a copy of the requests to be applied int he migration context with other things.. if the sriov migration code is not playing nice with that, then it's wrong19:05
sean-k-mooneyartom: we wont clobber anything19:05
sean-k-mooneybut we will end up trying to claim pci device twice19:05
artomsean-k-mooney, well no, dansmith's point was that we still skip the actual claiming (right?)19:05
artomJust don't mess with any DB stuff19:05
sean-k-mooneyand then whe might not move them form claimed to allocated correctly and leak pci deivces19:05
dansmithartom: I don't think I made that claim19:06
sean-k-mooneye.g. once that are claimed for the instance but not allocated to it19:06
dansmithartom: I might have asked that19:06
mriedem"I don't think I made that claim"19:06
* mriedem cues rimshot19:06
artomWe need to stop overloading "claim" >_<19:06
dansmithI still don't understand why this needs to be different for cold and live migration, with respect to the accounting19:06
melwittdonnyd: right... that does sound strange, but I'm definitely not that knowledgable about what is reasonable to expect there. I'll run those details by penick and find out if it rings a bell19:06
sean-k-mooneyclaimed and allocated are states in the pci resouce tracker.19:06
sean-k-mooneyand claimes generally refers to the RT19:07
*** markvoelker has quit IRC19:07
artomdansmith, it doesn't :) But SRIOV live migration was implemented without it, so now we're in this mess19:07
sean-k-mooneywe have moved to using allocaiton to refer to placmeent19:07
dansmithartom: yeah, sounds like that is the real problem here19:07
donnydI think the last time we dug into it, it had something to do with the requests library only being able to use one core, which in turn make sense because my controllers cores aren't super fast19:08
sean-k-mooneyi can proably make the sriov stuff work with move claims. but i would prefer to keep move claimes for cold migration19:08
sean-k-mooneysriov migration did not use claims because we did not need to and live migration never used them before19:08
*** markvoelker has joined #openstack-nova19:08
mriedemso something was bolted on instead and now we have a mess19:09
donnydI thought the http.store option in glance would allow compute to grab from a http server (like apache or something), but i have no idea how to configure it19:09
mriedemis the summary yeah?19:09
dansmithrightm19:09
dansmiththat's the problem19:09
dansmith"live migration is different so I can be more different"19:09
artommriedem, I think sean-k-mooney would object to the "bolted on" wording, but yeah19:10
sean-k-mooneywhen we were first proposing sirov migration we were not planning to use move claime for numa migration19:10
artomIt's also on me for not having reviewed that spec/code19:10
sean-k-mooneyi am testing your code right now by the way19:11
artomCould have said "hey let's use claims since we'll need them for NUMA LM anyways"19:11
sean-k-mooneywell we dont you could alos do the calimes the way we do but anyway should i start looking at how to convert the sriov code19:11
dansmithwhich cores were reviewing that?19:12
dansmithI don't think it was me,19:12
mriedemjay and stephen19:12
dansmithokay19:12
melwittdonnyd: the last comment confuses me because compute downloads images over glance API (http), so what is the difference between that and the http.store option you mention, I wonder?19:12
dansmithSeems like jay was actually in favor of managing the pci stuff like the rest of it: https://review.opendev.org/#/c/620115/19:15
dansmith"If we managed PCI devices using the same system we do for CPU, memory and local disk, then we could do the resource *claim* in the scheduler's claim_resources() "19:16
dansmithactually,19:16
dansmithmaybe he means more placement-like by that19:16
sean-k-mooneyyes19:16
sean-k-mooneyhe wanted to avoid the move cliams becuae he wante use to use placment as much as possible19:17
dansmithI don't really see anywhere that he said that, but I don't doubt it19:17
artomI really didn't want to start a witch hunt :(19:19
sean-k-mooneythat was my preference too by the way so im biased in recolection but i think if we need too i can adapt the sriov code to get the device form the move claim if needed19:19
dansmithartom: no, just looking for context19:20
artomChoices were made, I'm sure at the time with the available info they were optimal19:20
artomdansmith, ack19:20
dansmithartom: this is the problem with bolting on incremental "okay we'll justsupport live migration with pci if they're like this" kind of features19:20
dansmithafter you do that a couple times, you end up here19:20
donnydmelwitt: well I am really just guessing because I don't understand how the http.store option works... I was hopeful that I could configure the filestore to save images in a particular location, and then have that exposed directly by apache (or something of the like) https://opendev.org/openstack/glance_store/src/branch/master/glance_store/_drivers/http.py19:20
sean-k-mooneywell we evenually wanted to use the multiple port binding workflow for could migration and then move the sriov port claiming to always use the live migration flow when we did that19:21
* dansmith goes for a shearing19:22
artomdansmith, isn't there a "law" about that?19:22
mriedemooo i'm scheduled for that tomorrow19:22
mriedemwe're sympatico19:22
artomProducts reproduce the communication structures of their organisations, or something like that?19:22
donnydmelwitt: but i honestly don't know because I have never had to get that far into glance... in my normal uses cases... it just works19:22
artomSo us, being distributed, loose and non-cohesive, will produce features that are the same?19:22
artomAha, https://en.wikipedia.org/wiki/Conway%27s_law19:23
mriedem"organizations that are a clusterfuck will produce software that is a clusterfuck"19:24
mriedemgot it19:24
artomI mean, yes. :P19:24
sean-k-mooneyfyi i just migrated an instace with a numa toployg and an sriov macvtap port19:25
melwittdonnyd: understood. I'm grepping around and also can't find the config option in glance about how to disable checksum verification for the image_cache. I'll need to chat with penick and get back to you. I'll ask him if he knows anything about http.store while I'm at it19:25
sean-k-mooneyi need to now look at the xml and see what happend19:25
sean-k-mooneybut it succeded19:25
donnydmelwitt: much appreciate... or just any possible way to make it faster would be a massive help19:26
mriedemmelwitt: that stuff should be disabled by default in nova19:26
mriedemhttps://docs.openstack.org/nova/latest/configuration/config.html#glance.verify_glance_signatures19:26
melwittmriedem: yeah, he said it's different than the signature verify19:26
melwittand I just can't find it anywhere19:26
mriedemis this based on his ocata cloud?19:26
melwittit might be19:27
mriedemhttps://docs.openstack.org/nova/pike/configuration/config.html#libvirt.checksum_base_images19:27
mriedemthat stuff has been removed19:27
melwittoh, dangit19:27
mriedemdagnabbit even19:27
melwittyeah, for sure19:28
melwittthere goes that lead :(19:28
sean-k-mooneyartom: the xml looks correct and it was updated19:28
donnydwhat would be real slick is to just mount the same share for glance filestore backend and nova image cache19:29
mriedemmake way19:29
donnydthat way once the image is uploaded the hypervisor would have immediate access19:29
openstackgerritMatt Riedemann proposed openstack/nova master: Add nova.compute.utils.delete_image  https://review.opendev.org/63760519:30
openstackgerritMatt Riedemann proposed openstack/nova master: Refactor ComputeManager.remove_volume_connection  https://review.opendev.org/64218319:30
openstackgerritMatt Riedemann proposed openstack/nova master: Add power_on kwarg to ComputeDriver.spawn() method  https://review.opendev.org/64259019:30
openstackgerritMatt Riedemann proposed openstack/nova master: Add Destination.allow_cross_cell_move field  https://review.opendev.org/61403519:30
openstackgerritMatt Riedemann proposed openstack/nova master: Change HostManager to allow scheduling to other cells  https://review.opendev.org/61403719:30
openstackgerritMatt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_dest compute method  https://review.opendev.org/63329319:30
openstackgerritMatt Riedemann proposed openstack/nova master: Add PrepResizeAtDestTask  https://review.opendev.org/62789019:30
openstackgerritMatt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_source compute method  https://review.opendev.org/63483219:30
openstackgerritMatt Riedemann proposed openstack/nova master: Add PrepResizeAtSourceTask  https://review.opendev.org/62789119:30
openstackgerritMatt Riedemann proposed openstack/nova master: Add finish_snapshot_based_resize_at_dest compute method  https://review.opendev.org/63508019:30
openstackgerritMatt Riedemann proposed openstack/nova master: Add FinishResizeAtDestTask  https://review.opendev.org/63564619:30
openstackgerritMatt Riedemann proposed openstack/nova master: Execute CrossCellMigrationTask from MigrationTask  https://review.opendev.org/63566819:30
openstackgerritMatt Riedemann proposed openstack/nova master: Plumb allow_cross_cell_resize into compute API resize()  https://review.opendev.org/63568419:30
openstackgerritMatt Riedemann proposed openstack/nova master: Filter duplicates from compute API get_migrations_sorted()  https://review.opendev.org/63622419:30
openstackgerritMatt Riedemann proposed openstack/nova master: Start functional testing for cross-cell resize  https://review.opendev.org/63625319:30
openstackgerritMatt Riedemann proposed openstack/nova master: Handle target host cross-cell cold migration in conductor  https://review.opendev.org/64259119:30
openstackgerritMatt Riedemann proposed openstack/nova master: Validate image/create during cross-cell resize functional testing  https://review.opendev.org/64259219:30
openstackgerritMatt Riedemann proposed openstack/nova master: Add zones wrinkle to TestMultiCellMigrate  https://review.opendev.org/64345019:30
mriedemgibi: stephenfin: dansmith: ^ i've rebased and moved the already +2'ed and less controversial things to the bottom to help move those through19:31
sean-k-mooneyartom: http://paste.openstack.org/show/765671/19:32
donnydmriedem: you are supposed to wait for Friday afternoons for that kind of action.. so my CI has something to do... it gets lonely and bored19:32
sean-k-mooneythat the xml before and after the migration19:32
mriedemdonnyd: i wanted to test your CI's image download capabilities19:33
*** itlinux has quit IRC19:33
donnydLOL - thanks19:33
*** itlinux has joined #openstack-nova19:33
artomsean-k-mooney, I think the best thing would be to write up a comprehensive test report19:35
artomI'm fairly confident that the XML part works, that's been there since Stein19:35
artomIt's more the new surprises - pci_requests from the migration_context, etc19:35
artomIt doesn't help that the path series keeps changing under you19:36
artomBut yeah19:36
artomI think testing by a more or less independent third party would be awesome :)19:36
artomI need to drop for a bit while I reconnect to phone's hotspot19:36
artombrb19:36
*** artom has quit IRC19:36
sean-k-mooneywell im not sure i qualify in that regard but ill test it none the less19:37
sean-k-mooneythe migration context is here http://paste.openstack.org/show/765743/19:42
sean-k-mooneyand the pci request in the instance extra table are empty19:42
*** itlinux has quit IRC19:42
*** artom has joined #openstack-nova19:43
sean-k-mooneymysql> select pci_requests from  instance_extra as ie where ie.instance_uuid='b304fedf-aa31-4990-9a74-29729eed336d';19:43
sean-k-mooney+--------------+19:43
sean-k-mooney| pci_requests |19:43
sean-k-mooney+--------------+19:43
sean-k-mooney| []           |19:43
sean-k-mooney+--------------+19:43
artomYey :(19:44
artomAt least we know we weren't wrong to worry19:44
sean-k-mooneythe pci device will will be claimed with the instance uuid in the pci tracker19:45
sean-k-mooneybut its not correct.19:45
artomYeah, but won't that screw up future operations?19:45
artomSuddenly you can evacuate to a host with no SRIOV, for example19:45
sean-k-mooneyi would have to do more testing but non nessiarally19:46
artomStill seems dangerous19:47
artomI mean, I want NUMA LM to land a *lot*, but there's a minimum level of quality here ;)19:47
artomAnd that's not part of it19:47
sean-k-mooneyya i know. the request_spec still has the orginal pci_request in it and we do not support pci hot attach19:49
sean-k-mooneythat is the request spec http://paste.openstack.org/show/765824/19:49
openstackgerritMatt Riedemann proposed openstack/nova master: FUP for I66d8f06f19c5c631e33208580428aa843abb38d2  https://review.opendev.org/67895119:49
sean-k-mooneystephenfin: by the way i think we need to apply https://review.opendev.org/#/c/675776/19:52
sean-k-mooneyi am seeing some other issue that that will likely fix19:52
sean-k-mooneyartom: with the same code if i use a non numa flavor the pci reuests are not removed19:54
sean-k-mooneyartom: we might be able to use the temporay mutation thing to ensure that the change is not saved19:55
*** jmlowe has joined #openstack-nova19:56
artomsean-k-mooney, errr...19:56
artomWe want to save the *other* stuff19:56
*** nweinber has quit IRC19:56
mriedemsean-k-mooney: so definite -1 here right? https://review.opendev.org/#/c/635669/39/nova/compute/resource_tracker.py@30719:59
mriedemcan someone do that?19:59
sean-k-mooneymriedem: yes19:59
sean-k-mooneycommeted and -1'd20:02
artomSheesh, rub it in, will'ya20:06
mriedemi can run a -2 into it if that helps20:07
mriedem*rub20:07
openstackgerritMatt Riedemann proposed openstack/nova master: Update help for image_cache_manager_interval option  https://review.opendev.org/67895420:07
sean-k-mooneyrubbing salt in a wound helps it heal :P20:07
artommriedem, no, rubbing a -2 does something entirely different20:07
*** prometheanfire has quit IRC20:08
*** prometheanfire has joined #openstack-nova20:09
*** dpawlik has joined #openstack-nova20:09
*** weshay is now known as weshay_pto20:11
sean-k-mooneyin other new https://bugs.launchpad.net/nova/+bug/1804502 is a pain20:12
openstackLaunchpad bug 1804502 in OpenStack Compute (nova) "Rebuild server with NUMATopologyFilter enabled fails (in some cases)" [Undecided,In progress] - Assigned to David Hill (david-hill-ubisoft)20:12
sean-k-mooneyand yes i know its a feature not a bug but i think i need to go figure out how to make that work for U ...20:12
mriedemcfriesen has a duplicate of that i think20:12
sean-k-mooneyya its a long knonw issue20:13
sean-k-mooneynone of the filters are really aware of rebuild/resize20:13
sean-k-mooneyand they all reate the request as a new spawn20:13
*** dpawlik has quit IRC20:13
sean-k-mooneyso we need double the capasity yada yada yada20:13
sean-k-mooneyits fixable it just a pain20:14
sean-k-mooneyand we have had more important things like moveing this stuff to placment20:14
mriedemthe filters are definitely aware of rebuild now20:14
sean-k-mooneynot all of them20:14
sean-k-mooneyor rahter they have to be coded to be aware20:15
mriedemcorrect, https://github.com/openstack/nova/blob/master/nova/scheduler/filters/numa_topology_filter.py#L2620:15
sean-k-mooneywe now at least go back to the schduler when the image changes20:15
dansmithartom: even if there's technically nothing that breaks because we nuke pci_requests, I still don't think it makes sense to land it in a state where we're clobbering data, which I think is what you're getting at20:15
*** bbowen has quit IRC20:15
sean-k-mooney yes it needs to run on rebuild but it does not have special handeling for rebuild20:16
mriedemdansmith: i woudln't be surprised if a secondary operatoin on the instance after the clobbered instance live migratoin blows up20:17
dansmithmriedem: right, I said that earlier20:17
mriedemhave seen all kinds of fun stuff with things like that and accidentally persisting crap to the request spec20:17
dansmithmriedem: I just didn't go looking for it20:17
sean-k-mooneyi can test that if ye like20:17
sean-k-mooneyi still have it20:17
dansmitheven if we can't poke something, it's still not okay to clobber them I think20:17
sean-k-mooneybut we obviously should still fix it20:18
dansmithanything on the compute or cell conductor that needs pci requests wouldn't be able to get it from the reqspec20:18
dansmithand as it is with pci, it's all very untested, so it'd be easy to miss something we're breaking20:18
sean-k-mooneyyes that is true20:18
mriedemi'm not even sure what you'd test after the clobbered live migration to show it fails, or how we'd recreate that in a functional test to prevent future regressions, but i know that having sean manually test all these weird things isn't sustainable20:19
dansmithobviously20:20
mriedemi guess you could like cold migrate back to the source host and verify there are any claimed pci devices for the instance20:20
sean-k-mooneythe pci device are still tracked in the pci tacker as used by the instace20:21
mriedemoh great20:21
sean-k-mooneyactully i should check that20:21
mriedemdidn't we just talk about something like this...vpmem?20:21
sean-k-mooneykind of20:21
*** artom has quit IRC20:23
sean-k-mooneyyes just check we are correctly tracking thing at the host level in the pci tracker at least20:23
*** gbarros has joined #openstack-nova20:24
*** artom has joined #openstack-nova20:25
donnydmriedem: melwitt I have the image cache dir all setup and running20:25
*** tbachman has quit IRC20:25
donnydwill see tomorrow if it fixes the issue20:25
zigoI got 3 unit test failures in Nova when building Stein in sid, because the tests are expecting from libvirt:20:29
zigo<target bus="virtio" dev="vda"/>20:29
zigowhen libvirt really returns:20:29
zigo<target dev="vda" bus="virtio"/>20:29
zigoHas this been fixed in master?20:29
zigomriedem: ^20:30
*** gbarros has quit IRC20:31
mriedemzigo: no idea, but it's likely some test using dicts and not handling sort order on the keys20:32
*** gbarros has joined #openstack-nova20:32
zigoExactly.20:32
mriedemopen a bug with the test traceback20:32
zigoSure.20:32
zigomriedem: Still in Launchpad, or in Storyboard?20:32
zigoLaunchpad, it seems.20:33
sean-k-mooneyif we cold migrate after live migration it failts with a port update error20:33
sean-k-mooneyError: Failed to perform requested operation on instance "sriov-macvtap", the instance has an error status: Please try again later [Error: Port update failed for port 485937a6-7611-4948-8420-4bfd73e15ea8: Unable to correlate PCI slot 0000:01:10.1].20:34
*** xek has quit IRC20:34
zigomriedem: https://bugs.launchpad.net/nova/+bug/184166720:36
openstackLaunchpad bug 1841667 in OpenStack Compute (nova) "failing libvirt tests: need ordering" [Undecided,New]20:36
zigoFeel free to rewrite the title if you find it not good enough.20:36
*** damien_r has joined #openstack-nova20:36
sean-k-mooneywell that is unexpected..20:38
sean-k-mooneyit errored after seting the host to the other host e.g. when migrating form cloud-4->cloud-3 it was errord on cloud-3 but libivrt did not have a domain on cloud-320:41
sean-k-mooneyit still had a domian on cloud-4 but was shutoff20:41
sean-k-mooneyafter a hardreboot it went form error to running on cloud-320:41
*** damien_r has quit IRC20:41
*** gbarros has quit IRC20:44
mriedemzigo: what version of lxml is being used?20:46
mriedemprometheanfire: efried: wasn't there a recent issue with newer lxml and nova unit tests?20:47
sean-k-mooneyya i delete that after the hard reboot it was running on cloud-3 with the device claimed on cloud-4 so that is not good.20:47
sean-k-mooneyim going to unstack and call it a night.20:47
efriedmriedem: yes, sean-k-mooney was looking into it.20:47
sean-k-mooneymriedem: ya im ment to be looking into that20:47
efriedhad to do with attribute ordering.20:47
sean-k-mooneyit was what i had planned to do today...20:47
mriedemyup https://bugs.launchpad.net/nova/+bug/184166720:47
openstackLaunchpad bug 1841667 in OpenStack Compute (nova) stein "failing libvirt tests: need ordering" [Undecided,New]20:47
zigomriedem: 4.4.1.20:48
mriedemefried: sean-k-mooney: did we have a bug for it?20:48
zigoThat's kind of new in Sid...20:48
sean-k-mooneythere was one filed yes20:48
sean-k-mooneymriedem: prometheanfire filed it i think20:48
*** brault has quit IRC20:48
mriedemhttps://bugs.launchpad.net/nova/+bug/183866620:48
openstackLaunchpad bug 1838666 in OpenStack Compute (nova) "lxml 4.4.0 causes failed tests in nova" [Undecided,Confirmed]20:48
mriedemgot it20:48
zigomriedem: Uploaded 10 days ago: https://tracker.debian.org/pkg/lxml20:49
*** lpetrut has joined #openstack-nova20:50
*** lpetrut has quit IRC20:51
*** lpetrut has joined #openstack-nova20:51
zigoWow, with my brand new laptop, it only takes me 5 minutes to run all Nova unit tests now! :)20:51
mriedemsounds like a challenge20:51
sean-k-mooneyya its nice when you can throw more cores at it and it scales well20:53
openstackgerritEric Fried proposed openstack/nova master: Raise when inventory retrieval & refresh fails  https://review.opendev.org/67895720:53
openstackgerritEric Fried proposed openstack/nova master: Remove @safe_connect from _get_provider_aggregates  https://review.opendev.org/67895820:53
openstackgerritEric Fried proposed openstack/nova master: Remove @safe_connect from _get_sharing_providers  https://review.opendev.org/67895920:53
openstackgerritEric Fried proposed openstack/nova master: Invalidate cache when _refresh_associations fails  https://review.opendev.org/67896020:53
efriedmriedem: This ^ may be sufficient for bug 1841481, assuming "the next refresh" is quick enough for us to repopulate the cache. Otherwise we may wish to wrap _refresh_associations in a one-loop @retry (but not on ClientException). Let me know your thoughts if you please.20:53
openstackbug 1841481 in OpenStack Compute (nova) "Race during ironic re-balance corrupts local RT ProviderTree and compute_nodes cache" [Medium,Triaged] https://launchpad.net/bugs/1841481 - Assigned to Eric Fried (efried)20:53
* efried ==> school runs20:53
zigosean-k-mooney: Got a 6 core Xeon (so 12 threads...)...20:55
sean-k-mooneyavoids the whos system is faster trap.20:56
*** lpetrut has quit IRC20:58
zigoThe nice thing is that I do have enough NVME space to have a full Debian mirror ...20:59
zigodeb file:///home/ftp/debian ... :P20:59
sean-k-mooneythats like somewhere betten 230-400GB right21:00
sean-k-mooneyi used to use apt-cacheng to cache the things i actully was using21:00
sean-k-mooneyi should set that up again.21:00
zigosean-k-mooney: 463 GB right now, only amd64 + sources, all available suites.21:00
zigoThe point is, it directly fetches on drive, avoiding network socket indirections...21:01
sean-k-mooneyright we had a ubunut mirror for the intel nfv ci and i rememebr it was somere in that range21:01
zigosean-k-mooney: I used approx.21:01
sean-k-mooneybut that was like 3 years ago21:01
sean-k-mooneyso i assuemd it grew21:01
mriedemmelwitt: time for you to bug dansmith https://review.opendev.org/#/c/507486/6921:03
sean-k-mooneyprometheanfire: do you know if the lxml thing only happins on a specific python verions21:03
sean-k-mooneyim running the test uder py36 at the momemt21:03
mriedemzigo: are you testing on py27 or py36?21:04
sean-k-mooneyi can test both but py36 is faster for me to test with21:04
sean-k-mooneyi have 4 test failrue on py3621:04
sean-k-mooneyall of which are not using our xml matcher21:06
sean-k-mooneythere doing self.assertEqual(diska_xml.strip(), actual_diska_xml.strip())21:06
sean-k-mooneyor assert called with21:06
*** spsurya has quit IRC21:07
*** itlinux has joined #openstack-nova21:07
*** damien_r has joined #openstack-nova21:07
*** damien_r has quit IRC21:07
prometheanfiresean-k-mooney: I don't know :(21:08
sean-k-mooneyit broke on py36 so i can reporduce21:08
sean-k-mooneyso its fine21:08
zigomriedem: python 3.721:08
zigosean-k-mooney: py3721:09
sean-k-mooneyit likely not python specfic21:09
zigoRight.21:09
sean-k-mooneyactully at least one of the tests fail on py3 and pass on py2721:11
sean-k-mooneyso maybe it is but ill fix it for both in anycase21:11
*** itlinux has quit IRC21:13
donnydso here is some interesting information about the image cache21:14
donnydI keep getting this error thrown21:14
donnydhttps://www.irccloud.com/pastebin/9eB2NYg4/21:14
sean-k-mooneyya i have seen that21:15
donnydmost notably this nova.exception.DiskNotFound: No disk at /var/lib/nova/instances/_base/1675f8d33da6cfdd6354d2078b61c3c11ff417d021:15
donnydhowever that image is there, and is continually downloaded over and over21:15
sean-k-mooneyi think mdbooth was fixing a cese were if we fail to resize an image we delete the base image backing file in the cache21:15
donnydhttps://www.irccloud.com/pastebin/yllMTx02/21:16
donnydits not a resize21:17
donnyd-rw-r--r--  1 libvirt-qemu kvm  7054688256 Aug 27 21:13 1675f8d33da6cfdd6354d2078b61c3c11ff417d021:17
donnyd-rw-r--r--  1 nova         nova          0 Aug 27 21:17 1675f8d33da6cfdd6354d2078b61c3c11ff417d0.part21:17
mriedemwonder if there is an access issue?21:17
mriedemthe DiskNotFound might just be a shitty generic exception that is thrown21:18
mriedem*raised21:18
donnydwell its owned by nova till download and then changed to libvirtd-qemu21:18
*** markvoelker has quit IRC21:19
mriedemraising from here though so not an access issue https://github.com/openstack/nova/blob/master/nova/virt/images.py#L5821:19
donnydIf I mount the whole directory for nova/instances this error goes away21:20
sean-k-mooneydonnyd: normally nova is in the libvirt group21:21
*** gbarros has joined #openstack-nova21:21
sean-k-mooneyso nova should be able to read and write to image owned by nova21:22
donnydWell there isnt an issue if I purge the cache, it seems to download and then run with it fine21:22
sean-k-mooney*libvirt/qemu21:22
donnydBut each hypervisor that tries to read this image throws that error and then redownloads21:23
*** bbowen has joined #openstack-nova21:24
*** ivve has quit IRC21:25
sean-k-mooneythe way the xml match works is dumb/ not documented21:27
sean-k-mooneywhy do retrun nothing mean it matched21:29
sean-k-mooneynow that i know that i can actully use it21:30
sean-k-mooneyoh this comes from testtools...21:30
donnydwell its reporting back exactly what qemu-img is telling it21:32
sean-k-mooneydonnyd: what os are you on21:32
donnyd sudo qemu-img info /var/lib/nova/instances/_base/1675f8d33da6cfdd6354d2078b61c3c11ff417d021:33
donnydqemu-img: Could not open '/var/lib/nova/instances/_base/1675f8d33da6cfdd6354d2078b61c3c11ff417d0'21:33
donnydubuntu 18.0421:33
sean-k-mooneycan you check if apparmor or systmed is blockign it21:33
sean-k-mooneye.g. check dmesg21:33
donnydpossible...21:34
donnydnot sure what i am looking for apparmor to say21:35
sean-k-mooneysomething like this21:36
sean-k-mooneyaudit: type=1400 audit(1565883657.055:70): apparmor="ALLOWED" operation="open" profile="libreoffice-soffice" name="/proc/31189/comm" pid=31189 comm="soffice.bin" requested_mask="r" denied_mask="r" fsuid=1000 ouid=100021:36
sean-k-mooneyexceet you are looking for apparmor="DENIED"21:36
donnydwhen i grep fro DENIED i return nothing21:37
sean-k-mooneyits proably not an apparmor thing but it could be if the file permession allow you to read the file as root21:37
openstackgerritArtom Lifshitz proposed openstack/nova master: Introduce live_migration_claim()  https://review.opendev.org/63566921:38
openstackgerritArtom Lifshitz proposed openstack/nova master: New objects for NUMA live migration  https://review.opendev.org/63482721:38
openstackgerritArtom Lifshitz proposed openstack/nova master: LM: Use Claims to update numa-related XML on the source  https://review.opendev.org/63522921:38
openstackgerritArtom Lifshitz proposed openstack/nova master: NUMA live migration support  https://review.opendev.org/63460621:38
openstackgerritArtom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration  https://review.opendev.org/64002121:38
openstackgerritArtom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration  https://review.opendev.org/67259521:38
openstackgerritArtom Lifshitz proposed openstack/nova master: DNM: Run LM integration tests with NUMA flavor  https://review.opendev.org/67888721:38
mriedemsean-k-mooney: https://blueprints.launchpad.net/nova/+spec/image-metadata-prefiltering is now in a runway21:46
sean-k-mooneymriedem: ok21:47
*** mriedem has quit IRC21:48
sean-k-mooneythe nova-next failure in the middle patch looks unrelated so i just rechecked it21:49
sean-k-mooneythe rest are clean21:49
sean-k-mooneyif you see anything ill adress it tomorow21:50
*** trident has quit IRC22:05
*** gbarros has quit IRC22:08
*** dpawlik has joined #openstack-nova22:10
sean-k-mooneyprometheanfire: i think i have fixed the lxml issue locally22:13
*** trident has joined #openstack-nova22:13
*** dpawlik has quit IRC22:15
*** rcernin has joined #openstack-nova22:15
sean-k-mooneyfungi: ya that looks fine22:15
sean-k-mooney* cve description22:15
*** markvoelker has joined #openstack-nova22:15
fungithanks again sean-k-mooney!22:16
*** markvoelker has quit IRC22:20
openstackgerritsean mooney proposed openstack/nova master: fix lxml compatiblity issues  https://review.opendev.org/67896422:25
sean-k-mooneyzigo: efried prometheanfire ^22:25
sean-k-mooneyi think that should fix it22:25
zigosean-k-mooney: Nice ! :)22:27
zigoI've already blacklisted the unit tests in my last upload to Sid though, but if it passes, I'll add your patch instead.22:27
* zigo is currently rebuilding Ceph Nautilus for Buster.22:27
*** mlavalle has quit IRC22:29
efriedsean-k-mooney: assertXmlEqual is the one-step alias for what you're doing in there.22:29
sean-k-mooneyyep22:30
sean-k-mooneyi knew that was a thing too but blanked when writing it22:30
sean-k-mooneybut ill fix that tommrow22:30
sean-k-mooneyhave people seen "ValueError: invalid literal for int() with base 10: '12.1'"22:37
sean-k-mooneyoslo_utils imageutils22:37
sean-k-mooneyhttps://zuul.opendev.org/t/openstack/build/177a7183dd234f68886e736ca4d82cd5/log/compute/logs/screen-n-cpu.txt.gz?severity=4#136722:40
sean-k-mooneyapparently that is a thing22:41
sean-k-mooney>>> int('12.1')22:41
sean-k-mooneyTraceback (most recent call last):22:41
sean-k-mooney  File "<stdin>", line 1, in <module>22:42
sean-k-mooneyValueError: invalid literal for int() with base 10: '12.1'22:42
efriedI think it may be telling you that '12.1' is an invalid literal for int() with base 10.22:43
sean-k-mooneythis is the oslo code https://github.com/openstack/oslo.utils/blob/master/oslo_utils/imageutils.py#L10822:43
sean-k-mooneythey are trying to caset what i assume is a sting to an int22:44
sean-k-mooneyint('12') work22:44
sean-k-mooneybut it does not truncate22:45
efriedI'm guessing the caller either omitted units or forgot to convert or forgot to round.22:45
*** gbarros has joined #openstack-nova22:45
sean-k-mooneywell this is the nova call site https://github.com/openstack/nova/blob/master/nova/virt/images.py#L9522:46
sean-k-mooneyand out is the standard out from running "'env', 'LC_ALL=C', 'LANG=C', 'qemu-img', 'info', path£22:47
*** avolkov has quit IRC22:47
sean-k-mooneyso either the qemu output has changed to include decimal22:47
sean-k-mooneyor this has always been broken22:47
sean-k-mooneyi think qemu has changed its proably both22:48
sean-k-mooneythat is from a f29 job with virt preview repor so its basically the nightly build of qemu and libvirt22:48
*** brault has joined #openstack-nova22:49
sean-k-mooneyoslo does not handel that import correctly and we pass the output form qemu directly22:49
*** brault has quit IRC22:53
*** tkajinam has joined #openstack-nova23:06
*** markvoelker has joined #openstack-nova23:15
*** markvoelker has quit IRC23:20
*** prometheanfire has quit IRC23:33
*** prometheanfire has joined #openstack-nova23:33
*** macz has quit IRC23:36
*** tbachman has joined #openstack-nova23:46
*** markvoelker has joined #openstack-nova23:55
« Monday, 2019-08-26 Index Wednesday, 2019-08-28 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!