Wednesday, 2020-04-29

« Tuesday, 2020-04-28 Index Thursday, 2020-04-30 »
*** ociuhandu has joined #openstack-nova00:04
*** ociuhandu has quit IRC00:12
*** mlavalle has quit IRC00:23
*** JamesBenson has joined #openstack-nova00:25
*** JamesBenson has quit IRC00:30
openstackgerritBrin Zhang proposed openstack/nova master: Add nested resource providers limit for multi create  https://review.opendev.org/72388400:36
brinzhang_bauzas: I was updated https://review.opendev.org/723884, as you said in bug 1874664, I copied and modified.00:41
openstackbug 1874664 in OpenStack Compute (nova) "Instance multi-create doesn't support available resources spread between children RPs" [Medium,Confirmed] https://launchpad.net/bugs/1874664 - Assigned to Wenping Song (wenping1)00:41
*** threestrands has joined #openstack-nova00:45
*** Liang__ has joined #openstack-nova00:59
openstackgerritsean mooney proposed openstack/nova master: silence amqp heartbeat warning  https://review.opendev.org/72418801:04
*** abaindur_ has joined #openstack-nova01:09
*** abaindur has quit IRC01:12
*** abaindur_ has quit IRC01:14
melwittsean-k-mooney: re: that log message, I dunno. we suggested it in the past to the oslo.messaging ppl and they thought it's something that should be logged, at info. iirc they didn't want to downgrade it to debug either01:16
melwittI recognize that maybe there's a way we could hide it on our side but I guess I dunno what to think about that01:16
*** JamesBenson has joined #openstack-nova01:47
*** ociuhandu has joined #openstack-nova01:50
*** JamesBenson has quit IRC01:52
*** ociuhandu has quit IRC02:03
*** ircuser-1 has joined #openstack-nova02:05
openstackgerritxuyuanhao proposed openstack/nova master: fix bug/1875624  https://review.opendev.org/72421302:17
*** gyee has quit IRC02:52
*** mkrai has joined #openstack-nova02:55
*** sapd1_x has joined #openstack-nova03:02
*** psachin has joined #openstack-nova03:22
*** JamesBenson has joined #openstack-nova03:27
*** JamesBenson has quit IRC03:32
*** ociuhandu has joined #openstack-nova03:39
*** ircuser-1 has quit IRC03:40
*** JamesBenson has joined #openstack-nova03:48
*** JamesBenson has quit IRC03:52
*** ociuhandu has quit IRC03:53
*** threestrands has quit IRC04:17
*** JamesBenson has joined #openstack-nova04:29
*** mkrai has quit IRC04:31
*** mkrai has joined #openstack-nova04:32
*** JamesBenson has quit IRC04:33
*** evrardjp has quit IRC04:35
*** evrardjp has joined #openstack-nova04:35
*** ociuhandu has joined #openstack-nova04:39
*** ociuhandu has quit IRC04:48
*** ratailor has joined #openstack-nova05:00
*** dklyle has quit IRC05:02
*** gryf has joined #openstack-nova05:07
*** bnemec has quit IRC05:15
*** slaweq has joined #openstack-nova05:19
*** tetsuro has joined #openstack-nova05:28
*** tetsuro has quit IRC05:32
*** mkrai_ has joined #openstack-nova05:37
*** mkrai has quit IRC05:39
*** links has joined #openstack-nova05:40
*** mkrai_ has quit IRC05:51
*** mkrai has joined #openstack-nova05:51
*** mkrai has quit IRC05:53
*** mkrai has joined #openstack-nova05:53
*** ociuhandu has joined #openstack-nova06:01
*** slaweq has quit IRC06:06
*** udesale has joined #openstack-nova06:10
*** ociuhandu has quit IRC06:15
*** maciejjozefczyk has joined #openstack-nova06:28
*** rpittau|afk is now known as rpittau06:32
bauzasgibi: on PTO this morning only FYI06:35
*** ociuhandu has joined #openstack-nova06:36
*** slaweq has joined #openstack-nova06:41
gibibauzas: hi. ack.06:48
*** ttsiouts has joined #openstack-nova06:51
*** brinzhang has quit IRC06:53
*** dustinc has quit IRC06:53
*** brinzhang has joined #openstack-nova06:53
*** brinzhang has quit IRC06:56
*** brinzhang has joined #openstack-nova06:56
*** belmoreira has joined #openstack-nova06:57
*** ociuhandu has quit IRC06:59
*** dpawlik has joined #openstack-nova06:59
*** bbowen has quit IRC07:00
*** bbowen has joined #openstack-nova07:00
*** larainema has joined #openstack-nova07:02
*** links has quit IRC07:03
*** links has joined #openstack-nova07:04
*** nightmare_unreal has joined #openstack-nova07:06
*** ircuser-1 has joined #openstack-nova07:10
*** tesseract has joined #openstack-nova07:11
*** brinzhang has quit IRC07:13
*** brinzhang has joined #openstack-nova07:13
*** takamatsu has joined #openstack-nova07:16
*** mkrai has quit IRC07:26
*** mkrai has joined #openstack-nova07:27
openstackgerritKevin Zhao proposed openstack/nova master: [WIP] CI: add tempest-integrated-compute-aarch64 job  https://review.opendev.org/71443907:27
*** rcernin has quit IRC07:27
*** tosky has joined #openstack-nova07:29
*** ociuhandu has joined #openstack-nova07:29
*** mkrai has quit IRC07:42
*** mkrai has joined #openstack-nova07:45
*** ralonsoh has joined #openstack-nova07:48
openstackgerritJiri Suchomel proposed openstack/nova-specs master: Add spec for downloading images via RBD  https://review.opendev.org/57280507:58
*** mkrai has quit IRC08:01
*** mkrai_ has joined #openstack-nova08:01
*** ccamacho has joined #openstack-nova08:05
*** martinkennelly has joined #openstack-nova08:18
brinzhang_gibi: hi, https://review.opendev.org/#/c/723884/2/api-guide/source/accelerator-support.rst@69 what do you mean?08:20
brinzhang_gibi: Or other words, I seems not understand this sentence "I would make this cyborg specific in this doc.", and Line 7608:22
*** avolkov has joined #openstack-nova08:33
*** jaosorior has quit IRC08:45
*** ociuhandu has quit IRC08:50
*** ttsiouts has quit IRC08:50
*** ociuhandu has joined #openstack-nova08:50
*** ociuhandu has quit IRC08:50
*** ttsiouts has joined #openstack-nova08:51
brinzhang_gibi: I think I get your comment, will be update08:51
*** ociuhandu has joined #openstack-nova08:53
openstackgerritJiri Suchomel proposed openstack/nova-specs master: Add spec for downloading images via RBD  https://review.opendev.org/57280508:53
*** mkrai has joined #openstack-nova09:07
*** artom has quit IRC09:08
openstackgerritxuyuanhao proposed openstack/nova master: the vms can not be force deleted when vm_status is soft-delete and task-state=deleting  https://review.opendev.org/72426009:08
*** mkrai_ has quit IRC09:08
*** artom has joined #openstack-nova09:08
*** mkrai has quit IRC09:15
*** mkrai_ has joined #openstack-nova09:15
openstackgerritBrin Zhang proposed openstack/nova master: Add nested resource providers limit for multi create  https://review.opendev.org/72388409:17
brinzhang_gibi: updated, although the sentence is somewhat redundant, but it is easier to understand, please review again, thanks09:17
*** dtantsur|afk is now known as dtantsur09:24
*** efried has quit IRC09:27
stephenfingibi, bauzas: Could I get you folks to take a look at these patches for me, one of which has been around for a loooong time https://review.opendev.org/#/c/706013/ https://review.opendev.org/#/c/530905/09:33
*** xek has joined #openstack-nova09:40
*** ociuhandu has quit IRC09:47
*** ociuhandu has joined #openstack-nova09:48
*** ociuhandu has quit IRC09:53
*** udesale has quit IRC10:03
*** udesale has joined #openstack-nova10:03
*** brinzhang_ has quit IRC10:11
*** ociuhandu has joined #openstack-nova10:12
openstackgerritBrin Zhang proposed openstack/nova master: Add nested resource providers limit for multi create  https://review.opendev.org/72388410:15
openstackgerritStephen Finucane proposed openstack/nova master: Fix 500 error while passing 4-byte unicode data  https://review.opendev.org/40751410:16
*** Liang__ has quit IRC10:25
*** brinzhang has quit IRC10:29
*** rpittau is now known as rpittau|bbl10:55
openstackgerritMerged openstack/nova master: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available  https://review.opendev.org/68277210:59
openstackgerritMerged openstack/nova master: Feature matrix: update AArch64 information  https://review.opendev.org/71597910:59
*** ociuhandu has quit IRC11:06
*** ociuhandu has joined #openstack-nova11:07
*** ociuhandu has quit IRC11:13
*** tetsuro has joined #openstack-nova11:16
*** ttsiouts has quit IRC11:25
*** ttsiouts has joined #openstack-nova11:27
openstackgerritjayaditya gupta proposed openstack/nova master: Support for --force flag for nova-manage placement heal_allocations command use this flag to forcefully call heal allocation for a specific instance  https://review.opendev.org/71539511:28
*** mkrai_ has quit IRC11:29
*** mkrai has joined #openstack-nova11:29
*** derekh has joined #openstack-nova11:30
*** raildo has joined #openstack-nova11:39
*** JamesBenson has joined #openstack-nova11:40
*** ttsiouts has quit IRC11:43
*** raildo has quit IRC11:49
*** raildo has joined #openstack-nova11:50
*** tetsuro has quit IRC11:52
*** tetsuro has joined #openstack-nova11:53
*** tetsuro has quit IRC11:54
*** martinkennelly has quit IRC11:59
*** sapd1_x has quit IRC12:07
*** ociuhandu has joined #openstack-nova12:08
*** ttsiouts has joined #openstack-nova12:16
*** ttsiouts has quit IRC12:21
gibistephenfin: did a review on both12:22
stephenfingibi++ ta12:23
*** derekh has quit IRC12:26
* bauzas is back around12:29
bauzas(from PTO morning)12:29
*** martinkennelly has joined #openstack-nova12:31
*** rpittau|bbl is now known as rpittau12:31
gibigmann: I left feedback on https://review.opendev.org/#/c/72364512:36
*** ociuhandu has quit IRC12:44
*** ociuhandu has joined #openstack-nova12:45
*** ociuhandu has quit IRC12:49
*** nweinber has joined #openstack-nova12:53
*** ttsiouts has joined #openstack-nova12:55
*** links has quit IRC12:59
*** ociuhandu has joined #openstack-nova12:59
*** bbobrov has joined #openstack-nova13:00
*** jsuchome has joined #openstack-nova13:01
openstackgerritJiri Suchomel proposed openstack/nova master: Add ability to download Glance images into the libvirt image cache via RBD  https://review.opendev.org/57430113:09
*** derekh has joined #openstack-nova13:12
bauzasgibi: gmann: fwiw, we need to merge https://review.opendev.org/#/c/723645/ by a RC2 :(13:15
bauzasI mean, by merging it back to Ussuri for a next RC13:16
bauzasbecause if not, nova-status for Ussuri won't check it13:16
*** ratailor has quit IRC13:16
gibibauzas: yes, we have to do that13:17
bauzasgibi: then I'm adding an ussuri-rc-potential tag to the bug13:18
gibibauzas: good point, thanks13:20
gibigmann: have you tried the new policy upgrade check in a devstack? I'm trying it but I see that the enforcer is None here https://review.opendev.org/#/c/723645/7/nova/cmd/status.py@37813:24
*** udesale_ has joined #openstack-nova13:24
*** udesale has quit IRC13:27
*** mkrai has quit IRC13:28
*** ttsiouts has quit IRC13:29
gibigmann: linked printouts in the review13:29
sean-k-mooneygibi: printouts as in paper?13:31
gibisean-k-mooney: :)13:31
gibisean-k-mooney: printouts as the stuff my debugger printed13:31
sean-k-mooneyah13:31
sean-k-mooneyi have actully done some spec reviews in paper form when i needed to compare and contrast 3 interrealted specs and ran out of monitor space on 3 monitors..13:32
sean-k-mooneyit works but its a pain and should be avoided unless you hate trees13:32
bauzasgibi: I need to work on a next devstack :)13:34
bauzasI have some hardware, I should try to use it13:34
gibisean-k-mooney: I did that last time when reading heavy telco specifications13:35
*** psachin has quit IRC13:36
*** avolkov has quit IRC13:42
artombelmoreira, hello again, have more time this morning (well, afternoon, for you)13:42
belmoreirahi artom, tell me13:42
artombelmoreira, so, stephenfin has proposed an online data migration here: https://review.opendev.org/#/c/537414/26/nova/objects/compute_node.py@533 to get rid of really old dict JSON blobs in instance_extra.numa_topology and replace them with ovo JSON blobs13:43
*** mkrai has joined #openstack-nova13:44
artomThe proposal is to use SQL string filtering, which is slow - so for deployments with a large number of instances (anything over 1000, based on what zzzeek was saying yesterday), this might be a painful migration to run13:45
artombelmoreira, CERN's probably the largest operator (that we know of), so I was wondering if you could have any input on that13:46
artomLike, maybe online data migrations taking forever isn't such a big deal?13:46
belmoreiralet me have a look13:46
artomThanks :)13:47
belmoreirafor what I see it should generate something like 'select * from compute_nodes where deleted = 0 and numa_topology like "%nova_object.name%";'13:52
belmoreirabeing "compute_nodes" table, this will be done per cell13:52
*** Liang__ has joined #openstack-nova13:53
stephenfinbelmoreira: pretty much13:53
dansmithbelmoreira: he linked you to the wrong one.. we're doing compute nodes, but we're also doing all instances in the next file13:53
stephenfinthough13:53
artombelmoreira, oh, sorry, yeah, the compute nodes one isn't the big deal, there are rarely over a thousand of those13:54
stephenfin 'select * from instance_extra where deleted = 0 and numa_topology is not null and numa_topology like "%nova_object.name%";'13:54
artomThe instances are the open question13:54
belmoreiraWe have ~200 nodes per cell. If that is the correct query (I'm expecting that sql alchemy introduces much more stuff) executing it in a cell of ~200 is fast enough (39 ms)13:54
artomstephenfin, did you find out about the filtering order? Like, if we put deleted=0 first, does it reduce the amount of instances the string filter has to work over?13:55
stephenfinartom: the docs that I read said that the SQL engine would do that optimization itself13:55
belmoreirastephenfin artom are we talking about instance_extra as well?13:56
artomstephenfin, ok, that's reassuring13:56
artombelmoreira, we are13:56
stephenfinbelmoreira: yes, both13:56
*** bnemec has joined #openstack-nova13:56
*** dklyle has joined #openstack-nova13:57
belmoreiraok, I started with the compute_node.py :)13:57
*** Liang__ is now known as LiangFang13:57
artombelmoreira, that was my bad, I linked the wrong thing13:58
*** mriedem has joined #openstack-nova13:58
belmoreiraI'm checking how much time takes the query that stephenfin mentioned13:59
openstackgerritStephen Finucane proposed openstack/nova master: nova-audit: Use cliff instead of homegrown argparse bleh  https://review.opendev.org/72433214:00
*** ttsiouts has joined #openstack-nova14:00
artomI have to say, this shows the power of operator involvement upstream14:00
dansmithartom: man, I really wish this had just been done as migrate-on-load14:00
artomUs devs get "access" to actual large deployment, and the operators get devs not being stupid ;)14:01
artomdansmith, I guess stephenfin's response to that would be "but then we'll never know for sure that we can remove the compat routines"14:01
stephenfin'zactly14:02
dansmithartom: I think we'd load this whenever we do update available resource, so it'd be migrated within one cycle14:02
artomdansmith, for all instances?14:02
dansmithand if not, we could have just added it to the preloads and forced it to happen14:02
gmannbauzas: gibi oh, i checked locally but i might need to init the policy explicitly depends on when cmd are run. let me test those scenario also. I will do after my internal meeting.14:02
dansmithartom: yes, parallelized by compute node14:03
stephenfindansmith: have we an in-tree example of this migrate-on-load pattern?14:03
dansmithI can point you at one for sure, but let me see if we have any in current master14:04
artomThe fact that belmoreira appears to have RIP'ed running that query is worrisome ;)14:04
sean-k-mooneystephenfin: one thing we could do if this was expensive is to not do it as a normal online data migration in the sense of a global change14:06
dansmithstephenfin: https://github.com/openstack/nova/blob/58aaffade9f78c5fdc6b0d26ec26b70908b7a6aa/nova/objects/migration.py#L89-L8914:06
sean-k-mooneybut instead have a nova manage command that would allow you do it per host14:06
dansmithstephenfin: we generate uuid for migration on load if not present14:06
sean-k-mooneyso you could slowly space our the migration host by host until you are done14:07
dansmithsean-k-mooney: we already have batch limits in place, so that doesn't help14:07
dansmithother than create more debt in the form of a new command14:07
belmoreiraI was running these in our DBs. Again this is per cell. Some have much more than 1000 instances but for those we don't have numa topology defined. The ones that have numa_topology defined only run batch processing so they are large instances. Meaning that we have x4 or x8 times the number of compute nodes.14:07
belmoreirain both cases queries take few ms14:08
sean-k-mooneywell ya that basically what i was thinkign of just limiting the rows it touches but if we can already do that then sure14:08
stephenfinbelmoreira++ excellent, thanks for checking that up14:08
artombelmoreira, yep, that's very helpful, thanks!14:08
belmoreiraartom stephenfin thanks for ping me on this14:09
sean-k-mooneywell there are two parts first the query to get all the instance that need to be migrated then we need to lock the db and update all recoreds14:09
sean-k-mooneyyou can actully do that in one update query14:10
sean-k-mooneyin general bug not in this case14:10
artombelmoreira, just to make sure I understand, you're saying that your numa_topology-having instances are only about 4 to 8 instances per compute host, right?14:10
gibigmann: ack, thanks14:10
dansmithbelmoreira: I'm not sure I got what you said.. you don't have a db with tons of instances with a numa topology to test this on right?14:10
sean-k-mooneyif i parsed it correctly the cells with numa instance only run large vms so they have fewer isntances then the other cells14:11
dansmithright14:12
belmoreiraartom yes14:12
*** ttsiouts has quit IRC14:12
artombelmoreira, aha, ok - well, it tells us that the sql engine (or sqlalchemy itself?) is smart and checks the more restrictive stuff first - iow having a numa_topology at all14:12
artomBut it doesn't tell us what happens if there are thousands of numa_topology-having instances... :(14:13
sean-k-mooneyto be fair i would suspect that is likly common of many deployments. e.g. most nuam instance will be large flavors and therefore there will be less of them then standard instances14:13
artomsean-k-mooney, that does seem very likely...14:13
dansmithstephenfin: without the migrate on load, any idea what I can do to tickle those instances to rewrite them?14:13
belmoreiradansmith you're right. I don't have a cell with a lot of instances. <2000 instances only14:13
dansmithbelmoreira: gotcha, thanks14:13
sean-k-mooneybelmoreira: could you do a count(*) on instance that have a numa topology so we can get a feel for the amount you have14:14
belmoreirasean-k-mooney yes, give a sec14:14
sean-k-mooneyjust wondering if its in the 100s or roughly what you were messuring14:14
sean-k-mooneyartom: i mean we can always fake it if we need too14:15
artomsean-k-mooney, fake what?14:16
sean-k-mooneya db with 1000s of numa instances14:16
artomTrue14:16
belmoreiraour typical use case:14:16
belmoreiraselect count(*) from instance_extra where deleted = 0; #71114:16
belmoreiraselect count(*) from compute_nodes nodes; #17614:16
belmoreiraselect * from instance_extra where deleted = 0 and numa_topology is not null and numa_topology like "%nova_object.name%"; #took 38.2 ms14:16
sean-k-mooneyok so 700 ish is not bad14:17
sean-k-mooneyits not 2000 but its not 1014:17
artomSo based on those number it might actually be OK to go ahead as is...14:18
sean-k-mooneyif we were to assume this was liniar then i think it would be accpetable14:18
artom*numbers14:18
dansmithso, if we just convert to migrate-on-load, we don't take any overhead for string searching at all, migrate only the instances that need it, and are assured that they have migrated all instances by the end of the cycle, right?14:19
dansmitheven with nova-manage, we can't make them run it without a blocker migration (which is also expensive)14:19
belmoreirain this case nova_topology is not defined, but may help in the analysis:14:19
belmoreiraselect count(*) from instance_extra where deleted = 0; #157314:19
belmoreiraselect count(*) from compute_nodes nodes; #8814:19
belmoreiraselect * from instance_extra where deleted = 0 and numa_topology is not null and numa_topology like "%nova_object.name%"; #took 106 ms14:19
*** tkajinam has joined #openstack-nova14:19
dansmithif we did that, stephenfin could just remove his nova-manage bits entirely, no impact for anyone that doesn't have old instances14:19
stephenfinYeah, I can do that too. Let me see if I can wrangle something up14:20
artomdansmith, the end of cycle stuff is complicated by FFUs and operators skipping releases, no?14:20
sean-k-mooneyya if we migrate on load that also works but the only realy implciation is keeping that for a relase or two for FFU14:20
dansmithartom: ah, yeah, good point...14:21
artomSo 2 or 3 cycles, I guess14:21
dansmithartom: if we had migrate-on-load for a while, then it'd be fine, but that's the problem trying to do the switch quickly14:21
sean-k-mooneywe coudl do both e.g. do the migrate on load and then wen we drop that provide a nova manange command with a blocker migration14:21
dansmithsean-k-mooney: stephenfin has been trying to get this done for a long time so we're trying not to make this a career-long arc14:22
artomdansmith, it's a thing he'll bequeath to his grandchildren14:22
sean-k-mooneyyep i know if this was a few weeks ato i woudl have suggeted doing the migrate on load in ussuri and then the blocker/migrate command in victoria14:23
stephenfin826 days14:23
*** priteau has joined #openstack-nova14:23
stephenfinoh my, I appear to have broken zuul https://review.opendev.org/#/c/724332/14:23
artomDayyyyuuuuum14:24
sean-k-mooneyhehe let me check with infra14:24
* artom screenshots14:25
sean-k-mooneyzuul is being restart14:27
sean-k-mooneyit hit an out of memory issue and they are currently trying to fix it14:28
*** dpawlik has quit IRC14:28
sean-k-mooneyso hold rechecks for a few minutes while they sort this out14:29
* artom prefers to blame stephenfin 14:29
sean-k-mooneywell clearly his patch ate all the memory14:29
sean-k-mooneyok zuul is back up. changes running before 14:00 UTC have been requed anything uploaded or approved bettween 14:00 and 14:30 needs to be rechecked14:35
sean-k-mooneyinfra are going to send a staus update for the same shortly14:35
-openstackstatus- NOTICE: Zuul had to be restarted, all changes submitted or approved between 14:00 UTC to 14:30 need to be rechecked, we queued already those running at 14:0014:35
*** efried has joined #openstack-nova14:35
*** mlavalle has joined #openstack-nova14:41
kashyapsean-k-mooney: Since you've reviewed an older version (PS-5) and a newer one (PS-7), I'll just address the PS-7 bits here: https://review.opendev.org/#/c/631154/14:52
kashyapsean-k-mooney: That okay?14:52
*** ociuhandu has quit IRC14:52
* kashyap goes ahead with that plan to respond14:55
sean-k-mooneykashyap: am sure14:55
sean-k-mooneyi think i coppied most of the relevent bits although if you read both and just resond on 7 that is fine with me14:56
sean-k-mooneyor well update it in version 814:56
kashyapsean-k-mooney: While I comment in the spec, on the "increased memory usage" bit -- I knew that thing, but haven't explicitly mentioned it because it requires precise tests, in what scenarios, etc14:56
kashyapYou can't just put a generic: "in all cases memory is increased"14:57
sean-k-mooneykashyap: not really even with 1 pci root port dan found it used more memeory the pc14:57
kashyapIt requires more testing; so somebody ought to do the "performance testing guy's job"...14:57
sean-k-mooneyso i think in all configurtion it has more memory overhead14:57
kashyapsean-k-mooney: Right; I'll mention that, but need to carefully write it in context and with a config example14:58
sean-k-mooneyor we can jsut say we expect that q35 will use more memroy as we have never seen a case where it uses less14:58
kashyapsean-k-mooney: Also I don't want us to get sucked into that black-hole and get derailed...14:58
kashyapBut that begs the question: "how much more memory than before"14:58
sean-k-mooneysure an that we can leave to the operator14:59
kashyapWhich requires clear a example benchmark14:59
sean-k-mooneyor perfomance guy14:59
kashyapOkido; I actually first mentioned it locally and then removed it, as I was still thinking of it14:59
* kashyap goes to respond...14:59
sean-k-mooneyi mainly just want them to be aware that they should consider it when upgrading so they can factor it in to there host memory reservation and capastity planning15:00
kashyapYeah, definitely.  Thx for the taking time respond.15:01
*** trident has quit IRC15:04
*** trident has joined #openstack-nova15:05
*** LiangFang has quit IRC15:05
*** ociuhandu has joined #openstack-nova15:15
*** mkrai has quit IRC15:25
*** spatel has joined #openstack-nova15:32
spatelsean-k-mooney: do you know how much CPU would be enough to reserve for hypervisor? using isolcpus option?15:35
*** jaosorior has joined #openstack-nova15:44
sean-k-mooneyi dont adviase using isolcpus15:45
sean-k-mooneygenerally 1 phsycial core is more then enough on a compute node or 1 per numa nodes if you want to do affintiy of interupts15:46
spatelsean-k-mooney: but mostly for NFV they suggest using isolcpus for isolation15:47
sean-k-mooneyspatel:  i generally recommend you use the vcpu_pin_set or in train+ cpu_dedicate_set and cpu_shared_set to do the reservation15:48
spatelyou are saying 2 cpu core would be more than enough for hypervisor (per NUMA)?15:48
sean-k-mooneyspatel: you should only use isolcpus on realtime hosts and only on the cpus used for pinned vms15:48
spatelI am doing all cpu pinning (dedicated) option for all my workload15:49
spatelwe need performance not quantity.15:49
spatelI am planning to use isolcpus + vcpu_pin_set  (both option to allocate dedicated CPU)15:51
*** gyee has joined #openstack-nova15:53
openstackgerritJiri Suchomel proposed openstack/nova master: Add ability to download Glance images into the libvirt image cache via RBD  https://review.opendev.org/57430115:55
spatelsean-k-mooney: ^^15:55
sean-k-mooney1 phsyical core(2 hyperthreads) is normally enouch for a compute node if you are not using heavy telemetry15:56
sean-k-mooneyspatel: you can use isolcpus + vcpu_pin_set but only if the vm is pinned15:56
sean-k-mooneywhen you use isolcpus it disables the linux kernel shcduler for those cores15:57
sean-k-mooneyso if you have floating vms then they wont float15:58
spatelsean-k-mooney: sounds good, yes we do pinned VM (currently i have assigned 8 cores but wanted to see what people mostly recommend )15:58
sean-k-mooneyin generally isolcpus is only a good idea if you are running realtime wrokloads15:58
sean-k-mooneyspatel:  you might want to look into tuned by the way15:58
sean-k-mooneyit supports configuring this via userspace/sysfs15:59
spateltuned profile?15:59
sean-k-mooneyhttps://github.com/redhat-performance/tuned/tree/master/profiles/cpu-partitioning15:59
*** kberger_ has joined #openstack-nova15:59
sean-k-mooneyisolcpus is a deprecated kernel argument16:00
sean-k-mooneyhttps://github.com/redhat-performance/tuned/blob/master/profiles/cpu-partitioning/cpu-partitioning-variables.conf16:00
*** kberger_ has quit IRC16:00
melwittgmann: I'm trying to understand a bit about how/why two grenade jobs would run on stable/ussuri (we don't have an example yet). and I looked at the tempest change and realized I don't understand why it ran two grenade jobs on the name change patch https://review.opendev.org/722551 could you please explain why two jobs run on openstack/tempest? I thought it would have been only one16:00
*** kberger_ has joined #openstack-nova16:00
sean-k-mooneytuned uses the the sysfs/cgroups interface to achive the same effect without the drawbacks16:01
spatelsean-k-mooney: ohh good to know :)16:01
spatelwill look into that16:01
sean-k-mooneynormally i would jsut set  isolated_cores=2,4-7 and not set no_balance_cores=5-1016:01
sean-k-mooneyalthough no_balance_cores=5-10 would be useful for realtime hosts or ovs-dpdk16:02
spatelDo i need to restart machine to set this values ?16:02
*** KeithMnemonic has quit IRC16:03
gmannmelwitt: sure. for nova stable/ussuri, it will be both job running if you recheck any ussuri backport (or testing patch) until 724189 is merged. this is because compute template in Tempest switched to new job (https://review.opendev.org/#/c/722551/3/.zuul.yaml@543) and nova stable/ussuri ./.zuul.yaml have old job also listed for irrelevant file16:03
sean-k-mooneyspatel: am i dont think so16:04
sean-k-mooneyyou would on teh kernel command line but not with tuned16:04
gmannmelwitt: https://github.com/openstack/nova/blob/stable/ussuri/.zuul.yaml#L40016:04
*** noonedeadpunk has left #openstack-nova16:04
spatelsean-k-mooney: yes kernel does require reboot but lets me test in tuned16:04
melwittgmann: sorry I mean as an aside, why did two jobs run on https://review.opendev.org/722551 ? I realized I didn't understand that16:05
gmannmelwitt: on Tempest side it is still running because Tempest gate run 'integrated-gate-py3' template which is running all service tests. and that template is on openstack-zuul-jobs side so once i update that template then Tempest also will have single new job16:05
melwittthanks16:06
gmanncompute and service specific template are taken care by 722551 but integrated-gate-py3 template is not yet.16:06
*** ociuhandu has quit IRC16:07
gmannbest things we did in grenade side is we alias the grenade-py3 to new zuulv3 native job to avoid running legacy + new jobs during this migration. It is same zuulv3 jobs running twice with different name so will not cause issue.16:07
*** ociuhandu has joined #openstack-nova16:08
*** ociuhandu has quit IRC16:13
*** nightmare_unreal has quit IRC16:16
*** jsuchome has quit IRC16:25
*** jamesden_ has joined #openstack-nova16:31
*** jamesdenton has quit IRC16:31
*** evrardjp has quit IRC16:35
*** evrardjp has joined #openstack-nova16:35
*** tesseract has quit IRC16:36
*** dtantsur is now known as dtantsur|afk16:36
*** rpittau is now known as rpittau|afk16:37
*** tkajinam has quit IRC16:57
openstackgerritStephen Finucane proposed openstack/nova master: WIP: objects: Add migrate-on-load behavior for legacy NUMA objects  https://review.opendev.org/72438117:09
stephenfindansmith: That's not complete, but when you've a chance can you sanity check to see if that's what you're after? ^17:10
* stephenfin -> 🏃17:10
openstackgerritsean mooney proposed openstack/nova master: silence amqp heartbeat warning  https://review.opendev.org/72418817:10
*** ociuhandu has joined #openstack-nova17:12
artomThat just looks like stephenfin harpooned a dude coming at him17:12
sean-k-mooneyhehe you dont know what stephenfin gets up to on his runs17:14
sean-k-mooneyits one way to keep otheres away17:15
sean-k-mooneyi mean would you approch a person running with a harpoon17:15
artomDepends17:16
artomAm I a masochistic whale?17:16
*** ociuhandu has quit IRC17:17
* sean-k-mooney that feels like a trap17:17
* sean-k-mooney like does this make my butt look big17:17
*** avolkov has joined #openstack-nova17:21
*** maciejjozefczyk has quit IRC17:22
*** priteau has quit IRC17:32
*** ralonsoh has quit IRC17:41
openstackgerritsean mooney proposed openstack/nova master: [WIP] add workaround to disable multiple port bindings  https://review.opendev.org/72438617:42
openstackgerritsean mooney proposed openstack/nova master: [DNM] testing with force_legacy_port_binding workaround  https://review.opendev.org/72438717:42
*** udesale_ has quit IRC17:42
sean-k-mooneyby the way how do people feel about ^ as a temproy workaround for this long standing issue while we figure out how to fully fix this17:49
sean-k-mooneyi think my other patches are still the wright approch https://review.opendev.org/#/c/602432/ and https://review.opendev.org/#/c/640258 but i have not looked at this in a year and was not planning to but i guess i can try to find time to look at this again17:51
*** ociuhandu has joined #openstack-nova18:00
*** ociuhandu has quit IRC18:31
*** ociuhandu has joined #openstack-nova18:33
*** derekh has quit IRC18:36
*** ociuhandu has quit IRC18:37
*** iurygregory has quit IRC18:38
*** martinkennelly has quit IRC18:53
*** ociuhandu has joined #openstack-nova19:12
*** ociuhandu has quit IRC19:25
*** ociuhandu has joined #openstack-nova19:26
*** ociuhandu has quit IRC19:32
*** jangutter has quit IRC19:33
*** jangutter_ has joined #openstack-nova19:34
*** damien_r has quit IRC19:34
*** priteau has joined #openstack-nova19:45
*** avolkov has quit IRC19:45
*** avolkov has joined #openstack-nova19:50
*** priteau has quit IRC19:51
*** amodi has joined #openstack-nova19:51
*** belmoreira has quit IRC19:58
*** maciejjozefczyk has joined #openstack-nova20:27
openstackgerritGhanshyam Mann proposed openstack/nova master: Add nova-status upgrade check and reno for policy new defaults  https://review.opendev.org/72364520:43
*** nweinber has quit IRC20:45
gmanngibi: i updated the upgrade check with more safer way to cover all cases how policy is initialized or not. I have added test also for that and waiting for grenade job result which run upgrade check. also I am preparing new devstack but some issue, please let me know if it run fine now on your env- https://review.opendev.org/72364520:55
gmanni mean with the policy file you showed in http://paste.openstack.org/show/792881/20:56
*** ccamacho has quit IRC20:59
*** ociuhandu has joined #openstack-nova21:01
*** slaweq has quit IRC21:01
*** xek has quit IRC21:02
JamesBensonAll: Does anyone know if Nova supports the Tesla M2050 or M2070 for vGPU?21:03
*** ociuhandu has quit IRC21:06
melwittJamesBenson: I don't know but here's the doc we have if you didn't already find https://docs.openstack.org/nova/latest/admin/virtual-gpu.html21:08
JamesBensonmelwitt:  Yes, thanks, I saw that resource too.  I have some older m1000e's that I got and just not sure if I should bother installing openstack or just have them as one off's for testing code.21:09
*** slaweq has joined #openstack-nova21:09
melwittok. bauzas would know but he's off by now today. sean-k-mooney might know ^21:10
*** slaweq has quit IRC21:14
*** raildo has quit IRC21:18
*** rcernin has joined #openstack-nova21:32
*** rcernin has quit IRC21:33
*** rcernin has joined #openstack-nova21:34
*** spatel has quit IRC21:40
*** mriedem has left #openstack-nova21:48
openstackgerritmelanie witt proposed openstack/nova-specs master: Re-propose nova-audit spec for Victoria  https://review.opendev.org/72443021:48
*** mlavalle has quit IRC22:09
*** hamzy_ has joined #openstack-nova22:11
*** mlavalle has joined #openstack-nova22:12
*** hamzy has quit IRC22:13
*** mlavalle has quit IRC22:20
*** mlavalle has joined #openstack-nova22:26
*** ociuhandu has joined #openstack-nova22:36
melwittgmann: heya, would you mind looking over this review? it looks sane to me but could use your api validation expertise https://review.opendev.org/40751422:37
melwittI'm not clear on whether it could possibly cause any backward compat issues. it means to only target the 500 error casd22:38
melwitt*case22:38
*** tkajinam has joined #openstack-nova22:51
*** ociuhandu has quit IRC22:58
*** spatel has joined #openstack-nova23:03
*** spatel has quit IRC23:08
*** tosky has quit IRC23:11
*** ociuhandu has joined #openstack-nova23:33
*** ociuhandu has quit IRC23:53
« Tuesday, 2020-04-28 Index Thursday, 2020-04-30 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!