Friday, 2020-02-14

*** penick has quit IRC00:06
*** slaweq has joined #openstack-nova00:11
*** jmlowe has joined #openstack-nova00:15
*** slaweq has quit IRC00:16
*** ociuhandu has joined #openstack-nova00:17
*** ociuhandu has quit IRC00:22
*** jmlowe has quit IRC00:33
*** yedongcan has joined #openstack-nova00:51
*** migawa is now known as migawa|AFK00:52
*** migawa|AFK is now known as migawa00:52
*** ileixe has joined #openstack-nova00:55
*** abaindur has joined #openstack-nova00:58
huaqiangstephenfin: thank you and all for the review and patient.00:59
huaqiangs/patient/patience/ :)01:11
*** slaweq has joined #openstack-nova01:11
*** ileixe has quit IRC01:15
*** slaweq has quit IRC01:16
*** ileixe has joined #openstack-nova01:16
brinzhangefried: I think https://review.opendev.org/#/c/580336 all things are ready in SPEC, and also provide it's PoC codes https://review.opendev.org/#/c/693828/ agreement reached at Shanghai PTG https://etherpad.openstack.org/p/nova-shanghai-ptg(Line253)01:28
johnsomHi nova folks. FYI, I opened a bug that anti-affinity appears to have been broken since at least August. https://bugs.launchpad.net/nova/+bug/1863190 Let me know if there is more information I should collect while I have the instance up and running.01:31
openstackLaunchpad bug 1863190 in OpenStack Compute (nova) "Server group anti-affinity no longer works" [Undecided,New]01:31
*** gyee has quit IRC01:36
*** vishalmanchanda has joined #openstack-nova01:54
*** dave-mccowan has joined #openstack-nova01:56
melwittjohnsom: did you boot the two instances at the same time or near the same time? did you use multi-create?02:09
johnsomThey were close in time, but individual calls to the nova api02:10
johnsomOur normal active/standby boot sequence02:10
melwittok. I'm wondering if it's a race basically02:11
*** slaweq has joined #openstack-nova02:11
melwittlike if you wait for the first instance to become active and then do another request, would it fail02:11
johnsomYeah, probably. This used to work.02:11
johnsomYeah, I don’t know about waiting for active. That can take some time.02:12
johnsomWe have seen that up to five minutes, mostly scheduler time. I don’t think we want to hold up the boot process for the secondary that long02:13
melwittno sorry, I meant in your devstack02:14
melwittI can try it later too but not tonight. I was just asking in case you had that easy environment already up02:15
johnsomAh, as a test. I can try that. It will be tomorrow however, it is dinner time. Lol02:15
melwittsame02:15
johnsomIf there is a list of thing you would like me to try, comment on the bug and I will run them tomorrow02:15
melwittthere's a thing called the "late affinity check" in nova-compute that is on by default that should handle races02:15
melwittI don't know off the top of my head what could have regressed this.02:16
*** slaweq has quit IRC02:16
melwittI'll try out some things with a devstack tomorrow to start finding out what's going on02:18
melwittefried: potential regression alert ^ fyi https://bugs.launchpad.net/nova/+bug/186319002:18
openstackLaunchpad bug 1863190 in OpenStack Compute (nova) "Server group anti-affinity no longer works" [Undecided,New]02:18
johnsomOk, thanks. I will hold this instance through tomorrow in case we need it02:18
melwittk thanks02:21
*** jawad_axd has joined #openstack-nova02:22
*** jawad_axd has quit IRC02:27
*** HagunKim has joined #openstack-nova02:29
*** tbachman has quit IRC02:30
*** ileixe has quit IRC02:32
*** ileixe has joined #openstack-nova02:34
*** jawad_axd has joined #openstack-nova02:43
*** migawa is now known as migawa|AFK02:45
*** migawa|AFK is now known as migawa02:47
*** jawad_axd has quit IRC02:47
*** tbachman has joined #openstack-nova02:50
*** migawa is now known as migawa|AFK02:53
openstackgerritLiang Fang proposed openstack/nova-specs master: Support volume local cache  https://review.opendev.org/68907002:55
openstackgerritBrin Zhang proposed openstack/nova master: Introduce scope_types in os-instance-action policy  https://review.opendev.org/70775102:58
*** abaindur has quit IRC03:02
*** abaindur has joined #openstack-nova03:03
*** abaindur has quit IRC03:08
*** slaweq has joined #openstack-nova03:11
*** slaweq has quit IRC03:16
*** mkrai has joined #openstack-nova03:43
*** penick has joined #openstack-nova03:43
*** penick has quit IRC03:52
*** migawa|AFK is now known as migawa03:56
*** slaweq has joined #openstack-nova04:11
*** dave-mccowan has quit IRC04:13
*** slaweq has quit IRC04:16
*** mkrai has quit IRC04:57
*** udesale has joined #openstack-nova05:04
*** nicolasbock has quit IRC05:09
*** slaweq has joined #openstack-nova05:11
*** slaweq has quit IRC05:16
*** mkrai has joined #openstack-nova05:26
*** evrardjp has quit IRC05:34
*** evrardjp has joined #openstack-nova05:34
*** psachin has joined #openstack-nova05:37
*** CeeMac has quit IRC05:50
*** ratailor has joined #openstack-nova05:53
*** migawa is now known as migawa|AFK05:53
*** migawa|AFK is now known as migawa05:54
*** abaindur has joined #openstack-nova05:59
*** xiaolin has quit IRC06:00
*** xiaolin has joined #openstack-nova06:10
*** slaweq has joined #openstack-nova06:11
*** slaweq has quit IRC06:16
*** abaindur has quit IRC06:32
*** abaindur has joined #openstack-nova06:32
openstackgerritBrin Zhang proposed openstack/nova master: Introduce scope_types in os-instance-action policy  https://review.opendev.org/70775106:41
openstackgerritBrin Zhang proposed openstack/nova master: Introduce scope_types in os-instance-action policy  https://review.opendev.org/70775106:42
*** jhesketh has quit IRC06:48
*** jhesketh has joined #openstack-nova06:50
*** penick has joined #openstack-nova06:55
*** slaweq has joined #openstack-nova07:11
*** jhesketh has quit IRC07:15
*** slaweq has quit IRC07:16
*** penick has quit IRC07:17
*** tetsuro has joined #openstack-nova07:22
*** jawad_axd has joined #openstack-nova07:22
*** tetsuro has quit IRC07:23
*** jhesketh has joined #openstack-nova07:25
*** mkrai has quit IRC07:26
*** bbowen_ has joined #openstack-nova07:29
*** bbowen has quit IRC07:30
*** imacdonn has quit IRC07:54
*** imacdonn has joined #openstack-nova07:55
*** ivve has joined #openstack-nova07:56
*** ralonsoh has joined #openstack-nova07:57
openstackgerritBrin Zhang proposed openstack/nova master: Add test coverage of existing os-instance-actions policies  https://review.opendev.org/70777707:57
*** maciejjozefczyk has joined #openstack-nova07:58
openstackgerritBrin Zhang proposed openstack/nova master: Introduce scope_types in os-instance-action policy  https://review.opendev.org/70775107:58
*** slaweq has joined #openstack-nova08:00
*** mkrai has joined #openstack-nova08:01
*** abaindur_ has joined #openstack-nova08:05
*** abaindur has quit IRC08:08
*** amoralej|off is now known as amoralej08:10
*** abaindur_ has quit IRC08:11
*** abaindur has joined #openstack-nova08:12
*** lpetrut has joined #openstack-nova08:14
*** tesseract has joined #openstack-nova08:21
*** slaweq has quit IRC08:33
*** slaweq has joined #openstack-nova08:37
*** tkajinam has quit IRC08:42
openstackgerritGuo Jingyu proposed openstack/nova-specs master: Proposal for a safer noVNC console with password authentication  https://review.opendev.org/62312008:59
openstackgerritBrin Zhang proposed openstack/nova master: Add test coverage of existing os-instance-actions policies  https://review.opendev.org/70777709:03
openstackgerritGuo Jingyu proposed openstack/nova-specs master: Proposal for a safer noVNC console with password authentication  https://review.opendev.org/62312009:07
*** rcernin has quit IRC09:12
*** xek has joined #openstack-nova09:16
openstackgerritBrin Zhang proposed openstack/nova master: Introduce scope_types in os-instance-action policy  https://review.opendev.org/70775109:28
*** martinkennelly has joined #openstack-nova09:31
*** klippo has joined #openstack-nova09:32
*** derekh has joined #openstack-nova09:32
*** beagles has quit IRC09:54
*** rcernin has joined #openstack-nova09:59
openstackgerritLee Yarwood proposed openstack/nova master: DNM - Test TEMPEST_EXTEND_ATTACHED_ENCRYPTED_VOLUME  https://review.opendev.org/70759310:02
*** HagunKim has quit IRC10:09
openstackgerritMerged openstack/nova-specs master: Add nova-audit spec  https://review.opendev.org/69322610:11
*** ociuhandu has joined #openstack-nova10:12
*** rcernin has quit IRC10:17
*** abaindur has quit IRC10:18
*** abaindur has joined #openstack-nova10:18
openstackgerritSylvain Bauza proposed openstack/nova-specs master: Proposes NUMA topology with RPs  https://review.opendev.org/55292410:23
bauzasstephenfin: sean-k-mooney: efried: I made another iteration to clarify ^10:23
*** abaindur has quit IRC10:23
*** psachin has quit IRC10:25
*** vishalmanchanda has quit IRC10:28
openstackgerritSundar Nadathur proposed openstack/nova master: Delete ARQs for an instance when the instance is deleted.  https://review.opendev.org/67373510:29
openstackgerritSundar Nadathur proposed openstack/nova master: Enable hard/soft reboot with accelerators.  https://review.opendev.org/69794010:29
openstackgerritSundar Nadathur proposed openstack/nova master: Enable start/stop of instances with accelerators.  https://review.opendev.org/69955310:29
openstackgerritSundar Nadathur proposed openstack/nova master: Enable and use COMPUTE_ACCELERATORS trait.  https://review.opendev.org/69955410:29
openstackgerritSundar Nadathur proposed openstack/nova master: Bump compute rpcapi version and reduce Cyborg calls.  https://review.opendev.org/70422710:29
openstackgerritSundar Nadathur proposed openstack/nova master: Add cyborg tempest job.  https://review.opendev.org/67099910:29
*** psachin has joined #openstack-nova10:33
*** ccamacho has quit IRC10:40
*** belmoreira has quit IRC10:46
openstackgerritLee Yarwood proposed openstack/nova master: api: Introduce microverion 2.82 allowing boot from volume rescue  https://review.opendev.org/70143010:47
openstackgerritLee Yarwood proposed openstack/nova master: compute: Extract _get_bdm_image_metadata into nova.utils  https://review.opendev.org/70521210:47
*** mkrai has quit IRC10:54
*** ociuhandu has quit IRC10:54
*** ociuhandu has joined #openstack-nova10:59
*** ociuhandu has quit IRC11:01
*** ociuhandu has joined #openstack-nova11:01
*** abaindur has joined #openstack-nova11:05
*** N3l1x has joined #openstack-nova11:09
*** abaindur has quit IRC11:10
*** psachin has quit IRC11:10
*** psachin has joined #openstack-nova11:12
openstackgerritGuo Jingyu proposed openstack/nova-specs master: Proposal for a safer noVNC console with password authentication  https://review.opendev.org/62312011:53
*** udesale_ has joined #openstack-nova11:56
*** ccamacho has joined #openstack-nova11:57
*** tkajinam has joined #openstack-nova11:58
*** udesale has quit IRC11:59
*** ociuhandu has quit IRC12:07
*** tbachman has quit IRC12:11
*** ociuhandu has joined #openstack-nova12:12
*** tbachman has joined #openstack-nova12:12
*** nicolasbock has joined #openstack-nova12:13
*** psachin has quit IRC12:15
openstackgerritMerged openstack/nova stable/rocky: Mask the token used to allow access to consoles  https://review.opendev.org/70425512:18
*** ociuhandu has quit IRC12:18
*** maciejjozefczyk has quit IRC12:21
*** maciejjozefczyk has joined #openstack-nova12:37
sean-k-mooneybauzas: i think you missed some of efried's comments. like explainging the fallback query.12:41
sean-k-mooneyefried: also did you see my suggestion to have teh numa=false host be added to an aggreate so we can use forbiden aggreates for that case https://review.opendev.org/#/c/552924/21/specs/ussuri/approved/numa-topology-with-rps.rst@23512:42
*** tetsuro has joined #openstack-nova12:42
sean-k-mooneyover all however i think there is enough we agree on that we can proceed with this and adject with a fix up patch later based on implementaion discussions12:46
sean-k-mooneyi am off today so im not going to be around but i might see pings in the evening as my irc client will be connected12:47
*** ratailor has quit IRC12:50
*** yedongcan has quit IRC12:52
*** ociuhandu has joined #openstack-nova12:53
openstackgerritBrin Zhang proposed openstack/nova-specs master: Proposal for a safer noVNC console with password authentication  https://review.opendev.org/62312012:59
*** dave-mccowan has joined #openstack-nova13:02
*** mkrai has joined #openstack-nova13:06
*** ociuhandu has quit IRC13:11
*** tkajinam has quit IRC13:17
*** ivve has quit IRC13:28
*** ociuhandu has joined #openstack-nova13:31
*** psachin has joined #openstack-nova13:37
*** mkrai has quit IRC13:37
*** bengates has joined #openstack-nova13:37
*** ociuhandu has quit IRC13:45
bauzassean-k-mooney: again, I explained this13:48
bauzassean-k-mooney: oops my bad, I didn't passed my draft comments13:48
bauzassean-k-mooney: see them13:48
*** ociuhandu has joined #openstack-nova13:49
bauzasstephenfin: can you please review it ? https://review.opendev.org/#/c/552924/13:49
*** ociuhandu has quit IRC13:55
*** ociuhandu has joined #openstack-nova13:58
*** amoralej is now known as amoralej|lunch13:59
*** tetsuro has quit IRC14:04
*** ociuhandu has quit IRC14:19
openstackgerritBalazs Gibizer proposed openstack/nova stable/queens: Mask the token used to allow access to consoles  https://review.opendev.org/70784514:19
*** ociuhandu has joined #openstack-nova14:19
openstackgerritBrin Zhang proposed openstack/nova-specs master: Proposal for a safer noVNC console with password authentication  https://review.opendev.org/62312014:20
*** amoralej|lunch is now known as amoralej14:30
*** jawad_axd has quit IRC14:38
*** jawad_axd has joined #openstack-nova14:39
*** jawad_axd has quit IRC14:41
*** psachin has quit IRC14:48
*** ociuhandu has quit IRC14:55
bauzasefried: stephenfin: the spec is waiting for your comments :) https://review.opendev.org/#/c/552924/15:03
melwittjohnthetubaguy, bauzas: thank you both for your review on the nova-audit spec ++15:03
*** ociuhandu has joined #openstack-nova15:05
openstackgerritBrin Zhang proposed openstack/nova-specs master: Proposal for a safer noVNC console with password authentication  https://review.opendev.org/62312015:08
melwittalex_xu: hey, did you mean to hold off on +W on the unified limits spec? https://review.opendev.org/60220115:10
*** abaindur has joined #openstack-nova15:16
*** nweinber has joined #openstack-nova15:18
*** abaindur has quit IRC15:21
*** mlavalle has quit IRC15:23
*** _mlavalle_1 has joined #openstack-nova15:23
openstackgerritAlexandre arents proposed openstack/nova master: Avoid allocation leak when deleting instance stuck in BUILD  https://review.opendev.org/70236815:25
alex_xumelwitt: I'm ok with spec, just see bauzas whether want to look at that again15:31
bauzasalex_xu: the unified limits one ?15:31
melwittalex_xu: cool, thanks15:31
bauzasI can take a look15:31
*** ociuhandu has quit IRC15:31
alex_xunp15:32
*** eharney has joined #openstack-nova15:33
*** martinkennelly has quit IRC15:39
efriedalex_xu, johnthetubaguy: and also stephenfin: bauzas: gibi: Would you please have a look at https://review.opendev.org/#/c/580336 (delete on termination) and see if it's ready to +A today?15:42
*** maciejjozefczyk has quit IRC15:42
bauzasefried: /me opens a tab15:43
efriedsean-k-mooney, bauzas: I'm looking at the NUMA RP spec again right now. Based on comments I skimmed on my phone, I think we might have missed the boat on the `False` thing. But checking...15:44
*** nweinber has quit IRC15:44
bauzasefried: when we discussed this with stephenfin, we agreed on splitting between non-NUMA and NUMA-aware nodes15:44
bauzasso, if you say 'no' for a node, then you shouldn't get NUMA-aware instances15:45
bauzasit's like 'no, I don't want to get a beer', but then the bar tender gives you one15:45
efriedbauzas: exactly, and there was a design hole for that in PS21.15:46
gibiefried: ack15:46
*** _mlavalle_1 has quit IRC15:48
openstackgerritBrin Zhang proposed openstack/nova-specs master: Support re-configure deleted_on_termination in server  https://review.opendev.org/58033615:48
*** mlavalle has joined #openstack-nova15:49
*** nweinber has joined #openstack-nova15:49
bauzasefried: ok, look then at the new revision and tell me then if you see some hole15:50
efriedbauzas: now I think I understand that you got rid of the fallback query completely, and on purpose. But that doesn't work. The upgrade issue that gibi and stephenfin identified still exists.15:50
efriedbauzas: Specifically: without the 'fallback query', NUMA-aware flavors can't land on `None` hosts.15:50
efriedBut *with* the fallback query, NUMA-aware flavors will *also* (incorrectly) land on `False` hosts.15:51
bauzasefried: ah, possibly15:51
bauzasshit15:51
efriedSo we need a) the fallback query, and b) some other design element to fix that second problem.15:51
efriedsean-k-mooney's suggestion of using an aggregate... that will work, but it seems like a big hammer to me.15:52
bauzasor using another trait then15:52
efriedI think I would prefer marking `False` hosts with a `NO_NUMA_HERE` trait... yes15:52
bauzaslike HW_NON_NUMA15:52
bauzasok, I think we can do this15:52
efriedIt's not pretty. But it's also *temporary*. Once all computes are upgraded to V where the conf is mandatory, we can get rid of that, because we also get rid of the fallback query.15:52
efriedplease be sure to mention ^ that.15:53
efriedsean-k-mooney: does that work for you?15:53
bauzasa trait for which nodes then ?15:53
efriedfor U+ nodes where the conf opt is marked False explicitly.15:53
bauzasfor those having 'false' right?15:53
efriedy15:53
bauzasok, because then we will use the trait even after V15:54
efriedno15:54
efriedI mean, it wouldn't *hurt* to keep it, but it won't be needed after V.15:54
bauzasif so, I'd prefer to have a trait on 'none'15:54
efriedcan't15:54
bauzasactually you're right15:54
efriedbecause we can't muck with T hosts15:55
bauzasand yeah we can still have the trait15:55
bauzasin V15:55
bauzasand for example, stoppint to have it when we want (V or W)15:55
efriedWe could. But we wouldn't need it anymore. But we can make that decision in V. Just mention that we *could* get rid of it if we wanted to.15:55
bauzasagreed15:55
bauzasokay I think I can write *something*15:55
alex_xubauzas: efried that spec I only not sure about the policy, that API is admin-only API. but I think his usecase is for normal user. maybe need to add new policy15:55
alex_xubrinzhang: ^15:55
bauzasmelwitt: +Wd https://review.opendev.org/#/c/602201/1715:55
gibiefried, brinzhang: I have to API related comments in https://review.opendev.org/#/c/580336/15:56
melwittthanks bauzas!15:56
openstackgerritBrin Zhang proposed openstack/nova-specs master: Proposal for a safer noVNC console with password authentication  https://review.opendev.org/62312015:57
brinzhangalex_xu, bauzas, melwitt, gibi, stephenfin, efried: the nova-support-webvnc-with-password-anthentication spec was ready, please review, hope it can be merged before freeze SPEC day (Feb 14), thanks.15:57
brinzhangalex_xu: thanks for your review of the noVNC spec15:58
alex_xubrinzhang: np, quick reminder the title https://review.opendev.org/#/c/623120/27//COMMIT_MSG15:59
brinzhangalex_xu: will update15:59
openstackgerritBrin Zhang proposed openstack/nova-specs master: Proposal for a safer remote console with password authentication  https://review.opendev.org/62312016:01
brinzhangalex_xu: done.16:01
efriedbrinzhang: you're going to have a busy evening :)16:02
efriedbrinzhang: did you see gibi's comment on https://review.opendev.org/#/c/580336/ above?16:02
brinzhangefried: I will see it16:02
*** bengates has quit IRC16:06
*** ociuhandu has joined #openstack-nova16:06
*** xek has quit IRC16:10
gmannefried: sean-k-mooney : sorry missed your ping on microversion need discussion. I did not read the full context but if it is failing later and change converting 2xx->4xx , then we do not need microversion.16:10
efriedgmann: cool, thanks.16:11
efriednot sure what you mean by 'failing later'. The failure would happen via an API check.16:11
efriedand we're looking to convert a 4xx->2xx16:12
efriedIOW a flavor+image combination that we would previously bounce would now be accepted.16:12
gibiefried, brinzhang, alex_xu: +A-d the vnc password spec16:12
efriedthanks gibi16:12
gmannefried: ohk, and it is bounce currently in what situation ?16:13
openstackgerritMerged openstack/nova-specs master: Add Unified Limits Spec  https://review.opendev.org/60220116:13
efriedgmann: today you're not allowed to specify a hw:numa_nodes that would result in an uneven split, unless you also specify that split.16:13
efriedgmann: what sean-k-mooney is proposing is to tweak the algorithm so it's able to do that uneven split for you.16:14
*** N3l1x has quit IRC16:14
efriedgmann: e.g. if your flavor+image says vcpus=5, hw:numa_nodes=2, previously we would fail. With the proposed "bug fix" we would succeed and give you a 3/2 split.16:15
brinzhanggibi: how about "The ``delete_on_termination`` field is optional, if not specified, the server's BDM does not contains ``delete_on_termination``."16:15
brinzhanggibi: thanks +A for vnc passwd spec16:15
gmannefried: sean-k-mooney ah, got it. ok in that case, we should bump version for interoperability .16:16
*** nweinber has quit IRC16:16
efriedokay. I think that makes sense.16:16
efriedergo, not happening in U :(16:16
gmanngibi: brinzhang re: delete_on_termination spec - but volume id in url is old volume and in body it is new volume so those cannot be same.16:17
*** udesale_ has quit IRC16:17
brinzhanggmann: yes, it's true16:18
gmannso we have to keep expecting the volume_id in request as mandatory.16:19
gmannconsidering delete_on_termination as false by default make sense for me and consider only if it is explicitly mentioned in request16:20
brinzhang - If only ``volumeId`` is required, just doing a swap volume.16:20
brinzhang- If only ``delete_on_termination`` is required, just update the voume  denoted by the ``volume_id`` in the URL path not the request body.16:20
brinzhangthese sentences make sense16:21
brinzhangThe default value of delete_on_termination is False, doesnot impact anything of the target volume, so I also think it ok have the default value16:22
brinzhanggibi: what do you think?16:22
bauzasefried: quick question, when we ask for a forbidden trait for all request groups, can we just say '&required=!HW_NON_NUMA' or need to ask for every request group ?16:24
bauzaseg. &required_PROC1=!HW_NON_NUMA&required_PROC2=!HW_NON_NUMA16:25
bauzasor sean-k-mooney or anyone that can answer this question ^16:25
efriedbauzas: In this case it's for the fallback query.16:25
openstackgerritLee Yarwood proposed openstack/nova stable/rocky: Force refresh instance info_cache during heal  https://review.opendev.org/67927116:25
efriedSo it can just be in the same group with the VCPU and MEMORY_MB resources.16:25
gmannbrinzhang: second statement seems like updating 'delete_on_termination' for already attached volume but via PUT which is nothing but swap volume API not updating the existing attachment.16:26
bauzasefried: no, the fallback query is literrally Train16:26
efriedthe point is we *want* to land on unreshaped hosts -- but only those with `None`16:26
bauzasactually, you're right16:27
gmannbrinzhang: gibi i think, we should only target for newly replaced volume request can accept the 'delete_on_termination' for new volume.16:27
brinzhanggmann: you mean we donot to change the old volume's delete_on_termination?16:27
bauzasI'm already requiring NUMA-aware nodes, I don't need to ask in this query but in the fallback one16:27
bauzasefried: thanks16:28
gmannbrinzhang: gibi sp that we can cover all the case where user can attach a volume (server boot, attach volume, swap volume) with 'delete_on_termination' info16:28
* bauzas is fried16:28
efriedright, so:16:28
efried- U hosts with `False` we a) don't reshape, and b) mark with HW_NO_NUMA16:28
efried- pre-U and `None` hosts we a) don't reshape, and b) *don't* mark with HW_NO_NUMA (in the case of pre-U hosts, we couldn't mark them with anything even if we wanted to)16:28
efried- The fallback query *forbids* HW_NO_NUMA so we *won't* land on U/False hosts.16:28
efriedcorrect16:28
brinzhanggmann: I know you mean, but the old volume is not support to configure the 'delete_on_termination' after it was attached, or booted(bfv)16:30
brinzhanggmann: I think change the old volume's 'delete_on_termination' make sense16:30
gmannbrinzhang: but user can do it always while it was attached originally in server boot or POST /servers/{server_id}/os-volume_attachments/{volume_id}16:33
brinzhanggmann: I know, but it just a decision while the server booting or attaching a volume, while the server run a long time, the user may will change it.16:34
openstackgerritMerged openstack/nova-specs master: Proposal for a safer remote console with password authentication  https://review.opendev.org/62312016:35
gibigmann: as I read the goal of the spec is to change the delete_on_termination flag of an already attached volume. But now I see another usecase when you swap a volume then you need to define what should be the delete_on_terminate value of the new volume16:35
gibibrinzhang: do you need both use case?16:35
brinzhanggmann: In our project, we met such customers who were unwilling to suspend the virtual machine business and wanted to change the volume configuration, but were unwilling to keep mounted volumes when cleaning up the servers.16:36
gmannyeah, i feel later case make sense and with that we cover all cases where user can attach volume and specify the 'delete_on_termination'16:36
melwittI'm just in the peanut gallery but I would think yes, do both right? if you're swapping in a new volume, you'd want to be able to say whether it should be deleted on termination16:37
melwittotherwise that would be the one case where you can't say it16:37
*** amoralej is now known as amoralej|off16:37
gmannyeah, for new volume yes. but i am not getting to do it for old attached volume.  or at least via this API which is nothing but swap volume.16:38
brinzhangLooking back at the history of the spec, we can't find a better nova API to make changes to this. The current API is suggested by mriedem.16:40
melwittI think the main thing they want to do is change for old already attached volume :) but I do agree, adding this would be overloading the swap volume API (not ideal), as lyarwood mentioned on the review awhile back16:40
gibibrinzhang: so you confirm that you would like to have both use cases covered16:40
gmanntrue.16:40
gmannbrinzhang: you cannot delete the attachment and re-attach with 'delete_on_termination' ?16:41
gibinot for bfv16:41
*** Sundar has joined #openstack-nova16:41
gmannohk16:41
brinzhanggmann: of course, I can, but while the admin need to manager more than 100+ instance, it will be have some problem to maintain16:43
gmanni am reading the recommended use case of this API from api-ref in warning section - https://docs.openstack.org/api-ref/compute/?expanded=update-a-volume-attachment-detail16:43
brinzhanggibi, gmann, melwitt: so you all suggestion is only do the swap's volume to set the 'delete_on_termination' for the new volume?16:44
gmannand making this APIs for this new case means we are over-scoping this API which can lead this to more-confused API than it is today16:44
gibibrinzhang: I would be happy to support changing delete_on_terminate for attached volume I just see that gmann has a problem with the API design16:44
gibias a note: my EndOfBusiness is closing really fast16:45
gmannhttps://docs.openstack.org/api-ref/compute/?expanded=update-a-volume-attachment-detail#update-a-volume-attachment16:46
gmann'This API is typically meant to only be used as part of a larger orchestrated volume migration operation initiated in the block storage service'16:46
gibiif you can figure out a good place for this in the API I will support both use case, if there is no good place for the first case in the API then I can also support implementing only the second case in U16:46
lyarwoodFWIW I think this is the correct place for the API16:46
lyarwoodonce we've removed the swap stiff16:47
lyarwoodstuff*16:47
lyarwoodI wanted to look into moving it into an instance action as with extend_volume etc16:47
openstackgerritSylvain Bauza proposed openstack/nova-specs master: Proposes NUMA topology with RPs  https://review.opendev.org/55292416:47
bauzasI'm burned from this week16:48
* efried clicks16:48
gmannor in PUT /servers ?16:48
bauzasefried: I'll leave you look again if you agree with https://review.opendev.org/55292416:48
bauzasstephenfin: sean-k-mooney: your help could be nice ^16:48
efriedlooking right now16:48
bauzasgibi: too if you're not burned too ^16:48
bauzasefried: not a big deal, just wrote another paragraph https://review.opendev.org/#/c/552924/22..23/specs/ussuri/approved/numa-topology-with-rps.rst16:49
brinzhanggmann: PUT /servers is the PS3 solution16:49
brinzhangOr I can say is our first idea for the 'delete_on_termination'16:50
*** martinkennelly has joined #openstack-nova16:50
gmanngibi: lyarwood brinzhang melwitt :in reality this swap volume  API has been always in question and diverted the use case from it was originally. I remember one failure in this API when one of our customer  were doing swap to vole2 and swap back to vol1 and so on. i do not know why :)16:51
gibisorry folks I have to leave.16:51
openstackgerritBrin Zhang proposed openstack/nova-specs master: Support re-configure deleted_on_termination in server  https://review.opendev.org/58033616:51
lyarwoodgmann: yeah direct use of the API isn't supported, that's why I want to hide it behind https://docs.openstack.org/api-ref/compute/?expanded=run-events-detail#create-external-events-os-server-external-events16:51
gmannif we delete this API then it will be much better i think :)16:52
bauzasgibi: I'm just about to second you16:52
brinzhanggibi: thanks, good night.16:53
*** martinkennelly has quit IRC16:56
Sundarsean-k-mooney: I updated https://review.opendev.org/673735 to move the deletion of ARQs during rescheduling to https://review.opendev.org/#/c/673735/38/nova/conductor/manager.py@598 .16:56
efriedbauzas: I've got a couple of really minor updates I'd like to make and then +2. Are you around for just a couple more minutes?16:57
bauzasefried: I can but then I'll have to leave16:57
*** jmlowe has joined #openstack-nova16:57
* bauzas won't tell what he has to do for tonight, I hate this f* day16:58
Sundardansmith, efried, gibi: Hope you're all ok with this approach.16:58
bauzasefried: have you clicked on submitting your comments ?16:59
efriedbauzas: 2 mins16:59
bauzascool16:59
brinzhangefried, gmann, melwitt: It's too later for me, I have to leave, if you can get AGGREMENT please leave some comments, or let this SPEC go.16:59
brinzhangefried, gmann, melwitt: Thank you very much.16:59
dansmithSundar: I'll have to see it, but I think that makes sense for reschedule17:00
*** penick has joined #openstack-nova17:01
brinzhangdansmith: Agree, add the delete arqs operation in  _cleanup_when_reschedule_fails(), I think make sense.17:02
efriedbauzas: now17:04
bauzasack17:04
efriedbauzas: do you want me to make the edit?17:05
openstackgerritIlya Etingof proposed openstack/nova master: Add JSON schema and test for network_data.json  https://review.opendev.org/70313317:05
bauzasefried: nah, can do17:05
efriedk17:05
bauzasAtom is still open17:05
artom'sup? Oh.17:06
efrieddo you seriously have a notifier for 'atom'?17:06
bauzasLOL17:06
artomNah, I just happen to be procrastinating in here17:06
efried:)17:07
efriedshoot, if you want to read some specs...17:07
efriedon second thought, never mind. Don't want to risk a -1 at this stage...17:07
artomHaha17:07
artomSo, it's short-term procrastination, waiting for a command to run17:08
efriedoh17:08
efriedsee, I call that "working"17:08
efriedback in the day, when waiting for compiles, play some guitar.17:09
*** lpetrut has quit IRC17:09
efried"working".17:09
artomAlso, I thought spec deadline was yesterday?17:09
efriedartom: ahem, "this week".17:11
bauzasrunning tox and then uploading17:13
*** ociuhandu has quit IRC17:16
openstackgerritSylvain Bauza proposed openstack/nova-specs master: Proposes NUMA topology with RPs  https://review.opendev.org/55292417:18
* bauzas calls a day with ^17:18
* bauzas now has to rush to a flower shop, damn it17:19
openstackgerritMerged openstack/nova master: Skip to run all integration jobs for policy-only changes.  https://review.opendev.org/70726817:19
melwittbrinzhang: no, I was saying I think it should be allow delete_on_termination for both new swap volume and old volume17:20
efriedbauzas: Looks good, thanks. I'll wrangle the +A from here (hopefully).17:22
efriedstephenfin: I think that +A is pretty much yours https://review.opendev.org/#/c/552924/17:22
*** tesseract has quit IRC17:29
*** evrardjp has quit IRC17:34
*** evrardjp has joined #openstack-nova17:34
*** tbachman has quit IRC17:41
*** tbachman_ has joined #openstack-nova17:41
*** mvkr has quit IRC17:42
*** gmann is now known as gmann_afk17:42
*** nweinber has joined #openstack-nova17:42
*** igordc has joined #openstack-nova17:43
openstackgerritMerged openstack/nova stable/queens: Block deleting compute services with in-progress migrations  https://review.opendev.org/69971817:45
*** eharney has quit IRC17:45
efriedgibi: if you happen to check back in, you could also be that +A https://review.opendev.org/#/c/552924/17:45
*** penick has quit IRC17:48
artomefried, FWIW I think stephenfin's off this afternoon17:49
artomSo it's up to... not sure who, actually17:50
efriedartom: ugh, okay, thanks for the info. Guess I need to decide whether his/gibi's previous +2s encompass enough of the current iteration for me to proxy them.17:52
efried...yeah, no, I can't do that.17:54
artomdansmith lulz17:54
dansmithwhat?17:55
artomSorry, was trolling. We're out of cores in the correct timezone to +A the NUMA RPs spec17:56
efriedI don't think attempting to drag dansmith back in is the right answer.17:57
artomI know - hence the "trolling"17:57
artomOh great, func tests are broken on queens17:58
artomI know upstream doesn't care, but Red Hat does :P17:58
efriedgmann_afk: is that ^ something you were working on?17:58
*** eharney has joined #openstack-nova17:58
efriedartom: gmann had some fixes in stable, but perhaps they didn't make it all the way back to q. Are you aware or want me to dig?17:58
efriedartom: nm, apparently q was included https://review.opendev.org/#/q/topic:fix-stable-gate+status:merged18:00
artomefried, I found some things, but yeah, they're for tempest, and exist in queens18:00
efriedso if still broken, I guess it's something else? (or you need a rebase?)18:00
*** nweinber has quit IRC18:00
artomI'm basically getting https://bugs.launchpad.net/ubuntu/+source/nova/+bug/180726218:01
openstackLaunchpad bug 1807262 in nova (Ubuntu) "stein unit tests fail with sqlalchemy.exc.NoSuchTableError: migration_tmp" [High,Fix released]18:01
efriedartom: meaning you need to backport a req bump to sqla 0.12.0?18:02
*** derekh has quit IRC18:03
artomefried, apparently18:04
* artom attempts to confirm18:04
efriedartom: I'll join you in -requirements, but I thought that was a thing we don't do.18:04
artomefried, yeah, it seems wide-hitting to me18:05
efriedhow is it just cropping up, then?18:05
efriedartom: I guess you might want to confirm that that version of sqla fixes your issue before pursuing.18:05
artomefried, backporting a thing to downstream queens, func test failed, since we suck and don't have an easy way to run downstream func tests locally, I figured a backport to queens might hit the same issue18:06
artomInstead, I hit a different one18:06
artom*This* one :)18:06
efriednote that nova/U bumped past 0.12.0 to 0.13.0 for unrelated reasons https://review.opendev.org/#/c/690704/ so you would need a stable-only patch starting in train.18:06
artomAnd actually bumping to 0.12.0 does nothing18:07
efriedokay then.18:07
efriedartom: what are you backporting downstream?18:07
artomefried, https://review.opendev.org/#/q/Iafba419fe86446ffe636721f523fb619f8f787b318:07
efriedartom: well, confirming that 0.12.0 shouldn't fix the issue, the fact that that's already in r/s/t and those guys aren't using 0.12.0...18:08
artomefried, yeah. And actually, q is em, so I could try and push it18:09
artomSee what the gate says, maybe it's local to me...18:09
efriedswhat I would do.18:09
openstackgerritArtom Lifshitz proposed openstack/nova stable/queens: Add functional regression recreate test for bug 1839560  https://review.opendev.org/70788618:09
openstackbug 1839560 in OpenStack Compute (nova) stein "ironic: moving node to maintenance makes it unusable afterwards" [High,Fix committed] https://launchpad.net/bugs/1839560 - Assigned to Matt Riedemann (mriedem)18:09
openstackgerritArtom Lifshitz proposed openstack/nova stable/queens: Restore soft-deleted compute node with same uuid  https://review.opendev.org/70788718:09
artomAnd while that runs, I need lunch18:10
* efried also bbiab18:10
*** eharney has quit IRC18:33
*** jmlowe has quit IRC18:46
*** jdillaman has quit IRC18:52
*** ociuhandu has joined #openstack-nova19:01
*** cmurphy is now known as cmorpheus19:05
*** ociuhandu has quit IRC19:06
*** ralonsoh has quit IRC19:07
*** mvkr has joined #openstack-nova19:10
*** Sundar has quit IRC19:14
*** tbachman_ has quit IRC19:22
*** mvkr has quit IRC19:29
*** eharney has joined #openstack-nova19:54
*** jmlowe has joined #openstack-nova20:03
*** nweinber has joined #openstack-nova20:09
*** eharney has quit IRC20:10
*** gmann_afk is now known as gmann20:12
gmannefried: artom it was working till morning with latest fix of stachviz.20:14
*** igordc has quit IRC20:15
gmannthis stable/queens got in today- https://review.opendev.org/#/c/699718/20:15
*** spatel has joined #openstack-nova20:16
spatelsean-k-mooney: question if you around20:16
spatelwhen we do cpu pinning in that case does openstack make sure it properly map with sibling?20:17
artomgmann, thanks, I'll try with and without that change20:17
gmannartom: but i would not be surprise if there is new one :) by seeing current frequency of 1-2 issue per day on stable branches.20:18
artomgmann, ack20:19
artomgmann, yeah, no, still happening without that last change20:21
artomgmann, however, it seems it's only happening on my machine...20:22
gmannartom: until you have updated stable constraint locally or something. may be recreate tox can help ?20:24
artomgmann, I've been doing *nothing but* recreates :)20:24
gmann:)20:24
* artom tries rm -rf'ing some "cruft"20:25
*** nweinber has quit IRC20:31
*** xek has joined #openstack-nova20:32
spatelDoes anyone know what would be the advantage of hw:cpu_thread_policy=require20:43
spatelcurrently i have default which is "prefer"20:44
spatelI am seeing performance issue so trying to understand what would be the advantage20:44
efriedspatel: let me know if this doesn't help: https://docs.openstack.org/nova/latest/admin/cpu-topologies.html#customizing-instance-cpu-thread-pinning-policies20:48
spatelefried: i am on same page but i am trying to understand language20:48
efriedah20:48
efriedthis is not my area of expertise. Normally I would call on stephenfin or sean-k-mooney, but I think they're out at the moment. artom, how's your understanding here?20:49
spatelcurrently i am seeing on vm siblings are not correctly align with my physical cores...20:50
spatellook like cpu_thread_policy=require may fix that but wanted to understand first before i just go and play20:50
artomspatel, it's getting into the nitty gritty of low level CPU arch, which is not my strong point20:51
artomBut IIUC putting workloads on thread siblings (policy=require) means the content for the hardware that's shared between the two threads20:52
artomI *think* there would be 2 decoding units, but a single execution unit20:52
artomSo if your workload is CPU heavy, I would advice against putting it on thread sibligns, and use policy=isolate (at the cost of unusable CPU threads)20:53
spatelwe are running Erlang application which is CPU and memory bound20:53
artom*means there's contention20:53
artomspatel, OK, so try `isolate`, see if there's an improvement20:54
spateland i want to create single VM on single compute (want to assign all resources to vm)20:54
spatelartom: i am reading all other document and they are saying for performance use policy=require it will make sure your process stay with sibling..20:54
spateltrying to understand this language20:55
artomspatel, "The presence of an SMT implementation like Intel Hyper-Threading can boost performance by up to 30% for some workloads. However, thread siblings share a number of components and contention on these components can diminish performance for other workloads."20:55
artomspatel, there's no magic answer, it depends on the workload20:55
spatelhmm!20:56
*** nweinber has joined #openstack-nova20:56
spatelJust FYI - i am running "qemu-kvm-ev-2.12.0-33.1.el7_7.4.x86_64"  hope this version is fully stable20:56
artomPresumably it is, if you got from your OS's regular repos/sources20:57
artomBut I'm not a qemu developer, so I don't know off the top of my head (and qemu developers themselves probably don't)20:57
*** tbachman has joined #openstack-nova20:58
*** jmlowe has quit IRC21:00
*** nweinber has quit IRC21:04
*** spatel has quit IRC21:08
*** abaindur has joined #openstack-nova21:08
*** abaindur has quit IRC21:08
*** abaindur has joined #openstack-nova21:09
*** eharney has joined #openstack-nova21:15
*** mvkr has joined #openstack-nova21:20
*** dave-mccowan has quit IRC21:23
*** tbachman has quit IRC21:24
abaindurhello, seeing an issue with SRIOV. When we delete a VM, it's VF is released back, but then we see the VF running DHCP. This causes the VF to get assigned an IP and change default route out of the host, screwing things up21:32
abaindurDont know if this is openstack related, or NIC driver related, or some Linux setting21:32
abaindurJan 30 16:36:44 compute-02 systemd: Starting DHCP interface eth40...21:34
abaindurNetworkManager is disabled... and the physical NIC for the VFs has NM_CONTROLLED=no and BOOTPROTO=none21:35
*** jmlowe has joined #openstack-nova21:35
*** nicolasbock has quit IRC21:54
*** TristanSullivan has joined #openstack-nova21:57
*** jmlowe has quit IRC22:04
*** xek has quit IRC22:13
*** slaweq has quit IRC22:15
*** xek has joined #openstack-nova22:15
*** ociuhandu has joined #openstack-nova22:40
*** ociuhandu has quit IRC22:42
*** mlavalle has quit IRC22:43
*** ociuhandu has joined #openstack-nova22:43
*** xek has quit IRC22:47
*** mlavalle has joined #openstack-nova22:47
*** ociuhandu has quit IRC22:48
*** slaweq has joined #openstack-nova23:11
*** slaweq has quit IRC23:15
*** mlavalle has quit IRC23:30
*** jmlowe has joined #openstack-nova23:48
*** abaindur has quit IRC23:52
*** ociuhandu has joined #openstack-nova23:55
*** spatel has joined #openstack-nova23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!