Thursday, 2021-06-03

*** artom has quit IRC		00:00
*** artom has joined #openstack-nova		00:24
*** rloo has quit IRC		00:45
*** rloo has joined #openstack-nova		00:46
*** rloo has quit IRC		00:55
*** martinkennelly_ has quit IRC		00:56
*** martinkennelly has quit IRC		00:56
gmann	melwitt: replied on this - https://review.opendev.org/c/openstack/nova/+/762013/2/doc/source/configuration/policy-concepts.rst#220	01:19
gmann	melwitt: I agree on your point. I will add the current limitation for force_host and requested_destination	01:20
gmann	zero_disk and external-network seems good. later one depends on neutron policy but that is open for project reader also	01:20
opendevreview	Ghanshyam proposed openstack/nova master: Remove PROJECT_ADMIN limitation from zero-disk and external-network policy https://review.opendev.org/c/openstack/nova/+/794360	01:37
gmann	melwitt: ^^	01:38
opendevreview	Ghanshyam proposed openstack/nova master: Improve policy doc for supported scope info https://review.opendev.org/c/openstack/nova/+/762013	02:18
gmann	melwitt: stephenfin ^^ updated	02:18
*** whoami-rajat has quit IRC		03:06
*** brinzhang has joined #openstack-nova		03:25
*** brinzhang_ has joined #openstack-nova		03:28
*** brinzhang has quit IRC		03:28
*** vishalmanchanda has joined #openstack-nova		04:29
*** abhishekk has joined #openstack-nova		04:52
*** hemanth_n has joined #openstack-nova		05:02
opendevreview	melanie witt proposed openstack/nova master: Make test_refresh_associations_* deterministic https://review.opendev.org/c/openstack/nova/+/794396	05:06
melwitt	gmann: did you mean to remove the table when you went from PS2 to PS3? https://review.opendev.org/c/openstack/nova/+/762013/2..3/doc/source/configuration/policy-concepts.rst#b307	05:10
melwitt	added a comment on the review ^	05:17
*** suzhengwei__ has joined #openstack-nova		05:35
*** ralonsoh has joined #openstack-nova		06:01
*** luksky has joined #openstack-nova		06:13
*** slaweq has joined #openstack-nova		06:16
*** kaisers has joined #openstack-nova		06:18
*** kaisers_ has joined #openstack-nova		06:35
*** kaisers has quit IRC		06:42
*** whoami-rajat has joined #openstack-nova		06:43
*** slaweq[m] has joined #openstack-nova		06:50
*** slaweq has quit IRC		06:57
*** david-lyle has quit IRC		06:57
*** slaweq[m] is now known as slaweq		07:15
*** andrewbonney has joined #openstack-nova		07:18
*** tosky has joined #openstack-nova		07:20
*** lucasagomes has joined #openstack-nova		07:26
*** wenpingsong has joined #openstack-nova		07:26
*** rpittau\|afk is now known as rpittau		07:34
*** lucasagomes has quit IRC		07:38
bauzas	hmpf, reminder that we have an office hour in 15 mins , I guess	07:44
*** XinxinShen has joined #openstack-nova		07:53
gibi	yes	07:58
gibi	in 2 mins :)	07:58
gibi	I'm almost ready to start	07:58
*** martinkennelly has joined #openstack-nova		07:58
gibi	...	07:58
*** martinkennelly_ has joined #openstack-nova		07:58
gibi	#startmeeting nova_extra	08:00
opendevmeet	Meeting started Thu Jun 3 08:00:16 2021 UTC and is due to finish in 60 minutes. The chair is gibi. Information about MeetBot at http://wiki.debian.org/MeetBot.	08:00
opendevmeet	Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.	08:00
opendevmeet	The meeting name has been set to 'nova_extra'	08:00
gibi	welcome o/	08:00
suzhengwei__	\o/	08:01
XinxinShen	o/	08:01
gibi	... waiting a minute to lets everybody join	08:01
gibi	So a quick summary. We agree to have this meeting time slot one a Months in every first Thursday here in #openstack-nova	08:03
gibi	s/one/once/	08:03
gibi	but we haven't talked about yet what would be a good format for it	08:04
gibi	should we go through the normal meeting agenda	08:04
gibi	talking about bugs, CI, release, stable,	08:04
*** lucasagomes has joined #openstack-nova		08:04
gibi	or just go with an open agenda, like in an office hour	08:04
gibi	what is the preference of the people present?	08:05
gibi	I can do both if needed	08:05
suzhengwei__	This is the first extra meeting. My colleague didn't prepare for it enough.	08:07
gibi	suzhengwei__: no worries	08:07
suzhengwei__	Open agenda will be all right.	08:07
gibi	If no oppinions about the format then I would go with the simple open agenda	08:08
gibi	suzhengwei__: cool, then we are on the same page	08:08
suzhengwei__	ok	08:08
gibi	and we can change later if needed	08:08
gibi	I have one agenda point for today	08:08
gibi	just want to refresh the process we have in nova about the release	08:09
gibi	so we just had Milestone 1 last week	08:09
gibi	that that point of time there is no special deadline in nova	08:10
gibi	the next Milestone will happen in 5 weeks from now	08:10
gibi	at Milestone 2 we will do spec freeze	08:10
*** derekh has joined #openstack-nova		08:11
gibi	it means that if you have open specs then you have to get merged before M2 or it will need to be re-proposed to the Y release during the autumn	08:11
gibi	I'm planning to have a spec review day before M2	08:11
gibi	similarly how we had such day before M1	08:11
gibi	those features that got approved before M2 could be implemented and merged before M3	08:12
gibi	at M3 we will have feature freeze	08:12
bauzas	sorry for being late, got some issue with my laptop notification	08:12
gibi	bauzas: no worried :0	08:12
gibi	:)	08:12
gibi	basically this is the importnat deadlines	08:13
gibi	the exact dates are #link https://releases.openstack.org/xena/schedule.html	08:13
gibi	is there any question about these?	08:13
suzhengwei__	Got it.	08:14
gibi	cool	08:15
gibi	that was what I prepared for today.	08:15
bauzas	next open office hour would be around the spec freeze	08:15
gibi	Is there any topic you want to discuss?	08:15
bauzas	so, I have a concern	08:16
gibi	bauzas: god point	08:16
gibi	good even	08:16
gibi	bauzas: tell us	08:16
bauzas	how can we help contributors that are not in the nova meeting for their own specs ?	08:16
bauzas	how can we discuss ?	08:16
gibi	bauzas: what problem do you see that we need to dicuss in a meeting? stuck reviews?	08:17
bauzas	and if we want to have priorities between some reviews, how could we know which ones ?	08:17
bauzas	gibi: my point is that sometimes it's nice to discuss directly in IRC when you have some spec questions	08:18
gibi	sure	08:18
gibi	but it does not need to be a in the form of a meeting	08:18
bauzas	yup	08:18
bauzas	so, I wonder how we could help the contributors that aren't around in general	08:18
gibi	still if there is question now, I'm happy to hear them even during this meeting	08:18
*** suzhengwei__ has quit IRC		08:18
gibi	bauzas: I'm OK to do discussion primarily in the spec review	08:19
*** suzhengwei has joined #openstack-nova		08:19
gibi	and if we hit a wall there then we can try to find a common time between the spec author and the reviewers to resolve	08:19
gibi	the block	08:19
bauzas	yup	08:20
gibi	suzhengwei, XinxinShen do you have a view on this?	08:20
gibi	do you feel it is hard to discuss issues about the specs?	08:21
bauzas	in general, it takes more than one week	08:23
bauzas	for example, when I review a spec, I provide some comment	08:23
bauzas	but then the reply could be on the next day	08:23
bauzas	and then I'd only see it honestly by the end of the week	08:24
suzhengwei	Is this channel often open to all? I means if i want to get help, would someone react in time?	08:24
bauzas	suzhengwei: yes, in general, we do it this way	08:24
suzhengwei	good	08:24
bauzas	suzhengwei: for example, when I'm reviewing some gibi's spec, I'm pinging him	08:24
gibi	suzhengwei: it depends on the time zone. like I tend to be here between UTC 7:00 - UTC 17:00 on workdays	08:25
bauzas	telling him that I had some questions	08:25
bauzas	so, sometimes we directly discuss on IRC for a spec	08:25
suzhengwei	cool	08:25
bauzas	but like gibi said, we're both on the same TZ	08:25
bauzas	so it's simple	08:25
gibi	suzhengwei: I keep my client up during the night so I will see pings next day	08:26
bauzas	me too	08:26
XinxinShen	cool, We should ensure that the spec author and reviewer can communicate in time.	08:28
gibi	suzhengwei: if you ping us, one thing to help is not just pinging but stating your problem right away	08:28
suzhengwei	yes	08:29
suzhengwei	I am doing some work about instance HA, and interested in the topic "Support vm evacuation while server status is suspended, paused".	08:30
suzhengwei	I wonder if some one has processed it.	08:31
bauzas	I still have open specs to look at	08:31
suzhengwei	If not, I would like to do it.	08:31
*** luksky has quit IRC		08:31
gibi	suzhengwei: interesting ideas	08:31
*** luksky has joined #openstack-nova		08:32
gibi	suzhengwei: I think we don't support these today, but if with some compromise we could	08:32
gibi	we will loose the in memory state of the paused instance but we could rebuild it still on another compute host	08:33
gibi	and if the source compute is dead already then the in memory state is lost anyhow	08:33
suzhengwei	host failure triggers evacuation. And active instance loose the in memory state too.	08:34
gibi	suzhengwei: yes, so I think it is OK to lose that for the pause instance too	08:34
gibi	suzhengwei, bauzas: does suspend saves some state to the disk?	08:34
bauzas	good question	08:34
bauzas	I honestly don't have the answer straight out of my mind	08:35
gibi	anyhow if it save something to disk and that disk is on shared storage then we might even recover that saves state on the destination host	08:35
bauzas	the crucial bit to remember with instance HA is that the host is already done	08:35
gibi	but as a first step I would loose that too	08:35
bauzas	gone*	08:35
gibi	bauzas: yepp	08:35
bauzas	so, yeah, ephemeral storage can't be somehow persisted	08:35
gibi	except if it is on shared storage ^^ ;)	08:36
bauzas	you need to have either shared storage or volumes	08:36
gibi	yepp	08:36
bauzas	yeah, but in general, you need to assume a crash	08:36
bauzas	so any memory that's not synced is lost	08:36
gibi	bauzas: yeah, this is why first I would assume that the suspended state is lost as well	08:36
gibi	to avoid an inconsistent suspended state to be loaded	08:37
bauzas	if suspend stores on disk, we're ok	08:37
bauzas	kashyap: around ?	08:37
kashyap	bauzas: Mornin, yes	08:37
kashyap	How can I be useful? :)	08:37
bauzas	kashyap: we are in office hour and we have a question about suspended instances	08:37
* kashyap reads back		08:38
bauzas	with the libvirt driver and qemu, what happens to the memory state when suspending ?	08:38
bauzas	do we suspend on disk ?	08:38
bauzas	I'd be inclined to say so	08:38
kashyap	bauzas: gibi: Yes: suspend usually indeed means save-the-state-to-a-file-on-disk	08:38
kashyap	Your inclination is correct :)	08:39
bauzas	the problem is that we can't tell whether the instance is on shared storage or not	08:39
gibi	kashyap: and what do you think, moves such suspended state between compute host make sense?	08:39
bauzas	gibi: keep in mind evacuate is a rebuild	08:40
kashyap	gibi: You mean moving such suspended state between different compute hosts make sense?	08:40
gibi	kashyap: yeaht that is my queston	08:40
gibi	bauzas: in case of suspend we see the vm_state on the dest being suspended so we can look for the state file on the disk. if it is there then we know that it was on shared storage	08:41
bauzas	gibi: sure but then we leak the state of the host	08:41
bauzas	this isn't predictable	08:41
gibi	OK, I agree this can be a can of worm	08:42
gibi	suzhengwei: in case of evacuating a suspended VM, is it OK to you to loose the suspended state?	08:42
kashyap	gibi: I need to think a bit more about it. (libvirt has managedSave() API that does the suspend thingie, which already Nova uses. So we have the primitives...)	08:44
suzhengwei	If host down, active suspend instance both loose their memory.	08:44
kashyap	gibi: suzhengwei: What is the main use-case here? The ability to start suspended instances on any compute host from a given pool?	08:44
gibi	suzhengwei: if you don't want to recover the suspended state that is saved to disk, then I think your proposal is pretty simple and straight forward	08:44
suzhengwei	Instance Ha, try best to recover the workload as much as possible.	08:45
bauzas	interestingly, I found some nova admin docs https://docs.openstack.org/nova/latest/admin/node-down.html	08:45
gibi	kashyap: we looked at it from evacuation perspective. VM is suspended to disk (on shared storage), the host dies, user evacuates VM	08:45
kashyap	gibi: I see; that makes sense	08:46
bauzas	gibi: I honestly feel we can just support recreating a new instance	08:46
kashyap	bauzas: Isn't that what already 'rebuild' is?	08:47
kashyap	Ah, you said that already above :)	08:47
bauzas	kashyap: yup, the question was about the memory state	08:47
suzhengwei	If host down, the suspend instance can be active agian on the origin node. So I think it makes sence to evacuate suspend instances.	08:47
suzhengwei	can not	08:48
bauzas	I guess here suzhengwei's concern is that we limitate evacuate on active instances	08:48
bauzas	right?	08:48
bauzas	that's the problem we're trying to solve ?	08:48
gibi	I think so	08:48
suzhengwei	yes	08:48
gibi	and I'm totally supportive to extend evac to support paused and suspended instances. It is simple if we allow loosing the running state	08:49
bauzas	I just remembered we have a --on-shared-storage flag https://docs.openstack.org/nova/latest/admin/evacuate.html#evacuate-a-single-instance	08:49
bauzas	since evacuate is an admin action, op can use it	08:49
bauzas	on purpose	08:49
bauzas	so we already do the check automatically	08:50
gibi	bauzas: onSharedStorage is deprecated in 2.13	08:50
gibi	bauzas: today we automatically detect it I guess	08:51
bauzas	gibi: because we detect this ?	08:51
bauzas	yeah	08:51
gibi	"Starting since version 2.14, Nova automatically detects whether the server is on shared storage or not. Therefore this parameter was removed."	08:51
gibi	yepp	08:51
bauzas	ok, so I guess we can consider adding suspend	08:51
gibi	suzhengwei: I suggest to propose a small spec about this. I'm happy to review it	08:51
bauzas	if the target host is on shared storage, we could just try to boot with the suspended state	08:51
bauzas	for paused, the implication would be that the evacuated instance would become active	08:52
bauzas	for suspend, too	08:52
gibi	bauzas: active, or stopped	08:52
gibi	bauzas: we can decide	08:52
bauzas	yup, that's the point	08:52
gibi	bauzas: but true, it cannot be pasued any more	08:52
gibi	paused	08:52
*** Luzi has joined #openstack-nova		08:52
gibi	I don't want to stop the discussion, but we have 8 minutes left. If there any other topic to discuss?	08:53
gibi	/If/Is/	08:53
kashyap	bauzas: gibi: One last:	08:53
gibi	kashyap: go	08:53
suzhengwei	I think stopped is better. No matter pause or suspend, users can not acess the instance directly.	08:53
gibi	suzhengwei: I can accept that	08:53
kashyap	gibi: suzhengwei: On whether it makes sense of moving suspended instances between compute hosts, a thumb-rule can be: "follow the same rules for hardware matching as for a live migration between the hosts"	08:54
kashyap	(I mean, to uncover any "gotchas")	08:54
gibi	kashyap: ahh you have a point, this state can be hw dependent	08:54
kashyap	FWIW, I also just checked the above w/ a QEMU migration developer; and he agrees.	08:54
bauzas	kashyap: since evacuate is a rebuild, we can't predict this	08:54
bauzas	gibi: sorry, I wasn't explicit but when I said 'we're gonna try to unsuspend from disk", I was thinking of hardware capabilities	08:55
gibi	OK, then I propose not to try to recover the suspended state during evac. At list not in the first step	08:55
kashyap	bauzas: gibi: Hm, so looks like this needs to be fleshed out in a design document	08:55
gibi	/list/least/	08:55
bauzas	kashyap: the evacuate workflow is waaaaay different from live-migrate	08:56
bauzas	you can't just check the source host at first ;)	08:56
bauzas	and compare both	08:56
bauzas	the scheduler is just giving you a target and then good luck with it	08:56
kashyap	bauzas: I see; fair enough	08:56
gibi	so in summary	08:57
bauzas	but yeah, we're 4 mins	08:57
bauzas	left	08:57
gibi	so in summary	08:57
gibi	suzhengwei: please propose a spec. I don't see any problem supporting evac for paused and suspended VMs. But they will lose the in memory or suspended state. They will be fresh VMs on the dest host in stopped state	08:57
kashyap	Yeah; makes sense.	08:58
suzhengwei	I will.	08:58
kashyap	(On spec)	08:58
gibi	suzhengwei: cool, thanks	08:58
gibi	any last words before we stop the meeting? ;)	08:58
suzhengwei	nothing from me.	08:59
gibi	XinxinShen: ?	08:59
XinxinShen	nothing for me. thanks.	08:59
gibi	then thanks for joining. please continue discussion if needed	08:59
gibi	I just stop the meeting log here	09:00
gibi	#endmeeting	09:00
opendevmeet	Meeting ended Thu Jun 3 09:00:04 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)	09:00
opendevmeet	Minutes: http://eavesdrop.openstack.org/meetings/nova_extra/2021/nova_extra.2021-06-03-08.00.html	09:00
opendevmeet	Minutes (text): http://eavesdrop.openstack.org/meetings/nova_extra/2021/nova_extra.2021-06-03-08.00.txt	09:00
opendevmeet	Log: http://eavesdrop.openstack.org/meetings/nova_extra/2021/nova_extra.2021-06-03-08.00.log.html	09:00
bauzas	right on time	09:00
kashyap	gibi: Remind me again, what's the case against trying to "resume" the saved state on the destination, instead of a fresh rebuild (which loses the state)? Because it also involves dealing with the state file transportation?	09:02
gibi	kashyap: if we need to take care of HW compatibility then we have to blow up the evac process	09:03
kashyap	gibi: Oh, yikes; yes. How come I forgot so quickly	09:03
gibi	live migration is special :)	09:03
kashyap	Tell me all about it.	09:04
gibi	:D	09:04
kashyap	gibi: I'm not the person for GIFs, but this captures it: https://kashyapc.fedorapeople.org/Live-Migration.gif	09:05
kashyap	(Not sure if it's loading correctly for you)	09:05
gibi	ooh, that is good	09:06
gibi	I will distribute it :)	09:06
kashyap	You're welcome ;-)	09:06
gibi	thanks	09:06
kashyap	gibi: Does it load smoothly for you? For me not here, strangely	09:06
gibi	it works for me from firefox	09:06
gibi	smooth and on auto repeat	09:06
kashyap	Ah, nod. (It goes in super slow-motion on FF and Chromium.)	09:07
bauzas	gibi: I think we have a formal contract with evacuate	09:10
bauzas	gibi: this is badly named as it's a recreate	09:10
bauzas	if you really want to have persisted storage, use volumes	09:10
bauzas	because if not, we'll recreate the instance, even on shared storage	09:11
*** wenpingsong has quit IRC		09:12
gibi	yeah, we should just drop evac and change rebuild to detect if the compute is dead and then rebuild the instance on anther host. (partly joking)	09:12
bauzas	gibi: and chicken will have teeth	09:13
gibi	OK, I guess this is some French saying :)	09:14
bauzas	but yeah, this extra API action is confusing	09:14
bauzas	and the wording is horrible	09:14
bauzas	gibi: for your european culture https://blogs.transparent.com/french/when-pigs-fly-and-chicken-have-teeth-impossible-french-expressions/	09:15
gibi	thank you	09:16
bauzas	gibi: I guess you probably have other expressions like this in your own language	09:16
*** XinxinShen has quit IRC		09:17
bauzas	either way, dropping for a few	09:18
gibi	Ok for me this chikent thing would be : "when red snow falls (... with black dots) "	09:19
gibi	which was OK in my childhood, but recently we got sand from Sahara sometimes and that can make the rain red, so I can imagine that we can get red snow in the same way	09:21
*** abhishekk has quit IRC		09:24
*** swp20 has joined #openstack-nova		09:36
bauzas	gibi: indeed, when you live close to the mountains, you see red snow ;)	09:46
bauzas	actually, more likely orange snow	09:46
gibi	yeah it is orange-red rain that happens here	09:46
gibi	it makes everything dirty :/	09:47
bauzas	gibi: https://photos.app.goo.gl/nCCAirhi8sZ3vDyc9 some pic from last winter ;)	09:47
*** jangutter has joined #openstack-nova		09:48
gibi	It looks like it made with a sepia filter :)	09:48
bauzas	indeed, but no, just a regular pic ;)	09:48
gibi	crazy	09:49
bauzas	and this was on a morning, not during sunset	09:49
bauzas	imagine a lost skiing resort as the lifts were closed with this sky	09:50
bauzas	probably a good place and time for a new Hitchcook movie	09:51
bauzas	I loved the abandoned atmosphere :)	09:51
*** jchorin_ has joined #openstack-nova		09:52
*** jangutter_ has quit IRC		09:55
*** jangutter has quit IRC		10:01
*** jangutter has joined #openstack-nova		10:01
sean-k-mooney	its almost mars like	10:02
sean-k-mooney	how did the first meeting go this morning	10:03
* sean-k-mooney scrolls back		10:03
sean-k-mooney	suzhengwei: we spoke about evacuation for suspended and stopped instace at the ptg. i assume you added that topic but were not able to attend when we discussed it	10:08
sean-k-mooney	suzhengwei: the feedback at the time was to submit a spec, intoduce a new microversion and have the state transtion as follows	10:09
sean-k-mooney	Shelved -> Shelved or Shelved_offloaded ?	10:09
sean-k-mooney	Paused -> Stopped	10:09
sean-k-mooney	Suspended -> Stopped	10:09
sean-k-mooney	active -> active	10:09
sean-k-mooney	actully Shelved -> Shelved or Shelved_offloaded ? is not relevnet	10:10
sean-k-mooney	well ok it kind of is but in a subtel way	10:11
sean-k-mooney	shelved instance still have a host but you have a choice of if you will shleve it to a different host or shleve offload it	10:11
sean-k-mooney	suzhengwei: hence the ? as its an open question which would be the correct behavior but that should be adressed in the spec	10:12
sean-k-mooney	suzhengwei: im happy to review the sepc if you propose it so feel free to ping me or add me to the review once its pushed.	10:13
*** jangutter has quit IRC		10:14
*** osmanlic- has joined #openstack-nova		10:14
*** jangutter_ has joined #openstack-nova		10:15
*** osmanlicilegi has quit IRC		10:17
*** bauzas has quit IRC		10:23
*** bauzas has joined #openstack-nova		10:23
gibi	sean-k-mooney: thanks sean-k-mooney for remembering that we talked about that on the ptg, I already forgot	10:23
gibi	but we concluded the same thing today :)	10:23
sean-k-mooney	gibi: well a least we were consitnet in our feedback	10:24
gibi	yepp	10:24
sean-k-mooney	so thats a good sign at least	10:24
*** suzhengwei has quit IRC		10:24
*** admin1 has joined #openstack-nova		10:29
sean-k-mooney	lyarwood: by the way since https://review.opendev.org/c/openstack/nova/+/794252 is chanigng the virt dirver interface its nice to email the list about it	10:39
sean-k-mooney	which is ironic since we dont support out of tree virt drivers	10:39
lyarwood	sean-k-mooney: ack yeah will do	10:45
*** suzhengwei has joined #openstack-nova		10:53
*** suzhengwei has left #openstack-nova		10:58
*** halali_ has quit IRC		11:11
*** osmanlic- has quit IRC		11:20
*** osmanlicilegi has joined #openstack-nova		11:20
*** hemna has quit IRC		11:29
*** halali_ has joined #openstack-nova		11:32
*** halali_ has quit IRC		11:40
hemanth_n	hi can I get final reviews + workflow on this clean backport patch on queens https://review.opendev.org/c/openstack/nova/+/761825 .. already Elod reviewed and gave +2, thanks elod	11:43
*** hemna has joined #openstack-nova		11:45
sean-k-mooney	melwitt: lyarwood can you take a look at ^	11:50
sean-k-mooney	granted at this point we likely wont pull this downstream since we wont have any new releases based on 13 at this point but it would still be good to fix for those on queens still	11:51
jchorin_	Hi everyone! I am trying to enable TLS on the nova API. Setting `cert` and `key` in nova.conf did not change anything except for the novncproxy. Is there another way to do it?	12:10
*** hemanth_n has quit IRC		12:10
*** halali_ has joined #openstack-nova		12:13
sean-k-mooney	jchorin_: is the nova api using the copy of the nova.conf you updated	12:15
sean-k-mooney	jchorin_: some installer create mulitple	12:15
jchorin_	Yes that is the one installed with the nova package and used by the process when started	12:18
sean-k-mooney	jchorin_: how are you running nova-api	12:19
sean-k-mooney	with the eventlet console script or via a wsgi server like appach or uwsgi	12:19
sean-k-mooney	* apache mod_wsgi	12:20
sean-k-mooney	jchorin_: i assume you set it in the default section https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.cert	12:22
sean-k-mooney	jchorin_: if its runnign under uwsgi or mod_wsgi you likely need to do the cert configurtion in the wsgi server instead	12:22
sean-k-mooney	jchorin_: like this https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova/templates/nova-api-wsgi.conf.j2#L47-L51	12:24
jchorin_	I added it in the default section yes.	12:25
jchorin_	It is started with `python3 /usr/bin/nova-api ...`	12:25
jchorin_	But there is no nova file in apache's site-available	12:25
sean-k-mooney	gibi: dansmith do ye know if we ever supported tls nativly when runing nova-api directly issted of under an external wsgi server	12:26
sean-k-mooney	jchorin_: yes unless you created one there would not be	12:27
sean-k-mooney	jchorin_: that is running the api using hte python built in webserver with eventlets handeling multiplexing connections	12:28
sean-k-mooney	jchorin_: kolla ansible runs it nova-api using apache and mod_wsgi and https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova/templates/nova-api-wsgi.conf.j2 is the template they use to render the apache configuration for the site	12:29
sean-k-mooney	jchorin_: i woudl recommend doing that if you are manually installing instead as it perfroms better	12:30
sean-k-mooney	well in many cases at least	12:30
jchorin_	You mean to use wsgi? Yes it is entirely fine for me to use it. Do I only need to add this file to the site-available?	12:31
sean-k-mooney	well that s in jinga 2 format so you need to add a rendered equivalent but yes	12:31
sean-k-mooney	then symlink it to site-enabled	12:32
jchorin_	However, the server will still be running right?, So both TLS and non-TLS will be reachable	12:32
gibi	sean-k-mooney: hm, I'm not expert on the TLS front, what deploymnet I see was basically using a TLS proxy in front of the API servies	12:32
*** Luzi has quit IRC		12:32
sean-k-mooney	gibi: yep i think that is what we did before we could run the api with a wsgi server via a web server	12:33
sean-k-mooney	jchorin_: what os are you on	12:34
jchorin_	Ubuntu	12:34
sean-k-mooney	ok are you following https://docs.openstack.org/nova/wallaby/install/controller-install-ubuntu.html#install-and-configure-components	12:34
sean-k-mooney	that is indeed mssing a tls section	12:35
jchorin_	This one yes	12:35
sean-k-mooney	its also using sytemd to run nova-api presumable directly as you noted above	12:36
*** kaisers has joined #openstack-nova		12:43
*** swp20 has quit IRC		12:44
jchorin_	Alright, I'll take a look, thanks for your help!	12:44
sean-k-mooney	i have just been looking and out side of the installer that have implemented support for tls endpoing like kolla-ansible and tripleo i do not see any docs for how to deploy openstack with tls endpoints	12:45
sean-k-mooney	so this is an operational gap in our docs as a whole	12:46
jchorin_	Yes, I could not find any information except for the `cert` and `key` settings in the conf	12:47
sean-k-mooney	the general approch however seams to be the same across most installer project, use a webserver to handel the tls/ssl laywer and hand off to the openstack service	12:47
*** kaisers_ has quit IRC		12:48
*** nicolasbock has joined #openstack-nova		12:55
*** rloo has joined #openstack-nova		12:56
*** rloo has quit IRC		12:59
*** rloo has joined #openstack-nova		12:59
*** jchorin_ has quit IRC		13:07
*** jchorin has joined #openstack-nova		13:11
sean-k-mooney	gibi: kashyap fyi i filed the whishlist bug for the realtime emulator threads here https://bugs.launchpad.net/nova/+bug/1930706	13:11
opendevmeet	Launchpad bug 1930706 in OpenStack Compute (nova) "nova allows suboptimal emulator tread pinning for realtime guests" [Wishlist,Triaged]	13:11
sean-k-mooney	just closing out that ar form the last meeting	13:11
* kashyap clicks		13:11
sean-k-mooney	speaking of which i think the new review priortiy docs have merged? im going to go update the patch to project config	13:12
jchorin	sean-k-mooney : Unfortunately I got "could not bind to address 0.0.0.0:8774", as the serverlet is still running and I do not use kolla. Is there a configuration setting for changing the port or should I do it differently?	13:12
sean-k-mooney	you should stop the serverlet	13:14
sean-k-mooney	so stop the nova-api systemd service and disable it	13:14
sean-k-mooney	then run it only under apache	13:15
kashyap	sean-k-mooney: Thanks for writing it up; it might be worth it to explicilty flag the workaround for operators who're not well-versed in this area	13:15
sean-k-mooney	kashyap: i think i said that in the comment i left	13:16
sean-k-mooney	e.g. use emulator thread policy	13:16
kashyap	sean-k-mooney: Ah, the second comment, yes	13:16
sean-k-mooney	i realised i did not add it orginally	13:17
kashyap	sean-k-mooney: The "not" at the start confused me; I know you meant it as "note"	13:17
sean-k-mooney	i think the bug desciription can be updated if you want to add it	13:17
sean-k-mooney	kashyap: ya i did	13:17
sean-k-mooney	i just fixed that	13:17
kashyap	Ah, cool.	13:18
sean-k-mooney	but if you want ot update the description please feel free too	13:18
kashyap	sean-k-mooney: Yeah, good idea to update the description; /me goes to do it :)	13:18
kashyap	sean-k-mooney: Mind if I also touch up a few typos?	13:19
sean-k-mooney	go for it	13:19
kashyap	sean-k-mooney: One clarif:	13:20
kashyap	In this bit: "When the emulator thread is spawned on core 1 since it has less priority then the vcpu thread it will only run if the guest vcpu idels resulting in the iablity for qemu to process device attach"	13:20
kashyap	sean-k-mooney: Can you pleease rephrase the last part? The "idels and "iability" are ambigious to parse for me	13:21
kashyap	sean-k-mooney: Added the "Workaround" section at the end of the description; please double-check.	13:29
dansmith	sean-k-mooney: no I don't know	13:32
ozzzo	Sean has helped me a lot so I've gotten pretty good at translating Seanglish: "guest vcpu idles resulting in the inability"	13:41
kashyap	ozzzo: Ah, thanks. I'm mostly pretty good at parsing him, sometimes I miss the mark :)	13:42
*** abhishekk has joined #openstack-nova		13:44
kashyap	sean-k-mooney: Disregard my above question (thx to ozzzo); updated the bug.	13:46
sean-k-mooney	ozzzo: hehe seanspeak is a select dialect :)	13:47
opendevreview	Merged openstack/nova stable/queens: Update pci stat pools based on PCI device changes https://review.opendev.org/c/openstack/nova/+/761825	14:00
*** dklyle has joined #openstack-nova		14:11
gmann	melwitt: no, it was by mistake. thanks for catching it. fixing it now	14:32
opendevreview	Ghanshyam proposed openstack/nova master: Improve policy doc for supported scope info https://review.opendev.org/c/openstack/nova/+/762013	14:46
gmann	melwitt: stephenfin updated ^^	14:47
* stephenfin looks		14:51
*** abhishekk has quit IRC		15:08
*** abhishekk has joined #openstack-nova		15:08
*** abhishekk has quit IRC		15:08
*** erlon has quit IRC		15:13
*** jchorin has quit IRC		15:25
gibi	FYI: I will be mostly off on Monday and Tuesday next week, but I will chair the nova meeting on Tuesday	15:47
opendevreview	Balazs Gibizer proposed openstack/nova master: Detect extended_resource_request neutron API extension https://review.opendev.org/c/openstack/nova/+/793618	16:06
opendevreview	Balazs Gibizer proposed openstack/nova master: Reject server create with extended resource req https://review.opendev.org/c/openstack/nova/+/793619	16:06
opendevreview	Balazs Gibizer proposed openstack/nova master: Reject server operations with extended resource req https://review.opendev.org/c/openstack/nova/+/793620	16:06
opendevreview	Balazs Gibizer proposed openstack/placement master: Add support for RP re-parenting and orphaning https://review.opendev.org/c/openstack/placement/+/784020	16:08
opendevreview	Balazs Gibizer proposed openstack/nova master: Add same_subtree field to RequestLevelParams https://review.opendev.org/c/openstack/nova/+/791503	16:08
opendevreview	Balazs Gibizer proposed openstack/nova master: Bump min placement microversion to 1.36 https://review.opendev.org/c/openstack/nova/+/791504	16:12
*** rpittau is now known as rpittau\|afk		16:12
bauzas	gibi: ack, and thanks for chairing	16:13
bauzas	last time I did, I felt the breath of the loneliness :p	16:13
opendevreview	Balazs Gibizer proposed openstack/nova master: Support same_subtree in allocation_canadidate query https://review.opendev.org/c/openstack/nova/+/791505	16:13
gibi	bauzas: ahh I remember that meeting, I sit in the inner park of the hospital waiting my turn in the CT. It was a bit surreal both in IRC space and in real space too :)	16:15
opendevreview	Balazs Gibizer proposed openstack/nova master: Support the new port resource_request format https://review.opendev.org/c/openstack/nova/+/787208	16:15
bauzas	gibi: :D	16:16
opendevreview	Balazs Gibizer proposed openstack/nova master: Transfer RequestLevelParams from ports to scheduling https://review.opendev.org/c/openstack/nova/+/791506	16:16
opendevreview	Balazs Gibizer proposed openstack/nova master: [func test] ports with both bw and pps resources https://review.opendev.org/c/openstack/nova/+/792394	16:16
gibi	this time I will be off to the lake during the weekend + Monday. But I need to be back in the city on Tuesday so I can do the meeting	16:18
gibi	(or I could do the meeting from the lake side but meh :D)_	16:18
opendevreview	Balazs Gibizer proposed openstack/nova master: [func test] move unshelve test to the proper place https://review.opendev.org/c/openstack/nova/+/793621	16:19
sean-k-mooney	gibi: you you know one of us could run the meeting for you and you could relax	16:20
gibi	sean-k-mooney: thanks, yes. but no need this time	16:20
gibi	sean-k-mooney: the original plan was that we are back on Wednesday morning, and then I would ask somebody to run it	16:21
gibi	sean-k-mooney: but turned out that we have to get back on Tuesday morning	16:21
*** kaisers has quit IRC		16:22
gibi	so I think I will use Tuesday to do some chores and meeting fits well to that list :)	16:22
*** ralonsoh has quit IRC		16:24
*** lucasagomes has quit IRC		16:25
melwitt	gibi: easy test-requirements update for placement, dunno if you also ran into this when running func tests the first time https://review.opendev.org/c/openstack/placement/+/787289	16:31
opendevreview	Balazs Gibizer proposed openstack/nova master: Remove unnecessary mocks from unit test https://review.opendev.org/c/openstack/nova/+/794637	16:35
gibi	melwitt: I don't remember I saw it but the change itself does not hurt	16:36
gibi	could be that I have that package already locally	16:37
gibi	yepp, I have it already that package installed globally in my dev machine	16:39
gibi	that is why I did not see it	16:39
opendevreview	Lee Yarwood proposed openstack/nova master: libvirt: Set driver_iommu when attaching virtio devices to SEV instance https://review.opendev.org/c/openstack/nova/+/794639	16:49
lyarwood	^ should be WIP, I'll finish it up later with volume attach tests	16:51
lyarwood	and maybe a seperate functional test	16:51
sean-k-mooney	lyarwood: the set_driver_iommu_for_device should be called form vif_driver.get_config more then likely	17:04
sean-k-mooney	lyarwood: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py#L560	17:06
lyarwood	sean-k-mooney: I'm assuming that wasn't done originally to ensure all devices end up with it set not just when disks and vifs we define	17:06
*** derekh has quit IRC		17:06
sean-k-mooney	lyarwood: well curently you are just alwasy checkign it without considering the vif model correct	17:07
lyarwood	sean-k-mooney: no that's checked in the designer	17:07
lyarwood	sean-k-mooney: at least that it's virtio	17:08
sean-k-mooney	ya just looked at that now	17:08
sean-k-mooney	i dont see any reason not to put it in get config other then we will need to store the config and then return it	17:09
sean-k-mooney	although you probably could do it here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py#L528	17:09
sean-k-mooney	we really need to get rid of the non os-vif code path	17:11
lyarwood	sean-k-mooney: I'd rather do this explicitly in both hot plug flows first and then refactor the vif part later if you think it's worth it	17:11
lyarwood	sean-k-mooney: we can't do this in the volume drivers as we don't pass in the instance so it would be weird to change this just for vifs IMHO	17:12
sean-k-mooney	well i dont like spreading out the config genration like that	17:12
sean-k-mooney	i dont think that the attach or detach function should have to care about this explictly	17:12
sean-k-mooney	to me that is a poor speeration of concerns	17:13
lyarwood	Yup I agree but the original SEV code was already doing this during spawn	17:15
lyarwood	how about we fix this and then refactor both sets of config generation to do this correctly	17:15
lyarwood	I really don't want to do that as part of the bugfix we need to backport	17:16
sean-k-mooney	we could yes by passing the flavor/image metadata to _get_volume_config	17:16
sean-k-mooney	and then moving this into vif_driver.get_config	17:16
lyarwood	right I'm not going to do that in the backportable change	17:16
lyarwood	someone else can	17:16
sean-k-mooney	well honestly its not really a big change is it	17:17
lyarwood	but to me that's something that can follow on master	17:17
lyarwood	_get_volume_config isn't generating the actual config so if we were to do the same thing as vifs we'd be passing that down into the volume drivers right?	17:18
lyarwood	otherwise there's no difference to what the change is already doing in attach_volume	17:18
sean-k-mooney	there is a change presumable _get_volume_config is used for every operation that gets the confi for a volume	17:19
sean-k-mooney	as vif_driver.get_config is	17:20
sean-k-mooney	so if we put the if and call to the desinger in _get_volume_config and vif_driver.get_config	17:20
sean-k-mooney	we can never forget to do it	17:20
sean-k-mooney	what i dislike about your current patch is that it leave open the need to do this on all relevent code paths	17:21
lyarwood	the original did that	17:21
lyarwood	anyway let me respin and see what the fallout is	17:22
sean-k-mooney	did what	17:22
lyarwood	nvm	17:22
sean-k-mooney	for get_volume config its just passing the info and doing the if here right https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1887	17:24
lyarwood	right I thought you were suggesting changing the actual get_config methods in the volume drivers	17:25
lyarwood	this is still going to cause test fallout that I'd rather avoid for backports but I'll give it a go	17:25
sean-k-mooney	no	17:25
sean-k-mooney	and for vif.py its just replaceing the returns here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/vif.py#L578-L601	17:26
sean-k-mooney	with an asignment to conf and dong the if then returning	17:26
sean-k-mooney	lyarwood: the rest of the approch im ok with	17:26
sean-k-mooney	lyarwood: due to the changing of the signiture of an privete funcion _get_volume_config	17:27
sean-k-mooney	i would hope the fallout from tha twould be relitivly minimal	17:28
lyarwood	sean-k-mooney: what about get_base_config for the vif?	17:28
sean-k-mooney	you dont need to modify that	17:28
sean-k-mooney	you could but you cna do it in get_config	17:28
sean-k-mooney	lyarwood: that said https://github.com/openstack/nova/blob/cd084aeeb8a2110759912c1b529917a9d3aac555/nova/virt/libvirt/vif.py#L180	17:29
sean-k-mooney	it has the flaovr and image_meta	17:29
sean-k-mooney	and that shoudl be called in all code paths	17:29
lyarwood	right I was looking at the end of that method	17:30
sean-k-mooney	so ya you can just add it there	17:30
sean-k-mooney	lyarwood: to be honest i tought it was already there	17:30
lyarwood	ah the only issue is checking if sev is enabled from here is going to be awkward	17:31
sean-k-mooney	you need to do it here though https://github.com/openstack/nova/blob/cd084aeeb8a2110759912c1b529917a9d3aac555/nova/virt/libvirt/vif.py#L192	17:31
sean-k-mooney	lyarwood: becuase you dont have aceese to sev_enabled?	17:31
lyarwood	_sev_enabled yeah	17:31
sean-k-mooney	that is just https://github.com/openstack/nova/blob/cd084aeeb8a2110759912c1b529917a9d3aac555/nova/virt/libvirt/driver.py#L6716-L6743	17:32
sean-k-mooney	you can put that in desinger.py	17:32
sean-k-mooney	if you wanted too	17:32
lyarwood	right but that's checking _host	17:32
sean-k-mooney	well no	17:32
sean-k-mooney	it need _host	17:32
sean-k-mooney	ok well we have _host in vif.yp	17:33
lyarwood	lol	17:33
sean-k-mooney	but if you want to do this in a follow up then ok	17:33
sean-k-mooney	i think this would still be a clearer way to do this	17:33
sean-k-mooney	we likely woudl just want to pass host as a parmater to sev_enabled	17:34
sean-k-mooney	actully you could put the if check in set_driver_iommu_for_device	17:35
sean-k-mooney	and pass host, flavor, image_meta and conf	17:35
sean-k-mooney	anyway i guess leave it for now	17:36
sean-k-mooney	at some point we really need to get ride of the desginer.py as well	17:37
lyarwood	tbh this is the first time I've really looked at it	17:37
sean-k-mooney	the designer.py has always been a pet pev of mine	17:37
sean-k-mooney	i want driver.py to be smaller	17:38
sean-k-mooney	but designer.py does not relaly add value today	17:38
lyarwood	brb baby bedtime	17:38
sean-k-mooney	it would be beter to merge it with config.py	17:38
sean-k-mooney	it was seperated out so that config.py could just be the xml object creation and designer.py woudl have the logic to do some of the compostion	17:39
sean-k-mooney	but we never too the xml logic out of vif.py or the volume modules	17:40
sean-k-mooney	so really designer just complicated things	17:40
opendevreview	Merged openstack/nova stable/wallaby: rbd: Get rbd_utils unit tests running again https://review.opendev.org/c/openstack/nova/+/790836	17:42
opendevreview	melanie witt proposed openstack/nova stable/victoria: rbd: Get rbd_utils unit tests running again https://review.opendev.org/c/openstack/nova/+/794624	18:04
melwitt	fix for intermittent unit test failure if anyone interested https://review.opendev.org/c/openstack/nova/+/794396	18:31
*** vishalmanchanda has quit IRC		18:39
sean-k-mooney	melwitt: oh it was somethime slow an now did not match?	18:53
sean-k-mooney	or something like that that change the behavior of _refresh_associations	18:53
melwitt	sean-k-mooney: yeah, learned of it cause it hit my patch heh https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b0d/794186/1/check/openstack-tox-py38/b0dd1dd/testr_results.html	18:53
sean-k-mooney	that was going to be my next question good catch	18:53
melwitt	after awhile I realized it's because refresh associations records the time at which it ran, and the test was relying on the fact that that time would be sufficiently soon enough after "now" was recorded prior	18:55
sean-k-mooney	that looks like now = time.time() was previously unused	18:56
sean-k-mooney	oh i guess its later in the test	18:56
sean-k-mooney	ya it is	18:56
melwitt	no yeah it's used	18:57
sean-k-mooney	part of me wishes there was a clearn helper funciton or somthing for this but ya this looks correct to me	18:57
sean-k-mooney	for example mocking time.time via a decorator at the top of the function	18:58
melwitt	oh yeah, I guess that would work without needing reset of the mocks throughout, cause all you'd change is the return_value and not asserting calls	18:59
sean-k-mooney	e.g. @mock.patch.object(time, 'time', return_value=time.time())	18:59
sean-k-mooney	yep	19:00
sean-k-mooney	you could still reset the time mock if needed	19:00
sean-k-mooney	but they dont check that as far as i can see	19:00
melwitt	yeah. I was hoping to not though	19:01
sean-k-mooney	i dont think you need too	19:01
melwitt	yeah, agree	19:01
sean-k-mooney	@mock.patch.object(time, 'time', return_value=time.time())/def test_refresh_associations_time(self, log_mock,time_mock):	19:01
sean-k-mooney	shoudl fix it	19:01
sean-k-mooney	return_value=time.time() shoudl be evaluated before the mock is appied and resovled into a what time.time() returns	19:02
melwitt	I'll give it a go	19:05
*** whoami-rajat has quit IRC		19:10
*** efried has quit IRC		19:14
*** efried has joined #openstack-nova		19:14
*** ralonsoh has joined #openstack-nova		19:16
opendevreview	melanie witt proposed openstack/nova master: Make test_refresh_associations_* deterministic https://review.opendev.org/c/openstack/nova/+/794396	19:28
opendevreview	melanie witt proposed openstack/nova master: Make test_refresh_associations_* deterministic https://review.opendev.org/c/openstack/nova/+/794396	19:30
opendevreview	melanie witt proposed openstack/nova stable/victoria: zuul: Replace grenade and nova-grenade-multinode with grenade-multinode https://review.opendev.org/c/openstack/nova/+/794674	19:33
opendevreview	melanie witt proposed openstack/nova stable/ussuri: zuul: Replace grenade and nova-grenade-multinode with grenade-multinode https://review.opendev.org/c/openstack/nova/+/794675	19:37
*** ralonsoh has quit IRC		20:28
*** andrewbonney has quit IRC		21:14
*** admin1 has left #openstack-nova		21:40
*** whoami-rajat has joined #openstack-nova		21:56
*** eharney has quit IRC		21:57
*** brinzhang0 has joined #openstack-nova		22:11
*** brinzhang_ has quit IRC		22:18
*** rloo has quit IRC		22:28
*** tosky has quit IRC		23:00
*** luksky has quit IRC		23:07
opendevreview	melanie witt proposed openstack/nova stable/train: zuul: Replace grenade and nova-grenade-multinode with grenade-multinode https://review.opendev.org/c/openstack/nova/+/794686	23:53
opendevreview	melanie witt proposed openstack/nova stable/train: zuul: Replace grenade and nova-grenade-multinode with grenade-multinode https://review.opendev.org/c/openstack/nova/+/794686	23:57
melwitt	sigh	23:59
*** martinkennelly has quit IRC		23:59
*** martinkennelly_ has quit IRC		23:59
opendevreview	melanie witt proposed openstack/nova stable/train: zuul: Replace grenade and nova-grenade-multinode with grenade-multinode https://review.opendev.org/c/openstack/nova/+/794686	23:59

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!