Wednesday, 2019-07-31

« Tuesday, 2019-07-30 Index Thursday, 2019-08-01 »
*** tbachman has quit IRC00:04
*** slaweq has joined #openstack-nova00:11
*** slaweq has quit IRC00:16
*** tbachman has joined #openstack-nova00:23
*** tetsuro has joined #openstack-nova00:55
*** mkrai_ has joined #openstack-nova01:01
*** ricolin has joined #openstack-nova01:01
*** tbachman has quit IRC01:01
*** tbachman has joined #openstack-nova01:03
*** tbachman has quit IRC01:08
*** ricolin has quit IRC01:31
*** spatel has joined #openstack-nova01:33
*** spatel has quit IRC01:36
openstackgerritMerged openstack/nova master: Add functional recreate test for bug 1764556  https://review.opendev.org/56204102:00
openstackbug 1764556 in OpenStack Compute (nova) ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,In progress] https://launchpad.net/bugs/1764556 - Assigned to melanie witt (melwitt)02:00
*** tbachman has joined #openstack-nova02:00
*** slaweq has joined #openstack-nova02:11
*** mkrai_ has quit IRC02:11
*** slaweq has quit IRC02:16
*** tetsuro has quit IRC02:16
*** tetsuro has joined #openstack-nova02:52
*** rcernin has quit IRC02:57
*** mkrai_ has joined #openstack-nova02:59
*** rcernin has joined #openstack-nova03:13
*** tetsuro has quit IRC03:28
*** psachin has joined #openstack-nova03:37
*** tbachman has quit IRC03:40
*** tbachman has joined #openstack-nova03:45
*** tetsuro has joined #openstack-nova03:46
*** BjoernT has joined #openstack-nova03:55
*** mkrai_ has quit IRC03:57
*** mkrai has joined #openstack-nova03:57
*** slaweq has joined #openstack-nova04:11
*** slaweq has quit IRC04:16
*** whoami-rajat has joined #openstack-nova04:19
*** udesale has joined #openstack-nova04:20
*** udesale has quit IRC04:21
*** udesale has joined #openstack-nova04:21
*** ratailor has joined #openstack-nova04:27
*** Luzi has joined #openstack-nova04:30
*** BjoernT has quit IRC04:36
*** tkajinam has quit IRC05:04
*** tkajinam has joined #openstack-nova05:05
*** bhagyashris has joined #openstack-nova05:15
*** ganso has quit IRC05:19
*** ganso has joined #openstack-nova05:20
*** boxiang_ has quit IRC05:21
*** boxiang has joined #openstack-nova05:22
*** gyee has quit IRC05:26
*** jaosorior has quit IRC05:31
openstackgerritSundar Nadathur proposed openstack/nova master: ksa auth conf and client for Cyborg access  https://review.opendev.org/63124205:36
openstackgerritSundar Nadathur proposed openstack/nova master: WIP: Add Cyborg device profile groups to request spec.  https://review.opendev.org/63124305:36
openstackgerritSundar Nadathur proposed openstack/nova master: WIP: Create and bind Cyborg ARQs.  https://review.opendev.org/63124405:36
openstackgerritSundar Nadathur proposed openstack/nova master: WIP: Get resolved Cyborg ARQs and add PCI BDFs to VM's domain XML.  https://review.opendev.org/63124505:36
openstackgerritSundar Nadathur proposed openstack/nova master: Define new exceptions related to device profiles and ARQs.  https://review.opendev.org/67373305:36
openstackgerritSundar Nadathur proposed openstack/nova master: Refactor some methods for reuse by Cyborg code.  https://review.opendev.org/67373405:36
openstackgerritSundar Nadathur proposed openstack/nova master: Delete ARQs for an instance when the instance is deleted.  https://review.opendev.org/67373505:36
*** ratailor has quit IRC05:39
*** threestrands has joined #openstack-nova05:44
openstackgerritSundar Nadathur proposed openstack/nova master: WIP: Add Cyborg device profile groups to request spec.  https://review.opendev.org/63124305:47
openstackgerritSundar Nadathur proposed openstack/nova master: WIP: Create and bind Cyborg ARQs.  https://review.opendev.org/63124405:47
openstackgerritSundar Nadathur proposed openstack/nova master: WIP: Get resolved Cyborg ARQs and add PCI BDFs to VM's domain XML.  https://review.opendev.org/63124505:47
openstackgerritSundar Nadathur proposed openstack/nova master: Delete ARQs for an instance when the instance is deleted.  https://review.opendev.org/67373505:47
*** ratailor has joined #openstack-nova05:48
*** mkrai has quit IRC06:08
*** mkrai_ has joined #openstack-nova06:08
*** slaweq has joined #openstack-nova06:11
*** belmoreira has joined #openstack-nova06:13
*** slaweq has quit IRC06:16
*** kashyap has joined #openstack-nova06:22
*** jaosorior has joined #openstack-nova06:23
kashyapaspiers: Morning, on the SEV experiment on Leap 15.0, I don't know you're getting stuck at the UEFI boot06:23
kashyapaspiers: We can debug it when you're on.  I'm hunkering down for a few hours to investigate something, so will be slow in the morning to respond.06:24
*** mkrai_ has quit IRC06:26
*** mkrai__ has joined #openstack-nova06:26
openstackgerritMerged openstack/nova master: Add functional regression test for bug 1778305  https://review.opendev.org/58240706:32
openstackbug 1778305 in OpenStack Compute (nova) "Nova may erronously look up service version of a deleted service, when hostname have been reused" [Medium,In progress] https://launchpad.net/bugs/1778305 - Assigned to melanie witt (melwitt)06:32
*** artom has quit IRC06:32
*** artom has joined #openstack-nova06:33
*** zhengMa has joined #openstack-nova06:37
*** slaweq has joined #openstack-nova06:59
*** rcernin has quit IRC07:02
*** ratailor has quit IRC07:07
*** ratailor has joined #openstack-nova07:08
*** xek has joined #openstack-nova07:09
*** takamatsu has quit IRC07:17
*** brault has joined #openstack-nova07:19
*** maciejjozefczyk has joined #openstack-nova07:19
*** brault has quit IRC07:23
*** udesale has quit IRC07:24
*** udesale has joined #openstack-nova07:24
*** tesseract has joined #openstack-nova07:27
*** belmoreira has quit IRC07:28
openstackgerritYongli He proposed openstack/nova master: Add server sub-resource topology API  https://review.opendev.org/62147607:28
*** tssurya has joined #openstack-nova07:31
*** mkrai__ has quit IRC07:34
*** pcaruana has quit IRC07:35
*** mkrai__ has joined #openstack-nova07:35
*** ociuhandu has joined #openstack-nova07:35
*** cdent has joined #openstack-nova07:36
*** igordc has joined #openstack-nova07:37
*** rpittau|afk is now known as rpittau07:37
*** igordc has quit IRC07:38
*** ociuhandu has quit IRC07:40
*** ociuhandu has joined #openstack-nova07:44
*** ociuhandu has quit IRC07:44
openstackgerritSundar Nadathur proposed openstack/nova master: Refactor some methods for reuse by Cyborg code.  https://review.opendev.org/67373407:51
openstackgerritSundar Nadathur proposed openstack/nova master: WIP: Add Cyborg device profile groups to request spec.  https://review.opendev.org/63124307:51
openstackgerritSundar Nadathur proposed openstack/nova master: WIP: Create and bind Cyborg ARQs.  https://review.opendev.org/63124407:51
openstackgerritSundar Nadathur proposed openstack/nova master: WIP: Get resolved Cyborg ARQs and add PCI BDFs to VM's domain XML.  https://review.opendev.org/63124507:51
openstackgerritSundar Nadathur proposed openstack/nova master: Delete ARQs for an instance when the instance is deleted.  https://review.opendev.org/67373507:51
*** betherly has joined #openstack-nova08:01
*** belmoreira has joined #openstack-nova08:01
*** dpawlik has joined #openstack-nova08:02
*** mkrai__ has quit IRC08:03
*** jaosorior has quit IRC08:03
*** mkrai__ has joined #openstack-nova08:03
*** lpetrut has joined #openstack-nova08:04
*** tetsuro has quit IRC08:11
*** ttsiouts has joined #openstack-nova08:11
*** pcaruana has joined #openstack-nova08:13
*** mkrai__ has quit IRC08:15
*** ociuhandu has joined #openstack-nova08:15
*** tetsuro has joined #openstack-nova08:17
*** tkajinam has quit IRC08:19
*** ttsiouts has quit IRC08:21
*** ttsiouts has joined #openstack-nova08:22
openstackgerritMerged openstack/nova master: Don't generate service UUID for deleted services  https://review.opendev.org/58240808:25
*** ttsiouts has quit IRC08:27
*** mkrai__ has joined #openstack-nova08:27
*** takamatsu has joined #openstack-nova08:32
*** ttsiouts has joined #openstack-nova08:36
*** derekh has joined #openstack-nova08:40
*** ricolin has joined #openstack-nova08:40
*** dikonoor has joined #openstack-nova08:43
*** takamatsu_ has joined #openstack-nova08:48
*** threestrands has quit IRC08:48
*** takamatsu has quit IRC08:49
*** panda has quit IRC08:58
*** panda has joined #openstack-nova08:59
*** jaosorior has joined #openstack-nova09:00
*** belmoreira has quit IRC09:03
*** ttsiouts has quit IRC09:05
*** priteau has joined #openstack-nova09:05
*** tetsuro has quit IRC09:06
*** ttsiouts has joined #openstack-nova09:09
*** ociuhandu has quit IRC09:12
*** ociuhandu has joined #openstack-nova09:12
*** ralonsoh has joined #openstack-nova09:13
*** zhengMa has quit IRC09:17
*** zhengMa has joined #openstack-nova09:22
*** ttsiouts has quit IRC09:23
*** ttsiouts has joined #openstack-nova09:24
*** zhengMa has quit IRC09:25
*** happy-ms-ge has joined #openstack-nova09:26
*** ttsiouts has quit IRC09:28
*** ivve has joined #openstack-nova09:28
*** takamatsu_ has quit IRC09:30
*** dikonoor has quit IRC09:36
*** takamatsu has joined #openstack-nova09:37
*** happy-ms-ge has quit IRC09:39
*** betherly has quit IRC09:43
*** ttsiouts has joined #openstack-nova09:45
*** bhagyashris has quit IRC09:48
*** jaosorior has quit IRC09:50
openstackgerritBrin Zhang proposed openstack/python-novaclient master: Microversion 2.75: Support Specifying AZ to unshelve  https://review.opendev.org/66513609:52
*** ttsiouts has quit IRC10:00
*** ttsiouts has joined #openstack-nova10:01
*** ttsiouts has quit IRC10:06
*** takamatsu has quit IRC10:08
openstackgerritBrin Zhang proposed openstack/python-novaclient master: Microversion 2.75: Support Specifying AZ to unshelve  https://review.opendev.org/66513610:08
*** ratailor has quit IRC10:09
*** ratailor has joined #openstack-nova10:10
*** sapd1_x has joined #openstack-nova10:22
*** takamatsu has joined #openstack-nova10:24
*** lpetrut has quit IRC10:24
*** cf__ has quit IRC10:26
aspierskashyap: it's not urgent right now but I would like to get to the bottom of it. If you have any ideas let me know10:33
*** bbowen has joined #openstack-nova10:34
kashyapaspiers: Will do.  Currently mired in parsing getDomainCapabilities(); /me is slow with it10:34
*** jaosorior has joined #openstack-nova10:34
aspierskashyap: parsing how?10:35
aspiersthe UEFI bits?10:35
kashyapaspiers: Yeah, /me gets a pastebin10:36
kashyapMaybe you can get me out of the ditch :D10:36
kashyapaspiers: http://paste.openstack.org/show/755161/10:37
kashyapEssentially:10:38
kashyap    # Only available from for libvirt 5.4 or above10:38
kashyap    $> virsh domcapabilities --machine q35 | xpath -q -e "//enum[@name='firmware']/value/text()"10:38
kashyap    efi10:38
kashyapAnd this one (available for much older libvirts):10:39
kashyap    $> virsh domcapabilities --machine q35 | xpath -q -e "//loader/value/text()"10:39
kashyap    /usr/share/edk2/ovmf/OVMF_CODE.fd10:39
*** betherly has joined #openstack-nova10:44
*** pcaruana has quit IRC10:45
kashyapaspiers: Ignore the pastebin; just spotted more problems w/ it10:46
aspiersOK10:46
aspiersI get Cannot open file '-q' at /usr/lib/perl5/vendor_perl/5.18.2/XML/XPath.pm line 53.10:46
aspiersmy xpath doesn't like -q or -e10:46
aspiersfrom perl-XML-XPath-1.13-99.19.x86_6410:47
*** belmoreira has joined #openstack-nova10:50
*** ricolin_ has joined #openstack-nova11:01
*** ricolin has quit IRC11:03
*** ratailor has quit IRC11:10
*** ratailor has joined #openstack-nova11:11
openstackgerritLuyao Zhong proposed openstack/nova master: Indroduce new LibvirtConfigGuestPMEM  https://review.opendev.org/67295611:13
openstackgerritLuyao Zhong proposed openstack/nova master: Populates existing pmems in libvirt device manager  https://review.opendev.org/67295711:13
openstackgerritLuyao Zhong proposed openstack/nova master: Report pmem namespaces resources by provider tree  https://review.opendev.org/63455111:13
openstackgerritLuyao Zhong proposed openstack/nova master: Support instance creation and resize with pmems  https://review.opendev.org/67295811:13
*** ttsiouts has joined #openstack-nova11:23
*** fanzhang has joined #openstack-nova11:35
*** ivve has quit IRC11:40
fanzhangHey, I recently hit the bug https://bugs.launchpad.net/nova/+bug/1836204, and I was wondering why can't we just delete the mdevs after destroy the guest, and always create  new mediated devices when booting instance? Is there any special reason why not clean mdevs up after deleting vms?11:41
openstackLaunchpad bug 1836204 in OpenStack Compute (nova) "The allocation of VGPU has race problem" [High,Triaged] - Assigned to Alex Xu (xuhj)11:41
openstackgerritKashyap Chamarthy proposed openstack/nova master: libvirt: Parse the 'os' element from domainCapabilities  https://review.opendev.org/67379011:43
kashyapaspiers: Okay, an idea for your problem11:44
* aspiers is listening11:44
kashyapaspiers: I learnt from Laszlo (OVMF maintainer) in the past that there's a QEMU debug port which captures OVMF error messages11:44
kashyapaspiers: But, it requires you explicitly adding it in.  This waY:11:44
kashyap  <qemu:commandline>11:45
kashyap    <qemu:arg value='-global'/>11:45
kashyap    <qemu:arg value='isa-debugcon.iobase=0x402'/>11:45
kashyap    <qemu:arg value='-debugcon'/>11:45
kashyap    <qemu:arg value='file:/tmp/ovmf.fedora.q35.log'/>11:45
kashyap  </qemu:commandline>11:45
kashyapaspiers: To your test.  (Yes, we're doing QEMU 'commandline' passthrough - because we're debugging)11:45
kashyaps/test/guest/11:45
kashyapThen, the error message can be read there11:45
aspiersI'm not expecting any error though11:46
aspiersit just stops at the UEFI shell prompt11:46
aspiersas if that's what was supposed to happen11:46
kashyapHmm.  What is different in Leap, compared to SLES?11:46
*** udesale has quit IRC11:46
aspiersIt's the same guest image11:46
aspiersonly the hypervisor stack changed11:47
kashyapaspiers: And previously your host was Ubuntu?11:47
aspierswhat?!11:47
kashyapHehe11:47
aspierswash your mouth out!11:47
aspiersheathen11:47
aspiers;-)11:47
*** udesale has joined #openstack-nova11:47
kashyapaspiers: I'm just trying to understand the env 'before' and 'after'11:48
aspiersIt was working fine on SLES12 SP4, broken on Leap 15.011:48
kashyapAh, okay.11:48
aspierswhich are supposed to be close IIRC11:48
*** jaosorior has quit IRC11:48
kashyapRight, Leap is upstream11:48
kashyapaspiers: How about you try to create a guest from scratch on Leap 15.0?11:49
*** ociuhandu has quit IRC11:49
kashyapIdeally 'virt-install' + "kickstart" (assuming SUSE has such a concept, too)11:49
aspiersautoyast11:50
kashyapRight11:50
aspiersI think I was writing ks.cfg files in the 1990s though11:50
aspierscertainly a lot in the early 2000s11:50
kashyapI just a plain minimal '@core' only11:50
*** ratailor has quit IRC11:51
aspiersautoyast was way more powerful, at least until 201011:51
kashyapSee the bits after "fkstart" here: https://kashyapc.fedorapeople.org/Create-a-SecureBoot-enabled-VM.bash11:51
aspiersno idea these days11:51
kashyapaspiers: You wouldn't believe, 10 years ago I _was_ using SUSE :-)11:51
aspiershaha I would11:51
kashyapWhy wouldn't you believe?  It's perfectly believable11:51
aspiersmany people did11:51
aspiersespecially in the 1990s11:51
kashyapAnd my laptop has SUSE chamelon stickers, too11:52
aspiers:-O11:52
aspiersyou didn't get fired for that?11:53
aspiersX-D11:53
*** boxiang has quit IRC11:53
kashyapNot to mention: https://kashyapc.fedorapeople.org/geeko-key.jpg11:53
kashyapaspiers: So back to the problem, I have no other ideas why it's stuck like that.  Maybe as an experiment: re-install the image from scratch on Leap 15, _and_ try _that_ image on SLES12 SP411:55
aspierswell the image was built in OBS using kiwi11:55
aspiersnot manually11:55
sean-k-mooneyaspiers: at least kashyap uses fedora as his distro on his laptop unlike me...11:55
kashyapHmm11:55
aspiersah you patched the win key, nice11:56
aspierssean-k-mooney: don't tell me, Ubuntu11:56
kashyapI'm also an official Fedora packager.  Also the Virt Stack support is unparalleled on Fedora. "Eating your own cooking" and all that.11:56
sean-k-mooneyaspiers: technically pop-OS but its based on ubuntu11:56
aspiersWe have a few SUSE employees who use Ubuntu as their base OS. My mind explodes every time I see it11:56
kashyapaspiers: Unrelated, before I go for a break, I still need tests here, but please tell me if I did anything glaringly wrong: https://review.opendev.org/#/c/673790/11:57
sean-k-mooneyaspiers: we also have people that use windows and mac11:57
aspierssean-k-mooney: I can almost understand that more11:57
sean-k-mooneyas long as you get your work donw people dont really care11:57
kashyapsean-k-mooney: Not quite...11:57
aspiersbut if you're gonna use Linux, why not use the distro you're being paid to improve?11:57
sean-k-mooneyim not being paid to imporve it11:58
kashyapI'd say, if you're at a Linux company, doing Linux-based dev.  "Suffer the pain" and use Linux.11:58
sean-k-mooneyi dont work on rhel11:58
kashyapNo ifs, buts or maybes.11:58
aspierskashyap: I'm with you on this11:58
kashyapIt's not about "working on RHEL".11:58
sean-k-mooneyand i didnt join redhat for any of its  products. i joined sole to work with people i enjoyed working with upstream11:58
* aspiers senses this conversation could rathole :)11:58
sean-k-mooneyi am waiting for cento8 to come out however11:59
kashyapHeh, yes11:59
sean-k-mooneyi would like to try it out11:59
aspierskashyap: you meant to add <enum> to the sparc fixture?12:00
aspierskashyap: https://review.opendev.org/#/c/673790/1/nova/virt/libvirt/config.py@165 that [] looks wrong12:01
aspierssince you're not appending <os> to a list12:01
aspierskashyap: also please think about +1'ing https://review.opendev.org/#/c/348394/ as per my newest comment12:05
*** takamatsu has quit IRC12:10
*** pcaruana has joined #openstack-nova12:10
*** tbachman has quit IRC12:11
kashyapaspiers: Oh, right.  That [] is wrong.  Indeed, one 'os' element12:13
aspiersThank god I'm actually some use as a reviewer occasionally12:14
kashyapaspiers: Don't be silly, you know you give useful remarks a lot of times12:14
aspierssometimes I wonder ;-)12:14
aspiersjournalctl -f | lnav is truly a gift from the gods12:14
aspiersif you haven't discovered the joys of lnav, do so immediately12:14
aspiershttp://lnav.org/12:15
kashyapaspiers: On that other bug from Dirk -- I'll repost the patch with CentOS fix, too.  And think more12:15
aspierskashyap: thanks. Yeah just upload a new patchset?12:15
kashyapaspiers: Yea, brewing12:15
kashyapaspiers: I take it that you've read my full comment on that patch; thanks12:16
kashyapaspiers: On lnav - I noticed it, but haven't used it much12:16
aspiersYou should, it's insanely good12:16
aspiersworth it for the histogram feature alone12:16
sean-k-mooneyyep12:16
sean-k-mooneyi found it like a year ago12:16
sean-k-mooneyits really nice12:16
aspierssean-k-mooney: did you know about "lnav -i extra" ?12:16
aspiersmy 3rd party repo gets included12:17
sean-k-mooneyi havent use that really but i like its ablity to filter things12:17
sean-k-mooneyno but -q should be on by default12:17
aspierssean-k-mooney: haha, did you see that issue?12:17
kashyapaspiers: Installed; and began using.  I heard it from sean-k-mooney, actually12:17
sean-k-mooneyis there an issue open for it12:17
kashyapaspiers: Meanwhile, I take it that you haven't spotted anything _else_ that is spooky in https://review.opendev.org/#/c/673790/12:18
aspierssean-k-mooney: https://github.com/tstack/lnav/issues/436 but it got closed because the author *still* isn't getting the point12:18
aspiersabout to comment12:18
sean-k-mooneyi have basically locked up my terminal after leaving the devstack logs open for a few days in it12:18
sean-k-mooneyanyway be right be time to grab coffee and post something by snail mail...12:20
sean-k-mooneyat least its not fax...12:20
* kashyap --> late lunch12:20
*** kashyap has quit IRC12:20
sean-k-mooneymy insurance company will acatully accept email copies of scans provided you eventually send them the forms which is nice12:21
* aspiers hates paper12:21
aspiersJul 29 17:06:56 devstack systemd[1]: devstack@n-sch.service: Main process exited, code=dumped, status=11/SEGV12:22
aspiersummm... WTF?!12:22
*** ociuhandu has joined #openstack-nova12:22
sean-k-mooneythat sound like the issue matt was hitting yesterday12:23
sean-k-mooneyare you using an old kernel or python 2?12:23
aspierspython 212:23
aspiers4.12.14-95.24-default12:23
sean-k-mooneyon kernel? 4.15 ish?12:23
sean-k-mooneyok older12:23
aspiersis there a workaround/solution?12:23
sean-k-mooneyya there may be an issue with one of our deps with python 2 and older kernels12:24
* aspiers reads scrollback12:24
sean-k-mooneywe havnt track it down yet but if you can provide logs/version number it might help12:24
sean-k-mooneyyou can try python312:24
aspiersthere's nothing useful in the log12:25
sean-k-mooneyjust he seg fault12:25
aspiersoh god12:25
aspiers[35889.416395] cinder-volume[178403]: segfault at 7f2154837db7 ip 00007f20f299c9ca sp 00007ffea13d86d0 error 6 in libpython2.7.so.1.0[7f20f28b0000+1ac000]12:25
aspiersnot just nova12:25
sean-k-mooneyya12:25
sean-k-mooneyare you using nested virt12:25
aspiersno12:25
sean-k-mooneyok so that rules out one possibelissue12:26
sean-k-mooneymatt was seeing similar isssues with python 2.7 on ubuntu 18.04 yesterday and last week12:26
aspiersI guess I can try switching to python 312:27
aspiersyikes12:27
sean-k-mooneyif leap 15 has python3 i would try that12:27
aspiersthis is not leap12:27
aspierssles12 sp412:27
*** fanzhang has quit IRC12:27
aspiersbut yes leap and sles have both had python3 for years12:27
sean-k-mooneymake sure you use either 3.5 or 3.612:28
sean-k-mooneywe never really supported 3.4 and 3.7 may have issues12:28
aspiersoh crap12:28
aspiersdefault is 3.4.612:28
sean-k-mooney3.4 might work12:28
sean-k-mooneywe just never offially supported it12:28
sean-k-mooneywe went strait to 3.512:28
aspiersok12:28
*** takamatsu has joined #openstack-nova12:31
stephenfingmann: Could you take a look at https://review.opendev.org/#/c/661391/ (DevStack stuff) when you've a chance?12:32
*** mchlumsky has joined #openstack-nova12:33
aspierssean-k-mooney: https://github.com/tstack/lnav/issues/436#issuecomment-51682910612:34
*** ivve has joined #openstack-nova12:35
gmannstephenfin: sure. do you have nova patch to test that ? only thing i want to verify is IPv6 job i added recently on nova gate works fine. I removed the setting of novnc addresses from base devstack job and let /lib/nova handle that.12:36
gmannstephenfin: of let me add the same job on devstack on top of that patch which will test both.12:36
gmanns/of/or12:36
*** priteau has quit IRC12:39
*** psachin has quit IRC12:41
*** jaosorior has joined #openstack-nova12:43
aspierssean-k-mooney: I have a coredump now12:46
aspierssean-k-mooney: also https://github.com/aspiers/lnav-formats/blob/master/oslo_log.json12:46
aspiersalthough I'm not sure that was necessary, it might be in other repos12:46
*** lpetrut has joined #openstack-nova12:50
*** ricolin_ is now known as ricolin12:50
stephenfingmann: Yeah, I think this one should do it https://review.opendev.org/#/c/667388/12:51
gmannstephenfin: thanks. rechecked that.12:54
*** Luzi has quit IRC12:54
sean-k-mooneyaspiers: with python 3.4?12:56
aspiersno12:56
aspiershttp://paste.openstack.org/show/755163/12:56
sean-k-mooneyaspiers: also did you add oslo_log support to lnav as a plugin if so i might have to try that12:57
aspierssean-k-mooney: that's what the link above is12:57
aspierssean-k-mooney: it gets installed via "lnav -i extra"12:57
aspiersjust type that12:57
sean-k-mooneyya i clicked and got confused with your core dump comment but imm looking at teh paste now12:58
*** lpetrut has quit IRC12:59
*** lpetrut has joined #openstack-nova12:59
sean-k-mooneyhum well i cant really tell what cased that seg fault other thin it appers to be a result of a getattr call on a randomp python object. so ill blame eventlest by default if you can repoduce with the pytho debuger instad of gdb that might help13:03
sean-k-mooneyaspiers: can you try running the conductor with pdb13:05
aspierssean-k-mooney: I'm trying to install the python gdb extensions so I can get the Python stacktrace13:05
aspierssean-k-mooney: I guess that might work too13:05
*** ivve has quit IRC13:06
sean-k-mooneyi think its dieing pretty early in the nova-condoctor console script that is generated via pbr/setup tools13:07
aspierssean-k-mooney: what's the best way to start it?13:07
aspiersoh wow13:08
aspiers● devstack@n-sch.service        loaded failed failed Devstack devstack@n-sch.service13:08
aspiers● devstack@n-super-cond.service loaded failed failed Devstack devstack@n-super-cond.service13:08
aspiers● devstack@s-proxy.service      loaded failed failed Devstack devstack@s-proxy.service13:08
aspiersI've had 6 crashes in the last hour13:08
sean-k-mooneyi think just "pdb  /usr/bin/python /usr/bin/nova-conductor --config-file /etc/nova/nova.conf" maybe with sudo13:08
sean-k-mooneyya one of our deps really dislikes python2.7 apparently13:11
sean-k-mooneywas this a clean sles 14 deployment13:12
sean-k-mooneyim wondering if i can quickly spin up an container or something and try and repoduce13:12
*** tbachman has joined #openstack-nova13:13
aspiersgah13:17
aspiersI just lost connection to the node13:18
*** brtknr has left #openstack-nova13:18
sean-k-mooneydid you check dmesg before13:18
sean-k-mooneythere may have been other core dumps or crashes13:18
aspierscoredumpctl shows them all13:19
aspiersall 613:19
sean-k-mooneyall openstack services13:19
*** tesseract has quit IRC13:20
*** tesseract has joined #openstack-nova13:24
aspiersno, just a few13:25
*** ttsiouts has quit IRC13:25
*** ttsiouts has joined #openstack-nova13:26
*** bbowen has quit IRC13:26
sean-k-mooneyi ment were the 6 core dumps form opensack not did all the openstack services core dump13:26
aspiersyes all openstack I think13:26
*** mriedem has joined #openstack-nova13:27
aspiersbut I can't check because my connection died13:27
sean-k-mooneymaybe ssh coredumped...13:27
*** bbowen has joined #openstack-nova13:27
aspiersnope13:27
aspiersno longer pingable13:27
sean-k-mooneyit could be a full kernel panic13:28
aspierswell aren't you the optimist :)13:28
sean-k-mooneydo you have acess to the system to check?13:28
aspiersmaybe a PSU caught fire, and now the whole DC is a raging inferno13:28
aspiersmy colleague is checking via remote console13:28
aspiersI'm going back to figure out why cloud-init is so broken13:29
*** bbowen has quit IRC13:29
sean-k-mooneysure :) let me know if they find anything via the remote console13:29
*** bbowen has joined #openstack-nova13:30
aspierswill do13:30
*** BjoernT has joined #openstack-nova13:30
*** ttsiouts has quit IRC13:31
*** ivve has joined #openstack-nova13:37
openstackgerritMatt Riedemann proposed openstack/nova stable/stein: Add functional recreate test for bug 1764556  https://review.opendev.org/67381213:39
openstackbug 1764556 in OpenStack Compute (nova) stein ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,Confirmed] https://launchpad.net/bugs/176455613:39
*** ociuhandu has quit IRC13:39
*** ociuhandu has joined #openstack-nova13:39
openstackgerritMatt Riedemann proposed openstack/nova stable/stein: Add functional regression test for bug 1778305  https://review.opendev.org/67381313:40
openstackbug 1778305 in OpenStack Compute (nova) stein "Nova may erronously look up service version of a deleted service, when hostname have been reused" [Medium,Confirmed] https://launchpad.net/bugs/177830513:40
*** spatel has joined #openstack-nova13:40
*** ttsiouts has joined #openstack-nova13:41
*** BjoernT_ has joined #openstack-nova13:42
*** belmoreira has quit IRC13:43
*** BjoernT has quit IRC13:44
openstackgerritMatt Riedemann proposed openstack/nova stable/stein: Don't generate service UUID for deleted services  https://review.opendev.org/67381413:44
*** belmoreira has joined #openstack-nova13:45
*** spatel has quit IRC13:45
*** liuyulong has joined #openstack-nova13:46
openstackgerritMatt Riedemann proposed openstack/nova stable/rocky: Add functional recreate test for bug 1764556  https://review.opendev.org/67381613:48
openstackbug 1764556 in OpenStack Compute (nova) stein ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,In progress] https://launchpad.net/bugs/1764556 - Assigned to Matt Riedemann (mriedem)13:48
openstackgerritMatt Riedemann proposed openstack/nova stable/rocky: Add functional regression test for bug 1778305  https://review.opendev.org/67381813:52
openstackbug 1778305 in OpenStack Compute (nova) stein "Nova may erronously look up service version of a deleted service, when hostname have been reused" [Medium,In progress] https://launchpad.net/bugs/1778305 - Assigned to Matt Riedemann (mriedem)13:52
mriedemlyarwood: could you poke https://review.opendev.org/#/c/673532/1 and the one above?13:54
*** eharney has joined #openstack-nova13:55
*** belmoreira has quit IRC13:56
openstackgerritMatt Riedemann proposed openstack/nova stable/rocky: Don't generate service UUID for deleted services  https://review.opendev.org/67382113:57
*** spatel has joined #openstack-nova13:59
openstackgerritMatt Riedemann proposed openstack/nova stable/queens: Add functional recreate test for bug 1764556  https://review.opendev.org/67382414:00
openstackbug 1764556 in OpenStack Compute (nova) stein ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,In progress] https://launchpad.net/bugs/1764556 - Assigned to Matt Riedemann (mriedem)14:00
openstackgerritMatt Riedemann proposed openstack/nova stable/queens: Add functional regression test for bug 1778305  https://review.opendev.org/67382514:01
openstackbug 1778305 in OpenStack Compute (nova) stein "Nova may erronously look up service version of a deleted service, when hostname have been reused" [Medium,In progress] https://launchpad.net/bugs/1778305 - Assigned to Matt Riedemann (mriedem)14:01
lyarwoodmriedem: yup will try to get to them today14:02
mriedemthanks14:03
*** spatel has quit IRC14:03
openstackgerritMatt Riedemann proposed openstack/nova stable/queens: Don't generate service UUID for deleted services  https://review.opendev.org/67382714:04
*** boxiang has joined #openstack-nova14:12
*** belmoreira has joined #openstack-nova14:15
*** brinzhang_ has quit IRC14:17
*** brinzhang_ has joined #openstack-nova14:18
*** brinzhang_ has joined #openstack-nova14:18
openstackgerritMatt Riedemann proposed openstack/nova stable/pike: Add functional recreate test for bug 1764556  https://review.opendev.org/67383014:20
openstackbug 1764556 in OpenStack Compute (nova) stein ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Medium,In progress] https://launchpad.net/bugs/1764556 - Assigned to Matt Riedemann (mriedem)14:20
boxiangmriedem: efried: gibi_off: two patches about live-migration and evacuation need to review https://review.opendev.org/#/c/649963/ https://review.opendev.org/#/c/651969/14:20
mriedemit'd be nice if mdbooth could look at the evacuate one since he also wrote https://review.openstack.org/#/c/605436/ - would also be nice if cfriesen were around to review those since i know starlingx has patches for these same issues14:21
*** brinzhang has quit IRC14:21
mriedemi'd prefer a functional test on https://review.opendev.org/#/c/651969/13 since i don't really trust unit tests for this kind of stuff14:21
mriedemgibi is out until next week i think14:22
boxiangok, so I will add a functional test for this https://review.opendev.org/#/c/651969/1314:23
* mdbooth notes that his already has that: https://review.opendev.org/#/c/605436/9/nova/tests/functional/regressions/test_bug_1735407.py14:23
mdboothMeh14:23
boxianghi mdbooth14:23
mdboothDuplicated effort is the best effort14:23
boxiangcan you take a look for my two patches, thanks14:24
*** betherly has quit IRC14:24
mdboothboxiang: Not any time soon, unfortunately. Working on stuff that might get used instead.14:25
*** takamatsu has quit IRC14:28
*** sapd1_x has quit IRC14:30
openstackgerritMatt Riedemann proposed openstack/nova stable/pike: Add functional regression test for bug 1778305  https://review.opendev.org/67383214:31
openstackbug 1778305 in OpenStack Compute (nova) stein "Nova may erronously look up service version of a deleted service, when hostname have been reused" [Medium,In progress] https://launchpad.net/bugs/1778305 - Assigned to Matt Riedemann (mriedem)14:31
mriedemmdbooth: i meant the live migration patch, not the evacuate one14:31
*** trident has quit IRC14:32
openstackgerritMatt Riedemann proposed openstack/nova stable/pike: Don't generate service UUID for deleted services  https://review.opendev.org/67383314:33
sean-k-mooneyby the way https://review.opendev.org/#/c/669738/ has been sitting for a while is mainly functional tests14:33
sean-k-mooneymdbooth: are you ok with this^14:34
sean-k-mooneymdbooth: i have your patch rebased on top14:34
sean-k-mooneymriedem: i also added functional test as you requested since you last looked14:35
mdboothsean-k-mooney: Yep. IIRC that's what we discussed previously.14:35
sean-k-mooneyi think the backport is still on hold for the orginal change14:35
mriedemdustinc: in case you haven't seen since i didn't vote, i've got some concerns in the base ironic+sdk patch https://review.opendev.org/#/c/642899/3314:36
*** trident has joined #openstack-nova14:36
sean-k-mooneyi belive my change can be applied to stable/X without breaking anything then the other chagne can be stacked on top14:36
mriedemugly tracebacks in the logs when nova-compute starts up before ironic-api is available14:36
*** boxiang has quit IRC14:36
*** boxiang has joined #openstack-nova14:37
dustincmriedem: been a bit distracted but did notice and will be looking into it when I get some time. Much appreciated!14:37
*** tbachman has quit IRC14:38
*** dpawlik has quit IRC14:38
*** trident has quit IRC14:47
*** kashyap has joined #openstack-nova14:49
*** trident has joined #openstack-nova14:50
*** belmoreira has quit IRC14:51
*** boxiang has quit IRC14:57
*** prometheanfire has joined #openstack-nova14:58
prometheanfireI suspect that the new lxml has some ordering issues or something14:58
prometheanfirehttp://logs.openstack.org/60/673160/3/check/cross-nova-py36/547e78c/testr_results.html.gz14:58
*** trident has quit IRC15:00
*** trident has joined #openstack-nova15:03
*** belmoreira has joined #openstack-nova15:03
openstackgerritMerged openstack/nova master: Fix AttributeError in RT._update_usage_from_migration  https://review.opendev.org/66768715:04
openstackgerritMerged openstack/nova master: Fix RT init arg order in test_unsupported_move_type  https://review.opendev.org/66768815:04
*** zhengMa has joined #openstack-nova15:06
*** zhengMa has left #openstack-nova15:06
*** trident has quit IRC15:08
openstackgerritZhengMa proposed openstack/nova master: Check container_format on instance creation  https://review.opendev.org/67340715:15
*** trident has joined #openstack-nova15:16
*** mlavalle has joined #openstack-nova15:18
sean-k-mooneyprometheanfire: those xml fragment look identical so ya maybe15:19
*** zhengMa has joined #openstack-nova15:20
*** trident has quit IRC15:24
prometheanfirekk, testing https://review.opendev.org/67384815:25
*** bbowen has quit IRC15:26
*** trident has joined #openstack-nova15:27
mriedemsean-k-mooney: comments in https://review.opendev.org/#/c/669738/15:33
mriedemmaybe i'm just dumb15:34
*** tbachman has joined #openstack-nova15:34
*** lpetrut has quit IRC15:34
*** cdent has quit IRC15:34
*** trident has quit IRC15:35
sean-k-mooneymriedem: thanks ill chack15:36
aspierssean-k-mooney: the new filter editor in lnav 0.8.5 is awesome (press tab)15:37
*** trident has joined #openstack-nova15:38
*** ttsiouts has quit IRC15:42
*** ttsiouts has joined #openstack-nova15:42
openstackgerritMatt Riedemann proposed openstack/nova stable/stein: Fix AttributeError in RT._update_usage_from_migration  https://review.opendev.org/67385615:45
*** tssurya has quit IRC15:46
*** ttsiouts has quit IRC15:47
*** belmoreira has quit IRC15:50
*** gyee has joined #openstack-nova15:52
*** nafiux has joined #openstack-nova15:56
*** efried is now known as efried_rollin15:57
nafiuxGood morning team, I’m trying to configure allocation ratio for CPU and Memory, but I’m getting this error from nova-scheduler: “Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up.”15:57
openstackgerritMerged openstack/nova master: Introduces the openstacksdk to nova  https://review.opendev.org/64366415:57
openstackgerritMerged openstack/nova master: Use OpenStack SDK for placement  https://review.opendev.org/65602315:57
*** icarusfactor has joined #openstack-nova15:58
nafiuxAny hint on how to setup that properly?15:58
*** ociuhandu has quit IRC15:59
nafiuxI’m using kolla-ansible for deployment, and I already created the “/etc/kolla/config/nova/localhost/nova.conf” file which contains: cpu_allocation_ratio = 16.0 & ram_allocation_ratio = 5.0 under [DEFAULT]15:59
*** factor has quit IRC15:59
*** mkrai__ has quit IRC16:06
*** Sundar has joined #openstack-nova16:06
mriedemwhich release? do you even have compute nodes reporting into placement? i.e. does "openstack resource provider list" return anything?16:11
mriedemhttps://docs.openstack.org/osc-placement/latest/16:12
mriedemhttps://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#allocation-ratios16:12
*** lpetrut has joined #openstack-nova16:14
*** lpetrut has quit IRC16:15
mriedemefried_rollin: random musing in https://review.opendev.org/#/c/673706/ but i'll likely abandon that change and open a new bug for the other spurious warning in there16:15
*** lpetrut has joined #openstack-nova16:15
*** gyee has quit IRC16:16
*** trident has quit IRC16:16
*** gyee has joined #openstack-nova16:17
*** rpittau is now known as rpittau|afk16:18
nafiux“openstack resource provider list” isn’t found, should I install any additional python CLI package?16:19
nafiuxOh, I saw… “pip install osc-placement”, let me try.16:20
*** xek has quit IRC16:20
nafiuxYes, I’m getting a node there16:20
*** trident has joined #openstack-nova16:21
SundarHi all, compute/manager.py::build_and_run_instance() (https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1859) requires the request_spec as an argument. But many unit test cases (e.g. https://github.com/openstack/nova/blob/master/nova/tests/unit/compute/test_compute.py#L9305) call it without the request_spec, which causes some u16:26
Sundarnit tests to fail. Is this really kosher?16:26
*** mkrai__ has joined #openstack-nova16:30
*** nafiux has quit IRC16:34
*** mkrai__ has quit IRC16:34
*** ociuhandu has joined #openstack-nova16:35
*** igordc has joined #openstack-nova16:36
melwittSundar: that unit test is not doing something ideal by passing {} for those args. in the case where the args are not relevant to what's being tested, it's recommended to use mock.ANY instead. what do you mean by "causes some unit tests to fail"?16:37
*** lpetrut has quit IRC16:37
*** panda has quit IRC16:39
*** panda has joined #openstack-nova16:41
openstackgerritMatt Riedemann proposed openstack/nova master: rt: soften warning case in _remove_deleted_instances_allocations  https://review.opendev.org/67387316:45
mriedemSundar: first thing is i'd avoid copying unit tests from the test_compute module,16:46
mriedembase your tests off test_compute_mgr if you're looking for existing tests to start from16:47
*** takamatsu has joined #openstack-nova16:47
mriedemtest_compute.py is historically running actual services and using the db, so it's more functional than unit-y16:47
mriedemreserve more complicated tests for actual functional tests with fixtures and such16:47
*** trident has quit IRC16:48
mriedemlike mel said, if your test doesn't care about the param, pass a Mock, else pass an actual nova.objects.RequestSpec object, not a dict16:48
*** zhengMa has quit IRC16:48
mriedemrequest specs as dicts are super legacy16:48
*** lpetrut has joined #openstack-nova16:48
*** ricolin has quit IRC16:50
*** trident has joined #openstack-nova16:51
*** ociuhandu has quit IRC16:52
*** BjoernT_ has quit IRC16:56
*** derekh has quit IRC16:57
*** takamatsu has quit IRC16:59
*** lpetrut has quit IRC17:01
*** BjoernT has joined #openstack-nova17:01
*** BjoernT has quit IRC17:05
Sundarmriedem, melwitt: Sorry, was in a call. I am referring to existing Nova unit tests e.g. https://github.com/openstack/nova/blob/master/nova/tests/unit/compute/test_compute.py#L9305. My patches do not anything with flavors etc. but are failing due to this issue: http://logs.openstack.org/44/631244/25/check/openstack-tox-py27/f76bf93/job-output.txt.gz17:09
Sundar#_2019-07-31_10_21_40_68484717:09
Sundarmriedem, melwitt: Sorry, was in a call. I am referring to existing Nova unit tests e.g. https://github.com/openstack/nova/blob/master/nova/tests/unit/compute/test_compute.py#L9305. My patches do not anything with flavors etc. but are failing due to this issue: http://logs.openstack.org/44/631244/25/check/openstack-tox-py27/f76bf93/job-output.txt.gz17:09
Sundar#_2019-07-31_10_21_40_68484717:09
*** icarusfactor has quit IRC17:10
Sundar[Yikes, message got duplicated]17:10
SundarIOW, these are not new test cases I am introducing.17:10
*** factor has joined #openstack-nova17:11
melwittyeah, so that makes sense considering the code you added. you'll just need to adjust the unit tests to accommodate the new code, like making the test pass Mock objects instead of empty dicts like it is doing right now17:11
melwittthis is a common thing, when adding new code, often existing unit tests will need to be adjusted along with the change. in this particular case, you uncovered a legacy thing (passing {} for objects) that should not be that way17:14
*** nafiux has joined #openstack-nova17:16
* stephenfin notes mypy will help highlight these issues in our tests once we can start using it in U17:18
Sundarmelwitt: Yes, I know. But there are a lot of unit tests like that. :(  Was hoping to get some alternatives.17:19
*** ociuhandu has joined #openstack-nova17:21
Sundarmelwitt, mriedem: I could adjust my code to check if request_spec is None. Hope that is also acceptable.17:22
*** ociuhandu has quit IRC17:26
melwittif there's a chance for request_spec to be None in reality, then that would work. if not, it's not really ok to do something in the impl just to get around unit tests that should be fixed. I'm not as well-versed about request_spec as mriedem, he probably would know most whether request_spec could possibly be None in this case17:27
*** udesale has quit IRC17:27
*** factor has quit IRC17:29
Sundarmelwitt: Kinda expected that response, but tried my luck anyway. ;)  Thank you.17:31
*** igordc has quit IRC17:32
mriedemSundar: for tests that hit this code https://review.opendev.org/#/c/631244/25/nova/compute/manager.py@1865 you should be passing a real RequestSpec object. for *existing* tests hitting that code, if they are passing {} or None or something, then they need to be updated to pass either a RequestSpec object or a Mock17:40
mriedemor, to simply mock out _create_and_bind_arqs17:41
mriedemsince those existing tests don't care about the details of that method17:41
mriedemand your *new* tests would call _create_and_bind_arqs directly17:41
mriedemwith a RequestSpec object17:41
mriedem- Failed: 15917:41
mriedemyikes17:41
mriedemi'm guessing that's a lot of cumulative failures since it's later in the series17:41
mriedemi.e. https://review.opendev.org/#/c/631243/2717:42
Sundarmriedem: Thanks. I counted 272 errors of that type over all. Mocking _create_and_bind_arqs is certainly easier.17:45
melwittSundar: a bit of advice, what I would do in this situation is write the impl and get the tempest tests passing (I assume something has been or needs to be set up that runs cyborg in it) first, get some review feedback once tempest coverage is passing, then fix the unit test and func test failures after you get some code review. that way you don't have to go through the work to fix hundreds of tests when the impl might change during17:45
melwitt code review17:45
mriedemyeah, or just mock out the thing semi-globally to start,17:46
mriedeme.g. in the patch that is failing tests on _create_and_bind_arqs, just stub that out in the test class setUp17:47
mriedemmelwitt: the tricky thing with tempest and this series is there is no tempest coverage with cyborg yet,17:47
mriedemwhen there is, it'll be a fake cyborg driver,17:47
mriedembut cyborg isn't hooked up with all of that yet, not even placement17:47
mriedemso tempest is a loooong ways off17:47
melwittoh :\ ok17:48
Sundarmelwitt: Makes sense. mriedem: We are setting up tempest: https://review.opendev.org/#/c/670999/17:48
*** maciejjozefczyk has quit IRC17:48
SundarBut I agree it is tough to predict when it'll land.17:49
openstackgerritDustin Cowles proposed openstack/nova master: Provider config file schema and loader  https://review.opendev.org/67334117:49
openstackgerritDustin Cowles proposed openstack/nova master: Provider config file schema and loader  https://review.opendev.org/67334117:52
*** ralonsoh has quit IRC17:57
*** kashyap has quit IRC18:04
*** tesseract has quit IRC18:05
mriedemmelwitt: looks like https://review.opendev.org/#/c/668123/3 is ready to go in queens18:07
mriedemlee was wanting that one18:07
melwitta-ha, cool. will review18:09
openstackgerritMerged openstack/nova master: Get rid of args to RBDDriver.__init__()  https://review.opendev.org/66856418:10
openstackgerritKashyap Chamarthy proposed openstack/nova master: libvirt: Handle alternative UEFI firmware binary paths  https://review.opendev.org/34839418:15
openstackgerritKashyap Chamarthy proposed openstack/nova master: libvirt: Parse the 'os' element from domainCapabilities  https://review.opendev.org/67379018:15
*** Sundar has quit IRC18:16
*** altlogbot_1 has quit IRC18:19
*** altlogbot_2 has joined #openstack-nova18:20
nafiuxTeam, what should I do to properly configure the overcommit of resources? I’m getting this error: “Got no allocation candidates from the Placement API. This could be due to insufficient resources or a temporary occurrence as compute nodes start up.”18:23
*** bbowen has joined #openstack-nova18:25
mriedemefried_rollin: since i'm not sure how this whole series works, i didn't vote on https://review.opendev.org/#/c/671072/2 but not sure if that should be using a set or list for forbidden_aggregates18:30
mriedemnafiux: you said you listed some resource providers, did you check their inventories to see that they have the allocation_ratio values you configured in nova? you also didn't say what release you were using.18:30
mriedemhttps://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-inventory-list18:30
nafiuxThanks @mriedem. I’m using stein. Let me take a look on that documentation.18:31
nafiuxHere what I have: https://pastebin.com/MdXsJYs318:33
mriedemnafiux: and https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#allocation-ratios if you didn't read that yet18:33
*** spatel has joined #openstack-nova18:33
spatelsean-k-mooney: are you there?18:33
mriedemyeah so your compute nodes configured ram_allocation_ratio is being reported in the MEMORY_MB inventory for the provider18:33
mriedem| MEMORY_MB      |              5.0 |    49139 |      512 |         1 |        1 | 49139 |18:33
mriedemthat looks fine18:33
sean-k-mooneyspatel: kind of yes18:33
mriedemnafiux: is this a fresh deploy or do you already have some servers on that node?18:34
nafiuxIt’s a fresh deployment, I don’t have any instance running at this point. I used kolla-ansible. I checked the /etc/nova/nova.conf inside the nova_scheduler container and I was able to see the allocation values:18:35
nafiuxdocker exec -it nova_scheduler bash18:35
spatelGood, I have machine with 64G and with this setting in grub  hugepagesz=2M hugepages=2750018:35
spatelI have this flavor setting  - properties                 | hw:cpu_policy='dedicated', hw:mem_page_size='large', hw:numa_nodes='2'18:35
nafiuxvi /etc/nova/nova.conf, and I can see inside [DEFAULT] the cpu_allocation_ratio = 16.0 and ram_allocation_ratio = 5.018:35
nafiuxI’m able to launch a server with 16GB of Ram and 8 VPCU, but I’m not able to launch a second one.18:36
nafiuxLet me launch one, and provide you the report again.18:36
*** igordc has joined #openstack-nova18:36
spatelflavor has 8vcpu / 12G memory18:37
spateltrying to create 2 VM but getting error, but i am able to create 1 vm ( not able to create 2)18:37
sean-k-mooneyyou should have enough hugepages although the kerenl may not alwasys blance it perfectly18:38
sean-k-mooneyyou have  enough clearance however that that should not be the issue18:38
sean-k-mooneyhow many cpu to you have18:39
spatel32 core18:39
-spatel- [root@ostack-compute-bld-gen-2-31 ~]# cat /sys/devices/system/node/node*/meminfo | fgrep Huge18:39
-spatel- Node 0 AnonHugePages: 0 kB18:39
-spatel- Node 0 HugePages_Total: 1375018:39
-spatel- Node 0 HugePages_Free: 1050018:39
-spatel- Node 0 HugePages_Surp: 018:39
-spatel- Node 1 AnonHugePages: 0 kB18:39
-spatel- Node 1 HugePages_Total: 1375018:39
-spatel- Node 1 HugePages_Free: 1050018:39
-spatel- Node 1 HugePages_Surp: 018:39
nafiux@mriedem: I was able to launch the first one, but whe I try to launch the second one I got the “Got no allocation candidates from the Placement API.” error18:39
sean-k-mooneyyou have ~20G free per numa node and you need 12 so it should not be hugepage related18:40
sean-k-mooneyyou have 32 cores is htat 2 16 core cpus or 2 8 core cpus with hyperthread or 1 32 core cpus18:41
mriedemnafiux: how much disk in the flavor? or are these volume-backed servers?18:41
mriedemnafiux: the placement-api also logs the allocation candidate filtering it does so you should trace the requestid through the placement logs - might need to enable debug18:41
spatelsean-k-mooney: that is what i am trying to find.. what is wrong here :(18:41
sean-k-mooneyspatel: it would be useful to know what the vcpu_pin_set is set to in your nova.conf18:42
mriedemthat should tell you why it kicked out the provider18:42
nafiuxOuch, I think that is it! Is the hard disk. Which actually doesn’t overcommit.18:42
nafiuxI have assigned a lot in the flavor, let me reduce it and test it again.18:42
mriedemyeah your provider only has ~300 GB of disk18:43
*** takamatsu has joined #openstack-nova18:43
spatelgrep  vcpu_pin_set /etc/nova/nova.conf    <--- empty18:43
spatelI have queen cluster and same config works there...18:43
mriedemnafiux: if you were doing volume-backed servers then you're probably fine, lots of vcpu and ram on that node18:43
spateli am running new cluster in stein18:43
mriedembut very little disk18:43
spatelwhich causing issue18:43
nafiux@mriedem yes, that is true, only 300 GB free, and the flavor has 160 GB, so only 1 instance with that capacity will be allowed.18:44
mriedem\o/18:45
nafiuxDoes "volume-backed" means the actuall physical hard disk of the hypervisor/nova node?18:46
mriedemno18:46
mriedemmeans the root disk is a cinder volume18:46
mriedemhttps://docs.openstack.org/nova/latest/user/launch-instance-from-volume.html18:47
nafiuxOh, I don’t have installed cinder. That will help a lot, if I understand correctly?18:48
mriedemi don't know how to answer that question :)18:48
mriedemif you have a compute node with 300 GB of disk and flavors with 160 GB of disk, you're not going to get far18:48
mriedemso you'd have to carve down your flavors to have smaller disk,18:48
mriedemor add more storage to your compute nodes,18:49
mriedemor use cinder to back the root disk18:49
nafiuxLol, yeah. How servers backed with cinder volumes behave? Do they use as needed and not the actual size specified at creation time?18:50
mriedemfor volume-backed servers the root gb on the flavor doesn't really apply18:53
mriedemthe size of the volume is whatever it is when it's created18:53
sean-k-mooneymriedem: im not sure i did a good job of explaing but it tried to explain the issue again im my reponce to your comments18:53
sean-k-mooneyhttps://review.opendev.org/#/c/669738/218:54
mriedemthe user can pre-create a volume and provide that to nova when creating the server, or the user can give some parameters to nova and nova will create the volume and attach it to the server18:54
sean-k-mooneybut yes i can factor the functional test our into there own regression module18:54
mriedemso many words....but i started that18:54
sean-k-mooneyand add teh rebuild/shelve tests18:54
mriedemnafiux: some people use cinder exclusively for root volumes of their servers, some people use shared storage for their compute nodes, some mix both - it takes all kinds18:55
sean-k-mooneynafiux: cinder backed volume behvior depend on teh dirver. some cinder backends support thin provisioning allowing over commit some dont18:57
sean-k-mooneyand some time the same dirver change behavior based on your config setting which makes it all the more fun to debug18:58
nafiuxThanks @mridem & @sean-k-mooney. I will read more about those topics, so, in general which are the different options for volumens to the instances?18:58
sean-k-mooneythere are two sets of volumes, volumes that have independent lifetime form the vm usually called data volumes19:01
sean-k-mooneythos are create via teh cinder api.19:01
sean-k-mooneyand then there are volumes that have ther life time cupled to a vm19:01
nafiuxOk, which is the equivalent to the root volumes in AWS?19:02
nafiuxCan I have a root volume with cinder?19:02
sean-k-mooneyyes19:02
sean-k-mooneywe call that boot form volume or bfv for short19:02
sean-k-mooneyin that case you tell nova that the root disk should not be create using the compute nodes default driver and instad should be provide by cinder19:03
nafiuxOh, so, due my current setup for learning, I will leverage cinder for both bfv and additional volumes as needed19:03
sean-k-mooneyyou can optionl specifythat that volume should be deleted when the vm is or not19:03
sean-k-mooneyi wasnt following the full conversation but you can use cinder for both19:04
nafiuxOk, yea, is just because I have a lot of CPU & Memory but few hard disk19:04
sean-k-mooneythe default backend for non bfv guest is to create a local file on the compute node for the root disk of the instance19:04
sean-k-mooneyyou can also tell the compute node to use a differnt default image backend such as ceph19:05
sean-k-mooneyin which case nova will create a remote ceph volume out of band of cinder19:05
nafiuxOk, I need to read a lot :D cinder, ceph, and will back later with more specific questions, this is really amazing.19:06
sean-k-mooneyother peopl in a similar situation somethime mount the default location where libvirt stores its disk image on nfs19:06
sean-k-mooneyso as mriedem said there are lots of way to do it and people do many different things but its better to keep it simple when learning19:07
*** whoami-rajat has quit IRC19:08
nafiuxAgree. Thanks for the support.19:08
mriedemnafiux: more info on cinder in their docs https://docs.openstack.org/cinder/latest/19:11
mriedemthere is also the ops guide which is older but relevant https://docs.openstack.org/operations-guide/19:11
sean-k-mooneyor if you like video content there are way more videos then you will have time to watch on teh offical youtube channel19:12
sean-k-mooneyhttps://www.youtube.com/user/OpenStackFoundation19:12
nafiuxSure! I will take a look on this too: https://docs.openstack.org/arch-design/design-storage/design-storage-concepts.html19:13
*** bbowen has quit IRC19:31
*** liuyulong has quit IRC19:35
*** takamatsu has quit IRC19:47
efried_rollinmriedem: responded. TL;DR: a Set would be better, but a list will work just fine.19:47
*** efried_rollin is now known as efried19:47
*** eharney has quit IRC19:48
efriedI guess that's not FUPable since it's an OVO/RPC thing. No takesy backseys.19:49
efriedso if you feel strongly about it you should -119:49
mriedemdone19:52
mriedemi've made more comments up that series but non-voting b/c i'm a coward19:53
mriedemi'd really like dan to review that series but he's out this week19:53
efriedmriedem: FYI Dan was involved earlier, which is how we ended up with the query and filter in their current form.19:54
mriedemyeah i knew he blew up the single megapatch19:55
efriedbut also was the one who suggested the db function as it now exists.19:56
efriedmuch nicer than it was before19:56
mriedemb/c it's mostly a copy of the one he added for his earlier (first) pre-filters19:56
mriedemfwiw most of my comments are "can you document this?"19:57
mriedemi also feel like forbidden is the wrong word for the user-facing side of this feature, it's really about host isolation19:57
mriedembut it's probably too late to change that now19:57
mriedemunless i'm just too slow to connect the dots, which is possible19:58
mriedemlots of negative logic in this feature makes it hard for me to reason about19:58
mriedemdon't not give me the hosts i don't not want19:58
efriedI'll have another look at the documentation, but I thought it was ultimately explained the right way for users.19:59
efriedThe logic is necessarily hairy and compound-negative-confusing19:59
efriedbut the ux winds up being really crisp and clean imo19:59
*** nafiux has quit IRC20:02
efried...you're right, the doc https://review.opendev.org/#/c/667952/10/doc/source/reference/forbidden-aggregates.rst still talks about forbidden aggregates. Though really, the explanation is very good, and "forbidden aggregates" is really just used as the name-of-feature.20:02
*** mchlumsky has quit IRC20:06
mriedemyeah i'm not -1 on it, it's just a bit confusing20:16
mriedemsome of the wording20:16
efriedI think you've got a valid point that we could s/forbidden/isolated/ and it would be more comprehensible. The docs could be fixed later, but if we want to rename the conf opt we should do it now.20:18
prometheanfiresean-k-mooney: ya, looks like it's lxml https://review.opendev.org/67384820:21
mriedemefried: question in https://review.opendev.org/#/c/670182/120:23
*** BjoernT has joined #openstack-nova20:23
efriedmriedem: responded20:24
*** BjoernT_ has joined #openstack-nova20:25
*** BjoernT has quit IRC20:28
*** med_ has quit IRC20:28
openstackgerritMatt Riedemann proposed openstack/nova stable/rocky: Fix python3 compatibility of rbd get_fsid  https://review.opendev.org/66451320:29
mriedemamorin: are you still working on https://review.opendev.org/#/c/667294/ ?20:31
*** eharney has joined #openstack-nova20:35
*** trident has quit IRC20:44
*** mriedem has quit IRC20:44
openstackgerritMerged openstack/nova master: neutron: log something more useful in _get_instance_nw_info  https://review.opendev.org/67369420:45
*** mriedem has joined #openstack-nova20:45
*** trident has joined #openstack-nova20:46
*** mriedem has quit IRC20:54
*** mriedem has joined #openstack-nova20:55
*** dillaman has joined #openstack-nova21:11
*** jdillaman has quit IRC21:15
*** jdillaman1 has joined #openstack-nova21:19
*** dillaman has quit IRC21:23
*** nafiux has joined #openstack-nova21:34
*** takamatsu has joined #openstack-nova21:40
efriedmriedem: RequestContext question for ya21:43
efriedI've been tracking where nova-compute talks to Placement to do things like creating the compute node RP.21:43
*** spatel has quit IRC21:44
efriedIIUC those interactions use a RequestContext that gets created when the service starts up, because they're done in a periodic, not based on a request coming from e.g. conductor or whatever.21:44
efriedwhich would be fine, except...21:45
efriedit doesn't seem to be the same context that's being used for logging.21:45
mriedemb/c the request id changes etc21:45
mriedemwhich is because of a thread local store for the context way up in oslo.context21:45
mriedemand it's mega confusing21:45
efriedrightright, but it's a *different* context?21:45
mriedemas mentioned https://bugs.launchpad.net/nova/+bug/1835958/comments/721:46
openstackLaunchpad bug 1835958 in OpenStack Compute (nova) "Nova sync power state on large clusters causes poor performance" [Undecided,New]21:46
efriedviz: http://logs.openstack.org/86/672986/2/check/nova-live-migration/0171fa9/logs/screen-n-cpu.txt.gz#_Jul_30_22_09_35_33485321:46
efriedhere ^ you can see that the log line is showing one request_id, but the context we're using for the placement request has a different request_id.21:46
mriedemreq-428963d8-bc71-4823-8e35-97db13886690 is just some admin context created when the compute service starts up right?21:47
mriedemhttps://github.com/openstack/nova/blob/master/nova/compute/manager.py#L132121:48
mriedemso you're using a random context there with a random request_id21:48
efriedreq-42... is the placement-side local request_id, just ignore that one.21:48
mriedembecause https://github.com/openstack/oslo.context/blob/master/oslo_context/context.py#L27321:48
mriedemi was looking at that one because21:49
mriedemJul 30 22:09:35.334349 ubuntu-bionic-rax-ord-0009499845 nova-compute[28986]: INFO nova.scheduler.client.report [None req-fb42d5e6-1c0d-41d0-88c5-a4f1ac127a41 None None] [req-428963d8-bc71-4823-8e35-97db13886690] Created resource provider record via placement API for resource provider with UUID 5dcb9522-0513-440a-80db-d985b5d45704 and name ubuntu-bionic-rax-ord-0009499845.21:49
efriedreq-fb42d5e6-1c0d-41d0-88c5-a4f1ac127a41 is the oslo.context one that oslo.log is paying attention to21:49
efriedreq-8a7e2cc5-3870-4118-b55f-4533d7a4de80 is the one in the RequestContext being used for the placement communication.21:49
efriedyeah, I know, I had to track that one down as well. I started this whole thing because I was wondering why tf we needed to print that reqid at all.21:50
efriedit's the one being sent *back* to us in the placement response headers, which turns out to be the (not-global!) request_id on the placement side21:50
efriedand because of this whole snafu, it's currently the only way for us to correlate the requests on the nova & placement sides21:50
efriedbut that flies in the face of the entire global_request_id concept21:51
efriedif we get global_request_id working, we don't need that at all because we'll be able to correlate with global_request_id, which is the whole friggin point.21:51
efriedso my question is this21:51
mriedemidk what oslo.log is doing really but i know things get wonky with this https://github.com/openstack/oslo.context/blob/master/oslo_context/context.py#L30021:52
efriedwhat about if, when we create the RequestContext on compute startup, we a) set its global_request_id, and b) pass overwrite=True so oslo.context replaces it in the threadlocal21:52
efriedyeah, b) would call update_store and replace that.21:53
efriedI think having talked through this I've got enough to do that ^ in a WIP and see what shakes out.21:54
mriedem....ok21:55
openstackgerritMerged openstack/nova master: Move adding vlans to interfaces to privsep.  https://review.opendev.org/63543621:56
mriedemi know melwitt https://review.opendev.org/#/c/524306/ and gibi https://review.opendev.org/#/c/542891/ have muddled in this and it's a mess21:56
mriedemso i guess welcome to the tarpits21:56
efriedmriedem: I see that get_context() and get_admin_context() explicitly set overwrite=False. The former even says it's to not mess with logging. Why??21:57
mriedemoh eric, i could tell you but that would be too easy21:57
*** BjoernT_ has quit IRC21:58
mriedemif you can hunt down vishy from 2012 maybe he can answer https://review.opendev.org/#/c/2923/21:58
mriedemor i guess alaski from 2016 https://review.opendev.org/#/c/378941/21:59
mriedemthe admin context stuff is also a bit legacy from when db apis were explicitly blocked unless you had an admin context, which alex_xu removed a long time ago22:01
mriedembut still affects project_id filtering in the model_query22:01
mriedemas melwitt can probably go into detail about22:01
mriedemas for the logging and thread local stuff, ....idk22:01
*** slaweq has quit IRC22:04
*** rcernin has joined #openstack-nova22:05
openstackgerritEric Fried proposed openstack/nova master: Store a RequestContext on service to reuse for periodic tasks  https://review.opendev.org/52430622:05
openstackgerritEric Fried proposed openstack/nova master: Poison context usage in periodic tasks  https://review.opendev.org/54289122:05
efriedmelwitt, gibi_off: FYI, rebased & resolved merge conflicts ^.  I'm going to throw one on top that overwrites the threadlocal context and creates a global_request_id so our logging might be sane. We'll see what explodes.22:07
efriedmriedem: fyi ^22:07
*** mlavalle has quit IRC22:07
openstackgerritEric Fried proposed openstack/nova master: WIP: Generate and log global_request_id properly  https://review.opendev.org/67392422:10
openstackgerritEric Fried proposed openstack/nova master: Move iptables rule fetching and setting to privsep.  https://review.opendev.org/63650822:13
openstackgerritEric Fried proposed openstack/nova master: Move dnsmasq restarts to privsep.  https://review.opendev.org/63928022:13
openstackgerritEric Fried proposed openstack/nova master: Move router advertisement daemon restarts to privsep.  https://review.opendev.org/63928122:13
openstackgerritEric Fried proposed openstack/nova master: Move calls to ovs-vsctl to privsep.  https://review.opendev.org/63928222:13
openstackgerritEric Fried proposed openstack/nova master: Move setting of device trust to privsep.  https://review.opendev.org/63928322:13
openstackgerritEric Fried proposed openstack/nova master: Move final bridge commands to privsep.  https://review.opendev.org/63958022:14
openstackgerritEric Fried proposed openstack/nova master: Cleanup the _execute shim in nova/network.  https://review.opendev.org/63958122:14
openstackgerritEric Fried proposed openstack/nova master: We no longer need rootwrap.  https://review.opendev.org/55443822:14
openstackgerritEric Fried proposed openstack/nova master: Cleanup no longer required filters and add a release note.  https://review.opendev.org/63982622:14
openstackgerritMerged openstack/nova master: Remove Request Spec Migration upgrade status check  https://review.opendev.org/67358022:24
efriedo/22:31
*** nafiux has quit IRC22:35
*** nafiux has joined #openstack-nova22:37
*** mriedem has quit IRC22:43
*** threestrands has joined #openstack-nova22:55
*** tkajinam has joined #openstack-nova22:57
*** panda has quit IRC23:03
*** panda has joined #openstack-nova23:04
*** slaweq has joined #openstack-nova23:11
*** slaweq has quit IRC23:15
*** vishwanathj has quit IRC23:36
*** betherly has joined #openstack-nova23:42
*** eharney has quit IRC23:45
*** betherly has quit IRC23:47
*** takamatsu has quit IRC23:59
« Tuesday, 2019-07-30 Index Thursday, 2019-08-01 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!