Wednesday, 2021-02-10

« Tuesday, 2021-02-09 Index Thursday, 2021-02-11 »
*** pmannidi has quit IRC00:07
*** pmannidi has joined #openstack-nova00:13
*** gmann_afk is now known as gmann00:19
*** macz_ has quit IRC00:27
openstackgerritGhanshyam proposed openstack/nova master: DNM:try l-c with direct deps  https://review.opendev.org/c/openstack/nova/+/77278000:36
*** pmannidi_ has joined #openstack-nova00:43
*** eharney has quit IRC00:45
*** pmannidi has quit IRC00:45
*** pmannidi_ has quit IRC00:48
*** macz_ has joined #openstack-nova00:52
*** spatel has quit IRC00:53
openstackgerritMerged openstack/placement master: policy: Deprecate 'admin_api' rule  https://review.opendev.org/c/openstack/placement/+/77233400:55
openstackgerritMerged openstack/placement master: policy: Add note about keystone's expansion of roles  https://review.opendev.org/c/openstack/placement/+/77275200:56
*** macz_ has quit IRC00:56
*** rchurch has quit IRC01:05
*** mlavalle has quit IRC01:07
*** rchurch has joined #openstack-nova01:07
*** sapd1 has joined #openstack-nova01:13
*** macz_ has joined #openstack-nova01:13
*** martinkennelly has quit IRC01:17
*** macz_ has quit IRC01:18
*** ircuser-1 has joined #openstack-nova01:29
*** pmannidi has joined #openstack-nova01:33
*** nweinber has quit IRC01:48
*** macz_ has joined #openstack-nova01:55
*** macz_ has quit IRC01:59
*** artom has quit IRC02:13
*** rcernin has quit IRC02:31
*** rcernin has joined #openstack-nova02:40
*** bbowen has quit IRC02:49
*** pmannidi has quit IRC02:49
*** pmannidi has joined #openstack-nova02:52
*** adriant has quit IRC02:58
*** adriant has joined #openstack-nova02:58
*** macz_ has joined #openstack-nova03:02
*** spatel has joined #openstack-nova03:04
*** macz_ has quit IRC03:07
*** rchurch has quit IRC03:14
*** rchurch has joined #openstack-nova03:15
*** artom has joined #openstack-nova03:19
*** macz_ has joined #openstack-nova03:23
*** hemanth_n has joined #openstack-nova03:25
*** macz_ has quit IRC03:28
*** khomesh24 has joined #openstack-nova03:41
*** macz_ has joined #openstack-nova03:44
*** psachin has joined #openstack-nova03:45
*** macz_ has quit IRC03:49
*** zzzeek has quit IRC03:56
*** zzzeek has joined #openstack-nova03:56
*** dviroel has quit IRC04:07
*** pmannidi has quit IRC04:07
*** pmannidi has joined #openstack-nova04:10
*** macz_ has joined #openstack-nova04:26
*** macz_ has quit IRC04:30
*** jmlowe has quit IRC04:37
*** amodi has quit IRC04:37
*** lemko has quit IRC04:37
*** hoonetorg has joined #openstack-nova04:40
*** jmlowe has joined #openstack-nova04:42
*** amodi has joined #openstack-nova04:42
*** lemko has joined #openstack-nova04:42
*** masterpe has quit IRC04:45
*** macz_ has joined #openstack-nova04:46
*** macz_ has quit IRC04:51
*** whoami-rajat__ has joined #openstack-nova04:57
*** openstackstatus has quit IRC04:58
*** openstack has joined #openstack-nova05:00
*** ChanServ sets mode: +o openstack05:00
*** hoonetorg has quit IRC05:05
*** pmannidi has quit IRC05:13
*** pmannidi has joined #openstack-nova05:16
*** masterpe has joined #openstack-nova05:18
*** spatel has quit IRC05:19
*** hoonetorg has joined #openstack-nova05:19
*** ratailor has joined #openstack-nova06:05
*** vishalmanchanda has joined #openstack-nova06:12
*** pmannidi has quit IRC06:19
*** pmannidi has joined #openstack-nova06:22
*** k_mouza has joined #openstack-nova06:22
*** k_mouza has quit IRC06:26
*** gyee has quit IRC06:54
*** eharney has joined #openstack-nova06:55
gibigmann: awesome, thanks07:14
*** khomesh24 has quit IRC07:26
*** pmannidi has quit IRC07:26
*** macz_ has joined #openstack-nova07:27
*** pmannidi has joined #openstack-nova07:29
*** macz_ has quit IRC07:32
*** dklyle has quit IRC07:38
*** rpittau|afk is now known as rpittau07:51
*** ralonsoh has joined #openstack-nova07:55
*** lpetrut has joined #openstack-nova08:04
*** sapd1 has quit IRC08:05
*** tesseract has joined #openstack-nova08:17
*** andrewbonney has joined #openstack-nova08:21
*** slaweq|away is now known as slaweq08:29
*** nightmare_unreal has joined #openstack-nova08:30
*** pmannidi has quit IRC08:41
*** zenkuro has joined #openstack-nova08:49
*** pmannidi has joined #openstack-nova08:50
*** rcernin has quit IRC08:56
*** tosky has joined #openstack-nova09:12
stephenfinCan someone hit this RBAC release note before I shove that context out of my brain? https://review.opendev.org/c/openstack/placement/+/774654/09:15
kashyaplyarwood: Hmm, I'm hitting this one on F32 DevStack - https://bugs.launchpad.net/devstack/+bug/188519809:22
openstackLaunchpad bug 1885198 in devstack "table `broute' is incompatible, use 'nft' tool" [Undecided,New]09:22
* kashyap tries w/o the DEVSTACK_PARALLEL patch, to rule it out09:23
*** macz_ has joined #openstack-nova09:28
openstackgerritVlad Gusev proposed openstack/nova stable/stein: Avoid lazy-loading instance.flavor in cold migration  https://review.opendev.org/c/openstack/nova/+/77479609:28
*** macz_ has quit IRC09:33
*** derekh has joined #openstack-nova09:33
lyarwoodkashyap: the failure was before that,09:37
lyarwood2021-02-09 18:08:18.069 | Cloning into '/home/stack/src/cloud/nova'...09:37
lyarwood2021-02-09 18:11:10.431 | error: RPC failed; curl 18 transfer closed with outstanding read data remaining09:37
kashyaplyarwood: Yeah, just noticed the 'curl' thing; ugh09:38
lyarwoodkashyap: why is it cloning into /home/stack/src/cloud/nova?09:38
kashyaplyarwood: Because I used a custom data dir out of habit09:38
lyarwoodstephenfin: did you want to add a link as suggested by lbragstad ?09:40
kashyaplyarwood: But it's not the dir that is reason; still looking why the 'curl' failed - perhaps my poor n/w succumbed to a Nova repo clone09:41
stephenfinlyarwood: ah, I hadn't seen that. Looking09:48
*** macz_ has joined #openstack-nova09:49
*** macz_ has quit IRC09:54
gibistephenfin: regarding the RBAC; the patch looks good to me. What do you want to do with Lance's comment?09:55
stephenfinI'm reworking it now, but perhaps I can use a follow-up?09:55
stephenfinOr not09:55
stephenfinIt'll just take me a bit to write this09:55
stephenfinthere's a lot of policies to account for :)09:56
stephenfin*there are09:56
gibistephenfin: do the write up and then ping me, I will push the patch through09:56
stephenfin(y)09:56
gibithanks09:56
*** martinkennelly has joined #openstack-nova10:00
*** jangutter has joined #openstack-nova10:07
kashyaplyarwood: Now it fails by trying not find a tenant n/w -- http://paste.openstack.org/show/802501/10:07
*** pmannidi has quit IRC10:07
* kashyap tries to shave this mini yak too10:07
lyarwoodkashyap: how many nics does your vm have?10:08
kashyaplyarwood: Just one; shall I add one more?  /me looks at his working older DevStack setup10:08
lyarwoodyeah I've always had two, I assume that's why it's unhappy10:09
kashyaplyarwood: Gah.  But ... in the name Zeus, I got it working w/ one NIC on an F31 deployment10:10
kashyapLemme try; sorry for the noise10:10
*** macz_ has joined #openstack-nova10:10
*** pmannidi has joined #openstack-nova10:11
*** macz_ has quit IRC10:15
openstackgerritMerged openstack/placement stable/ussuri: Fix Placement Doc  https://review.opendev.org/c/openstack/placement/+/76260710:25
openstackgerritMerged openstack/placement stable/train: Fix Placement Doc  https://review.opendev.org/c/openstack/placement/+/76260810:25
*** dtantsur|afk is now known as dtantsur10:38
*** belmoreira has joined #openstack-nova11:00
*** dviroel has joined #openstack-nova11:02
openstackgerritStephen Finucane proposed openstack/placement master: policy: Add releasenote for RBAC work  https://review.opendev.org/c/openstack/placement/+/77465411:10
openstackgerritStephen Finucane proposed openstack/placement master: Implement secure RBAC for reshaper  https://review.opendev.org/c/openstack/placement/+/77482711:10
stephenfingibi, lyarwood: ^11:11
stephenfinAlso spotted a policy that hadn't been updated. Don't know if that was intentional from lbragstad but I suspect not11:11
gibistephenfin: ack11:11
gibistephenfin: all looks good to me11:15
*** zenkuro has quit IRC11:20
*** zenkuro has joined #openstack-nova11:22
sean-k-mooneykashyap: you should aways use at least 2 nics for a devstack setup11:22
sean-k-mooney1 for ssh/managment and a second for ovs br-ex11:22
sean-k-mooneyyou can do it with one but devstack will try and move the ip and add it to the ovs bridge11:23
sean-k-mooneywhich can cause ssh to drop11:23
sean-k-mooneyso unless you run devstack in tmux or screen that can break things11:23
sean-k-mooneyyou used to be able to just not add any interface but they changed something recently11:24
sean-k-mooneyhavent spend time figuring out what11:24
*** k_mouza has joined #openstack-nova11:27
kashyapsean-k-mooney: I do run in a 'tmux', IIRC11:27
kashyapsean-k-mooney: In the past, I recall testing just fine w/ one network that is _not_ the 'default' libvirt network.  I created a separate NAT br to use for OpenStack n/w11:27
sean-k-mooneythen it should complete stacking then if you just have one nic11:28
sean-k-mooneydevstack pretty much just works if you use it today on ubuntu and centos. i got it to stack on fedroa too but that is less stable generally due to package conflicts11:29
sean-k-mooneye.g. between pip and dnf11:29
kashyapFedora is the beast I know; I'm gonna skip CentOS.  First, helping a different colleague11:36
kashyapsean-k-mooney: Unrelated: BTW, did you know that the "tsx=off" on kernel command-line will _not_ have any effect in certain cases?11:37
sean-k-mooneylike when its off already or the kernel does not have it  :P11:38
kashyapsean-k-mooney: Off on the kernel command-line; the processor itself has it11:38
sean-k-mooneycentos, centos stream and ubunut all work well11:38
kashyapsean-k-mooney: Reading the upstream documentation in the kernel commit, it points out that "tsx=off" will only have effect on newer CPUs that are not vulnerable to MDS (which is indicated by a microcode) -- https://github.com/torvalds/linux/commit/95c5824f75f311:39
sean-k-mooneykashyap: sure so it turns it off on procees that dont have it disabled by microcode11:39
sean-k-mooneyand for older cpus they are assuming you have teh microcode updated11:40
sean-k-mooneywhich sound resounable11:40
sean-k-mooneyif you want it turned off and there is a newer microcode that does that then you should update11:40
kashyapYeah; what a bloody mess this is11:41
openstackgerritStephen Finucane proposed openstack/nova master: Use neutronclient's port binding APIs  https://review.opendev.org/c/openstack/nova/+/70629511:42
sean-k-mooneykashyap: i just do this http://paste.openstack.org/show/802510/11:42
* kashyap clicks11:43
kashyapsean-k-mooney: Also, aside: even "tsx=off" does not remove it from /proc/cpuinfo -- and we should not rely on it anyway.  Because it only shows what the hardware has.11:44
sean-k-mooneywell that is the only thing we can check11:45
sean-k-mooneywhat would we use instead11:45
sean-k-mooneydmidecode?11:45
sean-k-mooneythat the only thing i can think off but i dont think that will help11:46
sean-k-mooneyif /proc/cpuinfo is wrong libvirt will be11:46
sean-k-mooney/proc/cpuinfo change when you have hypterthreading disable/enable in the bios not sure about at the kernel level11:46
sean-k-mooneyis that the delta11:47
sean-k-mooneybios vs kernel disable11:47
kashyapThere is no BIOS switch for it here.11:48
kashyapsean-k-mooney: But /proc/cpuinfo only shows what the hardware itself has -- the kernel parameter 'tsx=off' will _not_ physicallly remove it from /proc/cpuinfo; it just tells the OS not to use it11:49
kashyapSo, anyway, I'm jotting down some notes on this to see find out a reliable approach to check11:49
sean-k-mooneykashyap: well that is not always the case11:50
kashyapsean-k-mooney: Also, there _are_ other ways: We should instead check 'virsh domcapabilities' on the compute node to see the effect of 'tsx=off' parameter.11:50
sean-k-mooneykashyap: if you disabel hyperthreading in the bios ht is removed form /proc/cpuinfo11:50
kashyapsean-k-mooney: That's not the same -- TSX has no BIOS option.11:51
sean-k-mooneyright thats why i was asking is this a delta between disabeling a feature at the bios vs kernel11:51
sean-k-mooneye.g. /proc/cpuinfo is showing what the bios reports11:51
kashyapYep11:52
sean-k-mooneyoh i have tsx on my laptop11:53
sean-k-mooneyor at least hle11:53
sean-k-mooneywht the other one11:53
*** tbachman has quit IRC11:53
*** tbachman has joined #openstack-nova11:53
kashyap'rtm'11:53
sean-k-mooneyya i have both hle and rtm11:54
*** hemanth_n has quit IRC11:56
sean-k-mooneyskylake so it should be enabled by defult which it does11:58
sean-k-mooneykashyap: i tought you said the upstream kernel disabled tsx?11:58
kashyapsean-k-mooney: It did (like I said, but its effect depends on the processor); what kernel are you running?11:59
sean-k-mooneyim running 5.10.811:59
sean-k-mooneyfrom fedora 3211:59
kashyapsean-k-mooney: Yours is older11:59
kashyapsean-k-mooney: You need v5.11 or above11:59
kashyapI even mentioned it in today's agenda meeting notes :D11:59
*** lpetrut has quit IRC11:59
sean-k-mooneyso you ment it was disabled in an unreleased kernel12:00
sean-k-mooney5.11 is not released yet12:00
kashyapsean-k-mooney: Wait, it was disabled in Oct 201912:00
kashyapThere have been plenty kernels released since then: https://github.com/torvalds/linux/commit/95c5824f75f312:00
sean-k-mooneywell fedora has it enabled12:00
sean-k-mooneyand the kernel config has CONFIG_X86_INTEL_TSX_MODE_OFF=y12:01
sean-k-mooneyi guess this is what you ment by you cant trust /proc/cpuinfo12:02
sean-k-mooneylet me check in a vm12:02
kashyapsean-k-mooney: How are you saying it is "enabled"?  If you're looking in /proc/cpuinfo, it is not12:02
kashyaps/it is not/don't trust that/ :)12:02
kashyapsean-k-mooney: Yes, as you see, your config has it _OFF=y12:02
sean-k-mooneykashyap: that is the only interface we really have in userspace without poking registers12:03
sean-k-mooneyif we cant trust that we cant trust libvirt12:03
kashyapsean-k-mooney: Wait.  libvirt developers are telling me to use 'virsh domcapabilities' to detect what effect the 'tsx=off' parameter has12:04
kashyapsean-k-mooney: The only other option is to poke for the MSR -- which I have to look, even if it is possible12:05
kashyapsean-k-mooney: BTW, this not showing up "/proc/cpuinfo" is nothing new -- another flag is "mds-no" -- it also won't show up in /proc/cpuinfo12:06
sean-k-mooneycpuid show  RTM: restricted transactional memory     = true12:06
kashyapI wrote the upstream QEMU docs for it; /me looks12:06
*** lpetrut has joined #openstack-nova12:07
kashyapFWIW: https://lists.nongnu.org/archive/html/qemu-devel/2020-02/msg07164.html (qemu-cpu-models.rst: Document -noTSX, mds-no, taa-no, and tsx-ctrl)12:07
kashyapsean-k-mooney: So you had it off in the kernel; but the hardware supports it.  But `cpuid` shows RTM true?12:08
sean-k-mooneyyes cpu id shows true12:08
sean-k-mooneyand its off in the kernel12:09
*** khomesh24 has joined #openstack-nova12:10
sean-k-mooneyhttp://paste.openstack.org/show/802511/12:11
sean-k-mooneySkylake-Client-IBRS has it enabled12:11
kashyapYes, libvirt/QEMU introduced: "Skylake-Client-IBRS-noTSX"12:15
*** pmannidi has quit IRC12:15
*** pmannidi has joined #openstack-nova12:18
sean-k-mooneyya so right now i do have rtm and hle in a vm on my laptop12:20
sean-k-mooneywith libvirt 6.1.012:21
* kashyap --> hungry; back later12:23
*** zenkuro has quit IRC12:25
*** zenkuro has joined #openstack-nova12:25
*** mgariepy has quit IRC12:46
*** iurygregory has quit IRC12:51
sean-k-mooneystephenfin: requirements_cehcks is unhappy but everything else passed https://review.opendev.org/c/openstack/os-vif/+/77469413:03
*** ratailor has quit IRC13:04
*** macz_ has joined #openstack-nova13:07
*** macz_ has quit IRC13:11
openstackgerritMerged openstack/nova master: db: Compact Queens database migrations  https://review.opendev.org/c/openstack/nova/+/75908813:12
*** rcernin has joined #openstack-nova13:13
*** nweinber has joined #openstack-nova13:14
*** mgariepy has joined #openstack-nova13:14
*** rcernin has quit IRC13:18
*** pmannidi has quit IRC13:23
*** pmannidi has joined #openstack-nova13:27
*** macz_ has joined #openstack-nova13:27
*** macz_ has quit IRC13:32
*** nightmare_unreal has quit IRC13:47
*** macz_ has joined #openstack-nova13:48
*** macz_ has quit IRC13:53
*** spatel has joined #openstack-nova13:57
*** khomesh24 has quit IRC14:07
*** macz_ has joined #openstack-nova14:09
*** iurygregory has joined #openstack-nova14:13
*** macz_ has quit IRC14:14
*** pmannidi has quit IRC14:34
*** pmannidi has joined #openstack-nova14:35
*** psachin has quit IRC14:35
openstackgerritStephen Finucane proposed openstack/nova master: Use neutronclient's port binding APIs  https://review.opendev.org/c/openstack/nova/+/70629514:41
openstackgerritStephen Finucane proposed openstack/python-novaclient master: Add support for microversion v2.88  https://review.opendev.org/c/openstack/python-novaclient/+/77057314:45
*** macz_ has joined #openstack-nova14:47
*** macz_ has quit IRC14:51
*** zenkuro has quit IRC15:07
*** macz_ has joined #openstack-nova15:08
sean-k-mooneygibi: you can see nova security bug correct.15:08
*** zenkuro has joined #openstack-nova15:08
sean-k-mooneyis there a group i can check i want to see who is on the list15:09
gibisean-k-mooney: hm, I saw some before, but then I was explicitly included in them. I think I'm not in the secu group in lauchpad15:09
sean-k-mooneyah ok im being cauios with a bug im filing so im going to mark it as a security one initally but id like your input15:10
sean-k-mooneyill add you to cc directly just in case15:10
gibisean-k-mooney: yes that will work15:10
*** macz_ has quit IRC15:13
*** rcernin has joined #openstack-nova15:14
*** mgariepy has quit IRC15:15
*** rcernin has quit IRC15:18
openstackgerritStephen Finucane proposed openstack/placement master: Move policy deprecation to base rules  https://review.opendev.org/c/openstack/placement/+/77278415:22
stephenfingmann: I found/fixed the bug on that and am +2 on it now ^15:23
*** khomesh24 has joined #openstack-nova15:23
stephenfinlyarwood: When you're back, would appreciate you weighing in on https://review.opendev.org/c/openstack/nova/+/751368 (stable patch)15:24
gmannstephenfin: ah i see.  thanks.15:26
stephenfinit was so close to be perfect :D15:26
stephenfin*being15:26
gmannI thought we have broken something :)15:27
*** zenkuro has quit IRC15:28
*** macz_ has joined #openstack-nova15:29
*** zenkuro has joined #openstack-nova15:29
*** martinkennelly has quit IRC15:32
openstackgerritStephen Finucane proposed openstack/nova master: cmd: Remove 'nova-manage db ironic_flavor_migration'  https://review.opendev.org/c/openstack/nova/+/77377615:32
*** martinkennelly has joined #openstack-nova15:32
*** lpetrut has quit IRC15:33
stephenfingibi: You were +2 on that previously but I'd to address a doc failure. Care to revisit? Only change is to 'doc/source/cli/nova-manage.rst' ^15:33
gibisure15:33
stephenfinthanks15:33
*** macz_ has quit IRC15:33
stephenfingibi: We're _almost_ done with merging the main DB migration squash series (about time). We can probably look at removing that tweak to tox.ini once it's done, I guess?15:34
*** dklyle has joined #openstack-nova15:34
gibistephenfin: yepp, I can propose the rewert on top of the db series15:35
stephenfincool15:35
*** sapd1 has joined #openstack-nova15:35
* stephenfin rebases API DB series15:35
*** pmannidi has quit IRC15:43
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Add manage.py script  https://review.opendev.org/c/openstack/nova/+/77141915:44
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Liberty database migrations  https://review.opendev.org/c/openstack/nova/+/75939915:44
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Mitaka database migrations  https://review.opendev.org/c/openstack/nova/+/75940015:44
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Newton database migrations  https://review.opendev.org/c/openstack/nova/+/75940115:44
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Ocata database migrations  https://review.opendev.org/c/openstack/nova/+/75940215:44
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Pike database migrations  https://review.opendev.org/c/openstack/nova/+/75940315:44
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Queens database migrations  https://review.opendev.org/c/openstack/nova/+/75940415:44
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Rocky database migrations  https://review.opendev.org/c/openstack/nova/+/75940515:44
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Stein database migrations  https://review.opendev.org/c/openstack/nova/+/75940615:44
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Train database migrations  https://review.opendev.org/c/openstack/nova/+/77142015:44
*** mgariepy has joined #openstack-nova15:44
stephenfinlyarwood, bauzas, melwitt: Can one of you take a look at https://review.opendev.org/c/openstack/nova/+/773727/ and https://review.opendev.org/c/openstack/nova/+/773728/ to fix the PPC64 gate, please15:44
*** pmannidi has joined #openstack-nova15:46
bauzasstephenfin: kinda try15:47
openstackgerritStephen Finucane proposed openstack/os-vif master: Resolve dependency issues  https://review.opendev.org/c/openstack/os-vif/+/77469415:48
openstackgerritBalazs Gibizer proposed openstack/nova master: Revert "Run the db migration tests in the same test worker"  https://review.opendev.org/c/openstack/nova/+/77488915:48
gibistephenfin: here you are ^^15:49
stephenfin+215:49
*** macz_ has joined #openstack-nova15:49
gibiI would keep the elastic signature still in place to see if the compaction really give us the speed bump on the gate I measured locally15:50
bauzasstephenfin: https://review.opendev.org/c/openstack/nova/+/773728/ isn't needed for fixing the ppc64 gate job, righrt?15:53
*** macz_ has quit IRC15:53
stephenfinbauzas: Yes, that's the crucial bit in fact15:54
*** khomesh24 has quit IRC15:54
bauzasok, then I misunderstood it15:54
bauzaswill review it15:54
stephenfinhttp://lists.openstack.org/pipermail/openstack-discuss/2021-January/020088.html15:54
stephenfintl;dr: I added a patch to disable the USB controller if there were no USB devices in the XML, however, libvirt adds a USB keyboard and mouse automatically for guests on PPC64 hosts15:55
stephenfinso it now fails15:55
bauzasaha I see15:55
bauzasit wants to add both the kb and the mouse but there are no longer usb controllers15:56
stephenfinyup15:56
bauzasgotcha, will look15:56
stephenfinso we skip that logic on PPC64 now since we know we'll always need a controller on that platform15:56
lyarwoodstephenfin: sorry wasn't watching irc, I'll take a look at that and the other stuff from earlier after our next call15:59
*** bbowen has joined #openstack-nova16:08
*** efried has quit IRC16:08
*** macz_ has joined #openstack-nova16:10
*** efried has joined #openstack-nova16:11
*** mlavalle has joined #openstack-nova16:16
*** mlavalle has quit IRC16:16
*** mlavalle has joined #openstack-nova16:17
dansmithgibi: that's nifty, I didn't know about that grouping thing16:32
dansmithgibi: did you happen to notice if the IO-related timeouts were specific to one provider?16:33
dansmithtripleo was having job timeouts on one of their jobs that did a lot of IO and vexxhost switched our flavor to double the IOPs16:34
gibidansmith: the failures was spread amongst providers. at least I haven't noticed a pattern16:35
dansmithokay16:35
gibiI assumed it was IO limited as I was able to reproduce the extreme slowness locally if I IO deprived the machine16:35
dansmithmy parallel devstack ran all of those migrations in parallel for a very nice speedup on local hardware,16:36
dansmithso maybe with 75% faster due to compaction we'll get a nice bump in the io-limited workers too16:36
dansmithgibi: yeah probably16:36
gibiyeah I saw you parallel devstack patch, that also looks like a nice bump. I will try it at some point16:36
dansmithyeah, hopefully we can merge it soon16:37
dansmithit's super nice to be able to stack locally in five minutes :)16:37
* gibi goes and try a stack without and then with dansmith's patch locally16:38
*** ianw has quit IRC16:39
*** ianw has joined #openstack-nova16:39
dansmithdevstack also runs db sync for keystone three times for some reason I have yet to figure out,16:39
dansmithand while they don't have hundreds of migrations, it's non-trivial16:39
*** nweinber has quit IRC16:43
*** nweinber has joined #openstack-nova16:44
openstackgerritElod Illes proposed openstack/nova stable/ussuri: Add upgrade check about old computes  https://review.opendev.org/c/openstack/nova/+/77076516:52
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Record the machine_type of instances in system_metadata  https://review.opendev.org/c/openstack/nova/+/76753316:52
openstackgerritLee Yarwood proposed openstack/nova master: nova-manage: Add machine_type get command  https://review.opendev.org/c/openstack/nova/+/76954816:52
openstackgerritLee Yarwood proposed openstack/nova master: nova-status: Add hw_machine_type check for libvirt instances  https://review.opendev.org/c/openstack/nova/+/77064316:52
openstackgerritLee Yarwood proposed openstack/nova master: nova-manage: Add machine_type update command  https://review.opendev.org/c/openstack/nova/+/77489616:52
openstackgerritLee Yarwood proposed openstack/nova master: WIP nova-manage: Add machine_type list_unset command  https://review.opendev.org/c/openstack/nova/+/77489716:52
openstackgerritLee Yarwood proposed openstack/nova master: WIP libvirt: Add a config update workflow func test for [libvirt]hw_machine_type  https://review.opendev.org/c/openstack/nova/+/77489816:52
openstackgerritLee Yarwood proposed openstack/nova master: WIP docs: Add admin docs for configuring and updating machine types  https://review.opendev.org/c/openstack/nova/+/77489916:52
*** jangutter_ has joined #openstack-nova17:00
*** jangutter has quit IRC17:01
*** nweinber has quit IRC17:11
*** jangutter has joined #openstack-nova17:12
*** nweinber has joined #openstack-nova17:12
*** jangutter_ has quit IRC17:15
openstackgerritStephen Finucane proposed openstack/nova master: tests: Posion nova.objects.fields.Architecture.from_host  https://review.opendev.org/c/openstack/nova/+/76992017:15
openstackgerritStephen Finucane proposed openstack/nova master: tests: Move 'nova.virt.libvirt.utils.get_arch' stub to fixture  https://review.opendev.org/c/openstack/nova/+/77490417:15
openstackgerritStephen Finucane proposed openstack/nova master: trivial: Remove unnecessary mock  https://review.opendev.org/c/openstack/nova/+/77490517:15
stephenfinlyarwood: Fixed that, I think ^17:16
lyarwoodawesome, I'll take a look in the morning once the tests have passed17:19
stephenfinack17:20
*** pmannidi has quit IRC17:30
*** pmannidi has joined #openstack-nova17:32
gibidansmith: I saw ~17% speedup locally in devstack stacking with parellelism enabled. http://paste.openstack.org/show/802536/17:33
gibidansmith: two interesting observations; 1) osc time went up when run in parallel 2) unacounted time become negative :)17:33
dansmithgibi: is that with the db migrations squashed?17:34
dansmithmerged I mean17:34
gibidansmith: squashed up until Queens17:34
gibidansmith: I will rerun now with all the db squashes17:35
dansmithokay, I'm not really sure all of the component timing actually works with async because they're run in subshells and the timings don't really get reflected to the parent, AFAIK17:35
gibiahh, I see17:36
dansmiththat's 25% increase from top to bottom,17:37
*** nweinber has left #openstack-nova17:37
dansmithI'm not sure why you'd see a difference with async patches applied but disabled, should be identical, unless you're in a VM with neighbor noise affecting the numbers17:37
gibidansmith: could be noise and could be that apt did updated things at the first run but not at the subsequent runs17:39
dansmithah okay yeah17:39
dansmithcompacting the migrations will definitely reduce the gain from the async tasks, since they're doing less, thus less to parallelize17:39
dansmithbut still better overall with both of course17:40
gibiyeah17:40
gibialso based on experience having tempest enabled make also a significant difference17:40
dansmithdid you pull the latest async? it should dump async-specific numbers after the component timing17:41
dansmithyeah, definitely.. we rebuild the tempest venv several times :/17:41
gibidansmith: I pulled https://review.opendev.org/c/openstack/devstack/+/774022/517:41
dansmithokay then you should have it when async is enabled17:42
*** tesseract has quit IRC17:43
dansmithlike this: https://zuul.opendev.org/t/openstack/build/4f1863e956364177bf2152920636ff0b/log/job-output.txt#2493217:43
gibioh I see now in the logs, I just forget to copy it17:44
gibihttp://paste.openstack.org/show/802538/17:44
dansmiththat's similar to what I see locally, even before the compaction, interesting17:44
gibithe bigger half of the compation is already merged and I have it in that run17:46
dansmithI'm not sure I've RECLONE=yes since that stuff started to merge, so I'll try17:47
dansmithmaybe with the lower IO related to compaction we're parallelizing more things that won't compete, like just python startup and sql generation or something17:48
dansmithbut regardless, the two of these things should be noticeably better for the gate which is awesome17:48
gibithis is the run with all the db compation pulled http://paste.openstack.org/show/802539/17:48
dansmithcool17:49
dansmithhard to complain about 80% faster :)17:49
gibi:)17:49
*** sapd1 has quit IRC17:49
gibiI'm not complaining :d17:49
gibi:D17:49
dansmithbauzas: are you still working on this? https://review.opendev.org/c/openstack/nova/+/76145217:54
bauzasdansmith: it's my next duty after the routed networks17:55
dansmithokay cool17:55
dansmithjust making sure17:55
bauzasdansmith: but i'll need you to look at it ;)17:55
bauzasI got some issue with the CI17:55
dansmithack, it's in merge conflict now anyway, but poke me when you get back to it and have something for me to dig into17:56
bauzasso I'll rebase with the next API version, and then uploading it, so we would get new CI17:56
dansmithyeah cool17:56
*** derekh has quit IRC18:00
openstackgerritMerged openstack/nova master: db: Compact Rocky database migrations  https://review.opendev.org/c/openstack/nova/+/75908918:02
openstackgerritMerged openstack/nova master: libvirt: Stub out 'nova.virt.libvirt.utils.get_arch'  https://review.opendev.org/c/openstack/nova/+/77372718:04
stephenfindansmith: gibi: FYI the DB compaction series doesn't seem to have fixed the issue :-( https://review.opendev.org/c/openstack/nova/+/774889/18:12
stephenfinassuming I'm reading that correctly18:12
dansmithmaybe something else is getting hung up?18:12
stephenfinquite possibly. I haven't investigated yet18:12
stephenfinFair to say that probably shouldn't go in until we root cause it though18:13
gibiI'm a sad panda18:13
gibistephenfin: you are readig it right18:13
gibithe I have to assume that the way I was able to reproducing it locally was actually reproduced the symptom but in a different way how it happens on the gate18:15
gibimaybe we can try to dump logs at each step of test_walk_versions to see if there is a certain step that has high chance to hang18:16
gibibut not today18:21
gibio/18:21
*** rpittau is now known as rpittau|afk18:23
*** k_mouza has quit IRC18:33
*** pmannidi has quit IRC18:38
*** spatel has quit IRC18:40
*** belmoreira has quit IRC18:41
*** pmannidi has joined #openstack-nova18:43
*** spatel has joined #openstack-nova18:44
*** dtantsur is now known as dtantsur|afk18:45
*** hemna has quit IRC18:53
*** hemna has joined #openstack-nova18:54
*** whoami-rajat__ has quit IRC18:57
*** zenkuro has quit IRC18:57
*** zenkuro has joined #openstack-nova18:58
*** ralonsoh has quit IRC19:17
*** rchurch has quit IRC19:17
*** rchurch has joined #openstack-nova19:19
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Ocata database migrations  https://review.opendev.org/c/openstack/nova/+/75940219:20
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Pike database migrations  https://review.opendev.org/c/openstack/nova/+/75940319:20
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Queens database migrations  https://review.opendev.org/c/openstack/nova/+/75940419:20
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Rocky database migrations  https://review.opendev.org/c/openstack/nova/+/75940519:20
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Stein database migrations  https://review.opendev.org/c/openstack/nova/+/75940619:20
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Train database migrations  https://review.opendev.org/c/openstack/nova/+/77142019:20
*** zzzeek has quit IRC19:26
*** zzzeek has joined #openstack-nova19:27
sean-k-mooneyoh your talking about the db walk timeout ? or test_models_sync19:38
sean-k-mooneythe time out ligitablity might just be a side effect of a slow node. gibi tried to extend the time previousl but the other issue in test_models_sync are new to me19:39
*** andrewbonney has quit IRC19:39
sean-k-mooneythat kind of looks like the db tables did not create properly19:41
stephenfingmann: Could you take a look at https://review.opendev.org/c/openstack/placement/+/774827/ too. Seems it was missed previously19:41
stephenfinThat will unblock the release note19:41
*** pmannidi has quit IRC19:48
*** slaweq has quit IRC19:49
*** pmannidi has joined #openstack-nova19:49
*** zzzeek has quit IRC19:53
openstackgerritMartin Kopec proposed openstack/nova stable/rocky: DNM - testing tempest-slow job  https://review.opendev.org/c/openstack/nova/+/77500319:55
*** zzzeek has joined #openstack-nova19:57
*** zzzeek has quit IRC20:21
*** gyee has joined #openstack-nova20:21
*** zzzeek has joined #openstack-nova20:22
gibisean-k-mooney: the db compation should have been sped up the db migration test a lot20:27
gibisean-k-mooney: but it still times out20:27
sean-k-mooneyyep it should have helped but it was not gurenteed to fix it20:27
sean-k-mooneyjust make it much much less likely20:27
dansmithin terms of IO, I guess the only thing that it necessarily does less of is when we drop stuff20:28
gibiwe do a lot less testing now20:28
dansmithI guess I'm not sure how much thrashing there really is.. the initial migration was always the bulk of it20:28
dansmithyeah, right, okay in that test where we walk the migrations, that's right20:29
sean-k-mooneydansmith: the table_walk i think is doing each migration one by one20:29
sean-k-mooneyso that "shoudl" be run the compacted migration + stuff form this cycle20:29
sean-k-mooneyat least when they are all merged20:29
dansmithyeah, I was thinking about the total IO load of starting with an empty DB and getting to a current schema,20:30
dansmithbut the test will obviously do fewer things20:30
sean-k-mooneydansmith: so pararrle devstack + osc as a service reduced stack time 40%20:38
sean-k-mooneygranted that wont be a 40% speed up in job time as we spend or 50% of it running tempest20:38
sean-k-mooneybut that should still be 10-15% maybe overall20:39
sean-k-mooneythat pretty dramatic20:39
sean-k-mooney+ the developer workflow improment20:40
sean-k-mooneythat said i have been used to sub 20 min stack times basiclay for ever due to how i use it but 5-10 min stack times is still great20:41
dansmithyep, very non-trivial savings multiplied across every job runm20:41
dansmithand local dev time improvement is huge20:42
sean-k-mooneyya i used to lement not being able to use offline mode or wehn i could not just restart the service due to db changes20:42
sean-k-mooneybut if its that quick honestly i would restack more often then i do20:43
dansmith5 minutes is on the order of "hang on let me stack so I can answer that CLI usage question in realtime"20:43
sean-k-mooneyyep20:43
sean-k-mooneyi have done that in meeting wtih a 15min time but 5 mins is shorter then ill stack this while i get coffee20:44
*** pmannidi has quit IRC20:53
*** pmannidi has joined #openstack-nova20:56
*** xek has joined #openstack-nova21:06
sean-k-mooneyis the build_requests table in the api db used anymore?21:47
sean-k-mooneyit seams to always be empty21:47
sean-k-mooneymaybe we automatically delete the entry once it builds21:48
*** atmark has joined #openstack-nova21:50
atmarkhello, is it possible to rename a cell ?21:50
sean-k-mooneycell0 no the other cells im not sure21:52
sean-k-mooneywithout db change i dont think there is any nova manager command you can updated21:52
sean-k-mooneythe host_mappings just use the cell_id im not sure what if anything uses the cell name corm the cell_mappings tables21:54
artomatmark, also, *why*?21:54
sean-k-mooneyya even if you can do it you proably should not do it unless you have a good reason too21:56
sean-k-mooneyi belive we map cells to placment aggreates at some point. im not sure if that is by name or by id but i would not assume there is nothing that depends on the cell name without fully checking21:57
sean-k-mooneyyou cant rename Avaiableity zones with out breaking things for example. gratted they exists at the api level and cells do not but the cell name might be storeed somewhere although we should really be usign the id.21:59
*** xek has quit IRC22:02
gmannstephenfin: done. +A22:02
*** pmannidi has quit IRC22:03
*** pmannidi has joined #openstack-nova22:06
*** spatel has quit IRC22:09
openstackgerritMerged openstack/nova master: libvirt: Always enable USB controller on PPC64  https://review.opendev.org/c/openstack/nova/+/77372822:12
openstackgerritGhanshyam proposed openstack/placement master: Move policy deprecation to base rules  https://review.opendev.org/c/openstack/placement/+/77278422:13
gmannstephenfin: updated this for reshaper policy - https://review.opendev.org/c/openstack/placement/+/77278422:15
*** zzzeek has quit IRC22:37
*** zzzeek has joined #openstack-nova22:37
*** zzzeek has quit IRC22:50
*** vishalmanchanda has quit IRC22:51
*** zzzeek has joined #openstack-nova22:52
*** zzzeek has quit IRC23:02
*** zzzeek has joined #openstack-nova23:04
openstackgerritGhanshyam proposed openstack/nova master: Switch to new rolevar for run-temepst role  https://review.opendev.org/c/openstack/nova/+/77504023:11
*** pmannidi has quit IRC23:14
*** pmannidi has joined #openstack-nova23:15
*** zzzeek has quit IRC23:27
sean-k-mooneyoh dead code i can channel my inner stephenfin and delete it :)23:27
sean-k-mooneythe only thin that uses _load_legacy_requests is get_by_instance and the only thing that uses that is test code23:28
sean-k-mooneyhttps://github.com/openstack/nova/blob/5b52db409b00e6257e83598efa0d8528ebe380e0/nova/objects/instance_pci_requests.py#L116-L14623:28
*** zenkuro has quit IRC23:28
sean-k-mooneywe use get_by_instance_uuid instead now23:28
sean-k-mooneysince we started storign the request spec in teh api_db like 4? 5? cycles ago23:29
sean-k-mooneyits been a very long time since it was stored in teh instace_system_metadata table23:29
*** zzzeek has joined #openstack-nova23:29
sean-k-mooneyi think maybe melwitt moved it back in queens/rocky? maybe older then that23:31
melwittsean-k-mooney: afaik request spec has always been in the api db i.e. it didn't exist prior to that. I didn't move anything related to it :)23:32
sean-k-mooneyits existed since like juno  it was made an object back in 2014 https://github.com/openstack/nova/commit/2d998c8df2048167c2db3694d3aae4605ff2625c23:33
melwittI mean request spec23:34
*** zzzeek has quit IRC23:34
sean-k-mooneyoh sorry i said request spec23:34
sean-k-mooneyi ment instace_pci_request23:35
sean-k-mooneysorry im currently trying to figure out why the numa policy i am setting is not makeing it to the db23:35
*** zzzeek has joined #openstack-nova23:36
sean-k-mooneywhen i grab it out of the db {"count": 1, "spec": [{"physical_network": "public"}], "alias_name": null, "is_new": false, "numa_policy": null, "request_id": "6ca3ee2f-59d7-4828-b95a-f503f7eb261d", ...23:36
melwittare you doing like request_spec.pci_requests = <stuff> and then request_spec.save()?23:37
sean-k-mooneythe instance_extra table to be exact its reset to null23:37
melwitthow are you setting them23:37
sean-k-mooneyam im not doing the save but ill show you one sec23:37
sean-k-mooneyhttps://review.opendev.org/c/openstack/nova/+/773792/1/nova/network/neutron.py#215123:38
sean-k-mooneywe are modifying it just above too  spec[pci_request.PCI_TRUSTED_TAG] = str(trusted)23:39
sean-k-mooneyno not quite23:39
sean-k-mooneythe oter values are all passed in23:39
melwittoh, I can see this object doesn't work like the typical23:40
sean-k-mooneywell it should get commits with the rest of the requests23:40
sean-k-mooneycreate_resource_requests23:40
sean-k-mooneyis invoked during the build process and we pass back a set of resouce requests23:41
sean-k-mooneytechnically i gues there realy is no reason i could not pass the affinity policy to the consturor i guess23:41
sean-k-mooneybut what im doing should work23:41
melwittgenerally, the pci tracker is the thing that writes all pci related things to the db, so my first thought is maybe it doesn't handle numa_policy yet?23:42
sean-k-mooneyi really hope https://github.com/openstack/nova/commit/59d94633518e6f6272e9f0654bb908e332f97a96 added that23:45
sean-k-mooneyor else i have to talk to stephenfin about his commit naming23:45
melwittoh, hm23:45
melwittlol23:45
sean-k-mooneyi mean i know im missing unit/functional tests in that change and when i add them i might find out why its not working but still23:47
sean-k-mooney                numa_policy = port_numa_policy or affinity_policy23:47
sean-k-mooney                if numa_policy:23:47
sean-k-mooney                    request.numa_policy = numa_policy23:47
sean-k-mooneythat should work23:47
melwittyeah.. I don't see anything either23:48
sean-k-mooneyi think tomorrow ill just start writing a functional test. i need to anyway but i was hoping to test it manually once first to make sure it worked23:49
sean-k-mooneybecause its being lost before it gets to the db it does not.23:50
melwittseems like it would have to mean port_numa_policy or affinity_policy are both None?23:50
sean-k-mooneyyep23:50
sean-k-mooneyaffinity_policy is None23:50
melwittor empty in some way23:50
sean-k-mooneywell none or empty ya23:50
sean-k-mooneyport_numa_policy in my testing is required23:50
sean-k-mooneywell23:50
sean-k-mooneythe policy on the port is required23:51
sean-k-mooneythat does not mean i dont have a different bug23:51
*** CeeMac has quit IRC23:51
sean-k-mooneyim reading the value form the port here https://review.opendev.org/c/openstack/nova/+/773792/1/nova/network/neutron.py#205323:52
sean-k-mooneyi fixed the spelling localy23:52
melwittis it possible is null or empty from neutron itself?23:52
sean-k-mooneynot any more alos https://review.opendev.org/c/openstack/nova/+/773792/1/nova/network/neutron.py#303923:53
melwittor the constant is wrong?23:53
sean-k-mooneyi also extended the nova VIF object23:54
sean-k-mooneyto store it and if i look in the network info cache its set to required23:54
sean-k-mooneyand i create the port before i booted the vm23:54
sean-k-mooneyso i could check it was set23:54
sean-k-mooneyso the VIF object makes it to the db fine23:54
melwittoh ok. hrm23:55
sean-k-mooneyya its weird23:55
sean-k-mooneyi did spell consttants wong but thats what i fixed locally23:55
sean-k-mooneyoh....23:55
sean-k-mooneythat would do it https://review.opendev.org/c/openstack/nova/+/773792/1/nova/network/neutron.py#203923:56
sean-k-mooneyif i dont ask for the field... then my  numa_policy = port.get(consttants.NUMA_POLICY, None) is always going to be None23:57
sean-k-mooneywe normlly dont filter like that so i never check for it23:57
melwittoh oops23:58
melwittat least you found it :)23:58
sean-k-mooneyyep23:58
sean-k-mooneyi did have a probalem before where i also forgot to trun the extention on in neutorn fully23:59
sean-k-mooneyso i would create the port, osc would show the policy in the port in the respocne but neutorn never saved it to the db on its end23:59
« Tuesday, 2021-02-09 Index Thursday, 2021-02-11 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!