| *** markvoelker has joined #openstack-nova | 00:00 | |
| *** markvoelker has quit IRC | 00:05 | |
| *** dtruong has quit IRC | 00:07 | |
| *** dtruong has joined #openstack-nova | 00:07 | |
| *** owalsh has joined #openstack-nova | 00:13 | |
| *** TxGirlGeek has quit IRC | 00:17 | |
| alex_xu | mriedem_away: thanks :) | 00:19 |
|---|---|---|
| *** markvoelker has joined #openstack-nova | 00:22 | |
| *** mlavalle has quit IRC | 00:25 | |
| *** mlavalle has joined #openstack-nova | 00:27 | |
| mriedem_away | i love SNES | 00:31 |
| mriedem_away | and am playing FF1 on PS1 lately because | 00:32 |
| mriedem_away | but i was actually physically at a park with my kid | 00:32 |
| mriedem_away | with air and sun | 00:32 |
| mriedem_away | wild | 00:32 |
| *** mriedem_away is now known as mriedem | 00:32 | |
| *** mriedem has quit IRC | 00:33 | |
| *** gyee has quit IRC | 00:36 | |
| openstackgerrit | Brin Zhang proposed openstack/nova master: Remove the invalid AZ server create test https://review.opendev.org/678116 | 00:38 |
| *** ozzzo has left #openstack-nova | 00:42 | |
| *** markvoelker has quit IRC | 00:47 | |
| *** brinzhang_ has joined #openstack-nova | 00:51 | |
| *** brinzhang has quit IRC | 00:54 | |
| *** bbowen has joined #openstack-nova | 00:58 | |
| *** larainema has joined #openstack-nova | 01:00 | |
| *** bhagyashris has joined #openstack-nova | 01:11 | |
| openstackgerrit | Brin Zhang proposed openstack/nova master: Specify availability_zone to unshelve https://review.opendev.org/663851 | 01:17 |
| *** tbachman has quit IRC | 01:24 | |
| *** tbachman has joined #openstack-nova | 01:26 | |
| *** sapd1_x has joined #openstack-nova | 01:30 | |
| *** gbarros has joined #openstack-nova | 01:42 | |
| *** mmethot has joined #openstack-nova | 01:57 | |
| openstackgerrit | Bhagyashri Shewale proposed openstack/nova master: Ignore root_gb for BFV in simple tenant usage API https://review.opendev.org/612626 | 01:58 |
| *** gbarros has quit IRC | 02:05 | |
| openstackgerrit | Merged openstack/nova master: Add docstring to check_availability_zone function https://review.opendev.org/677858 | 02:11 |
| *** dave-mccowan has joined #openstack-nova | 02:35 | |
| *** markvoelker has joined #openstack-nova | 02:40 | |
| *** boxiang has quit IRC | 02:44 | |
| *** zhubx has joined #openstack-nova | 02:44 | |
| *** markvoelker has quit IRC | 02:46 | |
| *** psachin has joined #openstack-nova | 03:02 | |
| *** KeithMnemonic1 has quit IRC | 03:06 | |
| *** KeithMnemonic has joined #openstack-nova | 03:12 | |
| *** markvoelker has joined #openstack-nova | 03:20 | |
| *** markvoelker has quit IRC | 03:25 | |
| openstackgerrit | Guo Jingyu proposed openstack/nova master: Define and catch InstanceAgentNotRunning for set-password failure https://review.opendev.org/661466 | 03:30 |
| openstackgerrit | zhufl proposed openstack/nova master: [Trivial]Remove used helper get_vif_devname_with_prefix https://review.opendev.org/678136 | 03:38 |
| openstackgerrit | Brin Zhang proposed openstack/nova master: Specify availability_zone to unshelve https://review.opendev.org/663851 | 03:43 |
| *** mlavalle has quit IRC | 03:47 | |
| openstackgerrit | Merged openstack/nova master: objects: Rename 'nova.objects.instance_numa_topology' https://review.opendev.org/671789 | 03:54 |
| openstackgerrit | Merged openstack/nova master: libvirt: Remove unnecessary try-catch around 'getCPUMap' https://review.opendev.org/671790 | 03:55 |
| openstackgerrit | Merged openstack/nova master: libvirt: Remove unnecessary argument https://review.opendev.org/674396 | 03:55 |
| openstackgerrit | Merged openstack/nova master: claims: Remove useless caching https://review.opendev.org/671791 | 03:55 |
| *** ricolin has quit IRC | 04:14 | |
| *** ricolin has joined #openstack-nova | 04:21 | |
| *** udesale has joined #openstack-nova | 04:23 | |
| *** mkrai has joined #openstack-nova | 04:33 | |
| *** ratailor has joined #openstack-nova | 04:34 | |
| *** markvoelker has joined #openstack-nova | 04:40 | |
| *** markvoelker has quit IRC | 04:45 | |
| *** ash2307 has left #openstack-nova | 04:53 | |
| *** jaosorior has joined #openstack-nova | 05:01 | |
| *** ricolin has quit IRC | 05:02 | |
| *** ricolin has joined #openstack-nova | 05:03 | |
| *** janki has joined #openstack-nova | 05:08 | |
| *** dave-mccowan has quit IRC | 05:16 | |
| *** adriant has joined #openstack-nova | 05:23 | |
| *** tetsuro has joined #openstack-nova | 05:27 | |
| *** tetsuro has quit IRC | 05:27 | |
| openstackgerrit | zhufl proposed openstack/nova master: [Trivial]Remove unused helper get_vif_devname_with_prefix https://review.opendev.org/678136 | 05:40 |
| *** udesale has quit IRC | 05:55 | |
| *** udesale has joined #openstack-nova | 05:55 | |
| *** boxiang has joined #openstack-nova | 05:56 | |
| *** boxiang has quit IRC | 05:57 | |
| *** boxiang has joined #openstack-nova | 05:57 | |
| *** zhubx has quit IRC | 05:58 | |
| *** ociuhandu has joined #openstack-nova | 06:03 | |
| *** udesale has quit IRC | 06:06 | |
| *** ociuhandu has quit IRC | 06:07 | |
| openstackgerrit | Yongli He proposed openstack/nova master: Add server sub-resource topology API https://review.opendev.org/621476 | 06:16 |
| openstackgerrit | Yongli He proposed openstack/nova master: Add server sub-resource topology API https://review.opendev.org/621476 | 06:19 |
| *** markvoelker has joined #openstack-nova | 06:40 | |
| *** markvoelker has quit IRC | 06:45 | |
| *** ash2307 has joined #openstack-nova | 06:47 | |
| *** shilpasd has joined #openstack-nova | 06:56 | |
| *** jawad_axd has joined #openstack-nova | 07:07 | |
| *** takamatsu has joined #openstack-nova | 07:08 | |
| *** ricolin has quit IRC | 07:15 | |
| *** rcernin has quit IRC | 07:15 | |
| *** trident has quit IRC | 07:25 | |
| *** lpetrut has joined #openstack-nova | 07:29 | |
| *** trident has joined #openstack-nova | 07:31 | |
| *** sapd1_ has joined #openstack-nova | 07:33 | |
| *** N3l1x has quit IRC | 07:34 | |
| *** ash2307 has left #openstack-nova | 07:36 | |
| *** xek has joined #openstack-nova | 07:37 | |
| *** sapd1 has quit IRC | 07:37 | |
| *** AJaeger has joined #openstack-nova | 07:40 | |
| AJaeger | nova stable cores, could you review https://review.opendev.org/677178, https://review.opendev.org/677176 https://review.opendev.org/677177 ? Those remove experimental openSUSE 42.3 jobs since the distro is end-of-life now. | 07:41 |
| lyarwood | AJaeger: ack, I'll take a look now | 07:43 |
| *** ociuhandu has joined #openstack-nova | 07:44 | |
| lyarwood | AJaeger: nothing on stable/stein? | 07:45 |
| AJaeger | lyarwood: no, that one uses the newer opensuse-150 job already | 07:45 |
| AJaeger | thanks, lyarwood | 07:45 |
| *** ociuhandu has quit IRC | 07:46 | |
| lyarwood | Ib0b49fead613d06001df348831c022edfe8d0fd1 - got it, thanks | 07:46 |
| *** ralonsoh has joined #openstack-nova | 07:57 | |
| *** takashin has quit IRC | 08:06 | |
| *** dtantsur|afk is now known as dtantsur | 08:10 | |
| *** rpittau|afk is now known as rpittau | 08:14 | |
| *** jangutter has quit IRC | 08:15 | |
| *** tkajinam has quit IRC | 08:19 | |
| *** ociuhandu has joined #openstack-nova | 08:20 | |
| *** ociuhandu has quit IRC | 08:25 | |
| *** ivve has joined #openstack-nova | 08:26 | |
| *** lpetrut has quit IRC | 08:31 | |
| *** shilpasd has quit IRC | 08:33 | |
| *** jangutter has joined #openstack-nova | 08:33 | |
| *** mdbooth has joined #openstack-nova | 08:35 | |
| *** mkrai has quit IRC | 08:39 | |
| *** mkrai has joined #openstack-nova | 08:43 | |
| openstackgerrit | Yongli He proposed openstack/nova master: Add server sub-resource topology API https://review.opendev.org/621476 | 08:47 |
| *** sapd1_x has quit IRC | 08:47 | |
| *** janki has quit IRC | 08:48 | |
| cervigni | hello | 08:51 |
| cervigni | I am playing around with vgpu | 08:51 |
| *** mdbooth has quit IRC | 08:52 | |
| cervigni | It seems that the nova scheduler does not assign properly the cpu-> gpu | 08:52 |
| cervigni | essentially the first vgpu that is always created by the scheduler, goes to my 2nd gpu | 08:52 |
| cervigni | while the cores are assigned to first cpu (first numa node) | 08:52 |
| cervigni | is there a way to tweak the scheduling algorithm or to specify somehow that there is an affinity between CPU and VGPUs ? | 08:53 |
| cervigni | as it stands, instances always get first cpus / second gpu | 08:53 |
| cervigni | therefore there is a big numa issue and performance penality where instances are communicating with the "opposite" numa node in the other cpu socket | 08:54 |
| *** mdbooth has joined #openstack-nova | 08:54 | |
| cervigni | it seems to me a real nova scheduling issue, but I would like someone to confirm that is the case before opening a bug report | 08:56 |
| openstackgerrit | Mitsuhiro Tanino proposed openstack/nova master: Smaller project quota value than user quota value can't be set https://review.opendev.org/678183 | 08:56 |
| *** shilpasd has joined #openstack-nova | 08:57 | |
| cervigni | GPU 0 has a pci id lower than GPU 1, but for reasons I don't understand, the VGPUs are always first assigned to GPU1 | 08:57 |
| cervigni | |===============================+======================+======================| | 08:57 |
| cervigni | | 0 Tesla V100-PCIE... On | 00000000:37:00.0 Off | Off | | 08:57 |
| cervigni | | N/A 38C P0 27W / 250W | 39MiB / 16383MiB | 0% Default | | 08:57 |
| cervigni | +-------------------------------+----------------------+----------------------+ | 08:57 |
| cervigni | | 1 Tesla V100-PCIE... On | 00000000:86:00.0 Off | Off | | 08:57 |
| cervigni | | N/A 35C P0 29W / 250W | 51MiB / 16383MiB | 0% Default | | 08:57 |
| cervigni | +-------------------------------+----------------------+----------------------+ | 08:57 |
| *** mdbooth has quit IRC | 08:57 | |
| *** mdbooth has joined #openstack-nova | 08:58 | |
| openstackgerrit | Mitsuhiro Tanino proposed openstack/nova master: Smaller project quota value than user quota value can't be set https://review.opendev.org/678183 | 08:59 |
| *** mdbooth has quit IRC | 08:59 | |
| *** mdbooth has joined #openstack-nova | 09:00 | |
| openstackgerrit | Mitsuhiro Tanino proposed openstack/nova master: Smaller project quota than user quota shouldn't be set https://review.opendev.org/678183 | 09:05 |
| *** owalsh is now known as owalsh|away | 09:16 | |
| *** cervigni has quit IRC | 09:27 | |
| *** ociuhandu has joined #openstack-nova | 09:30 | |
| *** ociuhandu has quit IRC | 09:34 | |
| *** brinzhang_ has quit IRC | 09:46 | |
| *** ociuhandu has joined #openstack-nova | 09:47 | |
| openstackgerrit | Bhagyashri Shewale proposed openstack/nova master: Ignore root_gb for BFV in simple tenant usage API https://review.opendev.org/612626 | 09:53 |
| *** psachin has quit IRC | 10:04 | |
| *** psachin has joined #openstack-nova | 10:06 | |
| *** maciejjozefczyk has joined #openstack-nova | 10:06 | |
| *** maciejjozefczyk has quit IRC | 10:07 | |
| *** ociuhandu has quit IRC | 10:10 | |
| *** jaosorior has quit IRC | 10:10 | |
| *** roukoswarf has quit IRC | 10:24 | |
| *** rouk has joined #openstack-nova | 10:24 | |
| *** mrjk has quit IRC | 10:27 | |
| *** jaosorior has joined #openstack-nova | 10:27 | |
| *** bhagyashris has quit IRC | 10:30 | |
| *** ociuhandu has joined #openstack-nova | 10:43 | |
| *** mkrai_ has joined #openstack-nova | 10:44 | |
| *** mkrai has quit IRC | 10:47 | |
| *** stakeda has quit IRC | 10:47 | |
| *** mkrai__ has joined #openstack-nova | 10:48 | |
| stephenfin | alex_xu: You should probably look at https://review.opendev.org/#/c/674894/ given your comments at https://review.opendev.org/#/c/664420/42/nova/scheduler/request_filter.py@206 | 10:48 |
| *** mkrai_ has quit IRC | 10:52 | |
| *** mkrai__ has quit IRC | 10:55 | |
| *** lpetrut has joined #openstack-nova | 10:56 | |
| *** zhubx has joined #openstack-nova | 10:59 | |
| *** zhubx has quit IRC | 11:01 | |
| *** zhubx has joined #openstack-nova | 11:02 | |
| *** boxiang has quit IRC | 11:03 | |
| *** tesseract has joined #openstack-nova | 11:12 | |
| *** bauzas has quit IRC | 11:18 | |
| *** bauzas has joined #openstack-nova | 11:18 | |
| efried | sean-k-mooney: I agree. I skimmed over your review last night, sounds like what I was expecting. | 11:18 |
| sean-k-mooney | efried: i might try and submit a basic patch to show how to do it as i suggested instead | 11:19 |
| sean-k-mooney | efried: i assume we are talking about cyborg privsep | 11:19 |
| efried | yes | 11:19 |
| sean-k-mooney | i think i was a little harsh in the review but i also have strong feeling about this topic. | 11:20 |
| *** jaosorior has quit IRC | 11:26 | |
| efried | Swhy I was hoping you would look at it. | 11:26 |
| *** tbachman has quit IRC | 11:35 | |
| *** hemna has joined #openstack-nova | 11:40 | |
| *** jaosorior has joined #openstack-nova | 11:43 | |
| sean-k-mooney | efried: well i review in cyborg octionally and is mostly negitve when i do. not alway but my +/- raition there is proably not that positive. so i do try to make sure my review that i do there are at least constructive rather then saying "no this is all terible" and suggest how it can be improved. but if i do my usuall 50 billion comments on each indiviaual issue that can come across badly if in a | 11:47 |
| sean-k-mooney | "drive by" review. | 11:47 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Use SDK for setting instance id https://review.opendev.org/659690 | 11:53 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Use SDK for add/remove instance info from node https://review.opendev.org/659691 | 11:53 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Use SDK for getting network metadata from node https://review.opendev.org/670213 | 11:53 |
| efried | sean-k-mooney: Any update on that libvirt attribute ordering bug? Not sure if I missed something going past... | 11:54 |
| *** markvoelker has joined #openstack-nova | 11:57 | |
| sean-k-mooney | efried: its still on my todo list. i was dealing with a downstream backport the last day or two but i should get to it today or monday. | 11:58 |
| efried | cool, thanks. It's a "requirements freeze" thing fwiw. | 11:58 |
| sean-k-mooney | oh ya when is that next week? | 11:59 |
| sean-k-mooney | i was aware of that but i proably should expidite looking at it | 11:59 |
| sean-k-mooney | non-client lib frezee shoudl m3 -2 weeks | 11:59 |
| efried | oh, I thought it was, but no, it's the week of Sept 9 | 11:59 |
| sean-k-mooney | and oslo shoudl be m3 -3 | 12:00 |
| sean-k-mooney | i guess requirement freeze is m3-1 | 12:00 |
| sean-k-mooney | actully it looks like its m3 | 12:01 |
| sean-k-mooney | so september 12th | 12:01 |
| sean-k-mooney | but ill try and repoduce anyway this/next week and get a patch up so we dont run up against it | 12:01 |
| sean-k-mooney | is this a shorter release by the way. it feels a little compressed. the freeze dates are all 1 week later then i expect in my head | 12:03 |
| sean-k-mooney | maybe im just used to makeing sure everythin is in os-vif on time and i naturally try to have the last week to make sure we dont have any bugs before the freeze | 12:04 |
| *** dave-mccowan has joined #openstack-nova | 12:06 | |
| *** larainema has quit IRC | 12:09 | |
| *** tbachman has joined #openstack-nova | 12:35 | |
| *** tbachman has quit IRC | 12:38 | |
| *** davee_ has joined #openstack-nova | 12:40 | |
| *** tbachman has joined #openstack-nova | 12:40 | |
| *** spatel has joined #openstack-nova | 12:43 | |
| *** ociuhandu has quit IRC | 12:44 | |
| *** dougsz has joined #openstack-nova | 12:47 | |
| *** udesale has joined #openstack-nova | 12:48 | |
| *** spatel has quit IRC | 12:48 | |
| dougsz | nova-manage db online_data_migrations <- Do I need to run that per cell, or is that a top level only thing? Ref: https://docs.openstack.org/nova/latest/user/upgrade.html | 12:49 |
| AJaeger | nova stable cores, could I get a second +2 on https://review.opendev.org/677178, https://review.opendev.org/677176 https://review.opendev.org/677177, please ? Those remove experimental openSUSE 42.3 jobs since the distro is end-of-life now. | 12:51 |
| *** Bidwe_jay has joined #openstack-nova | 12:52 | |
| sean-k-mooney | AJaeger: do we have a replacement suse job either based on tumbelweed or a more recent leap release | 12:54 |
| sean-k-mooney | AJaeger: or rather do we have opensuse node sets so a replacment job could be built if needed | 12:55 |
| AJaeger | sean-k-mooney: yes, in master and stein already | 12:57 |
| *** gbarros has joined #openstack-nova | 12:57 | |
| sean-k-mooney | cool | 12:57 |
| AJaeger | but I didn't want to add new distro support to old stable branches that only have an experimental job | 12:57 |
| *** aarents has joined #openstack-nova | 12:57 | |
| sean-k-mooney | ya makes sense | 12:58 |
| AJaeger | sean-k-mooney: there's job tempest-full-py3-opensuse150: | 12:58 |
| AJaeger | (still experimental) in master and stein | 12:58 |
| sean-k-mooney | im not sure experimental jobs add much value on stable as they will be used even less often then master. but its still better then nothing | 12:59 |
| *** roukoswarf has joined #openstack-nova | 12:59 | |
| *** rouk has quit IRC | 13:00 | |
| sean-k-mooney | or in other words removing an experimal job form stable that is on an unsupproted os/is broken is pretty safe | 13:01 |
| AJaeger | agreed | 13:02 |
| *** ratailor has quit IRC | 13:04 | |
| *** eharney has joined #openstack-nova | 13:12 | |
| *** jaosorior has quit IRC | 13:14 | |
| *** spatel has joined #openstack-nova | 13:14 | |
| *** ociuhandu has joined #openstack-nova | 13:14 | |
| *** spatel has quit IRC | 13:17 | |
| *** ociuhandu has quit IRC | 13:19 | |
| *** ivve has quit IRC | 13:24 | |
| *** BjoernT has joined #openstack-nova | 13:27 | |
| *** zhubx has quit IRC | 13:31 | |
| *** zhubx has joined #openstack-nova | 13:31 | |
| *** bnemec has joined #openstack-nova | 13:34 | |
| *** bnemec is now known as beekneemech | 13:35 | |
| *** mriedem has joined #openstack-nova | 13:37 | |
| *** artom has joined #openstack-nova | 13:40 | |
| *** ociuhandu has joined #openstack-nova | 13:41 | |
| *** psachin has quit IRC | 13:47 | |
| *** jawad_axd has quit IRC | 13:49 | |
| *** jawad_axd has joined #openstack-nova | 13:53 | |
| *** ociuhandu has quit IRC | 13:54 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Do not log token in the novncproxy logs https://review.opendev.org/678234 | 13:57 |
| *** jawad_axd has quit IRC | 13:58 | |
| *** ociuhandu has joined #openstack-nova | 14:00 | |
| *** tbachman_ has joined #openstack-nova | 14:01 | |
| *** tbachman has quit IRC | 14:05 | |
| *** tbachman_ is now known as tbachman | 14:05 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: Update SDK fixture for openstacksdk 0.35.0 https://review.opendev.org/678237 | 14:05 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Process [compute] in $NOVA_CPU_CONF in nova-next https://review.opendev.org/672800 | 14:11 |
| alex_xu | stephenfin: nice | 14:11 |
| alex_xu | mriedem: I replied your email, let me know what can make you have more confidence on this | 14:13 |
| dansmith | artom: you've got a bunch of legit unit test fails | 14:15 |
| artom | dansmith, yeah, those have been fixed locally | 14:16 |
| dansmith | artom: I'm also concerned about the validity of some of the other tests that aren't failing even though you're using the wrong dst_numa_config in some of the code and/or tests | 14:16 |
| dansmith | artom: okay cool, I just lightly dusted the bottom several patches with more comments | 14:16 |
| artom | dansmith, I think that's because I was asserting obj.dst_numa_config and setting it as well | 14:16 |
| dansmith | up to the one you said was the top of the things you've updated | 14:16 |
| artom | So the code sets dst_numa_config, and the test checks it as well | 14:17 |
| dansmith | artom: okay, I guess just setting a non-field property on the object, okay | 14:17 |
| artom | dansmith, yeah | 14:17 |
| dansmith | well, let's make sure to get all those right :D | 14:17 |
| artom | dansmith, so you're saying I need to squash the RPC changes into the top patch? | 14:17 |
| dansmith | I guess no real testing was done with the updated set since the top weren't updated anyway | 14:17 |
| artom | I split them to keep patch size as small as possible | 14:17 |
| artom | But looks like it'll be unavoidable here :( | 14:18 |
| artom | dansmith, yeah, grep took care of that | 14:18 |
| dansmith | artom: squash the rpc one with the top one because of the version/capability advertisement? | 14:18 |
| artom | dansmith, yeah | 14:18 |
| dansmith | that's probably too obtuse of a solution.. I mean it'll work I guess, but hopefully we can split them a little more than that | 14:19 |
| dansmith | flipping their order is probably better | 14:19 |
| artom | Ohhh | 14:20 |
| dansmith | get the code in place, just don't allow it to be called from RPC until the second one, where you actually bump the versions and add the params | 14:20 |
| dansmith | and remove the extra service version bump of course | 14:20 |
| dansmith | amirite? | 14:20 |
| *** dpawlik has quit IRC | 14:20 | |
| artom | Makes sense, though I need to think through how that works for params | 14:20 |
| dansmith | it'll require some tweaking of things I'm sure, but that general ordering should work for you I think | 14:21 |
| dansmith | artom: I also think you probably need to maintain a -W on the bottom patch, if you're going to have it in various states of not-really-working (or I can -2 it) | 14:21 |
| artom | It's that sort of expected as the series goes through review? | 14:21 |
| dansmith | artom: since it's going to mostly require a manual verification (which right now would fail hard), we probably want to wait to land much of it until it's actually proven to be working | 14:22 |
| artom | But I can do that, sure | 14:22 |
| artom | Ah, in that sense | 14:22 |
| artom | Yeah, makes sense | 14:22 |
| dansmith | just for someone not really paying attention, | 14:22 |
| dansmith | because your bottom few patches look devilishly small | 14:22 |
| artom | I'm devious like that :D | 14:23 |
| artom | Actually the func test on top is only failing because I explicitly added an assertFalse because I wanted to see the logging | 14:23 |
| dansmith | the claim one wouldn't be terrible to land in isolation, but the next one is an object schema commitment, which is still not ready to land, so... | 14:23 |
| artom | I can remove that so we can get at least some confidence in the gate | 14:24 |
| dansmith | okay I'm not sure how that can be the case with you using the wrong field names in the object, | 14:24 |
| dansmith | unless it's really that fake | 14:24 |
| artom | dansmith, well ok, it *was* failing because of the assertFalse | 14:24 |
| dansmith | oh okay I see | 14:24 |
| artom | *now* it might be broken for real :D | 14:24 |
| dansmith | I hope it is :) | 14:24 |
| dansmith | otherwise it doesn't do much for confidence :) | 14:24 |
| artom | dansmith, yeah: "AttributeError: 'module' object has no attribute 'NUMAHostInfo'" | 14:25 |
| efried | artom: qq: Est-ce que c'est vrai que, si je dis, "on ne permet pas l'un ou l'autre," même si tu connais ce que je veux dire, ce n'est pas strictement correcte? Il faudrait dire, "on ne permet pas ni l'un ni l'autre"? | 14:27 |
| artom | efried, on ne permet ni l'un ni l'autre | 14:27 |
| efried | sorry, yah | 14:27 |
| artom | Ou même "aucun des deux n'est permis" | 14:28 |
| dansmith | artom: uh, that's not the one I was worried about, so maybe there's another problem? | 14:28 |
| artom | dansmith, it probably just hit that one first | 14:28 |
| dansmith | point is, I didn't catch that one | 14:28 |
| *** mlavalle has joined #openstack-nova | 14:29 | |
| artom | dansmith, oh, wait, that's from an unrelated stephenfin commit | 14:29 |
| artom | dansmith, tell you want, before addressing any other feedback, I'll un-WIP the func test | 14:29 |
| artom | *what | 14:30 |
| dansmith | un-WIP and fix? | 14:30 |
| artom | Yeah, fix was implied :) | 14:30 |
| dansmith | juuuuust checking :) | 14:30 |
| artom | Haha, there are limits to my lazyness ;) | 14:30 |
| dansmith | well, you can see my level of confidence in your laziness I guess | 14:30 |
| artom | ;_; | 14:31 |
| dansmith | efried: are you a candidate reviewer for this numa live migration stuff? seems like it's probably somewhat in your wheelhouse and sphere of interest | 14:32 |
| dougsz | In cells v2, do you run online_data_migrations only when *all* cells are upgraded? | 14:32 |
| dansmith | dougsz: you can run them against individual cells as they're upgraded or afterwards against all of them | 14:32 |
| *** davee_ has quit IRC | 14:33 | |
| AJaeger | efried, so are fine with https://review.opendev.org/677178, https://review.opendev.org/677176 and https://review.opendev.org/677177, please ? | 14:33 |
| efried | dansmith: sphere of interest, as dictated by my employer, yes. Wheelhouse, not in the slightest. I've been hoping people who actually know wtf they're looking at will do the dirty work here. | 14:34 |
| efried | AJaeger: I'm not stable. dansmith might be able to push those for you, though. | 14:35 |
| dansmith | efried: um, it's numa and little bit of rpc | 14:35 |
| dansmith | efried: but I'm trying to make sure it's clean and then will have to look for another reviewer to +W and figured that might be you | 14:35 |
| dansmith | efried: it's really not very complicated.. much less so than I originally thought when discussing the meta details | 14:35 |
| dougsz | dansmith: thanks - so to upgrade one cell, i run online_data_migrations in the cell, and then again at the top level? | 14:35 |
| dansmith | dougsz: "run again at the top level" meaning "run again against all cells once they're all updated" ? | 14:36 |
| efried | dansmith: I will dig in if I need to, but I'm still way low on the learning curve of both numa and rpc. | 14:36 |
| mriedem | alex_xu: ack, and replied | 14:36 |
| dansmith | efried: opportunity? :) | 14:36 |
| mriedem | alex_xu: tl;dr do whatever you want | 14:37 |
| efried | yeah | 14:37 |
| dansmith | heh | 14:37 |
| alex_xu | I want to sleep now :) | 14:38 |
| artom | efried, FWIW, the crux of it is converting parts of https://libvirt.org/formatdomain.html#elementsNUMATuning and https://libvirt.org/formatdomain.html#elementsCPUTuning to objects and back | 14:39 |
| artom | So it's not touching any of the nasty code in nova/virt/hardware.py | 14:39 |
| artom | Or just get alex_xu to do it, if he's more up to speed on the libvirt'y things ;) | 14:40 |
| mriedem | dansmith: i'm likely your other core for numa live migration | 14:40 |
| AJaeger | efried: ah, thanks | 14:40 |
| dansmith | mriedem: I didn't think you'd want to | 14:41 |
| dansmith | mriedem: and/or I didn't want to know what pound of flesh you'd want in return :D | 14:41 |
| mriedem | i've gone through it before...in rocky probably at tihs point | 14:41 |
| dansmith | yeah, I know | 14:41 |
| mriedem | and i'm more familiar with the live migration and rt claims flows that many | 14:41 |
| mriedem | *than | 14:41 |
| dansmith | I know, but .. see above :) | 14:41 |
| mriedem | however, i also have confidence issues with it | 14:42 |
| mriedem | so i've been waiting for another core to dig in first, root out the obvious stuff, and someone (intel/windriver) to come along and say they've actually tested it in a lab | 14:42 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Mask the token used to allow access to consoles https://review.opendev.org/220622 | 14:42 |
| dansmith | well, I'm doing the former | 14:43 |
| mriedem | yeah, | 14:43 |
| dansmith | and I think it's just going to be artom's hand testing | 14:43 |
| mriedem | and i think as of last friday sean-k-mooney said in his free time at home when he shouldn't be on his own personal hardware was trying to do the latter | 14:43 |
| dansmith | at least initially | 14:43 |
| dansmith | or that | 14:43 |
| artom | mriedem, Dean told me he's brought it up the chain to try and get resources to test it, but not sure that's going to pan out | 14:43 |
| mriedem | i know intel is strapped for cash and resources these days | 14:43 |
| mriedem | it's sad | 14:43 |
| mriedem | sad folks | 14:44 |
| alex_xu | mriedem: thanks anyway, sorry for pushing hard | 14:44 |
| sean-k-mooney | reads back | 14:44 |
| dansmith | it's not going to happen in time to land this either I'm sure | 14:44 |
| dansmith | lol | 14:44 |
| dougsz | dansmith: maybe I'm confused. If I run `nova-manage db online_data_migrations` using the cell conductor nova.conf, I update only the cell, if I run `nova-manage db online_data_migrations` using the top level nova.conf, I upgrade all cells? | 14:44 |
| dansmith | dougsz: no, hence my question ;) | 14:44 |
| artom | FWIW, I do have that whitebox tempest plugin code | 14:44 |
| artom | It's nowhere upstream yet, but that's next on my list | 14:44 |
| dansmith | dougsz: the former per cell is how it works, which is why I asked why you'd run again "at the top level" | 14:44 |
| alex_xu | artom: emm..not sure what is it yet | 14:45 |
| artom | Move whitebox under openstack-qa, find hardware on which to run it. Fort Nebula is a good candidate | 14:45 |
| sean-k-mooney | i do plan to test artoms stuff next week. i need to resintall the os on those servers. but i have already set up one of them | 14:45 |
| AJaeger | dansmith: could you put the following three on your review list, please? https://review.opendev.org/677178, https://review.opendev.org/677176 and https://review.opendev.org/677177 all remove an experimental opensuse job for an eol distro from stable branches | 14:45 |
| mriedem | dougsz: iow, you have to run that command per cell | 14:45 |
| *** jawad_axd has joined #openstack-nova | 14:46 | |
| dansmith | mriedem: he said he used the cell conductor config, which is the per-cell bit, so I think he gets that | 14:46 |
| mriedem | ok | 14:46 |
| dansmith | I guess we do have some migrations in the api database, | 14:46 |
| dansmith | so I guess running against the api config is also necessary, I dunno why I wasn't thinking about that initially | 14:47 |
| dansmith | dougsz: ^ | 14:47 |
| artom | alex_xu, NUMA live migration, you looked at it a bit before, and there's some overlap with your vpmem work I think. dansmith and I are hunting for a second non-RH core | 14:48 |
| dougsz | dansmith, mriedem: thanks, I think that makes sense. So per cell, *and* at the top level to do it everywhere. | 14:48 |
| mriedem | "to do it everywhere" isn't a thing | 14:48 |
| mriedem | the command doesn't broadcast across all cells | 14:48 |
| dansmith | right, | 14:48 |
| dansmith | that's the part I was taking exception with | 14:48 |
| mriedem | dan is just saying there are some migrations in the API DB which is "at the top" | 14:49 |
| mriedem | "top of the topology" iow | 14:49 |
| alex_xu | artom: yea, i can onboard the review, probably need one or two day, rampup your patches | 14:49 |
| artom | alex_xu, thank you, appreciated :) | 14:49 |
| dougsz | mriedem: Ok, thanks, and I trigger those top level migrations via nova-manage api_db sync. I think I get it | 14:50 |
| alex_xu | artom: btw, I don't think it will overlap with vpmem anymore, but will check again | 14:50 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Update SDK fixture for openstacksdk 0.35.0 https://review.opendev.org/678237 | 14:50 |
| efried | mriedem: more sdk turducken :( ^ | 14:50 |
| mriedem | dougsz: online_data_migrations is not the same as the sync command | 14:50 |
| mriedem | sync == upgrade db table schemas | 14:51 |
| *** jawad_axd has quit IRC | 14:51 | |
| mriedem | online_data_migrations == migrate existing data in the table to some new format if necessary | 14:51 |
| dougsz | got it | 14:51 |
| *** shilpasd has quit IRC | 14:53 | |
| dougsz | final question, If I upgrade a single cell, to get it to move to latest RPC version, I need to SIG-HUP the top level Nova services as well? | 14:53 |
| *** lpetrut has quit IRC | 14:54 | |
| mriedem | AJaeger: i got them | 14:54 |
| efried | SIGHUP is still broken, until https://review.opendev.org/#/c/641907/ | 14:54 |
| AJaeger | thanks, mriedem ! | 14:55 |
| *** ociuhandu has quit IRC | 14:55 | |
| dougsz | yeah, just thinking, I upgrade one cell, I need to restart all top level services because SIG-HUP doesn't work | 14:55 |
| mriedem | dougsz: well, you should be pinning the rpc upgrade level for the computes at the top level so when you upgrade your computes in the cell, the top level (api/conductor) isn't sending messages the computes won't understand | 14:55 |
| mriedem | dougsz: are you updating the control plane services first or last? | 14:56 |
| dougsz | control plane first | 14:56 |
| efried | stephenfin: I'm waiting for a new push of the cpu-resources series, right? | 14:56 |
| mriedem | https://docs.openstack.org/nova/latest/user/upgrade.html | 14:56 |
| stephenfin | yup | 14:56 |
| mriedem | dougsz: https://docs.openstack.org/nova/latest/user/upgrade.html | 14:56 |
| stephenfin | working on it as we speak | 14:56 |
| dougsz | yeah, trying to follow that in a cells context | 14:56 |
| efried | stephenfin: I'm hoping that if I read the code once a day for a week, I'll understand it. | 14:56 |
| efried | Currently it's you making changes I don't understand to code I don't understand. | 14:57 |
| mriedem | dougsz: cellsv2 yeah? | 14:57 |
| dougsz | yeah | 14:57 |
| mriedem | dougsz: what services are running in the cells? just nova-compute? | 14:57 |
| *** xek has quit IRC | 14:58 | |
| dougsz | + nova-ssh, nova-conductor | 14:58 |
| mriedem | idk what nova-ssh is | 14:58 |
| mriedem | anyway, if you upgrade the control plane first, you want to make sure those are configured for [upgrade_levels]/compute (either auto or whatever your n-1 release you're upgrading from is) | 14:59 |
| mriedem | that way the controller services won't send N version messages to N-1 services | 14:59 |
| mriedem | i.e. pin the rpc versions to the lowest common version, roll through upgrades, and then unpin | 14:59 |
| dougsz | ok, thanks mriedem, that bit makes sense. I will go and have a play | 15:00 |
| *** dpawlik has joined #openstack-nova | 15:00 | |
| *** AJaeger has left #openstack-nova | 15:00 | |
| *** tbachman has quit IRC | 15:05 | |
| *** jawad_axd has joined #openstack-nova | 15:11 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: allow getting resource request of every bound ports of an instance https://review.opendev.org/655110 | 15:12 |
| *** tbachman has joined #openstack-nova | 15:12 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pass network API to the conducor's MigrationTask https://review.opendev.org/655111 | 15:14 |
| *** jawad_axd has quit IRC | 15:16 | |
| dansmith | efried: sighup still does the thing he needs, just inefficiently, IIRC | 15:16 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add request_spec to server move RPC calls https://review.opendev.org/655721 | 15:16 |
| dansmith | efried: it effectively restarts the whole process, which is good enough, just too heavy | 15:16 |
| efried | dansmith: I thought we decided SIGHUP also breaks privsep | 15:17 |
| dansmith | oh does it? | 15:17 |
| efried | something about either restarting or not restarting the privsep daemon... | 15:18 |
| dougsz | https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova/handlers/main.yml#L211 | 15:18 |
| efried | now I can't remember where I saw it. | 15:18 |
| dougsz | dansmith efried ^ That's the issue afaik | 15:18 |
| dansmith | not restarting is fine, as long as it doesn't lose a reference to the pipe, which I thought was global | 15:18 |
| *** damien_r has quit IRC | 15:18 | |
| *** tbachman has quit IRC | 15:18 | |
| dansmith | dougsz: oh, right, right, ignore me | 15:19 |
| dansmith | I forgot the rabbit hole went deeper | 15:19 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: re-calculate provider mapping during migration https://review.opendev.org/655112 | 15:19 |
| dougsz | np! | 15:19 |
| dansmith | the simple case is that it does a full restart for no reason, but I forgot it also breaks some state as a result | 15:19 |
| *** dave-mccowan has quit IRC | 15:20 | |
| efried | someone stephenfin with oslo-core stephenfin should merge that patch | 15:20 |
| stephenfin | whuh? | 15:21 |
| efried | https://review.opendev.org/#/c/641907/ | 15:21 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: update allocation in binding profile during migrate https://review.opendev.org/656422 | 15:21 |
| stephenfin | I'm going to finish this cpu-resources reshuffle, but I can take a look after that | 15:22 |
| efried | cool. No hurry, clearly. It's been open since like rocky. | 15:22 |
| efried | sure would be nice to stop having to tell people SIGHUP is broken tho. | 15:22 |
| beekneemech | If you figure it out you'll be ahead of the rest of us. :-) | 15:22 |
| efried | (oh, did I trigger a keyword?) | 15:23 |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: doc: remove confusing docs about aggregate allocation ratios https://review.opendev.org/678254 | 15:23 |
| beekneemech | Yeah, I added an oslo highlight after the Denver PTL discussion. | 15:23 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Extend NeutronFixture to handle migrations https://review.opendev.org/655114 | 15:24 |
| dansmith | beekneemech: efried: I think it might be worth testing that with a real nova | 15:24 |
| dansmith | before just firing it in | 15:24 |
| dansmith | has anyone done that? | 15:24 |
| mriedem | cpu-resources reshuffle is the worst dance move yet | 15:24 |
| efried | dansmith: I ripped it in a devstack, does that count? | 15:24 |
| dansmith | efried: does that mean "tested it" ? | 15:24 |
| mriedem | it means he farted i think | 15:25 |
| efried | I didn't try to hit the "stuff that actually breaks" paths cause I didn't understand them well enough | 15:25 |
| dansmith | then...no | 15:25 |
| efried | but I watched logs through SIGHUP before and after | 15:25 |
| efried | that analysis is in the patch | 15:25 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: prepare func test env for moving servers with bandwidth https://review.opendev.org/655109 | 15:26 |
| dansmith | you should be able to validate that before the patch SIGHUP breaks booting instances, and after it doesn't right? | 15:26 |
| *** macz has joined #openstack-nova | 15:26 | |
| efried | dansmith: I'm updating my devstack now, then will try that. | 15:27 |
| mriedem | due to not processing the vif plug event right? you might be waiting 5 minutes if you don't tweak the config | 15:28 |
| mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.vif_plugging_timeout | 15:28 |
| mriedem | i'd drop that to 60 seconds | 15:28 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Add emulated TPM support to Nova https://review.opendev.org/631363 | 15:28 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Add support for resize and cold migration of emulated TPM files https://review.opendev.org/639934 | 15:28 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Func test for migrate server with ports having resource request https://review.opendev.org/655113 | 15:28 |
| *** CeeMac has joined #openstack-nova | 15:28 | |
| CeeMac | afternoon | 15:28 |
| CeeMac | is there a way to configure nova to use cinder volumes for ephemeral storage and root disks? | 15:29 |
| mriedem | nope | 15:29 |
| CeeMac | that answers that then | 15:29 |
| mriedem | it's come up many a time | 15:29 |
| CeeMac | so, the only option for shared storage backend is to mount the instance disk path to NFS? | 15:30 |
| mriedem | you're talking about different things i think | 15:30 |
| CeeMac | its entirely possible | 15:30 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Make _rever_allocation nested allocation aware https://review.opendev.org/676138 | 15:31 |
| mriedem | the instance path files are not the same as the root disk - you can have the root disks in rbd but have the instance files (console log, config drive, etc) be on local disk on the compute node | 15:31 |
| mriedem | it gets confusing | 15:31 |
| CeeMac | i don't use ceph, if that helps | 15:31 |
| dansmith | artom: so, what do you think the timetable is for a respin of the numalm set? you said you've got unit tests fixed locally already, so can you do some swizzling of the top two patches and push that up today to get a run started? | 15:31 |
| mriedem | or you could have the instance path files on local disk or shared NFS/rbd, but the root disk is actually a volume in cinder | 15:31 |
| CeeMac | i was thinking in terms of when an instance is launched without specifying to use a cinder volume | 15:32 |
| mriedem | CeeMac: sure, just saying that's a common shared storage backend | 15:32 |
| artom | dansmith, working on updating the func test currently, since confidence seems to be the big thing | 15:32 |
| CeeMac | mriedem: got you, yeah its all very confusing. | 15:32 |
| *** rpittau is now known as rpittau|afk | 15:32 | |
| mriedem | CeeMac: assuming you're using libvirt, the image backend for the root disk is defined with this config https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.images_type | 15:33 |
| mriedem | so you're probably using raw/flat/qcow2 yeah? | 15:33 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support reverting migration / resize with bandwidth https://review.opendev.org/676140 | 15:33 |
| CeeMac | qcow2 | 15:33 |
| CeeMac | but i back images off onto volume, so it converts it raw iirc | 15:33 |
| mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.instances_path controls where the instance files are going to be then | 15:34 |
| CeeMac | right | 15:35 |
| CeeMac | thats what I was trying to get at i think | 15:35 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Func test for migrate re-schedule with bandwidth https://review.opendev.org/676972 | 15:35 |
| CeeMac | when an instance is launched without specifying to use volume backing for root disk, the root disk will go there too by default? | 15:36 |
| CeeMac | i.e. not through horizon | 15:36 |
| mriedem | horizon doesn't have anything to do with it (unless you mean horizon is somehow forcing you to use volume-backed servers) | 15:36 |
| mriedem | but yeah | 15:36 |
| mriedem | https://review.opendev.org/#/c/511965/ has some background details on the whole "configure nova to do volume-backed servers by default" thing which has had lots of discussion in the past like i said | 15:37 |
| mriedem | many attempts, many failures | 15:37 |
| sean-k-mooney | horizon default to createing a volume if cinder is enabled | 15:37 |
| mriedem | sean-k-mooney: ok but it's not required | 15:37 |
| mriedem | nor does everyone use horizon | 15:37 |
| sean-k-mooney | im not really sure why but it does. its not require correct | 15:37 |
| CeeMac | sean-k-mooney: thats my understanding too | 15:38 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support migrating SRIOV port with bandwidth https://review.opendev.org/676980 | 15:38 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Allow migrating server with port resource request https://review.opendev.org/671497 | 15:38 |
| sean-k-mooney | CeeMac: it defintly does not require it as i untick it all the time | 15:38 |
| CeeMac | my issue comes when using rancher to deploy VMs, socker-volume isn't block-storage aware | 15:38 |
| CeeMac | I'm happy with horizon defaulting to volume, as thats where I'd prefer for the instance to go | 15:39 |
| CeeMac | trying to work out a way to force all instances to back off to block storage regardless of how they're requested | 15:39 |
| CeeMac | was hoping there was an integrated cinder way | 15:39 |
| sean-k-mooney | CeeMac: i kind of wish it default to delete the volume when you delete the vm too however | 15:39 |
| CeeMac | but I'm guessing I'd need to go with mounting /var/lib/nova/instances to NFS on our SAN | 15:39 |
| CeeMac | which is where instances_path points to right now | 15:40 |
| CeeMac | in nova.config | 15:40 |
| sean-k-mooney | CeeMac: i think we have a config option that will help | 15:40 |
| sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.max_local_block_devices | 15:40 |
| sean-k-mooney | if you set that to 0 you have to use cinder volumes | 15:40 |
| sean-k-mooney | not sure if tha tis what you want | 15:40 |
| CeeMac | sean-k-mooney: that is a good start to prevent local disks being created and potentially filling up the (small) local disks on the compute | 15:41 |
| sean-k-mooney | the other options open to you if you nova no/limnited local stoage and have ceph is to enable the RBD image backend | 15:41 |
| CeeMac | but I haven't found a way to instruct rancher, or docker for that matter, to use volumes for images | 15:41 |
| CeeMac | sean-k-mooney: sadly no ceph here | 15:42 |
| CeeMac | iscsi backed cinder | 15:42 |
| sean-k-mooney | ah ok | 15:42 |
| sean-k-mooney | we dont have a cinder or iscis image backend to force all instace to use cinder unfortunetly | 15:42 |
| CeeMac | is NFS the only/best/supported way of backing off the instance_path, or is iscsi an option? | 15:43 |
| CeeMac | haha, i think you answered that question then. | 15:43 |
| sean-k-mooney | i think you would be stuck with nfs | 15:43 |
| CeeMac | you could be right, at least for the short term until I can get/hack a workaround for rancher to be able to call for volume on instance launch | 15:44 |
| CeeMac | thanks for the help mriedem and sean-k-mooney appreciate it | 15:44 |
| sean-k-mooney | rancher use docker to run all service rihgt | 15:44 |
| sean-k-mooney | i have not looked at rahcner os much | 15:44 |
| CeeMac | yeah | 15:45 |
| sean-k-mooney | but i think you can confugre docer to use remote sotorage | 15:45 |
| CeeMac | and docker-volume isn't block storage aware | 15:45 |
| CeeMac | you feed it a flavour and image effectively | 15:45 |
| sean-k-mooney | e.g. i think you should be able to have the non system docker use a volume plugin and mont /var/libvirt/instace... or what ever on a network share | 15:45 |
| *** gyee has joined #openstack-nova | 15:46 | |
| CeeMac | if rancher was running inside openstack yeah, but we're connecting to openstack through api to a project from outside | 15:46 |
| CeeMac | i'll have a ponder | 15:47 |
| sean-k-mooney | i know there used to be a tone of voluem plugins https://docs.docker.com/engine/extend/legacy_plugins/#volume-plugins but i dont know if any of thos worked with racner/still exists | 15:47 |
| CeeMac | i've seen a cinder driver for docker on github, but havent had a chance to test that yet | 15:48 |
| mriedem | sean-k-mooney: https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.max_local_block_devices is not what you think it is | 15:49 |
| mriedem | sean-k-mooney: that's only if the bdm has destination_type='local' | 15:49 |
| mriedem | unless you mean for the implicit image-defined bdm | 15:49 |
| sean-k-mooney | it prevent you spaning flavor tha trequest ephmeral/swap or root disk if you are using qcow | 15:49 |
| mriedem | source_type='image', destination_type='local' | 15:49 |
| sean-k-mooney | ya i ment you can use that to ensure only bfv guest can boot on that node | 15:50 |
| mriedem | *any* node | 15:50 |
| mriedem | it's config-driven api | 15:50 |
| sean-k-mooney | yeah thos are awsome... but ya | 15:50 |
| sean-k-mooney | i was looking to do that in the past but since i was deploying ceph for cinder i just deploy with the rbd image backend instead | 15:51 |
| *** udesale has quit IRC | 15:57 | |
| sean-k-mooney | mriedem: im going to step away for an hour or so to have breakfast/lunch but i plan to adress your comments on https://review.opendev.org/#/c/669738/ when i get back since they have been waiting a while. | 15:57 |
| *** udesale has joined #openstack-nova | 15:58 | |
| mriedem | breakfast/lunch? | 15:58 |
| mriedem | it's 5pm | 15:58 |
| sean-k-mooney | lyarwood: mriedem if we can land https://review.opendev.org/#/c/678098/ on stable stine that would make mnaser happy | 15:59 |
| sean-k-mooney | yes.. i normally eat once to twice a day. usually i try to grab something at around 3 | 15:59 |
| *** hamzy has quit IRC | 15:59 | |
| sean-k-mooney | and then around 8 | 15:59 |
| sean-k-mooney | or i jsut have one big meal around 6 and then snack on fruit/nuts/small things | 16:00 |
| *** kmalloc is now known as needscoffee | 16:00 | |
| *** needscoffee is now known as needsSoMuchMoreC | 16:00 | |
| *** needsSoMuchMoreC is now known as needscoffee | 16:00 | |
| *** dtantsur is now known as dtantsur|afk | 16:00 | |
| mriedem | i just got the image of you as the giant in jack and the beanstalk crushing human bones to make your bread | 16:02 |
| mriedem | "small things" | 16:02 |
| *** jangutter has quit IRC | 16:03 | |
| dansmith | artom: I looked at the top two patches with plans to prescribe an ordering and squash/split instructions, but I think it probably just needs to be fully squashed | 16:04 |
| artom | dansmith, yeah? I was hoping to avoid that, as it makes reviewing difficult | 16:04 |
| dansmith | I know | 16:04 |
| dansmith | I kept calling out lines as "this could be separate" and then removed those comments | 16:05 |
| artom | But you're most likely right (duh!), as a whole bunch of things need to happen in one chunk | 16:05 |
| dansmith | yup, any ordering that I could come up with was dependent on a patch ahead of everything that was like "add a bunch of single-line unrelated changes to various places" followed by a bigger one that hooked them all up | 16:06 |
| dansmith | and it just doesn't make sense | 16:06 |
| dansmith | I kept having to jump back and forth between those two patches to work that out and ultimately fail, which means reviewing isn't really improved | 16:06 |
| dansmith | fewer lines, but not in a meaningful way, so... | 16:06 |
| artom | Well, at least the commit message is already in bullet points | 16:07 |
| *** beekneemech has quit IRC | 16:07 | |
| dansmith | usually that's a bad sign, but in this case, I don't really see a useful way around | 16:07 |
| dansmith | mriedem might disagree, but hopefully not | 16:07 |
| dansmith | he likes to disagree with me periodically to make himself look smart though | 16:08 |
| dansmith | or to assert his smartness, maybe I should say | 16:08 |
| artom | Oh, nice save | 16:08 |
| dansmith | heh | 16:08 |
| mriedem | https://www.youtube.com/watch?v=YLP1_jwT9_M | 16:10 |
| dansmith | heh | 16:11 |
| *** tesseract has quit IRC | 16:11 | |
| dansmith | https://www.youtube.com/watch?v=oGfkPCZYfFw | 16:11 |
| artom | Have we finally devolved to communicating via only YouTube videos? | 16:12 |
| dansmith | trying to speak mriedem's language | 16:12 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: update allocation in binding profile during migrate https://review.opendev.org/656422 | 16:12 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Extend NeutronFixture to handle migrations https://review.opendev.org/655114 | 16:12 |
| *** dpawlik has quit IRC | 16:13 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Docs and functional test for max_local_block_devices https://review.opendev.org/678268 | 16:14 |
| * mriedem goes to get bbq | 16:14 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: prepare func test env for moving servers with bandwidth https://review.opendev.org/655109 | 16:15 |
| dougsz | One other question (sorry!). If I upgrade one cell of many, run online_data_migrations in that cell only, set upgrade_levels to auto (or N-1 level) to pin RPC version, will there be a performance penalty in that cell, until all cells move to the latest RPC version? | 16:16 |
| dansmith | dougsz: you need to move the upgrade pin only once all the cells are updated | 16:17 |
| dansmith | otherwise the conductors will start speaking newer versions to the old cells | 16:17 |
| dansmith | also, | 16:17 |
| dougsz | so auto is not good enough? | 16:17 |
| dansmith | if you use auto, then it will always get the minimum support version, you don't need to tweak it before and after | 16:18 |
| dansmith | online-data-migrations is unrelated to the rpc pin version fwiw | 16:18 |
| *** tbachman has joined #openstack-nova | 16:19 | |
| *** TxGirlGeek has joined #openstack-nova | 16:19 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Func test for migrate server with ports having resource request https://review.opendev.org/655113 | 16:19 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Make _rever_allocation nested allocation aware https://review.opendev.org/676138 | 16:19 |
| dougsz | ok, thanks, that makes sense, i can upgrade a single cell and take my time checking that it doesn't catch fire before upgrading the others | 16:20 |
| *** bnemec has joined #openstack-nova | 16:20 | |
| dansmith | theoretically yes | 16:21 |
| dougsz | :) | 16:21 |
| dansmith | I believe cern does this | 16:21 |
| openstackgerrit | Merged openstack/nova master: trivial: Rename exception argument https://review.opendev.org/671795 | 16:23 |
| openstackgerrit | Merged openstack/nova master: Remove 'hardware.get_host_numa_usage_from_instance' https://review.opendev.org/671797 | 16:23 |
| openstackgerrit | Merged openstack/nova master: Remove 'hardware.host_topology_and_format_from_host' https://review.opendev.org/671798 | 16:23 |
| openstackgerrit | Merged openstack/nova master: Remove 'hardware.instance_topology_from_instance' https://review.opendev.org/671799 | 16:23 |
| openstackgerrit | Merged openstack/nova master: Rework 'hardware.numa_usage_from_instances' https://review.opendev.org/672565 | 16:23 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support reverting migration / resize with bandwidth https://review.opendev.org/676140 | 16:24 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Func test for migrate re-schedule with bandwidth https://review.opendev.org/676972 | 16:27 |
| stephenfin | Nice. Delighted to get rid of all those awful functions ^ | 16:29 |
| stephenfin | This is all going swimmingly | 16:29 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support migrating SRIOV port with bandwidth https://review.opendev.org/676980 | 16:29 |
| *** dpawlik has joined #openstack-nova | 16:29 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Allow migrating server with port resource request https://review.opendev.org/671497 | 16:31 |
| *** dpawlik has quit IRC | 16:34 | |
| artom | dansmith, about the is_trackable_migration removal... | 16:36 |
| dansmith | yar | 16:36 |
| artom | We obviously need to get rid of it, because we want the resource tracker to track NUMA live migrations and update resource usage accordingly | 16:36 |
| artom | But we're removing it entirely, not just for NUMA live migrations. | 16:37 |
| dansmith | isn't this just for reporting purposes? | 16:37 |
| artom | Well, resource usage as well | 16:37 |
| dansmith | I've mostly dumped this out of my brain because placement, | 16:37 |
| artom | NUMATopologyFilter still uses it | 16:37 |
| dansmith | but does the scheduler actually use some of this information anymore? | 16:37 |
| *** tbachman has quit IRC | 16:37 | |
| dansmith | right, okay, the legacy things | 16:37 |
| artom | So because we'll be tracking all live migrations, we need to do claims for all live migrations, not just NUMA ones | 16:38 |
| artom | (which is what's currently in the patch series) | 16:38 |
| dansmith | so right now we don't report this for live migrations, nor do we claim, so the numa filter will send us things we don't have room for, and we also won't do claims to check that it was wrong, right? | 16:38 |
| artom | Yep | 16:38 |
| *** markvoelker has quit IRC | 16:39 | |
| dansmith | okay, but non-numa-affected live migrations are already protected and don't need anything else right? | 16:39 |
| dansmith | because we're depending on placement for the non-numa resources | 16:39 |
| artom | Yeah, that should be fine | 16:39 |
| artom | I was more worried about things like PCI | 16:39 |
| artom | Though I guess we don't support live migrations for those at all, so it the claim fails, it'll just be a plus | 16:40 |
| dansmith | no live migration with pci devices yeah? | 16:40 |
| dansmith | right | 16:40 |
| dansmith | so to continue my thought above, | 16:40 |
| *** tbachman has joined #openstack-nova | 16:40 | |
| artom | Though that's completely untested | 16:40 |
| dansmith | I think you can save that change until the very end, | 16:40 |
| dansmith | because once we're claiming, we're doing better than we are today, and then we can start reporting them, which is another incremental improvement to make the scheduler make better decisions | 16:40 |
| artom | The very end == the massive squashed patch, right? | 16:41 |
| dansmith | well, wherever | 16:41 |
| dansmith | just saing, | 16:41 |
| dansmith | it doesn't need to be synchronous with the change to claim I think | 16:41 |
| artom | Oh, right, that's a given. I've addressed all that feedback, now working on squashing | 16:42 |
| dansmith | okay | 16:42 |
| dansmith | seems like it might be a good candidate for being its own patch though.. right? | 16:42 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'hw:cpu_policy', 'hw:mem_page_size' extra specs from API samples https://review.opendev.org/675338 | 16:44 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start reporting PCPU inventory to placement https://review.opendev.org/671792 | 16:44 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: '_get_(v|p)cpu_total' to '_get_(v|p)cpu_available' https://review.opendev.org/672693 | 16:44 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Rewrap definitions of 'NUMACell' https://review.opendev.org/674395 | 16:44 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: hardware: Differentiate between shared and dedicated CPUs https://review.opendev.org/671800 | 16:44 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Rename 'fields' import to 'obj_fields' https://review.opendev.org/674103 | 16:44 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start reporting 'HW_CPU_HYPERTHREADING' trait https://review.opendev.org/675571 | 16:44 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta https://review.opendev.org/671801 | 16:44 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Fold in argument to '_update_provider_tree_for_vgpu' https://review.opendev.org/676729 | 16:44 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add reshaper for PCPU https://review.opendev.org/674895 | 16:44 |
| artom | dansmith, don't think so - it depends on the RPC changes, and once we start claiming, we need all the other things as well - rollback/application of the migration context, etc | 16:44 |
| dansmith | um | 16:45 |
| dansmith | artom: the reporting stands alone, no? | 16:45 |
| artom | dansmith, it uses the resource tracker, which is affected by claims | 16:45 |
| artom | You're making me doubt myself now | 16:46 |
| dansmith | artom: if we don't report (as we don't today) then the scheduler will keep making the same decisions it is today, but we can stop an operation if the claim tells us there isn't space | 16:46 |
| dansmith | we don't alter the claims as part of that routine do we? | 16:46 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start reporting PCPU inventory to placement https://review.opendev.org/671793 | 16:47 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: '_get_(v|p)cpu_total' to '_get_(v|p)cpu_available' https://review.opendev.org/672693 | 16:47 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Rewrap definitions of 'NUMACell' https://review.opendev.org/674395 | 16:47 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: hardware: Differentiate between shared and dedicated CPUs https://review.opendev.org/671800 | 16:47 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Rename 'fields' import to 'obj_fields' https://review.opendev.org/674103 | 16:47 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start reporting 'HW_CPU_HYPERTHREADING' trait https://review.opendev.org/675571 | 16:47 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta https://review.opendev.org/671801 | 16:47 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Fold in argument to '_update_provider_tree_for_vgpu' https://review.opendev.org/676729 | 16:47 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add reshaper for PCPU https://review.opendev.org/674895 | 16:47 |
| artom | dansmith, I might be misunderstanding you entirely, but reporting and claims go hand in hand - when we start claiming, we'll automatically get correct reporting | 16:47 |
| *** tbachman has quit IRC | 16:47 | |
| *** markvoelker has joined #openstack-nova | 16:47 | |
| artom | Well | 16:48 |
| artom | Unless we keep ignoreing live migration in the resource tracker | 16:48 |
| stephenfin | mriedem: How does one kick off tempest tests when they wouldn't normally run? https://review.opendev.org/#/c/677475/ | 16:48 |
| stephenfin | (because irrelevant-files, I'm assuming) | 16:49 |
| artom | dansmith, is that what you meant? Move the "stop ignoreing live migrations in the resource tracker" to its own small patch on top? | 16:49 |
| dansmith | artom: yes, but, I need to re-read this claims stuff | 16:50 |
| dansmith | maybe I'm making CLAIMS that are incorrect | 16:50 |
| artom | dansmith, ack. Sounds like a good place for me to get lunch | 16:50 |
| artom | dansmith, HI-OH | 16:50 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: setup.cfg: Cleanup https://review.opendev.org/677969 | 16:51 |
| artom | dansmith, want me to push the bottom fixes? | 16:51 |
| dansmith | I think it's because all the regular claim code has been removed | 16:51 |
| dansmith | artom: yes | 16:51 |
| dansmith | I thought the only bits left of the claiming stuff don't actually really interact that much with the resource tracker, | 16:52 |
| dansmith | and rather just see if the things being asked for exist on the compute.. meaning the pci device and the numa arrangement | 16:53 |
| dansmith | so I'm not sure why the reporting stuff is really related directly | 16:53 |
| *** bnemec has quit IRC | 16:53 | |
| *** dougsz has quit IRC | 16:54 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Introduce live_migration_claim() https://review.opendev.org/635669 | 16:54 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: New objects for NUMA live migration https://review.opendev.org/634827 | 16:54 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: add support for augmenting migrate_data with info from claims https://review.opendev.org/634828 | 16:54 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: add support for updating NUMA-related XML on the source https://review.opendev.org/635229 | 16:54 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | 16:54 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | 16:54 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional test for NUMA live migration https://review.opendev.org/672595 | 16:54 |
| efried | dansmith: I should only have to restart n-cpu to pick up a change in oslo.service, yah? | 16:54 |
| efried | at least, the change we care about | 16:54 |
| dansmith | efried: if you're just testing that service then obviously :) | 16:55 |
| efried | I didn't know if other services and their e.g. RPC objects would care. | 16:55 |
| efried | dansmith: Then I've confirmed the oslo.service fix doesn't fix the problem. | 16:56 |
| efried | (which doesn't mean it's wrong, just not complete) | 16:56 |
| dansmith | artom: okay, I see now, I forgot how this works, but I see that if we don't remove that skip we won't do the actual checking of anything, so ignore me | 16:57 |
| dansmith | efried: sweet | 16:57 |
| artom | dansmith, yeah, it's coupling great? | 16:58 |
| artom | *isn't | 16:58 |
| *** boxiang has joined #openstack-nova | 16:58 | |
| artom | Right, pushed, bottom 4 should be good, now lunch | 16:59 |
| *** ozzzo has joined #openstack-nova | 16:59 | |
| *** zhubx has quit IRC | 17:01 | |
| ozzzo | It looks like nova is keeping mysql connections open until they time out. Is there a way to stop these error messages? | 17:01 |
| ozzzo | Aborted connection 10726 to db: 'nova' user: 'nova' host: 'asdf' (Got timeout reading communication packets) | 17:02 |
| *** boxiang has quit IRC | 17:02 | |
| *** zhubx has joined #openstack-nova | 17:02 | |
| *** udesale has quit IRC | 17:05 | |
| *** udesale has joined #openstack-nova | 17:06 | |
| *** igordc has joined #openstack-nova | 17:09 | |
| *** udesale has quit IRC | 17:11 | |
| *** udesale has joined #openstack-nova | 17:12 | |
| mnaser | efried: if you have a few seconds, i wanted to know if you had thoughts about https://review.opendev.org/#/c/676540/ -- should i mark it no longer as deprecated as it's clearly being used? | 17:16 |
| efried | mnaser: or propose whatever alternative we would prefer. mriedem ^ | 17:17 |
| mnaser | i mean if it's clearly being consumed, then i don't mind making the change to stop marking is an deprecated.. | 17:18 |
| *** tbachman has joined #openstack-nova | 17:22 | |
| openstackgerrit | Merged openstack/nova stable/rocky: Remove experimental job on openSUSE 42.3 https://review.opendev.org/677176 | 17:25 |
| openstackgerrit | Merged openstack/nova stable/queens: Remove experimental job on openSUSE 42.3 https://review.opendev.org/677177 | 17:25 |
| openstackgerrit | Merged openstack/nova stable/pike: Remove experimental job on openSUSE 42.3 https://review.opendev.org/677178 | 17:25 |
| efried | mriedem, dansmith: Posted details on the SIGHUP patch https://review.opendev.org/#/c/641907/ | 17:28 |
| efried | We are indeed breaking the privsep pipe (/me says as if I know what I'm talking about) | 17:29 |
| efried | unclear where that would need to be fixed. | 17:29 |
| mriedem | stephenfin: yeah irrelevant-files | 17:29 |
| *** ralonsoh has quit IRC | 17:34 | |
| *** udesale has quit IRC | 17:35 | |
| *** hamzy has joined #openstack-nova | 17:35 | |
| sean-k-mooney | stephenfin: add a followup [DNM] patch that add whitespace to a file that would trigger tempest or temporally remove one fo the fiels from irrelevent-files. i generall prefer to have a DNM testing patch as i can just run the job i care about. | 17:36 |
| mriedem | efried: mnaser: i think we likely should undeprecate the direct file download stuff since clearly there are users and those users have made attempts to upstream an alternative solution which hasn't been approved, but not wholesale rejected either from what i understand, so it's probably not good to just rip it out from them when we're not providing an alternative | 17:37 |
| efried | wfm | 17:38 |
| sean-k-mooney | stephenfin: that chante will alter they way devstack install nova as nova installs all packages in requirements.txt and test-requiremnts.txt system wide by default so it should pass in the gate | 17:38 |
| sean-k-mooney | stephenfin: however if you enable installing in virtual envs it may not work anymore | 17:38 |
| sean-k-mooney | stephenfin: without a change to devstack nova would be relying on other pojects to install pymsql or psycopg2 for postgress | 17:39 |
| efried | mriedem: any advice on next steps for copying mysql logs? https://review.opendev.org/#/c/678051/ | 17:43 |
| *** gbarros has quit IRC | 17:45 | |
| efried | mriedem: | 17:45 |
| efried | Warning: Permanently added '166.78.116.9' (ECDSA) to the list of known hosts. | 17:45 |
| efried | sudo: no tty present and no askpass program specified | 17:45 |
| efried | rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] | 17:45 |
| efried | rsync error: error in rsync protocol data stream (code 12) at io.c(226) [Receiver=3.1.1] | 17:45 |
| sean-k-mooney | i dont think you want to copy them to the executer liek that i think you want to drop them into the working directory on the vm and have the normal log copy task copy them at the end | 17:47 |
| sean-k-mooney | i may be wrong about that but i think that is what i have seen done before | 17:48 |
| *** cz2 has joined #openstack-nova | 17:48 | |
| artom | sean-k-mooney, do you know if we would break anything if we started doing claims for all live migrations? I'm thinking of PCI stuff... | 17:49 |
| artom | Presumably claims for PCI aren't broken in and of themselves, since they're used to boot instances and for cold migrations | 17:50 |
| sean-k-mooney | we do claims for pci deivces for migration already | 17:50 |
| artom | *live*? | 17:50 |
| sean-k-mooney | yes | 17:50 |
| sean-k-mooney | well for neutron sriov ports | 17:51 |
| sean-k-mooney | we dont support live migration with other pci devices | 17:51 |
| sean-k-mooney | but for sriov migration we added cliams | 17:51 |
| sean-k-mooney | *sriov live migration | 17:51 |
| artom | Are we talking about the same claims? | 17:51 |
| sean-k-mooney | in the resouce tracker | 17:51 |
| artom | Yeah | 17:52 |
| sean-k-mooney | we do it as part of pre live migration at dest | 17:52 |
| artom | Could you point me to the review/code? | 17:52 |
| *** tbachman has quit IRC | 17:52 | |
| *** TxGirlGeek has quit IRC | 17:52 | |
| sean-k-mooney | lookign for it now | 17:52 |
| sean-k-mooney | https://review.opendev.org/#/c/616120/ | 17:53 |
| sean-k-mooney | and i think we use them in https://review.opendev.org/#/c/620115/28 | 17:53 |
| sean-k-mooney | the pci manager works differently then the numa toplogy in the RT | 17:54 |
| sean-k-mooney | but ehere is the claim in the compute manager | 17:54 |
| sean-k-mooney | https://review.opendev.org/#/c/620115/28/nova/compute/manager.py | 17:55 |
| openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Try to repro bug 1823251 with mysql logs https://review.opendev.org/678051 | 17:55 |
| openstack | bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] https://launchpad.net/bugs/1823251 | 17:55 |
| artom | sean-k-mooney, it's not actually creating a MoveClaim object though, is it? | 17:56 |
| artom | sean-k-mooney, I'm thinking of a scenario where we live-migration a NUMA instance with neutron SRIOV ports | 17:57 |
| sean-k-mooney | no its not as far as i rememeber | 17:57 |
| artom | Will the MoveClaim for the NUMA part conflict with the PCI claim for the SRIOV part | 17:57 |
| sean-k-mooney | no we claim them in the rt and store the pci addresses in the live migration data object | 17:58 |
| sean-k-mooney | i dont think we are using move claims at all | 17:58 |
| *** munimeha1 has joined #openstack-nova | 17:59 | |
| sean-k-mooney | we might use move claims for cold migration i have not check that in a long time | 17:59 |
| mriedem | efried: i think it's failing because the dest doesn't exist when your task runs | 17:59 |
| mriedem | that's done later here https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_51/678051/3/check/nova-tox-collect-py27/f64b3ba/ara-report/file/5ba0f501-9e58-4f71-859a-ef084a19e0be/#line-11 | 18:00 |
| efried | mriedem: infra said it was because sudo is disabled for tox by default. Latest respin (above) reparents to openstack-tox-with-sudo. | 18:00 |
| artom | sean-k-mooney, if you have the hardware to test those 2 together it would put many minds at east, I think | 18:00 |
| sean-k-mooney | i plan to do that yes | 18:00 |
| sean-k-mooney | ill do it monday or tuesday if you like | 18:01 |
| efried | mriedem: but maybe that too. Though I would have thought they would set up log dirs before invoking posts. | 18:01 |
| sean-k-mooney | i have 1 of my sriov systems setup with 18.04 so i need to install 18.04 on the second one an then install centos on a third host to test something else | 18:02 |
| sean-k-mooney | ill kick of those now and i should be able to test it monday if you bug me or tuesday if you dont | 18:03 |
| *** tbachman has joined #openstack-nova | 18:04 | |
| artom | sean-k-mooney, appreciated :) | 18:05 |
| mriedem | efried: maybe it is https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_51/678051/3/check/nova-tox-collect-py27/f64b3ba/ara-report/file/c8a99499-4049-4878-b0a9-13a9292437f4/#line-1 | 18:06 |
| mriedem | efried: so maybe your dest needs to be '{{ zuul_log_path }}' | 18:06 |
| mriedem | er + /mysql/ | 18:06 |
| mriedem | or {{ zuul_output_dir }}/logs/mysql | 18:07 |
| mriedem | https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_51/678051/3/check/nova-tox-collect-py27/f64b3ba/ara-report/file/f31abc72-92ae-4e7d-b37c-8c124f7f32bd/#line-1 | 18:07 |
| *** gbarros has joined #openstack-nova | 18:08 | |
| *** ozzzo has quit IRC | 18:09 | |
| efried | mriedem: okay, can try something along those lines if this one borks again. | 18:09 |
| mriedem | ok | 18:10 |
| mriedem | you know about the ara report right? | 18:10 |
| mriedem | to see what runs before and after you with what values | 18:10 |
| sean-k-mooney | artom: once i have it up an running i can give you remote acess to poke around if you want. i have a static ip so i can easily port forwad you to i the hosts. ubuntu is install now so i just need to add a couple of kernel arges to make sriov work then stack with your code | 18:13 |
| artom | sean-k-mooney, ack, that works | 18:14 |
| artom | sean-k-mooney, I'm thinking of this bit in the resource tracker: https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L283-L295 | 18:15 |
| artom | For NUMA LM, we call that during check_live_migration_at_destination | 18:15 |
| efried | mriedem: no, I knew it was there, but never really understood what it meant. | 18:16 |
| mriedem | btw, speaking of that nova.image.download module https://logs.opendev.org/84/676684/1/check/neutron-grenade-multinode/b0a76ac/logs/screen-n-api.txt#_Aug_15_13_25_04_744293 | 18:18 |
| mriedem | i'm not sure why that's happening | 18:18 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L294-L295 might be a proablem | 18:19 |
| sean-k-mooney | since we have already claimed them on the dest it could claim them twice | 18:19 |
| mriedem | oh gdi of course https://review.opendev.org/#/c/676684/ | 18:19 |
| mriedem | hehh | 18:19 |
| sean-k-mooney | i think currenlty we only use that for cold migration right | 18:19 |
| artom | sean-k-mooney, *currently* | 18:21 |
| artom | But NUMA LM is based on doing a MoveClaim | 18:21 |
| sean-k-mooney | right so we would have to modify this or it wont work with sriov live migration | 18:22 |
| sean-k-mooney | or rather might not | 18:22 |
| sean-k-mooney | will you be passing in somthing to indicate its a live migration | 18:23 |
| artom | Or just ignore pci_requests entirely if it's a live migration? | 18:23 |
| sean-k-mooney | that is what i was going to suggest | 18:23 |
| artom | Yeah, the migration will hava a type set | 18:23 |
| openstackgerrit | Andreas Jaeger proposed openstack/nova master: PDF documentation build https://review.opendev.org/676730 | 18:23 |
| *** gbarros has quit IRC | 18:23 | |
| sean-k-mooney | its nice that it is claiming alias based pci devices | 18:23 |
| sean-k-mooney | we dont currently support that in the sriov migration | 18:24 |
| sean-k-mooney | but that also means we woudl not update the xml | 18:24 |
| artom | Yeah, we can't claim resource we have no intention of using | 18:24 |
| sean-k-mooney | so what we need to to is 1 check that there is not alias based pci device and two not call the cpi claim bit if its a live migration | 18:24 |
| artom | Why is 2 alone not enough? | 18:25 |
| sean-k-mooney | it is but the other is a safty mesure | 18:25 |
| sean-k-mooney | e.g. we shoudl not attemt to migrate if you have alias based pci passthough | 18:25 |
| sean-k-mooney | its not supported at all | 18:25 |
| sean-k-mooney | we might already check that somewhere | 18:26 |
| sean-k-mooney | if we dont we should in the conductor | 18:26 |
| artom | Seems like that should be separate from NUMA LM tho | 18:26 |
| sean-k-mooney | ya it is. | 18:26 |
| artom | NUMA LM just cares about not stepping on SRIOV LM's toes | 18:27 |
| sean-k-mooney | so for numa the if(migration.type!=live) shoudl be enough | 18:27 |
| artom | We have functional tests for SRIOV LM? | 18:28 |
| artom | I wonder if we could try both in a func test | 18:28 |
| sean-k-mooney | we dont | 18:28 |
| artom | Func tests for neutron SRIOV at all? | 18:29 |
| sean-k-mooney | nope | 18:29 |
| sean-k-mooney | we have for flavor based pci passthough | 18:29 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/tests/functional/libvirt/test_pci_sriov_servers.py | 18:29 |
| sean-k-mooney | but no fucntional test for neutorn sriov | 18:29 |
| artom | So manual testing it is | 18:30 |
| sean-k-mooney | so we would have to extend the neutron mock | 18:30 |
| artom | Yeah | 18:30 |
| sean-k-mooney | but we could add them if we did | 18:30 |
| sean-k-mooney | i dont think we have any livemigration functional tests that use the libvirt fake driver | 18:30 |
| artom | We didn't | 18:31 |
| artom | NUMA LM's going to be the first | 18:31 |
| artom | You meant fakelibvirt, right? | 18:31 |
| sean-k-mooney | so your going to implement the multiple port bidnign workflow in the neutron fixtures | 18:31 |
| sean-k-mooney | yes | 18:31 |
| artom | Because we have a couple that use fake virt | 18:31 |
| sean-k-mooney | fakelibvirt | 18:31 |
| artom | sean-k-mooney, Ha. Ha. Ha. | 18:32 |
| sean-k-mooney | artom: you will need that form live migration testing i think | 18:32 |
| sean-k-mooney | you would for sriov lm | 18:32 |
| artom | sean-k-mooney, heh, one thing at a time :) | 18:32 |
| sean-k-mooney | for numa you could fall back to the old codepath | 18:32 |
| artom | If we decide neutron sriov func test is something we want, I don't mind doing it | 18:33 |
| artom | But for NUMA LM it's irrelevant | 18:33 |
| *** gbarros has joined #openstack-nova | 18:35 | |
| sean-k-mooney | it would be nice to have functional test for this at some point but i think we shoudl wait till U to do that | 18:36 |
| sean-k-mooney | at least that we we can do some firstpart testing beyond unit tests | 18:36 |
| mriedem | sean-k-mooney: https://review.opendev.org/#/c/678098/2 needs to mention the reason for the difference in the backport (in the commit message) | 18:37 |
| mriedem | otherwise it looks OK | 18:37 |
| sean-k-mooney | ok ill respin it now | 18:37 |
| sean-k-mooney | it was this https://github.com/openstack/os-vif/commit/ee124d2e98c2660ced7d71c1bd10525bf095a939 | 18:38 |
| mriedem | yeah i just ofund that | 18:38 |
| mriedem | Id8b71172fb06b435cf169a7e55c11233f22fa65b | 18:38 |
| *** gyee has quit IRC | 18:42 | |
| openstackgerrit | sean mooney proposed openstack/os-vif stable/stein: only disable mac ageing for ovs hybrid plug https://review.opendev.org/678098 | 18:44 |
| sean-k-mooney | something like ^ | 18:44 |
| sean-k-mooney | the other option we have would be to backport Id8b71172fb06b435cf169a7e55c11233f22fa65b | 18:44 |
| *** gyee has joined #openstack-nova | 18:46 | |
| mriedem | no we don't want to backport that big refactor | 18:49 |
| mriedem | +2 on the other | 18:49 |
| mriedem | melwitt: maybe you want to hit https://review.opendev.org/#/c/678098/ since lee is probably done | 18:49 |
| *** xek has joined #openstack-nova | 18:58 | |
| mriedem | melwitt: any changes of getting https://review.opendev.org/#/c/507486/ and/or https://review.opendev.org/#/c/640898/ updated soonish before i lose context on those? | 18:58 |
| mriedem | *chance | 18:58 |
| *** hamzy has quit IRC | 19:01 | |
| *** lyarwood has quit IRC | 19:04 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Introduce live_migration_claim() https://review.opendev.org/635669 | 19:04 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: New objects for NUMA live migration https://review.opendev.org/634827 | 19:04 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: add support for augmenting migrate_data with info from claims https://review.opendev.org/634828 | 19:04 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: add support for updating NUMA-related XML on the source https://review.opendev.org/635229 | 19:04 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | 19:04 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | 19:04 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional test for NUMA live migration https://review.opendev.org/672595 | 19:04 |
| *** gbarros has quit IRC | 19:07 | |
| *** bnemec has joined #openstack-nova | 19:10 | |
| *** bnemec is now known as beekneemech | 19:11 | |
| *** eharney has quit IRC | 19:13 | |
| sean-k-mooney | artom: the last patch ^ is the one i should deploy yes? | 19:16 |
| artom | sean-k-mooney, hol'up, things are in motion | 19:16 |
| sean-k-mooney | have you added the sriov if? | 19:16 |
| sean-k-mooney | ok | 19:16 |
| artom | But in general, yeah | 19:17 |
| artom | I did add the if | 19:17 |
| sean-k-mooney | im not quite ready yet to run it but its close | 19:17 |
| *** boxiang has joined #openstack-nova | 19:17 | |
| sean-k-mooney | i normlly run a vanila devstack install first then switch to the ting i want to test | 19:17 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Allow strict_proxies for sdk Connection https://review.opendev.org/676926 | 19:18 |
| efried | I think I oughtta rebase the ironic series on that ^ and switch it on. | 19:19 |
| *** zhubx has quit IRC | 19:19 | |
| *** zhubx has joined #openstack-nova | 19:20 | |
| *** gbarros has joined #openstack-nova | 19:22 | |
| *** ozzzo has joined #openstack-nova | 19:23 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: Introduces SDK to IronicDriver and uses for node.get https://review.opendev.org/642899 | 19:24 |
| *** boxiang has quit IRC | 19:24 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: Use SDK for node.list https://review.opendev.org/656027 | 19:24 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Use SDK for validating instance and node https://review.opendev.org/656028 | 19:24 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Use SDK for setting instance id https://review.opendev.org/659690 | 19:24 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Use SDK for add/remove instance info from node https://review.opendev.org/659691 | 19:24 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Use SDK for getting network metadata from node https://review.opendev.org/670213 | 19:24 |
| efried | so there's that | 19:24 |
| *** hamzy has joined #openstack-nova | 19:26 | |
| *** gbarros has quit IRC | 19:30 | |
| *** gbarros has joined #openstack-nova | 19:35 | |
| NobodyCam | Happy Friday Nova folks | 19:38 |
| dansmith | don't fall for it, it's a trap | 19:38 |
| NobodyCam | off the wall question; is it possible to the kvm vgamem option per flavor or image? | 19:39 |
| NobodyCam | lol | 19:39 |
| *** tbachman has quit IRC | 19:39 | |
| dansmith | see ^ | 19:39 |
| sean-k-mooney | NobodyCam: yes | 19:39 |
| sean-k-mooney | you can change it via the image | 19:39 |
| NobodyCam | Nice! :) | 19:40 |
| NobodyCam | Thank you | 19:40 |
| sean-k-mooney | https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt-image.json#L57-L62 | 19:40 |
| sean-k-mooney | also we dont test that so it might be broken | 19:40 |
| sean-k-mooney | but it should work | 19:40 |
| sean-k-mooney | let us know | 19:40 |
| sean-k-mooney | oh and apparently you can set a limit in the flavor | 19:41 |
| *** gbarros has quit IRC | 19:43 | |
| NobodyCam | I'll play with that | 19:44 |
| NobodyCam | <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/> | 19:44 |
| NobodyCam | is what I'm getting by default | 19:44 |
| sean-k-mooney | ya this is the code that uses it in the livbrt dirver | 19:45 |
| sean-k-mooney | https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/driver.py#L4809-L4846 | 19:45 |
| sean-k-mooney | apparently its supported by vmware too | 19:45 |
| sean-k-mooney | i think that default is coming form libvirt | 19:46 |
| NobodyCam | yea looking at the code I'm not sure that will get me what I'm looking for | 19:48 |
| sean-k-mooney | what are you trying to do | 19:49 |
| sean-k-mooney | oh you want to change vgamem | 19:49 |
| sean-k-mooney | not vram | 19:49 |
| sean-k-mooney | https://libvirt.org/formatdomain.html#elementsVideo looking at the docs | 19:50 |
| sean-k-mooney | vgamem is used for the size of the fallback framebuffer for the qxl type only | 19:51 |
| NobodyCam | oh, yea reading that now | 19:52 |
| sean-k-mooney | vram is basicaly the amount of ram reported to as on the grapics card to the guest | 19:52 |
| sean-k-mooney | so its normally the only one you care about changing | 19:53 |
| NobodyCam | yea, Let me test that, Thank you sean-k-mooney :) | 19:54 |
| sean-k-mooney | no worries. i added suport for requestion virtio a few weeks ago so it relitivly fresh in my mind since i wen though all the option at that time | 19:54 |
| NobodyCam | :) | 19:58 |
| *** ivve has joined #openstack-nova | 20:03 | |
| *** mgariepy has quit IRC | 20:06 | |
| *** eharney has joined #openstack-nova | 20:10 | |
| *** weshay is now known as weshay_MOD | 20:10 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: neutron: refactor nw info cache refresh out of associate_floating_ip https://review.opendev.org/678300 | 20:15 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Trap and log errors from _update_inst_info_cache_for_disassociated_fip https://review.opendev.org/678301 | 20:15 |
| *** igordc has quit IRC | 20:37 | |
| sean-k-mooney | artom: can you login to "ssh stack@jump.seanmooney.info -p 10024" | 20:48 |
| artom | sean-k-mooney, I cannot, looks like a timeout | 20:48 |
| artom | Also, this is a public channel | 20:48 |
| sean-k-mooney | ya i know i added your public keys | 20:48 |
| sean-k-mooney | i was hoping it would jsut work for you and you would be wondering how | 20:49 |
| artom | *suspicious Fry* I don't remember sending you my public key | 20:49 |
| sean-k-mooney | you didnt but i got them form your github | 20:49 |
| sean-k-mooney | i have the forwarding workin glocally but i have not test it form external and it look like it not right | 20:50 |
| artom | I have my pubkeys on github? | 20:52 |
| artom | Oh, the oshax thing? | 20:52 |
| sean-k-mooney | yep https://github.com/notartom.keys | 20:52 |
| sean-k-mooney | all gitub user have there public keys on github if you uploade them | 20:53 |
| artom | TIL | 20:53 |
| sean-k-mooney | want to try again | 20:53 |
| sean-k-mooney | i change the port the nat was applying too | 20:53 |
| sean-k-mooney | if it doesnt work no worries ill fix over the week end. still learnign how to configre my ubitui router | 20:54 |
| NobodyCam | FYI sean-k-mooney that setting seems to have taken care of my error ... Thank you (again) for your assistance :) | 20:54 |
| sean-k-mooney | NobodyCam: cool glad it still works | 20:55 |
| NobodyCam | hehehe ;p | 20:55 |
| artom | sean-k-mooney, hanging on debug1: Connecting to jump.seanmooney.info [93.107.226.179] port 10024. | 21:04 |
| artom | Just... open a tmux :) | 21:05 |
| sean-k-mooney | artom: ok its the rever nat that is not working yet then | 21:05 |
| sean-k-mooney | i mean i could also jsut use the simple ui for doing it rthar then doing it manually | 21:06 |
| sean-k-mooney | that might work | 21:07 |
| sean-k-mooney | ill fix it over teh weekend if not by connecting via my phone | 21:08 |
| sean-k-mooney | on the plus side i think numa migration is working with your patches | 21:09 |
| sean-k-mooney | i need to test it properly but i did just migrate a vm with hw:numa_nodes=2 | 21:10 |
| sean-k-mooney | and one with a sriov macvtap | 21:14 |
| sean-k-mooney | + numa | 21:14 |
| mriedem | huh, i saw this unit test job timeout https://18e51e15a34f17ffbc81-ffc80d196410a18186442d9badd30b78.ssl.cf2.rackcdn.com/677819/2/gate/openstack-tox-py27/3446844/job-output.txt | 21:18 |
| mriedem | and if you look at the tests being run, they are being run in serial after some point | 21:18 |
| *** xek has quit IRC | 21:25 | |
| *** needscoffee is now known as kmalloc | 21:32 | |
| sean-k-mooney | artom: so... your numa code | 21:33 |
| sean-k-mooney | have you refactored it much lately | 21:33 |
| artom | sean-k-mooney, it's currently half-broken as I work through feedback and tests failing... | 21:33 |
| sean-k-mooney | if by half broken you mean it nolonger updates the xml then yes | 21:33 |
| sean-k-mooney | yes it is | 21:33 |
| artom | Sounds about right. | 21:34 |
| artom | Fields got renamed, so various if checks are failing. | 21:35 |
| sean-k-mooney | on the plus side the vm was moved | 21:35 |
| sean-k-mooney | so its not anymore broken the without your code | 21:35 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Introduce live_migration_claim() https://review.opendev.org/635669 | 21:43 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: New objects for NUMA live migration https://review.opendev.org/634827 | 21:43 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: add support for augmenting migrate_data with info from claims https://review.opendev.org/634828 | 21:43 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: add support for updating NUMA-related XML on the source https://review.opendev.org/635229 | 21:43 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | 21:43 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | 21:43 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional test for NUMA live migration https://review.opendev.org/672595 | 21:43 |
| * artom calls it a week | 21:44 | |
| sean-k-mooney | o/ ill test that on monday | 21:44 |
| sean-k-mooney | or when your finsihed erworking it | 21:45 |
| *** igordc has joined #openstack-nova | 21:45 | |
| *** takamatsu has quit IRC | 21:55 | |
| *** rcernin has joined #openstack-nova | 22:11 | |
| *** KeithMnemonic has quit IRC | 22:13 | |
| *** ivve has quit IRC | 22:17 | |
| *** BjoernT has quit IRC | 22:18 | |
| *** kaisers has quit IRC | 22:21 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Find instance in another cell during floating IP re-association https://review.opendev.org/656594 | 22:24 |
| *** kaisers has joined #openstack-nova | 22:26 | |
| *** mriedem has quit IRC | 22:27 | |
| *** TxGirlGeek has joined #openstack-nova | 22:39 | |
| openstackgerrit | Merged openstack/python-novaclient master: Add --migration-type and --source-compute to migration-list https://review.opendev.org/675117 | 22:43 |
| *** tonyb[m] has joined #openstack-nova | 22:55 | |
| *** macz has quit IRC | 22:57 | |
| *** igordc has quit IRC | 23:00 | |
| *** rcernin has quit IRC | 23:11 | |
| *** rcernin has joined #openstack-nova | 23:12 | |
| *** beekneemech is now known as keanu | 23:26 | |
| *** keanu is now known as beekneemech | 23:27 | |
| melwitt | mriedem: yes, sorry, been caught up in downstream stuff of late. I will update those soon, probably this weekend | 23:39 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!