| *** slaweq has joined #openstack-nova | 00:11 | |
| *** slaweq has quit IRC | 00:16 | |
| *** betherly has joined #openstack-nova | 00:35 | |
| *** betherly has quit IRC | 00:39 | |
| *** bhagyashris has joined #openstack-nova | 00:47 | |
| *** betherly has joined #openstack-nova | 00:55 | |
| *** jmlowe has joined #openstack-nova | 00:58 | |
| *** betherly has quit IRC | 01:00 | |
| *** jbernard_ is now known as jbernard | 01:44 | |
| *** betherly has joined #openstack-nova | 01:46 | |
| *** nafiux has quit IRC | 01:48 | |
| *** betherly has quit IRC | 01:52 | |
| *** huaqiang has quit IRC | 01:53 | |
| *** zhouyao has joined #openstack-nova | 01:55 | |
| *** sapd1_ has quit IRC | 01:59 | |
| *** slaweq has joined #openstack-nova | 02:11 | |
| *** sapd1 has joined #openstack-nova | 02:13 | |
| *** slaweq has quit IRC | 02:16 | |
| *** zhouyao has quit IRC | 02:45 | |
| openstackgerrit | Merged openstack/nova master: Follow-up for I2936ce8cb293dc80e1a426094fdae6e675461470 https://review.opendev.org/672669 | 02:53 |
|---|---|---|
| openstackgerrit | Merged openstack/nova master: nit: fix the test case of migration obj_make_compatible https://review.opendev.org/673961 | 02:53 |
| *** sapd1 has quit IRC | 02:54 | |
| openstackgerrit | pengyuesheng proposed openstack/os-resource-classes master: Sync Sphinx requirement https://review.opendev.org/674164 | 02:58 |
| *** betherly has joined #openstack-nova | 02:59 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: Correct global_request_id sent to Placement https://review.opendev.org/674129 | 03:00 |
| *** betherly has quit IRC | 03:03 | |
| *** dpawlik has quit IRC | 03:04 | |
| *** ash2307 has left #openstack-nova | 03:12 | |
| *** betherly has joined #openstack-nova | 03:19 | |
| *** psachin has joined #openstack-nova | 03:21 | |
| *** betherly has quit IRC | 03:24 | |
| *** gyee has quit IRC | 03:26 | |
| *** psachin has quit IRC | 03:36 | |
| *** betherly has joined #openstack-nova | 03:39 | |
| *** betherly has quit IRC | 03:45 | |
| *** betherly has joined #openstack-nova | 03:50 | |
| *** betherly has quit IRC | 03:55 | |
| *** boxiang has joined #openstack-nova | 03:56 | |
| *** jojoda_ has joined #openstack-nova | 04:09 | |
| *** betherly has joined #openstack-nova | 04:11 | |
| *** slaweq has joined #openstack-nova | 04:11 | |
| *** ratailor has joined #openstack-nova | 04:16 | |
| *** slaweq has quit IRC | 04:16 | |
| *** betherly has quit IRC | 04:16 | |
| *** betherly has joined #openstack-nova | 04:21 | |
| *** betherly has quit IRC | 04:26 | |
| *** ratailor has quit IRC | 04:30 | |
| *** ratailor has joined #openstack-nova | 04:31 | |
| *** whoami-rajat has joined #openstack-nova | 04:34 | |
| *** abhishekk has joined #openstack-nova | 04:38 | |
| *** betherly has joined #openstack-nova | 04:41 | |
| *** betherly has quit IRC | 04:46 | |
| *** tkajinam has quit IRC | 05:00 | |
| *** sapd1 has joined #openstack-nova | 05:01 | |
| *** betherly has joined #openstack-nova | 05:02 | |
| *** tkajinam has joined #openstack-nova | 05:02 | |
| *** Luzi has joined #openstack-nova | 05:05 | |
| *** udesale has joined #openstack-nova | 05:07 | |
| *** betherly has quit IRC | 05:07 | |
| *** betherly has joined #openstack-nova | 05:18 | |
| *** yaawang has quit IRC | 05:19 | |
| *** betherly has quit IRC | 05:22 | |
| *** yaawang has joined #openstack-nova | 05:28 | |
| *** yaawang has quit IRC | 05:33 | |
| openstackgerrit | Merged openstack/nova stable/rocky: Fix python3 compatibility of rbd get_fsid https://review.opendev.org/664513 | 05:34 |
| *** markvoelker has joined #openstack-nova | 05:34 | |
| *** prometheanfire has left #openstack-nova | 05:37 | |
| *** markvoelker has quit IRC | 05:39 | |
| *** tkajinam_ has joined #openstack-nova | 05:53 | |
| *** tkajinam has quit IRC | 05:55 | |
| *** janki has joined #openstack-nova | 06:04 | |
| *** slaweq has joined #openstack-nova | 06:11 | |
| *** slaweq has quit IRC | 06:15 | |
| *** jojoda_ has quit IRC | 06:17 | |
| *** ratailor has quit IRC | 06:32 | |
| *** ratailor has joined #openstack-nova | 06:33 | |
| *** janki has quit IRC | 06:34 | |
| *** amodi has quit IRC | 06:34 | |
| *** janki has joined #openstack-nova | 06:35 | |
| *** jaosorior has joined #openstack-nova | 06:38 | |
| *** maciejjozefczyk has joined #openstack-nova | 06:54 | |
| *** jhesketh has joined #openstack-nova | 06:54 | |
| *** kdean has quit IRC | 06:56 | |
| *** rcernin has quit IRC | 06:58 | |
| *** slaweq has joined #openstack-nova | 07:05 | |
| *** shilpasd has joined #openstack-nova | 07:05 | |
| *** redrobot has quit IRC | 07:17 | |
| *** tesseract has joined #openstack-nova | 07:17 | |
| *** ratailor has quit IRC | 07:33 | |
| *** ratailor has joined #openstack-nova | 07:34 | |
| *** tssurya has joined #openstack-nova | 07:38 | |
| *** janki has quit IRC | 07:43 | |
| *** jcosmao has joined #openstack-nova | 07:47 | |
| *** cdent has joined #openstack-nova | 07:48 | |
| aspiers | o] | 07:51 |
| aspiers | wow, that was an unintentional dance move, let's try again | 07:52 |
| aspiers | o/ | 07:52 |
| kashyap | Hehe | 07:52 |
| kashyap | Morning, /me is especially feeling high-spirited today | 07:52 |
| kashyap | Maybe it's the accidental electric shock I got last night while plugging in a bad cable :D | 07:53 |
| cdent | one more shock and you will become the superhero you were born to be | 07:55 |
| kashyap | Hehe. That really shook me | 07:55 |
| *** ralonsoh has joined #openstack-nova | 08:10 | |
| *** threestrands has quit IRC | 08:25 | |
| *** avolkov has joined #openstack-nova | 08:25 | |
| *** brault has joined #openstack-nova | 08:40 | |
| *** brault has quit IRC | 08:41 | |
| aspiers | X-D | 08:42 |
| *** tkajinam_ has quit IRC | 09:01 | |
| *** derekh has joined #openstack-nova | 09:06 | |
| openstackgerrit | Huachang Wang proposed openstack/nova master: doc: correct the information of 'cpu_map' https://review.opendev.org/673272 | 09:13 |
| *** markvoelker has joined #openstack-nova | 09:13 | |
| *** Conqueror has joined #openstack-nova | 09:14 | |
| *** ksdean has joined #openstack-nova | 09:25 | |
| *** shilpasd has quit IRC | 09:30 | |
| openstackgerrit | Huachang Wang proposed openstack/nova master: Add the support of CPU feature 'AVX512-VNNI' https://review.opendev.org/673480 | 09:34 |
| *** takashin has left #openstack-nova | 09:35 | |
| *** ratailor has quit IRC | 09:35 | |
| *** ratailor has joined #openstack-nova | 09:36 | |
| *** jangutter has quit IRC | 09:36 | |
| openstackgerrit | Brin Zhang proposed openstack/nova master: WIP: Add user_id and project_id colume to Migration https://review.opendev.org/673990 | 09:39 |
| openstackgerrit | Brin Zhang proposed openstack/nova master: WIP: Filter migrations by user_id/project_id https://review.opendev.org/674243 | 09:39 |
| *** shilpasd has joined #openstack-nova | 09:43 | |
| *** ksdean has quit IRC | 09:46 | |
| *** markvoelker has quit IRC | 09:46 | |
| *** jangutter has joined #openstack-nova | 09:51 | |
| *** ociuhandu has joined #openstack-nova | 09:54 | |
| *** jawad_axd has joined #openstack-nova | 09:56 | |
| *** abhishekk has quit IRC | 10:12 | |
| *** priteau has joined #openstack-nova | 10:13 | |
| *** ociuhandu has quit IRC | 10:34 | |
| *** ociuhandu has joined #openstack-nova | 10:34 | |
| *** ociuhandu has quit IRC | 10:36 | |
| *** ociuhandu has joined #openstack-nova | 10:37 | |
| *** brault has joined #openstack-nova | 10:40 | |
| *** brault has quit IRC | 10:44 | |
| *** brault has joined #openstack-nova | 10:46 | |
| *** ratailor has quit IRC | 10:49 | |
| *** ksdean has joined #openstack-nova | 10:50 | |
| *** cdent has quit IRC | 10:50 | |
| *** ratailor has joined #openstack-nova | 10:55 | |
| *** brinzhang has quit IRC | 10:55 | |
| *** boxiang has quit IRC | 11:00 | |
| *** zhubx has joined #openstack-nova | 11:00 | |
| *** ricolin has quit IRC | 11:00 | |
| *** cdent has joined #openstack-nova | 11:01 | |
| *** jaosorior has quit IRC | 11:08 | |
| *** ksdean has quit IRC | 11:10 | |
| *** ksdean has joined #openstack-nova | 11:11 | |
| *** ccamacho has joined #openstack-nova | 11:18 | |
| *** zbr has quit IRC | 11:25 | |
| *** bhagyashris has quit IRC | 11:26 | |
| *** takamatsu is now known as mauro|call | 11:44 | |
| *** rpittau|afk is now known as rpittau | 11:56 | |
| openstackgerrit | Merged openstack/os-vif master: Bump the openstackdocstheme extension to 1.20 https://review.opendev.org/672857 | 11:58 |
| *** markvoelker has joined #openstack-nova | 12:00 | |
| *** tbachman has quit IRC | 12:01 | |
| *** amodi has joined #openstack-nova | 12:01 | |
| *** udesale has quit IRC | 12:02 | |
| *** udesale has joined #openstack-nova | 12:03 | |
| *** markvoelker has quit IRC | 12:03 | |
| *** markvoelker has joined #openstack-nova | 12:03 | |
| *** markvoelker has quit IRC | 12:04 | |
| *** panda is now known as panda|lunch | 12:05 | |
| *** redrobot has joined #openstack-nova | 12:07 | |
| *** markvoelker has joined #openstack-nova | 12:16 | |
| *** jawad_axd has quit IRC | 12:16 | |
| *** cdent has quit IRC | 12:18 | |
| *** Luzi has quit IRC | 12:21 | |
| *** ricolin has joined #openstack-nova | 12:23 | |
| *** ociuhandu has quit IRC | 12:24 | |
| *** dpawlik has joined #openstack-nova | 12:25 | |
| *** ratailor has quit IRC | 12:27 | |
| *** priteau has quit IRC | 12:28 | |
| openstackgerrit | Huachang Wang proposed openstack/nova master: doc: correct the information of 'cpu_map' https://review.opendev.org/673272 | 12:29 |
| *** cdent has joined #openstack-nova | 12:35 | |
| *** zbr has joined #openstack-nova | 12:35 | |
| *** markvoelker has quit IRC | 12:58 | |
| *** markvoelker has joined #openstack-nova | 12:58 | |
| openstackgerrit | Merged openstack/nova master: (Re-)enable vnc console tests in nova-multi-cell job https://review.opendev.org/667388 | 13:05 |
| *** panda|lunch is now known as panda | 13:06 | |
| *** priteau has joined #openstack-nova | 13:10 | |
| *** ociuhandu has joined #openstack-nova | 13:11 | |
| *** tbachman has joined #openstack-nova | 13:11 | |
| *** ccamacho has quit IRC | 13:18 | |
| *** eharney has joined #openstack-nova | 13:19 | |
| *** mriedem has joined #openstack-nova | 13:21 | |
| *** BjoernT has joined #openstack-nova | 13:22 | |
| *** markvoelker has quit IRC | 13:25 | |
| *** markvoelker has joined #openstack-nova | 13:25 | |
| *** BjoernT_ has joined #openstack-nova | 13:25 | |
| *** BjoernT has quit IRC | 13:27 | |
| *** mdbooth_ has joined #openstack-nova | 13:29 | |
| *** mdbooth_ has quit IRC | 13:29 | |
| *** mrch_ has joined #openstack-nova | 13:32 | |
| *** ivve has joined #openstack-nova | 13:43 | |
| *** mlavalle has joined #openstack-nova | 13:59 | |
| aspiers | is there an easy way to get encrypted ephemeral disks with devstack? | 14:12 |
| *** artom has quit IRC | 14:14 | |
| *** liuyulong has joined #openstack-nova | 14:15 | |
| *** artom has joined #openstack-nova | 14:16 | |
| melwitt | that would be a lyarwood question ^ | 14:16 |
| aspiers | melwitt: thanks! | 14:20 |
| aspiers | it seems that it requires lvm | 14:20 |
| aspiers | so just adding to [[post-config|$NOVA_CONF]] isn't enough... I think | 14:20 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Integrate 'pre-commit' https://review.opendev.org/665518 | 14:21 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Keep pre-commit inline with hacking and fix whitespace https://review.opendev.org/674057 | 14:21 |
| *** mrch_ has quit IRC | 14:22 | |
| lyarwood | aspiers: yeah there's only LVM support for that at the moment | 14:29 |
| aspiers | lyarwood: that's fine | 14:29 |
| aspiers | enable_plugin barbican | 14:29 |
| aspiers | NOVA_BACKEND=LVM | 14:29 |
| aspiers | is that enough, combined with the extra config options? | 14:29 |
| openstackgerrit | Merged openstack/nova stable/stein: Fix AttributeError in RT._update_usage_from_migration https://review.opendev.org/673856 | 14:29 |
| aspiers | [[post-config|$NOVA_CONF]] | 14:29 |
| aspiers | [ephemeral_storage_encryption] | 14:30 |
| aspiers | key_size = 256 | 14:30 |
| aspiers | cipher = aes-xts-plain64 | 14:30 |
| aspiers | enabled = True | 14:30 |
| *** shilpasd has quit IRC | 14:30 | |
| aspiers | lyarwood: something like that? | 14:30 |
| lyarwood | aspiers: yeah something like that | 14:30 |
| lyarwood | aspiers: I haven't played around with it in a while | 14:30 |
| aspiers | OK cool thanks I'll give it a go! | 14:30 |
| aspiers | ah I need the git URL after enable_plugin barbican | 14:32 |
| *** dpawlik has quit IRC | 14:38 | |
| *** tbachman has quit IRC | 14:47 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova-osprofiler-redis job to experimental queue https://review.opendev.org/674326 | 14:58 |
| *** mlavalle has quit IRC | 15:00 | |
| mriedem | you could look at the barbican zuul ci job setup for some of that | 15:04 |
| *** tssurya has quit IRC | 15:05 | |
| mriedem | they also have https://github.com/openstack/barbican/blob/master/devstack/local.conf.example | 15:05 |
| mriedem | not sure how accurate it is | 15:05 |
| mriedem | else https://github.com/openstack/barbican/blob/master/playbooks/legacy/barbican-devstack-base/run.yaml | 15:06 |
| *** ociuhandu has quit IRC | 15:09 | |
| *** ociuhandu has joined #openstack-nova | 15:09 | |
| *** liuyulong has quit IRC | 15:13 | |
| *** mdbooth has quit IRC | 15:16 | |
| *** markvoelker has quit IRC | 15:18 | |
| *** markvoelker has joined #openstack-nova | 15:20 | |
| *** eharney has quit IRC | 15:32 | |
| openstackgerrit | Merged openstack/nova master: trivial: Remove unused function parameter https://review.opendev.org/671796 | 15:32 |
| *** factor has joined #openstack-nova | 15:33 | |
| *** cdent has quit IRC | 15:37 | |
| *** zhubx has quit IRC | 15:38 | |
| *** zhubx has joined #openstack-nova | 15:38 | |
| *** zhubx has quit IRC | 15:40 | |
| *** zhubx has joined #openstack-nova | 15:41 | |
| *** jangutter has quit IRC | 15:51 | |
| *** maciejjozefczyk has quit IRC | 15:54 | |
| *** ociuhandu has quit IRC | 15:56 | |
| *** spatel has joined #openstack-nova | 15:59 | |
| spatel | sean-k-mooney: hey!! | 15:59 |
| *** ociuhandu has joined #openstack-nova | 16:00 | |
| spatel | i found issue and it was very stupid :( somehow my kickstart file got messed up and it created /var filesystem with only 150G, that is why i was not able to create more instance. | 16:00 |
| *** gyee has joined #openstack-nova | 16:06 | |
| *** udesale has quit IRC | 16:07 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Convert nova-lvm job to zuul v3 and modernize https://review.opendev.org/674345 | 16:13 |
| *** ash2307 has joined #openstack-nova | 16:14 | |
| *** pcaruana has quit IRC | 16:14 | |
| openstackgerrit | Merged openstack/nova master: Add method 'all_required_traits' to scheduler utils https://review.opendev.org/670297 | 16:20 |
| *** tbachman has joined #openstack-nova | 16:20 | |
| *** ociuhandu has quit IRC | 16:21 | |
| *** ociuhandu has joined #openstack-nova | 16:22 | |
| *** ociuhandu has quit IRC | 16:26 | |
| *** ociuhandu has joined #openstack-nova | 16:27 | |
| *** altlogbot_2 has quit IRC | 16:29 | |
| *** irclogbot_2 has quit IRC | 16:33 | |
| *** altlogbot_3 has joined #openstack-nova | 16:37 | |
| *** irclogbot_0 has joined #openstack-nova | 16:42 | |
| *** rpittau is now known as rpittau|afk | 16:46 | |
| *** epoojad has joined #openstack-nova | 16:47 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Don't mock something globally https://review.opendev.org/674351 | 16:48 |
| *** mlavalle has joined #openstack-nova | 16:52 | |
| *** ociuhandu has quit IRC | 16:57 | |
| *** ociuhandu has joined #openstack-nova | 16:57 | |
| *** tesseract has quit IRC | 16:58 | |
| *** derekh has quit IRC | 17:01 | |
| *** ociuhandu has quit IRC | 17:02 | |
| *** igordc has joined #openstack-nova | 17:03 | |
| *** jcosmao has left #openstack-nova | 17:03 | |
| *** ricolin has quit IRC | 17:07 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Convert nova-lvm job to zuul v3 https://review.opendev.org/674345 | 17:10 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Modernize nova-lvm job https://review.opendev.org/674356 | 17:13 |
| *** panda has quit IRC | 17:19 | |
| *** dpawlik has joined #openstack-nova | 17:19 | |
| *** panda has joined #openstack-nova | 17:20 | |
| *** openstackgerrit has quit IRC | 17:22 | |
| *** gyee has quit IRC | 17:24 | |
| *** ociuhandu has joined #openstack-nova | 17:26 | |
| aspiers | lyarwood: looks like a reboot loses devstack's /dev/loopN which are needed for the LVM volumes and devstack doesn't configure auto-setup on boot :-/ | 17:30 |
| *** priteau has quit IRC | 17:31 | |
| *** openstackgerrit has joined #openstack-nova | 17:31 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova-osprofiler-redis job to experimental queue https://review.opendev.org/674326 | 17:31 |
| *** dpawlik has quit IRC | 17:34 | |
| *** brault has quit IRC | 17:35 | |
| *** ksdean has quit IRC | 17:35 | |
| *** gyee has joined #openstack-nova | 17:36 | |
| openstackgerrit | Merged openstack/nova master: doc: correct the information of 'cpu_map' https://review.opendev.org/673272 | 17:38 |
| *** eharney has joined #openstack-nova | 17:41 | |
| *** dpawlik has joined #openstack-nova | 17:41 | |
| *** brault has joined #openstack-nova | 17:43 | |
| *** dpawlik has quit IRC | 17:45 | |
| *** brault has quit IRC | 17:48 | |
| *** ociuhandu has quit IRC | 17:49 | |
| *** ociuhandu has joined #openstack-nova | 17:50 | |
| *** ociuhandu has quit IRC | 17:50 | |
| *** ralonsoh has quit IRC | 17:51 | |
| eandersson | Anyone seen this error when cold migrating (Rocky) http://paste.openstack.org/show/755342/ | 18:00 |
| eandersson | > ValueError: Circular reference detected | 18:00 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: hardware: Differentiate between shared and dedicated CPUs https://review.opendev.org/671800 | 18:01 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support translating CPU policy extra specs, image meta https://review.opendev.org/671801 | 18:01 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Rename 'fields' import to 'obj_fields' https://review.opendev.org/674103 | 18:01 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Add proxy fields to NUMATopology object https://review.opendev.org/674104 | 18:01 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Rewrap definitions of 'NUMACell' https://review.opendev.org/674395 | 18:01 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Remove unnecessary argument https://review.opendev.org/674396 | 18:01 |
| mriedem | efried: looks like we need a warnings filter to avoid blowing up the console logs in functional test runs https://logs.opendev.org/61/673961/1/gate/nova-tox-functional-py36/a4cb2af/job-output.txt.gz#_2019-08-01_18_05_40_322077 | 18:09 |
| mriedem | b'/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/openstack/service_description.py:224: UnsupportedServiceVersion: Service placement has no discoverable version. The resulting Proxy object will only have direct passthrough REST capabilities.' | 18:10 |
| mriedem | getting all sorts of warnings from openstacksdk | 18:10 |
| mriedem | that or we actually have something to fix | 18:10 |
| mriedem | i'll push a patch to the sdk to fix this one https://github.com/openstack/openstacksdk/blob/master/openstack/config/cloud_region.py#L173 | 18:13 |
| mriedem | https://bugs.launchpad.net/nova/+bug/1838807 | 18:18 |
| openstack | Launchpad bug 1838807 in OpenStack Compute (nova) "UnsupportedServiceVersion: Service placement has no discoverable version. The resulting Proxy object will only have direct passthrough REST capabilities." [Medium,Confirmed] | 18:18 |
| eandersson | mriedem, you seen this before? http://paste.openstack.org/show/755342/ | 18:37 |
| mriedem | hmm, i've seen something like that in the past related to versioned notification payloads but that was fixed a long time ago | 18:39 |
| mriedem | https://github.com/openstack/nova/commit/3bf177a59cfd0b4e74dba256c3466ba2ea9bfbf7#diff-f009347572c3f7457d296a3b652002b5 | 18:40 |
| mriedem | so i wonder if there is a context in whatever you're seeing get serialized and that's blowing things up | 18:40 |
| eandersson | We for sure have that patch - this is stable/rocky (~5 commits behind) | 18:40 |
| mriedem | actually hmm | 18:41 |
| mriedem | legacy_spec = jsonutils.loads(jsonutils.dumps(legacy_spec)) | 18:41 |
| mriedem | i think there is a comment on that line about needing to do that to avoid a valuerror | 18:41 |
| mriedem | https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/tasks/migrate.py#L291 | 18:42 |
| eandersson | > legacy_props.pop('context', None) | 18:42 |
| eandersson | maybe we need to do the same on both? | 18:42 |
| mriedem | legacy_spec should be a dict form of the RequestSpec object w/o a context in it | 18:42 |
| mriedem | the dance done there is b/c of datetime fields afaict | 18:43 |
| mriedem | er per the comment from bauzas | 18:43 |
| mriedem | i also recently investigated this b/c i wanted to remove that... | 18:43 |
| mriedem | yeah https://review.opendev.org/#/c/582417/ | 18:44 |
| eandersson | Would that be possible to backport? | 18:46 |
| mriedem | no it has an rpc change | 18:46 |
| mriedem | but i'm not sure it would help you anyway | 18:46 |
| mriedem | note that we're still serializing/deserializing the ImageMeta object https://review.opendev.org/#/c/582417/7/nova/compute/rpcapi.py@741 | 18:46 |
| mriedem | i think, but don't know why, that is only in here for unit/functional tests that rely on the oslo.messaging rpc fake driver, but it doesn't make much sense to me because our functional tests pass things with datetimes over rpc all the time, | 18:47 |
| mriedem | like passing RequestSpec to the scheduler's select_destinations call | 18:47 |
| *** brault has joined #openstack-nova | 18:48 | |
| mriedem | do you have any out of tree changes to RequestSpec? | 18:49 |
| eandersson | Not that I am aware of | 18:49 |
| mriedem | is this an isolated incident? | 18:49 |
| eandersson | The only changes we do have are customer scheduler filters and weighers | 18:49 |
| eandersson | isolated so far yea | 18:49 |
| eandersson | but we haven't done a lot of migrations | 18:49 |
| eandersson | and this happened during like test run 3 | 18:50 |
| mriedem | hmm, ok so this is a cold migratoin (or resize) that is failing on one host and trying to reschedule to another and then failing when trying to serialize the legacy request spec b/c rpc casting to prep_resize on the alternative host | 18:50 |
| eandersson | *customer not customer | 18:50 |
| mriedem | the difference is probably in the reschedule | 18:50 |
| eandersson | *custom | 18:50 |
| eandersson | man | 18:50 |
| *** brault has quit IRC | 18:52 | |
| mriedem | i'll poke through that flow, we don't really hit migration reschedules in tempest integration runs unless it's in a multinode job | 18:53 |
| mriedem | and not intentionally | 18:53 |
| eandersson | Does it use placement for scheduling migrations? | 18:53 |
| eandersson | Due to how this fleet is set up it is a little too likely to hit Rescheduling errors at the moment in general. | 18:54 |
| eandersson | We heavily favor stacking VMs, plus Placement isn't properly configured, as we still rely on Aggregates for over-commit etc. | 18:55 |
| *** spatel has quit IRC | 18:55 | |
| eandersson | One of the downsides with doing a Mitaka to Rocky jump is that you miss things in releasenotes. :p | 18:56 |
| mriedem | placement is used for scheduling, | 18:56 |
| mriedem | so yes, the scheduler is going to ask placement for candidate hosts and the scheduler will pick a primary host after weighing and some alternates | 18:56 |
| mriedem | and those alternates are what get passed between conductor and compute during reschedules | 18:57 |
| mriedem | so after the initial scheduling request placement isn't really in the picture except for cleaning up allocations once the resize is confirmed/reverted | 18:57 |
| *** yaawang has joined #openstack-nova | 18:57 | |
| mriedem | i mean, i don't know what is triggering your reschedule - that should be clear from the compute logs, | 18:57 |
| eandersson | I have a good idea. | 18:57 |
| mriedem | but the valueerror thing is from something with the RequestSpec which doesn't really have anything to do with placement | 18:57 |
| mriedem | we have functional tests that trigger resize reschedules but they use fake rpc drivers and serialization fixtures for the context so those might be masking something, | 19:00 |
| mriedem | i'd have to stand up a 2 node devstack and hack it up to recreate | 19:00 |
| mriedem | if you could recreate, it would be good to dump that thing that gets serialized before it blows up | 19:01 |
| eandersson | Yep - I'll see if we can catch it | 19:01 |
| eandersson | Maybe even in the lab. | 19:01 |
| mriedem | i'm not sure if i can get something setup to recreate in the next hour | 19:01 |
| eandersson | I am also gonna re-run the tests in a env with a more optimal placement config. | 19:02 |
| mriedem | are there any special image metadata properties on the server being migrated? | 19:03 |
| mriedem | numa_topology, pci_requests? | 19:03 |
| eandersson | numa, nothing sepcial with pci | 19:05 |
| eandersson | I don't think we have anything special in image metadata, will need to double check | 19:06 |
| mriedem | hmm, the numa_topology object gets serialized in that thing as well, and that's likely not something we have in-tree functional tests for (instance + numa + resize reschedule) | 19:06 |
| mriedem | looking at the rocky code, on the first host attempt, conductor will serialize the request spec and pass it to compute, but compute won't pass it back to conductor on reschedule, so conductor has to build a new one https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/manager.py#L304 | 19:09 |
| mriedem | which will again get serialized and deserialized to dict form when calling prep_resize for the alternate host | 19:09 |
| mriedem | but the point being in rocky the request used for the first host isn't necessarily the same as the one for the alternate which could mean....something | 19:09 |
| *** portdirect has quit IRC | 19:13 | |
| *** portdirect has joined #openstack-nova | 19:13 | |
| *** yaawang has quit IRC | 19:25 | |
| donnyd | is max_concurernt_builds per hypervisor? | 19:25 |
| donnyd | for instance if i want to have only 2 instances per hypervisor schedule at any one point in time I would set that in nova-compute under max_concurernt_builds correct? | 19:28 |
| *** artom has quit IRC | 19:35 | |
| *** mgoddard has quit IRC | 19:37 | |
| *** dpawlik has joined #openstack-nova | 19:37 | |
| *** rtjure has quit IRC | 19:38 | |
| *** mgoddard has joined #openstack-nova | 19:38 | |
| *** irclogbot_0 has quit IRC | 19:39 | |
| eandersson | donnyd, yes | 19:40 |
| eandersson | https://github.com/openstack/nova/blob/master/nova/conf/compute.py#L610 | 19:40 |
| *** irclogbot_1 has joined #openstack-nova | 19:43 | |
| donnyd | thats what i thought, just confirming | 19:43 |
| donnyd | thanks eandersson | 19:43 |
| mriedem | cdent: looking at osprofiler data payed off https://bugs.launchpad.net/nova/+bug/1838817 | 19:48 |
| openstack | Launchpad bug 1838817 in OpenStack Compute (nova) "neutron: refreshing vif model for a server gets the same network multiple times" [Medium,Triaged] - Assigned to Matt Riedemann (mriedem) | 19:48 |
| *** slaweq has quit IRC | 19:55 | |
| *** whoami-rajat has quit IRC | 20:04 | |
| eandersson | mriedem, we had to set heal_instance_info_cache high because it was killing our control plane | 20:04 |
| eandersson | Is that why? :D | 20:04 |
| mriedem | that's not the only thing, | 20:04 |
| mriedem | but yeah it's expensive b/c building the instance nw info model means calling neutron separately for ports, networks, ips and subnets | 20:05 |
| mriedem | since there isn't a batch api to get all of that information | 20:05 |
| eandersson | It was getting real heavy on large sites with 1k nodes | 20:05 |
| mriedem | yeah b/c each node is hitting that refresh every minute | 20:06 |
| eandersson | We also ended up adding a variance | 20:06 |
| * mriedem remembers thinking many times how we should have a "performance tuning things at scale" | 20:06 | |
| mriedem | a doc | 20:06 |
| eandersson | Yea - let me know we have a lot of custom settings | 20:07 |
| mriedem | as does cern and others | 20:07 |
| mriedem | this kind of stuff comes up and we don't have a place to dump it really in the docs | 20:07 |
| eandersson | Yep | 20:07 |
| mriedem | similar with the long_rpc_timeout option | 20:07 |
| eandersson | Yea - we have it set to 300 :D | 20:07 |
| eandersson | 1k for dhcp agent lol | 20:07 |
| eandersson | (due to this bug https://github.com/openstack/neutron/commit/f9f6ae9c98a9ff9d3c0807b9fd51fb2ffe25fd5d) | 20:08 |
| mriedem | which thing do you have set to 300? | 20:09 |
| mriedem | heal instance info cache interval? | 20:09 |
| eandersson | 240 + variance on heal instance infop | 20:10 |
| eandersson | rcp_timeout is set to 300, but that could have been a mitaka thing | 20:10 |
| mriedem | rpc_response_timeout is the old one that defaults to 60 | 20:11 |
| mriedem | rpc_long_timeout is new as of i think rocky and is only used for certain long rpc calls | 20:11 |
| mriedem | like during live migratoin | 20:11 |
| eandersson | ah I see - didn't know about that one at all | 20:11 |
| mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.long_rpc_timeout | 20:11 |
| eandersson | Probably another thing we missed when jumping to so many versions. | 20:12 |
| eandersson | ah yea that fixes our problem | 20:12 |
| eandersson | We set it to 300 due to scheduling | 20:12 |
| mriedem | yup, that's using long_rpc_timeout now (select_destinations) | 20:12 |
| mriedem | long_rpc_timeout heartbeats the service every $rpc_response_timeout but the overall call timeout is now long_rpc_timeout | 20:13 |
| mriedem | i opened https://bugs.launchpad.net/nova/+bug/1838819 if you want to dump some stuff in there | 20:13 |
| openstack | Launchpad bug 1838819 in OpenStack Compute (nova) "Docs needed for tunables at large scale" [Undecided,New] | 20:13 |
| eandersson | It would probably be worth to get together during the next summit as well and talk about this (or maybe just a irc meeting) | 20:13 |
| eandersson | I tend to reach out to CERN a lot since they are running similar scale | 20:14 |
| mriedem | even dumping on the ML - lots of people won't be in shanghai | 20:15 |
| eandersson | Yea - including myself | 20:15 |
| mriedem | i'll throw that in hte ML quick | 20:15 |
| eandersson | Not sure why but I don't like emails | 20:15 |
| eandersson | Maybe I'll just join with my private email | 20:15 |
| mriedem | fire and forget... | 20:16 |
| mriedem | i'm ramping down to leave town for the weekend so i'm trying to fire things off | 20:16 |
| eandersson | Same | 20:17 |
| eandersson | It's why I am deep diving into some of the issues. | 20:17 |
| *** dpawlik has quit IRC | 20:19 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: neutron: avoid redundant show_network calls in _get_physnet_tunneled_info https://review.opendev.org/674422 | 20:32 |
| *** rtjure has joined #openstack-nova | 20:33 | |
| *** mriedem has quit IRC | 20:33 | |
| *** tbachman has quit IRC | 20:37 | |
| *** markvoelker has quit IRC | 20:42 | |
| *** ivve has quit IRC | 20:43 | |
| *** tbachman has joined #openstack-nova | 20:45 | |
| *** markvoelker has joined #openstack-nova | 20:52 | |
| *** markvoelker has quit IRC | 20:56 | |
| *** BjoernT_ has quit IRC | 21:09 | |
| *** tbachman has quit IRC | 21:09 | |
| *** slaweq has joined #openstack-nova | 21:11 | |
| *** tbachman has joined #openstack-nova | 21:12 | |
| *** slaweq has quit IRC | 21:15 | |
| *** _erlon_ has joined #openstack-nova | 21:15 | |
| *** jdillaman1 has quit IRC | 21:28 | |
| *** markvoelker has joined #openstack-nova | 21:48 | |
| *** betherly has joined #openstack-nova | 21:55 | |
| *** betherly has quit IRC | 22:01 | |
| *** jistr has quit IRC | 22:02 | |
| *** jistr has joined #openstack-nova | 22:09 | |
| *** slaweq has joined #openstack-nova | 22:11 | |
| *** slaweq has quit IRC | 22:15 | |
| *** jistr has quit IRC | 22:17 | |
| *** jistr has joined #openstack-nova | 22:19 | |
| *** eharney has quit IRC | 22:20 | |
| *** markvoelker has quit IRC | 22:21 | |
| *** sapd1 has quit IRC | 22:26 | |
| *** sapd1 has joined #openstack-nova | 22:26 | |
| *** sapd1 has quit IRC | 22:27 | |
| *** slaweq has joined #openstack-nova | 23:11 | |
| *** slaweq has quit IRC | 23:16 | |
| *** N3l1x has quit IRC | 23:17 | |
| *** markvoelker has joined #openstack-nova | 23:20 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Always set a global_request_id in RequestContext https://review.opendev.org/674138 | 23:21 |
| openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Always set a global_request_id in RequestContext https://review.opendev.org/674138 | 23:22 |
| *** _erlon_ has quit IRC | 23:25 | |
| *** betherly has joined #openstack-nova | 23:26 | |
| openstackgerrit | Merged openstack/os-resource-classes master: Sync Sphinx requirement https://review.opendev.org/674164 | 23:27 |
| *** betherly has quit IRC | 23:31 | |
| *** betherly has joined #openstack-nova | 23:47 | |
| *** betherly has quit IRC | 23:51 | |
| *** markvoelker has quit IRC | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!