*** mmethot has joined #openstack-nova | 00:40 | |
*** ccamacho has quit IRC | 00:42 | |
*** markvoelker has joined #openstack-nova | 00:46 | |
*** markvoelker has quit IRC | 00:51 | |
*** xek has quit IRC | 01:13 | |
*** bbowen has joined #openstack-nova | 01:16 | |
*** threestrands has joined #openstack-nova | 01:37 | |
*** hongbin has joined #openstack-nova | 01:44 | |
*** larainema has joined #openstack-nova | 02:12 | |
openstackgerrit | Takashi NATSUME proposed openstack/nova stable/stein: Remove descriptions of nonexistent hacking rules https://review.opendev.org/679695 | 02:15 |
---|---|---|
*** gbarros has quit IRC | 02:16 | |
openstackgerrit | Brin Zhang proposed openstack/nova master: Add delete_on_termination to volume-attach API https://review.opendev.org/673133 | 02:18 |
*** Tianhao_Hu has joined #openstack-nova | 02:23 | |
*** takashin has joined #openstack-nova | 02:26 | |
*** Tianhao_Hu has left #openstack-nova | 02:35 | |
*** mtanino has joined #openstack-nova | 02:36 | |
*** gbarros has joined #openstack-nova | 02:39 | |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: Use Claims to update numa-related XML on the source https://review.opendev.org/635229 | 02:42 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | 02:42 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | 02:42 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: DNM: extra logging https://review.opendev.org/679681 | 02:42 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | 02:42 |
*** gbarros has quit IRC | 03:22 | |
*** mkrai has joined #openstack-nova | 03:23 | |
*** hongbin has quit IRC | 03:26 | |
*** hongbin has joined #openstack-nova | 03:28 | |
*** sapd1_x has joined #openstack-nova | 03:37 | |
*** mtanino has quit IRC | 03:45 | |
*** mkrai has quit IRC | 03:48 | |
*** mkrai has joined #openstack-nova | 03:49 | |
openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Add a check for --config-drive option on nova boot https://review.opendev.org/653683 | 03:51 |
*** takashin has left #openstack-nova | 03:53 | |
*** mvkr has joined #openstack-nova | 04:14 | |
*** jaosorior has quit IRC | 04:24 | |
*** jaosorior has joined #openstack-nova | 04:25 | |
*** hongbin has quit IRC | 04:29 | |
*** hongbin has joined #openstack-nova | 04:49 | |
*** hongbin has quit IRC | 04:49 | |
*** Luzi has joined #openstack-nova | 04:59 | |
*** udesale has joined #openstack-nova | 04:59 | |
*** mkrai has quit IRC | 05:03 | |
*** mkrai has joined #openstack-nova | 05:03 | |
*** Garyx has quit IRC | 05:07 | |
*** Garyx has joined #openstack-nova | 05:09 | |
*** ratailor has joined #openstack-nova | 05:21 | |
*** jaosorior has quit IRC | 05:22 | |
*** sapd1_x has quit IRC | 05:30 | |
*** ivve has joined #openstack-nova | 05:35 | |
*** ratailor_ has joined #openstack-nova | 05:37 | |
*** prometheanfire has quit IRC | 05:39 | |
*** ratailor has quit IRC | 05:40 | |
*** ratailor__ has joined #openstack-nova | 05:40 | |
*** ratailor_ has quit IRC | 05:43 | |
*** shilpasd has joined #openstack-nova | 06:03 | |
*** xek has joined #openstack-nova | 06:11 | |
openstackgerrit | Luyao Zhong proposed openstack/nova master: db: Add resources column in instance_extra table https://review.opendev.org/678447 | 06:21 |
openstackgerrit | Luyao Zhong proposed openstack/nova master: object: Introduce Resource and ResouceList objs https://review.opendev.org/678448 | 06:21 |
openstackgerrit | Luyao Zhong proposed openstack/nova master: Add resources dict into _Provider https://review.opendev.org/678449 | 06:21 |
openstackgerrit | Luyao Zhong proposed openstack/nova master: Retrieve the allocations early https://review.opendev.org/678450 | 06:21 |
openstackgerrit | Luyao Zhong proposed openstack/nova master: Claim resources in resource tracker https://review.opendev.org/678452 | 06:21 |
openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver discovering PMEM namespaces https://review.opendev.org/678453 | 06:21 |
openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: report VPMEM resources by provider tree https://review.opendev.org/678454 | 06:21 |
openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Support VM creation with vpmems and vpmems cleanup https://review.opendev.org/678455 | 06:21 |
openstackgerrit | Luyao Zhong proposed openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456 | 06:21 |
openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/679640 | 06:21 |
openstackgerrit | Luyao Zhong proposed openstack/nova master: Add functional tests for virtual persistent memory https://review.opendev.org/678470 | 06:21 |
*** sapd1_x has joined #openstack-nova | 06:22 | |
*** mkrai has quit IRC | 06:31 | |
*** sapd1_x has quit IRC | 06:32 | |
*** ccamacho has joined #openstack-nova | 06:33 | |
*** jaosorior has joined #openstack-nova | 06:39 | |
*** mkrai has joined #openstack-nova | 06:39 | |
*** aloga has joined #openstack-nova | 06:55 | |
*** tridde has quit IRC | 07:00 | |
*** pcaruana has joined #openstack-nova | 07:06 | |
*** tesseract has joined #openstack-nova | 07:06 | |
*** trident has joined #openstack-nova | 07:09 | |
*** slaweq has joined #openstack-nova | 07:09 | |
*** damien_r has joined #openstack-nova | 07:15 | |
*** damien_r has left #openstack-nova | 07:17 | |
*** damien_r has joined #openstack-nova | 07:18 | |
*** threestrands has quit IRC | 07:18 | |
*** rcernin has quit IRC | 07:23 | |
*** kaisers has joined #openstack-nova | 07:24 | |
*** luksky has joined #openstack-nova | 07:40 | |
*** maciejjozefczyk has joined #openstack-nova | 07:41 | |
*** priteau has joined #openstack-nova | 07:47 | |
*** tkajinam has quit IRC | 08:02 | |
*** ratailor has joined #openstack-nova | 08:05 | |
*** ratailor__ has quit IRC | 08:07 | |
*** xek has quit IRC | 08:10 | |
*** xek has joined #openstack-nova | 08:10 | |
*** ociuhandu has joined #openstack-nova | 08:14 | |
*** jaosorior has quit IRC | 08:29 | |
stephenfin | alex_xu: Could you look at https://review.opendev.org/#/c/678861/ and https://review.opendev.org/#/c/678902/ ? | 08:32 |
*** ralonsoh has joined #openstack-nova | 08:37 | |
*** derekh has joined #openstack-nova | 08:38 | |
*** tesseract has quit IRC | 08:40 | |
*** ociuhandu has quit IRC | 08:43 | |
*** jaosorior has joined #openstack-nova | 08:59 | |
*** rcernin has joined #openstack-nova | 08:59 | |
*** avolkov has joined #openstack-nova | 09:06 | |
*** kaliya has joined #openstack-nova | 09:08 | |
*** kaliya has quit IRC | 09:09 | |
yaawang | stephenfin: Hi, cloud you review https://review.opendev.org/#/c/670298/ and https://review.opendev.org/#/c/670299/ and https://review.opendev.org/#/c/670300/ :) | 09:11 |
*** cdent has joined #openstack-nova | 09:17 | |
*** slaweq has quit IRC | 09:22 | |
*** slaweq has joined #openstack-nova | 09:23 | |
*** brinzhang_ has joined #openstack-nova | 09:24 | |
*** brinzhang_ has quit IRC | 09:25 | |
*** brinzhang_ has joined #openstack-nova | 09:25 | |
*** brinzhang has quit IRC | 09:27 | |
*** dtantsur|afk is now known as dtantsur | 09:42 | |
*** tkajinam has joined #openstack-nova | 09:48 | |
*** ianw has quit IRC | 10:00 | |
*** jaosorior has quit IRC | 10:05 | |
*** ianw has joined #openstack-nova | 10:08 | |
*** artom has joined #openstack-nova | 10:13 | |
luyao | stephenfin: Could you help review the rest of the patch series if you have time? :) https://review.opendev.org/#/q/topic:bp/virtual-persistent-memory+(status:open+) | 10:14 |
*** tkajinam has quit IRC | 10:18 | |
cdent | is 'nova-grenade-multinode' not working the latest "gate's not happy"? | 10:22 |
cdent | or is it just me | 10:23 |
*** ianw has quit IRC | 10:27 | |
gibi | cdent: could you link the failing job? | 10:27 |
cdent | gibi: https://review.opendev.org/#/c/679627/ https://zuul.opendev.org/t/openstack/build/af05b1eaef0047a6a2f2f466e93b5ac5 | 10:28 |
*** ianw has joined #openstack-nova | 10:28 | |
*** ianw has quit IRC | 10:28 | |
cdent | I've seen that few different times this morning, as I'm trying to catch up. haven't dug into it yet (still trying to catch up) | 10:29 |
*** ianw has joined #openstack-nova | 10:29 | |
openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Simplify 'fakelibvirt.HostInfo' object https://review.opendev.org/678861 | 10:30 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start checking compute usage in functional tests https://review.opendev.org/678902 | 10:30 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Rename 'fields' import to 'obj_fields' https://review.opendev.org/674103 | 10:30 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Fold in argument to '_update_provider_tree_for_vgpu' https://review.opendev.org/676729 | 10:30 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start reporting PCPU inventory to placement https://review.opendev.org/671793 | 10:30 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: '_get_(v|p)cpu_total' to '_get_(v|p)cpu_available' https://review.opendev.org/672693 | 10:30 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: hardware: Differentiate between shared and dedicated CPUs https://review.opendev.org/671800 | 10:30 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start reporting 'HW_CPU_HYPERTHREADING' trait https://review.opendev.org/675571 | 10:30 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta https://review.opendev.org/671801 | 10:30 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: Add reshaper for PCPU https://review.opendev.org/674895 | 10:30 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Always enable the NUMATopologyFilter https://review.opendev.org/679745 | 10:30 |
stephenfin | alex_xu: Added resize tests to https://review.opendev.org/671801 as promised | 10:32 |
gibi | cdent: did you found the exact failure in the logs? is it https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_27/679627/2/check/nova-grenade-multinode/af05b1e/logs/error.txt.gz ? or I | 10:32 |
gibi | or I'm on a wrong track | 10:32 |
*** ianw has quit IRC | 10:34 | |
*** ianw has joined #openstack-nova | 10:36 | |
cdent | gibi: yeah, that's the one I saw, while skimming the genade log | 10:37 |
*** ociuhandu has joined #openstack-nova | 10:37 | |
*** ianw_ has joined #openstack-nova | 10:41 | |
*** ianw_ has quit IRC | 10:41 | |
*** ianw_ has joined #openstack-nova | 10:43 | |
*** ianw_ has quit IRC | 10:44 | |
*** hoonetorg has quit IRC | 10:44 | |
*** ianw has quit IRC | 10:44 | |
openstackgerrit | Adam Spiers proposed openstack/nova master: Make _get_cpu_feature_traits() always return a dict https://review.opendev.org/679568 | 10:44 |
*** ianw has joined #openstack-nova | 10:44 | |
*** xek has quit IRC | 10:45 | |
*** xek has joined #openstack-nova | 10:45 | |
*** bbowen has quit IRC | 10:46 | |
*** ianw has quit IRC | 10:46 | |
*** ianw has joined #openstack-nova | 10:50 | |
*** cdent has quit IRC | 10:50 | |
*** tbachman has quit IRC | 10:51 | |
*** ianw has quit IRC | 10:52 | |
*** ianw has joined #openstack-nova | 10:52 | |
*** hoonetorg has joined #openstack-nova | 10:57 | |
*** HagunKim has quit IRC | 10:58 | |
*** dave-mccowan has joined #openstack-nova | 11:00 | |
*** dave-mccowan has quit IRC | 11:06 | |
*** luksky has quit IRC | 11:08 | |
*** markvoelker has joined #openstack-nova | 11:09 | |
openstackgerrit | sean mooney proposed openstack/nova master: multi numa nfv testing job https://review.opendev.org/679656 | 11:13 |
openstackgerrit | sean mooney proposed openstack/nova master: [DNM] test migration with pinning https://review.opendev.org/679754 | 11:13 |
*** markvoelker has quit IRC | 11:14 | |
sean-k-mooney | artom: ^ the second patch should get the pinning only case and i also am testign we can spawn with explcit small pages via the alternitive flavor | 11:15 |
sean-k-mooney | we are not migrating with the alternive flavor but i guess i could add a third patch to do that | 11:16 |
sean-k-mooney | e.g. hugepage+pinning+multi-numa+realtime | 11:16 |
sean-k-mooney | im going to test that config manually now however | 11:17 |
sean-k-mooney | artom: we might also want to invert the depens on relationship in a futre version so your change depnd on the zuul job instead of the other way aroudn | 11:19 |
*** nicolasbock has joined #openstack-nova | 11:19 | |
*** xek has quit IRC | 11:20 | |
artom | sean-k-mooney, we're not migrating with the alt flavor? That sucks, it was kinda the whole point | 11:20 |
sean-k-mooney | we migrate with the standard flaovr | 11:20 |
sean-k-mooney | we resize to the alt flavor | 11:20 |
sean-k-mooney | the standard flavor is multi-numa-128-1-2 | 11:21 |
sean-k-mooney | e.g. hugepages and 2 numa nodes | 11:21 |
artom | Oh | 11:21 |
artom | https://review.opendev.org/#/c/679754/1/playbooks/nfv/nfv.yaml@14 is fine anyways | 11:21 |
sean-k-mooney | the alt flaovr is the one with pinning and realtime + hugepages | 11:21 |
artom | It's what I had in mind | 11:21 |
sean-k-mooney | im using the secondary pinned-128-1-2 flavor to do some other testing as a side effect of the job | 11:22 |
sean-k-mooney | its unrelated to live migration | 11:22 |
sean-k-mooney | i just want to make sure that we can trun on all the things | 11:22 |
sean-k-mooney | which is chacked by the resize tests | 11:22 |
*** ociuhandu has quit IRC | 11:23 | |
*** francoisp has quit IRC | 11:25 | |
*** cdent has joined #openstack-nova | 11:26 | |
gibi | cdent: the bug with grenade + cinder seems to be already reported an tracked http://status.openstack.org/elastic-recheck/#1840355 | 11:28 |
cdent | the error message is the best ever... | 11:28 |
cdent | yes, yes it is | 11:28 |
sean-k-mooney | i mean its not wrong | 11:30 |
*** udesale has quit IRC | 11:37 | |
*** Garyx has quit IRC | 11:38 | |
*** Garyx has joined #openstack-nova | 11:41 | |
*** Garyx has quit IRC | 11:41 | |
*** Garyx has joined #openstack-nova | 11:41 | |
*** ociuhandu has joined #openstack-nova | 11:43 | |
yonglihe | stephenfin: Comments addressed. and thanks. https://review.opendev.org/#/c/621476/ | 11:43 |
*** xek has joined #openstack-nova | 11:48 | |
*** ociuhandu has quit IRC | 11:48 | |
*** jaosorior has joined #openstack-nova | 11:51 | |
*** udesale has joined #openstack-nova | 11:51 | |
*** bbowen has joined #openstack-nova | 11:51 | |
*** ricolin_ has joined #openstack-nova | 11:51 | |
*** ociuhandu has joined #openstack-nova | 11:52 | |
*** ricolin has quit IRC | 11:54 | |
*** bbowen_ has joined #openstack-nova | 11:54 | |
*** bbowen has quit IRC | 11:56 | |
*** dtantsur is now known as dtantsur|brb | 11:57 | |
sean-k-mooney | yaawang: the vcpu model selection series sure. im just grabing lunch but ill review it after | 12:01 |
yaawang | sean-k-mooney: thk :) | 12:14 |
*** larainema has quit IRC | 12:15 | |
*** markvoelker has joined #openstack-nova | 12:18 | |
*** brtknr has joined #openstack-nova | 12:19 | |
brtknr | Hi all, any users with experience using fedora-coreos on openstack? | 12:19 |
brtknr | do I need to explicitly inject ssh key to the instance? | 12:20 |
alex_xu | stephenfin: cool | 12:25 |
*** jistr has quit IRC | 12:29 | |
*** jistr has joined #openstack-nova | 12:29 | |
*** jistr has quit IRC | 12:30 | |
*** ociuhandu has quit IRC | 12:30 | |
*** ociuhandu has joined #openstack-nova | 12:30 | |
*** ociuhandu has quit IRC | 12:31 | |
*** ociuhandu_ has joined #openstack-nova | 12:31 | |
*** ociuhandu has joined #openstack-nova | 12:32 | |
*** weshay has joined #openstack-nova | 12:34 | |
*** tbachman has joined #openstack-nova | 12:35 | |
stephenfin | bauzas: Could you look at https://review.opendev.org/#/c/679745 ? | 12:36 |
*** ociuhandu_ has quit IRC | 12:36 | |
sean-k-mooney | brtknr: fedora core os does not have any password by defualt so if you dont inject ssh keys by selecting a key pair you wont be able to log in | 12:36 |
brtknr | sean-k-mooney: I selected a keypair when I created my instance but it is not letting me in as core user | 12:37 |
openstackgerrit | Silvan Kaiser proposed openstack/nova stable/stein: Exec systemd-run without --user flag in Quobyte driver https://review.opendev.org/660705 | 12:37 |
*** bbowen_ has quit IRC | 12:37 | |
*** bbowen has joined #openstack-nova | 12:40 | |
*** jistr has joined #openstack-nova | 12:40 | |
brtknr | sean-k-mooney: default user is core correct? | 12:40 |
sean-k-mooney | its would be core or fedora i suspect. i dont actully use fedora core os but i knwo that the cloud images do not have a password set | 12:41 |
openstackgerrit | Bhagyashri Shewale proposed openstack/nova master: Ignore root_gb for BFV in simple tenant usage API https://review.opendev.org/612626 | 12:44 |
*** cdent has quit IRC | 12:50 | |
*** nweinber has joined #openstack-nova | 12:52 | |
bauzas | stephenfin: sure | 12:54 |
*** redrobot has quit IRC | 12:55 | |
*** david-lyle has quit IRC | 13:02 | |
*** dklyle has joined #openstack-nova | 13:02 | |
*** pcaruana has quit IRC | 13:03 | |
*** bhagyashris has joined #openstack-nova | 13:07 | |
*** cdent has joined #openstack-nova | 13:15 | |
*** ratailor has quit IRC | 13:15 | |
*** luksky has joined #openstack-nova | 13:16 | |
*** ociuhandu has quit IRC | 13:21 | |
*** ociuhandu has joined #openstack-nova | 13:22 | |
sean-k-mooney | artom: Live Migration failure: 'NoneType' object has no attribute 'set' | 13:22 |
*** xek has quit IRC | 13:26 | |
sean-k-mooney | ill replicate this in the ci job but when the souce host only hase cpu enabled in the first numa node and the dest only in the second numa node it failes with that error | 13:27 |
*** tbachman has quit IRC | 13:27 | |
artom | sean-k-mooney, yeah, that's what I expected | 13:27 |
artom | sean-k-mooney, it's fixed in a local version, need to finish unit tests and push | 13:27 |
sean-k-mooney | ok im reconfriguring my vcpu_pin_set | 13:28 |
sean-k-mooney | and ill continue testing each feature independtly | 13:28 |
*** ociuhandu has quit IRC | 13:28 | |
*** ociuhandu has joined #openstack-nova | 13:29 | |
donnyd | sean-k-mooney: Any issues with NUMA I can help with today? | 13:29 |
*** mriedem has joined #openstack-nova | 13:29 | |
sean-k-mooney | donnyd: nope your ci is working fine. we are now testing different edgecases | 13:30 |
*** Garyx has quit IRC | 13:30 | |
sean-k-mooney | donnyd: i might add some new lables in the future e.g. multi-numa and nested-virt so that they are more expclit | 13:31 |
sean-k-mooney | donnyd: if you want to replicate the flaovr to a new on i can swap it over to use that | 13:31 |
donnyd | At some point we should probably create some sensical labels | 13:31 |
sean-k-mooney | yep | 13:32 |
*** tbachman has joined #openstack-nova | 13:32 | |
sean-k-mooney | if you can copy the flaovr you create and give it a new name i can add a multi-numa-ubuntu-bionic lable that uses it | 13:33 |
* cdent nominates stephenfin for nova ptl | 13:33 | |
donnyd | Ok, so you want to handle the project config end and I can do the flavor side? correct? | 13:33 |
sean-k-mooney | well i dont mind if you do the porject config stuff but im happy to do it too | 13:34 |
*** pcaruana has joined #openstack-nova | 13:34 | |
stephenfin | cdent: no. | 13:34 |
donnyd | I can take care of it np | 13:34 |
sean-k-mooney | cdent: he is obvioulsy the next oslo ptl :P | 13:34 |
cdent | maybe we should just dissolve things then :) | 13:35 |
artom | Did efried desist or something? | 13:35 |
sean-k-mooney | artom: no he just has not renominated himself | 13:35 |
sean-k-mooney | he has like 9 hours left | 13:35 |
sean-k-mooney | no one has nominated for nova | 13:35 |
efried | what what? | 13:36 |
* efried has just arrived | 13:36 | |
artom | We're grown ups, we can be trusted without an adult, right? | 13:36 |
*** redrobot_ has joined #openstack-nova | 13:36 | |
donnyd | 8cpu-8GBram-80GBdisk-multi-numa-nested-ubuntu-bionic | 13:36 |
donnyd | 8cpu-16GBram-80GBdisk-multi-numa-nested-ubuntu-bionic | 13:36 |
donnyd | so like those sean-k-mooney | 13:36 |
efried | artom: If I've learned anything from doing this, it's that a PTL is not useful for "adulting" the team. | 13:37 |
*** redrobot_ is now known as redrobot | 13:37 | |
efried | Run meetings, ack release patches, organize milestoney things, other paperwork. | 13:37 |
artom | efried, heh - from the outside looking in, and without wanting to sound pejorative, PTL, at least for nova, looks like a more... secretary-type? job | 13:38 |
artom | Perhaps admin/organizer would be a better word | 13:38 |
efried | just so | 13:38 |
sean-k-mooney | donnyd: they dont need to be that long be sure | 13:38 |
sean-k-mooney | donnyd: if thoes are teh flavor you created thats fine | 13:38 |
kashyap | Yeah, "secretary" undermines the actual (unthankful) work involved | 13:38 |
sean-k-mooney | but ill proably use the labels to hide some of the details | 13:38 |
donnyd | LOL - ok | 13:39 |
donnyd | 8c-8r-80d-multi-numa-nested-ubuntu-bionic | 13:39 |
donnyd | better? | 13:39 |
artom | kashyap, totally, I'm not downplaying the importance of the work | 13:39 |
sean-k-mooney | well i dont mind what the flaovr is called. | 13:39 |
kashyap | artom: Yeah, yeah. Just that I am a bit too OCD-ish (working on it) about words :D | 13:39 |
artom | Just saying that it's thankless, and not very technical (the T in PTL) from a software engineering/development POV | 13:39 |
donnyd | 8c-8r-80d-numa-nested-ubuntu-bionic | 13:39 |
sean-k-mooney | but the label i would prefer to jsut be muli-numa-ubuntu-bionic | 13:39 |
sean-k-mooney | and that can use the 8c-8r-80d-numa-nested-ubuntu-bionic flavor | 13:40 |
donnyd | well how would you tell the difference between a 16G label and a 8GB label | 13:40 |
sean-k-mooney | multi-numa-ubuntu-bionic and muli-numa-ubuntu-bionic-extend-memory | 13:40 |
cdent | donnyd: since you are here, I thought I would just mention "thank you, your VMs are _so_ much faster than the others" | 13:40 |
* donnyd accepts gratitude | 13:41 | |
efried | artom: that | 13:41 |
*** Luzi has quit IRC | 13:41 | |
sean-k-mooney | donnyd: lets submit a patch and work it out in the reivew | 13:41 |
donnyd | Ok, sounds great | 13:41 |
efried | I should perhaps remind those present that the PTL need not be a core. | 13:42 |
artom | PTL-less has happened before, what did other projects (or even Nova) do? | 13:42 |
kashyap | artom: Oh, totally, on the thanklessness. | 13:42 |
artom | For instance, running the meetings pretty much anyone can do, so just set up a rotation of volunteers | 13:42 |
artom | The release stuff is more finicky | 13:42 |
sean-k-mooney | efried: you sure | 13:42 |
sean-k-mooney | efried: i thought that was a requirement | 13:42 |
donnyd | I am also trying to come up with a way to each project team to have a play space, so they dont have to wait on a CI job.. but could do things on the same infra | 13:42 |
artom | I'd volunteer for the former (checks calendar... right, I can), not so much the latter | 13:42 |
artom | At least, not without some guidance | 13:43 |
kashyap | artom: On "PTL-less", try beating this dubious "record": | 13:43 |
efried | sean-k-mooney: not a requirement. I could go check, but I'm 98% sure. | 13:43 |
*** amrith has joined #openstack-nova | 13:43 | |
kashyap | artom: https://brussels-express.eu/fun-fact-belgium-owns-world-record-longest-period-without-government/ | 13:43 |
artom | (Actually, no, the Thursday afternoon meetings are 17:00 my time, can't make it) | 13:43 |
kashyap | [quote] | 13:44 |
kashyap | 541 days. That’s how long it took Belgian politicians to form an official government after the federal elections of June 13, 2010. It earned Belgium a Guinness World Record for going the longest time with no government | 13:44 |
kashyap | [/quote] | 13:44 |
kashyap | :D | 13:44 |
efried | artom: Delegating meetings would be totally nbd | 13:44 |
*** KeithMnemonic has joined #openstack-nova | 13:44 | |
efried | iirc, melwitt delegated the early ones - I think gibi ran those | 13:45 |
donnyd | Just make all the cores like knights of the round table and just roll responsibilities bi-weekly from core to core | 13:45 |
kashyap | NBD == No Big Deal (/me could only think of Network Block Device) | 13:45 |
*** brinzhang_ has quit IRC | 13:46 | |
gibi | efried, artom: I can chair the EU meetings if needed | 13:46 |
kashyap | gibi: LOL | 13:46 |
kashyap | Ah, wait - I misread it | 13:46 |
*** brinzhang_ has joined #openstack-nova | 13:46 | |
kashyap | gibi: You meant CET/CEST meetings; /me thought you were joking about the current political fiasco :D | 13:46 |
donnyd | kashyap: I was also wondering how networked disks became part of the conversation | 13:46 |
*** dtantsur|brb is now known as dtantsur | 13:46 | |
gibi | kashyap: sure, CET/CEST is more specific :) | 13:47 |
artom | Dudes, I dunno if that's what you're hinting at, but I'm not nominating myself for PTL. I'd feel like a total imposter. | 13:47 |
efried | artom: fake it til you make it. | 13:47 |
* efried <== imposter | 13:47 | |
*** pcaruana has quit IRC | 13:47 | |
sean-k-mooney | https://github.com/openstack/governance/blob/aa8e0ba19723269e1369e274128f801bb43e67c0/reference/charter.rst#candidates-for-ptl-seats i guess you are right | 13:47 |
cdent | artom: everyone is and always will be an imposter. anyone who says otherwise is not self-aware | 13:47 |
kashyap | artom: Yes, don't feel it; go for it, if you want to give it try. | 13:47 |
artom | Hardly | 13:47 |
kashyap | donnyd: I figured it out from a little tool called `wtf`: | 13:47 |
kashyap | $> rpm -qf `which wtf` | 13:47 |
kashyap | bsd-games-2.17-60.fc30.x86_64 | 13:47 |
artom | efried ^^ | 13:47 |
kashyap | donnyd: It's a tiny (less than 1MB package), and has a acryonym DB | 13:48 |
kashyap | So, I just type: | 13:48 |
kashyap | $> wtf is nbd | 13:48 |
kashyap | NBD: no big deal | 13:48 |
kashyap | no big deal | 13:48 |
artom | I'm also scared of commitment (says the man in an LTR with 2 kids) | 13:48 |
artom | (Man is a big word...) | 13:48 |
kashyap | artom: What is LTR? | 13:48 |
*** gbarros has joined #openstack-nova | 13:48 | |
artom | long term relationship | 13:49 |
kashyap | Aah, like that | 13:49 |
artom | I should clarify, I'm in a relationship *with my spouse* | 13:49 |
artom | Not with my kids - not in that sense, anyways | 13:49 |
efried | I was gonna say, two kids and LTR not married *is* fear of commitment. | 13:49 |
*** eharney has joined #openstack-nova | 13:49 | |
artom | Hah, maybe | 13:49 |
sean-k-mooney | artom: do you get the same tax break for beign in a long term relation and cohabitaing as you do when married | 13:50 |
*** tbachman has quit IRC | 13:51 | |
sean-k-mooney | i know in ireland you do but i dont think that is the case everywhere | 13:51 |
artom | sean-k-mooney, I think so - Quebec's pretty awesome for common law spouses that way | 13:51 |
artom | We file our taxes jointly and everything | 13:51 |
sean-k-mooney | ya you can do that in ireland too. although we dont have to file taxes unless we are self employed | 13:52 |
artom | Yeah, that stupidity we got from the US | 13:52 |
artom | Props up an entire fiscal accounting industry | 13:52 |
sean-k-mooney | PAYE (pay as you earn) make it way more simple | 13:53 |
bauzas | oh, it's surely not coming from the US | 13:53 |
sean-k-mooney | its not from the uk. we use the PAYE system because they did | 13:53 |
artom | sean-k-mooney, oh, we have PAYE | 13:54 |
donnyd | sean-k-mooney: How many OS do we want to support with NUMA? | 13:54 |
donnyd | ubuntu+centos | 13:54 |
artom | It's why there's a massive difference between my gross and net salary | 13:54 |
artom | But then there's a whole bunch of possible deductions after the fact | 13:54 |
artom | Registered retirement savings plans, daycare/school, etc etc | 13:54 |
artom | So they adjust at the end of the year | 13:55 |
artom | (well, April) | 13:55 |
sean-k-mooney | donnyd: ubunut is enough | 13:55 |
donnyd | kk | 13:55 |
sean-k-mooney | if you add centos thats fine too | 13:55 |
bauzas | artom: you wouldn't be surprised with our local tax filings then | 13:55 |
artom | bauzas, eh, I still think Quebec's is more complex | 13:55 |
sean-k-mooney | donnyd: we just need 1 flaovr on your end and we can create lables as we need them in project-config | 13:55 |
bauzas | artom: I personnally doubt that but okay | 13:56 |
artom | But... je fais mon bourgeois, I bring all the paperwork to an account, pay her some money, and if I pay her enough money, get some more money back from the govt after | 13:56 |
bauzas | yeah that's where US and Canadian folks are smarter than us | 13:56 |
sean-k-mooney | artom: ya we can file a p12 if we want to for that. but most things are handeled automatically | 13:57 |
bauzas | for us, we rather prefer to complain about how complicated our filings are | 13:57 |
artom | bauzas, speaking of complaining, I vividly remember a French woman in the local Ikea | 13:58 |
artom | You know that guttural sigh that you guys do? That "hhrrrrooooo" | 13:58 |
bauzas | and you don't imagine how much are forgetting to ask for a specific incentive they're allowed to | 13:58 |
artom | Dixit woman: "hrrroooo, qu'est ce c'est mal foutu" | 13:58 |
bauzas | LOL | 13:58 |
bhagyashris | mriedem: Hi, I would like your insights on https://review.opendev.org/#/c/612626/ | 13:59 |
bauzas | artom: I assume it's a very typical French person that's becoming rare : someone not living close to an Ikea store | 13:59 |
bauzas | I know those people exist, but I never met them yet | 13:59 |
*** pcaruana has joined #openstack-nova | 13:59 | |
*** cfriesen has joined #openstack-nova | 14:00 | |
luyao | efried: comments addressed for patch series https://review.opendev.org/#/q/topic:bp/virtual-persistent-memory+(status:open), thanks for reviewing | 14:00 |
bauzas | and I personnally consider my house as an Ikea showroom | 14:00 |
bauzas | stephenfin: not sure you saw my comment on https://review.opendev.org/#/c/679745/1 | 14:02 |
sean-k-mooney | efried: so as intersting as ^ is i take it you dont intend to run for PTL again or do you plan to serve for another term | 14:02 |
*** Garyx has joined #openstack-nova | 14:03 | |
bauzas | there is still 24 hours left | 14:05 |
bauzas | -ish | 14:05 |
*** amrith has quit IRC | 14:05 | |
sean-k-mooney | artom: let me know when you push your local version so i can test. im still getting the same error with the same vcpu_pin_set on each host | 14:05 |
bauzas | oh dear no | 14:05 |
donnyd | https://www.irccloud.com/pastebin/nh6VDxYb/ | 14:05 |
bauzas | PTL Nominations Aug 27, 2019 23:45 UTC Sep 03, 2019 23:45 UTC | 14:05 |
sean-k-mooney | bauzas: there are like 9 hours left | 14:05 |
donnyd | https://www.irccloud.com/pastebin/1dwaNShp/ | 14:05 |
bauzas | well, we can afford a PTL designation | 14:05 |
donnyd | sean-k-mooney: ^^^ | 14:06 |
donnyd | LMK if those look right to you | 14:06 |
stephenfin | bauzas: I didn't, but looking at it now I don't think that's we should do that there | 14:06 |
sean-k-mooney | donnyd: yep they look fine | 14:06 |
stephenfin | That's a much bigger discussion that what that patch is trying to fix (namely, making sure it's enabled for all the libvirt functional tests) | 14:06 |
donnyd | project config stuff will be up in a minute | 14:06 |
*** pcaruana has quit IRC | 14:07 | |
bauzas | stephenfin: I don't disagree, I'm just saying that enabling NUMATopologyFilter on *all* libvirt tests is a massive hammer with little justification | 14:07 |
sean-k-mooney | donnyd: cool ping me when you push it. | 14:07 |
bauzas | stephenfin: so, honestly, that's why the question is more widely like : if we go with enabling the filter for all tests, then why not enabling it by default ? | 14:07 |
sean-k-mooney | donnyd: i can proably make the job work on the smaller memory node too or use one of each | 14:08 |
bauzas | and I know this is a debatable question :) | 14:08 |
stephenfin | bauzas: It's already enabled for all but two of them, are it _should_ be enabled for one of those two | 14:08 |
dansmith | artom: so the multi-numa nfv thing is running live migrations with your patches yeah? | 14:08 |
efried | sean-k-mooney: I haven't decided yet. | 14:08 |
cdent | dansmith for ptl! | 14:09 |
sean-k-mooney | dansmith: yes im using depens on to pull in his patches | 14:09 |
stephenfin | It was enabled for all these tests in test_pci_sriov_servers.py and test_numa_servers.py. It should have been enabled for test_rt_servers.py. The only other one is test_reshape.py. I can disable it for that one, if you'd like? | 14:09 |
sean-k-mooney | dansmith: if you use depends-on in the same repo it create a merge commit assumeing there is not merge conflcits and runs on that | 14:09 |
stephenfin | or set it explicitly on the other three. Either one is good with me | 14:09 |
bauzas | stephenfin: uh | 14:09 |
dansmith | sean-k-mooney: yeah, I see that's mechanically true, but ... you guys have gone through and seen that it's working? | 14:09 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: Use Claims to update numa-related XML on the source https://review.opendev.org/635229 | 14:09 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | 14:09 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | 14:09 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | 14:09 |
sean-k-mooney | dansmith: yes | 14:10 |
artom | sean-k-mooney, ^^ | 14:10 |
sean-k-mooney | i have used it in the past | 14:10 |
*** pcaruana has joined #openstack-nova | 14:10 | |
artom | dansmith, yeah, huge props to sean-k-mooney for getting it set up (and donnyd's hardware!) | 14:10 |
dansmith | artom: does it look like it's all working? | 14:10 |
donnyd | https://review.opendev.org/679789 | 14:10 |
donnyd | I don't think I forgot anything | 14:10 |
sean-k-mooney | dansmith: hugepages are not fully working yet | 14:10 |
artom | dansmith, needs to run against the latest PSes, as it found issues with hugepages | 14:10 |
dansmith | artom: okay cool | 14:10 |
bauzas | stephenfin: honestly, dunno what to say | 14:11 |
bauzas | stephenfin: maybe just add a little more explanation in the commit msg and then I'll consider having the option set for all libvirt tests to be good enough | 14:11 |
sean-k-mooney | artom: does that patch have the fixes for hugepages. ill test it locally while we wait for ci | 14:12 |
donnyd | cdent: which part of FN do you think is faster. I know disk access should be in orders of magnitude faster from local NVME (in raid0) | 14:12 |
*** xek has joined #openstack-nova | 14:12 | |
sean-k-mooney | damb raid0 i guess for ci it makes sense | 14:13 |
stephenfin | bauzas: Sure, lemme do that | 14:13 |
*** mkrai has quit IRC | 14:13 | |
stephenfin | efried: yo, any chance you could sign these off for me? https://review.opendev.org/#/c/678902 https://review.opendev.org/#/c/678861 ? | 14:13 |
donnyd | sean-k-mooney: yea if something fails.. it just restarts the job on another hypervisor | 14:13 |
donnyd | so speed is the name of the game for CI... | 14:13 |
cdent | donnyd: it does seem like it mostly related to disk speed. Where I see it the most is: a) getting log files from the swift hosted on FN, b) the placement performance are consistently faster there | 14:13 |
sean-k-mooney | donnyd: devstack is hugely io bound | 14:13 |
*** ociuhandu has quit IRC | 14:13 | |
efried | stephenfin: looking | 14:13 |
stephenfin | bauzas: Need any help with https://review.opendev.org/#/c/678902, btw? That's probably the trickiest patch of the series | 14:14 |
donnyd | I would never do that for prod things - but for CI.. go as fast as possible is where I went with it | 14:14 |
*** xek_ has joined #openstack-nova | 14:14 | |
sean-k-mooney | donnyd: well ci is kind of prod. | 14:14 |
stephenfin | Well, it looks tricky cos it's big. It's not really though | 14:14 |
sean-k-mooney | donnyd: but i have no issue with raid0 as long as you dont put your database on it | 14:14 |
*** mkrai has joined #openstack-nova | 14:14 | |
bauzas | stephenfin: not sure I understand your question ? Why should I need help on something I reviewed ? | 14:15 |
luyao | dansmith: Could you look at https://review.opendev.org/#/c/678448/9 and https://review.opendev.org/#/c/678447/https://review.opendev.org/635229. Thanks. | 14:15 |
donnyd | sean-k-mooney: but if I lose something its not like losing a customers long running instance I can't get back | 14:15 |
stephenfin | bauzas: Damn, I meant https://review.opendev.org/#/c/671800/ | 14:15 |
sean-k-mooney | donnyd: ya that is ture | 14:15 |
donnyd | umm.. no.. the controller is setup a little bit more robust | 14:15 |
sean-k-mooney | donnyd: the could model however is to keep data in sepereate data volumes that are attached to a stateles instance. | 14:15 |
bauzas | stephenfin: oh this one is too hairy for now, I'll look at it tomorrow morning :) | 14:16 |
stephenfin | If you modify the address bar in Firefox, it doesn't change as you navigate pages until you refresh the page | 14:16 |
luyao | dansmith: sorry, they are https://review.opendev.org/#/c/678447/ and https://review.opendev.org/#/c/678448/9 | 14:16 |
sean-k-mooney | *cloud model | 14:16 |
stephenfin | So you end up copy-pasting the same thing multiple times | 14:16 |
stephenfin | bauzas: Heh, cool | 14:16 |
* bauzas doesn't forget he has to continue working on https://review.opendev.org/#/c/670112/ :( | 14:17 | |
*** xek has quit IRC | 14:17 | |
*** ociuhandu has joined #openstack-nova | 14:17 | |
donnyd | I need to get HA controllers running, however I have to make a choice between # of instances to run and potential for downtime... Heat is my battle here | 14:17 |
donnyd | I use forced air - NO A/C | 14:17 |
stephenfin | efried: I know you don't like adding config options without any users, but what about model fields? I feel I should split https://review.opendev.org/#/c/671800/ up so people can focus on the functional changes rather than the noise introduced by the additional field | 14:17 |
bauzas | oh shit, dadtaxi, yeah | 14:17 |
bauzas | stephenfin: I'm not worried by the size of the change in question | 14:18 |
bauzas | anyway, bbiab | 14:18 |
donnyd | If you think about it - I can push a button and redeploy the whole control plane.. so is two hours of downtime once a year or so worth losing 30 instances in CI | 14:19 |
donnyd | I made my bets on short downtime being more acceptable than losing CI resources | 14:19 |
*** mkrai has quit IRC | 14:19 | |
sean-k-mooney | donnyd: well contoler can run vms too | 14:19 |
sean-k-mooney | donnyd: what are you using to deploy | 14:19 |
donnyd | Its custom for my infra | 14:20 |
sean-k-mooney | ah ok | 14:20 |
*** tbachman has joined #openstack-nova | 14:20 | |
*** udesale has quit IRC | 14:20 | |
donnyd | I was using tripleo - but I couldn't get rabbit to act right | 14:20 |
sean-k-mooney | donnyd: ya i dont like how tripleo deplfoys rabbit | 14:21 |
*** udesale has joined #openstack-nova | 14:21 | |
sean-k-mooney | well i dont like how tripleo deploys things in general | 14:21 |
donnyd | well from my perspective there is a performance hit from running in containers - small but noticeable at my scale | 14:21 |
sean-k-mooney | but that is another mater | 14:21 |
donnyd | https://grafana.fortnebula.com/d/9MMqh8HWk/openstack-utilization?orgId=2&refresh=30s | 14:22 |
sean-k-mooney | really i have nver notticed any performac ehit | 14:22 |
donnyd | You can see I don't have much left on my controller for VM's | 14:22 |
sean-k-mooney | oh you mean from a ram proint of view | 14:22 |
luyao | efried: Would you like to test the patches about virtual persistent memory support? I have tested it on real hardware, and I'm preparing another OpenStack with pmems, so you can test on it if you want. I can send you how to access those servers after I set up the env. | 14:23 |
donnyd | Yea, memory is short in the controller | 14:24 |
efried | luyao: It would be nice if *someone* could do that testing in addition to you, just to get another pair of eyes on it. But I'm not sure I have the time to do it myself. | 14:24 |
sean-k-mooney | donnyd: thats becasue heat is partly because of heat | 14:24 |
donnyd | only has 96G | 14:24 |
efried | luyao: sounds like the kind of thing sean-k-mooney would enjoy :P | 14:24 |
sean-k-mooney | efried: i dont have pmem servers in my hallway | 14:25 |
efried | luyao: But I assume it requires specialized hardware, so needs to be inside Intel's firewall. | 14:25 |
efried | yeah. | 14:25 |
sean-k-mooney | so i would have to use qemu to fake pmem in a vm and then thest with nested virt | 14:25 |
donnyd | Due to the nature of forced air cooling, I can only do so much in the way of servers that can be turned on at a single point in time | 14:25 |
*** cdent has quit IRC | 14:26 | |
luyao | efried: yes, I will check if others can access the servers | 14:26 |
sean-k-mooney | efried: i could proably set that up but not until after i test all the other stuff on my list | 14:26 |
luyao | sean-k-mooney: you can use a normal file as a fake pmem | 14:26 |
sean-k-mooney | luyao: any progress on the intel ci for pmem | 14:26 |
sean-k-mooney | luyao: yes so if i spin up a vm with pme via the file backend | 14:26 |
sean-k-mooney | i could deploy devstask inside and test "hardware" pmem supprot | 14:27 |
*** jaosorior has quit IRC | 14:27 | |
luyao | sean-k-mooney: CI is in good process, they are testing the tempest plugin | 14:27 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Make scheduler filters customizable https://review.opendev.org/679745 | 14:27 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start reporting PCPU inventory to placement https://review.opendev.org/671793 | 14:27 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: '_get_(v|p)cpu_total' to '_get_(v|p)cpu_available' https://review.opendev.org/672693 | 14:27 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: hardware: Differentiate between shared and dedicated CPUs https://review.opendev.org/671800 | 14:27 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start reporting 'HW_CPU_HYPERTHREADING' trait https://review.opendev.org/675571 | 14:27 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta https://review.opendev.org/671801 | 14:27 |
sean-k-mooney | are we adding file backed pmem support this cycle to nova | 14:27 |
openstackgerrit | Stephen Finucane proposed openstack/nova master: Add reshaper for PCPU https://review.opendev.org/674895 | 14:27 |
sean-k-mooney | or jsut hardware pmem | 14:27 |
*** Sundar has joined #openstack-nova | 14:28 | |
efried | luyao: I'm pretty sure there's no reasonable way to get an outsider access to the internal lab. At least not in a reasonable time frame to make it worthwhile. | 14:28 |
efried | stephenfin: those two reviews are done | 14:28 |
stephenfin | ta | 14:29 |
efried | stephenfin: unused model fields are okay as long as we've got the series baked enough to be pretty sure we're going to use them. | 14:29 |
efried | stephenfin: We had a bit of a miscue on vpmem and merged a field we ended up redesigning out, so had to do an ugly migration patch. | 14:29 |
artom | efried, FWIW, given the amount of (very justifiable, I should say) grief I got over NUMA LM CI, I'd feel betrayed if something else got merged without seeing it run in public CI somewhere | 14:29 |
stephenfin | bauzas: Have respun https://review.opendev.org/#/c/679745/ | 14:30 |
efried | artom: heard and understood. Though arguably vpmem (niche specialized hardware) is different from NUMA+LM. | 14:31 |
*** rcernin has quit IRC | 14:32 | |
artom | efried, that's also a good point - still, releasing "we support vpmem now!" only to find out it doesn't work because the dev testing it missed a thing (no shade on luyao or you) doesn't look good :) | 14:33 |
artom | Anyways, just I'm just grumpy. We all know there's a pressing lack of specialized hardware in CI | 14:33 |
artom | And with things going more telco (at least from an RH customers POV), that's only going to get worse | 14:33 |
efried | agreed on all counts. | 14:34 |
luyao | sean-k-mooney: We don't support normal file backend. So there will be so many details. You can use a part of memory to simulate a pmem device, and specific kernel , libvirt and qemu are needed, so it may take long time to setup. | 14:34 |
sean-k-mooney | artom: if we support file backed pmem in nova we can emulate the specialised hardware in ci using file backed pemem on the node pool vms | 14:34 |
artom | I was in an interesting convo with an RH consultant posted at Bell, our local massive telco | 14:34 |
artom | There was some interest on their part to become more than a pure consumer | 14:34 |
*** ociuhandu has quit IRC | 14:34 | |
sean-k-mooney | luyao: ok | 14:35 |
artom | This is all very very preliminary, not even sure I should be talking about it, but I'd love it if that kinds of thing a. actually went somewhere and b. was more widespread in the telco world | 14:35 |
*** ociuhandu has joined #openstack-nova | 14:35 | |
*** gbarros has quit IRC | 14:35 | |
donnyd | sean-k-mooney: couldn't we attach like an ephemeral disk to the instance and use that as "fake pmem" | 14:36 |
sean-k-mooney | donnyd: no at least not the way you suggest | 14:36 |
donnyd | How about block storage? | 14:36 |
sean-k-mooney | donnyd: pmem device are exposed as dimms to the guest kernel | 14:36 |
donnyd | yea I know nothing about how it all works - /me goes to the corner to be quiet | 14:37 |
sean-k-mooney | so attaching block device ephemer or volumes wont present it to the guest correclty | 14:37 |
aspiers | kashyap: you around? | 14:37 |
kashyap | aspiers: Yep | 14:37 |
aspiers | kashyap: I've split out _guest_machine_type_for_sev(), but ... | 14:37 |
sean-k-mooney | donnyd: what we can do is have wemu use a file on the host disk to emulate pmem | 14:37 |
kashyap | (Although, I need to head out in an hour) | 14:37 |
sean-k-mooney | donnyd: libvirt/qemu support that | 14:37 |
aspiers | kashyap: now it occurs to me that this and the caller probably both belong in libvirt.utils | 14:37 |
sean-k-mooney | donnyd: nova does not currently | 14:37 |
kashyap | aspiers: :-) | 14:37 |
aspiers | does that make sense? | 14:38 |
kashyap | aspiers: I was also thinking something like that, sitting on _guest_machine_type_for_secure_boot() :D | 14:38 |
aspiers | all the rest of the machine type calculation is already in utils | 14:38 |
sean-k-mooney | donnyd: if we had that support we would jsut set some flave extra specls liek we did for numa sutff | 14:38 |
aspiers | kashyap: hell, we could even have machine_types.py | 14:38 |
kashyap | aspiers: Yeah, I think it does make sense -- keeps all the machine type-related context in the same place. | 14:38 |
aspiers | utils.py is a random mix of crap anyway | 14:38 |
sean-k-mooney | donnyd: pmem emultaion in qemu is really reall new however so even if openstack could do it most deploymes likely would not have a knew enough version | 14:39 |
kashyap | aspiers: Haha, "utils" is so generic, it's beyond description | 14:39 |
*** ociuhandu has quit IRC | 14:39 | |
kashyap | aspiers: More seriously, bearing future work in mind, I _guess_ there will be more special-casing based on machine types. So a machine_types.py doesn't sound far-fetched to me | 14:40 |
kashyap | (Thoug, I _was_ serious on "utils" being generic :D) | 14:40 |
aspiers | kashyap: maybe I could do that after this SEV change | 14:40 |
kashyap | aspiers: Yeah, don't let that bog you down, though. | 14:41 |
kashyap | (It's a nice-to-have for now; and I'd rather let the "Apply SEV-specific guest config when SEV is required | 14:42 |
kashyap | " through the finish line.) | 14:42 |
aspiers | kashyap: yeah I'm working on that patch now | 14:43 |
*** jaosorior has joined #openstack-nova | 14:46 | |
*** mlavalle has joined #openstack-nova | 14:46 | |
*** Garyx has quit IRC | 14:47 | |
*** gbarros has joined #openstack-nova | 14:49 | |
sean-k-mooney | artom: http://paste.openstack.org/show/770137/ | 14:51 |
*** Garyx has joined #openstack-nova | 14:52 | |
sean-k-mooney | almost everything is working at this point | 14:52 |
sean-k-mooney | what does not work is the case where the source and dest nodes dont have the same numa nodes available | 14:53 |
*** mkrai has joined #openstack-nova | 14:57 | |
*** francoisp has joined #openstack-nova | 14:58 | |
openstackgerrit | Balazs Gibizer proposed openstack/nova master: unit test: do not fill rp mapping for failed re-schedule https://review.opendev.org/679653 | 15:03 |
sean-k-mooney | artom: so ya if i use code with the same vcpu pin set on both host so both numa nodes are valid i can live migrate an instance with (pinning, hugepages, sriov, realtime, shared emulator threads) enable all at the same time | 15:05 |
sean-k-mooney | ill repoduce the failing case in the ci for you now | 15:06 |
*** ociuhandu has joined #openstack-nova | 15:07 | |
brtknr | i need someone who uses fedora-coreos on openstack to tell me why the --keypair argument doesnt get respected | 15:08 |
*** tbachman has quit IRC | 15:11 | |
*** jaosorior has quit IRC | 15:11 | |
*** mdbooth_ has joined #openstack-nova | 15:12 | |
*** luksky has quit IRC | 15:13 | |
openstackgerrit | sean mooney proposed openstack/nova master: [DNM] test non overlaping vcpu pin sets. https://review.opendev.org/679805 | 15:15 |
*** mdbooth has quit IRC | 15:15 | |
artom | sean-k-mooney, are you using the code I pushed earlier this morning? | 15:17 |
sean-k-mooney | yes | 15:17 |
sean-k-mooney | that fix everything except this last edgecase | 15:17 |
artom | You mean "hugepages: node 1 not found" is till happening? | 15:17 |
sean-k-mooney | yes | 15:17 |
*** mdbooth_ has quit IRC | 15:17 | |
openstackgerrit | sean mooney proposed openstack/nova master: [DNM] test non overlaping vcpu pin sets. https://review.opendev.org/679805 | 15:19 |
sean-k-mooney | i for got to disable the other jobs in ^ so now its fixed | 15:19 |
openstackgerrit | Merged openstack/nova master: Add <launchSecurity> and <driver iommu='on' /> to config.py https://review.opendev.org/636318 | 15:21 |
artom | I'm trying to understand what that error means | 15:21 |
openstackgerrit | sean mooney proposed openstack/nova master: [DNM] test migration with pinning https://review.opendev.org/679754 | 15:22 |
sean-k-mooney | artom: by the way the non WIP/DNM patches have the job in the perodic and experimental pipline so you need to use check experimenttal or somthing like that not recheck | 15:23 |
sean-k-mooney | artom: ill repoduce it locally again and dump the migration xml | 15:24 |
*** cdent has joined #openstack-nova | 15:25 | |
artom | sean-k-mooney, I *think* I'm getting it | 15:26 |
artom | I'll poke danpb downstream to make sure | 15:26 |
sean-k-mooney | ok | 15:27 |
*** pcaruana has quit IRC | 15:27 | |
*** mkrai has quit IRC | 15:30 | |
*** damien_r has quit IRC | 15:30 | |
*** jaosorior has joined #openstack-nova | 15:37 | |
*** mdbooth has joined #openstack-nova | 15:38 | |
*** xek_ has quit IRC | 15:42 | |
*** amrith__ has joined #openstack-nova | 15:42 | |
*** ociuhandu has quit IRC | 15:43 | |
*** ociuhandu has joined #openstack-nova | 15:44 | |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: Use Claims to update numa-related XML on the source https://review.opendev.org/635229 | 15:46 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | 15:46 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | 15:46 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | 15:46 |
*** amrith__ has quit IRC | 15:46 | |
*** ociuhandu has quit IRC | 15:47 | |
*** udesale has quit IRC | 15:47 | |
*** ociuhandu has joined #openstack-nova | 15:47 | |
*** amrith__ has joined #openstack-nova | 15:47 | |
*** amrith__ has quit IRC | 15:49 | |
efried | dustinc: Three sdk patches merging; problems on 4 & 5. | 15:49 |
*** amrith_ has joined #openstack-nova | 15:50 | |
*** gbarros has quit IRC | 15:50 | |
*** beekneemech is now known as bnemec | 15:51 | |
*** amrith_ has quit IRC | 15:52 | |
*** amrith_ has joined #openstack-nova | 15:52 | |
*** amrith_ has quit IRC | 15:53 | |
*** amrith_ has joined #openstack-nova | 15:53 | |
*** gbarros has joined #openstack-nova | 15:58 | |
*** jmlowe has quit IRC | 15:59 | |
*** tbachman has joined #openstack-nova | 16:01 | |
openstackgerrit | Merged openstack/nova master: libvirt: Simplify 'fakelibvirt.HostInfo' object https://review.opendev.org/678861 | 16:01 |
openstackgerrit | Merged openstack/nova master: libvirt: Start checking compute usage in functional tests https://review.opendev.org/678902 | 16:02 |
*** macz has joined #openstack-nova | 16:07 | |
*** gyee has joined #openstack-nova | 16:12 | |
*** amrith_ has quit IRC | 16:13 | |
*** BjoernT has joined #openstack-nova | 16:14 | |
*** dtantsur is now known as dtantsur|afk | 16:20 | |
donnyd | sean-k-mooney: Ok FN should have the right stuff to run the NUMA jobs now | 16:24 |
donnyd | https://review.opendev.org/#/c/679789/ | 16:24 |
*** ricolin_ has quit IRC | 16:25 | |
*** jaosorior has quit IRC | 16:25 | |
sean-k-mooney | donnyd: cool | 16:26 |
sean-k-mooney | ill try and swap the jobs over to use those during the week | 16:26 |
mriedem | gibi: i left comments on https://review.opendev.org/#/c/655113/23 but will withold voting for now | 16:28 |
mriedem | *withhold | 16:28 |
mriedem | https://review.opendev.org/#/c/679651/ and https://review.opendev.org/#/c/679653/ are pretty simple if another core can hit those - i'm going to have to rebase the cross-cell-resize series and i'd like to wait until after those are merged since they are going to merge conflict me as well | 16:30 |
*** ociuhandu_ has joined #openstack-nova | 16:30 | |
sean-k-mooney | donnyd: are you ok with leaveing the current lable as it is till the end of the week and ill swap over to the new lable at the weekend | 16:32 |
donnyd | Makes no nevermind to me | 16:32 |
donnyd | I can leave it intact until asked to refactor the current jobs | 16:32 |
*** ociuhandu has quit IRC | 16:33 | |
*** ociuhandu_ has quit IRC | 16:35 | |
*** mdbooth has quit IRC | 16:44 | |
*** jaosorior has joined #openstack-nova | 16:54 | |
openstackgerrit | Merged openstack/nova master: objects: Rename 'fields' import to 'obj_fields' https://review.opendev.org/674103 | 16:56 |
openstackgerrit | Merged openstack/nova master: Nice to have test coverage for If1f465112b8e9b0304b8b5b864b985f72168d839 https://review.opendev.org/679629 | 16:56 |
openstackgerrit | Merged openstack/nova master: Remove dead code https://review.opendev.org/679668 | 16:56 |
*** bbowen_ has joined #openstack-nova | 16:59 | |
*** jmlowe has joined #openstack-nova | 17:00 | |
*** derekh has quit IRC | 17:00 | |
*** bbowen has quit IRC | 17:02 | |
*** shilpasd has quit IRC | 17:02 | |
*** ociuhandu has joined #openstack-nova | 17:02 | |
*** ociuhandu has quit IRC | 17:07 | |
sean-k-mooney | artom: https://46b5dc7879590352c3ef-b484a2b89d5d2c358f068133dfb2fa14.ssl.cf5.rackcdn.com/679805/2/experimental/nova-nfv-multi-numa-multinode/b040d28/testr_results.html.gz :) that is for https://review.opendev.org/#/c/679805/2 | 17:11 |
sean-k-mooney | artom: the other two are going to fail due to capasity issue with concurrent tests | 17:12 |
artom | sean-k-mooney, yeah, I just posted on https://review.opendev.org/#/c/679656/7 summarized the CI success | 17:12 |
sean-k-mooney | but they did pass some of the tests | 17:12 |
sean-k-mooney | artom: ok cool | 17:13 |
artom | dansmith, mriedem, if you guys want to take a look ^^ I'd say https://review.opendev.org/#/c/672595/ is ready | 17:13 |
artom | Err, https://review.opendev.org/#/c/635669/49 | 17:13 |
artom | The first patch | 17:13 |
*** jaosorior has quit IRC | 17:14 | |
artom | So there's a failure in https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_54/679754/2/experimental/nova-nfv-multi-numa-multinode/7956dad/testr_results.html.gz, but it's because it failed to even create the server | 17:15 |
artom | The other live migration tests passed | 17:16 |
sean-k-mooney | artom: yep that is due to tempest_concurrency 2 | 17:16 |
*** jaosorior has joined #openstack-nova | 17:16 | |
*** jaosorior has quit IRC | 17:18 | |
mriedem | artom: the functional test patch is in merge conflict | 17:22 |
artom | mriedem, I know, working on it, it's https://review.opendev.org/678861 | 17:23 |
*** pcaruana has joined #openstack-nova | 17:29 | |
*** artom has quit IRC | 17:31 | |
*** artom has joined #openstack-nova | 17:31 | |
*** maciejjozefczyk has quit IRC | 17:31 | |
*** cdent has quit IRC | 17:38 | |
*** priteau has quit IRC | 17:43 | |
*** brinzhang has joined #openstack-nova | 17:45 | |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: Introduce live_migration_claim() https://review.opendev.org/635669 | 17:47 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: New objects for NUMA live migration https://review.opendev.org/634827 | 17:47 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: LM: Use Claims to update numa-related XML on the source https://review.opendev.org/635229 | 17:47 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | 17:47 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | 17:47 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | 17:47 |
artom | mriedem, ^^ | 17:47 |
*** mdbooth has joined #openstack-nova | 17:48 | |
*** brinzhang_ has quit IRC | 17:48 | |
efried | dansmith: Would you please have a look at the bottom two patches of https://review.opendev.org/#/c/678447/ when you get a chance? There's still a bit of work to be done up the series, but stephenfin and I are reviewing that. Want to be able to un-hold and send it once it's baked without having to bug you in emergency mode. | 17:48 |
openstackgerrit | Merged openstack/nova master: Use SDK for node.list https://review.opendev.org/656027 | 17:50 |
dansmith | artom: okay so it's passing everything or still failing legit on something? | 17:50 |
mriedem | landing that vpmems series in ~9 days makes me personally uncomfortable | 17:51 |
artom | dansmith, passing everything as far as I can tell | 17:51 |
dansmith | efried: yeah, just got off a call, so let me bio break and then I will | 17:51 |
dansmith | artom: oh okay, sweet | 17:51 |
efried | thank you dansmith | 17:51 |
*** luksky has joined #openstack-nova | 17:52 | |
*** markvoelker has quit IRC | 18:03 | |
openstackgerrit | Merged openstack/nova master: Use SDK for validating instance and node https://review.opendev.org/656028 | 18:17 |
*** mvkr has quit IRC | 18:18 | |
*** amrith has joined #openstack-nova | 18:18 | |
*** markvoelker has joined #openstack-nova | 18:20 | |
*** amrith has quit IRC | 18:22 | |
*** xek has joined #openstack-nova | 18:23 | |
*** markvoelker has quit IRC | 18:24 | |
*** amrith has joined #openstack-nova | 18:25 | |
openstackgerrit | Merged openstack/nova master: Use SDK for setting instance id https://review.opendev.org/659690 | 18:30 |
*** bbowen__ has joined #openstack-nova | 18:31 | |
*** bbowen_ has quit IRC | 18:35 | |
dansmith | is FF this thursday? | 18:37 |
mriedem | next | 18:37 |
dansmith | oh okay cool | 18:39 |
dansmith | artom: since you rebased it's a little hard to tell...can you summarize what you're now doing to indicate that both ends are new enough to do the numa LM? | 18:41 |
artom | dansmith, https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@6486 | 18:42 |
*** markvoelker has joined #openstack-nova | 18:43 | |
artom | If the destination got a migration flag from the conductor, it sets the numa_live_migration sentinel to True | 18:43 |
artom | It then passes that sentinel (either True or False) to check_can_live_migrate_source | 18:43 |
dansmith | is that all? | 18:45 |
artom | The source compute manager calls the driver's check_can_live_migrate_source - if that driver sets src_supports_numa_live_migration but the numa_live_migration sentinel was False or None, src_supports_numa_live_migration gets removed from migrate_data: https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@6561 | 18:45 |
*** mdbooth has quit IRC | 18:45 | |
artom | Which would then cause the check at https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@6519 back on the destination to be False | 18:46 |
dansmith | okay, so we've passed a new thing in conductor->destination, proving the conductor's rpc allows it, | 18:46 |
artom | Exactly | 18:46 |
dansmith | and then a new thing from destination->source, proving that destination's rpc allows it | 18:46 |
artom | Well, that's always been the case (conductor -> destination), it was the migration object | 18:47 |
dansmith | do we ever make a call from source->destination directly? | 18:47 |
artom | The new thing is destination -> source | 18:47 |
dansmith | like on rollback? | 18:47 |
dansmith | if so, I don't think you know that source's config allows the newer version in order to be able to make the newer rollback call right? | 18:48 |
* dansmith is trying to reload this all in his head post-long-weekend | 18:48 | |
artom | https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@7417 is the source -> destination call on rollback | 18:48 |
artom | No, not that | 18:49 |
dansmith | right, but the source will call to the destination during rollback to make the new drop claim call right? | 18:49 |
artom | https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@7406 is the drop claim call | 18:50 |
*** nweinber has quit IRC | 18:50 | |
artom | Which is entirely new in 5.3 | 18:50 |
*** nweinber has joined #openstack-nova | 18:50 | |
dansmith | right, but I don't think you know whether or not the source can call the destination at 5.3 until you need to rollback and make that call right? | 18:51 |
artom | Isn't that what the check on https://review.opendev.org/#/c/634606/70/nova/compute/manager.py@7405 does? | 18:52 |
artom | 'dst_numa_info' will not be in migrate_data if conductor, source, or dest is pinned | 18:52 |
mriedem | if dst_numa_info is in migrate_data which is built up on the dest wouldn't the rollback be safe to assume it can call the drop claim method? | 18:52 |
dansmith | no, because the source's config is different | 18:53 |
dansmith | that's the point of what I was asking above, | 18:53 |
dansmith | you're passing a flag from conductor->destination, and if it makes it to dest, you know conductor is allowed to send the new version | 18:53 |
dansmith | and you pass a flag from dest->source during the check phase, which proves the dest can send the newer version, | 18:53 |
*** gyee has quit IRC | 18:53 | |
dansmith | but if the source hasn't had its config unpinned and restarted, it will be unable to make the drop_claim call during rollback, but you've already claimed on the dest | 18:54 |
artom | Ohh, I see | 18:55 |
artom | Before including src_supports_numa_live_migration in migate_data, the sources needs to be sure it can speak 5.3 | 18:55 |
artom | *source | 18:55 |
dansmith | right | 18:56 |
artom | I don't have ideas besides "try a new dummy 5.3 method in a try/except" | 18:58 |
artom | Which is ugly as hell | 18:58 |
artom | Is there precedent for this kind of thing? | 18:58 |
dansmith | no, don't do that :) | 18:59 |
dansmith | mriedem: see what I mean ^ ? | 18:59 |
mriedem | i feel like the most we've ever done is just checking the compute service versions, but this is more than that yeah? so not only are the source and dest upgraded, but also checking the compute rpcapi client can_send_version is True from source to dest | 19:00 |
mriedem | so you want check_can_live_migrate_source to check the compute rpcapi client.can_send_version right? | 19:00 |
artom | mriedem, basically yeah | 19:00 |
mriedem | with a todo to drop that in U | 19:00 |
artom | Whenever 6.0 happens | 19:00 |
dansmith | well, | 19:01 |
dansmith | I don't want to call can_send_version from outside rpcapi | 19:01 |
artom | I'm still fuzzy on the whole compatibility thing, but IIUC all of 5.x needs to be cross-compatible? | 19:01 |
dansmith | but something to that effect | 19:01 |
*** igordc has joined #openstack-nova | 19:02 | |
dansmith | service_version isn't really enough here, | 19:02 |
dansmith | because it indicates what that service can receive, but not if the config *allows* sending newer stuff | 19:02 |
artom | Hrmm, source calls pre_live_migration on the dest | 19:04 |
artom | No, that's after | 19:04 |
mriedem | long after this | 19:04 |
artom | Yeah, thinking out loud, and confused by the 'pre-' prefix | 19:04 |
mriedem | https://docs.openstack.org/nova/latest/reference/live-migration.html | 19:04 |
artom | Yeah, you'd thunk it'd be on my bedroom wall by now | 19:05 |
mriedem | but instead it's just posters of grimace | 19:05 |
artom | Nah brah, it's Ahnold | 19:05 |
*** gyee has joined #openstack-nova | 19:07 | |
artom | dansmith, straight up checking CONF.upgrade_level.compute? | 19:12 |
dansmith | no. | 19:12 |
artom | Haha | 19:12 |
dansmith | bad. | 19:12 |
*** mdbooth has joined #openstack-nova | 19:12 | |
mriedem | self.compute_rpcapi.router.client(context).can_send_version or something like that, but abstract it | 19:14 |
mriedem | def can_drop_live_migration_claim_at_dest | 19:15 |
artom | It's not even that | 19:15 |
artom | It's def can_tell_dest_to_drop_calim | 19:15 |
*** Sundar has quit IRC | 19:16 | |
mriedem | yeah yeah | 19:17 |
mriedem | can_formally_please_to_be_requesting_you_rollback_a_live_migration_dest_claim_thank_you_very_much | 19:17 |
artom | version shouldn't leave rpcapi | 19:17 |
artom | So it would be something like can_send(method_name) | 19:18 |
artom | dansmith ^^ acceptable-eesh? | 19:18 |
mriedem | whatever, keep it simple since it gets dropped in U | 19:18 |
dansmith | well, that only works in this specific case, because most of the time it'll be | 19:18 |
dansmith | Can send? True...but... | 19:18 |
dansmith | but, I guess something like that is the quickets | 19:18 |
artom | I mean, unless we want to change the whole order, and have the conductor call the source, which then calls check_can_live_migrate_destination | 19:20 |
artom | Seems like overkill tho | 19:20 |
mriedem | you must be crazy | 19:22 |
sean-k-mooney | artom: why | 19:22 |
sean-k-mooney | the conductor currenlty calls check_can_live_migrate_destination direclty | 19:23 |
artom | sean-k-mooney, source needs to be sure it can send RPC 5.3 before putting src_supports_numa_live_migration in migrate_data | 19:23 |
artom | Becaues it calls drop_move_claim on rollback | 19:23 |
artom | And that's new in 5.3 | 19:23 |
sean-k-mooney | ok but the souce is called from the dest | 19:23 |
sean-k-mooney | so the test can tell the souce right | 19:24 |
mriedem | sean-k-mooney: read back from this: | 19:24 |
mriedem | (1:41:27 PM) dansmith: artom: since you rebased it's a little hard to tell...can you summarize what you're now doing to indicate that both ends are new enough to do the numa LM? | 19:24 |
artom | Yes, but the source needs to know *it* can send 5.3 - ie that it's not pinned to 5.2 for example | 19:24 |
artom | Hence this discussion on how do we achieve that without exposing rpcapi internals, or having a dummy method for the source to attempt to call | 19:24 |
*** ralonsoh has quit IRC | 19:25 | |
sean-k-mooney | mriedem: ok will do. but im concerned that if the mechanium we use for sriov livemigration and multipel port binidngs is not enough for numa migriton all 3 feature would be incorect | 19:25 |
mriedem | i have no idea how well sriov live migration handles rolling upgrades | 19:26 |
sean-k-mooney | it handesl exactly as well as multiple portbindings | 19:26 |
sean-k-mooney | as we did exeactly the same thing | 19:26 |
mriedem | checking the source and dest compute service versions yeah | 19:27 |
sean-k-mooney | we have service version checkes here https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L35-L58 and we call them here https://github.com/openstack/nova/blob/master/nova/conductor/tasks/live_migrate.py#L317-L344 | 19:27 |
*** gbarros has quit IRC | 19:27 | |
sean-k-mooney | mriedem: yes | 19:28 |
artom | I think SRIOV was different because it would outright forbid it everything wasn't updated | 19:28 |
artom | NUMA LM has to work across N/N-1 boundaries | 19:28 |
sean-k-mooney | so did the multiple port bindign yes | 19:28 |
dansmith | well, that doesn't necessarily mean it's good, | 19:28 |
dansmith | but if it didn't have the same amount of rpc involvement as this it would be different | 19:28 |
artom | Dunno about RPC, but actually yeah, they could all have the new service version, but still be pinned when sending | 19:29 |
sean-k-mooney | the numa migration should have the same rpc workflow | 19:29 |
*** jmlowe has quit IRC | 19:29 | |
mriedem | sriov live migration rolls back the claim on dest here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L7496 | 19:30 |
dansmith | it's not the workflow | 19:30 |
mriedem | which the source will call | 19:30 |
mriedem | so it's not really the same thing artom is dealing with, | 19:30 |
mriedem | since artom is calling a new method | 19:30 |
dansmith | probably better to just read the scrollback | 19:30 |
sean-k-mooney | mriedem: ah good point | 19:30 |
sean-k-mooney | we dont intoduce new calls jsut now object for sriov | 19:30 |
sean-k-mooney | and multiple port bindings | 19:31 |
mriedem | brinzhang: when it comes to your microversion API changes, you should probably focus on https://review.opendev.org/#/c/673133/ since i think it's closer and less complicated | 19:33 |
sean-k-mooney | i have read half the scrollback but dumb question. since the extra complexity comes form trying to support cases other that both host are upgraded to the new version. could we just prevent that. we could add a workaoruds config options to allow people to opt in and state if you set this you need to ensure the correct rpc versions are set. | 19:38 |
sean-k-mooney | it wont just magically work but it avoid this issue entirely. | 19:38 |
* sean-k-mooney goes back to reading | 19:38 | |
*** mdbooth_ has joined #openstack-nova | 19:39 | |
*** amrith has quit IRC | 19:39 | |
artom | sean-k-mooney, no, because in a rolling upgrade we want to allow old computes to be "live-evacuated" to new ones | 19:40 |
*** gbarros has joined #openstack-nova | 19:40 | |
sean-k-mooney | artom: right im saying why dont we not support that | 19:40 |
sean-k-mooney | but different question | 19:40 |
artom | sean-k-mooney, because it's a pretty crucial thing, IMHO :) | 19:41 |
sean-k-mooney | if we dont call the cleanup function on rollback | 19:41 |
artom | Anyways, the complexity comes from thinking about it | 19:41 |
artom | Which you need to do regardless | 19:41 |
sean-k-mooney | will the RT on the dest clean it up when the migration is rolled back | 19:41 |
artom | The code itself is clean-ish | 19:41 |
artom | sean-k-mooney, yeah, next periodic run | 19:41 |
sean-k-mooney | artom: ok so if we did not call the cleanup funtion it would get cleaned up eventually | 19:42 |
artom | Eventually | 19:42 |
mriedem | can i get another core to approve this simple refactor https://review.opendev.org/#/c/679651/ | 19:42 |
*** mdbooth has quit IRC | 19:42 | |
*** aojea has joined #openstack-nova | 19:46 | |
*** jmlowe has joined #openstack-nova | 19:47 | |
mriedem | dansmith: btw on https://review.opendev.org/#/c/633293/46 i gave a paste for what it would look like to use the existing prep_resize | 19:49 |
mriedem | at this point we can defer caring about that series until U i guess | 19:50 |
artom | dansmith, btw, I have a func test for RPC pinning, and it did find 1 bug. Wonder why it didn't pick up drop_move_claim() raising an RPC version error | 19:55 |
dansmith | artom: I dunno, probably worth looking though | 19:56 |
*** mdbooth_ has quit IRC | 19:56 | |
artom | Yep. | 19:56 |
dansmith | mriedem: okay, sorry I didn't really mean to block everything on that, I just wanted to ponder a bit but I never got anywhere | 19:56 |
openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider Config File: Update provider tree with new custom resources https://review.opendev.org/676522 | 19:57 |
*** mdbooth has joined #openstack-nova | 19:59 | |
*** pcaruana has quit IRC | 20:00 | |
*** bbowen__ has quit IRC | 20:01 | |
sean-k-mooney | ok i think im following what ye are suggesting | 20:03 |
sean-k-mooney | i also like mriedem function name but pep8 would hate it | 20:03 |
*** nweinber has quit IRC | 20:11 | |
*** slaweq has quit IRC | 20:17 | |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: NUMA live migration support https://review.opendev.org/634606 | 20:21 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | 20:21 |
openstackgerrit | Artom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration https://review.opendev.org/672595 | 20:21 |
artom | dansmith, mriedem, ^^ and at this point I need to done for a bit, because it's dad taxi and supper time | 20:21 |
*** amrith has joined #openstack-nova | 20:21 | |
artom | But if there's anything I'll try to address it later tonight | 20:22 |
sean-k-mooney | aspiers: som of the sev functional test are asserting behavior im not sure is correct or rather is failing when in integrate my image-metadata-prefilter series | 20:26 |
sean-k-mooney | aspiers: you seam to be asserting the ablit for the sev support to be removed while the agent is running. | 20:26 |
sean-k-mooney | the commen stats that thsi coudl happen as a result of a reconfiguration or kernel downgrade which woul nomally inlvoe the agent being resarted | 20:27 |
sean-k-mooney | aspiers: it looks like you are adding the sev tratis to the existing cpu tratis function which i am now cacheing as a property | 20:28 |
sean-k-mooney | https://review.opendev.org/#/c/666915/15/nova/virt/libvirt/driver.py | 20:28 |
sean-k-mooney | i can fix your functional test by reseting the property when you simulate teh restart of the compute service | 20:29 |
sean-k-mooney | but i wan to confim that you dont intend this to change outside of an agent restart | 20:29 |
*** artom has quit IRC | 20:30 | |
*** aojea has quit IRC | 20:35 | |
*** artom has joined #openstack-nova | 20:36 | |
sean-k-mooney | aspiers: im going to proceed with the assumetion i have interpred the test coorectly based on the comments and you are simulating an agent restart and just reset the chached traits. if you get a chance to review and confim that woudl be good. | 20:36 |
*** markvoelker has quit IRC | 20:45 | |
*** aojea has joined #openstack-nova | 20:46 | |
*** ociuhandu has joined #openstack-nova | 20:48 | |
*** trident has quit IRC | 20:52 | |
*** amrith has quit IRC | 20:56 | |
*** trident has joined #openstack-nova | 21:03 | |
*** bbowen has joined #openstack-nova | 21:09 | |
efried | nova FYI https://review.opendev.org/#/c/679862/ | 21:12 |
dansmith | +1 | 21:14 |
*** mdbooth_ has joined #openstack-nova | 21:15 | |
*** mdbooth has quit IRC | 21:17 | |
*** ociuhandu has quit IRC | 21:18 | |
openstackgerrit | sean mooney proposed openstack/nova master: libvirt: use domain capabilities to get supported device models https://review.opendev.org/666915 | 21:24 |
openstackgerrit | sean mooney proposed openstack/nova master: Add transform_image_metadata request filter https://review.opendev.org/665775 | 21:24 |
*** luksky has quit IRC | 21:27 | |
* mriedem deletes nova PTL candidacy patch | 21:27 | |
mriedem | har har jk lol | 21:27 |
dansmith | heh | 21:30 |
*** markvoelker has joined #openstack-nova | 21:31 | |
*** markvoelker has quit IRC | 21:35 | |
aspiers | sean-k-mooney: the comments which explicitly say it's simulating agent restart are correct | 21:37 |
aspiers | https://opendev.org/openstack/nova/src/branch/master/nova/tests/functional/libvirt/test_report_cpu_traits.py#L105 | 21:37 |
aspiers | https://opendev.org/openstack/nova/src/branch/master/nova/tests/functional/libvirt/test_report_cpu_traits.py#L163 | 21:37 |
aspiers | efried: you around? need some quick advice on https://review.opendev.org/#/c/644565 | 21:38 |
efried | aspiers: ... | 21:40 |
aspiers | I think https://review.opendev.org/#/c/644565/49/nova/virt/libvirt/driver.py@5089 needs to be added to utils.py instead | 21:40 |
aspiers | since it turns out that blockinfo.py also needs to call it | 21:40 |
aspiers | or more precisely, blockinfo.py needs to know the same machine type, and in both cases SEV needs to be taken into account | 21:41 |
efried | aspiers: You want my advice for this why? | 21:41 |
aspiers | Well, just a sanity check that sounds reasonable, because that would mean passing more info into blockinfo.py | 21:42 |
aspiers | Adding SEV checks to the machine type calculation requires extra info | 21:42 |
aspiers | e.g. the Host object to check whether SEV is supported | 21:42 |
aspiers | or the sev_enabled variable if already calculated by the caller | 21:42 |
efried | At a glance, I agree it looks weird for a method with this generic a name to need anything sev-specific. | 21:43 |
efried | but I've repeatedly punted on this machine type stuff for a reason. | 21:43 |
aspiers | The key case here is when there is no machine type specified in the image | 21:43 |
aspiers | Maybe I can hassle kashyap about it | 21:44 |
efried | you need to default the machine type differently depending on whether SEV is enabled?? | 21:44 |
aspiers | Yes | 21:44 |
efried | that... doesn't sound right | 21:44 |
aspiers | Normally 'pc' for x86_64, but 'q35' if SEV is enabled | 21:44 |
aspiers | That's in the spec | 21:44 |
aspiers | Ahh, actually the spec is out of date on this point | 21:45 |
efried | What sounds more likely to me is: | 21:45 |
efried | If the machine type is not specified, we default it to X. | 21:45 |
efried | X doesn't work for SEV, so if you do that, SEV will be disabled. | 21:45 |
efried | SEV requires machine type Y, so if you want to SEV, you have to specify machine type Y as well as {other stuff to get SEV} | 21:45 |
aspiers | No, the spec is technically correct | 21:45 |
aspiers | AFAICS your proposal would have a big (negative) impact on usability of SEV | 21:46 |
efried | put another way, the "is SEV enabled" check gets an additional condition for machine type. And the above is how that's reflected in the documentation. | 21:46 |
aspiers | It would make it impossible to turn SEV on or off just by choosing a different flavor | 21:46 |
aspiers | You'd have to also choose an SEV-specific image to go alongside it | 21:47 |
efried | um | 21:47 |
efried | you don't have to do that anyway? | 21:47 |
aspiers | No | 21:47 |
aspiers | You can have one image which can be used SEV or non-SEV | 21:47 |
efried | can you not specify machine type via flavor? | 21:47 |
aspiers | I don't think so | 21:48 |
aspiers | https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/utils.py#L543 | 21:49 |
efried | sorry, but if I'm understanding this correctly, I'm afraid that has to be the right call. | 21:49 |
efried | what are the reasons you would specify a machine type in your image? | 21:49 |
efried | possibly more importantly, what are the reasons you wouldn't? | 21:49 |
aspiers | Well, having the choice of SEV is one reason not to | 21:49 |
aspiers | Well I suppose it could be set to q35 and work non-SEV | 21:50 |
efried | and crucially, are we breaking anyone by setting a default different than the default they're expecting? | 21:50 |
aspiers | There are different levels of default | 21:50 |
aspiers | There's arch-specific defaults hardcoded into nova | 21:51 |
aspiers | There's the default per arch in nova.conf | 21:51 |
aspiers | There's even a QEMU default, which is changing to q35 at some point | 21:51 |
aspiers | (hence https://bugs.launchpad.net/nova/+bug/1780138) | 21:52 |
openstack | Launchpad bug 1780138 in OpenStack Compute (nova) "Don't assume the guest machine type to be of 'pc'" [Medium,Confirmed] - Assigned to Kashyap Chamarthy (kashyapc) | 21:52 |
efried | forgive me for having swapped all of this out, but do we have some flavor/image cross-validation code for this in place? | 21:52 |
aspiers | That's what I was in the process of adding | 21:53 |
efried | So if their image explicitly says machine type X and their flavor asks for SEV, we can punt? | 21:53 |
aspiers | Yes, that's what I agreed this morning with kashyap and sean-k-mooney | 21:53 |
efried | okay | 21:53 |
aspiers | I'm adding an InvalidMachineType exception | 21:53 |
aspiers | as per https://review.opendev.org/#/c/644565/49/nova/virt/libvirt/driver.py@5105 | 21:53 |
efried | and this is one where we're doing a request filtery thing (except no longer in request filter - now it's a RequestSpec loader thing?) to translate/add placement-isms | 21:54 |
aspiers | No, that's a different set of checks | 21:54 |
efried | yeah, I know, but I mean that's another part of this feature | 21:54 |
aspiers | It's part of the same series, yeah | 21:55 |
efried | and how many machine types will sev work with? | 21:55 |
aspiers | just q35 | 21:55 |
efried | beaut | 21:55 |
efried | so why don't we do this: | 21:56 |
efried | In the request spec loader thingy, if image meta doesn't specify a machine type, shove one in there. | 21:56 |
aspiers | I suppose SEV-capable images already need hw_firmware_type=uefi, so adding a requirement for another property isn't terrible | 21:56 |
aspiers | Yeah could do | 21:56 |
aspiers | See https://review.opendev.org/#/c/666616/41/doc/source/admin/configuration/hypervisor-kvm.rst@585 | 21:56 |
efried | that doesn't affect the image at large, does it? Just our local copy of it? | 21:56 |
aspiers | I'd add documentation of the extra constraint to that paragraph | 21:56 |
aspiers | Not sure I understand that question. At large? | 21:57 |
efried | I mean if you change RequestSpec.image.properties, it only affects this one request, not the real image. | 21:57 |
aspiers | It's somewhat unfortunate because it's not q35 per se which SEV requires, but rather virtio 1.0. But 'pc' only offers virtio 0.9, hence the need for q35 | 21:57 |
aspiers | Oh wait, you're suggesting that hw_machine_type=q35 should be added *on the fly*, not to the image via glance API? | 21:59 |
aspiers | I think that would be really bad, because it would override any hw_machine_type set on the real image, or any default in nova.conf | 21:59 |
aspiers | That overriding is what kashyap and sean-k-mooney were objecting to this morning, and I'm inclined to agree with them | 22:00 |
efried | okay, I'm happy to accept that that's a bad idea. | 22:00 |
efried | so back to this other thing | 22:00 |
aspiers | although this all feels a bit bikesheddy, because I have a hard time imagining that anyone would want to insist on 'pc', given that 'q35' has already been around for 11 years | 22:00 |
efried | you pointed out that there's already a requirement for the uefi thing in the image. | 22:00 |
aspiers | Yep | 22:00 |
aspiers | So we could just document an extra requirement for hw_machine_type=q35 | 22:01 |
efried | is that a property they have to set (to some value) regardless? | 22:01 |
efried | or does it also default if you don't set it? | 22:01 |
aspiers | It defaults via the code in libvirt/utils.py | 22:01 |
aspiers | which looks in nova.conf first, and then falls back to the hardcoded mapping | 22:01 |
aspiers | https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/utils.py#L537 | 22:01 |
aspiers | CONF.libvirt.hw_machine_type I mean | 22:02 |
efried | o-queso, I don't see the problem with requiring them to set two image props instead of one. | 22:02 |
aspiers | Yeah, that sounds reasonable to me too | 22:02 |
efried | whee | 22:02 |
aspiers | It's kind of in the same category as the UEFI requirement anyway | 22:02 |
efried | cool | 22:02 |
aspiers | "virtual hardware requirements" | 22:02 |
aspiers | OK that was very helpful, thanks a lot! | 22:03 |
efried | neat | 22:04 |
aspiers | BTW your "I don't know enough about libvirt to be helpful" stance just lost all credibility ;-) | 22:04 |
aspiers | not that I ever believed it anyway X-D | 22:04 |
efried | hah, read the transcript again. It should be even more clear that I know jack about libvirt. | 22:04 |
sean-k-mooney | efried: you can set the default machine_type in the nova.conf too | 22:04 |
aspiers | sean-k-mooney: that's what I just said ;-) | 22:05 |
sean-k-mooney | efried:so you dont have to set it in the image always | 22:05 |
efried | yeah, that ^ sounds like it would be a good option for deployers wishing to enable SEV on their hosts. | 22:05 |
efried | can you do the uefi thing in nova.conf too? | 22:05 |
sean-k-mooney | no | 22:05 |
efried | bummer | 22:05 |
sean-k-mooney | at least not that im aware of | 22:05 |
aspiers | that might be a nice feature to add | 22:05 |
sean-k-mooney | eventrually we do want to change the defualt machie type to Q35 by the way | 22:06 |
aspiers | that would actually be really easy | 22:06 |
aspiers | sean-k-mooney: yeah I mentioned that too | 22:06 |
sean-k-mooney | adding the bios/uefi | 22:06 |
sean-k-mooney | ya it should not be hard | 22:06 |
aspiers | sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1780138 | 22:06 |
openstack | Launchpad bug 1780138 in OpenStack Compute (nova) "Don't assume the guest machine type to be of 'pc'" [Medium,Confirmed] - Assigned to Kashyap Chamarthy (kashyapc) | 22:06 |
sean-k-mooney | although you do need to have the firmware ectra but i can see people wanting to eventurally mvoe the Q35+uefi only | 22:07 |
sean-k-mooney | ya we shoudl avoid assumign pc but its also wrong to assume q35 currently | 22:08 |
sean-k-mooney | we have a function that returns the correct machine type taking into account the default + config + image | 22:09 |
sean-k-mooney | if we dont know what to use we should call that rather then guessing | 22:09 |
*** mriedem has quit IRC | 22:10 | |
sean-k-mooney | aspiers: by the way https://review.opendev.org/#/c/666915/16/nova/tests/functional/libvirt/test_report_cpu_traits.py were the changes i need to make the sev code work with my series | 22:10 |
aspiers | sean-k-mooney: resetting the traits cache makes sense | 22:11 |
aspiers | sean-k-mooney: two questions about resetting domain caps cache | 22:11 |
aspiers | 1. how did my tests pass without that before? | 22:11 |
sean-k-mooney | technically i think you should have always been doing self.compute.driver._host._domain_caps = None in your orginal code | 22:12 |
sean-k-mooney | aspiers: i think it should not or it was a race | 22:12 |
aspiers | 2. why not do it on both tests? | 22:12 |
aspiers | AFAICS it can't be a race because we're only talking about a single thread here | 22:12 |
sean-k-mooney | the second test does not modify the capablities but it would be valid to do it for both | 22:12 |
openstackgerrit | Merged openstack/nova master: libvirt: Fold in argument to '_update_provider_tree_for_vgpu' https://review.opendev.org/676729 | 22:12 |
sean-k-mooney | aspiers: i ment between fucntional test and the order they execute in | 22:13 |
aspiers | oh ok the second test doesn't look up caps because it gives up as soon as it sees the kernel parameter file missing | 22:13 |
sean-k-mooney | yep | 22:13 |
aspiers | but the compute host isn't shared between the tests surely? | 22:14 |
sean-k-mooney | it depens on how your mocking. | 22:14 |
sean-k-mooney | it should not be but i have seen wierd issue | 22:14 |
aspiers | start_compute() is called by each test | 22:15 |
aspiers | and creates a fresh compute service | 22:15 |
sean-k-mooney | when i ran it under the debugger i found you were not geting to the updated domain caps because of https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L759-L760 | 22:16 |
aspiers | right | 22:16 |
aspiers | er | 22:17 |
sean-k-mooney | i dont really know why it passed before but self.compute.driver._host._domain_caps = None seams to fix it | 22:17 |
aspiers | actually I think it's https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L1191 | 22:18 |
aspiers | if the kernel param file is missing, it won't cache any caps | 22:18 |
aspiers | so when it appears later, it will call the fake getDomCaps API for the first time | 22:18 |
sean-k-mooney | no i steped throug it in the debugger | 22:18 |
aspiers | that's in the off_on case | 22:18 |
sean-k-mooney | it might be becasue of my traits code actully | 22:19 |
aspiers | in the on_off case, it wouldn't even care what caps were cached when it gets to off | 22:19 |
aspiers | but technically I think it's good to reset both caches in both tests | 22:19 |
aspiers | in case the order of tests ever got changed in the future | 22:20 |
sean-k-mooney | im calling this https://review.opendev.org/#/c/666915/16/nova/virt/libvirt/driver.py@6832 | 22:20 |
*** gbarros has quit IRC | 22:20 | |
aspiers | sorry, I mean the order of these https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L1188-L1193 | 22:20 |
sean-k-mooney | sure ill add that | 22:21 |
sean-k-mooney | it make it symetic and really we dont want the order of those check to break this | 22:21 |
aspiers | right | 22:21 |
aspiers | that static_traits looks fine to me | 22:21 |
aspiers | except for the blank line after the def ;-) | 22:22 |
sean-k-mooney | anyway the change looks sane to you? | 22:22 |
aspiers | yes | 22:22 |
aspiers | I didn't review the whole thing yet, but that file does | 22:22 |
*** gbarros has joined #openstack-nova | 22:22 | |
aspiers | oh, according to Gerrit I did review the whole thing | 22:22 |
aspiers | I forgot :) | 22:22 |
sean-k-mooney | there is no other change to that form the previous versin | 22:22 |
sean-k-mooney | yes | 22:22 |
sean-k-mooney | a week or so ago | 22:23 |
aspiers | k | 22:23 |
sean-k-mooney | it been wait since and the only change was resolving the merge conflcit with the sev code | 22:23 |
aspiers | not since PS9 | 22:23 |
aspiers | I hate how this old Gerrit makes it so hard to compare patchsets after rebase | 22:23 |
aspiers | that's the #1 thing I want a new Gerrit for | 22:24 |
openstackgerrit | sean mooney proposed openstack/nova master: libvirt: use domain capabilities to get supported device models https://review.opendev.org/666915 | 22:27 |
openstackgerrit | sean mooney proposed openstack/nova master: Add transform_image_metadata request filter https://review.opendev.org/665775 | 22:27 |
*** ociuhandu has joined #openstack-nova | 22:30 | |
*** BjoernT has quit IRC | 22:33 | |
*** aojea has quit IRC | 22:34 | |
*** ociuhandu has quit IRC | 22:35 | |
*** avolkov has quit IRC | 22:46 | |
*** mgoddard has quit IRC | 22:47 | |
*** mgoddard has joined #openstack-nova | 22:47 | |
*** tkajinam has joined #openstack-nova | 22:55 | |
*** xek has quit IRC | 23:03 | |
*** rcernin has joined #openstack-nova | 23:06 | |
*** threestrands has joined #openstack-nova | 23:26 | |
*** mlavalle has quit IRC | 23:30 | |
*** macz has quit IRC | 23:41 | |
*** gbarros has quit IRC | 23:49 | |
*** HagunKim has joined #openstack-nova | 23:52 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!