| *** takashin has joined #openstack-nova | 00:01 | |
| openstackgerrit | Tetsuro Nakamura proposed openstack/nova-specs master: Spec: Support filtering by forbidden aggregate https://review.openstack.org/603352 | 00:02 |
|---|---|---|
| *** brinzhang has joined #openstack-nova | 00:02 | |
| *** hamzy has joined #openstack-nova | 00:05 | |
| *** janki has quit IRC | 00:09 | |
| *** janki has joined #openstack-nova | 00:09 | |
| *** janki has quit IRC | 00:10 | |
| *** janki has joined #openstack-nova | 00:11 | |
| *** gyee has quit IRC | 00:19 | |
| *** jiapei has quit IRC | 00:24 | |
| *** new-comer has joined #openstack-nova | 00:37 | |
| new-comer | https://bugs.launchpad.net/nova/+bug/1792674 | 00:39 |
| openstack | Launchpad bug 1792674 in OpenStack Compute (nova) "[Rocky] fail to get VNC console" [Undecided,Confirmed] | 00:39 |
| *** yingjun has joined #openstack-nova | 00:43 | |
| *** new-comer has left #openstack-nova | 01:01 | |
| *** new-comer has quit IRC | 01:02 | |
| *** slaweq has joined #openstack-nova | 01:11 | |
| *** evrardjp has quit IRC | 01:12 | |
| *** slaweq has quit IRC | 01:15 | |
| *** mhen has quit IRC | 01:19 | |
| *** mhen has joined #openstack-nova | 01:21 | |
| *** Dinesh_Bhor has joined #openstack-nova | 01:29 | |
| *** mrsoul has quit IRC | 01:29 | |
| *** litao has joined #openstack-nova | 01:33 | |
| *** tetsuro has quit IRC | 01:35 | |
| *** hongbin has joined #openstack-nova | 01:37 | |
| openstackgerrit | Chen proposed openstack/nova master: doc trivial: additional info to admin-password-injection https://review.openstack.org/603414 | 01:43 |
| *** dpawlik has joined #openstack-nova | 01:56 | |
| *** dpawlik has quit IRC | 02:00 | |
| *** sapd1_ has joined #openstack-nova | 02:04 | |
| *** sapd1 has quit IRC | 02:07 | |
| *** Dinesh_Bhor has quit IRC | 02:22 | |
| *** yingjun has quit IRC | 02:28 | |
| *** Dinesh_Bhor has joined #openstack-nova | 02:31 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: test resize reschedule up-calls https://review.openstack.org/603389 | 02:38 |
| *** tetsuro has joined #openstack-nova | 02:43 | |
| *** rnoriega has quit IRC | 02:45 | |
| *** rnoriega has joined #openstack-nova | 02:46 | |
| *** vivsoni has joined #openstack-nova | 02:56 | |
| *** jistr has quit IRC | 03:00 | |
| *** jistr has joined #openstack-nova | 03:00 | |
| *** afazekas has quit IRC | 03:00 | |
| *** BlackDex has quit IRC | 03:01 | |
| *** BlackDex has joined #openstack-nova | 03:03 | |
| *** slaweq has joined #openstack-nova | 03:11 | |
| *** slaweq has quit IRC | 03:16 | |
| openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Add support changes-before for microversion 2.66 https://review.openstack.org/603549 | 03:18 |
| *** Dinesh_Bhor has quit IRC | 03:28 | |
| *** pvc has quit IRC | 03:28 | |
| openstackgerrit | Merged openstack/nova stable/rocky: Handle binding_failed vif plug errors on compute restart https://review.openstack.org/595317 | 03:32 |
| openstackgerrit | Merged openstack/nova stable/rocky: Restart scheduler in TestNovaManagePlacementHealAllocations https://review.openstack.org/598152 | 03:32 |
| *** Dinesh_Bhor has joined #openstack-nova | 03:34 | |
| *** vivsoni has quit IRC | 03:37 | |
| *** Dinesh_Bhor has quit IRC | 03:42 | |
| *** udesale has joined #openstack-nova | 03:52 | |
| *** cfriesen has quit IRC | 03:55 | |
| *** tetsuro has quit IRC | 03:58 | |
| *** vivsoni has joined #openstack-nova | 04:04 | |
| *** yingjun has joined #openstack-nova | 04:04 | |
| *** vivsoni has quit IRC | 04:06 | |
| *** vivsoni has joined #openstack-nova | 04:06 | |
| *** jaosorior_ is now known as jaosorior | 04:13 | |
| *** yingjun has quit IRC | 04:18 | |
| *** Dinesh_Bhor has joined #openstack-nova | 04:32 | |
| gmann | alex_xu: i will skip today API office hour due to fever. we will continue from next week. | 04:57 |
| *** hamzy_ has joined #openstack-nova | 05:01 | |
| *** hamzy has quit IRC | 05:02 | |
| *** hongbin has quit IRC | 05:06 | |
| alex_xu | gmann: ok, no problem | 05:06 |
| *** hamzy has joined #openstack-nova | 05:09 | |
| *** hamzy_ has quit IRC | 05:11 | |
| *** slaweq has joined #openstack-nova | 05:11 | |
| *** slaweq has quit IRC | 05:15 | |
| *** hamzy has quit IRC | 05:17 | |
| *** hamzy has joined #openstack-nova | 05:18 | |
| *** tetsuro has joined #openstack-nova | 05:20 | |
| *** hamzy has quit IRC | 05:23 | |
| *** hamzy has joined #openstack-nova | 05:24 | |
| *** rcernin has quit IRC | 05:30 | |
| *** rcernin has joined #openstack-nova | 05:30 | |
| *** ratailor has joined #openstack-nova | 05:31 | |
| *** yingjun has joined #openstack-nova | 05:36 | |
| *** luksky11 has joined #openstack-nova | 05:51 | |
| *** yingjun has quit IRC | 05:54 | |
| *** rcernin_ has joined #openstack-nova | 06:04 | |
| *** rcernin has quit IRC | 06:06 | |
| *** holser_ has joined #openstack-nova | 06:12 | |
| *** Luzi has joined #openstack-nova | 06:16 | |
| *** janki has quit IRC | 06:17 | |
| *** holser_ has quit IRC | 06:18 | |
| *** slaweq has joined #openstack-nova | 06:18 | |
| *** holser_ has joined #openstack-nova | 06:21 | |
| *** slaweq has quit IRC | 06:23 | |
| *** maciejjozefczyk has joined #openstack-nova | 06:26 | |
| *** dpawlik has joined #openstack-nova | 06:27 | |
| *** dpawlik has quit IRC | 06:28 | |
| *** dpawlik_ has joined #openstack-nova | 06:28 | |
| *** maciejjozefczyk has quit IRC | 06:30 | |
| *** dpawlik_ has quit IRC | 06:30 | |
| *** dpawlik has joined #openstack-nova | 06:30 | |
| *** dpawlik has quit IRC | 06:31 | |
| *** dpawlik_ has joined #openstack-nova | 06:31 | |
| *** rcernin has joined #openstack-nova | 06:34 | |
| *** rcernin_ has quit IRC | 06:36 | |
| *** yingjun has joined #openstack-nova | 06:36 | |
| *** deepak_mourya_ has quit IRC | 06:40 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.openstack.org/601047 | 06:53 |
| *** slaweq has joined #openstack-nova | 06:55 | |
| *** janki has joined #openstack-nova | 06:55 | |
| *** slaweq has quit IRC | 07:00 | |
| *** rcernin has quit IRC | 07:06 | |
| *** Dinesh_Bhor has quit IRC | 07:09 | |
| dpawlik_ | is someone using cellv2 but not as "simple" cell but multiple cells? | 07:11 |
| *** hamdyk has joined #openstack-nova | 07:11 | |
| *** slaweq has joined #openstack-nova | 07:16 | |
| *** hoonetorg has quit IRC | 07:17 | |
| *** cfriesen has joined #openstack-nova | 07:18 | |
| sorrison | dpawlik_: I know CERN are | 07:22 |
| dpawlik_ | sorrison: thx | 07:22 |
| dpawlik_ | sorrison: I got a feeling when I read documentation and the code on github, that all uses simple cell setup xD | 07:23 |
| sorrison | We (Nectar) have some cellsv2 stuff running in testing, still cellsv1 in production | 07:23 |
| dpawlik_ | there are no starts scripts for RDO/Debian Salsa/Ubuntu cloud archive to start super conductor and also conductor that take configuration from different file | 07:23 |
| sorrison | I'm *think* that the super conductor is just a normal conductor but it listens on the api level rabbit queue | 07:25 |
| sorrison | If you were running multiple cells then you'd have lots of different controllers running different conductors | 07:26 |
| bauzas | good morning stackers | 07:26 |
| bauzas | sorrison: this is correct | 07:27 |
| bauzas | dpawlik_: a super conductor is just a normal conductor with a specific configuration | 07:27 |
| dpawlik_ | bauzas: but its reading amqps queue from vhost / and its also talking with nova_cell0 db yes? | 07:30 |
| *** hoonetorg has joined #openstack-nova | 07:30 | |
| dpawlik_ | and the second conductor e.g. cell1 is reading from another vhost e.g. nova_cell1 and its also puting data to nova_cell1 db | 07:31 |
| dpawlik_ | I just describe my point of knowledge how it should be configured, is it true? | 07:31 |
| dpawlik_ | I just take it from devstack | 07:32 |
| dpawlik_ | and Im wondering why after executing the same commands like it is done on devstack, on my controller I have duplicated nova services | 07:32 |
| *** Dinesh_Bhor has joined #openstack-nova | 07:36 | |
| *** Gorian has quit IRC | 07:37 | |
| *** jamiec has quit IRC | 07:38 | |
| *** Gorian has joined #openstack-nova | 07:39 | |
| *** cfriesen has quit IRC | 07:39 | |
| *** helenafm has joined #openstack-nova | 07:39 | |
| bauzas | dpawlik_: sorry, I was on mute so I missed your ping | 07:40 |
| * bauzas facepalms | 07:40 | |
| bauzas | dpawlik_: hold on, we have some docs that explain this | 07:40 |
| dpawlik_ | bauzas: thx | 07:40 |
| bauzas | dpawlik_: https://docs.openstack.org/nova/latest/user/cells.html#setup-of-cells-v2 | 07:41 |
| bauzas | dpawlik_: and https://docs.openstack.org/nova/latest/user/cellsv2-layout.html | 07:42 |
| bauzas | hope that will help you understand the cells v2 concepts | 07:42 |
| dpawlik_ | So im doing the same how its described and TBH its not working well | 07:42 |
| dpawlik_ | Im still have duplication of nova services | 07:42 |
| dpawlik_ | yesterday I send it to dansmith how my cell mappings looks like | 07:43 |
| dpawlik_ | and let me quote "looks like only one mapping in there, so I guess you're good" | 07:43 |
| *** jamiec has joined #openstack-nova | 07:43 | |
| dpawlik_ | hmm, clean host, new day | 07:45 |
| dpawlik_ | bauzas: | 07:45 |
| dpawlik_ | bauzas: pro tip to setup cells | 07:45 |
| bauzas | how many services are you running in parallel ? | 07:46 |
| dpawlik_ | stop nova-conductor services before start setup cells_v2 | 07:46 |
| bauzas | dpawlik_: you're in an upgrade case or greenfields ? | 07:46 |
| *** maciejjozefczyk has joined #openstack-nova | 07:47 | |
| dpawlik_ | bauzas: for now Im just testing deployment queens release on new infra | 07:47 |
| dpawlik_ | for now I don't want to touch "upgrade" procedure | 07:47 |
| bauzas | but yeah, I'm assuming you populate the mappings first before starting the services | 07:47 |
| bauzas | since those services use the mappings you need to set | 07:48 |
| bauzas | dpawlik_: but we could make it clearer in docs, I presume | 07:48 |
| dpawlik_ | Now I will know | 07:48 |
| bauzas | dpawlik_: so yeah, that's why you had duplicate services | 07:48 |
| dpawlik_ | bauzas: Small notification on the top of the doc will be helpful for other people | 07:48 |
| bauzas | those first registered on startup | 07:48 |
| bauzas | dpawlik_: yeah, that makes sense in https://docs.openstack.org/nova/latest/user/cells.html#cells-v2 | 07:49 |
| bauzas | I could spin a change | 07:49 |
| dpawlik_ | dansmith: ^^ above is description why I have duplicated services on good cell mapping xD | 07:49 |
| dpawlik_ | bauzas: please :) | 07:50 |
| dpawlik_ | bauzas: if you can, write it please | 07:50 |
| openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Add support changes-before for microversion 2.66 https://review.openstack.org/603549 | 07:51 |
| *** sahid has joined #openstack-nova | 07:57 | |
| bauzas | dpawlik_: lemme stash my current work :) | 07:58 |
| *** yingjun has quit IRC | 07:59 | |
| *** takashin has left #openstack-nova | 08:02 | |
| *** maciejjozefczyk has quit IRC | 08:02 | |
| openstackgerrit | Tetsuro Nakamura proposed openstack/nova-specs master: Spec: allocation candidates in tree https://review.openstack.org/603585 | 08:02 |
| *** maciejjozefczyk has joined #openstack-nova | 08:05 | |
| *** yingjun has joined #openstack-nova | 08:05 | |
| openstackgerrit | Sylvain Bauza proposed openstack/nova master: cells: Be explicit in docs about service restarts https://review.openstack.org/603588 | 08:08 |
| bauzas | dpawlik_: ^ | 08:10 |
| *** tetsuro has quit IRC | 08:12 | |
| kashyap | Sigh, copy-pasting Git commit messages seem to break the line-wrapping in LaunchPad: https://bugs.launchpad.net/nova/+bug/1789868 | 08:16 |
| openstack | Launchpad bug 1789868 in OpenStack Compute (nova) "RFE: Add a virtio-rng device to Nova instances by default" [Medium,Triaged] - Assigned to Kashyap Chamarthy (kashyapc) | 08:16 |
| kashyap | The formatting is now fugly | 08:16 |
| dpawlik_ | bauzas: checking | 08:18 |
| *** mdbooth_ has joined #openstack-nova | 08:19 | |
| *** mdbooth_ is now known as mdbooth | 08:20 | |
| dpawlik_ | bauzas: maybe warning will be better than info | 08:22 |
| dpawlik_ | but it looks good | 08:22 |
| bauzas | not really | 08:23 |
| dpawlik_ | bauzas: im reading it from my perspective where I was wondering half day why my deployment doesn't work correctly | 08:24 |
| *** jpena|off is now known as jpena | 08:31 | |
| *** skatsaounis has quit IRC | 08:34 | |
| *** ralonsoh has joined #openstack-nova | 08:34 | |
| kashyap | bauzas: johnthetubaguy: Hey there, this trivial thing is ready for merge: https://review.openstack.org/#/c/602592/ | 08:38 |
| *** alex_xu has quit IRC | 08:38 | |
| *** luksky11 has quit IRC | 08:39 | |
| *** alexchadin has joined #openstack-nova | 08:43 | |
| *** skatsaounis has joined #openstack-nova | 08:44 | |
| *** mdbooth has quit IRC | 08:47 | |
| *** mdbooth has joined #openstack-nova | 08:48 | |
| *** derekh has joined #openstack-nova | 08:49 | |
| *** owalsh has quit IRC | 08:51 | |
| *** mdbooth_ has joined #openstack-nova | 08:54 | |
| *** owalsh has joined #openstack-nova | 08:56 | |
| *** mdbooth has quit IRC | 08:56 | |
| *** dtantsur|afk is now known as dtantsur | 08:57 | |
| *** mdbooth_ has quit IRC | 09:06 | |
| *** mdbooth has joined #openstack-nova | 09:06 | |
| *** luksky11 has joined #openstack-nova | 09:09 | |
| *** dpawlik_ has quit IRC | 09:13 | |
| *** dpawlik has joined #openstack-nova | 09:13 | |
| *** mdbooth_ has joined #openstack-nova | 09:20 | |
| *** mdbooth has quit IRC | 09:23 | |
| *** markvoelker has quit IRC | 09:25 | |
| *** elod has quit IRC | 09:26 | |
| openstackgerrit | Vlad Gusev proposed openstack/nova stable/queens: WIP libvirt: Reduce calls to qemu-img during update_available_resource https://review.openstack.org/603358 | 09:26 |
| *** elod has joined #openstack-nova | 09:27 | |
| openstackgerrit | Vlad Gusev proposed openstack/nova stable/queens: libvirt: Reduce calls to qemu-img during update_available_resource https://review.openstack.org/603358 | 09:33 |
| *** s10 has joined #openstack-nova | 09:33 | |
| *** maciejjozefczyk has quit IRC | 09:40 | |
| *** yingjun has quit IRC | 09:41 | |
| bauzas | kashyap: sorry, was on meeting | 09:46 |
| *** owalsh has quit IRC | 09:46 | |
| bauzas | kashyap: do we recreate the guest XML on start() ? | 09:48 |
| bauzas | kashyap: I'm confused by https://review.openstack.org/#/c/602592/3/releasenotes/notes/Use-virt-as-machine-type-for-ARMv7-cd2c252336057ec8.yaml | 09:48 |
| bauzas | but I can check | 09:48 |
| bauzas | kashyap: nevermind, I check | 09:49 |
| bauzas | checked* | 09:49 |
| *** Dinesh_Bhor has quit IRC | 09:49 | |
| bauzas | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2931 | 09:50 |
| bauzas | oops, bad link https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2895 | 09:51 |
| *** owalsh has joined #openstack-nova | 09:52 | |
| *** Dinesh_Bhor has joined #openstack-nova | 09:52 | |
| kashyap | bauzas: Sorry, was out to get some groceries | 09:58 |
| bauzas | nah no worries | 09:58 |
| kashyap | bauzas: Would you like me to update the text here: https://review.openstack.org/#/c/602592/3/releasenotes/notes/Use-virt-as-machine-type-for-ARMv7-cd2c252336057ec8.yaml | 09:59 |
| kashyap | bauzas: Since your question is answered (by yourself), or is the text good as-is for you? | 09:59 |
| kashyap | Ah, you +2ed it | 09:59 |
| *** maciejjozefczyk has joined #openstack-nova | 09:59 | |
| kashyap | Thanks for looking! | 10:00 |
| *** Dinesh_Bhor has quit IRC | 10:04 | |
| *** tbachman has quit IRC | 10:10 | |
| *** markvoelker has joined #openstack-nova | 10:30 | |
| *** tbachman has joined #openstack-nova | 10:36 | |
| *** Dinesh_Bhor has joined #openstack-nova | 10:37 | |
| *** hoangcx has quit IRC | 10:38 | |
| *** hoangcx has joined #openstack-nova | 10:38 | |
| *** Dinesh_Bhor has quit IRC | 10:39 | |
| *** pooja_jadhav has quit IRC | 10:40 | |
| *** tbachman has quit IRC | 10:40 | |
| *** tbachman has joined #openstack-nova | 10:48 | |
| *** pooja_jadhav has joined #openstack-nova | 10:56 | |
| *** pooja-jadhav has joined #openstack-nova | 10:57 | |
| *** markvoelker has quit IRC | 11:00 | |
| *** erlon has joined #openstack-nova | 11:02 | |
| *** tbachman has quit IRC | 11:09 | |
| *** ShilpaSD has quit IRC | 11:10 | |
| *** priteau has joined #openstack-nova | 11:16 | |
| *** imacdonn has quit IRC | 11:19 | |
| *** imacdonn has joined #openstack-nova | 11:20 | |
| *** jpena is now known as jpena|lunch | 11:20 | |
| *** cdent has joined #openstack-nova | 11:25 | |
| *** sambetts|afk is now known as sambetts | 11:35 | |
| *** med_ has joined #openstack-nova | 11:42 | |
| *** owalsh has quit IRC | 11:42 | |
| *** pvc has joined #openstack-nova | 11:47 | |
| *** tssurya has joined #openstack-nova | 11:48 | |
| pvc | good evening bauzas is vgpu supported on ocata? | 11:49 |
| bauzas | nope, queens only | 11:49 |
| pvc | queens and rocky right? | 11:50 |
| bauzas | but you can passthru in ocata | 11:50 |
| bauzas | queens and later, yes | 11:50 |
| pvc | thank you so much | 11:50 |
| *** udesale has quit IRC | 11:56 | |
| *** brinzhang has quit IRC | 11:56 | |
| *** markvoelker has joined #openstack-nova | 11:57 | |
| tssurya | johnthetubaguy or alex_xu: could you review https://review.openstack.org/#/c/596285/ whenever you have the time ? | 12:01 |
| sean-k-mooney | tssurya: thanks for working on https://review.openstack.org/#/c/603352/2/specs/stein/approved/alloc-candidates-negative-member-of.rst. i saw it yesterday when i went to write up the same spec :) | 12:11 |
| sean-k-mooney | tssurya: i have not had time to review properly yet but at a glance you seam to have captured the important points | 12:12 |
| tssurya | sean-k-mooney: you are surely talking to the wrong person :) | 12:13 |
| jaypipes | stephenfin: do you really have someone on your internal team named Jon Snow? | 12:13 |
| *** tbachman has joined #openstack-nova | 12:17 | |
| stephenfin | jaypipes: Within the company, yes, and the puns come thick and fast | 12:18 |
| *** med_ has quit IRC | 12:20 | |
| jaypipes | stephenfin: that is... well, that is awesome. | 12:20 |
| jaypipes | stephenfin: do you require him to carry a Valorian steel blade around the office? | 12:21 |
| * jaypipes thinks you should | 12:21 | |
| tssurya | I am thinking he must have the hardest time with the sentence "You know no'ting Jon Snow" | 12:22 |
| cdent | sean-k-mooney: you're looking tetsuro | 12:22 |
| stephenfin | jaypipes: No, but it appears he is the least knowledgeable person in the company | 12:22 |
| stephenfin | tssurya: Indeed :D | 12:22 |
| tssurya | :D | 12:22 |
| jaypipes | tssurya: ++ | 12:23 |
| *** jpena|lunch is now known as jpena | 12:24 | |
| *** markvoelker has quit IRC | 12:31 | |
| *** skatsaounis has quit IRC | 12:36 | |
| sean-k-mooney | cdent: tssurya sorry yes | 12:36 |
| sean-k-mooney | jaypipes: quick question for the device whitelist spec im working on. | 12:42 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: libvirt: Always escape IPv6 addresses when used in migration URI https://review.openstack.org/603737 | 12:42 |
| sean-k-mooney | did you want versioning, a schma or both. if both would you like it to also cover migrations. trying to refine scope | 12:42 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/pike: libvirt: Always escape IPv6 addresses when used in migration URI https://review.openstack.org/603738 | 12:43 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/ocata: libvirt: Always escape IPv6 addresses when used in migration URI https://review.openstack.org/603740 | 12:44 |
| jaypipes | sean-k-mooney: doesn't have to cover the evolution of the schema, just needs to have a schema and a version attribute in the schema so that we *can* evolve the schema over time. | 12:46 |
| sean-k-mooney | jaypipes: that is much simpler :) i was hopping you would say that. | 12:46 |
| jaypipes | sean-k-mooney: i.e. you don't need to go into the implementation details of evolving the schema over time. just make sure there's a schema and version. | 12:46 |
| sean-k-mooney | jaypipes: im debating about proposing jsonscma as the validation language for the schmea but it does not have built in migration support. oslo.config also can do the validation but forces ini and ovo gives the versioning but is way to verbose and annoying for configfiles. so of the eaisly accessable tools there was not one i could tink of that nativly was file format independent, did validation and | 12:49 |
| sean-k-mooney | hand native migration/versioning | 12:49 |
| jaypipes | sean-k-mooney: go with JSONSchema. | 12:50 |
| sean-k-mooney | jaypipes: thats what it current assumes | 12:50 |
| jaypipes | sean-k-mooney: you can always use YAML as the serialization format later if you want. | 12:50 |
| jaypipes | sean-k-mooney: but keep JSONSchema for validation. | 12:50 |
| sean-k-mooney | jaypipes: yes i also assumed that would be the default | 12:50 |
| sean-k-mooney | if we use json scema you can use json/toml/yaml as the file format and we dont need to care | 12:51 |
| sean-k-mooney | also i cant spell schema apparently | 12:51 |
| jaypipes | sean-k-mooney: well, let's face it, spelling isn't your strong suit. :P | 12:52 |
| openstackgerrit | Jay Pipes proposed openstack/os-traits master: clean up CUDA traits https://review.openstack.org/597170 | 12:53 |
| jaypipes | stephenfin: ^ | 12:53 |
| stephenfin | jaypipes: ack | 12:53 |
| *** alexchadin has quit IRC | 12:53 | |
| *** alexchadin has joined #openstack-nova | 12:54 | |
| *** vivsoni has quit IRC | 12:56 | |
| *** vivsoni has joined #openstack-nova | 12:56 | |
| *** kukacz_ is now known as kukacz | 13:00 | |
| openstackgerrit | huanhongda proposed openstack/nova stable/ocata: Cleanup RP and HM records while deleting a compute service. https://review.openstack.org/603749 | 13:02 |
| bauzas | folks, bailing out for 20-ish minutes to see first plays of basketball from my oldest | 13:02 |
| bauzas | ++ | 13:02 |
| *** ratailor has quit IRC | 13:03 | |
| *** gbarros has joined #openstack-nova | 13:03 | |
| *** holser_ has quit IRC | 13:04 | |
| sean-k-mooney | jaypipes: :) grammerly to the rescure. at least for the spec. | 13:04 |
| *** holser__ has joined #openstack-nova | 13:04 | |
| *** Kevin_Zheng has quit IRC | 13:08 | |
| *** alexchadin has quit IRC | 13:11 | |
| openstackgerrit | huanhongda proposed openstack/nova stable/ocata: Cleanup RP and HM records while deleting a compute service. https://review.openstack.org/603749 | 13:11 |
| jaypipes | sean-k-mooney: grammarly you mean? :) | 13:15 |
| jaypipes | sean-k-mooney: and I think you meant rescue, not rescure. P | 13:15 |
| *** alexchadin has joined #openstack-nova | 13:15 | |
| jaypipes | sean-k-mooney: don't MAKE me come over to Ireland to be your personal spelling assistant. | 13:15 |
| *** janki has quit IRC | 13:16 | |
| sean-k-mooney | jaypipes: you see if only they had a rest api that could be intergrated with my irc client life would be so much simpler. which reminds me i need to reinstall the aspell plugin... | 13:17 |
| dims | LOLOL | 13:17 |
| sean-k-mooney | jaypipes: :) im gong to push the very rough draft for some early feedback. ill be converting it to spec format and then starting on the actuall schema in json schema format in the next revision or two. | 13:19 |
| *** mriedem has joined #openstack-nova | 13:21 | |
| *** mdbooth_ has quit IRC | 13:22 | |
| openstackgerrit | Wenran Xiao proposed openstack/nova-specs master: Add suport selecting subnet when createing vm https://review.openstack.org/603344 | 13:24 |
| *** mdbooth has joined #openstack-nova | 13:26 | |
| *** claudiub has joined #openstack-nova | 13:28 | |
| BlackDex | Hello there. I'm trying to manipulate the libvirt/qemu/kvm parameters without needing to cold-migrate/hard-reboot an instances | 13:28 |
| BlackDex | I changed the xml of an instances using `virsh edit`, i do a live-migrate of that instances and the xml on the destination host contains the correct values, but looking at the process `ps fauxw` it still uses the old values. | 13:29 |
| BlackDex | How and where does nova determine which parameters to use for the live-migrate to spawn the instance to live-migrate to? | 13:30 |
| claudiub | you're basically trying to resize the libvirt/qemu/kvm instance without restarting it? | 13:30 |
| BlackDex | claudiub: yes | 13:30 |
| claudiub | heh, that sounds like live-resize | 13:30 |
| BlackDex | yea kinda is | 13:31 |
| BlackDex | but i haven't seen that feature yet ;) | 13:31 |
| BlackDex | i need it only for the iotune parameters | 13:31 |
| claudiub | it's because the proposal for that feature didn't merge: https://review.openstack.org/#/c/141219/ | 13:31 |
| BlackDex | the iotune parameters only seem to be changed when using volume-types | 13:31 |
| BlackDex | those seem te be dynamic when i change the min/max iops | 13:32 |
| BlackDex | but not with ephemeral disks | 13:32 |
| claudiub | hmm, i'm not sure those are set for ephemeral disks | 13:34 |
| BlackDex | you can set that in the flavor | 13:34 |
| BlackDex | quota:disk_read_iops_sec for instnace | 13:34 |
| BlackDex | that works for an empemeral disk | 13:35 |
| BlackDex | but, if you want to change that you need to resize/cold-migrate the instace for it to work | 13:35 |
| BlackDex | while volumes using a volume-type which are live-migrated will get a new iops setting if the volume-type has a qos set | 13:36 |
| claudiub | yeah, indeed, they are being set during resize. | 13:36 |
| BlackDex | i tried to change the mysql entries of the instance_extras table | 13:36 |
| BlackDex | the volume-type qos is also set during a live-migrate! | 13:37 |
| BlackDex | if it previously wasn't | 13:37 |
| *** udesale has joined #openstack-nova | 13:38 | |
| BlackDex | so, i changed in the mysql everything of the instance metadata info, but it doesn't use those values during a live-migrate | 13:38 |
| BlackDex | so i wondered where the parameters for qemu are comming from durning the live-migrate | 13:39 |
| BlackDex | it seems they do not come from the mysql database, or the libvirt xml | 13:39 |
| BlackDex | there are pulled from somewhere els | 13:39 |
| BlackDex | e | 13:39 |
| *** munimeha1 has joined #openstack-nova | 13:42 | |
| *** maciejjozefczyk has quit IRC | 13:46 | |
| *** maciejjozefczyk has joined #openstack-nova | 13:46 | |
| *** sapd1 has joined #openstack-nova | 13:52 | |
| openstackgerrit | sean mooney proposed openstack/nova-specs master: [WIP] generic device discovery policy https://review.openstack.org/603805 | 13:53 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Return a minimal construct for nova list when a cell is down https://review.openstack.org/567785 | 13:55 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Add scatter-gather-single-cell utility https://review.openstack.org/594947 | 13:55 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Resource retrieving: add changes-before filter https://review.openstack.org/599276 | 13:57 |
| *** janki has joined #openstack-nova | 13:59 | |
| openstackgerrit | Jay Pipes proposed openstack/os-traits master: clean up CUDA traits https://review.openstack.org/597170 | 14:04 |
| *** janki has quit IRC | 14:04 | |
| *** mlavalle has joined #openstack-nova | 14:04 | |
| *** munimeha1 has quit IRC | 14:09 | |
| *** awaugama has joined #openstack-nova | 14:12 | |
| *** Luzi has quit IRC | 14:19 | |
| *** cfriesen has joined #openstack-nova | 14:22 | |
| mriedem | dansmith: still humping approved stable changes through the gate, fyi | 14:23 |
| dansmith | ack | 14:23 |
| *** alexchadin has quit IRC | 14:25 | |
| *** s10 has quit IRC | 14:25 | |
| *** alexchadin has joined #openstack-nova | 14:30 | |
| *** gouthamr_ is now known as gouthamr | 14:35 | |
| *** udesale has quit IRC | 14:37 | |
| *** hamdyk has quit IRC | 14:38 | |
| mriedem | jaypipes: i'm +2 on the 2.66 changes-before change now https://review.openstack.org/#/c/599276/ | 14:39 |
| jaypipes | mriedem: yeah, I +W'd that a while ago :) | 14:39 |
| jaypipes | mriedem: "a while ago" == whole minutes ago. | 14:39 |
| mriedem | ah heh | 14:40 |
| mriedem | cool | 14:40 |
| *** alexchadin has quit IRC | 14:43 | |
| mriedem | gmann: is https://review.openstack.org/#/c/596285/ the last change for https://blueprints.launchpad.net/nova/+spec/api-extensions-merge-stein ? | 14:48 |
| *** munimeha1 has joined #openstack-nova | 14:49 | |
| cdent | mnaser: I think you mentioned you had a huge database that you could test a nova->placement migration script on, to get a sense of timing and the like? If that's true dansmith has made a thing: https://review.openstack.org/#/c/603234/ | 14:51 |
| mnaser | yup, i can try that out | 14:51 |
| mriedem | looks like it is, the only other extension with @wsgi.extends for servers is hide_server_addresses and i think that's deprecated? | 14:51 |
| dansmith | mnaser: it just dumps/imports, so it should be safe to run against a live source deployment.. it doesn't delete anything from the source | 14:51 |
| dansmith | but also, you could just measure the size of the tables and have a pretty good idea anyway | 14:52 |
| lbragstad | gmann is https://review.openstack.org/#/c/547850/ still being pursued? | 14:52 |
| *** marvin_mhg has quit IRC | 14:53 | |
| *** pvc has quit IRC | 14:55 | |
| mriedem | melwitt: btw, if you don't like geddy's voice, it's no crime to enjoy https://www.youtube.com/watch?v=iB4uwO1Dmf4 which is purely instrumental | 14:56 |
| *** alexchadin has joined #openstack-nova | 14:57 | |
| johnthetubaguy | gmann: good question, he did say he was planning on doing something about that | 14:59 |
| *** marvin_mhg has joined #openstack-nova | 15:00 | |
| mnaser | so looks like the biggest table is consumers | 15:01 |
| mnaser | i tried it on a relatively large ~100ish deployment and it took 2 minutes with a triple replicated galera cluster with pretty beefy controllers | 15:02 |
| bauzas | stupid question, but say I'm migrating some libvirt instance and would like to introspect its domain XML, should I lookup the source or the destination ? (in other words, shall I look at the migration allocation or the instance allocation) ? | 15:02 |
| *** janki has joined #openstack-nova | 15:03 | |
| mnaser | bauzas: afaik the domain xml should be the same for live migrations at least afaik | 15:03 |
| bauzas | for the moment, this is going to be a TODO since we don't support migrations for VGPU, but I need to make sure the code I'm writing for the reshape works | 15:03 |
| mnaser | dansmith: ^ fyi for some numbers, it's relatively painless comparing to something like.. cells v2 stuff | 15:03 |
| bauzas | mnaser: yup, but then the allocated PCI devices could differ | 15:03 |
| mnaser | bauzas: ah yes. that's a qurik i didn't think of | 15:04 |
| dansmith | mnaser: ack | 15:04 |
| mnaser | It would be nice if it could pull in environment variables | 15:04 |
| mnaser | But we could iterate on that later | 15:04 |
| mnaser | So we don’t have to write credentials on disk when we don’t have to | 15:05 |
| dansmith | mnaser: you should be able to do that as-is | 15:05 |
| dansmith | write an empty config file and make those variables be in the environment already | 15:06 |
| *** janki has quit IRC | 15:08 | |
| openstackgerrit | Merged openstack/nova master: Fix some typos in nova api ref doc https://review.openstack.org/603306 | 15:08 |
| *** tbachman has quit IRC | 15:08 | |
| mriedem | bauzas: the answer depends on what you need to know, | 15:10 |
| mriedem | if you're cleaning up something on the source during live migratoin, then look at the source xml, else look at the dest xml | 15:10 |
| bauzas | mriedem: no, I just want to reshape the existing allocations onto the right physical device | 15:10 |
| bauzas | mriedem: for that, I need to get the corresponding mediated devices | 15:11 |
| mriedem | umm, | 15:11 |
| mriedem | reshape *during* a live migration? | 15:11 |
| dansmith | mnaser: I can make the db and host things not clobber environment too and I guess add a flag that will allow the file to be missing or something | 15:11 |
| dansmith | credentials in environment is easy, but not more secure necessarily, so I guess I'm not sure why that's better, but alas :) | 15:12 |
| mriedem | bauzas: at the start of live migration, conductor moves the existing allocations from the instance to the migration record, and then the scheduler is going to allocate resources from the dest (tree) for the instance | 15:12 |
| mriedem | so i'm not sure why you'd need to reshape at all | 15:12 |
| bauzas | mriedem: I know about how we manage allocations during a migration | 15:13 |
| dansmith | mriedem: assume he means a symbolic reshape of allocations right? | 15:13 |
| dansmith | not an actual POST /reshaper operation | 15:13 |
| bauzas | the case I'm concerned is a cold migrate (because live migrating vGPUs is YAGNI) | 15:13 |
| mriedem | my point is, why? | 15:13 |
| dansmith | but a self-healing given the opportunity provided by a move | 15:13 |
| bauzas | right | 15:13 |
| dansmith | I dunno, I don't know what he's trying to heal exactly | 15:13 |
| bauzas | so, say cold migrate is a thing | 15:13 |
| mriedem | you can't heal a broken heart | 15:13 |
| bauzas | I mean, cold migrate a VGPU | 15:14 |
| bauzas | (which is not something there yet, but a bugfix I have targeted for Stein) | 15:14 |
| bauzas | then cold migrating the instance would mean 2 allocations for this | 15:14 |
| *** luksky11 has quit IRC | 15:15 | |
| bauzas | one having the consumer UUID being the migration UUID on the source host, the other one being the real allocation | 15:15 |
| bauzas | right? | 15:15 |
| mriedem | and during the cold migrate you want to move the existing vgpu allocation from the root node provider on the source host to the child vgpu provider on the dest node? | 15:15 |
| mriedem | the scheduler should take care of the latter | 15:15 |
| mriedem | and after we confirm the resize, we'll drop the former | 15:15 |
| mriedem | so i'm not sure why it matters | 15:15 |
| bauzas | mriedem: no, I'd then just move the allocation on the source host to be on the right vGPU provider on the same host | 15:15 |
| dansmith | wait what/ | 15:16 |
| *** munimeha1 has quit IRC | 15:16 | |
| dansmith | we will do the same migration-holds-source-allocation yeah? | 15:16 |
| bauzas | that's what I think yeah | 15:16 |
| * dansmith is confused | 15:16 | |
| mriedem | me too | 15:17 |
| mriedem | i don't see the issue | 15:17 |
| *** udesale has joined #openstack-nova | 15:17 | |
| bauzas | I'm probably confused too by what means a reshape | 15:17 |
| mriedem | it sounds like you're trying to auto-heal/reshape duringa cold migrate so we don't have to run the maybe more expensive reshape-on-compute-startup upgrade thing | 15:17 |
| bauzas | during the migration, we have 2 allocations on two different hosts, right? | 15:17 |
| bauzas | ah shit, that's what I fucked | 15:18 |
| dansmith | um, what? :) | 15:18 |
| dansmith | lol. | 15:18 |
| mriedem | maybe you want to just block az renames while the az has instances? | 15:18 |
| mriedem | that's probably easier | 15:18 |
| bauzas | can we have a migration running, and both services to be restarted ? | 15:18 |
| mriedem | sure | 15:18 |
| bauzas | in that case, it would trigger a reshape non? | 15:18 |
| mriedem | sure | 15:18 |
| bauzas | for both ? | 15:18 |
| bauzas | so in that interim period, we have two allocations, nope ? | 15:19 |
| *** alexchadin has quit IRC | 15:19 | |
| mriedem | if the allocations have moved from the instance to the migration record on the source host and the source host is restarted and a reshape happens, we should still move the allocations for the migration record | 15:19 |
| mriedem | the reshape in this case happens on distinct provider trees | 15:19 |
| bauzas | that's my point | 15:19 |
| bauzas | so in that case, I have a migration UUID that is the consumer | 15:20 |
| bauzas | on the source host | 15:20 |
| mriedem | if the resize is confirmed, we drop the source node tree allocations for the migration record, | 15:20 |
| mriedem | on revert we drop the target node tree allocations for the instance and move them back to the instance for the source tree | 15:20 |
| mriedem | but the reshape should be ok | 15:20 |
| bauzas | ok, but then I have to figure out the real instance UUID for the migration record | 15:20 |
| bauzas | that's a special case | 15:21 |
| mriedem | the migration consumer is just a fill in so someone doesn't quack quack seat back the source node resources | 15:21 |
| mriedem | honestly i don't know what problem you're trying to solve | 15:21 |
| mriedem | we should probably identify that a problem exists before discussing designs on how to fix it | 15:21 |
| bauzas | probably | 15:21 |
| bauzas | and I'm unclear | 15:22 |
| bauzas | I'll just leave a comment in my patch and I move on | 15:22 |
| dansmith | mnaser: how much of your two minutes is writing and reading that dumpfile on disk? would you prefer we just pipe it between the export and import (optionally?) and maybe tee it out for forensics? might be faster | 15:28 |
| *** dtantsur is now known as dtantsur|brb | 15:28 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove deprecated hide_server_address_states option https://review.openstack.org/603831 | 15:30 |
| mriedem | gmann: ^ | 15:31 |
| mnaser | dansmith: like 3-5 seconds for the dump. The write took the longest | 15:34 |
| dansmith | mnaser: 3-5 seconds overhead for writing the file? or 3-5 seconds to do the dump and the rest of the minute to write it out? | 15:35 |
| dansmith | on my tiny database it's immeasurable of course | 15:36 |
| dansmith | I get a constant 3.6s to do the dump and import regardless | 15:36 |
| dansmith | mnaser: I have a diff that does it pipely if you want to try it | 15:36 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove deprecated hide_server_address_states option https://review.openstack.org/603831 | 15:36 |
| *** gyee has joined #openstack-nova | 15:37 | |
| dansmith | mnaser: https://termbin.co/FXJg | 15:37 |
| dansmith | I also put a "tee $tmpfile |" in the middle of that pipe (hence the comment) but removed it for your test | 15:38 |
| *** helenafm has quit IRC | 15:40 | |
| *** tbachman has joined #openstack-nova | 15:44 | |
| *** dtantsur|brb is now known as dtantsur | 15:46 | |
| *** pcaruana has joined #openstack-nova | 15:47 | |
| openstackgerrit | Ben Nemec proposed openstack/nova master: WIP: Migrate upgrade checks to oslo.upgradecheck https://review.openstack.org/603499 | 15:56 |
| dansmith | tssurya: mriedem shall we meet about cells? we just talked about things last week and melwitt is not around this week | 16:00 |
| mnaser | dansmith: sorry i was in a meeting, the dump was really quick but the push was much slower because of (what i assume) replication | 16:01 |
| dansmith | ah the load, I see | 16:01 |
| dansmith | yeah I'm sure that's the heavy part | 16:02 |
| dansmith | it's heavy on my toy devstack even | 16:02 |
| dansmith | in that case, I'd just say we should leave separate the dump/load like it is now so that one can complete without the other, leaving a file you can just import if you want it | 16:03 |
| *** mdbooth_ has joined #openstack-nova | 16:04 | |
| jaypipes | stephenfin: so, unless I'm mis-reading your comments on the cpu-resource-tracking spec, you'd actually favor an "opt-in" approach to hyperthread usage. Is that correct? i.e. the virt driver defaults to *not* counting hyperthreads as CPU resources for guests unless some knob is turned on? | 16:05 |
| *** mdbooth has quit IRC | 16:05 | |
| stephenfin | jaypipes: For instances with dedicated CPUs, yes. However, we need to avoid breaking users | 16:06 |
| stephenfin | So that knob would have to default to on | 16:06 |
| stephenfin | The reason I want that is because a hyperthread != a core and I think the decision to model it as one was a mistake on day one | 16:06 |
| jaypipes | stephenfin: ack. no disagreement from me on that | 16:06 |
| jaypipes | stephenfin: agreed. | 16:07 |
| *** mdbooth_ has quit IRC | 16:10 | |
| openstackgerrit | sean mooney proposed openstack/nova-specs master: [WIP] generic device discovery policy https://review.openstack.org/603805 | 16:10 |
| *** mdbooth_ has joined #openstack-nova | 16:18 | |
| openstackgerrit | Merged openstack/nova stable/queens: Fix DB archiver AttributeError due to wrong table name attribute used https://review.openstack.org/599882 | 16:20 |
| openstackgerrit | Merged openstack/nova stable/queens: VMware: fix TypeError while get console log https://review.openstack.org/591365 | 16:20 |
| openstackgerrit | Merged openstack/nova stable/queens: Move conductor wait_until_ready() delay before manager init https://review.openstack.org/599200 | 16:20 |
| *** mdbooth_ has quit IRC | 16:22 | |
| *** dtantsur is now known as dtantsur|afk | 16:23 | |
| *** sahid has quit IRC | 16:24 | |
| *** pcaruana has quit IRC | 16:27 | |
| *** udesale has quit IRC | 16:28 | |
| mriedem | dansmith: i'm good to skip; plan on starting my poc for cross-cell resize today | 16:28 |
| dansmith | oh boy | 16:29 |
| openstackgerrit | Merged openstack/nova stable/ocata: Return 400 when compute host is not found https://review.openstack.org/590649 | 16:30 |
| stephenfin | jaypipes: If you don't have one already, a placement cheatsheet (in PDF and therefore printable) would be super useful | 16:31 |
| jaypipes | stephenfin: also, I agree with you about getting rid of emulator_threads_policy eventually, but is it 100% critical that I update the cpu-resource-tracking spec with information about that? | 16:31 |
| jaypipes | stephenfin: you mean cheatsheet about the REST API? | 16:32 |
| jaypipes | stephenfin: or something else? | 16:32 |
| jaypipes | stephenfin: or you mean a glossary of placement terms? | 16:32 |
| stephenfin | jaypipes: No, it can be be handled separately. However, I imagine you do need to take cpu overhead, which is incremented for each dedicated emulator thread core, into account | 16:33 |
| openstackgerrit | Jack Ding proposed openstack/nova master: Correct instance port binding for rebuilds/reboots https://review.openstack.org/603844 | 16:33 |
| stephenfin | jaypipes: Both. Glossary of terms followed by examples of either the REST API or using osc-placement | 16:33 |
| jaypipes | stephenfin: ack | 16:34 |
| stephenfin | jaypipes: Just an idea, obviously. I might even start drafting something myself some point this week | 16:34 |
| jaypipes | stephenfin: cool. let me know when you get to the "what is a consumer?" part... | 16:34 |
| *** holser__ has quit IRC | 16:36 | |
| stephenfin | jaypipes: FYI, the reason I brought up the whole "let's kill 'isolate' idea" was that that approach seemed easier than handling the CPU overhead thing *and* it reduced complexity for the user in the process, which feels like one of the goals | 16:36 |
| stephenfin | But again, not a blocker :) | 16:37 |
| *** panda has joined #openstack-nova | 16:38 | |
| jaypipes | stephenfin: yes, agreed. glad it's not a blocker though. | 16:39 |
| openstackgerrit | Sylvain Bauza proposed openstack/nova master: libvirt: mdevs returning parent and vendor PCI info https://review.openstack.org/562304 | 16:39 |
| bauzas | jaypipes: stephenfin: can one of you reapprove https://review.openstack.org/562304 ? I just rebased it out of the existing series and I need it for the reshaper change | 16:40 |
| bauzas | I'm surprised I lost my +2s on the rebase but meh | 16:40 |
| bauzas | probably because I changed the commit id | 16:41 |
| bauzas | oh yeah, I changed the message, hence why | 16:41 |
| jaypipes | stephenfin: feel free to re-+W bauzas patch | 16:41 |
| bauzas | jaypipes: thanks | 16:41 |
| bauzas | I need the parenting relationship for reshaping the VGPU resources to the right PGPU RP | 16:42 |
| stephenfin | bauzas: Done | 16:42 |
| bauzas | stephenfin: ack, thanks | 16:43 |
| *** derekh has quit IRC | 16:51 | |
| *** rabel has joined #openstack-nova | 16:53 | |
| *** gbarros has quit IRC | 16:53 | |
| openstackgerrit | Jay Pipes proposed openstack/nova-specs master: Standardize CPU resource tracking https://review.openstack.org/555081 | 17:01 |
| tobias-urdin | ping on a possible upgrade path bug q->r https://bugs.launchpad.net/nova/+bug/1793353 | 17:09 |
| openstack | Launchpad bug 1793353 in OpenStack Compute (nova) "broken upgrade path q->r requirement for oslo.db" [Undecided,New] | 17:09 |
| *** ralonsoh has quit IRC | 17:19 | |
| *** alex_xu has joined #openstack-nova | 17:26 | |
| *** sambetts is now known as sambetts|afk | 17:27 | |
| *** jiteka has joined #openstack-nova | 17:29 | |
| *** jpena is now known as jpena|off | 17:30 | |
| openstackgerrit | David Rabel proposed openstack/nova master: Really use source image format as default for snapshot_image_format https://review.openstack.org/603855 | 17:37 |
| jiteka | hello, could someone confirm that migrate feature is working with flavor using vCPU pinning and Numa in Mitaka ? | 17:51 |
| *** munimeha1 has joined #openstack-nova | 17:53 | |
| *** gbarros has joined #openstack-nova | 18:00 | |
| *** gbarros has quit IRC | 18:01 | |
| *** sapd1 has quit IRC | 18:01 | |
| openstackgerrit | Merged openstack/nova-specs master: Propose configurable maximum number of volumes to attach https://review.openstack.org/597306 | 18:01 |
| mriedem | jiteka: cfriesen might know | 18:10 |
| mriedem | there were some patches that mellanox and windriver worked on to make that work, but i can't remember which release in which those patches landed | 18:11 |
| mriedem | jiteka: cold migrate i mean, not live migration | 18:11 |
| mriedem | numa/pinned cpus does not work with live migration | 18:11 |
| mriedem | https://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/numa-aware-live-migration.html | 18:11 |
| *** tbachman has quit IRC | 18:11 | |
| *** claudiub has quit IRC | 18:14 | |
| *** gbarros has joined #openstack-nova | 18:16 | |
| openstackgerrit | Merged openstack/nova stable/ocata: Add recreate test for RT.stats bug 1784705 https://review.openstack.org/588076 | 18:28 |
| openstack | bug 1784705 in OpenStack Compute (nova) ocata "ResourceTracker.stats can leak across multiple ironic nodes" [High,In progress] https://launchpad.net/bugs/1784705 - Assigned to Matt Riedemann (mriedem) | 18:28 |
| *** tbachman has joined #openstack-nova | 18:30 | |
| mriedem | dansmith: would be good to get this pike backport in https://review.openstack.org/#/c/599883/ | 18:31 |
| dansmith | mriedem: hmm, did I do that? ISTR some reasoning there.. maybe sqla version? | 18:32 |
| mriedem | it was originally your code yes, but table.name is standard | 18:32 |
| *** gbarros has quit IRC | 18:36 | |
| mriedem | dansmith: does anything jump out at you for the source of the IndexError in this failure? http://logs.openstack.org/72/600372/1/gate/openstack-tox-py35/dcfd363/job-output.txt.gz | 18:44 |
| mriedem | looks like something between oslo.db/sqla/pymysql/eventlet switches context and we timeout after some huge amount of time | 18:44 |
| mriedem | nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_models_sync [664.512994s] ... FAILED | 18:45 |
| mriedem | nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_models_sync [39.644814s] ... ok | 18:45 |
| mriedem | clearly we're not just missing some short timeout window | 18:45 |
| dansmith | still waiting for the damn log to load | 18:46 |
| dansmith | wow | 18:48 |
| mriedem | yeah we have a lot of these failures in the gate across master, rocky and queens, | 18:48 |
| mriedem | https://bugs.launchpad.net/cinder/+bug/1793364 | 18:48 |
| openstack | Launchpad bug 1793364 in OpenStack Compute (nova) "mysql db opportunistic unit tests timing out intermittently in the gate (bad thread switch?)" [High,Confirmed] | 18:48 |
| mriedem | nova and cinder b/c they are using the same test fixtures | 18:49 |
| dansmith | hmm | 18:49 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Add contributor guide for upgrade status checks https://review.openstack.org/596902 | 18:50 |
| mriedem | maybe "connection.scalar(select([1]))"? | 18:50 |
| dansmith | does the index complaint surprise you? because it looks like a nonsense name | 18:50 |
| mriedem | well i see this too | 18:51 |
| mriedem | sqlalchemy.exc.ResourceClosedError: This result object does not return rows. It has been closed automatically | 18:51 |
| dansmith | yeah | 18:51 |
| mriedem | http://status.openstack.org/elastic-recheck/data/integrated_gate.html Overall Categorization Rate: 15.4% | 18:52 |
| mriedem | meaning we (openstack) have a shit load of uncategorized failures killing stuff in the gate | 18:52 |
| mriedem | which is why it's taking us days to merge code | 18:52 |
| *** BlackDex has quit IRC | 18:57 | |
| mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22%20%20connection.scalar(select(%5B1%5D))'%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d | 18:57 |
| *** BlackDex has joined #openstack-nova | 18:58 | |
| cfriesen | jiteka: I think cold migration should work in mitaka. I think there are some gotchas where resource tracking isn't accurate until the next audit, and maybe there are some resource tracking issues if you revert a resize. | 19:04 |
| *** gbarros has joined #openstack-nova | 19:04 | |
| *** munimeha1 has quit IRC | 19:05 | |
| tssurya | dansmith: ack, let's skip | 19:06 |
| cfriesen | jiteka: a bit more digging shows that PCI devices were not cold migratable until Newton. | 19:06 |
| mriedem | cfriesen: did you find a specific change that makes that work? | 19:07 |
| mriedem | b/c we could make a note of that on https://docs.openstack.org/nova/latest/admin/pci-passthrough.html | 19:07 |
| cfriesen | mriedem: 5129f48226c I think | 19:09 |
| cfriesen | actually, looks like there are multiple: https://bugs.launchpad.net/nova/+bug/1512880 | 19:10 |
| openstack | Launchpad bug 1512880 in OpenStack Compute (nova) newton "Failed cold migration with SR-IOV" [Medium,Fix released] | 19:10 |
| mriedem | yeah was just going to say that https://review.openstack.org/#/q/topic:bug/1512880+(status:open+OR+status:merged) | 19:10 |
| mriedem | i think it's fair to say it didn't work until newton | 19:11 |
| *** lucidguy has joined #openstack-nova | 19:27 | |
| lucidguy | I believe this bug applies to me, unfortunately I don't know how to resolve, assistance? https://bugs.launchpad.net/tripleo/+bug/1785568 | 19:28 |
| openstack | Launchpad bug 1785568 in OpenStack Compute (nova) "Multiple migration requests for same vm might fail" [Undecided,Incomplete] | 19:28 |
| mriedem | cfriesen: so if you have a minute, could you just update the note at the top of https://docs.openstack.org/nova/latest/admin/pci-passthrough.html about sriov to also mention that cold migration of servers with sriov ports attached didn't work until newton and reference that bug? | 19:29 |
| *** awaugama has quit IRC | 19:38 | |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Return a minimal construct for nova show when a cell is down https://review.openstack.org/591658 | 19:42 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Return a minimal construct for nova service-list when a cell is down https://review.openstack.org/584829 | 19:46 |
| *** tssurya has quit IRC | 19:51 | |
| openstackgerrit | Merged openstack/nova-specs master: fix spelling mistake https://review.openstack.org/573479 | 20:12 |
| *** erlon has quit IRC | 20:24 | |
| *** cdent has quit IRC | 20:27 | |
| cfriesen | anyone know anything about "bad magic number in nova.conf.crypto" when trying to build nova docs? | 20:32 |
| cfriesen | http://paste.openstack.org/show/730383/ | 20:33 |
| mriedem | i've seen that i think... | 20:35 |
| *** hemna_ has quit IRC | 20:36 | |
| mriedem | py34 isn't supported... | 20:37 |
| mriedem | not sure if that would be related | 20:37 |
| mriedem | also, | 20:37 |
| mriedem | nova.conf.crypto isn't in master | 20:37 |
| mriedem | is that a starlingx special? | 20:38 |
| mriedem | oh pike | 20:38 |
| mriedem | https://github.com/openstack/nova/blob/stable/pike/nova/conf/crypto.py | 20:38 |
| cfriesen | my regular "python" is 2.7.5. not sure why tox would use 3.4 unless it's searching for whether it's present. I'm on the master branch of upstream nova, though I have checked out pike in this repo. | 20:39 |
| *** priteau has quit IRC | 20:39 | |
| *** dklyle has quit IRC | 20:47 | |
| cfriesen | found a nova/conf/crypto.pyc file left over from a previous branch. deleted and retrying | 20:51 |
| *** gbarros has quit IRC | 20:53 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Mention SR-IOV cold migration limitation in admin docs https://review.openstack.org/603909 | 20:54 |
| mriedem | cfriesen: huh, i thought we used to have a thing in all tox runs that would clean out pycs | 20:55 |
| mriedem | but i also seem to remember someone removing that years ago | 20:55 |
| mriedem | oh right it's in the base testenv, | 20:55 |
| mriedem | but commands is overridden in the actual testenv:docs run | 20:55 |
| mriedem | cfriesen: oh were you building those docs ^ | 20:56 |
| mriedem | for the thing i just pushed? | 20:56 |
| cfriesen | heh...yep | 20:58 |
| mriedem | sorry, you didn't say anything so i assumed you were busy with something else | 20:58 |
| cfriesen | no worries. | 20:58 |
| openstackgerrit | Mohammed Naser proposed openstack/nova master: Default zero disk flavor to RULE_ADMIN_API in Stein https://review.openstack.org/603910 | 20:59 |
| cfriesen | mriedem: I almost feel like we should use PCI in there instead of SR-IOV, since it's the more generic form. Not all PCI passthrough is SR-IOV, but all SR-IOV is a form of PCI passthrough. | 21:00 |
| openstackgerrit | Mohammed Naser proposed openstack/nova master: Drop migration pre-check error for file_backed_memory https://review.openstack.org/603911 | 21:02 |
| mnaser | mriedem: ^ i thought we were in openstack-nova oops | 21:02 |
| cfriesen | mnaser: you are. :) | 21:02 |
| mnaser | oh no i was talking in #o-operators | 21:03 |
| mnaser | pushing stuffa nd wondering why the hell the bot wasn't notifying | 21:03 |
| cfriesen | that'll do it | 21:03 |
| mriedem | cfriesen: feel free to -1 my change and break my heart | 21:04 |
| mnaser | thats my nova commit for the cycle | 21:05 |
| mriedem | mnaser: i'm guessing you didn't run this through tests yet right? | 21:05 |
| * mnaser & | 21:05 | |
| cfriesen | mriedem: I commented on it | 21:05 |
| mnaser | mriedem: nope | 21:05 |
| mnaser | i probably should | 21:05 |
| mnaser | well it looks like tests should pass because the tests use a fixture of using RULE_ADMIN_API and making sure it fails | 21:07 |
| mnaser | so it was testing the then-future case | 21:07 |
| mriedem | mnaser: at least EnforceVolumeBackedForZeroDiskFlavorTestCase in tox -e functional | 21:07 |
| mnaser | mriedem: running | 21:08 |
| mriedem | comments inline | 21:09 |
| mnaser | lemme address after seeing if test breaks | 21:09 |
| openstackgerrit | Mohammed Naser proposed openstack/nova master: Default zero disk flavor to RULE_ADMIN_API in Stein https://review.openstack.org/603910 | 21:15 |
| *** _pewp_ has quit IRC | 21:18 | |
| *** cseader has quit IRC | 21:19 | |
| *** _pewp_ has joined #openstack-nova | 21:19 | |
| mriedem | lbragstad: are dashes in policy rule names frowned upon? | 21:19 |
| mriedem | e.g. os_compute_api:servers:cross-cell-resize? | 21:19 |
| mriedem | should be os_compute_api:servers:cross_cell_resize? | 21:19 |
| lbragstad | i don't think so | 21:19 |
| lbragstad | but when i went through a bunch of the projects last week there isn't really a standard | 21:20 |
| mriedem | shocking | 21:20 |
| lbragstad | inoright? | 21:20 |
| lbragstad | so - i guess if we're going to try and do this, i'd like to whack all the moles at once... | 21:20 |
| mriedem | since the prefix is os_compute_api i assume suffix should also be underscores | 21:20 |
| mriedem | oh and we have things like network:attach_external_network and create:zero_disk_flavor so nvm | 21:21 |
| mriedem | answered my own question | 21:21 |
| lbragstad | did you see my note about using the service name? | 21:21 |
| mriedem | no, where? | 21:21 |
| * lbragstad fetches link | 21:21 | |
| mriedem | i have'nt read the ML thread | 21:21 |
| lbragstad | http://lists.openstack.org/pipermail/openstack-dev/2018-September/134597.html | 21:22 |
| lbragstad | it starts there... | 21:22 |
| lbragstad | sorry if you we waiting to parse the lists later :) | 21:22 |
| mriedem | :( | 21:22 |
| lbragstad | but i'm trying to update that with all context of what i've found across projects (since operators are on that note, too) | 21:22 |
| mriedem | so you'd like to see compute:server:cross_cell_resize | 21:24 |
| mriedem | note this is an action, | 21:24 |
| mriedem | so it won't cleanly map to a method | 21:25 |
| lbragstad | right | 21:25 |
| lbragstad | which i guess is another reason to not put http methods in policy names? | 21:25 |
| mriedem | i mean, we could do compute:servers:resize:cross_cell | 21:25 |
| lbragstad | yeah - i don't think nova would be super special in that case... there are other projects have more than just <service>:<resource>:<action> | 21:26 |
| lbragstad | ironic and heat for example | 21:26 |
| mriedem | alright, this is months/releases away from landing so i'll just do that and move on | 21:27 |
| mriedem | this is the amount of progress i've made on this since last week | 21:27 |
| lbragstad | what's months away from landing? | 21:27 |
| mriedem | cross-cell resize | 21:27 |
| lbragstad | oh - sure | 21:28 |
| mriedem | not your thing | 21:28 |
| mriedem | being cross project, your thing is *years* away from landing :) | 21:28 |
| lbragstad | i'd like to get the convention "established" as soon as possible, because I think it's going to affect gmann's work too | 21:28 |
| lbragstad | right - let's be real here | 21:28 |
| mnaser | is there a documented process for when you end up with: "Instance 13a09ff5-7124-4b3c-8229-a8691a2f43b4 has allocations against this compute host but is not found in the database." | 21:41 |
| mnaser | context: dns was borked on a compute, live migration would complete, but the placement migration allocation thing fails because compute cant talk to placement | 21:41 |
| *** dklyle has joined #openstack-nova | 21:42 | |
| mriedem | that doesn't sound like a scenario in which you'd see that message | 21:43 |
| mnaser | well, it happened after fixing dns | 21:44 |
| mnaser | the compute node finally started talking to placement properly and now complains forever | 21:44 |
| mriedem | is that instance in the db? | 21:44 |
| mnaser | hmm | 21:44 |
| mriedem | the code calls out 2 cases that could happen: | 21:45 |
| mnaser | no, i think the exception i was seeing was unrelated | 21:45 |
| mnaser | i think it was no placement + vm deleted | 21:45 |
| mnaser | i think i jumped to a conclusion there | 21:45 |
| mriedem | 1. compute RT is racing with the scheduler where the scheduler created allocations in placement but didn't yet create the instance in the cell db, | 21:45 |
| mriedem | 2. the instance was deleted and archived/purged from the db, but the alloctions are still in placement for that node | 21:45 |
| mriedem | yeah, | 21:45 |
| mriedem | so if the compute couldn't talk to placement when the vm was deleted, we'd fail to cleanup the allocations | 21:46 |
| mriedem | https://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-allocation-delete | 21:46 |
| mnaser | thanks for making life easy and writing those clie stuf | 21:46 |
| mriedem | openstack resource provider allocation delete 13a09ff5-7124-4b3c-8229-a8691a2f43b4 | 21:46 |
| mriedem | if the instance is truly deleted | 21:46 |
| mriedem | if it were moved and allocations were just messed up, doing ^ would be bad | 21:47 |
| mnaser | ill double check they're all deleted | 21:47 |
| mriedem | might be good to update that log message with the same info | 21:47 |
| mnaser | tail -1000 /var/log/nova/nova-compute.log | grep 'has allocations against this compute host but is not found in the database.' | cut -d' ' -f13 | sort | uniq | xargs -n1 echo nova show | 21:47 |
| mnaser | magic | 21:47 |
| mriedem | "If you continue to see this message and have confirmed the instance is truly gone from the database, you can run 'openstack resource provider allocation delete <uuid>' to remove the allocation from the placement service." | 21:48 |
| mnaser | yeah just even document it somewhere i guess | 21:48 |
| mnaser | i mean | 21:48 |
| mriedem | well, | 21:48 |
| mnaser | couldnt nova confirm that it doesnt exist in the db | 21:48 |
| mriedem | not if it's a race | 21:49 |
| mnaser | and technically just delete it when it polls | 21:49 |
| mnaser | ah | 21:49 |
| mnaser | i guess we'd have to check N times that it's not there but that starts becoming silly i guess | 21:49 |
| mriedem | as noted, we create the allocations in scheduling before the instance is created in the cell db | 21:49 |
| mriedem | right | 21:49 |
| mnaser | or if its been running for a while | 21:49 |
| mriedem | putting it into some kind of faq could work, but people might not see that and come here asking the same thing | 21:49 |
| mnaser | like 300s age | 21:49 |
| mnaser | 300s isn't a race condition at that point i guess | 21:50 |
| mnaser | but i guess log is the easiest path | 21:50 |
| mriedem | rather than nova auto-delete your stuff, i'd prefer to just document weird edge cases for manual intervention | 21:50 |
| mriedem | we've also talked about (last week it came up too) a command to compare placement allocations against nova instances, | 21:51 |
| mriedem | the opposite of heal_allocations, | 21:51 |
| *** takashin has joined #openstack-nova | 21:51 | |
| mriedem | more of a "tell me what garbage is in the placement db b/c nova didn't clean up after itself" | 21:51 |
| mriedem | but we can't reliably do that until we have consumer types in placement... | 21:51 |
| mriedem | takashin: is this todo still valid https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3509 given we check the specified host is in the same cell above? https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3405 | 21:54 |
| mriedem | i think we can remove that todo; when we call objects.ComputeNode.get_first_node_by_host_for_old_compat the context is targeted to the cell in which the instance currently exists | 21:55 |
| mriedem | so if the specified host is in another cell, we'll get ComputeHostNotFound there | 21:56 |
| *** owalsh has joined #openstack-nova | 21:57 | |
| *** owalsh has quit IRC | 21:58 | |
| mriedem | johnthetubaguy: we should remove this rebuild parameter from the conductor migrate_server task API :) https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L263 | 21:58 |
| mriedem | that must have been something someone talked about way back in 2013 to unify the various move APIs | 21:59 |
| *** vishwanathj has joined #openstack-nova | 22:02 | |
| takashin | mriedem: you are right. We can remove the 'todo'. | 22:10 |
| mriedem | takashin: please push a change to remove it and i'll +2 | 22:10 |
| takashin | mriedem: okay. I will submit the patch. | 22:11 |
| vishwanathj | hi looking for guidance on how to limit the number of instances on certain set of hosts... | 22:13 |
| vishwanathj | not clear on sequence of steps to be executed on controller and compute nodes to get this working | 22:15 |
| *** mlavalle has quit IRC | 22:22 | |
| mriedem | there is a NumInstancesFilter | 22:31 |
| mriedem | vishwanathj: https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#numinstancesfilter | 22:32 |
| mriedem | and https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#aggregatenuminstancesfilter for aggregates | 22:32 |
| openstackgerrit | Jack Ding proposed openstack/nova master: Correct instance port binding for rebuilds/reboots https://review.openstack.org/603844 | 22:33 |
| mriedem | there is a known bug with those filters https://bugs.launchpad.net/nova/+bug/1740320 | 22:33 |
| openstack | Launchpad bug 1740320 in OpenStack Compute (nova) "nova-scheduler does not honor max_instances_per_host set to a host aggregate" [Undecided,Confirmed] | 22:33 |
| mriedem | race bug | 22:34 |
| *** hoonetorg has quit IRC | 22:41 | |
| *** vishwanathj has quit IRC | 22:41 | |
| *** rcernin has joined #openstack-nova | 22:44 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove an unnecessary comment https://review.openstack.org/603924 | 22:47 |
| takashin | mriedem: I submitted https://review.openstack.org/#/c/603924 . | 22:49 |
| sorrison | Anyone know how to re kick a gate job that failed due to a timeout. https://review.openstack.org/#/c/526558/ I tried a recheck but that aint it | 22:57 |
| sorrison | mriedem: ^ | 22:57 |
| *** hoonetorg has joined #openstack-nova | 22:58 | |
| mriedem | sorrison: the gate has been eating a ton of dairy | 23:02 |
| mriedem | plug 526558 into http://zuul.openstack.org/ | 23:02 |
| mriedem | it's queued up | 23:03 |
| mriedem | http://lists.openstack.org/pipermail/openstack-dev/2018-September/134867.html | 23:03 |
| sorrison | ok thanks | 23:04 |
| *** mvkr has joined #openstack-nova | 23:10 | |
| mriedem | takashin: got it, thanks | 23:19 |
| *** slaweq has quit IRC | 23:19 | |
| takashin | mriedem: Thank you for your review. | 23:20 |
| *** dklyle has quit IRC | 23:29 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Cross-cell resize https://review.openstack.org/603930 | 23:40 |
| mriedem | ship it! | 23:40 |
| *** mriedem is now known as mriedem_away | 23:44 | |
| gmann | lbragstad: i will check the ML today and get back to you. | 23:52 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!