| *** mlavalle has quit IRC | 00:01 | |
| *** hamzy has quit IRC | 00:01 | |
| openstackgerrit | Brin Zhang proposed openstack/nova master: Need further updates, no need to review https://review.openstack.org/599276 | 00:06 |
|---|---|---|
| *** hamzy has joined #openstack-nova | 00:14 | |
| prometheanfire | any clue on this? https://gist.github.com/prometheanfire/d1d2d93d7b1c97e4186389b664301a80 | 00:38 |
| prometheanfire | rocky nova-compute is not starting :| | 00:38 |
| *** brinzhang has joined #openstack-nova | 00:39 | |
| *** brinzh has joined #openstack-nova | 00:39 | |
| *** gbarros has joined #openstack-nova | 00:44 | |
| *** hoangcx has joined #openstack-nova | 00:45 | |
| *** tbachman has quit IRC | 00:46 | |
| *** brinzh has quit IRC | 00:49 | |
| *** tbachman has joined #openstack-nova | 00:49 | |
| * prometheanfire gives up | 00:51 | |
| *** Dinesh_Bhor has quit IRC | 00:54 | |
| *** Nel1x has joined #openstack-nova | 00:57 | |
| *** hongbin has joined #openstack-nova | 00:58 | |
| *** med_ has quit IRC | 01:05 | |
| *** med_ has joined #openstack-nova | 01:06 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova-specs master: Make scheduling weight more granular https://review.openstack.org/599308 | 01:08 |
| *** Dinesh_Bhor has joined #openstack-nova | 01:16 | |
| *** med_ has quit IRC | 01:21 | |
| *** rcernin has quit IRC | 01:24 | |
| *** rcernin has joined #openstack-nova | 01:24 | |
| *** bzhao__ has joined #openstack-nova | 01:27 | |
| *** fried_rice is now known as efried | 01:27 | |
| *** med_ has joined #openstack-nova | 01:27 | |
| *** erlon has quit IRC | 01:31 | |
| *** tetsuro has joined #openstack-nova | 01:47 | |
| *** tetsuro has quit IRC | 01:53 | |
| *** Dinesh_Bhor has quit IRC | 02:03 | |
| *** dave-mccowan has quit IRC | 02:13 | |
| *** Dinesh_Bhor has joined #openstack-nova | 02:29 | |
| *** erlon has joined #openstack-nova | 02:38 | |
| *** psachin has joined #openstack-nova | 02:42 | |
| openstackgerrit | Naichuan Sun proposed openstack/nova master: xenapi(N-R-P): support compute node resource provider update https://review.openstack.org/521041 | 02:47 |
| *** jiapei has joined #openstack-nova | 02:55 | |
| openstackgerrit | fupingxie proposed openstack/nova master: Support list for alias in pci section in nova.conf https://review.openstack.org/592243 | 02:58 |
| openstackgerrit | fupingxie proposed openstack/nova master: Add an example to add more pci devices in nova.conf https://review.openstack.org/592243 | 03:01 |
| *** Nel1x has quit IRC | 03:14 | |
| *** erlon has quit IRC | 03:17 | |
| *** ykarel has joined #openstack-nova | 03:24 | |
| *** r-daneel has joined #openstack-nova | 03:34 | |
| *** ykarel has quit IRC | 03:37 | |
| *** ykarel has joined #openstack-nova | 03:51 | |
| *** Dinesh_Bhor has quit IRC | 03:59 | |
| *** Dinesh_Bhor has joined #openstack-nova | 04:01 | |
| *** psachin has quit IRC | 04:04 | |
| *** med_ has quit IRC | 04:06 | |
| *** psachin has joined #openstack-nova | 04:06 | |
| *** Dinesh_Bhor has quit IRC | 04:07 | |
| *** ykarel has quit IRC | 04:08 | |
| *** udesale has joined #openstack-nova | 04:12 | |
| *** gbarros has quit IRC | 04:13 | |
| openstackgerrit | jichenjc proposed openstack/nova master: WIP: add check for deleted flag https://review.openstack.org/599492 | 04:16 |
| openstackgerrit | jichenjc proposed openstack/nova master: Move str to six.string_types https://review.openstack.org/599493 | 04:21 |
| *** Bhujay has joined #openstack-nova | 04:33 | |
| *** ykarel has joined #openstack-nova | 04:37 | |
| *** Dinesh_Bhor has joined #openstack-nova | 04:39 | |
| *** janki has joined #openstack-nova | 04:42 | |
| *** hongbin has quit IRC | 04:58 | |
| *** kaliya has quit IRC | 05:15 | |
| *** stakeda has joined #openstack-nova | 05:20 | |
| openstackgerrit | garyk proposed openstack/nova master: Docs: update link for remote debugging https://review.openstack.org/591316 | 05:32 |
| *** lei-zh has joined #openstack-nova | 05:39 | |
| *** links has joined #openstack-nova | 05:41 | |
| *** pas-ha has quit IRC | 05:48 | |
| *** pas-ha has joined #openstack-nova | 05:48 | |
| *** dtroyer has quit IRC | 05:49 | |
| *** mugsie has quit IRC | 05:49 | |
| *** zigo has quit IRC | 05:49 | |
| *** dtroyer has joined #openstack-nova | 05:49 | |
| *** Luzi has joined #openstack-nova | 05:57 | |
| openstackgerrit | garyk proposed openstack/nova master: Docs: update link for remote debugging https://review.openstack.org/591316 | 06:03 |
| *** elod has joined #openstack-nova | 06:06 | |
| *** sridharg has joined #openstack-nova | 06:07 | |
| *** Dinesh_Bhor has quit IRC | 06:07 | |
| *** links has quit IRC | 06:09 | |
| *** Bhujay has quit IRC | 06:16 | |
| *** sahid has joined #openstack-nova | 06:20 | |
| *** hoonetorg has joined #openstack-nova | 06:24 | |
| *** psachin has quit IRC | 06:25 | |
| *** Dinesh_Bhor has joined #openstack-nova | 06:30 | |
| *** psachin has joined #openstack-nova | 06:30 | |
| *** rcernin has quit IRC | 06:33 | |
| *** lpetrut has joined #openstack-nova | 06:33 | |
| *** Bhujay has joined #openstack-nova | 06:38 | |
| *** luksky has joined #openstack-nova | 06:58 | |
| jiapei | Good afternoon novaers, I don't know if any of you have encounter such a problem when install stein by devstack. The problem is "ComputeHostNotFound_Remote". My procedures are: 1. Jenkins pull devstack-stein code, 2. Jenkins use ./stack.sh to install 3. install successfully 4. next Jenkins job coming, Jenkins use ./unstack.sh and ./clean.sh to unistall, then Jenkins use ./stack.sh to install 5. failed; The logs is | 07:02 |
| jiapei | http://paste.openstack.org/show/729382/ | 07:02 |
| *** Dinesh_Bhor has quit IRC | 07:03 | |
| *** lpetrut has quit IRC | 07:08 | |
| *** sridharg has quit IRC | 07:08 | |
| *** Dinesh_Bhor has joined #openstack-nova | 07:09 | |
| openstackgerrit | sahid proposed openstack/nova stable/rocky: libvirt: skip setting rx/tx queue sizes for not virto interfaces https://review.openstack.org/599506 | 07:09 |
| *** dtantsur|afk is now known as dtantsur | 07:26 | |
| *** ccamacho has joined #openstack-nova | 07:30 | |
| *** holser_ has joined #openstack-nova | 07:32 | |
| *** alexchadin has joined #openstack-nova | 07:33 | |
| *** ykarel is now known as ykarel|lunch | 07:40 | |
| *** moshele has joined #openstack-nova | 07:43 | |
| *** tssurya has joined #openstack-nova | 07:45 | |
| *** gibi has joined #openstack-nova | 07:45 | |
| *** jpena|off is now known as jpena | 07:45 | |
| kashyap | stephenfin: Morning, thanks for the outputs; looking now | 07:46 |
| *** helenafm has joined #openstack-nova | 07:47 | |
| gibi | good morning nova | 07:47 |
| *** Dinesh_Bhor has quit IRC | 07:56 | |
| *** Dinesh_Bhor has joined #openstack-nova | 08:01 | |
| openstackgerrit | fupingxie proposed openstack/nova master: Delete allocations for instances that have been moved to another node https://review.openstack.org/582899 | 08:08 |
| *** priteau has joined #openstack-nova | 08:12 | |
| openstackgerrit | fupingxie proposed openstack/nova master: Delete allocations for instances that have been moved to another node https://review.openstack.org/582899 | 08:14 |
| openstackgerrit | fupingxie proposed openstack/nova master: Delete allocations for instances that have been moved to another node https://review.openstack.org/582899 | 08:18 |
| *** alexchadin has quit IRC | 08:19 | |
| *** Dinesh_Bhor has quit IRC | 08:27 | |
| *** derekh has joined #openstack-nova | 08:28 | |
| *** cdent has joined #openstack-nova | 08:34 | |
| openstackgerrit | garyk proposed openstack/nova master: Docs: update link for remote debugging https://review.openstack.org/591316 | 08:37 |
| *** ykarel|lunch is now known as ykarel | 08:45 | |
| *** psachin has quit IRC | 08:50 | |
| *** tonyb has quit IRC | 08:50 | |
| *** ttsiouts has joined #openstack-nova | 08:51 | |
| *** psachin has joined #openstack-nova | 09:03 | |
| *** ttsiouts has quit IRC | 09:06 | |
| *** ttsiouts has joined #openstack-nova | 09:06 | |
| *** Dinesh_Bhor has joined #openstack-nova | 09:12 | |
| *** davidsha has joined #openstack-nova | 09:19 | |
| *** stakeda has quit IRC | 09:21 | |
| *** sayalilunkad has joined #openstack-nova | 09:32 | |
| *** helenafm has quit IRC | 09:32 | |
| *** lei-zh has quit IRC | 09:33 | |
| openstackgerrit | Naichuan Sun proposed openstack/nova master: xenapi(N-R-P)(WIP): support compute node resource provider update https://review.openstack.org/521041 | 09:37 |
| *** jaosorior has joined #openstack-nova | 09:47 | |
| *** holser_ has quit IRC | 09:48 | |
| *** holser_ has joined #openstack-nova | 09:52 | |
| *** Dinesh_Bhor has quit IRC | 09:52 | |
| *** tonyb has joined #openstack-nova | 09:53 | |
| *** panda|rover has quit IRC | 09:54 | |
| *** panda has joined #openstack-nova | 09:59 | |
| *** panda has quit IRC | 10:04 | |
| *** psachin has quit IRC | 10:06 | |
| openstackgerrit | Chen proposed openstack/nova master: doc: update info for hypervisors https://review.openstack.org/599554 | 10:13 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: WIP: Use placement from separate repo in functional test https://review.openstack.org/599556 | 10:14 |
| openstackgerrit | fupingxie proposed openstack/nova master: Delete allocations for instances that have been moved to another node https://review.openstack.org/582899 | 10:16 |
| *** ttsiouts has quit IRC | 10:26 | |
| *** dave-mccowan has joined #openstack-nova | 10:41 | |
| *** tbachman has quit IRC | 10:42 | |
| *** mugsie has joined #openstack-nova | 10:46 | |
| *** cdent has quit IRC | 10:54 | |
| stephenfin | prometheanfire: Oh, yeah, I'm around now (and for the next 6 hours or so - I'm GMT) | 10:56 |
| *** erlon has joined #openstack-nova | 10:58 | |
| *** takamatsu has joined #openstack-nova | 11:00 | |
| *** jpena is now known as jpena|lunch | 11:02 | |
| *** luksky has quit IRC | 11:05 | |
| *** andreaf has joined #openstack-nova | 11:06 | |
| *** ttsiouts has joined #openstack-nova | 11:23 | |
| *** udesale has quit IRC | 11:24 | |
| kashyap | stephenfin: Most excellent: | 11:29 |
| kashyap | <controller type='pci' index='1' model='pcie-root-port'> | 11:30 |
| kashyap | <model name='pcie-root-port'/> | 11:30 |
| kashyap | ... | 11:30 |
| kashyap | </controller> | 11:30 |
| kashyap | stephenfin: That's the bit I was looking for. | 11:30 |
| kashyap | I was wondering if we (wrongly) set an explict controller model like: <model name='ioh3420'/> | 11:31 |
| *** beagles has joined #openstack-nova | 11:31 | |
| *** cdent has joined #openstack-nova | 11:33 | |
| *** panda has joined #openstack-nova | 11:34 | |
| *** alexchadin has joined #openstack-nova | 11:34 | |
| *** luksky has joined #openstack-nova | 11:40 | |
| sean-k-mooney | kashyap: ya perhaps. | 11:41 |
| *** jaypipes has joined #openstack-nova | 11:44 | |
| *** cdent has quit IRC | 11:49 | |
| *** sambetts_ has quit IRC | 11:55 | |
| *** dims has joined #openstack-nova | 11:56 | |
| *** sambetts_ has joined #openstack-nova | 11:57 | |
| *** jpena|lunch is now known as jpena | 12:03 | |
| *** ttsiouts has quit IRC | 12:04 | |
| *** ttsiouts has joined #openstack-nova | 12:05 | |
| *** ajo has joined #openstack-nova | 12:05 | |
| *** eharney has joined #openstack-nova | 12:06 | |
| *** ttsiouts_ has joined #openstack-nova | 12:07 | |
| *** trozet has joined #openstack-nova | 12:08 | |
| *** ttsiouts_ has quit IRC | 12:09 | |
| *** ttsiouts has quit IRC | 12:09 | |
| efried | Good morning nova | 12:10 |
| *** ttsiouts has joined #openstack-nova | 12:10 | |
| *** med_ has joined #openstack-nova | 12:13 | |
| *** tbachman has joined #openstack-nova | 12:14 | |
| *** tbachman has quit IRC | 12:14 | |
| *** tbachman has joined #openstack-nova | 12:16 | |
| kashyap | sean-k-mooney: Nova is not; so we're good there. | 12:19 |
| *** sahid has quit IRC | 12:23 | |
| *** sahid has joined #openstack-nova | 12:23 | |
| *** helenafm has joined #openstack-nova | 12:24 | |
| mnaser | fwiw: rocky has been stable in sjc1 with a bunch of new clients coming on and being part of nodepool at ~50 vms (+ all bfv) | 12:25 |
| mnaser | so congrats :) | 12:25 |
| *** trozet has quit IRC | 12:29 | |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Merge security groups extension response into server view builder https://review.openstack.org/585475 | 12:29 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Merge extended_status extension response into server view builder https://review.openstack.org/592092 | 12:29 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Add scatter-gather-single-cell utility https://review.openstack.org/594947 | 12:29 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Merge extended_volumes extension response into server view builder https://review.openstack.org/596285 | 12:30 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Making instance/migration listing skipping down cells configurable https://review.openstack.org/592428 | 12:30 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Add get_by_cell_and_project() method to InstanceMappingList https://review.openstack.org/591656 | 12:30 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Return a minimal construct for nova list when a cell is down https://review.openstack.org/567785 | 12:30 |
| *** ttsiouts has quit IRC | 12:32 | |
| efried | mnaser: ++ ! | 12:33 |
| *** ykarel is now known as ykarel|away | 12:34 | |
| *** ccamacho has quit IRC | 12:34 | |
| kashyap | sean-k-mooney: Hey, when you're about: from yesterday's XML snippet from stephenfin, when _not_ setting any 'num_pcie_ports', why do we get 4 root ports: http://paste.openstack.org/show/729350/ | 12:35 |
| gibi | mnaser: thanks for the good news | 12:36 |
| kashyap | sean-k-mooney: Ah, never mind, saw this: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5114-L5119 | 12:37 |
| mnaser | the next fun step is going to be upgrading our mtl region | 12:39 |
| mnaser | that will be the fun one | 12:39 |
| *** ttsiouts has joined #openstack-nova | 12:41 | |
| *** gbarros has joined #openstack-nova | 12:41 | |
| *** ykarel|away has quit IRC | 12:42 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Re-propose numa-aware-live-migration spec https://review.openstack.org/599587 | 12:47 |
| openstackgerrit | Brin Zhang proposed openstack/nova master: Need further updates, no need to review https://review.openstack.org/599276 | 12:52 |
| *** mriedem has joined #openstack-nova | 12:53 | |
| *** mdrabe has joined #openstack-nova | 12:56 | |
| *** gbarros has quit IRC | 13:04 | |
| *** gbarros has joined #openstack-nova | 13:05 | |
| *** awaugama has joined #openstack-nova | 13:06 | |
| *** munimeha1 has joined #openstack-nova | 13:06 | |
| *** lbragstad has joined #openstack-nova | 13:19 | |
| *** moshele has quit IRC | 13:25 | |
| openstackgerrit | Jay Pipes proposed openstack/nova-specs master: allow transferring ownership of instance https://review.openstack.org/599598 | 13:26 |
| jaypipes | melwitt: ^^ example of needed coordination between nova and placement. /me hopes the extraction won't be too much of a distraction | 13:27 |
| *** brinzhang has quit IRC | 13:30 | |
| openstackgerrit | Jay Pipes proposed openstack/nova-specs master: allow transferring ownership of instance https://review.openstack.org/599598 | 13:44 |
| *** ccamacho has joined #openstack-nova | 13:46 | |
| *** tbachman has quit IRC | 13:46 | |
| mriedem | jaypipes: can you push the spec and such against the old existing bp for the same thing? https://blueprints.launchpad.net/nova/+spec/transfer-instance-ownership | 13:54 |
| sean-k-mooney | jaypipes: i have been asked how to do that in the past. mainly for teaching cases where we wanted to be able to prepare a buncn of vms for people then give them the vm. | 13:56 |
| mriedem | this is more than just placement coordination, | 13:57 |
| mriedem | it's also cinder, neutron, glance, castellan/barbican right? | 13:57 |
| openstackgerrit | Jay Pipes proposed openstack/nova-specs master: allow transferring ownership of instance https://review.openstack.org/599598 | 13:58 |
| mriedem | plus maybe whatever is managing the vm? trove/heat? | 13:58 |
| jaypipes | mriedem: done. | 13:58 |
| sean-k-mooney | trove/heat should really only need to expose an api to initate the transfer the rest should be handeld in nova right? | 13:59 |
| *** udesale has joined #openstack-nova | 13:59 | |
| jaypipes | mriedem: I'm not trying to coordinate between cinder, neutron or glance in the spec. only placement and nova. I note the other integration points and why I'm not trying to add orchestration functionality to nova. | 13:59 |
| *** ttsiouts has quit IRC | 13:59 | |
| jaypipes | mriedem: I'm afraid absolutely nothing would get done at all if we try to boil the ocean like previous attempts have done. | 14:00 |
| mriedem | that would effectively break us | 14:00 |
| mriedem | if you create a vm which creates a volume and a port, | 14:00 |
| jaypipes | mriedem: what would effectively break us? | 14:00 |
| mriedem | and then change the owner of the vm, we'll fail to delete the volume/port | 14:00 |
| jaypipes | mriedem: why would you delete the volume/port? | 14:00 |
| mriedem | b/c that's what we do | 14:00 |
| mriedem | delete_on_termination=true for bfv, | 14:01 |
| sean-k-mooney | jaypipes: from a placement perspecitive did we settle on the idea that neutron and cinder resouces would be conumed by the instance e.g. the instance uuid is used as the consumer rather then neutorn port uuid ecta. | 14:01 |
| mriedem | and nova cleans up the ports it creates | 14:01 |
| jaypipes | mriedem: who said anything about terminating anything? | 14:01 |
| mriedem | we can assume that someone would eventually try to delete these resources | 14:01 |
| mriedem | if this is stricly baremetal instances b/c oath, then let's be clear about that | 14:01 |
| jaypipes | mriedem: let's discuss this on the review, eh | 14:02 |
| mriedem | but even baremetal instances can boot from volume now | 14:02 |
| jaypipes | mriedem: this is not strictly bm instances for oath, no... | 14:02 |
| mriedem | sure | 14:02 |
| jaypipes | I'm really not sure why you think that. | 14:02 |
| jaypipes | I'm not sure what about the spec as written gave you that impression. | 14:02 |
| *** johnsom has joined #openstack-nova | 14:06 | |
| *** r-daneel has quit IRC | 14:07 | |
| tobias-urdin | hm is there any easy way to figure out if an instance is volume backed using novaclient? | 14:08 |
| mriedem | yes | 14:09 |
| mriedem | image_ref is '' | 14:09 |
| mriedem | *image | 14:10 |
| mriedem | normally it's a dict with an id and link, | 14:10 |
| mriedem | but for volume-backed, it's just '' | 14:10 |
| mriedem | https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/views/servers.py#L332 | 14:11 |
| *** mlavalle has joined #openstack-nova | 14:13 | |
| *** itlinux has quit IRC | 14:15 | |
| mriedem | comments inline | 14:15 |
| mriedem | jaypipes: | 14:15 |
| *** Bhujay has quit IRC | 14:17 | |
| *** cdent has joined #openstack-nova | 14:18 | |
| tobias-urdin | mriedem: thanks! | 14:18 |
| *** sapd1_ has joined #openstack-nova | 14:19 | |
| *** eharney has quit IRC | 14:20 | |
| *** alexchadin has quit IRC | 14:22 | |
| tobias-urdin | mriedem: and if you need to get a list of all attached volume and know which one is the root volume? | 14:23 |
| tobias-urdin | sorry got it get_server_volumes() | 14:25 |
| tobias-urdin | if it's one volume that easy, otherwise should you rely that device=/dev/vda always is the first that is booting | 14:25 |
| tobias-urdin | ? | 14:26 |
| *** alexchadin has joined #openstack-nova | 14:26 | |
| *** ttsiouts has joined #openstack-nova | 14:27 | |
| mriedem | device name doesn't really mean anything, | 14:27 |
| mriedem | nova ignores it if supplied | 14:27 |
| mriedem | boot_index is what you'd want, but i don't think we expose that out of the api | 14:27 |
| mriedem | we certainly could, and probably should if we ever want to get device_name out of the API for volumes | 14:28 |
| tobias-urdin | hm ouch so the boot index cant be access through any api calls listing server info or block device mappings or similar | 14:30 |
| tobias-urdin | can the boot_index be changed from nova's perspective? because the only way I could work around that would be relying on the creation date of the volumes | 14:31 |
| sean-k-mooney | tobias-urdin: if you need a reliable way to assicate volumes with devices in the guest you need to use tags | 14:31 |
| tobias-urdin | sean-k-mooney: ok, don't think that helps what i'm trying to do. i need to get the root volume if its a volume backed instance | 14:33 |
| tobias-urdin | i guess other than creation date i could check if the volume was created from an image with the cinder api, but that could fail as well if somebody attaches a volume for recovery | 14:34 |
| *** jiapei has quit IRC | 14:34 | |
| sean-k-mooney | tobias-urdin: e.g. novas boot form volume form image or boot with a precreated volume | 14:34 |
| tobias-urdin | after checking i can't see nova populating the image field even when booting from a volume + image during creation | 14:36 |
| sean-k-mooney | mdbooth: wasn't someone working on ^^ | 14:36 |
| mriedem | which image field? | 14:37 |
| mriedem | i forgot about tags - yes you could use tags to say which is the root volume during boot from volume, but we don't expose the bdm tags out of the API either :) | 14:37 |
| tobias-urdin | whichever Server.image from novaclient provides | 14:37 |
| mriedem | i have related specs for both of those things i think | 14:37 |
| mriedem | yes that's on purpose - the server.image is '' if volume-backed | 14:38 |
| mriedem | the image backing the root volume is in the volume metadata | 14:38 |
| *** mchlumsky has joined #openstack-nova | 14:38 | |
| mriedem | in "volume_image_metadata" | 14:38 |
| mriedem | https://review.openstack.org/#/c/452546/ is related to getting device_name out of the API, | 14:39 |
| tobias-urdin | is that exposed out of the api and novaclient? | 14:39 |
| *** r-daneel has joined #openstack-nova | 14:39 | |
| sean-k-mooney | volume_image_metadata is a copy of the glance metadata for an image pluse i think an image ref of some kind i think ? | 14:39 |
| *** tbachman has joined #openstack-nova | 14:39 | |
| mriedem | tobias-urdin: which? volume_image_metadata? | 14:39 |
| mriedem | tobias-urdin: that's on the volume, so nova doesn't expose it, cinder does, | 14:40 |
| mriedem | but yes | 14:40 |
| mriedem | https://review.openstack.org/#/c/393930/ also related to exposing bdm tags out of the compute api | 14:41 |
| mdbooth | volume_image_metadata doesn't need to have a corresponding glance image, btw. | 14:41 |
| mriedem | which overlaps with https://review.openstack.org/#/c/452546/ | 14:41 |
| mdbooth | But it's used in the same way and has the same semantics. | 14:41 |
| mdbooth | mriedem: Yeah, it's weird we don't expose that. | 14:43 |
| mdbooth | tags via the rest api, that is. | 14:44 |
| mriedem | just hasn't been done - i've had the specs, been defeated, then found new agreement but haven't found new motivation for doing the work | 14:44 |
| * mdbooth knows that story :( | 14:45 | |
| *** sridharg has joined #openstack-nova | 14:45 | |
| tobias-urdin | found what i needed in volume_image_metadata atleast, just have to do some ugly assumptions for now based on volume_image_metadata and creation date since i cant access boot_index or block device mappings info | 14:46 |
| tobias-urdin | thanks for helping out :) | 14:46 |
| sean-k-mooney | tobias-urdin: this kind of think is something that could be a feature request to the openstacksdk/shade teams as getting the root volume for boot from volume is likely one of thoes things that will change alot depending on your env and could be handeld in a shade proxy api | 14:49 |
| mdbooth | tobias-urdin: In practise, it's probably going to be the one with volume_image_metadata. | 14:49 |
| mdbooth | Although I was reviewing a v2v tool the other day which added volume_image_metadata to multiple volumes, which would have broken that assumption, but I can't imagine that's common. | 14:50 |
| mdbooth | Creation data is less reliable, especially if you have persistent data on volume X, and created a root volume to contain an app to manipulate it some time later. | 14:51 |
| mdbooth | Unlike the first case, I'd expect that to be likely to happen in the wild. | 14:51 |
| tobias-urdin | sean-k-mooney: sort of pressing to get this stuff done (as always...) otherwise it would've been optimal to check the sdk first | 14:52 |
| tobias-urdin | mdbooth: yeah, i was thinking about combining if there is multiple volume_image_metadata just assume the first created.. atleast i would always get something | 14:53 |
| prometheanfire | stephenfin: hi :D | 14:56 |
| stephenfin | prometheanfire: o/ | 14:56 |
| mdbooth | Not strictly related, but afaict we attach volumes in a non-deterministic order on restart, except for the root volume. And that's ok. We should probably deliberately randomise it :) | 14:57 |
| *** dklyle has joined #openstack-nova | 14:58 | |
| tobias-urdin | while I'm at it... here's a sad question, can I somehow block the creation of image backed instances? (i.e images_type backed instances) | 14:58 |
| *** tbachman has quit IRC | 14:59 | |
| tobias-urdin | policy, super simple api hack or similar, everything's allowed but the best would be to not really touch anything critical | 14:59 |
| prometheanfire | stephenfin: maybe my deployment is just broken, gimme a few to get out of this meeting and I'll update you | 14:59 |
| tobias-urdin | I was hoping on setting images_type to None but that errors out upon initialization | 14:59 |
| mdbooth | tobias-urdin: Don't deploy glance? | 15:01 |
| tobias-urdin | (oh how I wish there was a cinder backend for images_type right now, wish i familar enough with the codebase to drive such a work) | 15:01 |
| tobias-urdin | mdbooth: how u mean? hm settings [glance]/api_servers to something invalid? | 15:02 |
| sean-k-mooney | tobias-urdin: well if you have ceph you could set nova to image_type ceph | 15:02 |
| tobias-urdin | sean-k-mooney: yea that's what we do today, however don't want users that way at all, weird i know but we want to default to cinder | 15:03 |
| efried | bauzas, mriedem: https://review.openstack.org/#/c/598365/ just needs config helps updated? | 15:03 |
| mriedem | haven't looked at reviews on it yet today | 15:04 |
| sean-k-mooney | tobias-urdin: its not that weird haveing a generic image_type cinder has come up in dublin ptg and last time in denver | 15:04 |
| bauzas | efried: yep, IMHO | 15:04 |
| sean-k-mooney | its just not that simple. | 15:04 |
| bauzas | efried: mriedem: once done, we could modify the default values | 15:05 |
| mriedem | i thought ovh already had a patch for that | 15:05 |
| tobias-urdin | sean-k-mooney: yea :( | 15:05 |
| efried | bauzas, mriedem: It's blocking our CI, so I would like to get it merged as soon as possible. | 15:05 |
| bauzas | orly N | 15:05 |
| bauzas | ? | 15:05 |
| bauzas | if so, let me +W it | 15:05 |
| mriedem | bauzas: https://review.openstack.org/#/c/532924/ | 15:06 |
| sean-k-mooney | mriedem: do you know if anyone is proposing or looking at a cinder image time for stein or makeing bfv the default? | 15:06 |
| efried | bauzas: Okay, thanks. We can do the conf helps in a fup? | 15:06 |
| *** beekneemech is now known as bnemec | 15:06 | |
| sean-k-mooney | *image_type=cinder | 15:06 |
| mriedem | efried: i can look and update | 15:06 |
| bauzas | efried: if needed, yep | 15:06 |
| mriedem | sean-k-mooney: as in the fabled libvirt cinder image backend of lore? | 15:06 |
| mriedem | no, no one is working on that | 15:07 |
| bauzas | mriedem: would you be possible to pass a new revision now, or just a new change later ? | 15:07 |
| bauzas | if the latter, no worries | 15:07 |
| kashyap | sean-k-mooney: When you get a minute, please remind me again: we can't set PCIe root ports via flavor metadata property, can we? | 15:07 |
| sean-k-mooney | mriedem: ya that is what i had rememberd form dublin | 15:07 |
| mriedem | bauzas: efried: i'll update it in a minute | 15:07 |
| bauzas | mriedem: and yeah, I remember this change | 15:07 |
| prometheanfire | stephenfin: ya, I made sure no pycache/pyc/pyo | 15:07 |
| sean-k-mooney | kashyap: if we can the glance metadef have not been created to document it. i wish we could and didn not have this in the nova config. ill check | 15:08 |
| kashyap | sean-k-mooney: Right, I presume we _can't_ today; I'll go look the code | 15:08 |
| bauzas | mriedem: and I also remember the spec https://review.openstack.org/#/c/552105/3/specs/rocky/approved/default-allocation-ratios.rst | 15:09 |
| stephenfin | prometheanfire: There's definitely some form of caching going on or your source is located somewhere else. That's the only reason for that stuff to happen | 15:09 |
| bauzas | mriedem: but I think it's a separate issue | 15:09 |
| kashyap | sean-k-mooney: A libvirt dev was asking that question: can Nova set the root ports via flavor; or just through a global knob | 15:09 |
| *** itlinux has joined #openstack-nova | 15:09 | |
| prometheanfire | stephenfin: even odder, I ran the py27 version of nova-compute, got the traceback containing py35 | 15:10 |
| prometheanfire | the main traceback was py27 though | 15:10 |
| stephenfin | prometheanfire: Can you paste the output of that? | 15:11 |
| prometheanfire | I think the real issue was that portion | 15:11 |
| sean-k-mooney | kashyap: this is the config generation code but github is not finding its usage https://github.com/openstack/nova/blob/c6218428e9b29a2c52808ec7d27b4b21aadc0299/nova/virt/libvirt/config.py#L1713-L1728 | 15:12 |
| sean-k-mooney | kashyap: codesearch did however http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/driver.py#n5117 | 15:13 |
| sean-k-mooney | kashyap: the dirver uses the conf value directly | 15:13 |
| kashyap | sean-k-mooney: So no metadata property | 15:13 |
| kashyap | Wonder if we should file a blueprint to add it | 15:13 |
| *** dklyle has quit IRC | 15:13 | |
| *** dklyle has joined #openstack-nova | 15:14 | |
| sean-k-mooney | kashyap: i would be +1 on that espcially if we can deprecate and remvoe the conf option | 15:14 |
| kashyap | sean-k-mooney: Yeah, we should be able to do that; instead of the global config. | 15:14 |
| kashyap | Damn, since I 'discovered' the bug, I get the pleasure of filing the Blueprint I guess :P | 15:14 |
| kashyap | sean-k-mooney: More seriously, paperwork question: does this require a spec? Since it's user-impacting? | 15:14 |
| kashyap | Or a spec-less blueprint is reasonable enough? | 15:15 |
| sean-k-mooney | kashyap: spec-less blueprint. extra specs are a gray area as they are not technical part of the versioned api but are user fasing so mriedem or someelse will likely comment on the blueprint if a specs is needed | 15:16 |
| prometheanfire | stephenfin: https://gist.githubusercontent.com/prometheanfire/6512134e799ec8c08c3f080150f60d19/raw/7cdb3a9a350c19540b1c930077e164786226636b/gistfile1.txt | 15:17 |
| kashyap | sean-k-mooney: Yep, noted; thanks for the discussion. | 15:17 |
| sean-k-mooney | by the way i have been using http://codesearch.openstack.org/ a lot more recently instead of using github to such for these things its pretty good | 15:18 |
| kashyap | Ah, nice. | 15:19 |
| kashyap | sean-k-mooney: BTW, seems like "hw:machine_type" isn't documented a flavor extra specs here: https://docs.openstack.org/nova/latest/user/flavors.html | 15:19 |
| kashyap | Is that so? | 15:19 |
| sean-k-mooney | kashyap: correct it is documented here https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt-image.json#L34-L38 | 15:20 |
| sean-k-mooney | accouding to the glace metadef registry its only valid on image not flavor | 15:20 |
| *** openstackgerrit has quit IRC | 15:20 | |
| kashyap | sean-k-mooney: Hmm, but the syntax is slightly different: hw:machine_type vs hw_machine_type (in nova.conf) | 15:21 |
| kashyap | Right, this seem to work: `openstack flavor set --property hw:machine_type=x86_64=q35 test.q35` | 15:21 |
| stephenfin | prometheanfire: Based on that, it seems oslo.service package in your virtualenv is starting a thread using the system oslo.service package. I've no idea why that would happen | 15:21 |
| *** Luzi has quit IRC | 15:21 | |
| stephenfin | prometheanfire: Might be worth asking on #openstack-oslo to see if anyone else has seen this before | 15:21 |
| sean-k-mooney | kashyap: hw: is the namesapced flavor syntax, images dont have namespaces so the namespace is prepended with an _ instead of : | 15:22 |
| sean-k-mooney | kashyap: https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt-image.json#L7-L11 tell you these are only valid in the image | 15:22 |
| stephenfin | prometheanfire: You've patched 'usr/lib64/python3.5/site-packages/nova', I assume? | 15:22 |
| kashyap | sean-k-mooney: Nod; I'll send a doc patch to document it here as well: https://docs.openstack.org/nova/latest/user/flavors.html | 15:22 |
| sean-k-mooney | kashyap: if its valid in both it looks like https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt.json#L7-L16 | 15:23 |
| prometheanfire | stephenfin: that file didn't exist at the time :P | 15:23 |
| prometheanfire | I uninstalled nova/oslo-service/oslo-db system-wide | 15:23 |
| prometheanfire | only available in the venv | 15:23 |
| sean-k-mooney | kashyap: documenting it there is fine but the autritive source is glance | 15:23 |
| prometheanfire | like I said, broken :P | 15:23 |
| stephenfin | prometheanfire: Very :) | 15:24 |
| kashyap | sean-k-mooney: Yep, noted. (And it seems to be valid for both, IIUC) | 15:24 |
| stephenfin | prometheanfire: Yeah, I'm not sure how much I can help with that. There's something funky going on with venvs that I don't understand. I don't think it's anything to do with the patch itself | 15:24 |
| *** gbarros has quit IRC | 15:24 | |
| *** gbarros has joined #openstack-nova | 15:25 | |
| prometheanfire | stephenfin: ya, at this point justmergeit | 15:27 |
| sean-k-mooney | kashyap: doing a code search i only see code for using it from the conf or the image not the flavor | 15:27 |
| sean-k-mooney | kashyap: http://codesearch.openstack.org/?q=machine_type&i=nope&files=&repos=nova | 15:27 |
| kashyap | sean-k-mooney: Yeah, I've just done a test, indeed it's so | 15:28 |
| kashyap | stephenfin: Hey, when you get a moment, yesterday you said this worked for you: | 15:28 |
| kashyap | $ openstack flavor create test.q35 | 15:28 |
| kashyap | $ openstack flavor set --property hw:machine_type=x86_64=q35 test.q35 | 15:28 |
| kashyap | $ openstack server create --flavor test.q35 --image test \ | 15:28 |
| kashyap | --nic net-id=$NIC_UUID test-q35 | 15:28 |
| kashyap | stephenfin: Did the second command really take effect? It shouldn't have worked. | 15:29 |
| stephenfin | kashyap: No, it didn't. I misread your comments | 15:29 |
| kashyap | I guess you had 'q35' via other means, like config | 15:29 |
| kashyap | Ah-ha! | 15:29 |
| stephenfin | Hence the second set of pastes | 15:29 |
| sean-k-mooney | kashyap: i think stephenfin had to use the config | 15:29 |
| *** Miouge has left #openstack-nova | 15:29 | |
| kashyap | Yep, it's all clear now. | 15:31 |
| *** amarao has quit IRC | 15:33 | |
| *** alexchadin has quit IRC | 15:35 | |
| *** alexchadin has joined #openstack-nova | 15:36 | |
| *** alexchadin has quit IRC | 15:36 | |
| *** alexchadin has joined #openstack-nova | 15:36 | |
| *** alexchadin has quit IRC | 15:37 | |
| *** alexchadin has joined #openstack-nova | 15:37 | |
| *** alexchadin has quit IRC | 15:37 | |
| *** alexchadin has joined #openstack-nova | 15:38 | |
| *** alexchadin has quit IRC | 15:38 | |
| *** moshele has joined #openstack-nova | 15:40 | |
| *** eharney has joined #openstack-nova | 15:45 | |
| *** ccamacho has quit IRC | 15:53 | |
| *** gbarros has quit IRC | 15:57 | |
| *** jlvilla-viva is now known as jlvillal | 15:58 | |
| *** openstackgerrit has joined #openstack-nova | 15:58 | |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Return a minimal construct for nova list when a cell is down https://review.openstack.org/567785 | 15:58 |
| *** gbarros has joined #openstack-nova | 15:59 | |
| *** macza has joined #openstack-nova | 16:08 | |
| *** alexchadin has joined #openstack-nova | 16:10 | |
| *** mdrabe has quit IRC | 16:10 | |
| *** ttsiouts has quit IRC | 16:13 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Document unset/reset wrinkle for *_allocation_ratio options https://review.openstack.org/599670 | 16:14 |
| *** ttsiouts has joined #openstack-nova | 16:14 | |
| mriedem | efried: bauzas: jaypipes: ^ follow up for config option wording | 16:14 |
| *** alexchadin has quit IRC | 16:14 | |
| *** holser_ has quit IRC | 16:15 | |
| bauzas | mriedem: thanks, and bingo | 16:15 |
| efried | mriedem: +A, nice one guv | 16:17 |
| *** Bhujay has joined #openstack-nova | 16:17 | |
| *** ttsiouts has quit IRC | 16:19 | |
| prometheanfire | huh, nova destroyed an instance when I was messing with placement stuff | 16:19 |
| *** tbachman has joined #openstack-nova | 16:19 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Don't persist zero allocation ratios in ResourceTracker https://review.openstack.org/599672 | 16:21 |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Document unset/reset wrinkle for *_allocation_ratio options https://review.openstack.org/599673 | 16:21 |
| jaypipes | mriedem: I'd rather have someone like mgagne look at that patch and give advice. | 16:22 |
| jaypipes | since we're not operators... | 16:22 |
| mriedem | sure, hence the big todo comment in the bottom change | 16:22 |
| sean-k-mooney | prometheanfire did you delete its allocation or something? | 16:24 |
| *** gyee has joined #openstack-nova | 16:24 | |
| sean-k-mooney | prometheanfire: i did not think we could kill nova instance by messing with placement so that sounds... unintended | 16:25 |
| prometheanfire | sean-k-mooney: no, it looks like libvirt forgot it exists | 16:25 |
| prometheanfire | or something | 16:25 |
| sean-k-mooney | is the instace still listed in openstack. e.g. opestack server list | 16:25 |
| prometheanfire | 2018-09-04 16:11:44.844 4079 INFO nova.compute.manager [req-ab55e9f6-b2a8-48ac-b50b-fde5b7af0892 - - - - -] [instance: 0e9aa374-3627-48ac-a410-4abd65564a80] Deleting instance as it has been evacuated from this host | 16:25 |
| prometheanfire | no clue why it was evacualted :| | 16:26 |
| prometheanfire | that's the first log line on start of nova-compute | 16:26 |
| sean-k-mooney | oh ok am ill go with ghosts | 16:26 |
| sean-k-mooney | or your other admins | 16:26 |
| prometheanfire | I'm the only admin :P | 16:26 |
| sean-k-mooney | then ill stick with my first answer | 16:26 |
| *** janki has quit IRC | 16:26 | |
| prometheanfire | what triggers a evacuation? | 16:26 |
| *** janki has joined #openstack-nova | 16:27 | |
| sean-k-mooney | you dont have watcher deployed or one of the other ha serices do you? | 16:27 |
| prometheanfire | nope | 16:27 |
| sean-k-mooney | prometheanfire: as far as i knew evacuate was an admin only api call | 16:27 |
| sean-k-mooney | so without manually invoking it i did not think we had a way to auto evacuate | 16:27 |
| prometheanfire | ghosts then | 16:28 |
| *** sahid has quit IRC | 16:28 | |
| sean-k-mooney | mriedem: jaypipes dansmith any idea what could cause an evacuation to happen without an admin doing it? | 16:28 |
| mgagne | jaypipes, mriedem: This change is already Workflow+1, wording is fine for me anyway. (re https://review.openstack.org/#/c/599670/) | 16:29 |
| *** helenafm has quit IRC | 16:30 | |
| prometheanfire | sean-k-mooney: https://gist.githubusercontent.com/prometheanfire/76fc4693feed8b99118feaebbadfaea4/raw/c2a1b60e7181d7b864af35730e9f7eadfa365fd1/gistfile1.txt | 16:30 |
| prometheanfire | sean-k-mooney: it looks like it tries to see if it was evacuated, gets the timeout/traceback and sees that as ok? | 16:30 |
| * sean-k-mooney im really glad that alt-l allow me to go to raw mode and click long links | 16:30 | |
| prometheanfire | sean-k-mooney: github gist links do suck | 16:31 |
| prometheanfire | 2018-09-04 16:34:15.499 4079 WARNING nova.compute.manager [req-2f9b1170-6748-4c94-af5f-00e8fc70d0e9 - - - - -] While synchronizing instance power states, found 4 instances in the database and 3 instances on the hypervisor. | 16:31 |
| prometheanfire | lol | 16:31 |
| sean-k-mooney | prometheanfire: had you previously evacuated instaces from that host? if so when the nova compute agent comes back up it will clean up any instaces that were not deleted | 16:31 |
| prometheanfire | is there a way to get nova to recreate the libvirt domain? | 16:31 |
| prometheanfire | sean-k-mooney: nope | 16:32 |
| prometheanfire | not that I can remember at all | 16:32 |
| jaypipes | mgagne: yes, I recognize the change is already +W. I just wanted a real operator to double-check the wording. :) | 16:32 |
| sean-k-mooney | prometheanfire: well it depends if you do opestack server show 0e9aa374-3627-48ac-a410-4abd65564a80 i assume it is gone? | 16:32 |
| mgagne | jaypipes: +1 | 16:33 |
| sean-k-mooney | prometheanfire: if not then it should be running somewhere else in your cloud | 16:33 |
| prometheanfire | sean-k-mooney: the server show still works, when I do a reboot I get this | 16:34 |
| prometheanfire | 2018-09-04 16:23:57.779 4079 ERROR nova.compute.manager [req-e7f9f2a5-7cbc-4776-a085-88245450abac bcebdc7b8dfd4d43b036d1b73df6d377 5488a33661454bd792ff8c62d31d07a0 - default default] [instance: 0e9aa374-3627-48ac-a410-4abd65564a80] Cannot reboot instance: Instance 0e9aa374-3627-48ac-a410-4abd65564a80 could not be found.: nova.exception.InstanceNotFound: Instance | 16:34 |
| prometheanfire | 0e9aa374-3627-48ac-a410-4abd65564a80 could not be found. | 16:34 |
| sean-k-mooney | prometheanfire: am from the admin view can you check the host its running on and see if libvirt see it? | 16:35 |
| prometheanfire | virsh list --all doesn't show it | 16:36 |
| prometheanfire | this had to be my dns server too | 16:36 |
| sean-k-mooney | prometheanfire: im guessing its partially deleted. you could try a force reset of the vm status followed by a hard reboot | 16:37 |
| prometheanfire | sean-k-mooney: where is the libvirt.xml stuff stored now? | 16:38 |
| *** janki has quit IRC | 16:38 | |
| prometheanfire | I could recreate the domain and it'd probably work | 16:38 |
| *** moshele has quit IRC | 16:38 | |
| prometheanfire | /etc/libvirt/qemu/ it looks like | 16:39 |
| sean-k-mooney | yes. if you know the instace name the xml might still be there if not the qemu args will be in /var/log/libvirt/qeum/instacne... | 16:40 |
| *** moshele has joined #openstack-nova | 16:40 | |
| prometheanfire | ya, that helps some | 16:44 |
| prometheanfire | the xml isn't there, but I can register a domain with it and it should be picked up then, hopefully... | 16:44 |
| kashyap | stephenfin: Hey, if you still have that env, can I ask to do one last test, please? | 16:44 |
| sean-k-mooney | prometheanfire: nova will rechreate the domain for you if you can start the vm. | 16:44 |
| kashyap | stephenfin: It is the following: | 16:45 |
| kashyap | stephenfin: Boot a guest w/ Q35, but now with _8_ PCIe root ports, using 'num_pcie_ports=8' in nova.conf | 16:45 |
| sean-k-mooney | prometheanfire: the easies thing to do would be to rest the state to active. issue a shudown the reset to active agin if need and start the instace | 16:45 |
| prometheanfire | sean-k-mooney: in that case reset state and start | 16:45 |
| prometheanfire | ya | 16:45 |
| prometheanfire | and we're back, odd that it happened, but ok | 16:49 |
| sean-k-mooney | prometheanfire: was your instance on ceph or cinder storage? | 16:50 |
| prometheanfire | no | 16:50 |
| prometheanfire | just a basic instance | 16:50 |
| *** udesale has quit IRC | 16:51 | |
| sean-k-mooney | :( in that case the evacuate deleted your data as it did a force rebuild to a different host | 16:51 |
| prometheanfire | sean-k-mooney: ssh was happy, so it didn't | 16:52 |
| prometheanfire | no clue what happened, but I'm fine | 16:52 |
| prometheanfire | even if it did, it's all in puppet for this node | 16:52 |
| *** luksky has quit IRC | 16:53 | |
| prometheanfire | my compute nodes still aren't reporting to placement though, not sure they ever did that right | 16:53 |
| sean-k-mooney | prometheanfire: well ssh would be fixed via cloud init | 16:53 |
| *** tssurya has quit IRC | 16:54 | |
| sean-k-mooney | if you data is intack however thats a good thing :) | 16:54 |
| prometheanfire | I meant the host key is the same | 16:54 |
| *** dkehn has quit IRC | 16:56 | |
| prometheanfire | sean-k-mooney: btw, did you see my comment about the database connection string in the apidb not being urlencoded? | 16:56 |
| openstackgerrit | Merged openstack/nova master: Combine error handling blocks in _do_build_and_run_instance https://review.openstack.org/545960 | 16:56 |
| prometheanfire | too far in backlog now | 16:57 |
| prometheanfire | update cell_mappings set database_connection='URLENCODED_CONNECTION_STRING' where uuid='0000' | 16:58 |
| *** SamYaple has joined #openstack-nova | 16:58 | |
| *** dkehn has joined #openstack-nova | 16:59 | |
| SamYaple | what actions, if any, can a user do that will trigger an 'instance.update' notification? having troble figuring it out from the code | 17:01 |
| *** derekh has quit IRC | 17:01 | |
| prometheanfire | There are no compute resource providers in the Placement service but there are 2 compute nodes in the deployment. This means no compute nodes are reporting into the Placement service and need to be upgraded and/or fixed. | 17:03 |
| *** ykarel has joined #openstack-nova | 17:03 | |
| prometheanfire | but the compute nodes have the placement info in their config, not sure what else to do there | 17:03 |
| *** sapd1_ has quit IRC | 17:05 | |
| mriedem | SamYaple: do you have notify_on_state_change set? | 17:05 |
| SamYaple | mriedem: yes | 17:05 |
| mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#notifications.notify_on_state_change | 17:05 |
| mriedem | to what? | 17:05 |
| SamYaple | prometheanfire: just checked scrollback, i just dealt with that issue! on a few nodes. It was caused by reprovisioning my compute nodes with teh same names but migrating the instances off with evacuate. when it came back the database was messed up | 17:05 |
| mriedem | prometheanfire: are you on pike or master/ | 17:06 |
| mriedem | ? | 17:06 |
| sean-k-mooney | prometheanfire: jaypipes or cdent might be able to help with that. im guessiog your missing the placement client and or config section in the nova config but honestly that is jsut a guess | 17:06 |
| prometheanfire | SamYaple: oh? | 17:06 |
| prometheanfire | mriedem: rocky | 17:06 |
| SamYaple | mriedem: so it *only* sends updates with state changes? if i read that correctly? (sorry, chasing down a bug in an inherited notifications reader with bad comments, so im trying to be explicit) | 17:07 |
| SamYaple | prometheanfire: yes, it had to do with the old instances that had been in an ERROR state before the node went down | 17:08 |
| prometheanfire | sean-k-mooney: there's a separate placement client? as in pypi type thing? | 17:08 |
| mriedem | prometheanfire: i relatively recently talked with dansmith about trying to parse/encode/decode the db connection string in the cell_mappings table and i think the consensus was if you have special stuff in the url, you need to encode it beforehand | 17:08 |
| sean-k-mooney | SamYaple: that kind of makes sense. since you used the same host name the compute agent will get teh compute servie record for the previous install | 17:08 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: WIP: Use placement from separate repo in functional test https://review.openstack.org/599556 | 17:08 |
| SamYaple | prometheanfire: ill try to pull my sql queries to fix it | 17:08 |
| prometheanfire | mriedem: ah, somehow I got it in the db | 17:08 |
| *** davidsha has quit IRC | 17:09 | |
| sean-k-mooney | prometheanfire: there is a placement osc plugin but when i said placement client i was referint to the devstack placement clinet service but honestly dont know what that does | 17:09 |
| mriedem | SamYaple: it looks like a hack way to trigger the instance.update notififcation is to change metadata on the instance | 17:09 |
| SamYaple | sean-k-mooney: would a possible workaround for my use case possibly be solved simply by deleting the service record between reprovisioning? | 17:10 |
| SamYaple | mriedem: ah! perfect! ok. so thats the hack this code is trying to tap into. that aligns on my end. thank you | 17:10 |
| prometheanfire | sean-k-mooney: ok, thought the nova-compute service handled registering itself | 17:10 |
| mriedem | prometheanfire: it does | 17:10 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: WIP: Use placement from separate repo in functional test https://review.openstack.org/599556 | 17:10 |
| mriedem | SamYaple: which release are you on? | 17:10 |
| SamYaple | mriedem: ocata/pike/queens across a few environments | 17:11 |
| sean-k-mooney | SamYaple:ya perhapos never tried it. with all the forign key on the table however it might be more hassel then its worth | 17:11 |
| mriedem | ok, because we didn't until recently cleanup after ourselves wrt placement when deleting a nova-compute service record | 17:11 |
| mriedem | thta's fixed now, but you'd have to make sure you have it, otherwise you have to cleanup placement entries yourself | 17:11 |
| sean-k-mooney | prometheanfire: if you have the right setting in your conf the nova-compute agent will take care of populating the placement api for you | 17:12 |
| SamYaple | ah, yes. this process has only been done on the older ocata clusters. We should be queens everywhere by end of year though. | 17:12 |
| prometheanfire | sean-k-mooney: nova-status upgrade check still shows the warning | 17:12 |
| SamYaple | mriedem: my sql are me cleaning up the placement entries :) | 17:12 |
| mriedem | fyi https://review.openstack.org/#/q/I7b8622b178d5043ed1556d7bdceaf60f47e5ac80 | 17:12 |
| SamYaple | mriedem: oh b-e-a-utiful. Super helpful. gonna do an internal backport on that for our ocata stuff. got a few hundred more nodes to go | 17:13 |
| sean-k-mooney | prometheanfire: for devstack the plamcenet section looks like this http://paste.openstack.org/show/729431/ | 17:13 |
| prometheanfire | sean-k-mooney: yep, looks right (looks like mine) | 17:14 |
| sean-k-mooney | prometheanfire: strange and openstack resource provider list is does not have the compute nodes? | 17:15 |
| prometheanfire | sean-k-mooney: http://paste.openstack.org/show/729432/ | 17:15 |
| mriedem | SamYaple: well, be warned, it's a hefty series of backports | 17:16 |
| *** jpena is now known as jpena|off | 17:16 | |
| mriedem | this is just pike https://review.openstack.org/#/q/topic:bug/1756179+(status:open+OR+status:merged)+branch:stable/pike | 17:16 |
| prometheanfire | sean-k-mooney: openstack service provider list | 17:16 |
| prometheanfire | that? | 17:16 |
| sean-k-mooney | no "openstack resource provider list" | 17:16 |
| sean-k-mooney | if you do not have that you are missing the placement osc plugin | 17:17 |
| prometheanfire | ok | 17:17 |
| mriedem | prometheanfire: are there errors in the nova-compute logs? | 17:17 |
| prometheanfire | mriedem: which errors you looking for? | 17:17 |
| sean-k-mooney | prometheanfire: https://github.com/openstack/osc-placement for future reference. it should be on pypi too | 17:17 |
| mriedem | prometheanfire: umm, any errors? | 17:17 |
| mriedem | presumably "failed to create resource provider in placement" or something like that | 17:18 |
| prometheanfire | mriedem: none now, I rebuilt all my systems from the ground up to fix the odd python issue | 17:18 |
| SamYaple | mriedem: sorry, i said backport. i mean hack something in based on that commit. i stay pretty close to upstream | 17:18 |
| *** tetsuro has joined #openstack-nova | 17:18 | |
| prometheanfire | mriedem: testing now via ' su - nova -c "/usr/lib/python-exec/python3.5/nova-compute --config-file /etc/nova/nova-compute.conf --config-file /etc/nova/nova.conf" ' | 17:18 |
| mriedem | SamYaple: ok, well, good luck. :) if that series of backports on pike also applies to ocata (i'm not sure if they all do), we could also backport them to ocata upstream | 17:19 |
| mriedem | all official like | 17:19 |
| sean-k-mooney | prometheanfire: you should get somting like "ResourceProviderCreationFailed" if the placement info is missing/wrong in the conf | 17:20 |
| prometheanfire | mriedem: http://paste.openstack.org/show/729433/ | 17:20 |
| prometheanfire | lgtm | 17:21 |
| *** med_ has quit IRC | 17:21 | |
| melwitt | . | 17:21 |
| sean-k-mooney | prometheanfire: that what i get when i comment it out but the agent keeps running which i was surrprised by | 17:21 |
| *** openstackgerrit has quit IRC | 17:22 | |
| *** openstackgerrit has joined #openstack-nova | 17:22 | |
| openstackgerrit | Merged openstack/nova master: Removing pip-missing-reqs from default tox jobs https://review.openstack.org/599442 | 17:22 |
| mriedem | prometheanfire: it's likely logged at debug | 17:23 |
| prometheanfire | ah, k | 17:23 |
| mriedem | do you see the resource provider in placement now? | 17:23 |
| prometheanfire | mriedem: I'm packaging osc-placement now | 17:23 |
| SamYaple | mriedem: ack. will do | 17:24 |
| *** med_ has joined #openstack-nova | 17:24 | |
| sean-k-mooney | mriedem: no its logging at error if it fails | 17:24 |
| prometheanfire | sean-k-mooney: mriedem does osc-placement need to be installed on the compute host? | 17:26 |
| cdent | prometheanfire: no, it's only an openstack client plugin, for humans | 17:27 |
| prometheanfire | that's what I thought | 17:27 |
| *** Bhujay has quit IRC | 17:31 | |
| sean-k-mooney | prometheanfire: its not needed but it makes debuging placment issues simpeler as you dont have to dive into the db | 17:32 |
| prometheanfire | openstack resource provider list - shows my two nodes | 17:34 |
| sean-k-mooney | prometheanfire: then placement is happy with them. what was the warning you were getting with nova status-check? | 17:34 |
| prometheanfire | one is generation 76 and one is generation 34 though | 17:35 |
| prometheanfire | sean-k-mooney: http://paste.openstack.org/show/729434/ | 17:35 |
| sean-k-mooney | cdent: do you no if nova-status upgrade check uses the api or try to connect to the db directly? | 17:36 |
| cdent | sean-k-mooney: api. it talks to the db for nova stuff, api for placement stuff | 17:37 |
| cdent | sean-k-mooney: yes, it is useful for debugging, but it doesn't have to be _on_ the compute node... | 17:37 |
| sean-k-mooney | prometheanfire: cdent yes. im wondering if nova-status if not finding the placement endoint or somthing in prometheanfire case | 17:38 |
| sean-k-mooney | cdent: sorry ^ was ment for you | 17:38 |
| sean-k-mooney | prometheanfire: do you have the placement setting in the nova.conf also or jsut /etc/nova/nova-cpu.conf? | 17:39 |
| prometheanfire | sean-k-mooney: I have it in nova.conf for all hosts (nova-api and compute hosts) | 17:40 |
| prometheanfire | online-data migrations fail too, I think it's looking in the wrong db here | 17:41 |
| sean-k-mooney | ok i was going to suggest using --config-file=/path/to/conf to make sure its not reading values form another location | 17:41 |
| cdent | sorry sean-k-mooney, I haven't been tracking prometheanfire's situation, just jumping at random points. in too many conversations at once | 17:41 |
| sean-k-mooney | cdent: no worries. | 17:42 |
| sean-k-mooney | prometheanfire: i would try "nova-status --debug --config-dir /etc/nova/nova.conf upgrade check" | 17:42 |
| prometheanfire | same | 17:43 |
| prometheanfire | sean-k-mooney: does placement look at the cell_mappings table of the api db? | 17:44 |
| sean-k-mooney | prometheanfire: i dont think placement does but if you have not do the cellsv2 discover host thing this might have issues | 17:45 |
| sean-k-mooney | prometheanfire: actull i think the output is missleading | 17:46 |
| sean-k-mooney | prometheanfire: i get http://paste.openstack.org/show/729436/ | 17:47 |
| prometheanfire | sean-k-mooney: that's diferent than mine | 17:48 |
| prometheanfire | nova-manage cell_v2 list_hosts | 17:49 |
| prometheanfire | shows my hosts as not mapped to a cell | 17:49 |
| sean-k-mooney | yes it is but 1 i got a success and its an all in one devstrck that is passing tempest test so it works | 17:49 |
| sean-k-mooney | so i think the test might be wrong | 17:49 |
| sean-k-mooney | hum ok well ill try to look into this more tomorow. my brain is fried for today so im going to call it a day o/ | 17:51 |
| prometheanfire | k | 17:52 |
| *** alexchadin has joined #openstack-nova | 17:56 | |
| *** gbarros has quit IRC | 17:57 | |
| prometheanfire | list cells backtraces | 17:57 |
| prometheanfire | http://paste.openstack.org/show/729437/ | 17:57 |
| mriedem | prometheanfire: you still don't have a resource provider in placement but no errors in the n-cpu logs? | 17:57 |
| prometheanfire | mriedem: shows my two nodes | 17:58 |
| mriedem | huh your cells must not have names | 17:59 |
| prometheanfire | my api_db has two entries for cell_mappings | 17:59 |
| prometheanfire | select * from cell_mappings; | 17:59 |
| mriedem | which is the TypeError | 17:59 |
| prometheanfire | ya, one of them has no name | 17:59 |
| mriedem | ok that's a bug clearly | 17:59 |
| mriedem | you want to report it or shall i? | 18:00 |
| *** alexchadin has quit IRC | 18:00 | |
| *** helenafm has joined #openstack-nova | 18:01 | |
| prometheanfire | I'm still figuring stuff out | 18:01 |
| prometheanfire | I think a lot of my bugs in this area are because I moved to cells before it was ready | 18:01 |
| prometheanfire | list_hosts now has the cell name :D | 18:02 |
| prometheanfire | mriedem: can I delete cell0 or is that still used? | 18:02 |
| mriedem | cell0 is required | 18:03 |
| mriedem | it's where instances records are created for things that fail to schedule | 18:03 |
| prometheanfire | k | 18:03 |
| mriedem | "for cell in sorted(cell_mappings, key=lambda _cell: _cell.name):" | 18:04 |
| mriedem | oops | 18:04 |
| mriedem | "because I moved to cells before it was ready" | 18:04 |
| mriedem | as in you moved in ocata? or rocky? | 18:04 |
| mriedem | or... | 18:04 |
| mriedem | and i'm assuming list_cells works now b/c you set a name in the DB? | 18:04 |
| prometheanfire | it does | 18:04 |
| mriedem | i just want to make sure we have a nova bug to track that, i'll open it | 18:04 |
| prometheanfire | online data migrations fail though | 18:05 |
| prometheanfire | it keeps on trying to look up the projects table in the nova database, not the nova_api database | 18:06 |
| *** r-daneel has quit IRC | 18:06 | |
| mriedem | projects table? | 18:06 |
| mriedem | got a paste? | 18:06 |
| melwitt | must be the placement db | 18:06 |
| melwitt | (projects table) | 18:07 |
| *** r-daneel has joined #openstack-nova | 18:07 | |
| prometheanfire | mriedem: http://paste.openstack.org/show/729438/ | 18:07 |
| mriedem | ok so the create_incomplete_consumers online data migration | 18:07 |
| mriedem | prometheanfire: is the placement db defined in nova.conf separately? in [placement_database]? | 18:07 |
| prometheanfire | mriedem: atm, no | 18:08 |
| mriedem | did you run nova-manage api_db sync before running online_data_migrations? | 18:08 |
| mriedem | you've gotta sync the api db schema before running the online data migrations | 18:08 |
| prometheanfire | yep | 18:08 |
| mriedem | https://bugs.launchpad.net/nova/+bug/1790695 btw | 18:08 |
| openstack | Launchpad bug 1790695 in OpenStack Compute (nova) "TypeError in nova-manage cell_v2 list_cells if a cell does not have a name" [High,Triaged] | 18:08 |
| *** helenafm has quit IRC | 18:09 | |
| prometheanfire | where does create_incomplete_consumers get the db that it's connecting to? | 18:09 |
| prometheanfire | also, running nova-manage with --debug doesn't print much debug info :P | 18:11 |
| *** ykarel is now known as ykarel|away | 18:13 | |
| mriedem | create_incomplete_consumers is getting an admin RequestContext which won't have any db connection set on it | 18:13 |
| mriedem | that context should then be changed by @db_api.placement_context_manager.writer | 18:14 |
| jaypipes | prometheanfire, mriedem: create_incomplete_consumers would need to query both the API database (which == placement DB) as well as the nova cell DBs, right? | 18:14 |
| mriedem | which should use the api_database if placement_database isn't configured | 18:14 |
| mriedem | jaypipes: there is nothing about that online data migration that is iterating cell dbs | 18:15 |
| mriedem | and no it doesn't need to hit the cell dbs | 18:16 |
| *** tetsuro has quit IRC | 18:16 | |
| mriedem | it creates missing consumer records for existing allocations records | 18:16 |
| *** tetsuro has joined #openstack-nova | 18:16 | |
| mriedem | what i don't know is if this is trying to connect to the placement database or the api db | 18:16 |
| mriedem | cdent wrote the placement db stuff so he'd likely need to look | 18:17 |
| jaypipes | mriedem: those are the same thing, no? | 18:17 |
| mriedem | no | 18:17 |
| mriedem | not if you configure the placement db | 18:17 |
| jaypipes | but isn't that a deliberate thing? has prometheanfire deliberately configured a placement DB? | 18:17 |
| cdent | mriedem: what's up? | 18:17 |
| prometheanfire | they are the same for me | 18:17 |
| jaypipes | right... | 18:17 |
| mriedem | jaypipes: no he hasn't, | 18:17 |
| mriedem | but i don't know that the code is correctly looking at this | 18:17 |
| cdent | which code? | 18:17 |
| mriedem | cdent: http://paste.openstack.org/show/729438/ | 18:18 |
| openstackgerrit | melanie witt proposed openstack/nova stable/rocky: Add functional test for affinity with multiple cells https://review.openstack.org/599731 | 18:18 |
| prometheanfire | that placement_database stuff only went in for rocky | 18:18 |
| openstackgerrit | melanie witt proposed openstack/nova stable/rocky: Make scheduler.utils.setup_instance_group query all cells https://review.openstack.org/599732 | 18:18 |
| cdent | thanks | 18:18 |
| mriedem | online_data_migrations trying to hit the projects table, | 18:18 |
| mriedem | but i don't know which db it's trying to hit | 18:18 |
| mriedem | prometheanfire: you're testing here is on rocky yes? | 18:18 |
| mriedem | *your | 18:18 |
| prometheanfire | yes | 18:18 |
| mriedem | i believe devstack in rocky is configuring the placement_database, but this should still work without placement_database, otherwise grenade wouldn't work | 18:19 |
| cdent | mriedem: devstack will only use placement_database is a flag var is set, which we set in nova-next, but not elsewhere (unless someone else has changed it) | 18:20 |
| cdent | #PLACEMENT_DB_ENABLED=True | 18:20 |
| mriedem | ok | 18:20 |
| *** tetsuro has quit IRC | 18:20 | |
| *** dtantsur is now known as dtantsur|afk | 18:21 | |
| prometheanfire | no way I can get more debug code? | 18:21 |
| prometheanfire | wasn't there a sqlalchemy setting for more debug? | 18:22 |
| mriedem | yes connection_debug or something | 18:22 |
| mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#api_database.connection_debug | 18:22 |
| prometheanfire | connection_trace = False (Boolean) Add Python stack traces to SQL as comment strings. | 18:23 |
| prometheanfire | that one?? | 18:23 |
| mriedem | there is also connection_debug | 18:23 |
| *** alexchadin has joined #openstack-nova | 18:23 | |
| mriedem | i don't know which is better | 18:23 |
| prometheanfire | ah, that one | 18:23 |
| mriedem | the banner hides everything | 18:23 |
| mriedem | cdent: i don't see what calls this https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/db_api.py#L27 except for the placement wsgi code | 18:24 |
| cdent | mriedem: there's a similar thing in nova-manage for db_sync https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L861-L866 . If that got missed elsewhere, could be a problem | 18:25 |
| prometheanfire | I'm fine re-creating the cell, but I think my instances would be dead at that point | 18:25 |
| mriedem | looks like it used to happen down in the db api code https://review.openstack.org/#/c/541435/ | 18:25 |
| mriedem | before ^ | 18:25 |
| mriedem | prometheanfire: this doesn't have anything to do with the cell db | 18:26 |
| cdent | What was the original command that started this investingation? | 18:27 |
| prometheanfire | ok | 18:27 |
| jaypipes | mriedem: are we able to get all the instance info from the nova API db for create_incomplete_consumers() then? | 18:27 |
| mriedem | cdent: nova-manage db online_data_migrations | 18:27 |
| prometheanfire | cdent: top of paste | 18:27 |
| mriedem | jaypipes: no | 18:27 |
| cdent | thanks | 18:27 |
| *** cfriesen has joined #openstack-nova | 18:27 | |
| *** alexchadin has quit IRC | 18:27 | |
| mriedem | jaypipes: you need the instance.user_id right? | 18:27 |
| jaypipes | mriedem: and instance.project_id. | 18:28 |
| jaypipes | mriedem: that's why I presumed we needed to hit the cell DB. | 18:28 |
| mriedem | we have the project_id in the instance_mappings table in the API DB | 18:28 |
| mriedem | but not the user_id | 18:28 |
| *** luksky has joined #openstack-nova | 18:28 | |
| mriedem | the online data migration, which you wrote i might add :) - relies on the consumer information in the allocations table, which is populated via running nova | 18:28 |
| mriedem | my guess is https://review.openstack.org/#/c/541435/ regressed something but i have no idea how prometheanfire would hit this but we don't in the gate | 18:29 |
| jaypipes | mriedem: we're talking about populating placement with missing *nova* information. | 18:29 |
| prometheanfire | mriedem: old and jankey install | 18:29 |
| prometheanfire | is there a way to re-init the cell db? | 18:29 |
| prometheanfire | I think my cell db for cell1 is the same as the general nova db | 18:30 |
| prometheanfire | not sure that's a good thing :| | 18:30 |
| mriedem | jaypipes: i don't know what you're talking about. looking at the online data migration, clearly it doesn't care about nova instance information from the cell dbs | 18:30 |
| mriedem | prometheanfire: yes that's expected | 18:30 |
| prometheanfire | ok, good | 18:30 |
| mriedem | jaypipes: this https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/objects/consumer.py#L44 | 18:31 |
| prometheanfire | cell0 has it's own db | 18:31 |
| mriedem | prometheanfire: yes | 18:31 |
| prometheanfire | and nova_api | 18:31 |
| cdent | mriedem: that seems like a good guess. is there a chance we don't have tests that exercise the online migrations? | 18:31 |
| mriedem | cdent: devstack runs them | 18:31 |
| mriedem | https://github.com/openstack-dev/devstack/blob/5da7e4a22ede5f3049e7607a54a0f5ca2b413a29/lib/nova#L787 | 18:31 |
| cdent | is there a paste of prometheanfire's nova.conf somewhere? | 18:31 |
| prometheanfire | no | 18:32 |
| jaypipes | mriedem: sorry, I was referring to https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1783 | 18:32 |
| mriedem | jaypipes: yeah totally different thing | 18:32 |
| jaypipes | mriedem: sorry | 18:32 |
| mriedem | np | 18:32 |
| prometheanfire | gimme a sec | 18:32 |
| *** moshele has quit IRC | 18:33 | |
| cdent | mriedem: but those migrations aren't checked for count and if a migration method fails all it does is print? | 18:33 |
| cdent | I might be reading this wrong: https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L677-L682 | 18:34 |
| *** tetsuro has joined #openstack-nova | 18:35 | |
| prometheanfire | cdent: http://paste.openstack.org/show/729439/ | 18:36 |
| *** eharney has quit IRC | 18:36 | |
| cdent | thanks | 18:37 |
| mriedem | well if it failed like it is in prometheanfire's paste, we'd notice | 18:38 |
| mriedem | but lemme check the logs | 18:38 |
| mriedem | maybe that Exception is masking it | 18:38 |
| prometheanfire | mriedem: one reason why I think it's an artifact of the old install | 18:38 |
| mriedem | F ME http://logs.openstack.org/08/599208/2/check/tempest-full/608d60a/controller/logs/devstacklog.txt.gz#_2018-09-02_15_04_31_949 | 18:39 |
| mriedem | yup, broken since rocky | 18:39 |
| mriedem | melwitt: time for RC10 | 18:39 |
| cdent | ugh | 18:39 |
| mriedem | well, added to and broken in rocky | 18:39 |
| * melwitt dies | 18:39 | |
| cdent | didn't papa python always teach us never to catch exception :( | 18:39 |
| mriedem | prometheanfire: you want to create the bug this time? | 18:40 |
| mriedem | it's nearly 2pm and i haven't had lunh yet | 18:40 |
| mriedem | *lunch | 18:40 |
| prometheanfire | mriedem: ya, I'm about to go get lunch I think | 18:40 |
| prometheanfire | mriedem: just the output of my migration in the bug? | 18:41 |
| cdent | if nobody else is aching to fix this tonight, I can do it tomorrow morning | 18:41 |
| prometheanfire | https://bugs.launchpad.net/nova/+bug/1790701 | 18:43 |
| openstack | Launchpad bug 1790701 in OpenStack Compute (nova) "online_data_migrations fail in rocky+" [Undecided,New] | 18:43 |
| prometheanfire | for now | 18:43 |
| prometheanfire | echo $? | 18:44 |
| prometheanfire | 0 | 18:44 |
| prometheanfire | LOL, exits 0 | 18:44 |
| prometheanfire | I'll add that to the bug | 18:44 |
| cdent | should we: a) make a devstack bug to make it be more unhappy when doing the migrations, or b) simply fix the exist code on the script? | 18:45 |
| prometheanfire | imo the exit code being fixed would should make devstack unhappy when it should be unhappy | 18:46 |
| cdent | yes | 18:46 |
| cdent | Until you mentioned the exit code my b was going to be something else that how now passed out of my mind... | 18:47 |
| prometheanfire | :D | 18:47 |
| cdent | so two fixes, in nova: exit code handling on that command, intialize the placement db properly | 18:47 |
| prometheanfire | fixing the error is still nice though | 18:47 |
| cdent | that's backportable then | 18:47 |
| prometheanfire | is that what's happening (api_db sync not init'ing the placement stuff)? | 18:48 |
| mriedem | i'll hack some stuff up | 18:48 |
| mriedem | this isn't api_db sync | 18:48 |
| cdent | then on top of that we need address that an online placement db migration probably shouldn't be in nova's db migrations (even if placement wasn't being extracted) | 18:48 |
| mriedem | but yes the online_data_migraitons command isn't configuring a global properly | 18:48 |
| prometheanfire | ok | 18:49 |
| prometheanfire | I'll be around if you want me to test, I should be able to take a snapshot of my master for rollback type testing too if you want | 18:49 |
| *** med_ has quit IRC | 18:49 | |
| prometheanfire | zfs <3 | 18:49 |
| cdent | mriedem: please ping me if you can't get around to it, or if you do add me to the review please | 18:49 |
| mriedem | sure | 18:50 |
| *** tetsuro has quit IRC | 18:53 | |
| *** moshele has joined #openstack-nova | 18:54 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Swallow fewer exceptions in _run_migration https://review.openstack.org/599744 | 18:55 |
| *** moshele has quit IRC | 18:56 | |
| mriedem | fwiw that blanket try/except has been around since the command was added https://review.openstack.org/#/c/278078/ | 18:56 |
| prometheanfire | mriedem: want a paste with my error with that patch | 18:58 |
| prometheanfire | it's longer | 18:58 |
| mriedem | sure | 18:58 |
| prometheanfire | mriedem: seems to fix the exit code problem too | 18:58 |
| mriedem | yeah b/c you're not getting a nova exception | 18:58 |
| *** itlinux has quit IRC | 18:58 | |
| mriedem | so it kills the command | 18:59 |
| mriedem | i expect my patch to make devstack fail | 18:59 |
| prometheanfire | I don't think it's that helpful (my paste) | 18:59 |
| prometheanfire | http://paste.openstack.org/show/729440/ | 18:59 |
| prometheanfire | but it's there anyway | 18:59 |
| mriedem | yeah it's what i'd expect | 18:59 |
| mriedem | i'll push the real fix in a follow up and then squash them | 18:59 |
| mriedem | so we can see the failure in devstack and the fix in the follow up | 19:00 |
| prometheanfire | wfm, going to lunch | 19:01 |
| prometheanfire | you should too | 19:01 |
| mriedem | eating while typing | 19:04 |
| mriedem | the healthy way | 19:04 |
| prometheanfire | :D | 19:06 |
| openstackgerrit | Jay Pipes proposed openstack/nova-specs master: Standardize CPU resource tracking https://review.openstack.org/555081 | 19:08 |
| openstackgerrit | Jay Pipes proposed openstack/nova-specs master: allow transferring ownership of instance https://review.openstack.org/599598 | 19:19 |
| *** sridharg has quit IRC | 19:21 | |
| *** moshele has joined #openstack-nova | 19:22 | |
| openstackgerrit | Jay Pipes proposed openstack/nova-specs master: allow transferring ownership of instance https://review.openstack.org/599598 | 19:33 |
| openstackgerrit | melanie witt proposed openstack/nova stable/queens: Fix the request context in ServiceFixture https://review.openstack.org/599762 | 19:41 |
| openstackgerrit | melanie witt proposed openstack/nova stable/queens: Honor availability_zone hint via placement https://review.openstack.org/599763 | 19:41 |
| openstackgerrit | melanie witt proposed openstack/nova stable/queens: Improve NeutronFixture and remove unncessary stubbing https://review.openstack.org/599764 | 19:41 |
| openstackgerrit | melanie witt proposed openstack/nova stable/queens: Add functional test for affinity with multiple cells https://review.openstack.org/599765 | 19:41 |
| openstackgerrit | melanie witt proposed openstack/nova stable/queens: Make scheduler.utils.setup_instance_group query all cells https://review.openstack.org/599766 | 19:41 |
| *** tbachman has quit IRC | 19:42 | |
| *** moshele has quit IRC | 19:47 | |
| mriedem | melwitt: rather than https://review.openstack.org/#/c/599763/ i'd probably just add that small 5 LOC to whatever you need that uses it and mention it in the commit message | 19:56 |
| melwitt | mriedem: ok, can do | 19:57 |
| melwitt | mriedem: it's a similar deal with the NeutronFixture changes. I could alternatively add a stub_network_* method call to my test. I didn't need it on master because of the NeutronFixture changes | 19:57 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Configure placement DB context manager for online_data_migrations https://review.openstack.org/599822 | 19:59 |
| mriedem | how big would the stub be on the patch in queens? | 20:00 |
| mriedem | but https://review.openstack.org/#/c/599764/ is a bit gross yeah | 20:01 |
| mriedem | prometheanfire: https://dal05.objectstorage.softlayer.net/v1/AUTH_3d8e6ecb-f597-448c-8ec2-164e9f710dd6/pkvmci/nova/44/599744/1/check/tempest-dsvm-full-xenial/9691036/devstacklog.txt.gz shows that failure as expected | 20:02 |
| mriedem | 2018-09-04 19:18:56.723 | ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'nova_cell0.projects' doesn't exist") [SQL: u'SELECT projects.id \nFROM projects \nWHERE projects.external_id = %(external_id_1)s'] [parameters: {u'external_id_1': '00000000-0000-0000-0000-000000000000'}] (Background on this error at: http://sqlalche.me/e/f405) | 20:02 |
| prometheanfire | :D | 20:03 |
| cdent | mriedem: is it normal for there to be NovaExceptions that we would want to only print? | 20:05 |
| mriedem | i'd say no | 20:05 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Configure placement DB context manager for online_data_migrations https://review.openstack.org/599744 | 20:05 |
| mriedem | i've removed it | 20:05 |
| cdent | huzza | 20:05 |
| cdent | h | 20:05 |
| mriedem | ^ is the old squasharoo | 20:05 |
| mriedem | oops guess i missed that in the squash | 20:06 |
| melwitt | mriedem: I mean I could add a fake_network.set_stub_network_methods(self) all to my test instead of backporting the changes that removed all of those calls | 20:08 |
| melwitt | *call | 20:08 |
| mriedem | melwitt: you mean like *add* to your backport test what was removed from here right? https://review.openstack.org/#/c/599764/1/nova/tests/functional/db/test_archive.py | 20:12 |
| mriedem | melwitt: if so, then yes just do that | 20:12 |
| mriedem | the neutron fixture backport is very weird otherwise | 20:12 |
| *** med_ has joined #openstack-nova | 20:13 | |
| melwitt | mriedem: yes, add fake_network.set_stub_network_methods(self) to my backport test | 20:13 |
| melwitt | k | 20:13 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Configure placement DB context manager for online_data_migrations https://review.openstack.org/599744 | 20:13 |
| mriedem | prometheanfire: ^ should make your dreams finally come true | 20:14 |
| prometheanfire | mriedem: mostly, 'nova-status upgrade check' still shows a warning | 20:15 |
| mriedem | paste me | 20:15 |
| prometheanfire | but online migrations work (with nothing migrated) | 20:15 |
| prometheanfire | mriedem: http://paste.openstack.org/show/729448/ | 20:15 |
| *** alexchadin has joined #openstack-nova | 20:16 | |
| prometheanfire | mriedem: I thought sean-k-mooney mentioned that being a false positive or something though | 20:17 |
| mriedem | hmm, how many nodes are shown with nova hypervisor-list? | 20:19 |
| prometheanfire | 2 | 20:22 |
| mriedem | and how many in openstack resource provider list? | 20:24 |
| prometheanfire | 2 | 20:26 |
| *** holser_ has joined #openstack-nova | 20:26 | |
| mriedem | hmm, looks like we have a bug then | 20:28 |
| mriedem | you want to write that up? | 20:28 |
| prometheanfire | sure | 20:29 |
| prometheanfire | I think this one is not pike only, I think I saw this in queens too | 20:29 |
| prometheanfire | maybe more, not sure | 20:29 |
| prometheanfire | s/pike/rocky | 20:29 |
| prometheanfire | I keep on calling rocky pike | 20:29 |
| cdent | criminey | 20:32 |
| mriedem | prometheanfire: and list_cells shows cell0 and cell1 right? | 20:34 |
| prometheanfire | https://bugs.launchpad.net/nova/+bug/1790721 | 20:34 |
| openstack | Launchpad bug 1790721 in OpenStack Compute (nova) "nova-status upgrade check shows warnings when it shouldn't" [Undecided,New] | 20:34 |
| prometheanfire | yep, shows both cells | 20:34 |
| mriedem | i think it's the same bug | 20:35 |
| mriedem | we're hitting the api db using this placement context manager, but it's not configured for the api db | 20:35 |
| mriedem | so it's hitting cell0 looking for resource providers | 20:35 |
| cdent | blargh. I thought nova-status use the api for placement checks? | 20:36 |
| prometheanfire | ok, so partial fix so far then (I think) | 20:36 |
| mriedem | cdent: there is a TODO from me in that same coe | 20:37 |
| mriedem | *code | 20:37 |
| cdent | ah | 20:37 |
| prometheanfire | mriedem: ya, read that :P | 20:37 |
| mriedem | we do hit the API to check that we can *talk* to placement | 20:37 |
| prometheanfire | in _count_compute_resource_providers | 20:37 |
| mriedem | yeah this one, "Check: Placement API" | 20:37 |
| mriedem | that makes sure we can at least get to placement | 20:37 |
| *** r-daneel_ has joined #openstack-nova | 20:38 | |
| *** r-daneel has quit IRC | 20:38 | |
| *** r-daneel_ is now known as r-daneel | 20:38 | |
| *** itlinux has joined #openstack-nova | 20:41 | |
| openstackgerrit | melanie witt proposed openstack/nova stable/queens: Add functional test for affinity with multiple cells https://review.openstack.org/599765 | 20:42 |
| openstackgerrit | melanie witt proposed openstack/nova stable/queens: Make scheduler.utils.setup_instance_group query all cells https://review.openstack.org/599766 | 20:42 |
| mriedem | i should probably add nova-status upgrade check to devstack first, but i think that has to be run *after* the subnodes, if any, are up, which means we need to call back into devstack from d-g | 20:43 |
| mriedem | like we do with discover_hosts | 20:43 |
| prometheanfire | iirc it's run right before online migrations | 20:44 |
| prometheanfire | at least the upgrade doc makes me think that | 20:44 |
| mriedem | you can run it on base install too | 20:44 |
| mriedem | to verify the deployment | 20:44 |
| prometheanfire | ah | 20:45 |
| openstackgerrit | melanie witt proposed openstack/nova stable/pike: Fix the request context in ServiceFixture https://review.openstack.org/599839 | 20:45 |
| openstackgerrit | melanie witt proposed openstack/nova stable/pike: Add functional test for affinity with multiple cells https://review.openstack.org/599840 | 20:45 |
| openstackgerrit | melanie witt proposed openstack/nova stable/pike: Make scheduler.utils.setup_instance_group query all cells https://review.openstack.org/599841 | 20:45 |
| openstackgerrit | melanie witt proposed openstack/nova stable/pike: Fix the request context in ServiceFixture https://review.openstack.org/599839 | 20:46 |
| openstackgerrit | melanie witt proposed openstack/nova stable/pike: Add functional test for affinity with multiple cells https://review.openstack.org/599840 | 20:46 |
| openstackgerrit | melanie witt proposed openstack/nova stable/pike: Make scheduler.utils.setup_instance_group query all cells https://review.openstack.org/599841 | 20:46 |
| *** cdent has quit IRC | 20:47 | |
| *** holser_ has quit IRC | 20:49 | |
| *** alexchadin has quit IRC | 20:51 | |
| *** Sundar has joined #openstack-nova | 20:51 | |
| *** eharney has joined #openstack-nova | 20:51 | |
| Sundar | melwitt: Please ping me when you have a moment. | 20:55 |
| melwitt | Sundar: hi | 20:56 |
| mriedem | prometheanfire: well we'll see if this notices it https://review.openstack.org/599847 | 20:56 |
| Sundar | melwitt: You had asked for a Nova spec for accelerator-related things. The only open AFAICS is how Cyborg will interact with placement: through virt drivers or by calling placement directly. | 20:56 |
| Sundar | Most other aspects are already addressed in the Cyborg/Nova scheduling spec (https://review.openstack.org/#/c/554717/) or in the ongoing os-acc spec (https://review.openstack.org/#/c/577438/). | 20:56 |
| Sundar | So, can the new spec just point to the older specs for those aspects? | 20:57 |
| prometheanfire | mriedem: just had to be sure, but at least that exit code works (got a 1) | 20:57 |
| Sundar | May be we need a bit more detail on how exactly the virt drivers will invoke os-acc. I can add that to os-acc spec. | 20:58 |
| melwitt | Sundar: you can and should add links to the other specs as references in the nova spec, but the nova spec should describe the proposed changes to nova as part of the interaction. the references can be for background reading and then the new spec will detail the nova changes that will be needed and those are what we will review (after reading the referenced other specs that you should add to the References section of the spec) | 21:00 |
| melwitt | we just want to be able to review the proposal for nova changes as a nova spec | 21:00 |
| * prometheanfire needs to start packaging earlier in the cycle so these bugs are hit in rc still | 21:01 | |
| *** imacdonn has quit IRC | 21:02 | |
| *** imacdonn has joined #openstack-nova | 21:02 | |
| Sundar | melwitt: OK. I'll take a stab. We can iterate from there as needed. Thank you. | 21:03 |
| melwitt | cool, thanks | 21:03 |
| melwitt | Sundar: btw, what day/time are you having the nova/placement interaction session at the cyborg room? | 21:03 |
| melwitt | Sundar: and would sometime between 11:10am and 12:30pm on thursday work for you for Cyborg/Nova session at the nova room? | 21:05 |
| *** erlon has quit IRC | 21:05 | |
| *** awaugama has quit IRC | 21:11 | |
| Sundar | melwitt: It doesn't look like the Cyborg times are decided yet! https://etherpad.openstack.org/p/cyborg-ptg-stein | 21:17 |
| Sundar | I'll ask Cyborg PTL and get this resolved | 21:17 |
| Sundar | I am fine with your proposed time on Thursday | 21:17 |
| melwitt | ok, just let me know so I can make a note on our etherpad so folks know when to show up at the cyborg room | 21:18 |
| Sundar | Yes, sure. Thanks! | 21:18 |
| *** luksky has quit IRC | 21:28 | |
| *** itlinux is now known as itlinux-away | 21:30 | |
| *** itlinux-away is now known as itlinux | 21:30 | |
| *** itlinux is now known as itlinux-away | 21:30 | |
| *** itlinux-away is now known as itlinux | 21:34 | |
| *** itlinux is now known as itlinux-away | 21:35 | |
| *** itlinux-away is now known as itlinux | 21:35 | |
| *** itlinux is now known as itlinux-away | 21:35 | |
| mriedem | lbragstad: we probably need your keystone eyeballs on this https://review.openstack.org/#/c/599598/3/specs/stein/approved/transfer-instance-ownership.rst@140 | 21:40 |
| mriedem | tl;dr should nova be responsible for checking that a given user is in a given project | 21:40 |
| lbragstad | mriedem looking | 21:40 |
| mriedem | i guess GET /v3/users/{user_id}/projects would be pretty easy though | 21:41 |
| *** munimeha1 has quit IRC | 21:47 | |
| *** itlinux-away is now known as itlinux | 22:01 | |
| *** itlinux has quit IRC | 22:02 | |
| *** Sundar has quit IRC | 22:06 | |
| *** r-daneel has quit IRC | 22:08 | |
| lbragstad | mriedem yeah - we have another API like that, too | 22:11 |
| lbragstad | https://developer.openstack.org/api-ref/identity/v3/index.html#list-role-assignments-for-user-on-project | 22:11 |
| lbragstad | so long as there is a role returned, then that might be enough for nova http://paste.openstack.org/raw/729458/ | 22:18 |
| *** priteau has quit IRC | 22:21 | |
| mriedem | lbragstad: ok and if nova is configured with admin / service user creds to keystone, can nova get the user information even if the current token in the request context is not for that user? | 22:21 |
| mriedem | i guess it probably depends on what auth nova's keystone creds are configured with | 22:22 |
| *** mchlumsky has quit IRC | 22:22 | |
| mriedem | but we use that today to verify a provided tenant id exists | 22:22 |
| mriedem | for certain apis that take a tenant id | 22:22 |
| lbragstad | yeah the token used to call that API is going to have to belong to the nova service user | 22:23 |
| lbragstad | right now that policy is protected by rule:admin_required | 22:24 |
| lbragstad | but - if that's too strict a default for nova, we do have work staged for stein to rework the authorization of that API https://bugs.launchpad.net/keystone/+bug/1750669 | 22:25 |
| openstack | Launchpad bug 1750669 in OpenStack Identity (keystone) "The v3 grant API should account for different scopes" [High,Triaged] | 22:25 |
| lbragstad | nova should have access to the token used to make the request via the context object | 22:27 |
| lbragstad | if you're expecting only system administrators to call this API, then you might be able to reuse that token to call the role API in keysotne | 22:28 |
| melwitt | that reminds me of a wishlist bug we have open where people would like us to validate the user_id when updating quota limits, same as we validate the project_id currently | 22:29 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix TypeError in nova-manage cell_v2 list_cells https://review.openstack.org/599861 | 22:30 |
| lbragstad | we have an api for that but it too would require administrator access | 22:32 |
| lbragstad | or it would require the nova service user to be an administrator | 22:32 |
| lbragstad | but, hopefully that's going to be changing soon with https://bugs.launchpad.net/keystone/+bug/1748027 | 22:33 |
| openstack | Launchpad bug 1748027 in OpenStack Identity (keystone) "The v3 users API should account for different scopes" [High,Triaged] - Assigned to sonu (sonu-bhumca11) | 22:33 |
| melwitt | I see. but the validation of project_id is a non-admin thing? | 22:33 |
| lbragstad | i guess it depends on how you're validating the project id | 22:33 |
| melwitt | I have to check. I don't remember how we're doing it | 22:34 |
| lbragstad | are you calling the GET /v3/projects/{project_id} | 22:34 |
| melwitt | lemme see | 22:34 |
| melwitt | yup, looks like it | 22:35 |
| lbragstad | nice - in that case we do rule:admin_required or project_id:%(target.project.id)s | 22:35 |
| lbragstad | making it accessible to admins and users with a role assignment on the project in the path | 22:35 |
| melwitt | https://github.com/openstack/nova/blob/master/nova/api/openstack/identity.py#L41 | 22:37 |
| lbragstad | if that ksa session is built with the nova service user, then it looks like you're already handling the case where nova doens't have the necessary permissions https://github.com/openstack/nova/blob/master/nova/api/openstack/identity.py#L61-L68 | 22:38 |
| lbragstad | so that's good | 22:38 |
| lbragstad | unless you're hoping to switch that case to false eventually | 22:39 |
| melwitt | trying to see if it's built with the nova service user (sorry, this is new to me) | 22:39 |
| lbragstad | https://github.com/openstack/nova/blob/master/nova/utils.py#L1189-L1191 ? | 22:40 |
| melwitt | yeah, and we're passing ksa_auth=context.get_auth_plugin() | 22:41 |
| lbragstad | looks like it supports being passed a ksa session and building one from config | 22:41 |
| mriedem | yes https://docs.openstack.org/nova/latest/configuration/config.html#keystone | 22:41 |
| mriedem | it's whatever you configure nova with | 22:41 |
| mriedem | if you don't and we can't verify, we would default to our old behavior which is, meh - hope you know what you're doing admin person! | 22:42 |
| melwitt | ah, ok | 22:42 |
| *** erlon has joined #openstack-nova | 22:42 | |
| lbragstad | huh - ok | 22:43 |
| melwitt | # we don't have enough permission to verify this, so default | 22:44 |
| melwitt | # to "it's ok". | 22:44 |
| melwitt | haha | 22:44 |
| lbragstad | so - the same creds used by ksm? | 22:44 |
| lbragstad | "move along citizen.. nothing to see here" | 22:44 |
| melwitt | haha | 22:45 |
| *** spartakos has joined #openstack-nova | 22:48 | |
| melwitt | should be whatever is in the [keystone] section of the nova.conf, though I don't see that configured in the gate nova.conf | 22:52 |
| melwitt | I see only [keystone_authtoken] for example, in here http://logs.openstack.org/99/584999/5/check/tempest-full/29581fd/controller/logs/etc/nova/nova_conf.txt.gz | 22:53 |
| lbragstad | yeah - that's what we look for in ksm for sure | 22:54 |
| melwitt | oh, ok | 22:54 |
| lbragstad | i'm wondering if that gets re-used for nova requests | 22:54 |
| lbragstad | look like it - it should be available to nova via the config object | 22:55 |
| *** rcernin has joined #openstack-nova | 22:55 | |
| melwitt | maybe this is how it gets re-used? https://github.com/openstack/nova/blob/master/nova/context.py#L59-L64 | 22:57 |
| melwitt | we do that as part of the context.get_auth_plugin() call, if there's no self.user_auth_plugin set https://github.com/openstack/nova/blob/master/nova/context.py#L160 | 22:58 |
| *** itlinux has joined #openstack-nova | 23:04 | |
| *** cfriesen has quit IRC | 23:06 | |
| *** priteau has joined #openstack-nova | 23:23 | |
| *** mlavalle has quit IRC | 23:33 | |
| *** itlinux is now known as itlinux-away | 23:34 | |
| *** brinzhang has joined #openstack-nova | 23:42 | |
| lbragstad | aha - interesting | 23:48 |
| *** itlinux-away is now known as itlinux | 23:50 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!