| *** macza has quit IRC | 00:09 | |
| openstackgerrit | melanie witt proposed openstack/nova master: Fixed concurrent access to direct io test file https://review.openstack.org/515091 | 00:11 |
|---|---|---|
| *** licanwei has quit IRC | 00:20 | |
| *** igordc has quit IRC | 00:33 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Add functional regression test for bug 1794996 https://review.openstack.org/623358 | 00:40 |
| openstack | bug 1794996 in OpenStack Compute (nova) rocky "_destroy_evacuated_instances fails and kills n-cpu startup if lazy-loading flavor on a deleted instance" [High,In progress] https://launchpad.net/bugs/1794996 - Assigned to Matt Riedemann (mriedem) | 00:40 |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Fix InstanceNotFound during _destroy_evacuated_instances https://review.openstack.org/623359 | 00:40 |
| *** mriedem has quit IRC | 00:45 | |
| *** rodolof has joined #openstack-nova | 00:46 | |
| *** Swami has quit IRC | 00:49 | |
| *** igordc has joined #openstack-nova | 01:01 | |
| *** gyee has quit IRC | 01:09 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Handle tags in _bury_in_cell0 https://review.openstack.org/621856 | 01:11 |
| *** igordc has quit IRC | 01:16 | |
| *** igordc has joined #openstack-nova | 01:16 | |
| *** igordc has quit IRC | 01:16 | |
| *** brinzhang has joined #openstack-nova | 01:21 | |
| *** wolverineav has quit IRC | 01:55 | |
| *** yedongcan has joined #openstack-nova | 01:58 | |
| *** betherly has joined #openstack-nova | 01:59 | |
| *** dave-mccowan has quit IRC | 02:00 | |
| *** betherly has quit IRC | 02:04 | |
| *** mrsoul has joined #openstack-nova | 02:12 | |
| *** Dinesh_Bhor has joined #openstack-nova | 02:12 | |
| *** Sundar has quit IRC | 02:16 | |
| openstackgerrit | melanie witt proposed openstack/nova-specs master: Propose counting quota usage from placement and API database https://review.openstack.org/509042 | 02:26 |
| *** cfriesen has quit IRC | 02:27 | |
| *** takashin has left #openstack-nova | 02:31 | |
| *** mhen has quit IRC | 02:34 | |
| *** mhen has joined #openstack-nova | 02:38 | |
| *** hongbin has joined #openstack-nova | 02:41 | |
| *** psachin has joined #openstack-nova | 02:53 | |
| *** imacdonn has quit IRC | 02:53 | |
| *** imacdonn has joined #openstack-nova | 02:53 | |
| *** awaugama has quit IRC | 03:00 | |
| *** betherly has joined #openstack-nova | 03:01 | |
| *** betherly has quit IRC | 03:05 | |
| *** dave-mccowan has joined #openstack-nova | 03:10 | |
| *** dave-mccowan has quit IRC | 03:19 | |
| *** lbragstad has quit IRC | 04:23 | |
| *** psachin has quit IRC | 04:41 | |
| *** wolverineav has joined #openstack-nova | 04:44 | |
| *** janki has joined #openstack-nova | 04:46 | |
| *** psachin has joined #openstack-nova | 04:58 | |
| *** diga has joined #openstack-nova | 05:04 | |
| *** rodolof has quit IRC | 05:10 | |
| *** licanwei has joined #openstack-nova | 05:12 | |
| openstackgerrit | Merged openstack/nova master: libvirt: Refactor handling of PCIe root ports https://review.openstack.org/620327 | 05:20 |
| openstackgerrit | Merged openstack/nova stable/queens: Make supports_direct_io work on 4096b sector size https://review.openstack.org/619220 | 05:20 |
| openstackgerrit | Merged openstack/nova stable/queens: Add regression test for bug #1764883 https://review.openstack.org/621199 | 05:20 |
| openstack | bug 1764883 in OpenStack Compute (nova) queens "Evacuation fails if the source host returns while the migration is still in progress" [Medium,In progress] https://launchpad.net/bugs/1764883 - Assigned to Lee Yarwood (lyarwood) | 05:20 |
| *** wolverineav has quit IRC | 05:21 | |
| *** takashin has joined #openstack-nova | 05:25 | |
| *** rodolof has joined #openstack-nova | 05:25 | |
| *** cfriesen has joined #openstack-nova | 05:26 | |
| *** rodolof has quit IRC | 05:31 | |
| *** wolverineav has joined #openstack-nova | 05:32 | |
| *** wolverineav has quit IRC | 05:36 | |
| *** ratailor has joined #openstack-nova | 05:57 | |
| *** sridharg has joined #openstack-nova | 05:57 | |
| *** Luzi has joined #openstack-nova | 06:03 | |
| *** diga has quit IRC | 06:04 | |
| *** Dinesh_Bhor has quit IRC | 06:04 | |
| *** takashin has left #openstack-nova | 06:30 | |
| *** yikun_ has joined #openstack-nova | 06:32 | |
| *** hongbin has quit IRC | 06:32 | |
| *** ohorecny2 has joined #openstack-nova | 06:37 | |
| openstackgerrit | Merged openstack/nova stable/queens: compute: Ensure pre-migrating instances are destroyed during init_host https://review.openstack.org/621200 | 06:37 |
| *** lei-zh has joined #openstack-nova | 06:38 | |
| openstackgerrit | Merged openstack/nova stable/pike: Update docs for _destroy_evacuated_instances https://review.openstack.org/621203 | 06:39 |
| *** Dinesh_Bhor has joined #openstack-nova | 06:53 | |
| *** tetsuro has joined #openstack-nova | 06:55 | |
| *** tetsuro_ has joined #openstack-nova | 06:58 | |
| *** tetsuro has quit IRC | 07:01 | |
| *** rcernin has quit IRC | 07:01 | |
| *** betherly has joined #openstack-nova | 07:08 | |
| *** dklyle has quit IRC | 07:09 | |
| *** dklyle has joined #openstack-nova | 07:10 | |
| *** betherly has quit IRC | 07:13 | |
| *** cfriesen has quit IRC | 07:14 | |
| *** tetsuro_ has quit IRC | 07:18 | |
| *** belmoreira has quit IRC | 07:21 | |
| *** tetsuro has joined #openstack-nova | 07:22 | |
| *** trident has quit IRC | 07:23 | |
| *** dpawlik has joined #openstack-nova | 07:24 | |
| *** belmoreira has joined #openstack-nova | 07:24 | |
| *** trident has joined #openstack-nova | 07:25 | |
| *** tetsuro_ has joined #openstack-nova | 07:28 | |
| *** tetsuro has quit IRC | 07:29 | |
| *** alexchadin has joined #openstack-nova | 07:35 | |
| *** tetsuro_ has quit IRC | 07:37 | |
| *** tetsuro has joined #openstack-nova | 07:40 | |
| *** dims has quit IRC | 07:44 | |
| *** dims has joined #openstack-nova | 07:47 | |
| *** awalende has joined #openstack-nova | 08:11 | |
| *** rambo_li has joined #openstack-nova | 08:12 | |
| *** Dinesh_Bhor has quit IRC | 08:14 | |
| *** rambo_li has quit IRC | 08:19 | |
| *** Dinesh_Bhor has joined #openstack-nova | 08:19 | |
| *** rambo_li has joined #openstack-nova | 08:23 | |
| *** awalende has quit IRC | 08:30 | |
| *** mcgiggler has joined #openstack-nova | 08:35 | |
| *** tetsuro_ has joined #openstack-nova | 08:36 | |
| *** tetsuro has quit IRC | 08:38 | |
| *** tetsuro_ has quit IRC | 08:55 | |
| *** yan0s has joined #openstack-nova | 08:55 | |
| *** sahid has joined #openstack-nova | 08:55 | |
| *** ratailor has quit IRC | 08:56 | |
| *** tetsuro has joined #openstack-nova | 08:58 | |
| *** ratailor has joined #openstack-nova | 08:58 | |
| *** KeithMnemonic has quit IRC | 09:00 | |
| *** KeithMnemonic has joined #openstack-nova | 09:01 | |
| *** ccamacho has joined #openstack-nova | 09:09 | |
| *** ccamacho has quit IRC | 09:09 | |
| *** tssurya has joined #openstack-nova | 09:10 | |
| *** Dinesh_Bhor has quit IRC | 09:11 | |
| *** Dinesh_Bhor has joined #openstack-nova | 09:12 | |
| *** sahid has quit IRC | 09:14 | |
| *** sahid has joined #openstack-nova | 09:14 | |
| *** tetsuro_ has joined #openstack-nova | 09:14 | |
| *** ccamacho has joined #openstack-nova | 09:15 | |
| *** tetsuro has quit IRC | 09:18 | |
| *** licanwei has quit IRC | 09:18 | |
| *** alexchadin has quit IRC | 09:24 | |
| *** k_mouza has joined #openstack-nova | 09:27 | |
| *** tetsuro_ has quit IRC | 09:29 | |
| *** tetsuro has joined #openstack-nova | 09:30 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Ignore MoxStubout deprecation warnings https://review.openstack.org/623309 | 09:30 |
| *** alex_xu has quit IRC | 09:30 | |
| *** lei-zh has quit IRC | 09:33 | |
| *** Dinesh_Bhor has quit IRC | 09:40 | |
| *** k_mouza has quit IRC | 09:42 | |
| *** tetsuro has quit IRC | 09:42 | |
| *** cdent has joined #openstack-nova | 09:42 | |
| *** tetsuro has joined #openstack-nova | 09:43 | |
| *** k_mouza has joined #openstack-nova | 09:43 | |
| cdent | stephenfin: I'm confused and frustrated and stuck on https://review.openstack.org/#/c/622972/ If you have any ideas, or can just fix it, that would be most helpful. | 09:43 |
| * cdent makes more coffee | 09:43 | |
| *** tetsuro_ has joined #openstack-nova | 09:54 | |
| *** tetsuro has quit IRC | 09:55 | |
| *** tetsuro has joined #openstack-nova | 10:03 | |
| *** tetsuro_ has quit IRC | 10:04 | |
| *** maciejjozefczyk has quit IRC | 10:05 | |
| *** maciejjozefczyk has joined #openstack-nova | 10:07 | |
| *** ttsiouts has joined #openstack-nova | 10:16 | |
| *** trident has quit IRC | 10:18 | |
| mdbooth | lyarwood: https://review.openstack.org/#/c/618478/ I agree with melwitt's comment that instance.host checking is a risk. Could you audit it? We're hopefully ok. | 10:20 |
| lyarwood | mdbooth: I've had a look already and I couldn't see a way for us to call cleanup on the source during a failure | 10:21 |
| *** trident has joined #openstack-nova | 10:21 | |
| lyarwood | mdbooth: we would call it on the dest but the whole point of the workaround is to ensure the directory is cleaned when it _isn't_ shared | 10:21 |
| lyarwood | mdbooth: so if it's enabled in that situation that's on the operator | 10:21 |
| mdbooth | lyarwood: I would personally start by looking at everywhere in ComputeManager which sets instance.host, and work back from there. Don't assume it necessarily makes sense. | 10:22 |
| lyarwood | mdbooth: there's zero point in looking at that if we don't call cleanup on the source during a failure | 10:22 |
| *** dpawlik has quit IRC | 10:23 | |
| *** dpawlik has joined #openstack-nova | 10:23 | |
| lyarwood | mdbooth: which we wouldn't do for LM, either post-copy or pre-copy AFAIK | 10:23 |
| lyarwood | I'll look again at where we are calling cleanup to confirm but I'm sure this isn't an issue | 10:23 |
| mdbooth | lyarwood: I strongly suspect it isn't an issue. | 10:24 |
| *** tetsuro_ has joined #openstack-nova | 10:24 | |
| * mdbooth is +0.8 ;) | 10:24 | |
| * lyarwood files a gerrit RFE for review range sliders | 10:25 | |
| lyarwood | DNM 0 [-------x--] 1 LGTM | 10:25 |
| mdbooth | lyarwood: +1 | 10:26 |
| *** tetsuro has quit IRC | 10:27 | |
| mdbooth | lyarwood: Are there any circumstances where we could end up calling cleanup on the source (e.g. init_host, periodic) where instance.host is erroneously set to dest? | 10:27 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Correct lower-constraints.txt and the related tox job https://review.openstack.org/622972 | 10:28 |
| *** k_mouza has quit IRC | 10:29 | |
| *** ttsiouts has quit IRC | 10:31 | |
| lyarwood | mdbooth: cleaning up evacuations that have failed in a weird way but again with this workaround enabled you'd only see console.log, kernel etc removed | 10:31 |
| *** ccamacho has quit IRC | 10:31 | |
| *** ttsiouts has joined #openstack-nova | 10:31 | |
| lyarwood | mdbooth: I can't say I've ever seen an evacuation fail in that way however | 10:31 |
| lyarwood | mdbooth: instance.host updated but the instance is still on the source | 10:32 |
| mdbooth | s/still on the source/isn't on the dest/ | 10:32 |
| mdbooth | although we still wouldn't have deleted the actual instance data as you say | 10:32 |
| mdbooth | So probably not the worst. | 10:33 |
| *** ccamacho has joined #openstack-nova | 10:41 | |
| *** brinzhang has quit IRC | 10:48 | |
| *** mdbooth has quit IRC | 10:48 | |
| *** alex_xu has joined #openstack-nova | 10:48 | |
| * alex_xu is in vacation for next whole week | 10:49 | |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: API microversion 2.68: Handles Down Cells https://review.openstack.org/591657 | 10:49 |
| gibi | alex_xu: have a nice vacation | 10:49 |
| *** sapd1_ has joined #openstack-nova | 10:50 | |
| *** k_mouza has joined #openstack-nova | 10:52 | |
| *** rambo_li has quit IRC | 10:57 | |
| *** wolverineav has joined #openstack-nova | 11:00 | |
| *** cdent has quit IRC | 11:01 | |
| *** tetsuro_ has quit IRC | 11:02 | |
| *** trident has quit IRC | 11:04 | |
| *** wolverineav has quit IRC | 11:05 | |
| *** trident has joined #openstack-nova | 11:06 | |
| *** maciejjozefczyk has quit IRC | 11:13 | |
| *** maciejjozefczyk has joined #openstack-nova | 11:15 | |
| *** sapd1_ has quit IRC | 11:17 | |
| *** sapd1_ has joined #openstack-nova | 11:20 | |
| *** cdent has joined #openstack-nova | 11:28 | |
| *** tbachman has quit IRC | 11:29 | |
| *** sc has joined #openstack-nova | 11:30 | |
| *** sapd1_ has quit IRC | 11:35 | |
| openstackgerrit | Dongcan Ye proposed openstack/nova master: Remove other volume snapshot type https://review.openstack.org/623456 | 11:36 |
| *** yedongcan has left #openstack-nova | 11:46 | |
| *** sapd1_ has joined #openstack-nova | 11:51 | |
| *** dtantsur|afk is now known as dtantsur\ | 11:54 | |
| *** dtantsur\ is now known as dtantsur | 11:54 | |
| *** sapd1_ has quit IRC | 11:56 | |
| *** slaweq has joined #openstack-nova | 12:03 | |
| openstackgerrit | sean mooney proposed openstack/os-vif master: add isolate_vif config option https://review.openstack.org/612534 | 12:22 |
| *** mcgiggler has quit IRC | 12:27 | |
| openstackgerrit | Gaudenz Steinlin proposed openstack/nova master: Extend volume for libvirt network volumes (RBD) https://review.openstack.org/613039 | 12:39 |
| *** trident has quit IRC | 12:39 | |
| *** kaisers has quit IRC | 12:39 | |
| *** trident has joined #openstack-nova | 12:41 | |
| *** psachin has quit IRC | 12:41 | |
| cdent | stephenfin: it looks like your change on the constraints job gets things to pass, which suggests that skipsdist is not working in a per-env setting? | 12:47 |
| stephenfin | cdent: Yup, I think I called that out in a comment/the commit message? | 12:47 |
| cdent | but it's also confusing that different behaviors are happening between my box and the gate. | 12:47 |
| stephenfin | cdent: It's a global option | 12:47 |
| *** tbachman has joined #openstack-nova | 12:47 | |
| stephenfin | I'd imagine it's the version of tox | 12:48 |
| cdent | they are the same | 12:48 |
| cdent | that's why it is confusing | 12:48 |
| stephenfin | Hmm. Have you wiped your .tox directory? | 12:48 |
| cdent | not the whole thing, but the specific env | 12:49 |
| cdent | the vm I was using is at home, so I cant' look now, but will check when I'm back home | 12:49 |
| stephenfin | I've no idea what's going on, in that case. Might be worth asking the tox devs, in case they've any ideas. The funkiness that pbr introduces could throw even them though | 12:50 |
| cdent | I'll also check to make sure that the job is sitll working as designed | 12:50 |
| *** Luzi has quit IRC | 12:51 | |
| cdent | yeah, pbr... | 12:51 |
| cdent | if it's happy as is, I'm happy to just move ... | 12:51 |
| cdent | on | 12:51 |
| cdent | wasted way more time on this than I wanted | 12:51 |
| *** tbachman has quit IRC | 12:52 | |
| cdent | "wasted way more time on this than I wanted" <- nova, defined | 12:55 |
| *** kaisers has joined #openstack-nova | 12:56 | |
| *** tbachman has joined #openstack-nova | 12:58 | |
| *** sapd1_ has joined #openstack-nova | 13:02 | |
| openstackgerrit | Silvan Kaiser proposed openstack/nova master: Exec systemd-run without --user flag in Quobyte driver https://review.openstack.org/554195 | 13:04 |
| *** sapd1_ has quit IRC | 13:07 | |
| *** artom has quit IRC | 13:11 | |
| *** cdent has quit IRC | 13:17 | |
| *** ratailor has quit IRC | 13:22 | |
| *** mlavalle has joined #openstack-nova | 13:24 | |
| *** janki has quit IRC | 13:24 | |
| openstackgerrit | Merged openstack/os-vif master: always create ovs port during plug https://review.openstack.org/602384 | 13:33 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP compute: reject migration request when source compute is disabled https://review.openstack.org/623489 | 13:34 |
| openstackgerrit | Merged openstack/nova master: Use tempest [compute]/build_timeout in evacuate tests https://review.openstack.org/623011 | 13:39 |
| *** edleafe- has joined #openstack-nova | 13:42 | |
| *** edmondsw has quit IRC | 13:42 | |
| *** edleafe has quit IRC | 13:43 | |
| *** edleafe- is now known as edleafe | 13:43 | |
| *** cfriesen has joined #openstack-nova | 13:45 | |
| *** _pewp_ has quit IRC | 13:49 | |
| sc | I need some help w/ configuring PCI passthrough in a newton based installation, I followed the newton admin guide but the scheduler returns 0 of 6 hosts. | 13:50 |
| sc | what do you suggest to look for? In the compute host dmesg reports the iommu as enabled and I see a nice list of PCI cards, including the GPU I would like to pass to my VM | 13:51 |
| *** tssurya has quit IRC | 13:59 | |
| *** ttsiouts has quit IRC | 14:01 | |
| *** ttsiouts has joined #openstack-nova | 14:01 | |
| *** ttsiouts has quit IRC | 14:06 | |
| *** eharney has quit IRC | 14:06 | |
| *** lbragstad has joined #openstack-nova | 14:06 | |
| *** edmondsw has joined #openstack-nova | 14:12 | |
| *** ttsiouts has joined #openstack-nova | 14:12 | |
| *** _hemna has quit IRC | 14:13 | |
| *** tbachman has quit IRC | 14:17 | |
| *** cdent has joined #openstack-nova | 14:18 | |
| *** ttsiouts has quit IRC | 14:18 | |
| *** mriedem has joined #openstack-nova | 14:25 | |
| *** lbragstad has quit IRC | 14:25 | |
| *** lbragstad has joined #openstack-nova | 14:29 | |
| openstackgerrit | Silvan Kaiser proposed openstack/nova master: Exec systemd-run without --user flag in Quobyte driver https://review.openstack.org/554195 | 14:31 |
| *** wolverineav has joined #openstack-nova | 14:36 | |
| *** kaisers_ has quit IRC | 14:37 | |
| *** wolverineav has quit IRC | 14:41 | |
| *** Dinesh_Bhor has joined #openstack-nova | 14:42 | |
| *** mvkr has quit IRC | 14:42 | |
| *** takamatsu has joined #openstack-nova | 14:43 | |
| *** tbachman has joined #openstack-nova | 14:47 | |
| *** ratailor has joined #openstack-nova | 14:49 | |
| openstackgerrit | Merged openstack/nova master: Update compute API.get() stubs for test_*security_groups https://review.openstack.org/615344 | 14:49 |
| *** awaugama has joined #openstack-nova | 14:49 | |
| openstackgerrit | Merged openstack/nova master: Update compute API.get() stubs for test_disk_config https://review.openstack.org/615345 | 14:49 |
| openstackgerrit | Merged openstack/nova master: Update compute API.get() stubs in test_access_ips https://review.openstack.org/615346 | 14:49 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/rocky: Add secret=true to fixed_key configuration parameter https://review.openstack.org/623507 | 14:50 |
| *** tssurya has joined #openstack-nova | 14:50 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: Add secret=true to fixed_key configuration parameter https://review.openstack.org/623509 | 14:51 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/pike: Add secret=true to fixed_key configuration parameter https://review.openstack.org/623510 | 14:52 |
| *** Dinesh_Bhor has quit IRC | 14:56 | |
| *** ccamacho has quit IRC | 15:00 | |
| openstackgerrit | Dan Smith proposed openstack/nova master: Only warn about not having computes nodes once in rpcapi https://review.openstack.org/623282 | 15:03 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Make compute rpcapi version calculation check all cells https://review.openstack.org/623284 | 15:03 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Make service.get_minimum_version_all_cells() cache the results https://review.openstack.org/623283 | 15:03 |
| dansmith | mriedem: melwitt: ^ this should be good now I think. also fixed something in the base patch that was causing a non-deterministic run in the later patches | 15:04 |
| *** slaweq has quit IRC | 15:04 | |
| melwitt | cool, thanks | 15:05 |
| *** eharney has joined #openstack-nova | 15:05 | |
| mriedem | on an unrelated note, i propose that we disable snapshot tests in the cells v1 job http://logs.openstack.org/47/623247/2/check/nova-cells-v1/18338f0/job-output.txt.gz | 15:07 |
| mriedem | snapshot with cells v1 in tempest is historically racy https://bugs.launchpad.net/nova/+bug/1620761 | 15:07 |
| openstack | Launchpad bug 1620761 in OpenStack Compute (nova) "test_create_second_image_when_first_image_is_being_saved intermittently times out in teardown in cells v1 job" [Undecided,Invalid] | 15:07 |
| mriedem | given the deprecated nature of cells v1 and the state of the gate, the less we can fail on the better | 15:07 |
| dansmith | mriedem: fine with me | 15:08 |
| mriedem | alright then | 15:08 |
| dansmith | I also wouldn't be opposed to making a statement in stein that cellsv1 is deprecated fully, we're not going to test it anymore, and we'll be ripping the code out at whatever pace we feel is achievable | 15:09 |
| dansmith | we know that the removal of v1 and n-net aren't trivial, so the actual removal will take time, but I think we could go ahead and drive the stake through the heart | 15:09 |
| *** lei-zh has joined #openstack-nova | 15:10 | |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: API microversion 2.68: Handles Down Cells https://review.openstack.org/591657 | 15:10 |
| *** ttsiouts has joined #openstack-nova | 15:12 | |
| *** N3l1x has joined #openstack-nova | 15:12 | |
| mriedem | shrug. we neutered quite a bit of the n-net specific apis in rocky, there are a couple lingering that i think would be priority to remove first, and maybe the os-cells api | 15:12 |
| *** dpawlik has quit IRC | 15:12 | |
| mriedem | i also wouldn't want that to distract from the other stuff that's already not getting attention, so close to the 2nd milestone | 15:12 |
| dansmith | that's why I'm saying we don't have to do anything, just remove the test and land a reno | 15:13 |
| dansmith | but that's just MHO | 15:13 |
| dansmith | the test job Imean | 15:13 |
| mriedem | remove the entire nova-cells-v1 job you mean? | 15:13 |
| mriedem | yeah | 15:13 |
| *** Luzi has joined #openstack-nova | 15:14 | |
| mriedem | it is using neutron at this point, so it doesn't really help cern as a n-net 50% user to keep it around | 15:14 |
| mriedem | since cern is cells v2 now | 15:14 |
| dansmith | yeah | 15:14 |
| dansmith | melwitt is on the worst call evar right now, so maybe just pause until she's done and can opine | 15:14 |
| * bauzas nods :p | 15:15 | |
| melwitt | haha :) yeah.... | 15:15 |
| *** jarodwl has quit IRC | 15:16 | |
| *** jarodwl has joined #openstack-nova | 15:17 | |
| melwitt | mriedem: definitely ok with me to disable snapshot tests. and no strong opinion about removing the entire job. I think we'd be ok removing the job considering everyone's moved/moving off of it. I think nectar is the only other one? and mgagne_? | 15:17 |
| mriedem | sorrison said they are on pike i think, and v1 but looking at v2 | 15:18 |
| mriedem | mgagne_ i think is still on mitaka | 15:18 |
| *** ccamacho has joined #openstack-nova | 15:18 | |
| dansmith | removing the tests several releases ahead of both of those would help send a message that it's time | 15:18 |
| dansmith | as if cern being on it wasn't enough | 15:18 |
| dansmith | continuing to whittle away at what it tests is also fine, but looking at how stressed the gate is, one fewer devstack runs for each nova patch seems like a decent win to me | 15:20 |
| melwitt | yeah, on one hand I think we could disable the snapshot tests and if all is well after that, then we have some coverage for sorrison and mgagne_ sake. but if the job keeps being a thorn and failing then look again at removing the entire job | 15:21 |
| mriedem | fwiw it's also tempest and devstack that run it | 15:21 |
| mriedem | alternative is, move it to experimental | 15:22 |
| mriedem | so it's run on deman | 15:22 |
| mriedem | *demand | 15:22 |
| *** _hemna has joined #openstack-nova | 15:22 | |
| dansmith | mriedem: yeah, that would be good | 15:22 |
| melwitt | yeah, that's a nice alternative I didn't think of | 15:22 |
| mriedem | caught me with my pants down https://review.openstack.org/#/q/topic:rm-nova-cells-v1-job+(status:open+OR+status:merged) | 15:22 |
| mriedem | ok i'll change those | 15:22 |
| dansmith | nova, tempest and devstack is a lot of running a heavy job for something we're just going to disable if any tests fail :/ | 15:22 |
| *** trident has quit IRC | 15:25 | |
| *** trident has joined #openstack-nova | 15:27 | |
| *** artom has joined #openstack-nova | 15:32 | |
| *** Luzi has quit IRC | 15:32 | |
| *** artom has quit IRC | 15:35 | |
| *** ShilpaSD has quit IRC | 15:36 | |
| *** eharney has quit IRC | 15:37 | |
| *** artom has joined #openstack-nova | 15:40 | |
| *** pchavva has joined #openstack-nova | 15:41 | |
| *** dpawlik has joined #openstack-nova | 15:41 | |
| lei-zh | Hi jaypipes sean-k-mooney, would you take a look at https://review.openstack.org/#/c/601596/ and https://review.openstack.org/#/c/622893/ when you have spare time | 15:42 |
| sean-k-mooney | lei-zh: yes ill add them to my review queue | 15:44 |
| lei-zh | sean-k-mooney: thanks | 15:44 |
| lei-zh | we have split the original spec into two specs, one general and one specific to libvirt driver implementation, hope this will be helpful for people to understand | 15:45 |
| sean-k-mooney | lei-zh: do you know waht is the state of the code/ci for this is? | 15:45 |
| sean-k-mooney | lei-zh: ok i was wondering why it was split but that makes sense | 15:45 |
| *** dpawlik has quit IRC | 15:46 | |
| lei-zh | sean-k-mooney: we have worked out some poc code and will continue to improve them | 15:47 |
| sean-k-mooney | cool is as part of the poc is there a way to test this without real persitent memory . i belive qemu can fake it with a file backed correct | 15:48 |
| *** k_mouza has quit IRC | 15:48 | |
| sean-k-mooney | just wondering how feasibel testing this would be | 15:48 |
| *** k_mouza has joined #openstack-nova | 15:49 | |
| lei-zh | sean-k-mooney: ci is also in the list but we plan to start working on that when the code is more ready | 15:49 |
| lei-zh | sean-k-mooney: currently my colleagues are testing this on real hardware, I'm not sure about qemu simulation, I will ask them about this and sync with you | 15:51 |
| *** eharney has joined #openstack-nova | 15:52 | |
| sean-k-mooney | ya no worries. it would be nice to test this upstream at somepoint but its not a blocker | 15:52 |
| lei-zh | yeah, thanks for pointing out that | 15:53 |
| *** ohorecny2 has quit IRC | 15:57 | |
| *** ratailor has quit IRC | 15:57 | |
| *** janki has joined #openstack-nova | 16:00 | |
| *** lei-zh has quit IRC | 16:03 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Move nova-cells-v1 to experimental queue https://review.openstack.org/623538 | 16:03 |
| mriedem | https://review.openstack.org/#/q/topic:bug/1807407+(status:open+OR+status:merged) | 16:04 |
| *** tbachman has quit IRC | 16:04 | |
| *** ttsiouts has quit IRC | 16:06 | |
| openstackgerrit | Merged openstack/nova stable/rocky: Add description of custom resource classes https://review.openstack.org/619122 | 16:07 |
| openstackgerrit | Merged openstack/nova stable/rocky: Mention meta key suffix in tenant isolation with placement docs https://review.openstack.org/617535 | 16:08 |
| cdent | mriedem++ on cellsv1 job experimental | 16:09 |
| sean-k-mooney | jaypipes: stephenfin hopefully a quick one but can ye look at https://review.openstack.org/#/c/612534/ | 16:10 |
| mriedem | cdent: jaypipes: i'm going to need you guys to tell me if we can remove the SchedulerReportClient lock https://review.openstack.org/#/c/623246/2//COMMIT_MSG@11 | 16:12 |
| mriedem | since between you guys and efried_cya_jan it was added | 16:12 |
| *** ttsiouts has joined #openstack-nova | 16:13 | |
| cdent | looking | 16:13 |
| jaypipes | mriedem: I don't see any reason for it. | 16:16 |
| jaypipes | mriedem: there's no state being fetched on init. | 16:17 |
| jaypipes | mriedem: was more than likely just a copy/pasta from something else. | 16:17 |
| *** ttsiouts has quit IRC | 16:18 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Ensure that allocated PF matches the used PF https://review.openstack.org/623543 | 16:18 |
| mriedem | jaypipes: did you see the change in which it was added? | 16:18 |
| mriedem | https://review.openstack.org/#/c/493536/ | 16:18 |
| mriedem | i guess it was added for safety precautions | 16:19 |
| mriedem | if we're meh on that now, then sure we can remove it | 16:19 |
| cdent | mriedem, jaypipes: based on the commit message on https://review.openstack.org/493536 it looks gratuitous | 16:19 |
| mriedem | if i'm going to remove the lock, i might as well also make it a singleton in the api since we don't need 70 of these created when it's used in 3 places | 16:19 |
| jaypipes | mriedem: from the commit message: "I've made the _client method synchronized so that in the unlikely event that the resource tracker is trying to do its update job while some other thing is happening, we won't waste the client. This may not be necessary, but probably doesn't harm anything." | 16:20 |
| mriedem | i know, i read it | 16:20 |
| cdent | given that I never synchronize anything, I'm guess somewhere along the way it was suggested as a "maybe you should..." and I just did, without any real data | 16:20 |
| jaypipes | cdent: either that, or copy/pasta from somewhere in the resource tracker way back when... not sure, either way, I'm pretty sure it's unnecessary. | 16:21 |
| cdent | we'll find out soon enough, I reckon | 16:21 |
| melwitt | bauzas: reminder that review comments on the reshaper patch await your reply https://review.openstack.org/599208 | 16:22 |
| cdent | generally speaking, unless we can come up with a really good reason for a lock, we should not | 16:23 |
| bauzas | melwitt: ok, I was thinking those details were pretty nitty, but I'll then provide a new revision | 16:23 |
| mriedem | bauzas: if you thought that, you could have said so weeks ago... | 16:24 |
| melwitt | bauzas: ok. you don't necessarily have to push a new revision but could make a comment in response so we know your thoughts | 16:24 |
| mriedem | or talked with artom | 16:24 |
| *** gyee has joined #openstack-nova | 16:25 | |
| *** mvkr has joined #openstack-nova | 16:27 | |
| *** rodolof has joined #openstack-nova | 16:30 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Ignore MoxStubout deprecation warnings https://review.openstack.org/623545 | 16:32 |
| *** tbachman has joined #openstack-nova | 16:35 | |
| *** jaypipes is now known as leakypipes | 16:39 | |
| *** macza has joined #openstack-nova | 16:39 | |
| *** erlon has quit IRC | 16:42 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Note the aggregate allocation ratio restriction in scheduler docs https://review.openstack.org/623546 | 16:52 |
| *** erlon has joined #openstack-nova | 16:54 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Note the aggregate allocation ratio restriction in scheduler docs https://review.openstack.org/623547 | 16:56 |
| *** sean-k-mooney has quit IRC | 16:59 | |
| *** tssurya has quit IRC | 17:01 | |
| *** nicolasbock has joined #openstack-nova | 17:04 | |
| *** sean-k-mooney has joined #openstack-nova | 17:05 | |
| artom | mriedem, bauzas, are we seriously OK with the length of those methods? I won't be a pain just for the hell of it, and I understand that sometimes old code grows beyond what we want, but brand new code? | 17:09 |
| nicolasbock | Hi, I have a server on Newton that after a migration attempt shows `task_state = resize_finish`, `status = RESIZE`. I can't resize confirm or start the server. I don't see any obvious error messages in the nova-compute logs on either the old nor the new hypervisor. | 17:09 |
| artom | I dunno, I just have trouble with that | 17:09 |
| nicolasbock | The resized disks have the same md5sum on either hypervisor | 17:09 |
| artom | If everyone's overruling me there's not much I can do, but I wanted to at least make that point | 17:09 |
| mriedem | artom: i'm not disagreeing with your comments | 17:09 |
| nicolasbock | Basically, I am wondering what the problem could be? | 17:10 |
| artom | mriedem, ok, cool, thanks :) | 17:10 |
| mriedem | nicolasbock: looks like something probably failed during the resize (maybe a db change?) where the task_state was not set to None | 17:10 |
| mriedem | so check the compute and conductor logs for errors | 17:10 |
| nicolasbock | the logs on the destination ? | 17:11 |
| nicolasbock | And thanks mriedem | 17:11 |
| mriedem | yeah finish resize happens on the dest machine | 17:12 |
| *** k_mouza has quit IRC | 17:12 | |
| mriedem | db errors would be in conductor logs though | 17:13 |
| nicolasbock | The compute only says: `nova-compute.log:2018-12-07 17:08:04.883 26498 INFO nova.compute.manager [-] [instance: 585b884f-29be-4bb1-8f95-8f5e4c02127d] During sync_power_state the instance has a pending task (resize_finish). Skip.` | 17:13 |
| mriedem | this is where that task_state would be set https://github.com/openstack/nova/blob/newton-eol/nova/compute/manager.py#L3913 | 17:14 |
| nicolasbock | I am going through the conductor logs now | 17:14 |
| mriedem | and then task_state=None when done https://github.com/openstack/nova/blob/newton-eol/nova/compute/manager.py#L3946 | 17:14 |
| mriedem | if something blew up in there, this should reset the task_state to None https://github.com/openstack/nova/blob/newton-eol/nova/compute/manager.py#L3954 | 17:14 |
| mriedem | but if you're having db problems, it's a crapshoot | 17:14 |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Note the aggregate allocation ratio restriction in scheduler docs https://review.openstack.org/623552 | 17:15 |
| *** k_mouza has joined #openstack-nova | 17:17 | |
| *** mriedem is now known as mriedem_lunch | 17:19 | |
| nicolasbock | I don't see any DB errors either. Strange. | 17:20 |
| *** dims has quit IRC | 17:21 | |
| *** dtantsur is now known as dtantsur|afk | 17:22 | |
| *** wolverineav has joined #openstack-nova | 17:23 | |
| *** wolverineav has quit IRC | 17:27 | |
| *** sahid has quit IRC | 17:30 | |
| *** janki has quit IRC | 17:33 | |
| *** k_mouza_ has joined #openstack-nova | 17:35 | |
| *** k_mouza has quit IRC | 17:37 | |
| *** k_mouza_ has quit IRC | 17:39 | |
| aspiers | mriedem_lunch: thanks a lot for the quick reply - will get you those confirmations about volume-backed and config drive ASAP | 17:50 |
| *** k_mouza has joined #openstack-nova | 17:55 | |
| *** k_mouza has quit IRC | 17:59 | |
| *** pchavva has left #openstack-nova | 18:00 | |
| openstackgerrit | Merged openstack/nova stable/rocky: Add functional recreate test for bug 1799727 https://review.openstack.org/614563 | 18:10 |
| openstack | bug 1799727 in OpenStack Compute (nova) rocky "CPU_Allocation_Ratio from nova.conf doesn't update exisiting providers" [High,In progress] https://launchpad.net/bugs/1799727 - Assigned to Matt Riedemann (mriedem) | 18:10 |
| *** sridharg has quit IRC | 18:11 | |
| openstackgerrit | Merged openstack/nova stable/rocky: Provide allocation_ratio/reserved amounts from update_provider_tree() https://review.openstack.org/614564 | 18:11 |
| openstackgerrit | Merged openstack/nova stable/rocky: Add regression test for bug 1796737 https://review.openstack.org/614587 | 18:11 |
| openstack | bug 1796737 in OpenStack Compute (nova) rocky "resize: hypervisor local_gb_used still reports usage even with volume-backed instances after fix for bug 1469179" [Medium,In progress] https://launchpad.net/bugs/1796737 - Assigned to Matt Riedemann (mriedem) | 18:11 |
| openstackgerrit | Merged openstack/nova stable/rocky: Properly track local root disk usage during moves https://review.openstack.org/614588 | 18:11 |
| openstackgerrit | Merged openstack/nova stable/rocky: Time how long select_destinations() takes in conductor https://review.openstack.org/608575 | 18:11 |
| openstackgerrit | Merged openstack/nova stable/rocky: Refix disk size during live migration with disk over-commit https://review.openstack.org/602477 | 18:11 |
| openstackgerrit | Merged openstack/nova stable/rocky: Handle IndexError in _populate_neutron_binding_profile https://review.openstack.org/610163 | 18:11 |
| melwitt | stephenfin: been meaning to ask you, the nova-next job stopped testing vnc console with TLS several months ago, see http://logs.openstack.org/92/621692/1/check/nova-next/28869ae/logs/screen-n-novnc-cell1.txt.gz#_Dec_04_01_01_59_749371 I tried digging into it and couldn't find what's going wrong, how it can't find the cert file. wanted to give you a heads up in case you had any clues | 18:12 |
| *** Swami has joined #openstack-nova | 18:13 | |
| *** wolverineav has joined #openstack-nova | 18:13 | |
| sean-k-mooney | melwitt: i wonder if devstack change its default or something and its not enableing tls anymore | 18:21 |
| melwitt | sean-k-mooney: yeah, it's supposed to set everything up so long as the tls-proxy service is enabled https://github.com/openstack-dev/devstack/blob/78a564bb0304b6f930e1491e7e116a0a0f6d9ab6/stack.sh#L848 | 18:23 |
| melwitt | and in the example log I linked, it is enabled http://logs.openstack.org/92/621692/1/check/nova-next/28869ae/logs/devstacklog.txt.gz#_2018-12-04_00_45_46_475 | 18:24 |
| melwitt | I looked around devstack months ago looking for a change but didn't find anything. I could have missed it | 18:24 |
| *** udesale has joined #openstack-nova | 18:26 | |
| *** udesale has quit IRC | 18:27 | |
| *** k_mouza has joined #openstack-nova | 18:35 | |
| *** k_mouza has quit IRC | 18:39 | |
| *** ccamacho has quit IRC | 18:40 | |
| *** yan0s has quit IRC | 18:43 | |
| *** mriedem_lunch is now known as mriedem | 18:49 | |
| *** dims has joined #openstack-nova | 18:50 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP Reject migration requests when src compute is down https://review.openstack.org/623489 | 18:53 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: add InstanceList.get_all_uuids_by_hosts() method https://review.openstack.org/623557 | 18:57 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: single pass instance info fetch in host manager https://review.openstack.org/623558 | 18:57 |
| leakypipes | belmoreira: ^^ would be great to get some initial feedback from you on these performance patches to the scheduler. See commit message from https://review.openstack.org/#/c/623558/ for details. | 18:58 |
| leakypipes | melwitt: ^^ | 18:58 |
| melwitt | kewl | 19:00 |
| openstackgerrit | Jack Ding proposed openstack/nova master: Preserve UEFI NVRAM variable store https://review.openstack.org/621646 | 19:02 |
| *** erlon has quit IRC | 19:06 | |
| *** rodolof has quit IRC | 19:06 | |
| *** rodolof has joined #openstack-nova | 19:07 | |
| * leakypipes hoping mriedem gets the joke on the above patch's git branch name... | 19:08 | |
| leakypipes | topic name, that is... | 19:09 |
| mriedem | call me ishaeml | 19:09 |
| mriedem | *ishmael | 19:09 |
| mriedem | dansmith: you may enjoy https://review.openstack.org/#/c/615347 and it also resolves a functional race bug | 19:11 |
| mriedem | http://status.openstack.org/elastic-recheck/#1800472 | 19:11 |
| *** wolverineav has quit IRC | 19:19 | |
| *** cdent has quit IRC | 19:32 | |
| *** tbachman has quit IRC | 19:38 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Drop nova-multiattach job https://review.openstack.org/623568 | 19:39 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop nova-multiattach job https://review.openstack.org/606981 | 19:45 |
| openstackgerrit | Ken'ichi Ohmichi proposed openstack/nova master: Add descriptions about microversions https://review.openstack.org/619144 | 19:48 |
| *** igordc has joined #openstack-nova | 19:53 | |
| cfriesen | is there an equivalent to "nova service-list" in OSC, which would show the state of each compute node's service separately? | 19:57 |
| cfriesen | never mind, found "openstack compute service list" | 19:58 |
| *** k_mouza has joined #openstack-nova | 20:05 | |
| *** tbachman_ has joined #openstack-nova | 20:07 | |
| *** k_mouza has quit IRC | 20:09 | |
| dansmith | mriedem: see comment | 20:10 |
| dansmith | +3 on the code part | 20:10 |
| *** radez has joined #openstack-nova | 20:18 | |
| radez | dansmith: thx, we're working on multi-tenancy with ironic trying to test using networking-ansible to isolate networks | 20:18 |
| radez | this works fine, though in our CI we're setting up two tenants and trying to deploy an instance using ironic in each of the tenants | 20:19 |
| dansmith | radez: on master or rocky? | 20:19 |
| radez | rocky | 20:19 |
| radez | when we schedule the first node it deploys just fine | 20:19 |
| dansmith | okay just to be clear.. nova with compute_driver=ironic, right? | 20:19 |
| *** sean-k-mooney has quit IRC | 20:20 | |
| radez | yes i believe so , let me dousble check that, I've been relying on ooo to setup ironic for me | 20:20 |
| dansmith | just want to make it clear you're using nova to schedule an instance that will be running on a node in ironic.. the way you worded it above made me first think you were just talking about standalone ironic | 20:21 |
| radez | sry, you're correct we're using nova to schedule and ironic node | 20:22 |
| dansmith | okay, sorry, continue :) | 20:22 |
| *** mriedem has quit IRC | 20:25 | |
| radez | gathering my details here to try and depict this correctly | 20:25 |
| radez | so we deploy the first node and that goes swimmingly. comes active we can connect to it. | 20:25 |
| radez | this is in one tenant | 20:25 |
| * dansmith nods | 20:25 | |
| radez | in the second tenant we try and deploy another node and the nova logs apprear to indicate that when it goes to get available nodes to deploy to it returns 1 node but it returns the already allocated node instead of the available node | 20:26 |
| radez | lemme gather a few logs that depict this | 20:27 |
| dansmith | are you sure it's selecting the same *node* and not the same nova-compute? | 20:27 |
| dansmith | because if you have only one machine running nova-compute, that compute service will be responsible for all the ironic nodes | 20:27 |
| dansmith | I assume that's not the problem because you would likely not even notice and not be here with an issue, but just ... in case it's relevant | 20:28 |
| *** sean-k-mooney has joined #openstack-nova | 20:28 | |
| dansmith | also, while you're collecting logs, this shouldn't have anything to do with the multi-tenancy part, as the scheduling, picking a node, reserving a node, etc doesn't really factor tenants into the calculus | 20:28 |
| *** mriedem has joined #openstack-nova | 20:28 | |
| radez | I *think* that's not the case because the ironic node uuids and the nova instance uuids match up in the logs | 20:29 |
| radez | fair enough, just laying out the context incase that was helpful | 20:29 |
| dansmith | okay, they should only really line up in the compute log I think, probably not in the scheduling log, fwiw | 20:29 |
| dansmith | ack, yep, just letting you know | 20:29 |
| dansmith | radez: my tacos are just about warmed, so keep collecting logs and dumping info here and I'll reply when my fingers are clean :) | 20:31 |
| dansmith | also, mriedem is here, he's real smart, and he was just telling me he was bored and looking for a good goose to chase :P | 20:31 |
| radez | lol, cool I just got a call I need to take that will distract me for about 30 mins. I'll finish collecting my logs and ping one of you in a bit, sry about that.... I appreciate your help! | 20:32 |
| mriedem | what's good for the goose is good for the gander | 20:33 |
| dansmith | radez: okay, also keep in mind it's friday afternoon so probably not many hours left in the day (and I'm sure you're ahead of the rest of us in EST) :D | 20:34 |
| *** eharney has quit IRC | 20:39 | |
| *** N3l1x has quit IRC | 20:40 | |
| dansmith | taco consumption complete | 20:42 |
| mriedem | dansmith: replied on the reno thing - i'm ok with dropping that if it would just cause more concern than it's worth | 20:43 |
| mriedem | that compat was really kind of out the window since pike i think | 20:43 |
| mriedem | when the api was made cells aware | 20:43 |
| dansmith | mriedem: yeah, so I'd just drop it if you're cool with it | 20:43 |
| dansmith | hah | 20:44 |
| mriedem | i'm very cool | 20:44 |
| dansmith | zuul says 135h remaining on my patch I submitted at like 7am this morning | 20:44 |
| mriedem | https://review.openstack.org/#/c/602804/ queued for at least 36 hours, hit a job timeout | 20:45 |
| dansmith | good lord | 20:46 |
| dansmith | maybe we should start enacting a french work-week to avoid overloading the gate | 20:46 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop pre-cellsv2 compat in compute API.get() https://review.openstack.org/615347 | 20:46 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove "API Service Version" upgrade check https://review.openstack.org/615348 | 20:46 |
| *** ralonsoh has quit IRC | 20:47 | |
| * mriedem thinks up a joke about expensive gas | 20:47 | |
| *** amodi has quit IRC | 20:54 | |
| *** N3l1x has joined #openstack-nova | 20:58 | |
| *** macza has quit IRC | 20:58 | |
| radez | ok, I'm back now, sry about that | 21:01 |
| radez | dansmith: mriedem: http://paste.openstack.org/show/736843/ | 21:02 |
| *** awaugama has quit IRC | 21:02 | |
| radez | this shows the uuids of the bm nodes and the nova instances that seems to be behaving like this | 21:02 |
| radez | and the nova logs where nova seems to assign the already assigned bm node to the new nova instance | 21:03 |
| radez | let me know if there's more logs I can get to help | 21:03 |
| dansmith | radez: so these nodes should all exist in placement with a single inventory item for the resource/node class, | 21:04 |
| dansmith | which means they should never get past the scheduler with two instances on the same node | 21:04 |
| radez | translation was helpful thx :) | 21:05 |
| radez | [instance: a840afbc-1314-44d6-83b3-c20b790b9322] Claim successful on node 3c3eb5ee-a358-421a-b2e9-7fd9a017ba13 | 21:05 |
| radez | hm, well now I think I've given you the logs that show that the associated ironic node and nova instance are properly associated together | 21:06 |
| *** brault has quit IRC | 21:06 | |
| *** awaugama has joined #openstack-nova | 21:07 | |
| mriedem | do the flavors have cpu/ram/disk set to anything !0? | 21:07 |
| radez | oh wait, sry I'm getting my self confused... right so does that log line I just pasted here show that nova is trying to | 21:07 |
| dansmith | radez: what log is this? just nova compute? | 21:07 |
| radez | assign ironic node 3c3eb5ee to nova instance a840afbc? | 21:08 |
| radez | yea nova-compute log | 21:08 |
| dansmith | radez: so probably want to look at the scheduler log | 21:08 |
| dansmith | radez: scheduler is what decides which node it is going to tell compute to use for each instance | 21:09 |
| *** rodolof has quit IRC | 21:09 | |
| radez | k lemme find a840afbc in the schedule logs | 21:10 |
| radez | http://paste.openstack.org/show/736844/ | 21:13 |
| openstackgerrit | Merged openstack/os-vif master: add isolate_vif config option https://review.openstack.org/612534 | 21:13 |
| radez | that also shows a840afbc trying to select 3c3eb5ee | 21:13 |
| radez | hypervisor list incase that's helpful? http://paste.openstack.org/show/736845/ | 21:14 |
| melwitt | radez: what version of rocky is this? we had a bug around ironic inventory update in the middle of rocky https://review.openstack.org/593678 but as long as you have 18.0.0.0rc2 or later, you should have the fix for that | 21:14 |
| melwitt | er, at the end of rocky | 21:14 |
| dansmith | radez: can you show more of the log so we can see both? | 21:14 |
| radez | yea, that was just a quick grep, lemme grab more of the scheduler log and check my version | 21:15 |
| dansmith | melwitt: that would only affect nodes in cleaning right? | 21:16 |
| melwitt | dansmith: I don't know, tbh | 21:18 |
| melwitt | I remember there were some ironic driver bugs we fixed near the end of rocky and was curious if those can be ruled out based on what version of rocky they're running | 21:19 |
| dansmith | the only way this should be happening, AFAIK, is if we're not exposing one inventory per node, or somehow not claiming in placement | 21:19 |
| melwitt | yeah, this bug was related to the claiming of a node via setting reserved == total | 21:20 |
| melwitt | and one spot where the newer placement api microversion needed to be set was missed | 21:20 |
| melwitt | so I was thinking that might possible cause a node to not be claimed when it was supposed to. but as for if it only comes up during cleaning, that I didn't know | 21:21 |
| melwitt | *possibly | 21:21 |
| mriedem | radez: what are the values for cpu/ram/disk for the flavors being used for these baremetal nodes? | 21:21 |
| melwitt | I had thought the reserved == total thing was claiming in general and not only for cleaning, but could easily be wrong as I don't know _that_ much about the ironic driver | 21:21 |
| radez | openstack-nova-scheduler-18.0.3 | 21:22 |
| radez | ok, got hte version, sry fighting with docker :/ | 21:22 |
| *** igordc has quit IRC | 21:22 | |
| melwitt | thanks radez. so yeah, the thing I wondered can be ruled out | 21:22 |
| dansmith | melwitt: reserved=total is only for making the node look unschedulable while being cleaned right? | 21:22 |
| *** igordc has joined #openstack-nova | 21:22 | |
| dansmith | melwitt: reserved=total does't make any sense when the node is actually allocated | 21:22 |
| mriedem | i'm wondering if you're hitting https://bugs.launchpad.net/nova/+bug/1796920 | 21:23 |
| openstack | Launchpad bug 1796920 in OpenStack Compute (nova) queens "Baremetal nodes should not be exposing non-custom-resource-class (vcpu, ram, disk)" [High,In progress] - Assigned to Matt Riedemann (mriedem) | 21:23 |
| radez | | properties | {u'memory_mb': u'4096', u'cpu_arch': u'x86_64', u'local_gb': u'40', u'cpus': u'4', u'capabilities': u'boot_option:local' | 21:23 |
| radez | both look liek that | 21:23 |
| mriedem | yeah so check that bug | 21:23 |
| mriedem | or https://review.openstack.org/#/c/609043/ | 21:23 |
| radez | getting the logs... | 21:23 |
| dansmith | mriedem: yeah, that's the kind of thing I was thinking of | 21:23 |
| melwitt | dansmith: I didn't know that. but what you say makes sense, you're probably right | 21:24 |
| dansmith | mriedem: we consume one CUSTOM_FOO, and then schedule the next one based on there looking like there's ram or something | 21:24 |
| mriedem | i'm very much half assing my involvement here, | 21:24 |
| mriedem | but that's something i'd check out | 21:24 |
| *** macza has joined #openstack-nova | 21:24 | |
| dansmith | I have to leave soon | 21:25 |
| radez | here's more of the scheduler log http://paste.openstack.org/show/736846/ | 21:28 |
| *** eharney has joined #openstack-nova | 21:31 | |
| dansmith | radez: okay but that's still just one instance, right? | 21:31 |
| dansmith | however, it looks like whatever host it chose got successfully claimed in placement, | 21:31 |
| dansmith | which can't happen if each host is exposing only one inventory item | 21:31 |
| dansmith | radez: can you correlate the "Attempting to claim.." line with the relevant context in the placement log? | 21:32 |
| radez | placement log is the scheduler log? | 21:32 |
| mriedem | no | 21:32 |
| dansmith | nope, a different service | 21:33 |
| mriedem | placement-api log | 21:33 |
| radez | oh, sec lemme find that log | 21:33 |
| dansmith | that should (I think) show you what is being claimed for the instance | 21:33 |
| dansmith | which really should be only one resource, something like CUSTOM_IRONIC_FOO=1 | 21:33 |
| radez | would that coorelate on the insance uuid? | 21:34 |
| dansmith | I'm not sure what the string looks like, so not sure if it dumps the consumer (instance) uuid or not | 21:35 |
| mriedem | if the flavor has vcpu/ram/disk, and the compute node is reporting vcpu/ram/disk inventory - which it will be still before stein, the claim will be on vcpu/ram/disk as well | 21:35 |
| dansmith | so I would go by date | 21:35 |
| dansmith | mriedem: wait, what? | 21:35 |
| dansmith | I thought it only reported the old resources in some compat situation? | 21:36 |
| radez | kk | 21:36 |
| mriedem | dansmith: no, | 21:37 |
| mriedem | we didn't neuter the ironic driver until stein, right around the ptg | 21:37 |
| dansmith | I think I recoiled in horror the last time we talked about this | 21:37 |
| mriedem | that's why we needed https://review.openstack.org/#/c/609043/ | 21:37 |
| dansmith | so maybe he just doesn't have the resource class in his flavor? | 21:37 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop pre-cellsv2 compat in compute API.get() https://review.openstack.org/615347 | 21:39 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove "API Service Version" upgrade check https://review.openstack.org/615348 | 21:39 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop old service version check compat from _delete_while_booting https://review.openstack.org/623589 | 21:39 |
| *** amodi has joined #openstack-nova | 21:39 | |
| mriedem | | properties | {u'memory_mb': u'4096', u'cpu_arch': u'x86_64', u'local_gb': u'40', u'cpus': u'4', u'capabilities': u'boot_option:local' | 21:39 |
| mriedem | those are extra specs? | 21:39 |
| dansmith | no, | 21:40 |
| mriedem | then correct the baremetal flavor doesn't have the custom resource class | 21:40 |
| dansmith | node detaisl Ithink | 21:40 |
| mriedem | radez: have you gone through this? https://docs.openstack.org/ironic/rocky/install/configure-nova-flavors.html | 21:40 |
| mriedem | especially: openstack flavor set --property resources:CUSTOM_BAREMETAL_SMALL=1 my-baremetal-flavor | 21:41 |
| dansmith | and/or.. who created your flavor? | 21:41 |
| radez | http://paste.openstack.org/show/736847/ | 21:41 |
| dansmith | o.O | 21:41 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop old service version check compat from _delete_while_booting https://review.openstack.org/623589 | 21:42 |
| radez | jlibosvar created the flavor, it looks like that custom baremetal small is not on it | 21:42 |
| mriedem | CUSTOM_BAREMETAL_SMALL is an example | 21:43 |
| dansmith | radez: it needs to be equal to whatever is set on the node | 21:43 |
| dansmith | right | 21:43 |
| radez | http://paste.openstack.org/show/736848/ | 21:43 |
| mriedem | the node has a resource_class field | 21:43 |
| radez | yea lemme see what's on the node | 21:43 |
| mriedem | yeah that flavor isn't going to cut it | 21:43 |
| dansmith | yeah | 21:43 |
| mriedem | https://docs.openstack.org/ironic/rocky/install/configure-nova-flavors.html ftw | 21:44 |
| radez | I don't think there is a resources property on the bm node either | 21:44 |
| mriedem | resource_class | 21:44 |
| mriedem | https://developer.openstack.org/api-ref/baremetal/?expanded=show-node-details-detail#show-node-details | 21:45 |
| dansmith | radez: that "how to configure flavors" doc above is probably your ticket out of here, right? | 21:45 |
| radez | | resource_class | None | 21:45 |
| mriedem | heh "This will be used by the openstack Placement Engine in a future release." | 21:45 |
| mriedem | jroll: maybe we should update the resource_class api-ref parameter description in ironic now... | 21:45 |
| radez | yea maybe he's just missed some of the setup steps. Lemme regroup and try and go back through the steps that have been taken in this env and I'll circle back back next week if we're still haveing trouble | 21:46 |
| dansmith | cool | 21:46 |
| radez | ok, thanks for your time! | 21:47 |
| * mriedem awaits his $2 in the mail | 21:49 | |
| dansmith | I need to remember to remember that we only closed that silly loop in stein | 21:52 |
| dansmith | all that discussion at the first denver ptg ... | 21:52 |
| * dansmith shakes his head | 21:52 | |
| *** wolverineav has joined #openstack-nova | 21:57 | |
| *** wolverineav has quit IRC | 21:57 | |
| *** wolverineav has joined #openstack-nova | 21:57 | |
| *** wolverineav has quit IRC | 22:04 | |
| mriedem | melwitt: thanks for hitting the stable branch reviews | 22:05 |
| melwitt | np | 22:06 |
| *** wolverineav has joined #openstack-nova | 22:08 | |
| *** slaweq has joined #openstack-nova | 22:11 | |
| *** KeithMnemonic has quit IRC | 22:16 | |
| *** rodolof has joined #openstack-nova | 22:16 | |
| *** trident has quit IRC | 22:22 | |
| *** trident has joined #openstack-nova | 22:22 | |
| *** gouthamr has quit IRC | 22:23 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove allocations before setting vm_status to SHELVED_OFFLOADED https://review.openstack.org/623596 | 22:29 |
| mriedem | gibi: efried_cya_jan: finally got that alternative fix up for that shelve gate race bug ^ | 22:29 |
| cfriesen | in compute/api.py in the @check_instance_state() decorator, if we don't specify a task_state does that mean that we don't care what the state is? | 22:35 |
| mriedem | no | 22:36 |
| mriedem | not specifying a task_state means task_state must be None | 22:36 |
| cfriesen | what if we have @check_instance_state(task_state=None) ? | 22:37 |
| openstackgerrit | Ben Nemec proposed openstack/nova master: Migrate upgrade checks to oslo.upgradecheck https://review.openstack.org/603499 | 22:38 |
| mriedem | cfriesen: if only the source code were freely available... | 22:38 |
| cfriesen | mriedem: yeah, well I'm staring at it and I'm trying to wrap my head around the logic since there are no useful comments. :) | 22:40 |
| cfriesen | I guess if we explicitly set it to None then we don't care, otherwise instance.task_state has to be None to pass. | 22:42 |
| melwitt | yeah, was about to say the same thing | 22:42 |
| melwitt | if task_state=None it isn't checking state | 22:43 |
| *** mriedem is now known as mriedem_afk | 22:46 | |
| *** rodolof has quit IRC | 22:52 | |
| *** slaweq has quit IRC | 22:52 | |
| *** N3l1x has quit IRC | 23:06 | |
| *** gouthamr has joined #openstack-nova | 23:06 | |
| *** wolverineav has quit IRC | 23:08 | |
| *** wolverin_ has joined #openstack-nova | 23:08 | |
| *** slaweq has joined #openstack-nova | 23:09 | |
| *** burt has quit IRC | 23:10 | |
| *** slaweq has quit IRC | 23:14 | |
| *** slaweq has joined #openstack-nova | 23:19 | |
| *** slaweq has quit IRC | 23:24 | |
| *** tbachman_ has quit IRC | 23:27 | |
| *** tbachman has joined #openstack-nova | 23:33 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!