| *** mriedem has quit IRC | 00:09 | |
| *** takashin has joined #openstack-nova | 00:16 | |
| *** brinzhang has joined #openstack-nova | 00:45 | |
| *** takashin has quit IRC | 01:13 | |
| *** whoami-rajat has joined #openstack-nova | 01:16 | |
| *** dannins has joined #openstack-nova | 01:19 | |
| *** tiendc has joined #openstack-nova | 01:31 | |
| *** logan- has quit IRC | 01:33 | |
| *** logan- has joined #openstack-nova | 01:37 | |
| *** Swami has quit IRC | 01:44 | |
| *** cdent has quit IRC | 01:44 | |
| *** bbowen_ has quit IRC | 02:04 | |
| *** hongbin has joined #openstack-nova | 02:19 | |
| *** JamesBenson has joined #openstack-nova | 02:34 | |
| *** cfriesen has quit IRC | 02:39 | |
| *** udesale has joined #openstack-nova | 02:42 | |
| *** rcernin has quit IRC | 02:43 | |
| *** samueldmq has quit IRC | 02:46 | |
| *** boxiang has quit IRC | 02:46 | |
| *** boxiang has joined #openstack-nova | 02:50 | |
| *** lbragstad has quit IRC | 02:54 | |
| *** rcernin has joined #openstack-nova | 03:29 | |
| *** cdent has joined #openstack-nova | 03:44 | |
| *** brinzhang has quit IRC | 03:44 | |
| *** brinzhang has joined #openstack-nova | 03:45 | |
| openstackgerrit | Chason Chan proposed openstack/python-novaclient master: Tiny fix of documentation https://review.opendev.org/657522 | 03:57 |
|---|---|---|
| *** cdent has quit IRC | 03:58 | |
| *** zhongjun2 has quit IRC | 04:05 | |
| *** tkajinam has quit IRC | 04:07 | |
| *** tkajinam has joined #openstack-nova | 04:08 | |
| *** tkajinam has quit IRC | 04:08 | |
| *** tkajinam has joined #openstack-nova | 04:08 | |
| *** KH-Jared has quit IRC | 04:14 | |
| *** ivve has quit IRC | 04:17 | |
| *** ricolin has joined #openstack-nova | 04:26 | |
| *** hongbin has quit IRC | 04:33 | |
| *** janki has joined #openstack-nova | 04:37 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix assert methods in unit tests https://review.opendev.org/657525 | 04:48 |
| *** KH-Jared has joined #openstack-nova | 05:04 | |
| *** JamesBenson has quit IRC | 05:11 | |
| *** ivve has joined #openstack-nova | 05:20 | |
| *** ricolin has quit IRC | 05:24 | |
| *** JamesBenson has joined #openstack-nova | 05:28 | |
| *** JamesBenson has quit IRC | 05:35 | |
| *** HW-Peter has quit IRC | 05:41 | |
| *** maciejjozefczyk_ has joined #openstack-nova | 05:57 | |
| *** KH-Jared has quit IRC | 06:06 | |
| *** lpetrut has joined #openstack-nova | 06:07 | |
| *** pcaruana has joined #openstack-nova | 06:21 | |
| *** slaweq has joined #openstack-nova | 06:36 | |
| *** rpittau|afk is now known as rpittau | 06:47 | |
| *** hoonetorg has quit IRC | 07:04 | |
| *** hoonetorg has joined #openstack-nova | 07:08 | |
| *** KH-Jared has joined #openstack-nova | 07:11 | |
| *** threestrands has joined #openstack-nova | 07:11 | |
| *** KH-Jared has quit IRC | 07:15 | |
| *** tosky has joined #openstack-nova | 07:16 | |
| *** ccamacho has joined #openstack-nova | 07:23 | |
| *** ivve has quit IRC | 07:25 | |
| *** tesseract has joined #openstack-nova | 07:26 | |
| *** rcernin has quit IRC | 07:27 | |
| *** ivve has joined #openstack-nova | 07:33 | |
| *** ivve has quit IRC | 07:44 | |
| *** ivve has joined #openstack-nova | 07:59 | |
| *** trident has quit IRC | 08:01 | |
| *** trident has joined #openstack-nova | 08:02 | |
| *** ccamacho has quit IRC | 08:03 | |
| *** threestrands has quit IRC | 08:18 | |
| *** tkajinam has quit IRC | 08:35 | |
| *** maciejjozefczyk_ has quit IRC | 08:37 | |
| *** maciejjozefczyk_ has joined #openstack-nova | 08:37 | |
| *** priteau has joined #openstack-nova | 08:44 | |
| *** ralonsoh has joined #openstack-nova | 08:48 | |
| openstackgerrit | Merged openstack/nova master: Use migration_status during volume migrating and retyping https://review.opendev.org/637224 | 09:07 |
| *** KH-Jared has joined #openstack-nova | 09:24 | |
| *** alex_xu has quit IRC | 09:28 | |
| *** jaosorior has joined #openstack-nova | 09:48 | |
| *** KH-Jared has quit IRC | 09:53 | |
| *** zbr has joined #openstack-nova | 10:04 | |
| *** Woutifier has joined #openstack-nova | 10:07 | |
| *** Woutifier has left #openstack-nova | 10:07 | |
| *** KH-Jared has joined #openstack-nova | 10:07 | |
| *** pcaruana has quit IRC | 10:19 | |
| *** yan0s has joined #openstack-nova | 10:24 | |
| *** bbowen_ has joined #openstack-nova | 10:41 | |
| *** udesale has quit IRC | 10:42 | |
| *** tbachman has quit IRC | 10:42 | |
| *** udesale has joined #openstack-nova | 10:42 | |
| *** bbowen_ has quit IRC | 10:50 | |
| *** pcaruana has joined #openstack-nova | 10:55 | |
| *** panda is now known as panda|lunch | 10:56 | |
| *** mdbooth has joined #openstack-nova | 10:57 | |
| *** priteau has quit IRC | 11:11 | |
| *** rpittau has quit IRC | 11:14 | |
| *** rpittau has joined #openstack-nova | 11:17 | |
| openstackgerrit | Merged openstack/nova master: Log when port resource is leaked during port delete https://review.opendev.org/657079 | 11:27 |
| openstackgerrit | Merged openstack/nova master: Fix assert methods in unit tests https://review.opendev.org/657525 | 11:27 |
| openstackgerrit | John Garbutt proposed openstack/nova-specs master: Policy Default Refresh spec https://review.opendev.org/547850 | 11:30 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/stein: Use migration_status during volume migrating and retyping https://review.opendev.org/657575 | 11:39 |
| openstackgerrit | John Garbutt proposed openstack/nova-specs master: Add Unified Limits Spec https://review.opendev.org/602201 | 11:39 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/rocky: Use migration_status during volume migrating and retyping https://review.opendev.org/657577 | 11:43 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: Use migration_status during volume migrating and retyping https://review.opendev.org/657579 | 11:43 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova stable/stein: Log when port resource is leaked during port delete https://review.opendev.org/657581 | 11:50 |
| *** lyarwood has quit IRC | 11:51 | |
| *** bbowen has joined #openstack-nova | 11:52 | |
| *** bbowen_ has joined #openstack-nova | 11:53 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova-specs master: Resource provider - request group mapping in allocation candidate https://review.opendev.org/597601 | 11:54 |
| *** ygk_12345 has joined #openstack-nova | 11:55 | |
| *** bbowen has quit IRC | 11:56 | |
| *** mriedem has joined #openstack-nova | 12:05 | |
| ygk_12345 | hi all | 12:05 |
| ygk_12345 | can someone help me how to get a vm's flavor from the nova database | 12:05 |
| mriedem | instance_extras.flavor column | 12:06 |
| mriedem | *instance_extra | 12:06 |
| mriedem | https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/models.py#L386 | 12:06 |
| gibi | ygk_12345: mysql nova_cell1 -e 'select flavor from instance_extra;' | 12:08 |
| gibi | ahh mriedem was faster | 12:08 |
| ygk_12345 | gibi: ok thanks got it :) | 12:09 |
| ygk_12345 | gibi: but is there a way to get that name of the flavor only in the output ? | 12:09 |
| mriedem | ygk_12345: original_name is provided in the output i think | 12:09 |
| mriedem | https://developer.openstack.org/api-ref/compute/?expanded=show-server-details-detail#show-server-details | 12:10 |
| mriedem | flavor.original_name | 12:10 |
| mriedem | that's available since microversion 2.47 | 12:10 |
| ygk_12345 | mriedem: is flavor a table ? | 12:10 |
| gibi | and there is "name" key in the flavor json in the database | 12:11 |
| mriedem | flavors is a table in the nova_api db yes | 12:11 |
| mriedem | but the flavor used to create the instance is not necessarily still in that table, | 12:11 |
| mriedem | or with the same values, i.e. extra specs | 12:11 |
| mriedem | the copy of the flavor used to create or last resize the instance is in the instance_extra table | 12:11 |
| ygk_12345 | gibi: yes I see but how to get it only in the query output. I am getting all the cluttered output | 12:12 |
| mriedem | ygk_12345: it's stored as a json blob in the instance_extra table | 12:12 |
| gibi | ygk_12345: ^^ | 12:12 |
| mriedem | so you'd have to deserialize it | 12:12 |
| mriedem | gibi: were you planning on backporting https://review.opendev.org/#/q/topic:bug/1819923+(status:open+OR+status:merged) ? i'm wondering about ordering for https://review.opendev.org/#/c/651945/ | 12:12 |
| *** tbachman has joined #openstack-nova | 12:12 | |
| ygk_12345 | gibi: mriedem now I see in the nova_api db. I will extract it somehow. Thanks for your time guys | 12:13 |
| mriedem | just remember what i said about the flavor used to create the server may not be in the flavors table anymore | 12:14 |
| gibi | mriedem: yes, I'm planning to backport heal port allocation to stein | 12:15 |
| gibi | mriedem: if https://review.opendev.org/#/c/651945/ merges then I will handle the conflicts | 12:16 |
| gibi | or we might consider https://review.opendev.org/#/c/651945/ also as a backport | 12:18 |
| *** nicolasbock has joined #openstack-nova | 12:20 | |
| *** mchlumsky has joined #openstack-nova | 12:28 | |
| *** alex_xu has joined #openstack-nova | 12:35 | |
| *** brinzhang has quit IRC | 12:41 | |
| *** _hemna has quit IRC | 12:47 | |
| *** hemna has joined #openstack-nova | 12:48 | |
| *** tiendc has quit IRC | 12:53 | |
| *** mmethot has joined #openstack-nova | 13:04 | |
| *** tssurya has joined #openstack-nova | 13:05 | |
| *** jdillaman has joined #openstack-nova | 13:12 | |
| *** lbragstad has joined #openstack-nova | 13:20 | |
| mriedem | tssurya: i'm reviewing your locked_reason patch so hold up if you're making changes | 13:22 |
| tssurya | mriedem: yea I am | 13:22 |
| tssurya | ok will hold up :) | 13:23 |
| * bauzas is back with a bit of jetlag | 13:23 | |
| tssurya | bauzas: hi-fi | 13:23 |
| bauzas | \o | 13:23 |
| * gibi survives the jetlag with coffee | 13:24 | |
| tssurya | gibi: thanks for the review on the locked_reason patch, sorry for the big patch size, couldn't help it | 13:24 |
| gibi | tssurya: no worries. at the end it was fairly easy to review | 13:25 |
| gibi | tssurya: thanks for working on that feature | 13:25 |
| *** panda|lunch is now known as panda | 13:25 | |
| tssurya | mriedem: while you are reviewing could you confirm if you are okay with the comment here? : https://review.opendev.org/#/c/648662/8/api-ref/source/servers-actions.inc@640 | 13:25 |
| *** hemna has quit IRC | 13:25 | |
| tssurya | gibi: :) | 13:25 |
| *** hemna has joined #openstack-nova | 13:26 | |
| *** mgoddard has quit IRC | 13:39 | |
| *** mgoddard has joined #openstack-nova | 13:40 | |
| *** lpetrut has quit IRC | 13:40 | |
| *** artom has quit IRC | 13:45 | |
| *** jaosorior has quit IRC | 13:49 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova stable/stein: Reproduce bug #1819460 in functional test https://review.opendev.org/657600 | 13:49 |
| openstack | bug 1819460 in OpenStack Compute (nova) stein "instance stuck in BUILD state due to unhandled exceptions in conductor" [Medium,Confirmed] https://launchpad.net/bugs/1819460 | 13:49 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova stable/stein: Fix exception type in test_boot_reschedule_fill_provider_mapping_raises https://review.opendev.org/657601 | 13:49 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova stable/stein: Handle placement error during re-schedule https://review.opendev.org/657602 | 13:49 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova stable/stein: Only call _fill_provider_mapping if claim succeeds https://review.opendev.org/657603 | 13:49 |
| mriedem | huh, auto_disk_config is a valid filter parameter for listing servers but not handled in the DB API at all from what i can tell | 13:55 |
| mriedem | oh well, it's xen only anyway | 13:55 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add a workaround config toggle to refuse ceph image upload https://review.opendev.org/657078 | 14:02 |
| *** boxiang has quit IRC | 14:02 | |
| *** boxiang has joined #openstack-nova | 14:03 | |
| *** tbachman has quit IRC | 14:03 | |
| *** jdillaman has quit IRC | 14:03 | |
| *** ccamacho has joined #openstack-nova | 14:06 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif stable/stein: Prevent "qbr" Linux Bridge from replying to ARP messages https://review.opendev.org/655678 | 14:08 |
| *** artom has joined #openstack-nova | 14:12 | |
| *** jistr is now known as jistr|call | 14:14 | |
| *** janki has quit IRC | 14:14 | |
| mriedem | tssurya: done | 14:17 |
| mriedem | might be a record | 14:17 |
| tssurya | mriedem: thanks! record for a hard rain of comments? | 14:18 |
| tssurya | :D | 14:18 |
| *** artom has quit IRC | 14:18 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif stable/rocky: Prevent "qbr" Linux Bridge from replying to ARP messages https://review.opendev.org/655692 | 14:18 |
| *** pcaruana has quit IRC | 14:19 | |
| mriedem | tssurya: yeah | 14:19 |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif stable/queens: Prevent "qbr" Linux Bridge from replying to ARP messages https://review.opendev.org/655694 | 14:19 |
| tssurya | mriedem: heh but thanks a lot for the time, I'll work on them | 14:19 |
| tssurya | mriedem: btw am I causing these tests to fail ? http://logs.openstack.org/62/648662/8/check/openstack-tox-py36/7604398/testr_results.html.gz | 14:20 |
| tssurya | I couldn't find the cause | 14:20 |
| mriedem | no | 14:21 |
| mriedem | http://status.openstack.org/elastic-recheck/#1823251 | 14:21 |
| tssurya | mriedem: oh cool thanks | 14:21 |
| *** jistr|call is now known as jistr | 14:23 | |
| *** tbachman has joined #openstack-nova | 14:25 | |
| *** ygk_12345 has quit IRC | 14:27 | |
| *** JamesBenson has joined #openstack-nova | 14:27 | |
| mriedem | dansmith: did you see my question/concern on the image type request filter patch regarding bfv instances? | 14:29 |
| mriedem | i didn't -1 yet | 14:29 |
| dansmith | no | 14:29 |
| *** mrch_ has joined #openstack-nova | 14:29 | |
| mrch_ | had anyone problems upping novacompute after upgrade queens>rocky (centos)? | 14:30 |
| *** mlavalle has joined #openstack-nova | 14:31 | |
| *** JamesBenson has quit IRC | 14:31 | |
| *** JamesBenson has joined #openstack-nova | 14:34 | |
| *** mdbooth has quit IRC | 14:35 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif stable/rocky: Prevent "qbr" Linux Bridge from replying to ARP messages https://review.opendev.org/655692 | 14:37 |
| *** pcaruana has joined #openstack-nova | 14:38 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif stable/queens: Prevent "qbr" Linux Bridge from replying to ARP messages https://review.opendev.org/655694 | 14:38 |
| *** efried is now known as efried_pto | 14:39 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: api-ref: fix mention of all_tenants filter for non-admins https://review.opendev.org/657620 | 14:49 |
| mriedem | sorrison: you know how you added the "os_compute_api:servers:allow_all_filters" policy rule? if using all_tenants as the filter parameter, that could still be filtered out for non-admins because all_tenants has it's own specific rules (os_compute_api:servers:index:get_all_tenants and os_compute_api:servers:detail:get_all_tenants) | 14:50 |
| mriedem | it seems that os_compute_api:servers:allow_all_filters should also apply to the all_tenants filter, yeah? | 14:51 |
| *** Sundar has joined #openstack-nova | 14:53 | |
| openstackgerrit | Dan Smith proposed openstack/nova master: Expose Hyper-V supported image types https://review.opendev.org/655137 | 15:01 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Make libvirt expose supported image types https://review.opendev.org/653454 | 15:01 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add ironic driver image type capabilities https://review.opendev.org/655729 | 15:01 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add vmware driver image type capabilities https://review.opendev.org/655730 | 15:01 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add xenapi driver image type capabilities https://review.opendev.org/655731 | 15:01 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add zvm driver image type capabilities https://review.opendev.org/655732 | 15:01 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add image type request filter https://review.opendev.org/656413 | 15:01 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Enable image type query support in nova-next https://review.opendev.org/656903 | 15:01 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add docs for image type support request filter https://review.opendev.org/657025 | 15:01 |
| dansmith | mriedem: I've been thinking about the tracing thing too | 15:03 |
| *** cfriesen has joined #openstack-nova | 15:03 | |
| *** Sundar has quit IRC | 15:04 | |
| dansmith | more about making the existing filters log what they're doing, but the timer thing is legit too | 15:05 |
| *** tosky has quit IRC | 15:06 | |
| *** mgariepy has quit IRC | 15:07 | |
| mriedem | i'd think that osprofiler should be handling the time spent in each filter, but not sure how granular that is | 15:08 |
| dansmith | meaning we need some sort of hook thing? | 15:10 |
| *** imacdonn has quit IRC | 15:11 | |
| *** imacdonn has joined #openstack-nova | 15:11 | |
| mriedem | i'm not sure if the @profiler.trace_cls decorator on the SchedulerAPI (rpc) gives us what we'd want there | 15:12 |
| mriedem | since select_destinations is a call it might, but not sure | 15:12 |
| *** mrch_ has quit IRC | 15:12 | |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: [Docs] Change the server query parameter display into a list docs. https://review.opendev.org/657624 | 15:12 |
| mriedem | there was a recent ML thread about nicer osprofiler visualizations but i can't find it | 15:14 |
| *** tbachman has quit IRC | 15:15 | |
| *** pcaruana has quit IRC | 15:16 | |
| mriedem | oh right, something like this http://logs.openstack.org/69/617269/3/check/tempest-smoke-py3-osprofiler-redis/d5563c0/osprofiler-traces/trace-58ed6da6-f2c8-418c-9c60-337bf00ad86a.html.gz | 15:16 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: [Docs] Change the server query parameter display into a list. https://review.opendev.org/657624 | 15:16 |
| mriedem | not sure how to easily find a server create trace in there | 15:16 |
| *** ccamacho has quit IRC | 15:16 | |
| *** hamzy has quit IRC | 15:20 | |
| dansmith | mriedem: well, I'll start something and you can add to it if you want | 15:21 |
| dansmith | it'd be really nice if we could see what the reqspec looks like before/after in a generic way, but that's probably too detailed and too fragile, vs. each one just logging what it's doing | 15:22 |
| mriedem | i guess http://logs.openstack.org/69/617269/3/check/tempest-smoke-py3-osprofiler-redis/d5563c0/osprofiler-traces/trace-46016681-2a3d-4ca2-adf7-c4941649d753.html.gz and filter on nova-scheduler | 15:22 |
| mriedem | doesn't give me the time per filter though | 15:23 |
| mriedem | just says that like select_destinations took 302ms | 15:23 |
| *** tbachman has joined #openstack-nova | 15:24 | |
| *** artom has joined #openstack-nova | 15:27 | |
| *** mgariepy has joined #openstack-nova | 15:29 | |
| *** ivve has quit IRC | 15:29 | |
| mriedem | ah timeutils already has something i was thinking to use | 15:29 |
| dansmith | I was just using stopwatch | 15:29 |
| mriedem | looks like you can just use oslo_utils.timeutils.time_it | 15:30 |
| dansmith | yeah, but I want to not log if the thing appears disabled | 15:31 |
| *** jdillaman has joined #openstack-nova | 15:31 | |
| mriedem | the decorator takes an enabled kwarg | 15:32 |
| dansmith | but we don't know until after it's run, unless we change something | 15:32 |
| *** yan0s has quit IRC | 15:32 | |
| dansmith | I was just going to have them return a boolean if they're enabled/effective | 15:32 |
| dansmith | like, if we're is_bfv=True, no point in logging 0.0 | 15:33 |
| dansmith | I could only log if above a threshold, but then you don't know if something is running or running fastly | 15:33 |
| mriedem | ok that's more granular than i was thinking at first, like this is pretty easy: | 15:33 |
| mriedem | http://paste.openstack.org/show/750876/ | 15:33 |
| *** mdbooth has joined #openstack-nova | 15:33 | |
| dansmith | yeah, that works for the two simple ones, but not for my new one | 15:33 |
| dansmith | because it has three reasons to not log | 15:34 |
| dansmith | give me a few and I'll push up what I was thinking | 15:34 |
| mriedem | map_az_to_placement_aggregate also has several reasons to not do anything | 15:34 |
| mriedem | so depends on how granular you want this | 15:34 |
| openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Add extra logging to request filters https://review.opendev.org/657629 | 15:38 |
| dansmith | mriedem: ^ | 15:38 |
| dansmith | generic timer and contextual logs from each filter about what change it's making | 15:39 |
| dansmith | which I think is what I'd want to validate or diagnose | 15:39 |
| *** tbachman has quit IRC | 15:39 | |
| *** tbachman has joined #openstack-nova | 15:43 | |
| *** sapd1_x has joined #openstack-nova | 15:45 | |
| *** hamzy has joined #openstack-nova | 15:48 | |
| *** tbachman has quit IRC | 15:48 | |
| mriedem | i think your unit tests in https://review.opendev.org/#/c/656413/ are going to blow up | 15:50 |
| *** rpittau is now known as rpittau|afk | 15:51 | |
| dansmith | I thought is_bfv was usage safe so it would just return false | 15:51 |
| dansmith | but I didn't run them | 15:51 |
| dansmith | or maybe we set it somewhere before we get too far, but I won't hit that in the unit tests | 15:52 |
| mriedem | conductor sets it before the scheduler for a move, and api for a new server, but your unit tests won't lazy-load the value | 15:52 |
| dansmith | yeah | 15:53 |
| dansmith | I need to remove the empty image from the volume case anyway | 15:54 |
| *** gyee has joined #openstack-nova | 15:55 | |
| openstackgerrit | Dan Smith proposed openstack/nova master: Enable image type query support in nova-next https://review.opendev.org/656903 | 15:58 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add docs for image type support request filter https://review.opendev.org/657025 | 15:58 |
| *** cdent has joined #openstack-nova | 16:04 | |
| *** beagles has quit IRC | 16:11 | |
| *** beagles has joined #openstack-nova | 16:13 | |
| *** manjeets__ is now known as manjeets | 16:21 | |
| *** jbernard_ is now known as jbernard | 16:21 | |
| openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Add extra logging to request filters https://review.opendev.org/657629 | 16:24 |
| *** whoami-rajat has quit IRC | 16:25 | |
| *** nicolasbock has quit IRC | 16:29 | |
| *** nicolasbock has joined #openstack-nova | 16:29 | |
| openstackgerrit | John Garbutt proposed openstack/nova-specs master: Add Unified Limits Spec https://review.opendev.org/602201 | 16:40 |
| *** ivve has joined #openstack-nova | 16:40 | |
| *** whoami-rajat has joined #openstack-nova | 16:43 | |
| *** tbachman has joined #openstack-nova | 16:53 | |
| *** sapd1_x has quit IRC | 16:58 | |
| *** udesale has quit IRC | 17:13 | |
| *** maciejjozefczyk_ has quit IRC | 17:17 | |
| *** ttsiouts has joined #openstack-nova | 17:22 | |
| dansmith | ah crap | 17:23 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add image type request filter https://review.opendev.org/656413 | 17:24 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Enable image type query support in nova-next https://review.opendev.org/656903 | 17:24 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add docs for image type support request filter https://review.opendev.org/657025 | 17:24 |
| openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Add extra logging to request filters https://review.opendev.org/657629 | 17:24 |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif stable/rocky: Prevent "qbr" Linux Bridge from replying to ARP messages https://review.opendev.org/655692 | 17:34 |
| *** ttsiouts has quit IRC | 17:47 | |
| *** ttsiouts has joined #openstack-nova | 17:48 | |
| dansmith | mriedem: have you been following the helm and health check stuff that wants to throw a non-nova tool on our bus to ping all our services all the time for health checks? | 17:50 |
| dansmith | I've been arguing for going the real http-based healthcheck route to avoid extra message bus load, the upgrade concerns, and to, you know, allow health checks to work in the absence of a working message bus, which is the most likely failure in any deployment :) | 17:51 |
| artom | <drive by> I'd argue that you'd still need to check message queue health, but that it's a separate thing and shouldn't be confused with Nova | 17:52 |
| artom | Cuz if your message queue is down, not much is going to work :) | 17:53 |
| dansmith | nova does this already, and, the http based check can report "hey I can't talk to rabbit" | 17:53 |
| dansmith | the http check returns a json blob of "things I know about" and "are they working or not" | 17:53 |
| dansmith | so, "can I talk to rabbit" "am I able to report my service status" "can I talk to libvirt right now" etc | 17:54 |
| dansmith | doing that over the message bus adds load to the most loaded thing, and doesn't work if the bus is down, | 17:55 |
| dansmith | but also won't work if conductor is down after a few minutes, because nova-compute is likely to have exhausted its threadpool trying to check in | 17:55 |
| dansmith | so it's really a fragile expectation I think | 17:55 |
| *** lbragstad has quit IRC | 17:59 | |
| *** lbragstad has joined #openstack-nova | 17:59 | |
| *** ralonsoh has quit IRC | 18:00 | |
| *** ttsiouts has quit IRC | 18:14 | |
| artom | dansmith, oh, we have a specific health check "API"? | 18:15 |
| artom | TIL | 18:15 |
| dansmith | no, but kube has a standard format that people have proposed adding to our services | 18:16 |
| dansmith | and I'm saying we should do _that_ | 18:16 |
| artom | Seems sensible to me | 18:16 |
| artom | Better than doing it through the back door | 18:16 |
| * dansmith screenshots for later | 18:16 | |
| artom | What, that I agreed with you? | 18:16 |
| dansmith | no, nevermind :) | 18:17 |
| artom | Oh I know. | 18:17 |
| artom | But... this being a professional channel and all. | 18:17 |
| cdent | "we should do _that_"++ | 18:22 |
| mriedem | dansmith: no i haven't been following it | 18:23 |
| dansmith | they already have an agent that sits on our bus and pings services, and unsurprisingly, have found that the version matters and things break if you don't have that right | 18:24 |
| dansmith | and I just want to make sure we're not getting into a situation where we bless that kind of activity, and/or start trying to worry about external consumers of our RPC APIs and hold versions around because of what might be out in the wild | 18:25 |
| dansmith | https://review.opendev.org/#/c/651140/ | 18:25 |
| artom | I mean, we could be evil and just remove ping() | 18:28 |
| dansmith | artom: we need it for our own use | 18:28 |
| dansmith | and note that that error dump was on the *nova* server side, as a result of pinging with an old version, | 18:28 |
| dansmith | so that tool potentially generates bug reports for us | 18:29 |
| dansmith | artom: compute pings conductor at startup and waits until it replies, in case it gets started earlier, since it'll fail without conductor being up | 18:29 |
| dansmith | I told them about ping because they wanted to add their own ping-like thing to our rpcapi that they would use to do that, | 18:29 |
| *** ttsiouts has joined #openstack-nova | 18:29 | |
| dansmith | so I was like "we already have this, so please don't add another one, but also... about that approach..." | 18:30 |
| artom | Wait, ping() is in the conductor rpcapi, right? | 18:31 |
| artom | So... how are they going to check for Nova API health? | 18:31 |
| artom | Which I assume is what they most care about... | 18:31 |
| openstackgerrit | Chris Dent proposed openstack/nova master: Make all functional tests reusable by other projects https://review.opendev.org/657659 | 18:31 |
| artom | Ah, it's in the base class | 18:33 |
| artom | So all our services have it | 18:33 |
| dansmith | they can't check api that way anyway | 18:34 |
| dansmith | because api doesn't listen | 18:34 |
| dansmith | but yeah, it's baked in for the services that do | 18:34 |
| *** ttsiouts has quit IRC | 18:35 | |
| artom | So that's a good argument: you can only use ping() on services that listen to the message queue, which API doesn't. | 18:35 |
| artom | And API is kinda important ;) | 18:35 |
| dansmith | one could argue that you can test nova-api externally already ... via the api | 18:36 |
| dansmith | but yeah, it's an argument.. the others I have are more important to me though :) | 18:36 |
| artom | Right, I'm trying to think of what's important to *them* :) | 18:36 |
| artom | Since you know, it's how you actually convince people ;) | 18:36 |
| artom | Reading aspiers's reply on https://review.opendev.org/#/c/653707/2/specs/rpc-health-checks.rst@43, I can see where the self-healing SIG people are coming from. You want a think that tells you directly, with no proxy, whether a single service is healthy or not. | 18:39 |
| artom | You then have your own intelligence on what to do. But you don't need further analysis to see where the actual failure is - well, less of it, anyways | 18:39 |
| artom | Doing it through the RPC back door is one less layer to think about | 18:39 |
| dansmith | it's dependent on the network and rabbit working though | 18:40 |
| dansmith | and hides the visibility of what might be wrong on the network | 18:40 |
| dansmith | like, if you get no route, you know the computer is down. if you get refused, you know the service is down | 18:41 |
| dansmith | via rpc, you'll just not hear back, which could mean pretty much anything, including "the bus is overloaded right now" | 18:41 |
| artom | Also true. | 18:42 |
| dansmith | regardless, I don't want them to be on our bus as a matter of principle, aside from the fact that I think they'll get bad data from doing so | 18:42 |
| artom | Anyways, I need to step out of this philosophical discussion, things need doing :) | 18:42 |
| *** purplerbot has quit IRC | 18:42 | |
| artom | Was fun thinking about, though | 18:42 |
| *** purplerbot has joined #openstack-nova | 18:42 | |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Microversion 2.73: Support adding the reason behind a server lock https://review.opendev.org/648662 | 18:43 |
| tssurya | mriedem: thanks again for the detailed review ^. I totally appreciate it. | 18:44 |
| *** zbr is now known as zbr|pto | 18:45 | |
| aspiers | dansmith: thanks for the quick and helpful reply. I think I'm totally fine with your suggestion of opening up new dedicated HTTP endpoints for health-checking RPC-only services, and the advantages you cite make sense | 18:49 |
| *** _hemna has joined #openstack-nova | 18:51 | |
| aspiers | dansmith, artom: FWIW, the problem the OSH folks encountered was not due to pinging with an old version - it was due to not knowing ping() exists and therefore deliberately invoking a non-existent RPC call, which (unsurprisingly) caused errors in the server logs | 18:53 |
| aspiers | The errors were actually masked for a while, but surfaced when something in oslo logging changed | 18:53 |
| artom | aspiers, ah, yeah that makes more sense with "Attempted method: pod_health_probe_method_ignore_errors" | 18:54 |
| dansmith | aspiers: it's the same thing that will happen if you do use ping, but with the wrong version | 18:54 |
| artom | That's definitely not a method we have ') | 18:54 |
| dansmith | which was my point | 18:54 |
| aspiers | dansmith: gotcha | 18:54 |
| artom | It does add an external thing to the upgrade process. Nova internally can support N/N+1 | 18:55 |
| artom | With the proposed health check, suddenly this new external thing has be to upgraded at the same time as part of the Nova upgrade | 18:55 |
| dansmith | artom: we actually only support N/N+1 internally for major versions on bridge releases | 18:55 |
| dansmith | we support N.0-N.x on regular releases | 18:56 |
| aspiers | artom: You lost me - what has to be upgraded? | 18:56 |
| dansmith | this ^ :) | 18:56 |
| artom | aspiers, and dansmith lost me :) | 18:56 |
| artom | What's a bridge release? | 18:57 |
| aspiers | Are you talking about a scenario where the ping() interface changes in future versions? | 18:57 |
| dansmith | we haven't had one in a while, but when we bump the major, we support the old and new majors on the server side | 18:57 |
| dansmith | it's a whole complex thing | 18:57 |
| dansmith | aspiers: ping is unlikely to change, but it could, but even if it doesn't, you have to track the supported major in your tool else you'll generate errors like the one in the commit message when we bump, even though ping hasn't changed | 18:58 |
| aspiers | dansmith: you mean because it won't support calls from (much) older versions? | 18:59 |
| dansmith | and during a major bump release, you'd have to know which computes are upgraded to send the new major | 18:59 |
| dansmith | and conductors don't support multiple majors | 18:59 |
| artom | Ah, so my confusion stemmed from thinking release = major bump, which isn't the case | 18:59 |
| dansmith | artom: right | 18:59 |
| dansmith | aspiers: from anything other than major.0 right | 18:59 |
| *** mgariepy has quit IRC | 18:59 | |
| aspiers | OK got it (I think) | 18:59 |
| artom | So technically queens computes can talk to stein conductors, because both are major 5 | 18:59 |
| aspiers | even though I suspect the liveness probe would be packaged within the same container as the service it's checking | 19:00 |
| artom | It would still have to be code-updated at the same time with the new major | 19:01 |
| aspiers | right | 19:01 |
| artom | Which is harder when it's not the same repo :) | 19:01 |
| dansmith | this is why you shouldn't do this :) | 19:01 |
| aspiers | whereas going through oslo.middleware (say) wouldn't have any of these problems | 19:01 |
| *** ttsiouts has joined #openstack-nova | 19:02 | |
| dansmith | and we'd have a defined health check schema we agree on and support | 19:03 |
| dansmith | i.e. list of tests | 19:03 |
| aspiers | Hmm, re-reading the scrollback - did you say nova already exposes some health data on an existing endpoint? | 19:04 |
| aspiers | I wasn't aware of that ... /me checks the API docs | 19:05 |
| dansmith | no | 19:05 |
| aspiers | Maybe it's just using the existing oslo.middleware mechanism | 19:05 |
| dansmith | no, I'm saying we *should* do that | 19:05 |
| dansmith | none of our services (other than API) even have http interfaces, so we need to do that thing | 19:05 |
| aspiers | Ah, wrong interpretation of "this" --> http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-05-07.log.html#t2019-05-07T17:53:25 | 19:05 |
| *** mgariepy has joined #openstack-nova | 19:05 | |
| dansmith | I'm saying this is what we should be doing | 19:05 |
| aspiers | Yeah OK, that's what I thought. Just got momentarily confused by the scrollback. | 19:06 |
| aspiers | Sorry :) | 19:06 |
| *** ttsiouts has quit IRC | 19:06 | |
| artom | I have to say, opening up internal services with an HTTP endpoint seems dangerous, security wise. | 19:09 |
| artom | Depends on how it's implemented, I guess | 19:09 |
| artom | But being able to DDoS the conductor doesn't seem fun | 19:09 |
| artom | Tough I guess any self-respecting operator will firewall the crap out of those | 19:10 |
| aspiers | artom: I doubt OS-Helm has any intention of carrying out DDoS attacks ;-) | 19:10 |
| aspiers | Exactly. They probably don't even need to listen outside localhost | 19:10 |
| *** tesseract has quit IRC | 19:10 | |
| aspiers | Remember this is k8s monitoring from the container host | 19:10 |
| aspiers | or similar | 19:10 |
| artom | aspiers, I don't doubt Helm's intentions, it's others I'm paranoid about | 19:11 |
| aspiers | It won't be open to others | 19:11 |
| aspiers | ... unless the operator wants it to be. We're on the same page :) | 19:11 |
| dansmith | artom: it's read-only, has no real info, and you can make it depth-1, so literally only one request at a time | 19:12 |
| aspiers | Right | 19:12 |
| artom | dansmith, what, you mean dumping a detailed 1MB JSON blob of stats isn't a good idea? ;) | 19:12 |
| aspiers | It actually links up quite nicely with dirk's idea of exposing more internal metrics for consumption by e.g. Prometheus | 19:13 |
| aspiers | Anyway, gotta go. Back tomorrow. | 19:14 |
| *** jobewan has joined #openstack-nova | 19:24 | |
| *** tssurya has quit IRC | 19:29 | |
| *** _hemna has quit IRC | 19:34 | |
| *** ttsiouts has joined #openstack-nova | 19:43 | |
| *** itlinux has joined #openstack-nova | 19:58 | |
| *** bbowen_ has quit IRC | 20:12 | |
| *** ttsiouts has quit IRC | 20:16 | |
| *** itlinux has quit IRC | 20:19 | |
| *** hamzy has quit IRC | 20:27 | |
| *** artom has quit IRC | 20:31 | |
| *** slaweq has quit IRC | 20:36 | |
| *** slaweq has joined #openstack-nova | 20:42 | |
| *** tssurya has joined #openstack-nova | 20:52 | |
| *** slaweq has quit IRC | 20:55 | |
| *** jdillaman has quit IRC | 21:00 | |
| *** slaweq has joined #openstack-nova | 21:13 | |
| *** ttsiouts has joined #openstack-nova | 21:18 | |
| *** ttsiouts has quit IRC | 21:23 | |
| *** slaweq has quit IRC | 21:25 | |
| *** nicolasbock has quit IRC | 21:25 | |
| *** nicolasbock has joined #openstack-nova | 21:25 | |
| *** bbowen_ has joined #openstack-nova | 21:30 | |
| *** hongbin has joined #openstack-nova | 21:35 | |
| *** mriedem has quit IRC | 21:36 | |
| cfriesen | is opendev.org down? | 21:49 |
| cdent | cfriesen: looks that way | 22:02 |
| *** JamesBenson has quit IRC | 22:03 | |
| *** slaweq has joined #openstack-nova | 22:11 | |
| *** ccstone has joined #openstack-nova | 22:20 | |
| *** slaweq has quit IRC | 22:24 | |
| openstackgerrit | John Garbutt proposed openstack/nova master: Move default policy target https://review.opendev.org/657696 | 22:43 |
| openstackgerrit | John Garbutt proposed openstack/nova master: Better policy unit tests https://review.opendev.org/657697 | 22:43 |
| openstackgerrit | John Garbutt proposed openstack/nova master: Add functional test for admin_actions https://review.opendev.org/657698 | 22:43 |
| *** panda has quit IRC | 22:44 | |
| *** jobewan has quit IRC | 22:45 | |
| *** panda has joined #openstack-nova | 22:46 | |
| *** zer0c00l_ has joined #openstack-nova | 22:48 | |
| openstackgerrit | John Garbutt proposed openstack/nova master: WIP: Integrating with unified limits https://review.opendev.org/615180 | 22:57 |
| *** threestrands has joined #openstack-nova | 23:00 | |
| *** tkajinam has joined #openstack-nova | 23:00 | |
| *** threestrands has quit IRC | 23:00 | |
| *** rcernin has joined #openstack-nova | 23:06 | |
| *** slaweq has joined #openstack-nova | 23:11 | |
| *** artom has joined #openstack-nova | 23:21 | |
| *** openstackstatus has joined #openstack-nova | 23:24 | |
| *** ChanServ sets mode: +v openstackstatus | 23:24 | |
| *** slaweq has quit IRC | 23:24 | |
| -openstackstatus- NOTICE: If your jobs failed due to connectivity issues to opendev.org they can be rechecked now. Services have been restored at that domain. | 23:26 | |
| *** hoonetorg has quit IRC | 23:33 | |
| *** ttsiouts has joined #openstack-nova | 23:35 | |
| *** hongbin has quit IRC | 23:41 | |
| *** mlavalle has quit IRC | 23:43 | |
| *** hoonetorg has joined #openstack-nova | 23:47 | |
| *** bbowen_ has quit IRC | 23:47 | |
| *** bbowen_ has joined #openstack-nova | 23:47 | |
| *** JamesBenson has joined #openstack-nova | 23:53 | |
| *** whoami-rajat has quit IRC | 23:55 | |
| *** JamesBenson has quit IRC | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!