| *** dave-mccowan has quit IRC | 00:58 | |
| *** factor has joined #openstack-sdks | 01:39 | |
| *** bobh has joined #openstack-sdks | 01:51 | |
| openstackgerrit | Adrian Turjak proposed openstack/keystoneauth master: add support for auth_receipts and multi-method auth https://review.opendev.org/675049 | 02:37 |
|---|---|---|
| *** bobh has quit IRC | 02:46 | |
| openstackgerrit | Adrian Turjak proposed openstack/keystoneauth master: add support for auth_receipts and multi-method auth https://review.opendev.org/675049 | 02:46 |
| *** bobh has joined #openstack-sdks | 02:50 | |
| *** bobh has quit IRC | 02:57 | |
| *** gkadam has joined #openstack-sdks | 03:39 | |
| *** gkadam has quit IRC | 03:40 | |
| openstackgerrit | Brin Zhang proposed openstack/python-openstackclient master: Microversion 2.76: Support Specifying AZ to unshelve https://review.opendev.org/665336 | 04:05 |
| *** goldyfruit_ has quit IRC | 04:11 | |
| *** goldyfruit_ has joined #openstack-sdks | 04:11 | |
| *** whoami-rajat has joined #openstack-sdks | 04:18 | |
| *** dave-mccowan has joined #openstack-sdks | 04:56 | |
| *** dave-mccowan has quit IRC | 05:01 | |
| *** gkadam has joined #openstack-sdks | 05:30 | |
| openstackgerrit | Vieri proposed openstack/openstackclient master: Add Python 3 Train unit tests https://review.opendev.org/676117 | 06:48 |
| *** slaweq has joined #openstack-sdks | 07:05 | |
| *** dtantsur|afk is now known as dtantsur | 07:32 | |
| *** jangutter has joined #openstack-sdks | 07:34 | |
| *** tosky has joined #openstack-sdks | 07:37 | |
| *** jpena|off is now known as jpena | 07:39 | |
| *** yolanda has quit IRC | 08:07 | |
| *** jpich has joined #openstack-sdks | 08:12 | |
| *** cdent has joined #openstack-sdks | 08:26 | |
| *** jpich has quit IRC | 09:06 | |
| *** jpich has joined #openstack-sdks | 09:07 | |
| *** jpich has quit IRC | 09:23 | |
| *** jpich has joined #openstack-sdks | 09:26 | |
| *** e0ne has joined #openstack-sdks | 09:34 | |
| *** jpich has quit IRC | 10:21 | |
| *** jpich has joined #openstack-sdks | 10:22 | |
| dtantsur | mmm, so apparently our image API is using an unused "params" argument to Resource.fetch and tests fail if you wire it in.. | 10:35 |
| openstackgerrit | Dmitry Tantsur proposed openstack/openstacksdk master: Add support for fields in baremetal get_* resources https://review.opendev.org/676154 | 10:36 |
| dtantsur | efried: needed for nova ^^ but won't pass unit tests because of the image API test weirdness | 10:36 |
| dtantsur | ooops, sorry, it was DNS API | 10:37 |
| mordred | dtantsur: this is the test_wait_for_task_wait issue? | 10:54 |
| openstackgerrit | Dmitry Tantsur proposed openstack/openstacksdk master: Add support for fields in baremetal get_* resources https://review.opendev.org/676154 | 10:55 |
| dtantsur | mordred: no. I've uncovered two actual bugs, I think ^^^ | 10:55 |
| mordred | ah - neat | 10:55 |
| dtantsur | reminder for us: never ignore arguments, somebody will try to provide them :) | 10:55 |
| mordred | ++ | 10:56 |
| *** goldyfruit_ has quit IRC | 11:13 | |
| *** jpena is now known as jpena|lunch | 11:25 | |
| *** slaweq_ has joined #openstack-sdks | 11:29 | |
| *** slaweq has quit IRC | 11:33 | |
| *** mriedem has joined #openstack-sdks | 11:52 | |
| *** ricolin has quit IRC | 12:01 | |
| openstackgerrit | Monty Taylor proposed openstack/openstacksdk master: Fix discovery cache sharing https://review.opendev.org/675121 | 12:10 |
| openstackgerrit | Monty Taylor proposed openstack/openstacksdk master: Rationalize endpoint_for and get_endpoint_from_catalog https://review.opendev.org/675130 | 12:10 |
| openstackgerrit | Monty Taylor proposed openstack/openstacksdk master: Add support for global_request_id https://review.opendev.org/674807 | 12:10 |
| openstackgerrit | Monty Taylor proposed openstack/openstacksdk master: Validate that connect_as connects as the project https://review.opendev.org/675178 | 12:10 |
| openstackgerrit | Monty Taylor proposed openstack/openstacksdk master: Replace catalog-v3.json with keystoneauth fixture https://review.opendev.org/675187 | 12:10 |
| openstackgerrit | Monty Taylor proposed openstack/openstacksdk master: Allow services to override proxy creation https://review.opendev.org/674110 | 12:10 |
| openstackgerrit | Monty Taylor proposed openstack/openstacksdk master: Rationalize endpoint_for and get_endpoint_from_catalog https://review.opendev.org/675130 | 12:11 |
| openstackgerrit | Monty Taylor proposed openstack/openstacksdk master: Add support for global_request_id https://review.opendev.org/674807 | 12:14 |
| openstackgerrit | Monty Taylor proposed openstack/openstacksdk master: Validate that connect_as connects as the project https://review.opendev.org/675178 | 12:14 |
| openstackgerrit | Monty Taylor proposed openstack/openstacksdk master: Rationalize endpoint_for and get_endpoint_from_catalog https://review.opendev.org/675130 | 12:16 |
| openstackgerrit | Monty Taylor proposed openstack/openstacksdk master: Replace catalog-v3.json with keystoneauth fixture https://review.opendev.org/675187 | 12:16 |
| mordred | sorry for the churn - I thought I could pull one of those patches out to be independent - but turns out I couldn't | 12:16 |
| *** bobh has joined #openstack-sdks | 12:16 | |
| *** bobh has quit IRC | 12:21 | |
| *** jpena|lunch is now known as jpena | 12:22 | |
| *** bobh has joined #openstack-sdks | 12:35 | |
| openstackgerrit | Maxim Babushkin proposed openstack/openstacksdk master: Add 'tag' support to compute with supported microversion https://review.opendev.org/675306 | 12:40 |
| *** slaweq__ has joined #openstack-sdks | 12:45 | |
| *** slaweq_ has quit IRC | 12:45 | |
| *** jangutter_ has joined #openstack-sdks | 12:49 | |
| *** factor has quit IRC | 12:51 | |
| *** factor has joined #openstack-sdks | 12:52 | |
| *** jangutter has quit IRC | 12:52 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/openstacksdk master: Add support for fields in baremetal get_* resources https://review.opendev.org/676154 | 13:03 |
| *** jpich has quit IRC | 13:03 | |
| *** jpich has joined #openstack-sdks | 13:03 | |
| *** goldyfruit has joined #openstack-sdks | 13:13 | |
| *** bobh has quit IRC | 13:14 | |
| *** gkadam has quit IRC | 13:17 | |
| *** lbragstad has joined #openstack-sdks | 13:31 | |
| *** slaweq__ is now known as slaweq | 13:33 | |
| efried | dtantsur: I was going to hit you up about that | 13:49 |
| efried | I was looking into it yesterday | 13:49 |
| dtantsur | I hope I didn't step on your shoes | 13:49 |
| efried | just got to my desk, so haven't caught up, except to see that you commented on the nova patch and seem to have identified the/a problem | 13:49 |
| dtantsur | yeah | 13:49 |
| efried | from what I could tell, it was happening during construction of the proxy | 13:50 |
| efried | is that the case? | 13:50 |
| dtantsur | efried: ignore the version blah-blah logging. the actual problem seems to be get_node(<>, fields=..). we never implemented the "fields" bit. | 13:50 |
| dtantsur | my patch above hopefully adds its support | 13:51 |
| * dtantsur waits for the CI | 13:51 | |
| efried | dtantsur: Okay, the thing that concers me is... | 13:51 |
| * efried gets log links... | 13:51 | |
| efried | dtantsur: Okay, here | 13:52 |
| efried | https://logs.opendev.org/99/642899/38/check/ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa/665016d/controller/logs/screen-n-cpu.txt.gz?level=ERROR | 13:52 |
| dtantsur | "StrictVersion instance has no attribute 'version'" is an annoying exception that *likely* hides the fact that ironic is not up yet | 13:53 |
| efried | Even when this job succeeds, the "Error: StrictVersion instance has no attribute 'version'" always shows up, as does the "VirtDriverNotReady: Virt driver is not ready." | 13:53 |
| dtantsur | the actual problem that fails you https://logs.opendev.org/99/642899/38/check/ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa/665016d/controller/logs/screen-n-cpu.txt.gz?level=ERROR#_Aug_13_05_19_11_540713 | 13:53 |
| efried | And we always see at least one "No compute node record for host" | 13:54 |
| dtantsur | efried: right, it's a bug in ironicclient that hides the ConnectionRefused or whatever | 13:54 |
| dtantsur | because ironic starts later than nova | 13:54 |
| efried | right, so | 13:54 |
| dtantsur | I may be confusing some details, but the overall picture is like this | 13:54 |
| efried | this error didn't start appearing in this patch until the latest PS | 13:54 |
| efried | let me show you a log from before... | 13:54 |
| dtantsur | I've seen it many times already. Not sure if it's permanent or depends on some ordering in devstack. | 13:55 |
| efried | Yeah, I'm just using it as a baseline. | 13:57 |
| efried | dtantsur: So this is what I consider "good": https://logs.opendev.org/99/642899/36/check/ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa/4b528c1/controller/logs/screen-n-cpu.txt.gz?level=ERROR | 13:57 |
| efried | That job succeeded ^ | 13:57 |
| dtantsur | same problem in a recent successful run: https://logs.opendev.org/40/675940/1/check/ironicclient-tempest/f57c0f3/controller/logs/screen-n-cpu.txt.gz?level=WARNING#_Aug_12_16_38_40_790400 | 13:57 |
| dtantsur | right | 13:57 |
| efried | So now comparing against the run in the same time frame on the *subsequent* patch, whose only difference is that it starts using .nodes... | 13:57 |
| efried | https://logs.opendev.org/27/656027/20/check/ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa/ed7caf9/controller/logs/screen-n-cpu.txt.gz?level=ERROR | 13:58 |
| efried | Now all of a sudden we're seeing this: Error: The baremetal service for 213.32.77.113:RegionOne exists but does not have any supported versions.: NotSupported: The baremetal service for 213.32.77.113:RegionOne exists but does not have any supported versions. | 13:58 |
| efried | which comes from _make_proxy | 13:58 |
| efried | i.e. when we first bootstrap the connection | 13:58 |
| *** whoami-rajat has quit IRC | 13:59 | |
| efried | which implies to me that the *previous* succeeding run wasn't hitting the changed code at all, which would be... bad :( | 13:59 |
| dtantsur | we better ask mordred but I suspect "does not have any supported version" may be the consequence of ironic being down still | 14:00 |
| efried | That never appears in any of the other runs though | 14:00 |
| efried | I looked through a dozen yesterday | 14:00 |
| efried | or | 14:00 |
| efried | I guess it could be timing. If the code path that hits .nodes() happens much earlier than get_node(), and the service is still down, and we try to make the proxy and fail discovery... but we should try again a bit later, shouldn't we?? | 14:01 |
| *** lbragstad has quit IRC | 14:02 | |
| dtantsur | I guess we do? I'd fix the get_node problem first, then see where we stand | 14:02 |
| efried | dtantsur: Yeah, see, the next patch in the series *should* be hitting that same get_node error, but it's not getting that far: https://logs.opendev.org/27/656027/21/check/ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa/60265c4/controller/logs/screen-n-cpu.txt.gz?level=ERROR | 14:03 |
| dtantsur | it cannot connect to keystone, maybe that's the cause? | 14:04 |
| efried | I suspect this just means the nova side needs to be more tolerant of service-not-yet-up. | 14:04 |
| dtantsur | yep | 14:04 |
| openstackgerrit | Merged openstack/openstacksdk master: DRY test_connection https://review.opendev.org/676009 | 14:08 |
| *** whoami-rajat has joined #openstack-sdks | 14:25 | |
| *** ricolin has joined #openstack-sdks | 14:28 | |
| efried | dustinc: ^ | 14:48 |
| dustinc | efried, dtantsur: thanks, will catch up when I get to work in about an hour | 14:52 |
| dtantsur | mordred, efried, https://review.opendev.org/#/c/676154/ passed now, please check | 14:57 |
| *** dave-mccowan has joined #openstack-sdks | 14:57 | |
| *** dave-mccowan has quit IRC | 15:04 | |
| openstackgerrit | Merged openstack/openstacksdk master: Fix discovery cache sharing https://review.opendev.org/675121 | 15:14 |
| efried | dtantsur: +A, thanks for the quick fix. | 15:35 |
| dtantsur | yvw | 15:36 |
| efried | dtantsur, mordred: still trying to nail down the difference in the discovery path between ksa and sdk | 15:47 |
| efried | specifically here https://logs.opendev.org/27/656027/21/check/ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa/60265c4/controller/logs/screen-n-cpu.txt.gz?level=WARNING#_Aug_13_04_59_18_053245 | 15:48 |
| efried | In this case we want ServiceUnavailable to be an exception, but I *think* it's proceeding and still trying to return a real proxy. | 15:48 |
| efried | I think allow_version_hack=False would get us DiscoveryFailure here | 15:49 |
| efried | ...but I'm not sure whether that would help :( | 15:49 |
| * dtantsur looks at mordred | 15:50 | |
| efried | mordred: I... kind of suspect we should be defaulting allow_version_hack=False from the sdk side. AFAICT it's always True today - even for the code paths where we're not explicitly setting it. | 15:55 |
| *** jpena is now known as jpena|off | 15:58 | |
| *** shubham_potale has joined #openstack-sdks | 16:06 | |
| efried | dustinc: When you're about, we should talk. | 16:23 |
| *** dtantsur is now known as dtantsur|afk | 16:29 | |
| dustinc | efried: around now | 16:29 |
| efried | dustinc: The nova/sdk series needs to be passing the ironic CI job before it's going to fly | 16:29 |
| efried | before yesterday, with the fields thing, the bottom patch was passing, but the second one (and up) was failing. | 16:30 |
| efried | and I still haven't been able to figure out exactly why. | 16:30 |
| dustinc | Thanks, I have not been giving it a lot of attention recently but am working on it today | 16:34 |
| efried | dustinc: In the CI env, the n-cpu service comes up before ironic | 16:35 |
| efried | so the first few attempts to connect will fail. | 16:35 |
| efried | This is a "normal" run: https://logs.opendev.org/40/675940/1/check/ironicclient-tempest/f57c0f3/controller/logs/screen-n-cpu.txt.gz?level=WARNING#_Aug_12_16_38_40_790400 | 16:35 |
| efried | And that's what things would look like, more or less, on the bottom patch (before the fields thing) | 16:37 |
| dustinc | thanks, checking it out right now | 16:37 |
| efried | But then with the second patch, we would start seeing stuff like this https://logs.opendev.org/27/656027/21/check/ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa/60265c4/controller/logs/screen-n-cpu.txt.gz?level=WARNING#_Aug_13_04_59_18_053245 | 16:37 |
| dustinc | (I knew about the fields support thing but totally forgot and thus added it back..) | 16:37 |
| efried | My current suspicion is that it's because we're using different retries/timeout than before. | 16:38 |
| dustinc | Hmm | 16:38 |
| *** jangutter_ has quit IRC | 16:39 | |
| efried | dtantsur|afk, dustinc, mordred: I think now that we have retry opts in ksa config [1][2], we should deprecate nova's equivalents [3] via alias so that they're used for both the old (iroincclient) and new (sdk) code paths, which will assure us parity at least in that regard. | 16:46 |
| efried | [1] https://review.opendev.org/666287 | 16:46 |
| efried | [2] https://review.opendev.org/672930 | 16:46 |
| efried | [3] https://opendev.org/openstack/nova/src/branch/master/nova/conf/ironic.py#L39-L63 | 16:46 |
| *** jpich has quit IRC | 16:47 | |
| *** ricolin_ has joined #openstack-sdks | 16:48 | |
| *** e0ne has quit IRC | 16:51 | |
| *** ricolin has quit IRC | 16:52 | |
| *** slaweq has quit IRC | 16:52 | |
| *** cdent has quit IRC | 16:53 | |
| *** mgariepy has quit IRC | 16:58 | |
| *** jangutter has joined #openstack-sdks | 16:59 | |
| openstackgerrit | Merged openstack/openstacksdk master: Add support for fields in baremetal get_* resources https://review.opendev.org/676154 | 16:59 |
| openstackgerrit | Eric Fried proposed openstack/openstacksdk master: Bump keystoneauth1 minimum to 3.16.0 https://review.opendev.org/676247 | 17:00 |
| efried | mordred, Shrews: I'm going to propose a sdk release now, mkay? | 17:00 |
| efried | accumulated a bunch of fixes nova (at least) needs. | 17:00 |
| mordred | efried: ++ | 17:01 |
| Shrews | up to mordred, but i have no problem with it | 17:01 |
| mordred | efried: actually ... | 17:01 |
| Shrews | will be able to restart nodepool with the object fixes, which will be nice | 17:01 |
| mordred | efried: should we also land https://review.opendev.org/#/c/674807/ too? | 17:01 |
| mordred | (that also contains the keystoneauth dep bump) | 17:02 |
| * mordred is fine cutting multiple releases too | 17:02 | |
| efried | mordred: could do, yes. Is there a reason that wasn't +W'd? | 17:03 |
| efried | mordred: also, that one bumps to 3.15.0, but we need .16. to pick up both retry changes. | 17:03 |
| efried | I could rebase my bump on that one and +W it. | 17:03 |
| mordred | ++ | 17:03 |
| *** ricolin_ is now known as ricolin | 17:04 | |
| *** jangutter has quit IRC | 17:10 | |
| openstackgerrit | Eric Fried proposed openstack/openstacksdk master: Bump keystoneauth1 minimum to 3.16.0 https://review.opendev.org/676247 | 17:11 |
| efried | mordred: ^ | 17:11 |
| efried | Shrews: too | 17:11 |
| efried | dustinc: I'm going to start hacking at that thing I said earlier, about deprecating the existing ironic retry opts. There are other ways to get the right timeouts into the sdk proxy, but they're messier. | 17:15 |
| efried | ...unless you want it | 17:15 |
| dustinc | efried: go for it, I am about to push a small fix to the bottom patch | 17:17 |
| efried | dustinc: are you taking out 'fields' for now? | 17:18 |
| dustinc | I was going to leave it expecting the SDK release to add support | 17:18 |
| dustinc | I can take it out for now if you want.. | 17:18 |
| efried | dustinc: meh | 17:19 |
| efried | I think we're not going to be able to land anything until we've got some of these issues sorted out | 17:19 |
| efried | so probably no hurry | 17:19 |
| efried | but you're going to want to put your series on top of this thing I'm doing | 17:20 |
| *** mgariepy has joined #openstack-sdks | 17:20 | |
| efried | which will also require the new sdk release | 17:20 |
| efried | for which I'm going to wait for the ksa bump | 17:20 |
| efried | so we got a little while until all those pieces come together. | 17:20 |
| *** shubham_potale has quit IRC | 18:04 | |
| *** e0ne has joined #openstack-sdks | 18:23 | |
| openstackgerrit | Merged openstack/openstacksdk master: Add support for global_request_id https://review.opendev.org/674807 | 18:35 |
| *** mgariepy has quit IRC | 19:08 | |
| *** mriedem has quit IRC | 19:08 | |
| *** mriedem has joined #openstack-sdks | 19:12 | |
| *** whoami-rajat has quit IRC | 19:33 | |
| *** slaweq has joined #openstack-sdks | 19:34 | |
| *** mgariepy has joined #openstack-sdks | 19:34 | |
| openstackgerrit | Merged openstack/openstacksdk master: Bump keystoneauth1 minimum to 3.16.0 https://review.opendev.org/676247 | 20:35 |
| *** e0ne has quit IRC | 20:38 | |
| efried | openstacksdk release 0.34.0 proposed: https://review.opendev.org/676276 | 20:44 |
| *** slaweq has quit IRC | 22:10 | |
| *** slaweq has joined #openstack-sdks | 22:26 | |
| *** tosky has quit IRC | 22:27 | |
| *** slaweq has quit IRC | 22:30 | |
| openstackgerrit | Merged openstack/openstacksdk master: Minor refactor cleanup of _make_proxy https://review.opendev.org/676015 | 22:50 |
| *** mriedem has quit IRC | 23:08 | |
| openstackgerrit | Adrian Turjak proposed openstack/keystoneauth master: add support for auth_receipts and multi-method auth https://review.opendev.org/675049 | 23:51 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!