| *** jamielennox_ has joined #openstack-shade | 00:02 | |
| *** jamielennox_ is now known as jamielennox | 00:06 | |
| *** gouthamr has quit IRC | 02:45 | |
| *** gouthamr has joined #openstack-shade | 02:46 | |
| *** calebb has quit IRC | 03:07 | |
| *** calebb has joined #openstack-shade | 03:08 | |
| *** gouthamr has quit IRC | 04:01 | |
| *** yfried has joined #openstack-shade | 06:15 | |
| *** yfried has quit IRC | 06:36 | |
| *** yfried has joined #openstack-shade | 06:55 | |
| *** iogg has joined #openstack-shade | 07:51 | |
| *** cdent has joined #openstack-shade | 09:35 | |
| *** iogg has quit IRC | 13:12 | |
| *** gouthamr has joined #openstack-shade | 13:13 | |
| *** dct360 has joined #openstack-shade | 13:29 | |
| *** openstackgerrit has joined #openstack-shade | 13:35 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Wait for volumes to detach before deleting them https://review.openstack.org/432770 | 13:35 |
|---|---|---|
| *** iogg has joined #openstack-shade | 14:12 | |
| openstackgerrit | Merged openstack-infra/shade master: Add accessor method to pull URLs from the catalog https://review.openstack.org/432358 | 14:20 |
| *** yfried has quit IRC | 14:36 | |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config master: Add support for overriding mistral service type https://review.openstack.org/434344 | 15:32 |
| *** yolanda has quit IRC | 15:34 | |
| *** yolanda has joined #openstack-shade | 15:35 | |
| dct360 | hello i'm having a problem with shade auth - has anyone seen this error before? http://paste.openstack.org/show/599064/ | 15:55 |
| *** abregman has joined #openstack-shade | 15:56 | |
| *** abregman has quit IRC | 15:56 | |
| *** abregman has joined #openstack-shade | 15:58 | |
| mordred | dct360: hrm. that doesn't look happy ... one sec | 16:03 |
| morgan_ | mordred: i was also looking at that, off the top of my head and compared to my setup... it looks ok. | 16:05 |
| mordred | yah | 16:06 |
| * mordred is trying to come up with a test script to produce more info | 16:06 | |
| mordred | that error, btw, means that os-client-config did not instantiate an auth_plugin when it was reading the config | 16:07 |
| mordred | but not instantiating an auth_plugin _should_ produce an exception | 16:08 |
| morgan_ | dct360: what is your auth_url | 16:08 |
| morgan_ | i know you've redacted it there, but i'm wondering if it's something wonky with it / typo | 16:08 |
| mordred | morgan_: this _should_ be an error before anything even tries to contact keystone | 16:09 |
| morgan_ | mordred: right, wondering if the auth_url is malformed | 16:09 |
| dct360 | mordred: it looks like this: https://ipaddress:5000/v2.0 | 16:09 |
| morgan_ | in a way that can't even be processed | 16:09 |
| morgan_ | that should be fine then. | 16:09 |
| mordred | morgan_: if you look in os_client_config.cloud_config.CloudConfig.get_session you can see the error (self._auth is None) | 16:10 |
| morgan_ | mordred: right. | 16:10 |
| *** morgan_ is now known as morgan | 16:10 | |
| mordred | morgan_: self._auth should never be None in this code flow - so YAY! | 16:10 |
| *** yfried has joined #openstack-shade | 16:10 | |
| *** yfried has quit IRC | 16:10 | |
| mordred | ooh. wait - I see something | 16:11 |
| morgan | ? | 16:14 |
| mordred | dct360: can you add: | 16:14 |
| mordred | import logging | 16:14 |
| mordred | log = logging.getLogger('os_client_config') | 16:15 |
| mordred | log.addHandler(logging.StreamHandler()) | 16:15 |
| mordred | log.setLevel(logging.DEBUG) | 16:15 |
| mordred | to the top of your script and try it again? I want to see if we get a debug message with "Deferring keystone exception" in it | 16:15 |
| dct360 | mordred: yes will try that | 16:16 |
| mordred | morgan: in os_client_config.config - if you look for that string, you'll see that in our ksc-fallback code we do not ever get an auth plugin (which makes sense) | 16:17 |
| morgan | ooooh | 16:17 |
| mordred | morgan: I'm thinking it's time for that code to DIAF - we just need to make sure killing it doesn't break OSC | 16:17 |
| morgan | yes, we need to kill that | 16:17 |
| mordred | morgan: I _ALSO_ think we should move the plugin loading out of get_one_cloud and into get_session | 16:18 |
| morgan | mordred: totally unrelated: https://usercontent.irccloud-cdn.com/file/RgvX1vAP/gnu.jpg looks good framed. | 16:18 |
| mordred | morgan: which I believe will fix many issues dtroyer has had in the past | 16:18 |
| mordred | morgan: nice! | 16:19 |
| morgan | mordred: likely. it would be better to have all that in get_session | 16:19 |
| morgan | since it is part of the session getting. | 16:19 |
| mordred | yup | 16:20 |
| mordred | the original motivation was to try to give people more informative errors sooner | 16:20 |
| mordred | but that's clearly not happening :) | 16:20 |
| dct360 | mordred: Deferring keystone exception: __init__() got an unexpected keyword argument 'verify' | 16:21 |
| mordred | awesome! | 16:21 |
| mordred | morgan: why is ksa failing on the verify param now I wonder? | 16:22 |
| morgan | mordred: not sure | 16:22 |
| mordred | dct360: thanks - you have found a bug and I _think_ we have enough info to track it down -although we may ping you again :) | 16:22 |
| morgan | mordred: hmmmm. | 16:24 |
| mordred | morgan: dct360 has the same versions of things installed that I do | 16:24 |
| dct360 | mordred: thanks and as you know i omitted 'verify: False' from my original clouds.yml paste | 16:25 |
| morgan | dct360: shouldn't matter if you have verify or not in this case, that error you're seeing is weird. | 16:25 |
| mordred | oh - you had verify: False in your paste? | 16:25 |
| mordred | wait- did you have it in the auth dict or up one level? | 16:25 |
| morgan | oooh | 16:26 |
| morgan | that could do it | 16:26 |
| mordred | (this is why we need to remove that exception handler ,we can give WAY better errors here) | 16:26 |
| dct360 | mordred: i have it right under project_name: | 16:27 |
| mordred | dct360: awesome - can you try moving it above auth: and indent to the same level as auth and try again? | 16:27 |
| morgan | dct360: indented the same level as project_name? or one level back? | 16:27 |
| dtroyer | morgan, mordred: let's talk about moving the plugin loading next week… maybe its time for occ 2.0 and some breaking down of the functionality so it can be netter composed rather than all-in-one? | 16:27 |
| mordred | dtroyer: +100 | 16:28 |
| morgan | woo | 16:28 |
| dct360 | morgan: same level | 16:28 |
| morgan | dct360: yeah make it the same indentation as the "auth:" bit. | 16:28 |
| morgan | as mordred said | 16:28 |
| dct360 | morgan: ok i'll try that | 16:29 |
| openstackgerrit | Monty Taylor proposed openstack/os-client-config master: Remove the keystoneclient auth fallback https://review.openstack.org/434372 | 16:30 |
| mordred | morgan: ^^ that should work for all non-osc users - main thing is to make sure it also does not break osc in any way | 16:31 |
| morgan | aye | 16:31 |
| mordred | which I think it won't, because I do not think osc uses that codepath | 16:31 |
| dct360 | mordred: morgan: keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authenticaton. (HTTP 401) | 16:31 |
| morgan | that looks more normal. | 16:32 |
| mordred | dct360: yay! that's a whole new thing - and now I'd check that your username/password/auth_url/project_name are all correct | 16:32 |
| dct360 | ok let me check that | 16:33 |
| dct360 | mordred: morgan: it's working now - many Thanks! | 16:41 |
| mordred | dct360: YAY!!! | 16:42 |
| *** abregman has quit IRC | 16:49 | |
| morgan | mordred: so, i think i will be finishing the move to register_uris today | 16:54 |
| morgan | mordred: only another ~60 occurences to fix. | 16:55 |
| mordred | WOOT! | 16:55 |
| morgan | mordred: i'll then make register_uri private and used for auth/discovery and test-case setup. | 16:55 |
| mordred | ++ | 16:55 |
| morgan | and since .register_uris errors if you call it more than once, it forces better habbits | 16:56 |
| morgan | sorry 160 occurences of "register" uri to convert. | 16:57 |
| morgan | mordred: ftr, this has been a slog | 17:02 |
| mordred | morgan: yah. thanks for doing that | 17:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Produce some debug details about nova usage objects https://review.openstack.org/434401 | 17:08 |
| mordred | hopefully that ^^ will help us find the nova usage issue | 17:08 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Try to capture novaclient debug logging https://review.openstack.org/434404 | 17:14 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Force nova microversion to 2.0 https://review.openstack.org/434416 | 17:36 |
| mordred | clarkb: ^^ there is an attempt at setting microversion to 2.0 | 17:41 |
| clarkb | mordred: thinking about the concern for potential for old api to break something a bit more, if you hardcode a newer microversion say the one prior to the breaking one, what is the behavior of novaclient when that version does not exist? does it fail or fall back? | 17:42 |
| mordred | clarkb: it falls back I believe | 17:43 |
| mordred | clarkb: or, more to the point, when I want to start supporting microversions I'll need to request a min and a max iirc | 17:44 |
| mordred | so if I say "I can handle 2.15 to 2.40" and the server only does 2.0 then we'll error | 17:44 |
| clarkb | gotcha | 17:44 |
| mordred | of course, shade will always be able to handle 2.0 | 17:44 |
| clarkb | so maybe the safe way to do this would be to hardcode a min and max now | 17:44 |
| clarkb | min 2.0, max last known good version? | 17:44 |
| clarkb | (that ignores that there may be intermediate versions which also break) | 17:44 |
| mordred | I mostly want to just unbreak the gate for now - so if the 2.0 hardcode works (I mean, half the public clouds only have 2.0 anyway, so it's not like we depend on anyhting later) | 17:45 |
| mordred | and deal with min/max setting as part of restification | 17:46 |
| mordred | (which will basically mean shade setting a max microversion it has branching logic to handle - then we can bump the microversion we can handle as we add branching logic for it where appropriate) | 17:46 |
| mordred | clarkb, Shrews: oh good. it also looks like maybe something changed with the cirros image that is normally loaded into glance? | 17:48 |
| mordred | we're getting AssertionError: 'no sensible image available' is not false | 17:48 |
| clarkb | mordred: I think they are trying to move to qcow2 and stop amiing | 17:48 |
| mordred | I'm so excited about that | 17:49 |
| Shrews | mordred: you get the bestest bugs | 17:50 |
| mordred | it's because we look for -uec as a suffic | 17:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Add image list to debug output on failure https://review.openstack.org/434427 | 17:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Consume cirros qcow2 image if it's there https://review.openstack.org/434428 | 17:52 |
| mordred | there is going to need to be a squash commit | 17:53 |
| mordred | but I'm leaving themall as commits right now so that people can look at them - if we ever actually get the jobs green again | 17:53 |
| *** yfried has joined #openstack-shade | 17:59 | |
| *** jamielennox is now known as jamielennox|away | 18:13 | |
| *** yfried has quit IRC | 18:16 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Consume cirros qcow2 image if it's there https://review.openstack.org/434428 | 18:44 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Consume cirros qcow2 image if it's there https://review.openstack.org/434428 | 18:55 |
| *** iogg has quit IRC | 19:08 | |
| *** yfried has joined #openstack-shade | 19:27 | |
| morgan | mordred: so, ran into another edge case in use of requests mock, surprised it only hit one test case. | 19:31 |
| morgan | mordred: need to pull some magic values out of the response_list | 19:31 |
| morgan | complete_qs = kwargs.pop('complete_qs', False) | 19:31 |
| morgan | request_headers = kwargs.pop('request_headers', {}) | 19:31 |
| morgan | real_http = kwargs.pop('_real_http', False) | 19:31 |
| morgan | those are special cases | 19:31 |
| morgan | though _real_http is probably something i can ignore | 19:31 |
| mordred | morgan: oh fun | 19:32 |
| morgan | yeah need to totally restructure .requests_uri | 19:32 |
| morgan | *facepalm* | 19:32 |
| morgan | erm uri_registry | 19:32 |
| morgan | so, i am wondering if i should make .register_uri just register the stuff in .uri_registry and force the test case to always call .register_uris even if it doesn't pass anything to it? | 19:33 |
| morgan | so we get all the magic response_list collapsing added in and avoid extra matchers. | 19:33 |
| * morgan is overthinking it i think | 19:33 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Fix several concurrent shade gate issues https://review.openstack.org/434428 | 19:49 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Actually normalize nova usage data https://review.openstack.org/434481 | 19:49 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: DNM - Patch to verify 2.0 usage payload behavior https://review.openstack.org/434482 | 19:50 |
| *** cdent has quit IRC | 19:51 | |
| *** jamielennox|away is now known as jamielennox | 20:36 | |
| -openstackstatus- NOTICE: We're currently battling an increase in log volume which isn't leaving sufficient space for new jobs to upload logs and results in POST_FAILURE in those cases; recheck if necessary but keep spurious rebasing and rechecking to a minimum until we're in the clear. | 20:57 | |
| *** yfried has quit IRC | 21:01 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/shade master: Convert use of .register_uri to .register_uris https://review.openstack.org/434516 | 21:04 |
| morgan | mordred: will be attacking test_object separately, but almost done | 21:05 |
| morgan | mordred: and we have 1 code path now for registering URIs vs. ~3 | 21:06 |
| mordred | \o/ | 21:06 |
| mordred | morgan: yah - test object - espeicaly task_upload - is where the dragons lie | 21:07 |
| morgan | well, thankfully it all passes now, and i'm not changing any of that | 21:07 |
| morgan | just making it use .register_uris | 21:07 |
| morgan | vs .register_uri | 21:07 |
| mordred | yup | 21:07 |
| mordred | morgan: isn't it amazing how having working tests can allow for large scale code rework with confidence? | 21:08 |
| mordred | morgan: especially when the mocking is outside of the code boundaries and not of the code api itself? | 21:08 |
| morgan | mordred: i'm telling you now, if i ever have to do this volumn of busy work again to restructure tests, I might drink myself to death | 21:12 |
| morgan | :P | 21:12 |
| mordred | morgan: fwiw, you can drink yourself to death WHILE restructuring tests | 21:14 |
| morgan | mordred: 41 more calls to register_uri to delete. | 21:20 |
| morgan | the countdown is real | 21:21 |
| mordred | morgan: the nice thing too is that I think your work with register_uris really could coalesce into a super nice fixture | 21:22 |
| *** yfried has joined #openstack-shade | 21:23 | |
| morgan | mordred: long term yes | 21:26 |
| mordred | yah. not today | 21:27 |
| morgan | mordred: i found a case you used self.adapter.register_uri instead of self.register_uri as well | 21:47 |
| morgan | now corrected | 21:47 |
| mordred | morgan: that's probably because I'm a bad person | 21:48 |
| morgan | well since the syntax for .register_uris is now massively different... | 21:49 |
| morgan | things like that are found more easily | 21:49 |
| morgan | ugh... now i'm getting a weird timeout | 21:49 |
| morgan | this *just* passed. | 21:49 |
| morgan | wtf. | 21:49 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/shade master: Convert test_object to use .register_uris https://review.openstack.org/434532 | 21:51 |
| morgan | mordred: ^ and phew | 21:51 |
| mordred | morgan: \o/ | 21:53 |
| morgan | mordred: i assume the broken gate things will be resolved at some point | 21:54 |
| morgan | because my brain feels fried. | 21:54 |
| mordred | yes! I has broken gate resolution worked out | 21:54 |
| morgan | cool | 21:54 |
| morgan | also post failures, but that i see is other things | 21:54 |
| *** gouthamr has quit IRC | 22:22 | |
| *** yfried has quit IRC | 22:28 | |
| morgan | mordred: ok i'm taking a break. i feel really fried with those last two patches. | 22:43 |
| morgan | i'll try and circle up on the rest of the removing keystoneclient mocks in the next day or so. | 22:43 |
| mordred | morgan: woot! - and totes understand | 22:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Actually normalize nova usage data https://review.openstack.org/434481 | 22:44 |
| morgan | only ~83 more keystoneclient mocks, unfortuantely these are the more beastly ones to tackle, catalog and role assignment stuff | 22:44 |
| *** gouthamr has joined #openstack-shade | 22:53 | |
| mordred | morgan: indeed. although those are also the ones that may feel more like thinking and less like repetitive drudgery | 22:56 |
| mordred | hopefully | 22:57 |
| morgan | nope, repetative drudgery the whole way down | 23:13 |
| morgan | its like turtles, but less cute | 23:13 |
| mordred | morgan: if you take more heavy hallucinagens, then the repetition becomes cute again - sort of covered in pastel geometric shapes | 23:59 |
| mordred | wait | 23:59 |
| mordred | that's a different conversation | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
