Wednesday, 2015-10-21

jamielennoxlhcheng the problem is knowing everywhere its in use and knowing the code base week enough to fix it. My current plan is to have a review that deletes the old stuff and see what fails00:00
harlowjathen clients can verify the signature themselves, and check the expiry and reuse the tokens ...00:00
*** stevemar_ has joined #openstack-oslo00:00
harlowjadoesn't seem so complicated :-P00:00
lhchengharlowja: we need only the  expiry, keep the service token in cache for some time (expiry - seconds buffer).00:03
harlowjaright00:04
lhchengthen make the client smart enough to refetch the token if its expired00:04
*** stevemar_ has quit IRC00:04
harlowjayup00:04
harlowjashitty part is u need to not have clients pass around tokens in there apis00:05
harlowjau need clients to say do this operation, no token provided, and the internals of the library/client do the token fetching/refetching...00:05
lhchengharlowja: as jamielennox mentioned, the challenge is finding all the code doing this and applying the logic.00:05
harlowjayup00:05
harlowjaits a shitty one to apply post library/client creation00:05
lhchengjamielennox: I wonder if that logic can be built-in into the session or keystoneclient object?00:06
jamielennoxlhcheng it is :) that's largely the point of sessions00:06
lhchengsession only caches the request connection right, not the token?00:07
jamielennoxThats essentially the reason we need a session and not just plug the auth plugins directly into the clients so we can do things like smart reauthentication00:08
jamielennoxWell the plugin caches the token and handles expiry00:08
jamielennoxThe session let's is so things like reauth on 40100:08
jamielennoxUs do00:08
jamielennoxAnd we are trying to make it so that services only ever have to deal with plugins, never touching the token directly00:10
*** krotscheck has quit IRC00:11
lhchengah aside from  auth_plugin managing the credential it does the token caching!00:12
lhchengokay, now it clearer now (the relationship between session and auth_plugin)00:12
lhchengjamielennox: theorically, I can follow the pattern in : https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L211-L25300:13
lhchengand apply it for ironic call00:13
jamielennoxlhcheng you shouldn't need to00:14
jamielennoxOne of the last things in liberty ksc was locking around the token fetch00:15
harlowjapeople pass around a session into every client api call?00:15
jamielennoxSo you should be able to just store the service token globally and reuse across threads00:15
jamielennoxDon't need to do the token_endpoint hack any more00:16
jamielennoxharlowja people create a client with a session, every api call uses that session00:16
jamielennoxPassing for every call would work just fine but it's a much bigger change from existing clients than people would accept00:17
harlowjaright, i guess there is the auth plugin  to clients that should know how to use that session and update it as needed00:17
jamielennoxBut on the plus side, because the clients no longer do their own auth creating client objects is cheap00:18
harlowjacool00:19
jamielennoxClose enough, session uses the plugin - but Same end result00:19
*** krotscheck has joined #openstack-oslo00:19
*** achanda has quit IRC00:23
lhchengjamielennox: ah right, the locking thing on the nova-neutron code can be removed00:24
*** achanda has joined #openstack-oslo00:26
openstackgerritJoshua Harlow proposed openstack/taskflow: Add a executor backed conductor and have existing impl. use it  https://review.openstack.org/17464300:33
*** jecarey has joined #openstack-oslo00:42
openstackgerritJoshua Harlow proposed openstack/taskflow: Add a executor backed conductor and have existing impl. use it  https://review.openstack.org/17464300:44
*** jecarey has quit IRC00:46
openstackgerritJoshua Harlow proposed openstack/taskflow: Use conductor entity class constant instead of raw string  https://review.openstack.org/23783100:48
openstackgerritJoshua Harlow proposed openstack/taskflow: Add a executor backed conductor and have existing impl. use it  https://review.openstack.org/17464300:49
openstackgerritJoshua Harlow proposed openstack/taskflow: Add a executor backed conductor and have existing impl. use it  https://review.openstack.org/17464300:51
openstackgerritJoshua Harlow proposed openstack/taskflow: Use conductor entity class constant instead of raw string  https://review.openstack.org/23783100:52
*** boris-42 has quit IRC00:58
*** kbyrne has quit IRC01:14
*** kbyrne has joined #openstack-oslo01:15
*** itisha has quit IRC01:31
*** salv-orlando has joined #openstack-oslo01:32
*** Fdaisuke has joined #openstack-oslo01:32
*** dims has quit IRC01:34
*** salv-orlando has quit IRC01:46
*** lhcheng has quit IRC01:59
*** stevemar_ has joined #openstack-oslo02:02
*** browne has quit IRC02:14
*** browne has joined #openstack-oslo02:15
*** browne has quit IRC02:18
*** jecarey has joined #openstack-oslo02:19
*** boris-42 has joined #openstack-oslo02:31
*** yamahata has quit IRC02:32
*** achanda has quit IRC02:53
*** achanda has joined #openstack-oslo02:56
*** achanda has quit IRC02:57
*** morgan has quit IRC03:00
*** waynr has quit IRC03:13
*** salv-orlando has joined #openstack-oslo03:45
openstackgerritLin Hua Cheng proposed openstack/taskflow: Move dispatch and consume/abandon to helper function  https://review.openstack.org/23573503:49
openstackgerritLin Hua Cheng proposed openstack/taskflow: Move dispatch and consume/abandon to helper function  https://review.openstack.org/23573503:50
*** salv-orlando has quit IRC03:55
*** rpodolyaka1 has joined #openstack-oslo04:04
*** links has joined #openstack-oslo04:11
*** rpodolyaka1 has quit IRC04:15
*** rpodolyaka1 has joined #openstack-oslo04:18
*** achanda has joined #openstack-oslo04:27
*** rpodolyaka1 has quit IRC04:32
*** jecarey_ has joined #openstack-oslo04:32
*** achanda has quit IRC04:32
*** jecarey has quit IRC04:34
*** achanda has joined #openstack-oslo04:35
*** rpodolyaka1 has joined #openstack-oslo04:36
*** salv-orlando has joined #openstack-oslo05:13
*** openstackgerrit has quit IRC05:16
*** morgan has joined #openstack-oslo05:16
*** openstackgerrit has joined #openstack-oslo05:16
*** yamahata has joined #openstack-oslo05:16
*** salv-orlando has quit IRC05:19
*** stevemar_ has quit IRC05:19
*** stevemar_ has joined #openstack-oslo05:19
*** stevemar_ has quit IRC05:22
*** vks has joined #openstack-oslo05:26
vkshow can we declare multilevel dictionary in config file. Is DictopT support this?05:27
*** rpodolyaka1 has quit IRC05:33
*** rpodolyaka1 has joined #openstack-oslo05:34
vksdhellmann, around?05:34
*** morgan has quit IRC05:45
*** yamahata has quit IRC05:46
*** morgan has joined #openstack-oslo06:01
*** nkrinner has joined #openstack-oslo06:06
*** zqfan_afk has joined #openstack-oslo06:19
*** rpodolyaka1 has quit IRC06:24
*** browne has joined #openstack-oslo06:25
*** e0ne has joined #openstack-oslo06:32
*** jamielennox is now known as jamielennox|away06:37
*** e0ne has quit IRC06:42
*** rpodolyaka1 has joined #openstack-oslo06:45
*** salv-orlando has joined #openstack-oslo06:46
*** e0ne has joined #openstack-oslo06:47
*** vks has quit IRC07:07
*** bnemec has quit IRC07:19
*** bnemec has joined #openstack-oslo07:25
*** e0ne has quit IRC07:27
*** jongchoi has joined #openstack-oslo07:31
*** e0ne has joined #openstack-oslo07:31
*** markus_z has joined #openstack-oslo07:36
*** rpodolyaka1 has quit IRC07:43
*** salv-orlando has quit IRC07:46
*** browne has quit IRC07:51
*** browne has joined #openstack-oslo07:58
*** dims has joined #openstack-oslo07:58
*** browne has quit IRC08:02
*** dims has quit IRC08:05
*** achanda has quit IRC08:10
*** yassine has joined #openstack-oslo08:12
*** achanda has joined #openstack-oslo08:13
*** achanda has quit IRC08:15
*** rpodolyaka1 has joined #openstack-oslo08:17
*** achanda has joined #openstack-oslo08:20
*** ihrachys has joined #openstack-oslo08:26
*** ihrachys has quit IRC08:33
*** e0ne has quit IRC08:37
*** zqfan_afk is now known as zqfan08:37
*** ihrachys has joined #openstack-oslo08:42
openstackgerritMatthew Booth proposed openstack/oslo.messaging: Fix a race calling blocking MessageHandlingServer.start()  https://review.openstack.org/23703808:45
openstackgerritMatthew Booth proposed openstack/oslo.messaging: Fix assumptions in test_server_wait_method  https://review.openstack.org/23703708:45
openstackgerritMatthew Booth proposed openstack/oslo.messaging: Rename MessageHandlingServer._executor for readability  https://review.openstack.org/23703608:45
*** jongchoi has quit IRC08:48
*** achanda has quit IRC08:48
*** links has quit IRC09:00
*** openstackgerrit has quit IRC09:01
*** openstackgerrit has joined #openstack-oslo09:01
*** amotoki has joined #openstack-oslo09:01
mdbooth"By default, the most simple executor is used - the blocking executor."09:02
mdbooth'most simple': heh09:03
*** ihrachys has quit IRC09:03
*** ihrachys has joined #openstack-oslo09:08
*** links has joined #openstack-oslo09:15
*** e0ne has joined #openstack-oslo09:17
openstackgerritKomei Shimamura proposed openstack/oslo.messaging: Add devstack plugin for kafka  https://review.openstack.org/22752209:18
*** openstackgerrit has quit IRC09:31
*** openstackgerrit has joined #openstack-oslo09:31
mdboothsileht: Restarting a MessagingHandlingServer09:32
mdboothLooking at _setup_listener, if we're not going to support restarting a MessageHandlingServer, I don't think anything is reusable there09:34
mdboothSo it's really a question of whether we want to support this. I'm going to need direction on this, as it's a question of design rather than correctness.09:35
mdboothsileht: What's the best way to ensure I do this properly?09:36
*** ihrachys has quit IRC09:36
*** salv-orlando has joined #openstack-oslo09:36
*** exploreshaifali has joined #openstack-oslo09:43
*** ihrachys has joined #openstack-oslo09:44
openstackgerritOleksii Zamiatin proposed openstack/oslo.messaging: Fix target resolution mismatch in neutron, nova, heat  https://review.openstack.org/23272009:54
*** salv-orlando has quit IRC09:59
*** ihrachys has quit IRC10:04
*** ihrachys has joined #openstack-oslo10:05
*** ozamiatin has joined #openstack-oslo10:09
*** jaypipes has joined #openstack-oslo10:26
*** salv-orlando has joined #openstack-oslo10:27
*** flaper87 has quit IRC10:29
openstackgerritMatthew Booth proposed openstack/oslo.messaging: Robustify locking in MessageHandlingServer  https://review.openstack.org/23534710:29
openstackgerritMatthew Booth proposed openstack/oslo.messaging: Fix test_two_pools_three_listener to not restart servers  https://review.openstack.org/23686210:29
*** rpodolyaka1 has quit IRC10:31
*** dims has joined #openstack-oslo10:34
*** flaper87 has joined #openstack-oslo10:35
*** flaper87 has quit IRC10:35
*** flaper87 has joined #openstack-oslo10:35
*** stevemar_ has joined #openstack-oslo10:35
*** achanda has joined #openstack-oslo10:36
openstackgerritMatthew Booth proposed openstack/oslo.messaging: Remove unused event in ServerThreadHelper  https://review.openstack.org/23797710:36
*** achanda has quit IRC10:37
*** stevemar_ has quit IRC10:38
openstackgerritMatthew Booth proposed openstack/oslo.messaging: Trivial locking cleanup in test_listener  https://review.openstack.org/23685910:40
*** salv-orlando has quit IRC10:40
*** links has quit IRC10:41
flaper87dims: hey there!10:42
flaper87dims: https://review.openstack.org/#/c/231039/10:42
flaper87All green now! It'd be great to get those 2 in10:42
flaper87I don't like having kilo things pending10:43
flaper87if you're ok, I can merge them since I'm part of stable-maint core too10:43
dimsflaper87: give me a few mins10:43
flaper87dims: sure! (fwiw, you had reviewed them already)10:43
dimsyep. done10:44
flaper87dims: ++10:44
flaper87thanks10:44
*** links has joined #openstack-oslo10:59
openstackgerritDavanum Srinivas (dims) proposed openstack/oslo.context: Ensure to_dict() supports non-ascii characters  https://review.openstack.org/23776711:01
kashyapdims: Hi11:14
kashyapWhen you're about (not sure you're following the discussion on #openstack-nova).  To not break the conv. I'll ask my question there -)11:14
*** jecarey_ has quit IRC11:29
*** salv-orlando has joined #openstack-oslo11:30
*** e0ne has quit IRC11:37
dimsack kashyap i'll look there11:37
*** exploreshaifali has quit IRC11:38
*** gordc has joined #openstack-oslo11:49
*** rpodolyaka1 has joined #openstack-oslo11:50
*** e0ne has joined #openstack-oslo11:54
*** thumpba has joined #openstack-oslo11:55
*** binarin has quit IRC11:57
*** _amrith_ is now known as amrith12:03
*** binarin has joined #openstack-oslo12:06
*** openstackgerrit has quit IRC12:16
*** openstackgerrit has joined #openstack-oslo12:16
*** jecarey has joined #openstack-oslo12:17
*** itisha has joined #openstack-oslo12:22
*** thumpba has quit IRC12:30
*** ihrachys has quit IRC12:31
*** ihrachys has joined #openstack-oslo12:32
*** ihrachys has quit IRC12:33
*** jecarey has quit IRC12:33
*** jecarey has joined #openstack-oslo12:34
dimsjecarey: ping12:35
*** edmondsw has joined #openstack-oslo12:35
dimsjecarey: would you have a few cycles to help with this? https://review.openstack.org/#/c/237767/12:35
*** ihrachys has joined #openstack-oslo12:37
*** ihrachys has quit IRC12:37
*** ihrachys has joined #openstack-oslo12:37
*** boris-42 has quit IRC12:38
*** jecarey has quit IRC12:38
*** fultonj has joined #openstack-oslo12:42
*** ihrachys has quit IRC12:42
*** kgiusti has joined #openstack-oslo12:46
*** thumpba has joined #openstack-oslo12:47
*** ndipanov has joined #openstack-oslo12:47
*** salv-orlando has quit IRC12:53
*** jecarey has joined #openstack-oslo12:55
*** ihrachys has joined #openstack-oslo13:00
*** jecarey has quit IRC13:02
*** jecarey has joined #openstack-oslo13:03
*** stevemar_ has joined #openstack-oslo13:03
*** stevemar_ has quit IRC13:07
*** salv-orlando has joined #openstack-oslo13:07
*** jecarey has quit IRC13:07
*** stevemar_ has joined #openstack-oslo13:09
*** rlrossit has joined #openstack-oslo13:11
*** stevemar_ has quit IRC13:12
*** ihrachys has quit IRC13:16
*** stevemar_ has joined #openstack-oslo13:17
*** jecarey has joined #openstack-oslo13:21
*** rpodolyaka1 has quit IRC13:24
*** rpodolyaka1 has joined #openstack-oslo13:31
*** dims has quit IRC13:45
*** dims has joined #openstack-oslo13:46
mdboothkgiusti: I have a pyngus question if you're about13:52
kgiustiI am about - and I have a pyngus answer (probably).13:53
mdboothkgiusti: I've run the oslo.messaging tests a ton of times recently, and I always get 2 local failures related to authentication13:53
mdboothThe first part of that is the authentication failure itself13:53
mdboothI haven't looked into that at all, but I expect it's some underlying library change outside python13:54
mdboothOr something13:54
*** stevemar_ has quit IRC13:54
mdboothAnyway, another day13:54
mdboothThe issue I'm looking at right now is why this results in a timeout, not in a failure13:54
mdboothI see in your patch which added pyngus 2.0 support, you've put the following in Connection.sasl_done:13:55
mdbooth# connection failure will be handled later13:55
mdboothThis doesn't appear to be handled later at all13:55
mdboothIn fact, I can't see anything which would handle an authentication failure13:55
* kgiusti looking...13:56
mdboothUnfortunately something from earlier have fallen out of my feeble brain13:57
mdboothHowever, what actually happens is it continues merrily and ends up blocking waiting on input which will obviously never arrive13:57
mdboothUntil the testrunner kills it13:57
kgiustimdbooth: ah - I see.13:57
mdboothSo, sasl authentication may succeed on your setup13:58
mdboothIn order to reproduce, you may have to sabotage it :)13:58
mdboothI always get 2 test failures13:58
mdboothThe one I'm currently looking at is oslo_messaging.tests.test_amqp_driver.TestCyrusAuthentication.test_authentication_default_username13:58
kgiustimdbooth: what the failure message/traceback for that?13:59
*** pradk has joined #openstack-oslo13:59
mdboothkgiusti: It hangs13:59
mdboothI can get a traceback with a ctrl-c13:59
mdbooth...sec13:59
kgiustimdbooth: ugh14:00
mdboothhttp://paste.fedoraproject.org/281919/4543603514:01
mdboothHowever, there are multiple threads in play14:01
mdboothLet me see if I can get you a better one from the debugger14:01
*** zqfan is now known as zqfan_afk14:02
mdboothkgiusti: Ok, so it actually hangs on the select in eventloop.run14:02
mdboothSo the issue seems to be that the connection failed unrecoverably, but nothing signaled this to the eventloop14:03
*** e0ne has quit IRC14:03
kgiustimdbooth: interesting, so it's waiting for data that will not arrive.   The failure detection should've kicked in when the eventloop processed the last bit of I/O14:04
mdboothWhich failure detection?14:04
kgiustimdbooth: can you try something for me?14:04
mdboothI couldn't find any failure detection14:04
kgiustimdbooth: I would expect a reply to arrive on that wait with an error result14:05
*** achanda has joined #openstack-oslo14:05
mdboothsasl_done has been called, btw14:05
kgiustimdbooth: with a non-zero outcome?14:06
mdboothYes14:06
*** sigmavirus24_awa is now known as sigmavirus2414:06
mdboothThe log message is output14:06
*** ozamiatin has quit IRC14:07
*** tristanC has quit IRC14:07
kgiustimdbooth: yeah... hrm.  the negative tests all have timeouts in the send request.14:07
mdboothThat's probably not valid...14:08
kgiustimdbooth: can you re-run that cyrus test with the env var PN_TRACE_FRM=1  ?14:08
kgiustimdbooth: yes I agree14:08
mdboothYup, sec14:08
*** jongchoi has joined #openstack-oslo14:08
*** achanda has quit IRC14:09
mdboothhttp://paste.fedoraproject.org/281925/1445436514:09
kgiustimdbooth: ha - no mech sent.  Ok I can repro that easily.14:10
mdboothCould that also relate to why it fails in the first place, btw?14:10
kgiustimdbooth: but I'm surprised pyngus signals EOS, yet the app never sees the close14:11
kgiustimdbooth: yes definitely14:11
kgiustimdbooth: and I've seen that error before14:11
mdboothkgiusti: Is this related to negotiation? Perhaps no common mech?14:11
kgiustimdbooth: sort of - the client isn't sending _any_ mechs.14:11
* mdbooth is utterly unfamiliar with this code, btw14:11
kgiustimdbooth: it's like the sasl lib doesn't have any configured mechs (?)14:12
mdboothkgiusti: Would that be an OS thing?14:13
mdboothOr configured by the calling app?14:13
mdboothDid I mention I'm utterly unfamiliar with this code? ;)14:13
*** ozamiatin has joined #openstack-oslo14:13
mdboothJust wondering if I should be looking for libraries which aren't even installed, or something in /etc14:13
kgiustimdbooth: the test should create a proper config.  What kind of system are you on?  fedoraish or debian-ish?14:14
mdboothFedora actual14:14
mdboothF2314:14
mdboothIt writes a config file at one point iirc14:15
mdboothYeah, it creates openstack.conf which contains:14:16
mdboothmech_list: DIGEST-MD5 SCRAM-SHA-1 CRAM-MD5 PLAIN14:16
mdboothI don't know if anything ever reads that, though14:16
mdboothI wonder...14:16
mdboothDoes it have any weird permissions?14:16
kgiustimdbooth: fyi: http://paste.fedoraproject.org/281928/54369691/14:16
kgiustimdbooth: that's my local system config14:17
mdboothLooks similar to mine14:17
mdboothAre you a Red Hatter by any chance, btw?14:17
kgiustimdbooth: re: permissions - the config files should be generated using the default mode of the process running the test14:18
*** rpodolyaka1 has quit IRC14:18
kgiustimdbooth: yep.14:18
mdboothSo... straced it14:18
kgiustimdbooth: kewl14:18
mdboothopenstack.conf is written, but never read14:18
*** ozamiatin has quit IRC14:19
* mdbooth wonders what would read it14:19
kgiustimdbooth: really?  Hmmm... should be read by the _broker_ side,  I don't think the client reads that, however14:20
mdboothI ran the test with strace -f -eopen14:20
mdboothIt's only opened once by any process, and that's to write it14:20
kgiustimdbooth: the test broker thread should read that config - it needs the sasldb at least.14:20
kgiustimdbooth: k - I'll try that locally too - see if there is a difference.  Also, can you hack the test setup in test_amqp_driver.py?14:21
mdboothNothing seems to read sasldb either14:21
mdboothSure14:21
kgiustimdbooth: in TestCyrusAuthentication.setUp() - add ANONYMOUS to the default list of mechs (line ~ 384).   Then rerun with that PN_TRACE_FRM=1 evn again.14:22
mdboothkgiusti: Ok, but that list is written to openstack.conf14:23
kgiustimdbooth: btw - are you on latest master?14:23
mdboothAnd I think the underlying issue is that nothing is reading that file14:23
mdboothYeah, latest master14:23
mdboothAh, nm14:24
kgiustimdbooth: yeah, good point.  So what cyrus rpms do you have installed? Just to be clear.14:24
mdboothIt's also passed as an arg to FakeBroken14:24
kgiustimdbooth: what?14:24
mdboothRunning14:24
mdboothkgiusti: With ANONYMOUS that runs successfully14:24
mdboothopenstack.conf might be a red herring14:24
kgiustimdbooth: does it now read that file (or sasldb?)14:25
mdboothSec, will strace it again14:25
mdboothopenstack.conf still not read14:25
mdboothSame for sasldb14:25
kgiustimdbooth: hrm.  Can you post the PN_TRACE_FRM=1 from the ANONYMOUS run?14:26
mdboothhttp://paste.fedoraproject.org/281936/4376271414:27
kgiustimdbooth: btw - I haven't tested this stuff on F23 - only up to F22.   I should try F23 locally - you may be seeing something related to the new system.14:27
mdboothcyrus-sasl-scram-2.1.26-25.2.fc23.x86_6414:27
mdboothcyrus-sasl-2.1.26-25.2.fc23.x86_6414:28
mdboothcyrus-sasl-plain-2.1.26-25.2.fc23.x86_6414:28
mdboothcyrus-sasl-lib-2.1.26-25.2.fc23.x86_6414:28
mdboothcyrus-sasl-gssapi-2.1.26-25.2.fc23.x86_6414:28
mdboothcyrus-sasl-md5-2.1.26-25.2.fc23.x86_6414:28
mdboothpython-saslwrapper-0.16-11.fc23.x86_6414:28
mdboothsaslwrapper-0.16-11.fc23.x86_6414:28
*** jongchoi has quit IRC14:28
mdboothkgiusti: Do you want me to raise a bug for this, btw?14:29
kgiustimdbooth: yes, let's do that.14:29
mdboothI'll raise 214:29
mdboothBut I think the hang is more serious14:29
kgiustimdbooth: yes definitely.   I need to address that ASAP14:30
kgiustimdbooth: hey can you "rpm -qa | grep "*proton*" for me?  Curious.14:30
mdboothNothing14:31
mdboothHowever, proton seemed to be available in tox's venv14:31
mdboothVersion 0.10 iirc14:31
kgiustimdbooth: bingo.  Try installing cyrus-sasl-devel, then blow away your tox env and rebuild it.14:31
mdboothI'll also have to delete the pyngus wheel from my pip cache :)14:32
mdboothsec14:32
mdboothOk, deleted pyngus and python_qpid_proton wheels14:33
kgiustimdbooth: I think tox is pulling down the proton bits via pypi.14:33
mdboothBut it compiles locally, no?14:34
kgiustimdbooth: if the proton c libs are not installed, the python proton setup.py will pull down the proton/c sources from apache and build them locally.14:34
kgiustimdbooth: but if the cyrus dev libs are _not_ available, proton won't be able to use the cyrus system.14:35
kgiustimdbooth: that's another bug in the test if that is the case.14:35
kgiustimdbooth: the test should skip cyrus tests if proton hasn't been built with the proper libs - I can fix that.14:36
*** zz_dimtruck is now known as dimtruck14:38
*** ihrachys has joined #openstack-oslo14:40
mdboothkgiusti: So, I just blew away all my local pip state (I use devpi-server and wheelhouse) and rebuilt14:44
*** rpodolyaka1 has joined #openstack-oslo14:44
mdboothpython_qpid_proton comes back, but I didn't notice it being compiled14:45
mdboothDoes it download binaries?14:45
* mdbooth would find that odd14:45
* kgiusti would too14:45
mdboothAnyway, it contains a single .so: _cproton.so14:45
kgiustimdbooth: no - it should pull down a tarball from apache14:45
mdboothAnd that is *not* linked directly to a sasl library14:46
mdboothlibssl, libcrypto, libgssapi_krb5, libkrb5, libk5crypto and libkrb5support14:46
mdboothBut no sasl14:46
kgiustimdbooth: no libsasl2.so?14:47
mdboothNo14:47
kgiustimdbooth: and the cyrus sasl dev libs were installed prior to the download?14:47
mdboothYup14:48
mdboothI might try to compile it manually, see what's going on14:48
kgiustimdbooth: pop into a venv and pip install pyngus14:48
mdboothYeah, it just installs the version from the local wheel14:49
mdboothwhich was presumably just built locally without my noticing14:50
*** links has quit IRC14:50
krotscheckdims, dhellmann: Question - I'm running into a lot of requests from the various services team to see functional tests for the oslo cors middleware. sdague proposed that it might be easiest if we add a generic devstack run to oslo_middleware to do that. There already seems to be a neutron run - is this something we can reuse and generify?14:50
*** stevemar_ has joined #openstack-oslo14:55
*** stevemar_ has quit IRC14:56
*** e0ne has joined #openstack-oslo14:56
*** nkrinner has quit IRC14:57
kgiustimdbooth: if you pull down the proton src http://qpid.apache.org/releases/qpid-proton-0.10/   goto the proton-c/bindings/python dir and try python setup.py build14:58
mdboothkgiusti: Just doing a little git archaeology. Looks like that timeout has been there in the failure case since your first patch in change Ib9279ed40b0f4cff62e1c742069c8f49f562565914:58
*** stevemar_ has joined #openstack-oslo14:59
kgiustimdbooth: I'm a sad panda14:59
mdboothSo I guess if the hang is real, nobody's hitting it hard enough to complain about it :)14:59
kgiustimdbooth: welcome to my nightmare.14:59
kgiustimdbooth: yeah, so here's the problem: a send (or notify) is queued up to pyngus for sending and the connection establishment is kicked off...15:00
*** ihrachys has quit IRC15:00
*** rpodolyaka1 has quit IRC15:01
* mdbooth is anticipating a scheduling-related deadlock...15:01
*** alejandrito has joined #openstack-oslo15:01
kgiustimdbooth: the connection fails to connect due to auth error.  At that point, the o.m. driver attempt to failover/reconnect in a (misguided) attempt to connection to other hosts which may have proper auth.15:01
*** alejandrito has quit IRC15:02
kgiustimdbooth: the pending sends are held on the queue until the connection is established or they time out.15:02
*** alejandrito has joined #openstack-oslo15:02
kgiustimdbooth: which is very, very bad.15:02
kgiustimdbooth: deadlock?  uh oh.  Where?15:02
mdboothkgiusti: Nah, just guessing. I've been having fun with deadlocks involving manual scheduling over the last few days15:03
kgiustimdbooth: cripes don't do that!  You've scared me enough for one day :)15:03
mdboothSorry :)15:03
*** f13o has joined #openstack-oslo15:03
kgiustimdbooth: np.  So what the driver _should_ do is cycle through all configured hosts and if none pass auth abort all pending requests.15:04
kgiustimdbooth: operator intervention is needed at that point.  probably.15:04
mdboothWell, I'm happy that I correctly called it when I decided I was out of my depth :)15:05
kgiustimdbooth: might me out of mine, too :)15:05
kgiustimdbooth: this was a big hole that the unit tests should've caught.  bad kenny.15:05
kgiustimdbooth: no soup for me.15:05
kgiustimdbooth: but the other problem - no cyrus lib linked in - that's something entirely different.15:06
kgiustimdbooth: hopefully easier to solve.15:07
mdboothYeah. Just writing up the hang bug now I've decided how to describe it.15:08
kgiustimdbooth: when you setup.py build proton-c/bindings/python - do you see the file "cyrus_sasl.c" included in the build, or "none_sasl.c"?15:10
mdboothNot got there, yet15:10
*** regXboi has joined #openstack-oslo15:12
*** ihrachys has joined #openstack-oslo15:12
mdboothhttps://bugs.launchpad.net/oslo.messaging/+bug/150851215:12
openstackLaunchpad bug 1508512 in oslo.messaging "oslo.messaging hangs if SASL authentication fails" [Undecided,New]15:12
*** stevema__ has joined #openstack-oslo15:13
*** mhorban_ has joined #openstack-oslo15:14
mdboothdims: If you've got a sec, I've got a patch with +2s blocked waiting on 2 trivial patches with only a single +215:15
dimsmdbooth: which one?15:15
mdboothTrivial patches are: https://review.openstack.org/#/c/237036/ and https://review.openstack.org/#/c/237037/15:15
mdboothThis one has 2 +2s: https://review.openstack.org/#/c/237038/15:16
dimsmdbooth: ack will be a few mins15:16
mdbooth(Thanks sileht and harlowja)15:16
*** stevemar_ has quit IRC15:16
*** Kiall has quit IRC15:19
*** ihrachys has quit IRC15:20
*** ihrachys_ has joined #openstack-oslo15:20
*** Kiall has joined #openstack-oslo15:20
openstackgerritTom Cammann proposed openstack/oslo.config: Allow naming of custom types  https://review.openstack.org/22139415:21
openstackgerritCyril Roelandt proposed openstack/oslo.messaging: Use yaml.safe_load instead of yaml.load  https://review.openstack.org/23813515:22
*** ihrachys_ has quit IRC15:23
*** ihrachys has joined #openstack-oslo15:23
*** ihrachys has quit IRC15:25
mdboothkgiusti: So, I *think* the issue is that somebody has uploaded a wheel for python-qpid-proton which doesn't contain the compiled bits15:28
mdboothBecause I have wheels configured15:28
mdboothI *think* I'm getting the pure python version by default15:28
mdboothI haven't confirmed this, but when I explicitly tell it not to use an existing wheel it links with sasl15:29
*** ihrachys has joined #openstack-oslo15:31
kgiustimdbooth: interesting - where is that wheel coming from, pypi?15:31
*** dims_ has joined #openstack-oslo15:31
mdboothkgiusti: I have read allusions to the existence of a suppository of wheels15:31
mdboothHowever, I have not confirmed its existence15:32
mdboothIt's mysterious, dark, and vaguely smelly15:33
kgiustimdbooth: trust no one...15:33
mdboothThe local thing is completely awesome, though15:33
* mdbooth caches everything15:33
kgiustimdbooth: I'm going to open another bug against the tests to skip the cyrus tests if the underlying proton lib hasn't been linked with cyrus.15:34
mdboothkgiusti: I'm literally just doing that15:35
kgiustimdbooth: thanks!15:35
*** dims has quit IRC15:35
mdboothI can confirm that, with the correctly linked python_qpid_proton, the tests all run successfully :)15:35
mdboothkgiusti: Thank you!15:35
kgiustimdbooth: np - I'm just glad other folks are finally banging on this driver.15:37
mdboothkgiusti: https://bugs.launchpad.net/oslo.messaging/+bug/150852315:38
openstackLaunchpad bug 1508523 in oslo.messaging "TestCyrusAuthentication runs even when python_qpid_proton is not linked against SASL libraries" [Undecided,New]15:38
mdboothdims_: Incidentally, sileht is concerned about removing the ability to restart servers, which he raises in this review: https://review.openstack.org/#/c/236862/215:39
mdboothThis is a reasonable concern, but I don't know how to proceed15:40
mdboothThe only thing I can think of it raising a spec and having the discussion in that review15:40
mdboothFrom my pov, I can restart or not15:41
mdboothMy first patch supported restarts, but I removed it after discussion15:41
dims_mdbooth: i'd go with sileht's view as he has been taking care of this stuff for a while now15:48
*** ndipanov has quit IRC15:52
*** markus_z has quit IRC15:52
*** ihrachys has quit IRC15:53
*** ihrachys has joined #openstack-oslo15:53
mdboothdims_: Ok, I'll put it back in.15:53
*** ozamiatin has joined #openstack-oslo15:54
*** yamahata has joined #openstack-oslo15:55
*** binarin has quit IRC15:56
*** sputnik13 has joined #openstack-oslo15:57
*** rlrossit has quit IRC15:59
*** ozamiatin has quit IRC15:59
*** stevema__ has quit IRC16:00
*** jecarey_ has joined #openstack-oslo16:00
*** jecarey_ has quit IRC16:02
*** jecarey_ has joined #openstack-oslo16:03
*** jecarey has quit IRC16:03
*** ihrachys has quit IRC16:04
*** exploreshaifali has joined #openstack-oslo16:05
*** rlrossit has joined #openstack-oslo16:10
*** bnemec has quit IRC16:10
*** yamahata has quit IRC16:11
*** yamahata has joined #openstack-oslo16:11
*** ihrachys has joined #openstack-oslo16:16
dims_folks, i've intiailized the etherpads, please take a look and add/update as needed - https://wiki.openstack.org/wiki/Design_Summit/Mitaka/Etherpads#Oslo16:16
*** e0ne has quit IRC16:16
*** e0ne has joined #openstack-oslo16:17
*** bnemec has joined #openstack-oslo16:17
*** achanda has joined #openstack-oslo16:28
*** jecarey_ has quit IRC16:31
*** jecarey_ has joined #openstack-oslo16:34
openstackgerritMerged openstack/oslo.messaging: Rename MessageHandlingServer._executor for readability  https://review.openstack.org/23703616:36
*** dims_ has quit IRC16:36
openstackgerritMerged openstack/oslo.messaging: Fix assumptions in test_server_wait_method  https://review.openstack.org/23703716:36
openstackgerritMerged openstack/oslo.messaging: Fix a race calling blocking MessageHandlingServer.start()  https://review.openstack.org/23703816:36
*** stevemar_ has joined #openstack-oslo16:36
*** dims has joined #openstack-oslo16:36
*** exploreshaifali has quit IRC16:38
*** stevemar_ has quit IRC16:41
*** amotoki has quit IRC16:42
*** ihrachys has quit IRC16:47
*** ihrachys has joined #openstack-oslo16:47
*** zigo_ has quit IRC16:59
*** zigo has joined #openstack-oslo17:00
*** sputnik13 has quit IRC17:11
*** sputnik13 has joined #openstack-oslo17:12
*** nikhil_k has joined #openstack-oslo17:15
*** nikhil has quit IRC17:15
*** fultonj has left #openstack-oslo17:16
*** salv-orl_ has joined #openstack-oslo17:18
*** ihrachys has quit IRC17:19
*** salv-orlando has quit IRC17:21
*** pasquier-s has quit IRC17:25
*** pasquier-s has joined #openstack-oslo17:27
*** sputnik13 has quit IRC17:27
*** tchaypo has quit IRC17:29
*** tchaypo has joined #openstack-oslo17:31
*** stevemar_ has joined #openstack-oslo17:32
*** ihrachys has joined #openstack-oslo17:33
*** sputnik13 has joined #openstack-oslo17:34
*** exploreshaifali has joined #openstack-oslo17:40
*** zqfan_afk is now known as zqfan17:42
*** amotoki has joined #openstack-oslo17:43
*** salv-orl_ has quit IRC17:45
*** rlrossit has quit IRC17:46
*** amotoki has quit IRC17:48
*** e0ne has quit IRC17:49
*** rlrossit has joined #openstack-oslo17:50
*** exploreshaifali has quit IRC17:53
*** harlowja has quit IRC17:57
*** stevemar_ has quit IRC17:58
*** harlowja has joined #openstack-oslo18:02
*** shardy is now known as shardy_afk18:02
*** dims has quit IRC18:03
*** dims has joined #openstack-oslo18:04
*** yamahata has quit IRC18:08
*** jecarey_ has quit IRC18:19
*** ihrachys has quit IRC18:19
*** jecarey has joined #openstack-oslo18:20
*** ihrachys has joined #openstack-oslo18:21
*** jecarey_ has joined #openstack-oslo18:23
*** jecarey has quit IRC18:24
*** salv-orlando has joined #openstack-oslo18:25
*** exploreshaifali has joined #openstack-oslo18:31
*** zqfan is now known as zqfan_afk18:31
*** thumpba has quit IRC18:34
*** amotoki has joined #openstack-oslo18:44
*** ihrachys has quit IRC18:47
*** amotoki has quit IRC18:49
*** ihrachys has joined #openstack-oslo18:52
*** sputnik13 has quit IRC18:53
*** sputnik13 has joined #openstack-oslo18:55
*** yamahata has joined #openstack-oslo18:57
*** gordc has quit IRC18:57
*** stevemar_ has joined #openstack-oslo19:07
*** yassine has quit IRC19:12
*** stevemar_ has quit IRC19:15
*** stevemar_ has joined #openstack-oslo19:16
*** achanda has quit IRC19:16
*** ihrachys has quit IRC19:18
*** stevemar_ has quit IRC19:20
*** ihrachys has joined #openstack-oslo19:25
*** shardy_afk is now known as shardy19:32
*** thumpba has joined #openstack-oslo19:35
*** thumpba has quit IRC19:41
*** ihrachys has quit IRC19:42
*** harlowja has quit IRC19:50
*** openstackstatus has joined #openstack-oslo20:07
*** ChanServ sets mode: +v openstackstatus20:07
*** dimtruck is now known as zz_dimtruck20:11
*** browne has joined #openstack-oslo20:15
*** jecarey_ has quit IRC20:15
*** jecarey has joined #openstack-oslo20:16
*** achanda has joined #openstack-oslo20:16
*** dims has quit IRC20:18
*** jaypipes has quit IRC20:18
*** dims has joined #openstack-oslo20:18
*** zz_dimtruck is now known as dimtruck20:19
*** jecarey has quit IRC20:20
*** achanda has quit IRC20:22
*** salv-orlando has quit IRC20:22
*** agentrooker has joined #openstack-oslo20:26
*** agentrooker has left #openstack-oslo20:26
*** jecarey has joined #openstack-oslo20:31
*** harlowja has joined #openstack-oslo20:39
*** harlowja_ has joined #openstack-oslo20:40
*** alejandrito has quit IRC20:42
*** achanda has joined #openstack-oslo20:42
*** browne has quit IRC20:44
*** harlowja has quit IRC20:44
*** amotoki has joined #openstack-oslo20:45
*** achanda has quit IRC20:46
*** salv-orlando has joined #openstack-oslo20:50
*** shardy has quit IRC20:54
*** salv-orlando has quit IRC20:55
*** achanda has joined #openstack-oslo20:57
*** rlrossit1 has joined #openstack-oslo21:05
*** rlrossit has quit IRC21:06
*** achanda has quit IRC21:10
*** stevemar_ has joined #openstack-oslo21:11
*** jongchoi_ has joined #openstack-oslo21:11
*** regXboi has quit IRC21:20
*** rlrossit1 has left #openstack-oslo21:22
*** kgiusti has quit IRC21:28
*** amotoki has quit IRC21:31
*** pballand has quit IRC21:44
*** abitha has joined #openstack-oslo21:45
*** avarner has quit IRC21:51
*** jongchoi_ has quit IRC22:02
*** ozamiatin has joined #openstack-oslo22:02
*** jongchoi_ has joined #openstack-oslo22:05
*** jamielennox|away is now known as jamielennox22:07
*** thumpba has joined #openstack-oslo22:07
*** jongchoi_ has quit IRC22:10
*** pradk has quit IRC22:10
*** salv-orlando has joined #openstack-oslo22:22
*** dimtruck is now known as zz_dimtruck22:35
*** sigmavirus24 is now known as sigmavirus24_awa22:35
*** jecarey has quit IRC22:36
*** thumpba has quit IRC22:38
*** salv-orlando has quit IRC22:40
*** thumpba_ has joined #openstack-oslo22:42
*** thumpba_ has quit IRC22:44
*** thumpba has joined #openstack-oslo22:48
*** otherwiseguy has quit IRC22:53
*** thumpba has quit IRC22:55
*** thumpba has joined #openstack-oslo22:58
openstackgerritJoshua Harlow proposed openstack/taskflow: Add a executor backed conductor and have existing impl. use it  https://review.openstack.org/17464323:02
openstackgerritJoshua Harlow proposed openstack/taskflow: Add a executor backed conductor and have existing impl. use it  https://review.openstack.org/17464323:04
*** thumpba has quit IRC23:04
*** pballand has joined #openstack-oslo23:05
*** pballand has quit IRC23:09
*** pballand has joined #openstack-oslo23:11
*** sheeprine has quit IRC23:11
*** otherwiseguy has joined #openstack-oslo23:11
*** sheeprine has joined #openstack-oslo23:13
openstackgerritJoshua Harlow proposed openstack/futurist: Allow for providing different run work sync functions  https://review.openstack.org/23827223:18
harlowja_mdbooth ^ might be helpful also23:19
*** sheeprine has quit IRC23:26
*** amotoki has joined #openstack-oslo23:28
*** dims_ has joined #openstack-oslo23:29
*** sheeprine has joined #openstack-oslo23:31
*** dims__ has joined #openstack-oslo23:31
*** dims has quit IRC23:31
*** amotoki has quit IRC23:33
*** harlowja_ has quit IRC23:33
*** dims_ has quit IRC23:34
*** harlowja has joined #openstack-oslo23:37
*** salv-orlando has joined #openstack-oslo23:37
openstackgerritJoshua Harlow proposed openstack/futurist: Allow lazy sync futures  https://review.openstack.org/23827423:38
harlowjamdbooth other idea is ^23:38
*** exploreshaifali has quit IRC23:38
*** harlowja_ has joined #openstack-oslo23:45
*** harlowja has quit IRC23:49
openstackgerritJoshua Harlow proposed openstack/futurist: Allow lazy sync futures  https://review.openstack.org/23827423:52
harlowja_mdbooth ok ^ should work (vs not, ha)23:52
*** sheeprine has quit IRC23:54
harlowja_actually that still has some issues, hmmm23:56
*** sheeprine has joined #openstack-oslo23:57
*** salv-orlando has quit IRC23:57

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!