Thursday, 2017-03-30

*** ddeja has quit IRC00:16
*** ddeja has joined #openstack-mistral00:20
*** thrash is now known as thrash|g0ne00:26
*** jkilpatr has quit IRC00:44
*** ddeja has quit IRC00:47
*** ddeja has joined #openstack-mistral00:48
*** jkilpatr has joined #openstack-mistral00:56
*** jkilpatr has quit IRC01:01
*** jkilpatr has joined #openstack-mistral01:06
*** jkilpatr has quit IRC01:16
*** jkilpatr has joined #openstack-mistral01:17
*** chlong has joined #openstack-mistral01:47
*** jkilpatr has quit IRC02:19
*** jkilpatr has joined #openstack-mistral02:19
*** jkilpatr has quit IRC02:36
*** jkilpatr has joined #openstack-mistral02:41
*** gongysh has joined #openstack-mistral04:33
*** gongysh has quit IRC05:01
*** dmellado has quit IRC05:14
*** thrash|g0ne has quit IRC05:14
*** chlong has quit IRC05:14
*** toure has quit IRC05:15
*** toure has joined #openstack-mistral05:16
*** dmellado has joined #openstack-mistral05:16
*** thrash has joined #openstack-mistral05:18
*** thrash has quit IRC05:18
*** thrash has joined #openstack-mistral05:18
*** sharatss has joined #openstack-mistral05:40
*** gongysh has joined #openstack-mistral06:04
*** zhurong has joined #openstack-mistral06:15
rakhmerovhi, does anybody know if there are some issues with CI?06:23
rakhmerovall our patches started failing unexpectedly06:23
rakhmerovdifferent tests with different errors06:23
*** gongysh has quit IRC06:36
*** openstackgerrit has joined #openstack-mistral06:37
openstackgerritIstvan Imre proposed openstack/mistral master: Dynamic workflow name evaluation.  https://review.openstack.org/43410606:37
*** zhurong has quit IRC06:38
openstackgerritIstvan Imre proposed openstack/mistral master: Dynamic workflow name evaluation.  https://review.openstack.org/43410606:38
*** tuan_ has joined #openstack-mistral06:51
tuan_Morning folks06:52
*** mgershen has quit IRC06:53
*** mgershen has joined #openstack-mistral06:57
rakhmerovtuan_: hi06:57
*** d0ugal has joined #openstack-mistral06:58
tuan_rakhmerov: Hi There07:00
tuan_:)07:00
rakhmerovd0ugal: good morning Dougal, seems like our CI is down07:01
rakhmerovtrying to figure out why07:01
tuan_Hi guys, if you have time, could you take a look to the refresh token expired of other clients (nova, neutron, etc.) in mistral07:01
d0ugalrakhmerov: morning07:06
d0ugaldang07:06
* d0ugal looks07:07
rakhmerovd0ugal: all of a sudden all patches coming to review started failing07:09
rakhmerova bunch of tests07:09
rakhmerovI have no idea why yet07:09
*** sharatss has quit IRC07:09
*** sharatss has joined #openstack-mistral07:10
d0ugalrakhmerov: yeah, very strange.07:10
rakhmerovI run same tests locally, they pass07:10
d0ugaloh, even more strange!07:10
d0ugalI just started them locally07:10
d0ugal32 failures in CI!07:12
d0ugalrakhmerov: They fail for me07:14
d0ugalrakhmerov: I guess your tox env is old07:14
d0ugal$ tox -r07:14
d0ugaland if they then start to fail for you, that means it is a issue with one of our requirements.07:14
*** shardy has joined #openstack-mistral07:14
d0ugalrakhmerov: so wait, don't run tox -r fist07:14
rakhmerovooh, hm..07:15
d0ugalrun $ .tox/py27/bin/pip freeze > saveit.txt07:15
d0ugalthen re-create the tox env07:15
d0ugalthen we have a starting point for comparison07:15
rakhmerovyeah, ok07:15
d0ugalTests have still not finished for me, but I have seen lots of errors so far07:16
rakhmerovyes07:17
d0ugalhah, I actually got 50 failues for py3407:18
rakhmerovaha07:18
therved0ugal, rakhmerov: Looks like SQLAlchemy07:19
d0ugalorly07:19
d0ugaltherve: has there been a recent sqlalchemy release? I don't see one.07:19
d0ugaloh, yes07:20
d0ugal1.1.7, 3 days ago07:20
therveWe were blocked on < 1.1 until recently too07:20
rakhmerovok07:20
d0ugalYeah, 1.1 was allowed here.07:22
d0ugalhttps://github.com/openstack/mistral/commit/95cf6bcaef88892548c050825b8d8d5a657ab12607:22
d0ugalI am just running the tests again with that reverted07:22
rakhmerovok, good07:22
d0ugalugh, for some reason pinning it in the requirements isn't working07:26
d0ugalI keep getting a newer version in the venv07:27
rakhmerovhow come? Do you just change requirements.txt and run tox again?07:28
rakhmerovd0ugal: ^07:28
d0ugalrakhmerov: yeah, I did - not sure why it wasn't picked up.07:30
d0ugal./.tox/py27/bin/pip install SQLAlchemy==1.0.1707:30
d0ugalI just used that to downgrade - running tests now :-D07:30
rakhmerovok, yes07:30
d0ugalI am still seeing errors07:31
rakhmerovI found a ML thread about SQLAlchemy 1.1, reading..07:31
d0ugalWaiting to see if it is the same number.07:31
rakhmerovI updated my env, same thing, a bunch of errors07:32
openstackgerritDougal Matthews proposed openstack/mistral master: Revert "Updated from global requirements"  https://review.openstack.org/45168107:35
d0ugalThere seem to be less errors with a downgraded sqlalchemy07:35
d0ugalbut there are still errors.07:35
rakhmerovhow many?07:37
d0ugalpy27 3 errors, py35 18 errors07:37
*** zhurong has joined #openstack-mistral07:38
rakhmerovhm...07:38
d0ugalpy35 had 50 failures without it07:40
d0ugalnot sure about py2707:41
d0ugalbrb07:43
rakhmerovok07:43
rakhmerovd0ugal: revert patch also failed with the same errors07:48
*** jpich has joined #openstack-mistral07:48
rakhmerovseemingly there's issues with yaql but I'm not sure yet..07:48
d0ugalrakhmerov: the revert didn't work.07:55
d0ugalhttp://logs.openstack.org/81/451681/1/check/gate-mistral-python27-ubuntu-xenial/26587bb/console.html#_2017-03-30_07_46_43_32568507:55
rakhmerovyeah, I know )07:55
rakhmerovI see issues with yaql07:55
rakhmerovdon't know yet if it's a yaql problem though07:55
d0ugalbut I wonder why reverting that doesn't change the sqlalchemy version?07:55
d0ugal:/07:55
rakhmerovooh, same version was installed for the tests?07:56
rakhmerov1.1.0?07:56
d0ugalyeah, see the link ^07:56
d0ugal1.1.707:56
rakhmerovd0ugal: https://github.com/openstack/requirements/blob/master/upper-constraints-xfails.txt07:57
rakhmerovhttps://github.com/openstack/requirements/blob/master/upper-constraints.txt07:58
d0ugalhuh, what does that file do?07:58
rakhmerovd0ugal: I don't know really..07:58
*** amoralej|off is now known as amoralej07:58
d0ugalit even lists mistral!07:58
rakhmerovyes07:58
rakhmerovI believe 1.1.7 gets installed because of upper-constraints.txt07:58
d0ugalI see07:59
rakhmerovit pins the version07:59
rakhmerovd0ugal: so seems like someone knows more than us about our failures :)08:00
d0ugalThis patch passed with 1.1.7 https://review.openstack.org/#/c/420687/08:00
rakhmerovI'll probably need to read that email thread to the end08:00
d0ugalhttps://github.com/openstack/requirements/commit/141abfb14d0f6f2a4d69cc9d3c1d0dd6eab312d308:00
d0ugalThe link in that commit might be relevant?08:01
rakhmerovwhich link?08:01
rakhmerovd0ugal: no, that commit installed SQLAlchemy==1.0.1708:02
rakhmerovhttp://logs.openstack.org/87/420687/3/check/gate-mistral-python35/f05cc31/console.html#_2017-03-06_11_00_52_72844808:03
*** openstackgerrit has quit IRC08:03
d0ugaloops, I read the wrong tab08:03
rakhmerovit's weird because I see a problem locally which doesn't seem to be related with SQLAlchemy..08:05
rakhmerovbrb08:05
sharatssd0ugal, rakhmerov hi08:09
d0ugalsharatss: hi08:10
sharatssd0ugal, as of now we are mapping the actions from respective clients of other projects and executing the actions right?08:11
d0ugalyeah08:11
sharatssd0ugal, what about the projects who have deprecated their cli and moved to openstackclient08:12
sharatss??08:12
d0ugalsharatss: hah, that is a good question :)08:12
d0ugalsharatss: They still have Python bindings right, they only deprecate the CLI I think08:12
sharatssd0ugal, but there will be no updates on the pythonclients08:14
d0ugalreally?08:14
d0ugalsharatss: which projects have said they wont do that?08:15
sharatssd0ugal, in neutron we had raised a BP for CLI and we were asked to move it to OSC since they were migrating everything to there08:15
sharatssd0ugal, projects like senlin and congress have completely deprecated the CLI08:16
d0ugalsharatss: sure, but have they deprecated their Python bindings?08:16
d0ugalThey are different things.08:16
sharatssd0ugal, i was thinking why dont we map actions directly from OSC?08:17
d0ugalsharatss: is OSC a library?08:18
d0ugalI thought it was just a CLI08:18
sharatssd0ugal, i think they still have the bindings as u said08:20
rakhmerovsharatss, d0ugal: I believe bindings won't go away any time soon08:20
rakhmerovotherwise, show the evidence..08:21
rakhmerovall projects currently have OSC plugins, but only plugins08:21
rakhmerovsharatss: and, as we've discussed lately, we'll be refactoring all actions anyway08:21
rakhmerovhopefully soon08:21
rakhmerovwe may consider to use OSC somehow08:22
thervesharatss, d0ugal: neutron uses openstacksdk for bindings nowadays08:23
therve(Somewhat)08:23
d0ugalopenstacksdk would make more sense I guess08:23
therved0ugal, Not sure I agree, because it's yet another bottleneck of reviews08:23
d0ugaltrue08:24
therveIf you use it as a library to build your client, ok08:24
therveBut neutron seems to stuff everything in there, doesn't make much sense to me08:24
d0ugallol08:24
therve(Especially since they actually don't put everything and you still have to use neutronclient anyway)08:24
*** gongysh has joined #openstack-mistral08:27
rakhmerovtherve: we've discussed this before with d0ugal. To me it seems like OSC is kind of limited because it forces to follow a certain structure of commands whereas many projects may choose do more. So, as far as I understand it, we can have a number of "typical" commands in CLI but those who want more advanced capabilities we'll still need to use our own client08:29
therverakhmerov, What do you mean by "more" ?08:29
therveOSC seems fairly flexible to me08:29
d0ugalmany projects so just provide OSC integration with their python client08:30
rakhmerovtherve: something that doesn't fit into "osc object verb params" structure08:30
d0ugallike mistral :)08:30
therverakhmerov, Right, but what does fit? :)08:30
rakhmerovmaking it up now... "osc workflow create my_wf" seems ok for OSC, but "osc analyze wf=my_wf filter=.." does not08:32
rakhmerovnot sure if I'm clear on that..08:32
rakhmerovmaybe we're talking about different things08:32
d0ugallol08:33
d0ugalI think most things can fit in with OSX08:33
d0ugalOSC08:33
rakhmerovmaybe08:33
* d0ugal says, having abused OSC a few times08:33
rakhmerovwe'll see08:33
rakhmerovmy point here is the following: we may come up with something very cool that it will be problematic to squeeze into OSC concepts08:34
d0ugalthat maybe sounds like a problem :)08:35
rakhmerovgenerally, standardization is OK unless it kills creativeness08:35
rakhmerov:)08:35
rakhmerovd0ugal: what sounds like a problem?08:36
d0ugalhaha, I was just joking08:36
d0ugalnevermind :)08:36
therverakhmerov, "osc workflow analyze" works though?08:36
rakhmerovtherve: yeah, but there may not be any "object", for example08:37
rakhmerovnot sure how it's addressed in OSC now08:37
rakhmerovor a command may work with different object types08:37
rakhmerovor we'll want to specify params in some tricky non-standard way08:38
therveWell, that last thing is not nice for your users08:38
therveAnd "object" can be "namespace" when appropriate08:38
rakhmerovtherve: anyway, I have to admit that I'm not well familiar with OSC now so my opinion is not a strong opinion (yet)08:39
rakhmerovthat's ok08:39
d0ugallol08:39
therverakhmerov, Yeah, I'm changing my opinion over time as well. The plugin interface works ok08:39
rakhmerovI'll be much more interesting person to talk to about it once I'll have a closer look at OSC :)08:39
therveIf we didn't have the overlap in naming, I'd love it08:39
rakhmerovI'm not getting rid of you though :) it's just true08:39
d0ugalThis is all a bit hypothetical - we should try and work with OSC but if we get blocked by something we will deal with it then08:40
d0ugalWe have bigger issues now :)08:40
rakhmerovd0ugal: that's where I'm getting to :)08:40
therveTotally08:40
rakhmerovtherve: we can get back to this conversation later08:40
rakhmerovtherve: that was useful for me anyway, thanks08:41
rakhmerovd0ugal: any new ideas on our issue?08:50
d0ugalrakhmerov: no, not really :/08:53
rakhmerovok08:53
rakhmerovd0ugal: I'm observing one really weird thing. If you run test test_task_defaults.test_task_defaults_retry_policy, for example, you'll see that 'fail' action runs many many times08:55
rakhmerovalthough it should run only 3 times08:55
rakhmerovd0ugal: my hypothesis is that something is broken in RPC08:55
d0ugalrakhmerov: that sounds likely. Unfortunately I don't know that part well yet09:00
*** zhurong has quit IRC09:01
rakhmerovor actually DB :)09:01
rakhmerovI've almost found a place..09:01
rakhmerovyeah, DB09:02
rakhmerovd0ugal: I guess something has changed in how SQLAlchemy works with graph updates09:03
rakhmerovwhat happens is: in Retry policy we store iteration # in DB, task_ex.runtime_context, which is JSON string09:04
rakhmerovbut it doesn't get updated when we increment it09:04
rakhmerovso policy runs forever09:04
rakhmerovd0ugal: and I know why only Mistral is suffering from this change :)09:12
d0ugalrakhmerov: oh cool, why?09:15
rakhmerovnobody else uses this feature )09:15
d0ugalhaha, makes sense09:15
rakhmerovd0ugal: remember you were surprised that not many other projects use transactions explicitly09:15
rakhmerovit's a part of the same thing..09:16
rakhmerovok, I'll tell more when I find the exact issue.. But at least it's clear now that it's because of SQLAlchemy09:17
d0ugalrakhmerov: oh, interesting09:19
d0ugalI guess I'll wait and see the patch :)09:19
*** sharat has joined #openstack-mistral09:21
*** sharatss has quit IRC09:21
*** sharat has quit IRC09:31
*** vienin has joined #openstack-mistral09:41
*** sharat has joined #openstack-mistral09:41
*** shardy has quit IRC09:47
rakhmerovd0ugal: so yes, I'm sure now about what happens but still don't know if it's a bug in SQLAlchemy or we just need to tweak something on our side10:04
rakhmerovanytime when we deal with json strings SQLAlchemy doesn't track state of these fields10:04
rakhmerovand doesn't see that the field has changed and therefore doesn't issue an SQL10:05
rakhmerovthat doesn't happen with other fields10:05
rakhmerovthat's why mostly 'with-items' and 'policy' tests fail, they all use these json strings10:05
rakhmerovwhat's interesting about it is that these fields are configured with custom SQLAlchemy types in the code10:06
rakhmerovand they do conversion on read/write to DB10:06
rakhmerovfrom dict to string and back10:06
rakhmerovmaybe that's where the problem is..10:06
rakhmerovd0ugal: I know exactly where the problem is, thinking how to fix it..10:35
rakhmerovmay take a while10:36
rakhmerovI'm off tomorrow so will try to fix it today10:36
d0ugalrakhmerov: :)10:36
d0ugalWhere is it?10:36
rakhmerovd0ugal: hold on, I'll give you a link10:37
rakhmerovhttps://github.com/zzzeek/sqlalchemy/blob/master/lib/sqlalchemy/ext/mutable.py#L64210:41
rakhmerovd0ugal: ^10:41
rakhmerovI wonder how it worked before actually..10:41
rakhmerovso here is the problem: if we have a dict like {'key1': {'nested_key1': 'nested_value1'}} then SQLA doesn't track deep changes properly10:42
rakhmerovas stated in this docstring10:43
rakhmerovI kind of understand what to do in order to fix it but not precisely..10:43
rakhmerovhere's where it's defined on our side: https://github.com/openstack/mistral/blob/master/mistral/db/sqlalchemy/types.py#L10710:45
d0ugalInteresting10:47
d0ugalMaybe it never really worked, only now we know it really doesn't :)10:47
d0ugalI need to go out for a bit, back later.10:48
rakhmerovme too10:49
*** gongysh has quit IRC11:03
*** openstackgerrit has joined #openstack-mistral11:06
openstackgerritXieYingYun proposed openstack/mistral master: Remove unnecessary setUp function in testcase  https://review.openstack.org/45176011:06
*** amoralej is now known as amoralej|lunch11:08
*** jkilpatr has quit IRC11:25
*** thrash has quit IRC11:37
*** thrash has joined #openstack-mistral11:41
*** thrash has quit IRC11:41
*** thrash has joined #openstack-mistral11:41
*** rbrady-afk is now known as rbrady11:45
*** dprince has joined #openstack-mistral11:49
*** sharatss_ has joined #openstack-mistral11:54
*** sharat has quit IRC11:54
*** catintheroof has joined #openstack-mistral12:21
*** shardy has joined #openstack-mistral12:49
*** shardy is now known as shardy_mtg12:50
*** amoralej|lunch is now known as amoralej12:52
*** shardy_mtg is now known as shardy12:56
*** gongysh has joined #openstack-mistral13:02
openstackgerritAnastasia Kuznetsova proposed openstack/mistral master: Fix work of task() without task name within on-clause cases  https://review.openstack.org/45134013:29
*** chlong has joined #openstack-mistral13:34
*** bobh has joined #openstack-mistral13:35
*** jaosorior is now known as jaosorior_away13:37
*** tuan_ has quit IRC13:52
*** tuan has joined #openstack-mistral13:54
*** szaher has quit IRC14:14
*** szaher has joined #openstack-mistral14:14
d0ugalrakhmerov: any luck?14:57
thervehttps://bitbucket.org/zzzeek/sqlalchemy/issues/3646/mutabledict-associatess-only-with-first sounds kind of worrying15:00
d0ugaltherve: yeah :/ I wonder what might have been missed due to this15:02
therveYeah15:03
therveTo be fair I think that using mutable is a horrible idea15:03
d0ugalyup15:03
*** xavierhardy has joined #openstack-mistral15:03
xavierhardyHi everyone, I have already asked the question but is it possible to authenticate to Mistral without Keystone, nor OpenID. We don't want to use keystone nor a third party website for authentication. Basic auth (username/password) would be enough.15:08
xavierhardybeing able to support custom authentication mechanism would be enough15:11
d0ugalxavierhardy: yeah, it is possible15:11
d0ugalyou just need to set auth_enable = False15:11
d0ugalthen you need to add basic auth with apache etc.15:11
xavierhardybut how do I get the users in Mistral then ?15:11
xavierhardyI want to apply policies then.15:12
xavierhardywith custom tenants, not coming from Keystone15:12
d0ugalwell, you can't ask for no auth but with auth :)15:13
xavierhardyBasic auth is not no auth15:13
xavierhardyWe have our own system, something not standard at all15:13
xavierhardyIf I have an option "auth_plugin" where I can define in Python what to do, I'm fine15:14
xavierhardyauth_plugin = my.custom.python:AuthenticationClass15:15
d0ugalno, I don't think there is a way to do that now15:15
xavierhardy:(15:15
*** gongysh has quit IRC15:17
*** vienin has quit IRC15:31
*** thrash is now known as thrash|biab15:31
*** weshay is now known as weshay_mtg15:43
*** Kimamisa has quit IRC15:46
*** sharatss_ has quit IRC15:59
*** thrash|biab is now known as thrash16:00
rakhmerovxavierhardy: that work was started by someone some time ago but it wasn't finished16:09
rakhmerovd0ugal: I didn't come up with a fix yet16:10
rakhmerovsorry16:10
rakhmerovwill try to find time tomorrow to fix it16:10
xavierhardyOK16:14
xavierhardyI guess you have some kind of abstraction for the authentication since you support both Keystone and OpenID, could I plug my code there ?16:14
xavierhardyIt would be to support custom authentication mechanisms in general.16:15
rakhmerovxavierhardy: yes, you can plug it there but we don't have any plugin architecture now, it's mostly hardcoded16:17
rakhmerovbtw, I don't think there's too much work to generalize it and make it pluggable16:18
rakhmerovxavierhardy: I'll talk to the person who was working on it, I believe he's mostly completed it but just didn't push upstream16:18
*** jaosorior_away is now known as jaosorior16:19
rakhmerovbecause they had the same requirement internally and AFAIK they addressed it16:19
xavierhardyOK16:19
*** shardy has quit IRC16:20
rakhmerovd0ugal: I have a pretty simple fix in mind, will try to check it tomorrow. In fact, we don't need nested structures, we can try to use plain dicts16:20
rakhmerovd0ugal, therve: weird, I just tried to remove extra level when using MutableDict, just left a simple dict with 1 level (i.e. {'k1': 'v1'}), and that doesn't work too16:29
rakhmerovas if there was actually a but in the new SQLA16:29
*** weshay_mtg is now known as weshay16:32
*** jaosorior has quit IRC16:50
*** chlong has quit IRC17:36
*** tuan has quit IRC17:37
*** tuan has joined #openstack-mistral17:41
*** FL1SK has quit IRC17:59
*** chlong has joined #openstack-mistral18:02
*** chlong has quit IRC18:11
*** tuan has quit IRC18:22
*** d0ugal has quit IRC18:24
*** chlong has joined #openstack-mistral19:00
*** chlong has quit IRC19:07
openstackgerritToure Dunnon proposed openstack/mistral-specs master: [WIP] Workflow Error Analysis  https://review.openstack.org/44321719:37
*** FL1SK has joined #openstack-mistral19:56
therverakhmerov, I pinged Mike Bayer, I hope he was some idea19:59
*** chlong has joined #openstack-mistral20:36
*** toure is now known as toure|gone20:55
*** catintheroof has quit IRC21:03
openstackgerritThomas Herve proposed openstack/mistral master: Workaround SQLAlchemy 1.1 issue  https://review.openstack.org/45198521:05
*** dprince has quit IRC21:05
*** jpich has quit IRC21:08
*** chlong has quit IRC21:09
*** harlowja has quit IRC22:02
*** bobh has quit IRC22:13
*** harlowja has joined #openstack-mistral23:24
*** bobh has joined #openstack-mistral23:41
*** openstack has joined #openstack-mistral23:44

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!