Wednesday, 2016-11-09

*** lucky_s has joined #openstack-mistral00:12
lucky_sHi dougal00:12
*** catintheroof has quit IRC00:14
lucky_scould you please again rebase on this patch https://review.openstack.org/#/c/375792/ . because i think it's again getting cachetools error00:14
*** catintheroof has joined #openstack-mistral00:15
*** catintheroof has quit IRC00:19
lucky_sjoin #openstack-watcher00:20
*** chlong has joined #openstack-mistral00:49
*** lucky_s has quit IRC01:56
*** Kiall has quit IRC02:22
*** Kiall has joined #openstack-mistral02:25
*** lucky_s has joined #openstack-mistral02:42
*** lucky_s has quit IRC04:06
openstackgerritMerged openstack/mistral-lib: Updated from global requirements  https://review.openstack.org/39532904:30
openstackgerritMerged openstack/mistral-dashboard: Updated from global requirements  https://review.openstack.org/39532804:31
*** janki has joined #openstack-mistral04:40
*** sharatss has joined #openstack-mistral04:45
*** hparekh has joined #openstack-mistral05:16
openstackgerritMerged openstack/python-mistralclient: Updated from global requirements  https://review.openstack.org/39536905:27
*** sharatss has quit IRC05:29
*** sharatss has joined #openstack-mistral05:35
*** sharatss has quit IRC05:53
*** sharatss has joined #openstack-mistral05:55
openstackgerritMerged openstack/mistral: Handle region_name in openstack actions  https://review.openstack.org/38918305:55
openstackgerritMerged openstack/mistral: Updated from global requirements  https://review.openstack.org/39518105:56
*** lucky_s has joined #openstack-mistral06:14
*** jtomasek has quit IRC06:55
rakhmerovkong: can we close https://bugs.launchpad.net/mistral/+bug/1634812 ?07:28
openstackLaunchpad bug 1634812 in Mistral "Retry policy failed to get upstream tasks context" [High,Confirmed] - Assigned to Lingxian Kong (kong)07:28
rakhmerovI believe it's fixed, right?07:28
rakhmerovI'll close it. Reopen it if it's not fixed (maybe I don't know something)07:29
lucky_sHi rakhmerov07:33
lucky_si want to discuss about how should i proceed for unit test case coverage blueprint07:34
lucky_sare you there ??07:35
rakhmerovyes07:35
rakhmerovhi07:35
rakhmerovfirst step I would suggest: show some stats as far as what's covered and what's not07:35
lucky_sAs mentioned in Blueprint all positive cases covered but i feel few negative cases not covered07:37
lucky_sin both mistral and it's client07:37
rakhmerovyes, maybe you could provide some list of things that you'd like to cover07:37
rakhmerovso that we could somehow track the progress07:37
rakhmerovotherwise it's not clear what the scope of this BP is07:37
rakhmerovd0ugal: hi )07:38
rakhmerovd0ugal: Congrats! You're a core :)07:38
lucky_srakhmerov, ok i will prepare stats what's covered and what's not and same update on registered blueprint .07:40
lucky_sis it ok ?07:40
rakhmerovyes, thanks07:40
rakhmerovlucky_s: ping me any time if you need to discuss something07:40
rakhmerovgenerally I think it's a very important work to do07:41
rakhmerovthank you for this07:41
lucky_srakhmerov: yeah , Thanks07:42
lucky_sHI rakhmerov : could you please review this patch https://review.openstack.org/#/c/348146/ . i explained about your query .08:02
rakhmerovok08:02
lucky_sThanks :)08:03
rakhmerovlucky_s: honestly, I still don't understand08:07
lucky_srakhmerov: ok , i am explaining here08:09
lucky_sis it ok ?08:09
rakhmerovby UTC and PDT you mean time zones?08:09
*** shardy has joined #openstack-mistral08:09
rakhmerovright?08:09
rakhmerovor something else08:09
lucky_syes time zones08:09
lucky_srakhmerov: have you checked the bug explanation ?08:15
rakhmerovmy confusion is the following: why should it be either UTC or PDT?08:16
rakhmerovshouldn't it be just a local server time?08:16
rakhmerovalways08:16
rakhmerovAside from the fact that this would be confusing to the user, if the timezone of the mistral08:20
rakhmerovengine were to be changed, the schedules would then be wrong08:20
rakhmerovlucky_s: I agree with the idea that all times should be in same time zone but I disagree with08:21
rakhmerov“Aside from the fact that this would be confusing to the user, if the timezone of the mistral08:21
rakhmerovengine were to be changed, the schedules would then be wrong08:21
rakhmerov08:21
rakhmerovI think "changing Mistral engine timezone" is not a thing that we should account for08:22
rakhmerovfor what reason should it change?08:22
rakhmerovddeja, d0ugal, kong, hparekh, mgershen: Guys, what do you think on this bug https://bugs.launchpad.net/mistral/+bug/1592164 ?08:22
openstackLaunchpad bug 1592164 in Mistral "cron-trigger shows execution times in local timezone" [Undecided,In progress] - Assigned to Lucky samadhiya (lucky-samadhiya)08:22
rakhmerovand the comments I left here08:23
rakhmerovlucky_s: another alternative could be just having a config option for that08:23
therverakhmerov, You should just use UTC everywhere08:24
rakhmerovsay "mistral_time_zone: local | UTC | PDT"08:24
rakhmerovtherve: why?08:24
therverakhmerov, Because that's the only timezone that you can trust08:24
therveIt doesn't have DST08:24
ddejarakhmerov: agree that we should have one timezone for created_at and next_execution_time08:24
*** jaosorior has joined #openstack-mistral08:25
rakhmerovddeja: right08:25
rakhmerovtherve: why is that important?08:25
ddejaand I agree with therve that using UTC is the safest08:25
therverakhmerov, Because you don't want to go back in time?08:25
lucky_srakhmerov: sorry  , my laptop got shutdown . i missed your conversation08:25
lucky_s:)08:25
therveBy using UTC, you can always translate to the user timezone in necessary, and you have a source of truth08:26
therveEveryone does that everywhere, that should be a no-brainer08:26
rakhmerovok, then it seems to me that we're talking about different things08:27
rakhmerovI guess we should split it into different things: 1) What time zone Mistral service uses (can be UTC, right) 2) As a user perspective I'd like to see everything in my local time08:28
rakhmerovon #1 - agree08:28
lucky_sagree08:28
rakhmerovon #2 - we need to implement it because it may be missing now08:28
rakhmerovif you all agree with these two points then yes, ok08:29
rakhmerovlet's proceed with this08:29
ddejarakhmerov: I guess it's the right way08:29
rakhmerovok08:29
therve#2 can (should?) be done in the client side though08:29
hparekhyeah08:29
rakhmerovI guess so08:29
ddejaso that we store time in the DB in UTC, but translate it to user on the client08:29
rakhmerovI just believe that it's not done yet08:29
rakhmerovyep08:29
rakhmerovlucky_s: +2 for this patch08:31
rakhmerovplease check if time gets translated as we discussed here08:31
lucky_sHi rakhmerov : i again missed your conversation . could you please let me what should i do now ??08:32
lucky_srakhmerov: Thanks for +2 :)08:32
rakhmerov(Wed November 09 2016 12:27:04) ok, then it seems to me that we're talking about different things08:34
rakhmerovrakhmerov08:34
rakhmerov(Wed November 09 2016 12:28:18) I guess we should split it into different things: 1) What time zone Mistral service uses (can be UTC, right) 2) As a user perspective I'd like to see everything in my local time08:34
rakhmerovrakhmerov08:34
rakhmerov(Wed November 09 2016 12:28:25) on #1 - agree08:34
rakhmerovlucky_s08:34
rakhmerov(Wed November 09 2016 12:28:43) agree08:34
rakhmerovrakhmerov08:34
rakhmerov(Wed November 09 2016 12:28:47) on #2 - we need to implement it because it may be missing now08:34
rakhmerovrakhmerov08:34
rakhmerov(Wed November 09 2016 12:29:06) if you all agree with these two points then yes, ok08:34
rakhmerovlucky_s: #2 is what I'd like you to do08:34
* d0ugal reads the bug08:35
hparekhrakhmerov: I have just put my comments on review.08:37
lucky_srakhmerov: i got it . so for #2 may be we can do in different patch08:38
hparekhI think we should have some good tests for this change.08:38
*** jpich has joined #openstack-mistral08:46
*** openstackgerrit has quit IRC08:48
*** openstackgerrit has joined #openstack-mistral08:48
rakhmerovyes08:52
*** lucky_s has quit IRC08:54
d0ugalI agree with the conculsion FWIW :)08:57
*** jrist has quit IRC08:59
*** jrist has joined #openstack-mistral09:00
kongrakhmerov: yeah, I believe it should be fixed09:20
openstackgerritpawnesh kumar proposed openstack/mistral: Replace uuid4() with generate_uuid() from oslo_utils  https://review.openstack.org/39434609:22
rakhmerovkong: ok09:23
openstackgerritMerged openstack/python-mistralclient: remove openstack/common/cliutils.py  https://review.openstack.org/39505509:24
*** janki has quit IRC09:25
*** janki has joined #openstack-mistral09:47
openstackgerritMerged openstack/python-mistralclient: remove apiclient from mistralclient  https://review.openstack.org/39508209:49
*** hparekh has quit IRC09:55
openstackgerritMerged openstack/mistral: Add type to tasks API  https://review.openstack.org/38289310:55
openstackgerritpawnesh kumar proposed openstack/python-mistralclient: Remove unused scripts in tools  https://review.openstack.org/39554611:30
openstackgerritpawnesh kumar proposed openstack/mistral: Remove unused scripts in tools  https://review.openstack.org/39554711:30
*** lucky_s has joined #openstack-mistral11:37
*** ddeja has quit IRC11:53
*** ddeja has joined #openstack-mistral11:57
*** catintheroof has joined #openstack-mistral12:16
*** catinthe_ has joined #openstack-mistral12:20
*** catintheroof has quit IRC12:21
*** d0ugal has quit IRC12:51
*** d0ugal has joined #openstack-mistral13:00
openstackgerritSharat Sharma proposed openstack/mistral: Added senlin action pack  https://review.openstack.org/39558713:18
*** lucky_s has quit IRC13:18
openstackgerritMerged openstack/mistral: Replace uuid4() with generate_uuid() from oslo_utils  https://review.openstack.org/39434613:20
*** jistr is now known as jistr|brb13:26
*** lucky_s has joined #openstack-mistral13:34
*** ^Gal^ has joined #openstack-mistral13:46
*** jistr|brb is now known as jistr13:46
*** catintheroof has joined #openstack-mistral13:47
*** catinthe_ has quit IRC13:50
*** janki has quit IRC14:07
*** xavierhardy has joined #openstack-mistral14:14
xavierhardyHi Mistral community, we have unveiled a very serious issue with tons of unclosed transactions in the DB14:14
xavierhardyhttps://bugs.launchpad.net/mistral/+bug/164046914:15
openstackxavierhardy: Error: malone bug 1640469 not found14:15
xavierhardyThe current commit is c0a450128e7fea3f984fabbbccb6e3473232f495 (stable/newton branch, probably not the latest)14:15
xavierhardyI just call "curl http://localhost:8989/v2/workflows" and that alone creates a dangling a transaction14:16
xavierhardyThe DB is a PostgreSQL DB.14:16
xavierhardyIs that familiar to anyone ?14:16
*** jrist has quit IRC14:17
*** jrist has joined #openstack-mistral14:20
xavierhardycould it be linked to that one: https://bugs.launchpad.net/mistral/+bug/1638805 ?14:23
openstackLaunchpad bug 1638805 in Mistral "Mistral DB layer rolls back TX even if oslo.db already did it" [High,In progress] - Assigned to Renat Akhmerov (rakhmerov)14:23
*** dprince has joined #openstack-mistral14:29
xavierhardycommit 856ecb252ea3ebfdeb93d2cdedbaadd515e0deac14:30
xavierhardyMerge: 76f6b92 c0266ee14:30
xavierhardyAuthor: Jenkins <jenkins@review.openstack.org>14:30
xavierhardyDate:   Mon Nov 7 09:53:22 2016 +000014:30
xavierhardy    Merge "Fix DB API transaction()" into stable/newton14:30
xavierhardycommit c0266ee87a3f2eedc08c6293b3e0e3e8b377c88914:30
xavierhardyAuthor: Renat Akhmerov <renat.akhmerov@gmail.com>14:30
xavierhardyDate:   Thu Nov 3 16:30:45 2016 +070014:30
xavierhardy    Fix DB API transaction()14:30
xavierhardy14:30
xavierhardy    * start_tx() should not be under 'try' because we can't do14:30
xavierhardy      any clean up actions if it fails. The best thing we can do is14:30
xavierhardy      to bubble up the error to the user.14:30
xavierhardy14:30
xavierhardyno, it's not14:33
xavierhardyI'm now able to reproduce it on the latest stable/newton commit14:33
*** jaosorior has quit IRC14:33
*** jaosorior has joined #openstack-mistral14:34
xavierhardyOK, it seems everywhere in the (REST) API component14:45
xavierhardywe must wrap all REST API functions with a transaction (we could batch them, but let's save that for later)14:46
*** lucky_s has quit IRC14:52
xavierhardyOK, adding with db_api.transaction() seems to fix it (I tried it on /workflows)15:01
xavierhardynow, only have a limited number of IDLE transactions, I guess the pecan thread/process each maintain an opened session15:01
xavierhardyI guess I'll add a wrapper to do that in rest_utils (just like wrap_wsme_controller_exception)15:10
d0ugalxavierhardy: Hey, just reading everything you sent.15:15
d0ugalxavierhardy: The first link you sent is a 404 - https://bugs.launchpad.net/mistral/+bug/164046915:17
openstackd0ugal: Error: malone bug 1640469 not found15:17
xavierhardyI filed it as a security issue15:18
xavierhardyyou need to log in.15:18
d0ugalxavierhardy: ah, I am logged in but I must not have access.15:18
xavierhardyOK, I'll check15:18
xavierhardyMay be notifiedDmitri ZimineNikolay Makhotkin15:18
xavierhardyindeed15:18
d0ugalrakhmerov: You might know about this ^15:18
xavierhardyI have added you15:19
xavierhardyboth15:19
d0ugalxavierhardy: Thanks15:20
d0ugalxavierhardy: do you think this is postgres specific?15:20
d0ugalxavierhardy: all my mistral envs are using mysql (unfortunately!)15:21
xavierhardyNo15:21
xavierhardyIt's not15:21
d0ugalok, cool - I shall try and reproduce it then in my env15:21
xavierhardySomeone started using Mistral seriously in production this morning, and it almost overloaded the DB.15:22
d0ugalxavierhardy: dang15:24
xavierhardyWe had seen this earlier in our custom action code, and fixed it; but we never thought this was affecting the whole REST API server.15:25
d0ugalxavierhardy: are you running mistral with no auth?15:27
xavierhardyfor the moment, yes15:28
xavierhardybut behind a proxy15:28
xavierhardywith auth15:28
d0ugalxavierhardy: rather than curl I am using `mistral workflow-list` because I have keystone auth enabled15:29
xavierhardyshould work too15:29
d0ugalxavierhardy: and this goes a bit beyond mysql knowledge, but I am using: SHOW FULL PROCESSLIST;15:30
d0ugalxavierhardy: and I have the workflow-list running in a loop in bash15:31
d0ugalthe number of rows in the process list does seem to be increasing15:31
xavierhardyin pgsql it's obvious15:31
xavierhardysee the bug tracker15:31
xavierhardyI have 50 rows without the fix, between 5 and 8 with the fix, and no more idle in transaction15:32
d0ugalxavierhardy: Yeah, I am seeing something similar I think15:33
xavierhardyyou mean with MySQL now ?15:34
d0ugalxavierhardy: Yeah, still looking15:34
xavierhardywith auth or without ?15:34
d0ugalwith auth15:34
d0ugalbut give me a min and I'll know more15:34
xavierhardytry with "with transactions()" all around the REST API code for /workflows and retry to see the difference15:35
xavierhardy*with db_api.transactions():15:38
*** weshay has joined #openstack-mistral15:38
d0ugalxavierhardy: I seem to be seeing exactly the same result15:43
d0ugalxavierhardy: so maybe I am just seeing connection pooling or something like that?15:43
d0ugalI don't really understand tbh15:43
d0ugalxavierhardy: Right, so I am seeing something15:48
d0ugalxavierhardy: without the db_api.transaction() it seems to get up to about 30 processes in mysql (I assume that means 30 transactions)15:48
d0ugalwith the db_api it gets up to 515:48
d0ugalso there must be something else cleaning up more regularly in the mysql case, I think?15:48
d0ugalxavierhardy: so, yeah, I think the change you are saying is good.15:49
*** rrecio has joined #openstack-mistral15:56
xavierhardyOK15:57
d0ugalI am adding details to the bug.15:57
d0ugal(of my findings)15:57
xavierhardyNo, transactions and sessions are 2 different things15:57
xavierhardybe careful15:57
xavierhardyfor a transaction, you need a lock15:57
xavierhardy(and also a session of course)15:57
xavierhardya session is just a connection on the DB15:57
xavierhardy(TCP connection)15:58
d0ugalRight15:58
xavierhardyHaving open sessions is fine as long as your DB supports. I think it is supported, I remember seeing a sql session pool somewhere15:58
xavierhardylocking tables all over the place without even committing anything (what happens here) is terrible and wrong15:59
xavierhardy*DB supports it15:59
xavierhardyhere is what I would patch16:00
xavierhardy@@ -167,6 +171,7 @@ class WorkflowsController(rest.RestController, hooks.HookController):16:00
xavierhardy             db_api.delete_workflow_definition(identifier)16:00
xavierhardy16:00
xavierhardy     @rest_utils.wrap_wsme_controller_exception16:00
xavierhardy+    @rest_utils.wrap_controller_transaction16:00
xavierhardy     @wsme_pecan.wsexpose(resources.Workflows, types.uuid, int,16:00
xavierhardy                          types.uniquelist, types.list, types.uniquelist,16:00
*** xavierhardy has quit IRC16:00
*** xavierhardy has joined #openstack-mistral16:01
*** xavierhardy has left #openstack-mistral16:01
*** xavierhardy has joined #openstack-mistral16:02
xavierhardyhi again16:02
xavierhardyI got disconnected for "flooding"...16:02
d0ugalxavierhardy: heh, yeah, pasting in here isn't wise :)16:02
xavierhardy(just copy-pasted some stuff)16:02
xavierhardyI'll send a link then16:02
d0ugalxavierhardy: thanks16:03
xavierhardyhttp://pastebin.com/2DF6GGqP16:03
d0ugalxavierhardy: Yup, that looks like a good fix. I guess we need to also do it on all the other files in mistral/api/controllers/v2/16:04
d0ugali.e. every controller16:04
xavierhardyyes16:04
xavierhardyOK16:05
xavierhardyI've also noticed that some calls miss one of the wrappers (separate PR then)16:05
xavierhardythe one to catch exceptions16:05
xavierhardy@rest_utils.wrap_wsme_controller_exception16:05
xavierhardyfor instance16:06
xavierhardyworkbook get_all16:06
xavierhardyis there a reason for that?16:07
d0ugalxavierhardy: Not that I know of, it would be good to fix that too16:09
xavierhardyOK16:10
*** sharatss has quit IRC16:19
openstackgerritXavier Hardy proposed openstack/mistral: Fix REST API dangling transactions  https://review.openstack.org/39569616:24
openstackgerritXavier Hardy proposed openstack/mistral: Fix missing exception decorators in REST API  https://review.openstack.org/39570516:32
xavierhardydone for tonight16:33
*** weshay is now known as weshay_brb16:36
d0ugalxavierhardy: Okay, thanks16:37
*** weshay_brb is now known as weshay16:45
*** chlong has quit IRC17:08
*** chlong has joined #openstack-mistral17:17
*** jpich has quit IRC17:33
openstackgerritMichal Gershenzon proposed openstack/mistral: Migrate mistral task_type  https://review.openstack.org/39574217:56
openstackgerritMerged openstack/mistral: Add more tests to mistral rally  https://review.openstack.org/39487918:00
*** jaosorior has quit IRC18:26
*** harlowja has quit IRC18:59
*** dprince has quit IRC19:00
*** catinthe_ has joined #openstack-mistral19:10
*** catintheroof has quit IRC19:12
*** clenimar has joined #openstack-mistral19:46
*** clenimar has quit IRC19:52
*** harlowja has joined #openstack-mistral21:49
openstackgerritWinson Chan proposed openstack/python-mistralclient: Add cancelled state to action executions  https://review.openstack.org/38470022:20
*** chlong has quit IRC22:47
*** catinthe_ has quit IRC23:04
*** harlowja_ has joined #openstack-mistral23:08
*** rrecio has quit IRC23:09
*** harlowja has quit IRC23:12
*** enykeev has quit IRC23:40
*** enykeev has joined #openstack-mistral23:41

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!