Wednesday, 2016-12-21

openstackgerritTony Breeds proposed openstack/mistral-dashboard: Add Constraints support  https://review.openstack.org/41330000:51
*** bobh has joined #openstack-mistral01:03
*** harlowja has joined #openstack-mistral01:06
openstackgerrityunfeng zhou proposed openstack/mistral-dashboard: add CONTRIBUTING.rst  https://review.openstack.org/41286501:06
openstackgerrityunfeng zhou proposed openstack/mistral-specs: add CONTRIBUTING.rst  https://review.openstack.org/41286601:07
*** harlowja has quit IRC01:44
*** sharatss has quit IRC03:18
*** sharatss has joined #openstack-mistral03:18
*** bobh has quit IRC03:22
rakhmerovkong_: please have a look at https://review.openstack.org/#/c/410567/ again03:58
rakhmerovI need these patches very much03:58
openstackgerritMerged openstack/mistral-dashboard: Updated from global requirements  https://review.openstack.org/41314904:12
*** thrash is now known as thrash|g0ne04:45
openstackgerritMerged openstack/mistral-dashboard: Workflow list - added missing fields  https://review.openstack.org/41251704:56
openstackgerritMerged openstack/mistral-dashboard: add CONTRIBUTING.rst  https://review.openstack.org/41286504:57
openstackgerritMerged openstack/mistral-specs: add CONTRIBUTING.rst  https://review.openstack.org/41286604:57
openstackgerritMerged openstack/mistral-dashboard: Add Constraints support  https://review.openstack.org/41330004:59
openstackgerritRenat Akhmerov proposed openstack/mistral: Refresh object state after lock acquisition in WithItemsTask  https://review.openstack.org/41341605:36
*** sharatss has quit IRC05:48
*** dkushwaha has joined #openstack-mistral05:55
*** dkushwaha has left #openstack-mistral05:56
*** Qiming has quit IRC06:01
*** Qiming has joined #openstack-mistral06:07
*** ist has joined #openstack-mistral06:28
*** sharatss has joined #openstack-mistral06:56
sharatssrakhmerov: hi07:03
rakhmerovhi07:04
rakhmerovhow can I help you, sir? :)07:04
*** jaosorior has joined #openstack-mistral07:17
sharatssrakhmerov: https://bugs.launchpad.net/mistral/+bug/1649366 do u think this is valid sir? :))07:20
openstackLaunchpad bug 1649366 in Mistral "body for std.email seems to actually be required - throws error when not present" [Undecided,Confirmed] - Assigned to Sharat Sharma (sharat-sharma)07:20
rakhmerovlet me see07:20
rakhmerovsharatss: Yes, the bug looks valid07:21
rakhmerovbut I think I've seen a patch for it already07:21
sharatssrakhmerov: my doubt is whether the body has to optional or the subject07:22
sharatssrakhmerov: definitely both should not be optional07:22
rakhmerovwell, you're talking about a different aspect07:22
rakhmerovthis bug says that the doc says "optional" but, in fact, it's not optional07:23
rakhmerovnot whether it should be optional or not07:23
rakhmerovit just states a mismatch07:23
rakhmerovfrom this perspective it's valid07:23
rakhmerovthe other thing is that how it should be07:24
rakhmerovI think the answer is: does SMTP protocol allow empty subject and body?07:24
rakhmerovif yes, then I see no reason why not keep them optional07:24
*** ist has quit IRC07:24
*** shardy has joined #openstack-mistral07:24
rakhmerovI'm pretty sure that subject can be empty in SMTP07:24
rakhmerovnot sure about body though07:25
rakhmerovmakes sense?07:25
sharatssrakhmerov: in code we have subject as optional.  should this be changed in doc as well or a code change is required to make body optional07:25
rakhmerovagain, I don't know at this point07:26
rakhmerovplease check what SMTP says07:26
rakhmerovif it allows them to be empty then we need to change our docs07:27
rakhmerovsorry07:27
rakhmerovthe code07:27
rakhmerovso that they are optional in both docs and implementation07:27
sharatssrakhmerov: i just checked07:28
sharatssrakhmerov: both can be optional07:28
rakhmerovok, so?07:28
rakhmerovok07:28
rakhmerovlet's do so07:28
sharatssrakhmerov: yea..07:29
rakhmerovok :)07:29
openstackgerritSharat Sharma proposed openstack/mistral: Make body of std.email optional  https://review.openstack.org/41346807:48
*** openstackgerrit has quit IRC07:48
*** _gryf has quit IRC08:16
*** _gryf has joined #openstack-mistral08:21
*** mgershen has joined #openstack-mistral08:22
*** mgershen has quit IRC08:36
*** jpich has joined #openstack-mistral08:52
*** gongysh has joined #openstack-mistral08:54
gongyshkong_, hi08:54
*** jaosorior has quit IRC08:58
*** jaosorior has joined #openstack-mistral08:58
*** chlong has quit IRC09:34
*** mgershen has joined #openstack-mistral09:47
rakhmerovkong_: thanks!10:04
*** mgershen has quit IRC10:05
*** ^Gal^ has joined #openstack-mistral10:06
ddejaHi rakhmerov10:20
rakhmerovhi10:20
ddejaI found the problem with failing tests10:21
rakhmerovok10:21
rakhmerovwhat is it this time?10:21
ddejalet me find the line in github10:21
ddejahttps://github.com/openstack/mistral/blob/master/mistral/engine/rpc_backend/rpc.py#L67-L7510:22
ddejarakhmerov: so, we are using global to store the engine_client object10:22
ddejaand we use wsgi to start the API10:22
ddejawsgi startes a number of workers with API, which are greenthreads10:23
rakhmerovthey are processes10:23
rakhmerovnot greenthreads10:23
rakhmerovAFAIK10:23
ddejahm from debbuging it looks like theye were greenthreads10:23
ddejabut that doesn't matter10:23
ddejathe point is10:24
rakhmerovok10:24
ddejawe got 2 threads using same engine_client object10:24
kong_rakhmerov: np :-)10:24
ddejaso in a very rare (but not rare enaught) situation when tow threads want to send something over RPC at the same time10:24
rakhmerov:)10:24
kong_the patches all look good10:24
kong_gongysh: hi10:24
gongyshhi10:25
kong_gongysh: how are u, man10:25
gongyshhow is the life?10:25
ddejarakhmerov: they both try to use same http socket10:25
kong_gongysh: not bad10:25
rakhmerovddeja: ok, go on pls10:25
gongyshkong_, me either.10:25
rakhmerovsame http socket?10:25
ddejawhich result in simultaneus fileread error10:25
rakhmerovoooh, damn...10:25
ddejarakhmerov: yes, rpc is using http sockets deep down ;)10:26
rakhmerovI think I understand10:26
ddejaso, my idea is to change the lines I've just send you10:26
gongyshkong_,  http://docs.openstack.org/developer/mistral/architecture.html10:26
ddejaso we don't use global, but local thread storage instead10:26
rakhmerovwait a sec..10:26
gongyshkong_, is the mistral arch doc right?10:26
rakhmerovlet me think10:26
ddejaso each thread got it's own client object10:27
rakhmerovlet's think10:27
rakhmerovabout pros and cons10:27
ddejabut I'm not sure if it's good idea, so that is why I write here instead of sending a patch :)10:27
rakhmerovclient object seems to be pretty cheap to create, right?10:27
rakhmerovnothing like a heavy initialization etc.10:27
ddejayes10:28
rakhmerovit's just a simple object parameterized with rpc info10:28
rakhmerovhm...10:28
ddejabut we don't want to spawn a lot of it, since each keeps an RPC connection to AMQP broker10:28
rakhmerovyes, that's what I thought too10:28
ddejabut one per process should be OK10:29
rakhmerovso, does this happen only in case of our Kombu RPC?10:29
rakhmerovor with oslo.m too?10:29
ddejaI've never seen it with oslo10:29
ddejathere may be some locking mechanism inside10:29
rakhmerovok, so we're talking only about Kombu10:29
ddejaor oslo internaly may have a lot of workers10:29
rakhmerovI see10:29
kong_gongysh: yes, i think so10:30
ddejarakhmerov: that is another approach10:30
ddejaI can make kombu_client thread-safe10:30
gongyshkong_ is the 'scheduler' an process?10:30
gongyshkong_, do we have such process?10:30
*** Qiming has quit IRC10:30
ddeja(well, I'd like to made it multi-thread support during Ocata cycle)10:31
kong_gongysh: it's a periodic task running inside mistral-engine actually10:31
rakhmerovddeja: since this is pluggable (currently o.m and kombu) we need to define requirements for implementation10:31
rakhmerovif that should be thread-safe it needs to be thread-safe :)10:31
rakhmerovfor all of them10:31
rakhmerovbecause a calling code doesn't know what it calls10:31
rakhmerovwhat impl10:31
ddejarakhmerov: yes. Right now I'm treating the o.m as an reference driver10:32
rakhmerovso the question is: 1) do we specify that implementation can be not thread-safe 2) opposite10:32
rakhmerovok10:32
ddejaOK, so I'll jump into the o.m10:32
ddejato check if there are some mechanism that prevents it10:33
ddejaor we are just lucky ;)10:33
rakhmerovddeja: I mean it's probably ok if we say "we allow them to be not thread-safe"10:33
rakhmerovjust brainstorming..10:33
ddejathen I'll proceed with a fix10:33
rakhmerovyes10:33
gongyshkong_ where is the event-engine?10:33
ddejarakhmerov: Ok, thanks a lot Renat!10:34
rakhmerovnp10:34
rakhmerovmany thanks to you10:34
* kong_ goes to pm to give rakhmerov and ddeja dedicated channel :-)10:34
rakhmerovyou do awesome job on ivestigating such issues :)10:34
rakhmerovkong_: no!10:34
rakhmerovwe're almost finished :)10:35
ddejakong_: we just finished I guess ;)10:35
*** Qiming has joined #openstack-mistral10:35
rakhmerovddeja: so, just one question: why did you say "one per process is enough"?10:35
rakhmerovnot sure I understand..10:35
rakhmerovand I'm surprised why you saw workers implemented as green threads10:35
ddejaI meant that: there should be no simultaneus reads error and on the other hand, we should not create too many connections10:36
ddejarakhmerov: I may misread something in logs10:36
rakhmerovif you look at oslo.service project you'll see that there's two things: ServiceLauncher and ProcessLauncher10:36
rakhmerovthe latter is for using with workers as separate threads10:36
ddejaand, ps aux | grep mistral shows more than one API service, so there are really threads :)10:36
rakhmerovmaybe for unit tests it runs differently10:36
rakhmerovooh, ok :)10:37
rakhmerovsystem threads10:37
rakhmerovthis is important :)10:37
rakhmerovI mean operating system threads10:37
ddejabut nevertheless, still they use same connection and sometimes try to use it at the same time, and it of course is not working :)10:37
rakhmerovhm...10:38
rakhmerovok10:38
rakhmerovso, I would say: it would be cool if we could make Kombu thread-safe, but I can't tell right away how problematic it is10:39
rakhmerovyou need to look into this..10:39
ddejarakhmerov: I know how to do this, that's not a problem10:39
rakhmerovawesome10:39
ddejaI was just not sure if it's right approach10:40
*** mgershen has joined #openstack-mistral10:47
ddejarakhmerov: I've checked - oslo is using something called 'connection pools' internally11:02
ddejait get's new connection from pool for each message11:02
ddejathat is why we don't see this error in oslo11:03
*** openstackgerrit has joined #openstack-mistral11:04
openstackgerritMerged openstack/mistral: Get rid of with_items.py module in favor of WithItemsTask class  https://review.openstack.org/41056711:04
rakhmerovddeja: yeah, typical pool pattern11:06
rakhmerovis it their own pool implementation?11:06
rakhmerovI wonder if we can reuse something like this11:06
rakhmerovfrom some generic lib (of course, not o.m)11:06
ddejarakhmerov: no, they are using pika-pool11:07
ddejabut what is worse11:07
ddejait seems like kombu is also using connection pools right now11:07
rakhmerov:)))11:07
rakhmerovthe game is getting more interesting? :)11:07
*** sharatss has quit IRC11:08
ddejarakhmerov: wait, no11:08
ddejaok11:08
ddejathere is a difference11:08
*** sharatss has joined #openstack-mistral11:08
ddejain docs for using pools: "from kombu.pools import producers"11:08
ddejain our code "from kombu import producers"11:08
ddeja(not excactly that, but this is the problem)11:09
ddejabut hmm, it looks like it points to the same place11:10
ddejaOK, so now I know nothing11:10
rakhmerov:)11:11
rakhmerovit happens11:11
* ddeja is getting back to debbuging11:12
ddejaI'll send a patch once I got it working11:12
*** gongysh has quit IRC11:13
openstackgerritGal Margalit proposed openstack/mistral-dashboard: Fixed: Dashboard: "Run action" functionality doesn't work  https://review.openstack.org/41315211:14
openstackgerritMerged openstack/mistral: Slightly improve 'with-items' tests  https://review.openstack.org/41067611:18
openstackgerritMerged openstack/mistral: Apply locking to control 'with-items' concurrency  https://review.openstack.org/41069011:18
openstackgerritMerged openstack/mistral: Fix 'with-items' task completion condition  https://review.openstack.org/41234111:18
rakhmerovddeja: ok11:21
rakhmerovddeja, d0ugal, kong_, hparekh: could some of you please look at https://review.openstack.org/#/c/413416/ ? It is small but very important11:21
rakhmerovgreat, thanks d0ugal11:25
d0ugalnp11:25
rakhmerovyou made me happy :)11:25
d0ugalhaha11:25
* d0ugal is reviewing all the things11:25
d0ugal(but mostly in TripleO so far)11:25
openstackgerritMerged openstack/mistral: Small adjustments in WithItemsTask  https://review.openstack.org/41276111:28
rakhmerov:)11:28
d0ugalI guess this is technically isn't backwards compatible.... but is that okay? https://review.openstack.org/#/c/412389/11:32
openstackgerritRenat Akhmerov proposed openstack/mistral: Add a test for 'with-items' task: count=100, concurrency=10  https://review.openstack.org/41357411:39
*** thrash|g0ne is now known as thrash11:45
*** mgershen has quit IRC11:48
openstackgerritRenat Akhmerov proposed openstack/mistral: Add rally tests for 'join': 100 and 500 parallel tasks  https://review.openstack.org/41358611:57
*** mgershen has joined #openstack-mistral12:04
*** shardy is now known as shardy_lunch12:05
openstackgerritMerged openstack/mistral: Refresh object state after lock acquisition in WithItemsTask  https://review.openstack.org/41341612:17
*** mgershen has quit IRC12:29
openstackgerritMerged openstack/mistral: Fix configuration generator  https://review.openstack.org/41225612:52
openstackgerritMerged openstack/mistral: Added test cases for a few possible scenarios  https://review.openstack.org/40911712:52
*** shardy_lunch is now known as shardy12:55
*** weshay_afk is now known as weshay12:56
*** mgershen has joined #openstack-mistral13:06
*** jtomasek has joined #openstack-mistral13:10
*** dprince has joined #openstack-mistral13:17
*** gongysh has joined #openstack-mistral13:30
*** gongysh has quit IRC13:34
*** shardy has quit IRC13:35
*** mgershen has quit IRC13:48
*** sharatss has quit IRC13:50
*** sharatss has joined #openstack-mistral13:51
*** jaosorior has quit IRC14:21
*** jaosorior has joined #openstack-mistral14:21
Qiminghi, guys14:47
QimingI'm studying mistral api14:48
Qimingit seems that a workflow update request can be posted to /v2/workflows directly, without providing an ID, am I understanding this correctly?14:49
Qimingoh, the verb is PUT, but ... still, the workflow ID is not required, right?14:49
*** dprince has quit IRC14:52
*** dprince has joined #openstack-mistral14:54
*** bobh has joined #openstack-mistral14:57
*** mgershen has joined #openstack-mistral15:00
*** bobh has quit IRC15:31
d0ugalQiming: correct.15:34
d0ugalQiming: Workflows are referenced by name - which is in the workflow definition15:35
d0ugalI don't really know why we need ID :)15:35
d0ugalMaybe it was just added incase we need it in the future?15:35
d0ugalrakhmerov: ^15:35
*** ^Gal^ has quit IRC15:39
*** jamielennox is now known as jamielennox|away15:46
*** openstackgerrit_ has joined #openstack-mistral15:47
*** openstackgerrit_ has quit IRC15:49
*** bobh has joined #openstack-mistral15:49
d0ugalis mistral.tests.unit.services.test_trigger_service.TriggerServiceV2Test.test_trigger_create faiing for anyone else?15:50
*** chlong has joined #openstack-mistral15:52
*** jamielennox|away is now known as jamielennox15:54
*** bobh has quit IRC16:00
*** jaosorior has quit IRC16:03
*** ^Gal^ has joined #openstack-mistral16:05
d0ugaljpich: so I am currently investigating https://github.com/openstack/mistral/blob/master/mistral/services/triggers.py#L28-L3416:23
jpichd0ugal: Is that what gets called in the problematic test?16:24
d0ugaljpich: yeah, well, create_cron_trigger16:24
d0ugaljpich: yeah, well, create_cron_trigger in that file gets called.16:24
d0ugaljpich: but the call to that small function is the only bit that looks suspicious (unless the issue is at the database leveel)16:24
*** ^Gal^_ has joined #openstack-mistral16:25
*** ^Gal^ is now known as Guest1026016:25
*** ^Gal^_ is now known as ^Gal^16:25
rakhmerovd0ugal, Qiming: on the API question Dougal's answer is 100% correct. As far as id, it's needed when it comes to e.g. multiple tenants16:28
rakhmerovtenants can have workflows with same names actually16:29
rakhmerovand the thing is that these workflows in some cases need to be accessible from many tenants16:29
d0ugaljpich: so, "local_time" is correct. https://github.com/openstack/mistral/blob/master/mistral/services/triggers.py#L2916:29
d0ugaljpich: it matches what the test expects.16:29
rakhmeroveither if they are created with scope=public or if a workflow is shared by one tenant with another one through resource sharing mechanism16:30
d0ugaljpich: something in the following two lines takes it back an hour16:30
*** ^Gal^ has quit IRC16:30
rakhmerovd0ugal: is this test failing for you locally?16:30
d0ugalrakhmerov: Yup16:30
rakhmerovhm..16:31
rakhmerovlet me check16:31
d0ugalrakhmerov: and jpich confirmed it as failing too, so it isn't only me16:31
*** bobh has joined #openstack-mistral16:31
*** bobh has quit IRC16:31
rakhmerovyes16:32
rakhmerovsame to me16:32
rakhmerovweird16:32
d0ugalrakhmerov: interesting ;)16:32
rakhmerovit passes on our gates, doesn't it?16:32
d0ugalrakhmerov: yup, it does.16:32
d0ugalrakhmerov: which OS are you running?16:32
rakhmerovMac OS16:32
rakhmerovSierra16:32
d0ugalk, good - so not specific to Feodra :)16:32
rakhmerovnope16:32
d0ugal(I think that is good?)16:32
rakhmerovyeah :)16:33
rakhmerovit's not only your trouble :)16:33
d0ugalhah16:33
d0ugalrakhmerov: it must be late for you?16:33
rakhmerovyes, kind of16:33
rakhmerovI'll go to bed soon16:33
rakhmerovso recently ddeja made changes related to how we store time16:35
d0ugalYeah, I guess that must be related.16:35
rakhmerovI wonder if that's related16:35
jpichd0ugal: I changed my timezone and the difference was 5 hours rather than one, some kind of UTC vs local time issue?16:36
jpichreference = datetime.datetime(2010, 8, 25, 0, 5) - actual    = datetime.datetime(2010, 8, 25, 5, 5)16:36
d0ugaljpich: interesting.16:36
rakhmerovjpich: yeah, right.. Let me find a patch that may have affected this16:38
openstackgerritDougal Matthews proposed openstack/mistral: Testing without utc  https://review.openstack.org/41371316:38
d0ugaljpich, rakhmerov: ^ that fixes the test for me. I am interested to see what happens when CI runs it.16:39
rakhmerovhttps://review.openstack.org/#/c/406740/16:39
rakhmerovI didn't have a chance to review this patch16:39
rakhmerovmaybe it introduces some issue16:39
d0ugalyeah, so that patch did introduce the line I just changed.16:40
rakhmerov:)16:40
jpichOther places in that patch seem to expect utc16:41
openstackgerritDougal Matthews proposed openstack/mistral: Testing without utc  https://review.openstack.org/41371316:41
d0ugalwow gerrit is slow today16:41
rakhmerovd0ugal: so now tests pass only if local time is UTC ?16:41
d0ugalrakhmerov: I don't really understand yet.16:42
rakhmerovok16:42
d0ugalI think the problem might be that we go from a datetime with a timzone to the epoch seconds which doesn't have a timezone16:42
rakhmerovgerrit is also very slow16:43
d0ugalthen we convert that back to a datetime16:43
rakhmerovooh, you said gerrit, yes (i thought about something different)16:43
d0ugal:-D16:43
rakhmerovmaybe, yes16:43
openstackgerritDougal Matthews proposed openstack/mistral: [WIP] Testing without utc  https://review.openstack.org/41371316:45
d0ugalI will need to stop shortly, but I guess I will look into this tomorrow.16:46
rakhmerovplease do16:46
rakhmerovI'm off for today too16:46
d0ugalThat CI test patch causes another totally unrelated unit test to fail for me locally16:52
d0ugallol16:52
d0ugalmistral.tests.unit.api.v2.test_tasks.TestTasksController.test_put_current_task_in_error16:52
d0ugaljpich: how do you change your tz? just in gnome settings?16:53
*** Kiall has quit IRC16:53
jpichd0ugal: rm /etc/localtime, ln -s /usr/share/zoneinfo/America/Chicago /etc/localtime (as sudo)16:53
d0ugalfancy16:54
jpichI like the style, changing tz by symlink :)16:54
d0ugalkinda scary16:54
d0ugalokay, I might use that tomorrow.16:55
jpichGerrit seems like it might need a reboot, getting a lot of proxy errors16:55
jpichGood luck!16:55
d0ugalYeah, gerrit is so so slow16:55
d0ugalpartly why I am leaving :)16:55
*** openstack has quit IRC17:02
*** openstack has joined #openstack-mistral17:04
*** jpich has quit IRC17:08
*** mgershen has quit IRC17:17
*** bobh has joined #openstack-mistral17:37
*** bobh has quit IRC17:46
*** dprince has quit IRC17:46
*** dprince has joined #openstack-mistral17:46
openstackgerritRenat Akhmerov proposed openstack/mistral: Add rally tests for 'join': 100 and 500 parallel tasks  https://review.openstack.org/41358617:53
*** jtomasek has quit IRC19:22
*** dprince has quit IRC19:34
*** weshay is now known as weshay_relocatin19:42
*** sharatss has quit IRC20:30
*** sharatss has joined #openstack-mistral20:30
*** chlong has quit IRC20:36
*** chlong has joined #openstack-mistral20:52
*** weshay_relocatin is now known as weshay21:05
*** sharatss has quit IRC21:58
openstackgerritLingxian Kong proposed openstack/mistral: Role based resource access control - get workflows  https://review.openstack.org/41379122:02
*** chlong has quit IRC22:04
*** bobh has joined #openstack-mistral22:14
*** bobh has quit IRC22:31
*** bobh has joined #openstack-mistral23:46
*** thrash is now known as thrash|g0ne23:48
*** catintheroof has joined #openstack-mistral23:48
*** bobh has quit IRC23:51

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!