| *** katkapilatova has quit IRC | 00:11 | |
| *** dprince has quit IRC | 00:58 | |
| *** bobh has joined #openstack-mistral | 01:10 | |
| *** bobh has quit IRC | 01:11 | |
| *** pengdake has joined #openstack-mistral | 01:27 | |
| *** harlowja has quit IRC | 01:59 | |
| *** zhurong has joined #openstack-mistral | 02:29 | |
| *** itlinux has quit IRC | 02:32 | |
| *** gkadam has joined #openstack-mistral | 02:42 | |
| *** gkadam has quit IRC | 02:46 | |
| *** hardikjasani has joined #openstack-mistral | 03:29 | |
| *** zhurong has quit IRC | 03:53 | |
| rakhmerov | pgaxatte: hi | 03:54 |
|---|---|---|
| rakhmerov | maybe they are disabled in the configuration? | 03:54 |
| rakhmerov | [cron_trigger] | 03:55 |
| rakhmerov | enabled = ... | 03:56 |
| rakhmerov | d0ugal: hi, https://review.openstack.org/#/c/550864 | 04:08 |
| *** harlowja has joined #openstack-mistral | 04:11 | |
| *** pengdake has quit IRC | 04:30 | |
| *** gkadam has joined #openstack-mistral | 04:33 | |
| *** harlowja has quit IRC | 04:50 | |
| *** gkadam has quit IRC | 04:55 | |
| *** pengdake has joined #openstack-mistral | 05:54 | |
| *** gkadam has joined #openstack-mistral | 06:35 | |
| *** hardikjasani has left #openstack-mistral | 06:47 | |
| *** gkadam has quit IRC | 07:03 | |
| *** gkadam has joined #openstack-mistral | 07:07 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/mistral-tempest-plugin master: Updated from global requirements https://review.openstack.org/553197 | 07:10 |
| *** gkadam has quit IRC | 07:15 | |
| *** apetrich has quit IRC | 07:36 | |
| *** apetrich has joined #openstack-mistral | 07:48 | |
| openstackgerrit | Merged openstack/mistral-lib master: Updated from global requirements https://review.openstack.org/553196 | 07:54 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-mistralclient master: Updated from global requirements https://review.openstack.org/553253 | 07:59 |
| *** gkadam has joined #openstack-mistral | 08:00 | |
| *** AlexeyAbashkin has joined #openstack-mistral | 08:13 | |
| pgaxatte | thanks rakhmerov, i'll check that | 08:26 |
| rakhmerov | yep | 08:30 |
| pgaxatte | it's not clear to me what part of mistral handles the cron-triggers, is it the event-engine? | 08:33 |
| *** jpich has joined #openstack-mistral | 09:03 | |
| pgaxatte | rakhmerov: that didn't seem to help | 09:03 |
| pgaxatte | it seems a bit random whether or not the cron are executed | 09:03 |
| pgaxatte | it seems like sometime it executes when i delete them | 09:04 |
| pgaxatte | but they are never really scheduled | 09:04 |
| pgaxatte | to test I'm using the pattern '* * * * *', maybe it's too much? | 09:04 |
| pgaxatte | (with a count) | 09:04 |
| openstackgerrit | Merged openstack/python-mistralclient master: Updated from global requirements https://review.openstack.org/553253 | 09:18 |
| d0ugal | pgaxatte: it is a bit odd - but the Mistral API process actually triggers the cron workflows. | 09:18 |
| d0ugal | pgaxatte: I would try a less frequent cron trigger - I have noticed a bug before if they are too frequent | 09:19 |
| pgaxatte | ok so do I even need event-engine? (everything else works anyway without it) | 09:19 |
| d0ugal | do you really need it every minute? | 09:19 |
| d0ugal | pgaxatte: no, I don't think you need the event-engine. | 09:19 |
| pgaxatte | no no it's just that I don't want to wait for the tests :D | 09:19 |
| d0ugal | :) | 09:19 |
| pgaxatte | side question then: what's the event-engine for? :D | 09:20 |
| d0ugal | pgaxatte: I guess you must be running Mistral master? | 09:21 |
| d0ugal | sorry, ignore that. It is in the last release too | 09:21 |
| d0ugal | (I was thinking of the notifier, it is in master only) | 09:21 |
| pgaxatte | d0ugal: I'm running pike | 09:22 |
| d0ugal | pgaxatte: oh, old ;) | 09:22 |
| pgaxatte | d0ugal: I can't go higher because of dependencies with higher version of oslo | 09:22 |
| d0ugal | pgaxatte: I gotta be honest and say I have never used the event engine, so I am not sure what it is for | 09:23 |
| d0ugal | I should know. | 09:23 |
| pgaxatte | d0ugal: paired with a newton infra so it's more archeology at this point :D | 09:23 |
| d0ugal | wow, brave :) | 09:23 |
| d0ugal | pgaxatte: I think it may allow you to trigger workflows based on events, but I might be wrong. | 09:24 |
| pgaxatte | d0ugal: yeah, migration is in the boxes we'll see how it goes :) | 09:24 |
| pgaxatte | anyway, I guess I'll let the event-engine aside :) | 09:24 |
| pgaxatte | d0ugal: sounds reasonable | 09:24 |
| d0ugal | I think they used the event-engine in this: https://www.openstack.org/videos/barcelona-2016/building-self-healing-applications-with-aodh-zaqar-and-mistral | 09:25 |
| *** shardy has joined #openstack-mistral | 09:25 | |
| d0ugal | I should watch it again and see :) | 09:25 |
| pgaxatte | oh thanks, i'll check that out! | 09:26 |
| pgaxatte | d0ugal: I'm going to schedule something less frequent like */10 * * * * without the event-engin and report back :) | 09:27 |
| d0ugal | pgaxatte: sounds good. | 09:28 |
| d0ugal | pgaxatte: My most frequent cron trigger runs every hour :) | 09:29 |
| d0ugal | pgaxatte: btw, one issue you may find is that the cron trigger creates quite high load - it polls the database every second. | 09:29 |
| pgaxatte | d0ugal: ultimately this is something our clients would use so if there a frequency restriction, it would be nice to know it in advance :) | 09:29 |
| d0ugal | I added a way to reduce that but it is only in queens. | 09:30 |
| pgaxatte | is that related to the "Starting Scheduler loop / Scheduler captured 0 delayed calls." spams in the logs? | 09:31 |
| d0ugal | Yeah | 09:31 |
| d0ugal | because you can actualy make the cron trigger run every second! | 09:31 |
| d0ugal | (use 6 *'s) | 09:31 |
| pgaxatte | that's intense (and undocumented :D) | 09:32 |
| d0ugal | The only way to restrict it was added in queens | 09:32 |
| d0ugal | WIth these two patches: https://github.com/openstack/mistral/commit/a1ab5d85cc2db7278c7b1977649f1cf51dc05eb5#diff-0c464ead4ce4f0d1c61c80bd163ab40c and https://github.com/openstack/mistral/commit/24be74635a2ffc45a9fe26b21a737a832b3e34f8 | 09:32 |
| d0ugal | One of them is quite big, but it is all indentation changes, so quite a simple change | 09:33 |
| *** pengdake has quit IRC | 09:34 | |
| pgaxatte | d0ugal: ok, thanks for the pointer | 09:35 |
| pgaxatte | I might have to upgrade to queens but I fear the oslo upgrade :) | 09:36 |
| d0ugal | Yeah, that would be tough. | 09:36 |
| pgaxatte | d0ugal: still no execution with a */10 schedule | 09:38 |
| pgaxatte | Next execution time: 2018-03-15 09:30:00 (UTC) | 09:38 |
| openstackgerrit | Renat Akhmerov proposed openstack/mistral master: Fix initializers of standard actions https://review.openstack.org/553298 | 09:38 |
| d0ugal | pgaxatte: hmm, that is odd | 09:39 |
| d0ugal | pgaxatte: anything in the api log? | 09:39 |
| *** pengdake has joined #openstack-mistral | 09:39 | |
| pgaxatte | some infrequent | 09:40 |
| pgaxatte | [Thu Mar 15 09:37:38.862895 2018] [wsgi:error] [pid 2967:tid 140132377466624] Unexpected error during heartbeart thread processing, retrying... | 09:40 |
| pgaxatte | [Thu Mar 15 09:37:38.866136 2018] [wsgi:error] [pid 2967:tid 140132377466624] [671ce285-e944-4612-83a5-9a8970447938] AMQP server on XXX.XXX.XXX.XXX:5672 is unreachable: . Trying again in 1 seconds. Client port: 33240 | 09:40 |
| *** pengdake has quit IRC | 09:40 | |
| pgaxatte | but that's no even close to the scheduled time | 09:40 |
| d0ugal | no, that sounds unrelated | 09:41 |
| d0ugal | pgaxatte: What tz in your server set to? | 09:41 |
| d0ugal | I do remember there was a bug if the server wasn't UTC - but I forget when that was fixed. | 09:41 |
| pgaxatte | /etc/localtime -> /usr/share/zoneinfo/Etc/UTC | 09:43 |
| pgaxatte | I'm on ubuntu 16.04 | 09:43 |
| *** pengdake has joined #openstack-mistral | 09:50 | |
| pgaxatte | d0ugal: I'm using mistral-api with wsgi, is there a way to increase the log level? I'm not sure just putting apache's LogLevel to debug is enough (I don't much more logs) | 10:02 |
| pgaxatte | or maybe I should run it outside apache | 10:02 |
| d0ugal | pgaxatte: There are a few options in the log to change the logging level. You can set debug=True, that is probably the easiest way | 10:09 |
| d0ugal | rakhmerov: Any tips for using join and branches that only sometimes join? | 10:09 |
| d0ugal | I want something like join: all-started :) | 10:10 |
| *** shardy has quit IRC | 10:18 | |
| pgaxatte | d0ugal: yeah I'm using debug=True, I'll see if running standalone improves the level of details | 10:18 |
| rakhmerov | d0ugal: there's only "join: N" where N defines a minimal number of branches that will trigger it | 10:19 |
| *** shardy has joined #openstack-mistral | 10:19 | |
| d0ugal | rakhmerov: it seems a bit limited. I guess I'll have to do less of this in parallel | 10:20 |
| rakhmerov | yeah, sorry | 10:20 |
| rakhmerov | this is the only option | 10:21 |
| d0ugal | no need to be sorry :) | 10:21 |
| rakhmerov | well, basically it's the only type of join we have | 10:21 |
| d0ugal | rakhmerov: I wonder if a "skip" task policy could be useful - then the task could still complete and be joined (but the aciton isn't executed) | 10:21 |
| rakhmerov | "all" and "one" are special cases of that | 10:21 |
| rakhmerov | one = 1, all = unlimited | 10:21 |
| rakhmerov | d0ugal: hm.. interesting idea | 10:21 |
| d0ugal | rakhmerov: it should be quite easy to implement too. I am not sure if is a good idea :) | 10:23 |
| rakhmerov | d0ugal: so something like.. | 10:25 |
| rakhmerov | skip: <% boolean expression %> | 10:26 |
| rakhmerov | ? | 10:26 |
| d0ugal | rakhmerov: yeah, exactly | 10:26 |
| rakhmerov | hm.. | 10:26 |
| rakhmerov | yeah.. | 10:26 |
| d0ugal | http://paste.openstack.org/show/701546/ | 10:26 |
| rakhmerov | worths a spec a least :) | 10:26 |
| d0ugal | so something like that paste ^ | 10:27 |
| rakhmerov | yes | 10:27 |
| rakhmerov | got it | 10:27 |
| d0ugal | http://paste.openstack.org/show/701547/ - this is how you might try to do it now, but it would fail | 10:27 |
| d0ugal | the second paste is exactly what I am doing :) | 10:27 |
| d0ugal | rakhmerov: or maybe we could do something with the tasks() yaql function | 10:29 |
| d0ugal | join: tasks(execution().id, state=RUNNING) | 10:29 |
| d0ugal | we would need a state that means "started" | 10:29 |
| rakhmerov | :) | 10:30 |
| rakhmerov | well, theoretically we can make N dynamic, yes | 10:30 |
| pgaxatte | d0ugal: haaa when I run mistral-wsgi-api directly, it executes the crons, on time! :) | 10:34 |
| pgaxatte | but i still don't have much debugging available | 10:35 |
| d0ugal | pgaxatte: interesting. I wonder why it would be different under wsgi | 10:36 |
| pgaxatte | d0ugal: me too, here my apache conf for reference. There doesn't seem to be anything really special: http://paste.openstack.org/show/qH5FMjHYmfgfzVHWWEa9/ | 10:38 |
| d0ugal | rakhmerov: oh, I just had one last idea. | 10:40 |
| d0ugal | rakhmerov: sorry if you are busy :-D | 10:40 |
| *** pengdake has quit IRC | 10:41 | |
| d0ugal | actually, I'm not sure I like this one. I was thinking of a new std action that waits for tasks to finish. std.wait(task_id, task_id, ...) | 10:42 |
| d0ugal | That could be a workaround, but I guess the real solution is to make join better. | 10:42 |
| d0ugal | rakhmerov: actually, I managed to do what I need - I just had to use a couple of extra joins. | 11:17 |
| *** Alexey_Abashkin has joined #openstack-mistral | 11:20 | |
| *** igormarnat_ has joined #openstack-mistral | 11:23 | |
| *** ttsiouts has quit IRC | 11:27 | |
| *** igormarnat has quit IRC | 11:27 | |
| *** igormarnat_ is now known as igormarnat | 11:27 | |
| *** AlexeyAbashkin has quit IRC | 11:28 | |
| *** FL1SK has quit IRC | 11:28 | |
| *** Alexey_Abashkin is now known as AlexeyAbashkin | 11:28 | |
| *** ttsiouts has joined #openstack-mistral | 11:29 | |
| *** thrash|g0ne is now known as thrash | 12:05 | |
| *** dprince has joined #openstack-mistral | 12:15 | |
| *** FL1SK has joined #openstack-mistral | 12:50 | |
| openstackgerrit | Brad P. Crochet proposed openstack/mistral master: Clarify usage of break-on and continue-on https://review.openstack.org/553386 | 12:56 |
| pgaxatte | d0ugal: just wanted to confirm the scheduler works with a simple echo scheduled every minute. Now I just have to figure out why it's not working when using apache | 13:55 |
| *** jaosorior has quit IRC | 14:03 | |
| *** bobh has joined #openstack-mistral | 14:10 | |
| *** bobh has quit IRC | 14:27 | |
| *** jaosorior has joined #openstack-mistral | 14:47 | |
| openstackgerrit | Toure Dunnon proposed openstack/mistral master: Workflow Error Analysis [WIP] https://review.openstack.org/455447 | 14:47 |
| *** jtomasek_ has joined #openstack-mistral | 14:54 | |
| *** bobh has joined #openstack-mistral | 14:57 | |
| *** bobh has quit IRC | 15:03 | |
| *** bobh has joined #openstack-mistral | 15:07 | |
| *** bobh has quit IRC | 15:09 | |
| *** bobh has joined #openstack-mistral | 15:09 | |
| *** thrash is now known as thrash|biab | 15:24 | |
| openstackgerrit | Toure Dunnon proposed openstack/python-mistralclient master: Workflow Error Analysis Client. https://review.openstack.org/452901 | 15:27 |
| *** itlinux has joined #openstack-mistral | 15:29 | |
| *** jtomasek_ has quit IRC | 15:39 | |
| *** rbrady is now known as rbrady-afk | 15:44 | |
| *** harlowja has joined #openstack-mistral | 16:03 | |
| *** thrash|biab is now known as thrash | 16:04 | |
| *** jaosorior has quit IRC | 16:21 | |
| pgaxatte | d0ugal, rakhmerov: could having multiple workers in apache cause a problem when running the periodic tasks? | 16:31 |
| pgaxatte | since there would be multiple threads trying to run the periodic tasks | 16:31 |
| *** AlexeyAbashkin has quit IRC | 16:32 | |
| d0ugal | pgaxatte: Good question. Possibly. But I would expect it to be run multiple times rather than 0 times. | 16:49 |
| pgaxatte | do you run it with mod_wsgi or standalone? | 16:50 |
| d0ugal | pgaxatte: standalone | 16:50 |
| d0ugal | pgaxatte: but I do have several API processes. http://paste.openstack.org/show/701955/ | 16:51 |
| pgaxatte | d0ugal: as soon as I run the server standalone, it starts running the periodic tasks that where not executed previously ^^ | 16:53 |
| pgaxatte | d0ugal: when I run one instance of the api and start a second one, it returns immediately | 16:55 |
| pgaxatte | d0ugal: something's fishy here :) | 16:55 |
| d0ugal | yeah, that is strange | 16:55 |
| d0ugal | pgaxatte: is the cron trigger the main feature you need? | 16:55 |
| d0ugal | Unfortunately, it is one of the least polished Mistral features imo | 16:55 |
| pgaxatte | d0ugal: no that makes sense: port already in use... | 16:55 |
| d0ugal | oh, of course :) | 16:55 |
| pgaxatte | d0ugal: yeah it kinda is :) | 16:55 |
| *** gkadam has quit IRC | 17:03 | |
| *** harlowja has quit IRC | 17:09 | |
| *** gkadam has joined #openstack-mistral | 17:25 | |
| *** shardy has quit IRC | 17:47 | |
| *** jpich has quit IRC | 17:49 | |
| *** rbrady-afk is now known as rbrady | 17:59 | |
| *** gkadam has quit IRC | 18:00 | |
| *** harlowja has joined #openstack-mistral | 18:20 | |
| *** rbrady has quit IRC | 18:23 | |
| *** rbrady has joined #openstack-mistral | 18:33 | |
| *** rbrady has joined #openstack-mistral | 18:33 | |
| *** rbrady has quit IRC | 18:39 | |
| *** AlexeyAbashkin has joined #openstack-mistral | 18:54 | |
| *** rbrady has joined #openstack-mistral | 18:54 | |
| *** rbrady has quit IRC | 18:54 | |
| *** rbrady has joined #openstack-mistral | 18:54 | |
| *** AlexeyAbashkin has quit IRC | 18:58 | |
| *** dprince has quit IRC | 19:56 | |
| openstackgerrit | Toure Dunnon proposed openstack/mistral master: Workflow Error Analysis [WIP] https://review.openstack.org/455447 | 19:59 |
| openstackgerrit | Toure Dunnon proposed openstack/python-mistralclient master: Workflow Error Analysis Client. https://review.openstack.org/452901 | 20:00 |
| openstackgerrit | Vitalii Solodilov proposed openstack/mistral master: Fixed the broken MySQL job https://review.openstack.org/547808 | 20:56 |
| *** mburrows has quit IRC | 21:21 | |
| *** itlinux has quit IRC | 21:39 | |
| *** jtomasek has quit IRC | 21:46 | |
| *** bobh has quit IRC | 21:53 | |
| *** bobh has joined #openstack-mistral | 21:56 | |
| *** bobh has quit IRC | 21:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!