| *** itlinux has joined #openstack-mistral | 00:12 | |
| *** AlexeyAbashkin has joined #openstack-mistral | 00:18 | |
| *** AlexeyAbashkin has quit IRC | 00:22 | |
| *** harlowja has quit IRC | 01:07 | |
| *** AlexeyAbashkin has joined #openstack-mistral | 01:18 | |
| *** AlexeyAbashkin has quit IRC | 01:22 | |
| *** pengdake has joined #openstack-mistral | 01:29 | |
| *** AlexeyAbashkin has joined #openstack-mistral | 02:18 | |
| *** AlexeyAbashkin has quit IRC | 02:23 | |
| *** zhurong has joined #openstack-mistral | 02:40 | |
| *** zhurong has quit IRC | 03:44 | |
| rakhmerov | d0ugal: hi | 04:34 |
|---|---|---|
| rakhmerov | sorry, missed your last message | 04:34 |
| *** harlowja has joined #openstack-mistral | 04:39 | |
| *** openstackgerrit has joined #openstack-mistral | 04:45 | |
| openstackgerrit | Renat Akhmerov proposed openstack/mistral master: Add a release note for the 'pause' command fix https://review.openstack.org/554794 | 04:45 |
| *** harlowja has quit IRC | 04:56 | |
| *** hardikjasani has joined #openstack-mistral | 04:57 | |
| *** zhurong has joined #openstack-mistral | 06:05 | |
| *** gkadam has joined #openstack-mistral | 06:50 | |
| *** jaosorior has quit IRC | 07:05 | |
| apetrich | rakhmerov, thanks for the release note | 07:12 |
| rakhmerov | np ) | 07:12 |
| *** jaosorior has joined #openstack-mistral | 07:44 | |
| *** AlexeyAbashkin has joined #openstack-mistral | 07:55 | |
| d0ugal | rakhmerov: np | 08:14 |
| d0ugal | rakhmerov: I was just going to show you a small tool I made | 08:14 |
| d0ugal | rakhmerov: http://paste.openstack.org/show/706040/ | 08:14 |
| d0ugal | Allows you to easily see a execution and it's tasks, actions and sub-workflows | 08:14 |
| d0ugal | The formatting isn't perfect, but I have found it really useful | 08:15 |
| d0ugal | It just uses mistralclient | 08:15 |
| *** danpawlik has joined #openstack-mistral | 08:15 | |
| danpawlik | Good morning, I was checking puppet integration manifests and as I see in my PS https://review.openstack.org/#/c/552833/ package mistral-event-engine is partialy installed because of mistral-engine package is installed (im talking about Ubuntu). Is this a bug or not? I see that in RDO they doesn't have any error | 08:18 |
| danpawlik | and http://logs.openstack.org/33/552833/2/check/puppet-openstack-integration-4-scenario003-tempest-ubuntu-xenial/f480d58/logs/dpkg-l.txt.gz package has flags iF | 08:18 |
| danpawlik | zigo: maybe you know more details | 08:18 |
| d0ugal | danpawlik: I believe it is an optional component, so that is probably why | 08:19 |
| d0ugal | but honestly I am not that familiar with the mistral-event-engine. I have been asked about it a couple of times recently, so I guess I should learn :) | 08:20 |
| rakhmerov | d0ugal: aah, cool. Reminds me about one BP that's been in progress for a while now ;) | 08:21 |
| d0ugal | rakhmerov: indeed, I needed something sooner | 08:21 |
| rakhmerov | I see | 08:21 |
| rakhmerov | d0ugal: can you share it somewhere? | 08:22 |
| d0ugal | rakhmerov: https://github.com/d0ugal/tripleo-util/blob/master/scripts/wf-report | 08:22 |
| rakhmerov | ok, thanks | 08:22 |
| d0ugal | rakhmerov: it is quite hacky, I'd like to tidy it up. I wonder if it would make sense as an addition to mistralclient | 08:22 |
| d0ugal | rakhmerov: I am not convinced we really need API changes for something like this - at least not for a first version | 08:22 |
| d0ugal | (we can change the API if we find some parts are too slow etc.) | 08:22 |
| rakhmerov | well, yeah, I think we wanted API changed because we predicted performance problems mostly | 08:24 |
| rakhmerov | as far I as I remember | 08:24 |
| rakhmerov | but as a prototype something like this would be enough for now | 08:24 |
| pgaxatte | o/ danpawlik, I asked recently about the event-engine :) | 08:24 |
| d0ugal | rakhmerov: right, but I don't think we need to add a new API endpoint - probably just add some options to what we have | 08:25 |
| rakhmerov | d0ugal: also, just letting you know that I won't be too active till the end of the week. Have to focus on some internal things for now | 08:25 |
| d0ugal | rakhmerov: sure | 08:25 |
| *** mcdoker181818 has joined #openstack-mistral | 08:25 | |
| danpawlik | good :) | 08:27 |
| danpawlik | pgaxatte: I guess zigo is one of the person who can make some modification in debianization :P | 08:27 |
| danpawlik | pgaxatte: he is a master of Debian and debian packages | 08:28 |
| zigo | danpawlik: I can. | 08:44 |
| zigo | danpawlik: How may I help? | 08:45 |
| zigo | danpawlik: What you reported looks like an error in *Ubuntu*. | 08:46 |
| zigo | I do Debian ... | 08:47 |
| zigo | My package switched to Python 3, they didn't in Ubuntu. | 08:47 |
| zigo | So it becomes quite different these days. | 08:47 |
| danpawlik | zigo: wow, I was thinking that Canonical takes debianization from debian :P | 08:49 |
| danpawlik | zigo: | 08:49 |
| zigo | danpawlik: They did, but not for Queens, since I switched all to Python 3. | 08:50 |
| danpawlik | zigo: there is small change in package debianization, pgaxatte will send you | 08:50 |
| zigo | danpawlik: By all means, please do, I'd love to have more contribs ! :) | 08:50 |
| zigo | BTW, switching to Py3 was mostly painless, I'm happy I did that. | 08:50 |
| zigo | Only manila-ui is completely broken, everything else just work ... | 08:51 |
| pgaxatte | danpawlik, zigo: actually my problem was about the event-engine which is not present on Debian | 08:52 |
| danpawlik | zigo: congrats ! | 08:53 |
| *** milind has joined #openstack-mistral | 08:58 | |
| openstackgerrit | Merged openstack/mistral master: Ensure workflow check is retried on DB error https://review.openstack.org/547069 | 08:59 |
| milind | hello, i am using openstack mistral for execution of workflows. The workflow consists of few actions. I have observed that the actions are taking time to invoke once the previous action execution completes. This is causing my project to slow down. Any idea how to troubleshoot this? Googled many things but didn | 09:00 |
| *** jpich has joined #openstack-mistral | 09:02 | |
| *** zhurong has quit IRC | 09:04 | |
| zigo | pgaxatte: So, I should add a new mistral-event-engine package, together with init/systemd script and all? | 09:12 |
| *** zhurong has joined #openstack-mistral | 09:13 | |
| pgaxatte | zigo: that would be nice :) I'm not sure I can submit anything with a guest account on salsa | 09:14 |
| zigo | pgaxatte: I don't see any console-script with event-engine in setup.cfg | 09:14 |
| zigo | pgaxatte: You can, you just do a pull request. | 09:15 |
| pgaxatte | alright, I'll try to do that and report back | 09:15 |
| zigo | pgaxatte: You can also join the openstack-team on Salsa, and I'll approve it, but then it's best if you do that on a branch. | 09:16 |
| zigo | pgaxatte: Also join the packaging list here: https://lists.alioth.debian.org/cgi-bin/mailman/listinfo/openstack-devel/ | 09:16 |
| pgaxatte | zigo: no problem, for now I'll just stick with making a branch and once I'm done i'll ping you | 09:16 |
| zigo | Cheers. | 09:17 |
| *** mcdoker181818 has quit IRC | 09:18 | |
| d0ugal | milind: Which version of Mistral are you using? | 09:51 |
| milind | how to check that ? is there any command.. i am using the mistral on docker container which i pulled from repo | 10:12 |
| milind | root@54582b021ef5:/# mistral --version mistral 3.1.2 | 10:13 |
| milind | 3.1.2 | 10:13 |
| d0ugal | milind: That is the python-mistralclient version. How did you install Mistral? You will need to check with the packaging you use | 10:20 |
| d0ugal | pip freeze | grep mistral | 10:20 |
| d0ugal | rpm -qa | grep mistral | 10:20 |
| d0ugal | or some debian command I forget :) | 10:21 |
| rakhmerov | d0ugal, apetrich, anyone else: who knows much about Python memory management? | 10:25 |
| *** pengdake has quit IRC | 10:25 | |
| d0ugal | rakhmerov: a bit, what is the question? | 10:25 |
| rakhmerov | sec.. | 10:25 |
| rakhmerov | we see the problem in production: Mistral processes take more and more memory over time reaching some really gigantic values like 20G | 10:26 |
| d0ugal | ouch | 10:26 |
| rakhmerov | we did a lot of tests in the past and concluded that Python objects are deallocated correctly | 10:26 |
| rakhmerov | it just seems like Python doesn't release memory correctly for some reason | 10:27 |
| pgaxatte | rakhmerov: did you try with different python versions? | 10:27 |
| rakhmerov | and we didn't find any ways to limit its memory consumption (like Xmax parameter in Java) | 10:27 |
| rakhmerov | pgaxatte: yes, seems like Python 3 doesn't have this problem ) | 10:27 |
| d0ugal | Interesting. | 10:27 |
| rakhmerov | at the first glance | 10:27 |
| rakhmerov | yeah | 10:27 |
| rakhmerov | Mistral itself doesn't seem to leak | 10:28 |
| d0ugal | I think that could suggest we have some reference cycles - and Python 3 is smarter about detecting them | 10:28 |
| d0ugal | rakhmerov: oh, if mistral doesn't leak - what does? | 10:28 |
| rakhmerov | Python | 10:28 |
| d0ugal | How do you know it isn't Mistral? | 10:28 |
| rakhmerov | I did tests that showed that a number of objects in Heap didn't grow over time (on similar API operations, for example) but the overall resident memory for the Python process was consistently growing | 10:29 |
| d0ugal | Interesting | 10:30 |
| d0ugal | rakhmerov: do you have your findings you can share somewhere? | 10:30 |
| rakhmerov | locally on my Mac it was ok, the memory was freed up after it reached a certain value (~1G) but on our production it doesn't seem to happen | 10:30 |
| rakhmerov | d0ugal: no, I didn't keep them, it was about a year ago | 10:31 |
| d0ugal | rakhmerov: oh, a long time :) | 10:31 |
| d0ugal | rakhmerov: Is the mistral under heavy load? or how long has it been running? | 10:31 |
| rakhmerov | d0ugal: so the fundamental question: is there any way to limit memory consumption for a python process? | 10:31 |
| d0ugal | rakhmerov: I am trying to see how I can reproduce it | 10:31 |
| d0ugal | rakhmerov: I have never needed to put limits on a Python process before - so I am not sure. | 10:32 |
| rakhmerov | d0ugal: well, by repeating highly memory consuming operations (e.g. task-list when there's a lot of tasks) | 10:33 |
| *** vienin has joined #openstack-mistral | 10:33 | |
| d0ugal | Every memory issue I have had has been in the user code - not Python itself :) | 10:33 |
| rakhmerov | ok | 10:33 |
| d0ugal | rakhmerov: how many tasks would you consider lots? (just so I can test something similar) | 10:33 |
| rakhmerov | I'm not sure if that matters | 10:34 |
| d0ugal | ok | 10:34 |
| *** mcdoker181818 has joined #openstack-mistral | 10:34 | |
| rakhmerov | as far as I remember, I was just doing something simple in the loop | 10:34 |
| d0ugal | I'll try that | 10:34 |
| rakhmerov | :) | 10:34 |
| d0ugal | rakhmerov: btw, I have had success with this in the past: https://mg.pov.lt/objgraph/ | 10:34 |
| milind | ok let me check | 10:34 |
| d0ugal | but it has been a while. | 10:34 |
| rakhmerov | ok, I thought may be you or some of your colleagues encountered something similar in the past | 10:35 |
| rakhmerov | d0ugal: yes, I used this lib too | 10:35 |
| apetrich | I've hit that before | 10:35 |
| rakhmerov | and some others | 10:35 |
| apetrich | but never able to get a satisfying solution | 10:35 |
| rakhmerov | apetrich: ooh really? | 10:35 |
| rakhmerov | apetrich: what did you observe? | 10:36 |
| apetrich | pretty much that. we would release objects but python would keep memory | 10:36 |
| rakhmerov | yeah | 10:37 |
| apetrich | in the end we did what things like phusion would do that limit a run of a worker for that many tasks, kill and respawn it | 10:37 |
| rakhmerov | right | 10:37 |
| vienin | Hi mistral team ! I'm just figuring out that "scope" property is not available for workbook creation, only for workflows. Is it requisite, or just missing ? Thanks for your pointers ! | 10:37 |
| rakhmerov | what was the operating system? | 10:37 |
| rakhmerov | apetrich: ^ | 10:37 |
| apetrich | linux | 10:38 |
| rakhmerov | ok, yep | 10:38 |
| apetrich | it was on our data gathering servers so ubuntu if I'm not mistaken | 10:38 |
| rakhmerov | I've read that on Win Python memory management works differently: it frees up memory immediately after gc.collect() | 10:38 |
| apetrich | I spent a silly amount of time drilling down that | 10:38 |
| rakhmerov | ok | 10:38 |
| rakhmerov | me too ) | 10:38 |
| rakhmerov | and now we're facing it again.. | 10:39 |
| rakhmerov | vienin: I believe this is a bug, can you file it pls so that we don't lose it? | 10:39 |
| apetrich | I didn't test on win | 10:39 |
| vienin | (in way to create "public" workbooks through the api) | 10:39 |
| rakhmerov | vienin: gotcha | 10:39 |
| vienin | Ok, you want I create the issue in launchpad, right ? | 10:40 |
| rakhmerov | vienin: you may also want to check our LP first, it seems to me that a similar bug may already exist | 10:40 |
| rakhmerov | vienin: yes | 10:40 |
| d0ugal | milind: the versions you sent me are not for mistral-server - how did you install mistral? | 10:40 |
| vienin | Ok, I found this issue https://bugs.launchpad.net/python-mistralclient/+bug/1746681 that can be related, but at all, do you have any other existing ticket in mind ? | 10:41 |
| openstack | Launchpad bug 1746681 in python-mistralclient "Scope attribute not shown in List, Get operations" [Undecided,New] - Assigned to Theodoros Tsioutsias (ttsiouts) | 10:41 |
| d0ugal | vienin: I am not aware of a way to make a public workbook. | 10:42 |
| d0ugal | vienin: workbooks are not very useful in the API, having access to them doesn't give you anything really | 10:42 |
| vienin | OK my bad :) So do you recommend to do use the workbook api to register workflows ? No pb for us to do that, workbook are just used to gather workflow in a single file | 10:44 |
| vienin | *to not use | 10:44 |
| milind | d0ugal, i am using mistral via docker.. it was already present in the docker image. There are 3 docker images i am using for mistral. one is having mistral-api, one is hosting mistral-executor and one is having mistral-engine. | 10:44 |
| d0ugal | milind: aha, are you using the docker config in mistral's git repo? if so, then you are using the git master version of mistral | 10:45 |
| d0ugal | vienin: we use workbooks to create our workflows, but we never really use the workbook api after that - we do everything with the workflow api | 10:45 |
| d0ugal | vienin: I guess the question I have is: what is it that you want to use the workbook api for? | 10:45 |
| d0ugal | vienin: when you create the workbook, is the scope for the workflow correct? | 10:46 |
| rakhmerov | d0ugal: yes but I see no reason why "scope" shouldn't be available at workbook creation | 10:46 |
| d0ugal | vienin: it does sound like you have spotted a bug - we should correct the scope for the workbooks | 10:46 |
| d0ugal | rakhmerov: agreed | 10:46 |
| rakhmerov | the corresponding workflows could naturally inherit that scope | 10:46 |
| vienin | <d0ugal> We are using the worbook api for register all our workflows, but we want those workflow public, as we want they "exists" in all domains/projects | 10:47 |
| vienin | Ok, make sens for us, can I report the issue and propose a fix ? | 10:48 |
| rakhmerov | vienin: sure, go ahead | 10:49 |
| milind | d0ugal: yes , i am using the docker config. The base repo listed is "mistral_base" | 10:49 |
| rakhmerov | we'd appreciate that | 10:49 |
| rakhmerov | vienin: https://bugs.launchpad.net/python-mistralclient/+bug/1746681 is not exactly about this, so please file a new one | 10:50 |
| openstack | Launchpad bug 1746681 in python-mistralclient "Scope attribute not shown in List, Get operations" [Undecided,New] - Assigned to Theodoros Tsioutsias (ttsiouts) | 10:50 |
| vienin | <rakhmerov> Ok | 10:51 |
| d0ugal | vienin: thanks! | 10:52 |
| vienin | <d0ugal> <rakhmerov> Thanks for your reactivity, see you ! | 10:52 |
| rakhmerov | np | 10:52 |
| *** zhurong has quit IRC | 10:55 | |
| *** vienin has quit IRC | 11:00 | |
| *** katkapilatova has joined #openstack-mistral | 11:21 | |
| *** milind has quit IRC | 11:33 | |
| *** pengdake has joined #openstack-mistral | 11:58 | |
| *** dprince has joined #openstack-mistral | 12:06 | |
| d0ugal | rakhmerov: I am trying hard, but failing to see my memory usage increase :) | 12:18 |
| *** pengdake has quit IRC | 12:18 | |
| d0ugal | I am running this, in 4 tmux panes.... | 12:18 |
| d0ugal | while true; do mistral task-list --limit=-1 | wc -l; done | 12:18 |
| d0ugal | and I have almost 6k tasks now | 12:18 |
| d0ugal | I also have a loop creating new executions | 12:19 |
| d0ugal | everything is getting quite slow :) | 12:19 |
| *** dprince has quit IRC | 12:29 | |
| *** dprince has joined #openstack-mistral | 13:05 | |
| *** mcdoker181818 has quit IRC | 13:48 | |
| *** bobh has joined #openstack-mistral | 14:03 | |
| openstackgerrit | Dougal Matthews proposed openstack/mistral master: [WIP] Experimental work adding a Zaqar event publisher https://review.openstack.org/547666 | 14:03 |
| *** itlinux has quit IRC | 14:16 | |
| *** itlinux has joined #openstack-mistral | 14:17 | |
| *** gkadam_ has joined #openstack-mistral | 14:18 | |
| *** gkadam has quit IRC | 14:20 | |
| *** itlinux has quit IRC | 14:22 | |
| *** pengdake has joined #openstack-mistral | 14:45 | |
| *** pengdake has quit IRC | 15:00 | |
| *** thrash is now known as thrash|biab | 15:13 | |
| *** gkadam_ has quit IRC | 15:15 | |
| *** thrash|biab is now known as thrash | 15:43 | |
| *** itlinux has joined #openstack-mistral | 16:07 | |
| *** katkapilatova has quit IRC | 16:07 | |
| *** danpawlik has quit IRC | 16:11 | |
| *** itlinux has quit IRC | 16:15 | |
| *** AlexeyAbashkin has quit IRC | 16:29 | |
| *** danpawlik has joined #openstack-mistral | 16:37 | |
| *** thrash is now known as thrash|biab | 16:40 | |
| *** danpawlik has quit IRC | 16:42 | |
| *** itlinux has joined #openstack-mistral | 16:50 | |
| openstackgerrit | Vitalii Solodilov proposed openstack/mistral master: Fixed the Mistral build job https://review.openstack.org/554987 | 16:52 |
| openstackgerrit | Vitalii Solodilov proposed openstack/mistral master: Fixed the Mistral Docker build job https://review.openstack.org/554987 | 16:56 |
| *** bradjones has quit IRC | 17:03 | |
| *** danpawlik has joined #openstack-mistral | 17:07 | |
| *** danpawlik has quit IRC | 17:11 | |
| *** danpawlik has joined #openstack-mistral | 17:20 | |
| *** danpawlik has quit IRC | 17:25 | |
| *** jpich has quit IRC | 17:33 | |
| *** danpawlik has joined #openstack-mistral | 17:41 | |
| *** danpawlik has quit IRC | 17:45 | |
| *** thrash|biab is now known as thrash | 17:49 | |
| *** AlexeyAbashkin has joined #openstack-mistral | 18:03 | |
| *** danpawlik has joined #openstack-mistral | 18:12 | |
| *** danpawlik has quit IRC | 18:17 | |
| *** AlexeyAbashkin has quit IRC | 18:21 | |
| *** EmilienM is now known as mimi | 18:21 | |
| *** mimi is now known as EmilienM | 18:21 | |
| *** harlowja has joined #openstack-mistral | 18:24 | |
| *** hardikjasani has quit IRC | 18:28 | |
| *** danpawlik has joined #openstack-mistral | 18:52 | |
| *** danpawlik has quit IRC | 18:56 | |
| *** AlexeyAbashkin has joined #openstack-mistral | 19:17 | |
| *** AlexeyAbashkin has quit IRC | 19:21 | |
| *** dprince has quit IRC | 19:23 | |
| *** jaosorior has quit IRC | 19:23 | |
| *** dprince has joined #openstack-mistral | 19:23 | |
| *** danpawlik has joined #openstack-mistral | 19:29 | |
| *** danpawlik has quit IRC | 19:33 | |
| *** dprince has quit IRC | 19:43 | |
| *** jaosorior has joined #openstack-mistral | 19:44 | |
| openstackgerrit | Kevin Pouget proposed openstack/mistral master: Add missing query paramater `scope` to the workbook api https://review.openstack.org/555030 | 19:47 |
| *** danpawlik has joined #openstack-mistral | 20:02 | |
| *** danpawlik has quit IRC | 20:06 | |
| *** AlexeyAbashkin has joined #openstack-mistral | 20:18 | |
| *** AlexeyAbashkin has quit IRC | 20:22 | |
| *** jaosorior_ has joined #openstack-mistral | 20:33 | |
| *** danpawlik has joined #openstack-mistral | 20:33 | |
| *** jaosorior has quit IRC | 20:37 | |
| *** danpawlik has quit IRC | 20:38 | |
| *** danpawlik has joined #openstack-mistral | 21:07 | |
| *** danpawlik has quit IRC | 21:12 | |
| *** AlexeyAbashkin has joined #openstack-mistral | 21:17 | |
| *** AlexeyAbashkin has quit IRC | 21:21 | |
| openstackgerrit | Vitalii Solodilov proposed openstack/mistral master: Add new indexes to optimize `task().result` expression function https://review.openstack.org/555066 | 21:41 |
| *** fultonj has joined #openstack-mistral | 21:44 | |
| *** danpawlik has joined #openstack-mistral | 21:45 | |
| fultonj | my execution contains a task which runs a command, the command has returned 0 and i'd like my workflow to move on, however, task seems to be stuck in the running state | 21:45 |
| *** rbrady is now known as rbrady-afk | 21:46 | |
| fultonj | as my task is the last task, i tried telling the execution that everything is ok | 21:46 |
| fultonj | mistral execution-update -s SUCCESS $UUID | 21:46 |
| fultonj | there's a longer story here, but i'm basically wondering if i can tell my workflow to move on | 21:49 |
| *** danpawlik has quit IRC | 21:50 | |
| *** itlinux has quit IRC | 21:51 | |
| *** danpawlik has joined #openstack-mistral | 21:53 | |
| *** danpawlik has quit IRC | 21:57 | |
| *** danpawlik has joined #openstack-mistral | 22:24 | |
| *** danpawlik has quit IRC | 22:28 | |
| *** threestrands has joined #openstack-mistral | 22:29 | |
| *** threestrands has quit IRC | 22:30 | |
| *** threestrands has joined #openstack-mistral | 22:30 | |
| *** threestrands has quit IRC | 22:30 | |
| *** threestrands has joined #openstack-mistral | 22:30 | |
| *** bobh has quit IRC | 22:30 | |
| *** d0ugal has quit IRC | 22:34 | |
| *** d0ugal has joined #openstack-mistral | 22:37 | |
| *** thrash is now known as thrash|g0ne | 22:39 | |
| *** danpawlik has joined #openstack-mistral | 23:04 | |
| *** jtomasek has quit IRC | 23:08 | |
| *** danpawlik has quit IRC | 23:08 | |
| *** itlinux has joined #openstack-mistral | 23:20 | |
| *** danpawlik has joined #openstack-mistral | 23:39 | |
| *** danpawlik has quit IRC | 23:44 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!