| *** openstack has joined #openstack-watcher | 00:08 | |
| *** openstackgerrit has quit IRC | 00:17 | |
| *** openstackgerrit has joined #openstack-watcher | 00:18 | |
| *** edleafe has quit IRC | 00:49 | |
| *** edleafe has joined #openstack-watcher | 00:52 | |
| *** xenogear has joined #openstack-watcher | 01:01 | |
| *** thorst_ has quit IRC | 01:15 | |
| *** thorst_ has joined #openstack-watcher | 01:17 | |
| *** thorst_ has quit IRC | 01:24 | |
| *** thorst_ has joined #openstack-watcher | 01:52 | |
| *** thorst_ has quit IRC | 02:18 | |
| *** thorst_ has joined #openstack-watcher | 02:19 | |
| *** thorst_ has quit IRC | 02:28 | |
| *** thorst_ has joined #openstack-watcher | 03:10 | |
| *** thorst_ has quit IRC | 03:14 | |
| *** thorst_ has joined #openstack-watcher | 03:15 | |
| *** xenogear has quit IRC | 03:17 | |
| *** thorst_ has quit IRC | 03:23 | |
| *** jwcroppe has joined #openstack-watcher | 04:02 | |
| *** thorst_ has joined #openstack-watcher | 04:21 | |
| *** thorst_ has quit IRC | 04:28 | |
| *** jwcroppe has quit IRC | 05:25 | |
| *** jwcroppe has joined #openstack-watcher | 05:25 | |
| *** thorst_ has joined #openstack-watcher | 05:26 | |
| *** jwcroppe has quit IRC | 05:30 | |
| *** thorst_ has quit IRC | 05:33 | |
| *** thorst_ has joined #openstack-watcher | 06:31 | |
| *** thorst_ has quit IRC | 06:38 | |
| *** apoorv has joined #openstack-watcher | 07:15 | |
| *** thorst_ has joined #openstack-watcher | 07:36 | |
| *** thorst_ has quit IRC | 07:43 | |
| *** vmahe has joined #openstack-watcher | 07:44 | |
| *** vincentfrancoise has joined #openstack-watcher | 08:33 | |
| *** junjie has quit IRC | 08:36 | |
| *** thorst_ has joined #openstack-watcher | 08:41 | |
| *** thorst_ has quit IRC | 08:48 | |
| *** junjie has joined #openstack-watcher | 08:50 | |
| *** dtardivel has joined #openstack-watcher | 09:29 | |
| *** openstackgerrit has quit IRC | 09:30 | |
| *** openstackgerrit_ has joined #openstack-watcher | 09:30 | |
| *** openstackgerrit_ is now known as openstackgerrit | 09:31 | |
| *** openstackgerrit has quit IRC | 09:31 | |
| *** openstackgerrit_ has joined #openstack-watcher | 09:31 | |
| *** openstackgerrit_ is now known as openstackgerrit | 09:32 | |
| *** openstackgerrit has quit IRC | 09:32 | |
| *** openstackgerrit_ has joined #openstack-watcher | 09:32 | |
| *** openstackgerrit_ is now known as openstackgerrit | 09:33 | |
| *** openstackgerrit has quit IRC | 09:33 | |
| *** openstackgerrit_ has joined #openstack-watcher | 09:33 | |
| *** openstackgerrit_ is now known as openstackgerrit | 09:34 | |
| tkaczynski | hi guys, might I have some clarification questions about watcher in general? I don't fully understand how it works | 09:39 |
|---|---|---|
| tkaczynski | specifically, how strategies are triggered. I've seen mentions about continuous audits, but can't find any reference about that | 09:40 |
| tkaczynski | how do I create an continuous audit? does it mean that the strategy will be triggered periodically? how often? | 09:41 |
| vincentfrancoise | tkaczynski: Continuous audits are not currently implemented | 09:42 |
| vincentfrancoise | The only mode that is working right now is the ONESHOT mode | 09:42 |
| vincentfrancoise | there are a couple a BPs that need to be implemented before we can get there | 09:43 |
| vincentfrancoise | And I think this is one of the objectives for Newton | 09:43 |
| tkaczynski | hmm, so what's the point of using ceilometer for example? I thought that the idea is that watcher would monitor e.g. the processor time and based on that it might trigger for example a live migration | 09:43 |
| vincentfrancoise | well this is for a static analysis of the cluster. | 09:45 |
| *** thorst_ has joined #openstack-watcher | 09:45 | |
| vincentfrancoise | Basically | 09:48 |
| vincentfrancoise | the use case as of now | 09:48 |
| vincentfrancoise | is that you have an admin that would manually trigger an audit of the system | 09:49 |
| vincentfrancoise | this audit would then use ceilometer via the strategies to compute the action plan | 09:49 |
| tkaczynski | so the ceilometer is responsible for gathering the information about cluster, like vms, network links, disks which is then available as cluster data model? | 09:50 |
| *** apoorv_ has joined #openstack-watcher | 09:50 | |
| *** acabot has joined #openstack-watcher | 09:51 | |
| vincentfrancoise | yes | 09:51 |
| tkaczynski | what about dynamic data, like processor usage etc? is this also available in ceilometer / cluster data model? | 09:52 |
| acabot | tomasz, here is the picture we discussed during the mid-cycle a, like processor usage etc? is this also available in ceilometer / cluster data model? | 09:53 |
| acabot | http://factory.b-com.com/www/watcher/doc/watcher/_images/sequence_trigger_audit_in_decision_engine.png | 09:53 |
| *** thorst_ has quit IRC | 09:53 | |
| acabot | if you remember our discussion, we said that every time an admin trigger an audit, we get the cluster state via Nova API and then for each resource we get metrics from ceilometer | 09:54 |
| acabot | and Ceilometer could be a bottleneck in this case so we plan to build a cache DB in the future | 09:54 |
| tkaczynski | yes, I remember that. but it's only now when I have a better understanding of the whole system and am able to ask the "right" questions | 09:55 |
| acabot | sure ;-) | 09:56 |
| acabot | thats why I'm bringing you back to this discussion ;-) | 09:56 |
| tkaczynski | so how is it going to work with continuous audits? I can imagine that if we will have hundreds (if not thousands) resources to monitor, we might want to collect a lot of data periodically, let's say every 5 seconds | 09:57 |
| tkaczynski | if this will all have to go through database and then be queried by strategies (again every few seconds), how will that scale? | 09:58 |
| tkaczynski | I know I'm not asking easy questions :) | 09:59 |
| tkaczynski | ... and you've probably had already tons of discussions about that... | 09:59 |
| vincentfrancoise | You can have a look at https://blueprints.launchpad.net/watcher/+spec/cluster-model-objects-wrapper for periodical data collection | 10:00 |
| vincentfrancoise | I didn't read it myself yet | 10:00 |
| vincentfrancoise | but I believe the specs are tackling the problem you are mentioning | 10:01 |
| acabot | today the continuous audit is not working and Watcher does not scale to thousands of nodes | 10:01 |
| acabot | we know that we have to collect data periodically regarding cluster state and cluster metrics, it is something we will do during the newton cycle | 10:02 |
| acabot | and be able to deliver a scalable solution for the Newton version | 10:03 |
| acabot | the BP vincentfrancoise mentionned is a way to deal with heterogeneous type of resources in the cluster data model | 10:03 |
| tkaczynski | how they say, "paper can take everything". I'd say it would be good to have at least some proof of concept that this will scale. it has to be designed with scalability in mind otherwise nobody would use watcher in production. just saying :) | 10:03 |
| acabot | I'm perfectly aware of that (ceilometer example) | 10:04 |
| acabot | the problem we have today is that Nova and Ceilometer APIs will impact the scalability of Watcher | 10:05 |
| acabot | so what we decided at the mid cycle was to discuss with nova and ceilometer teams to tackle this issue | 10:06 |
| acabot | during the Newton cycle | 10:06 |
| tkaczynski | I'm also thinking about how the new scoring module will fit the overall picture, also in the case of continuous audits. basically I can imagine that there will be several strategies working "in parallel", all of them might intensively use the scoring engines (different ones). given that the current proposal is to have a central scoring service, it might | 10:07 |
| tkaczynski | again become a bottleneck very quickly | 10:07 |
| acabot | more things we will be able to do with Watcher, more these teams will be considering our problems | 10:07 |
| acabot | tomasz I understand your point but we have to do it incrementaly | 10:08 |
| acabot | if you try to tackle the hardest problems at the beginning, you will never write a single line of code | 10:09 |
| acabot | so the idea of the scoring module is to provide a basic implementation that allows to plug a third party | 10:10 |
| openstackgerrit | Vincent Françoise proposed openstack/watcher-dashboard: Renamed 'TRIGGERED' state to 'PENDING' https://review.openstack.org/291080 | 10:10 |
| acabot | like TAP | 10:10 |
| tkaczynski | fully agreed about incremental work, but design decisions are being made now. wrong choice now means troubles in the future, that's why I'm asking all these questions. please don't take me wrong - I'm on your side. I am currently contributing to watcher, I want it to be good :) | 10:10 |
| acabot | you are absolutely right | 10:15 |
| acabot | and we need to take time to review specs in details | 10:15 |
| acabot | and of course do not hesitate to ask questions, there are no silly questions | 10:16 |
| openstackgerrit | Vincent Françoise proposed openstack/watcher-dashboard: Renamed 'TRIGGERED' state to 'PENDING' https://review.openstack.org/291080 | 10:43 |
| *** thorst_ has joined #openstack-watcher | 10:50 | |
| *** thorst_ has quit IRC | 10:58 | |
| *** wootehfoot has joined #openstack-watcher | 11:32 | |
| *** thorst_ has joined #openstack-watcher | 11:56 | |
| *** apoorv has quit IRC | 11:59 | |
| *** apoorv_ has quit IRC | 12:00 | |
| *** thorst_ has quit IRC | 12:03 | |
| *** apoorv_ has joined #openstack-watcher | 12:11 | |
| *** apoorv has joined #openstack-watcher | 12:12 | |
| *** brunograz has joined #openstack-watcher | 12:26 | |
| *** jwcroppe has joined #openstack-watcher | 12:34 | |
| *** thorst_ has joined #openstack-watcher | 12:35 | |
| *** apoorv_ has quit IRC | 12:37 | |
| *** apoorv has quit IRC | 12:37 | |
| tkaczynski | hi again. what's the best way to discuss some watcher architectural questions? I'm working on the scoring module BluePrint (and implementation to follow) and I would like to make sure we are on the same page. is everybody happy to discuss that here? I think we can at least try here before organizing any meetings | 12:59 |
| tkaczynski | dtardivel: acabot said that you are involved a lot in watcher architecture | 13:00 |
| *** alexchadin has joined #openstack-watcher | 13:15 | |
| dtardivel | tkaczynski: hi | 13:15 |
| dtardivel | tkaczynski: what's your matter ? | 13:19 |
| tkaczynski | ok, let me explain | 13:21 |
| openstackgerrit | Bruno Grazioli proposed openstack/watcher: Integrated consolidation strategy with watcher https://review.openstack.org/289259 | 13:21 |
| tkaczynski | in the scoring-module BluePrint I'm currently proposing that Scoring Module is a new Watcher service (similar to watcher-decision-engine or watcher-applier) | 13:22 |
| tkaczynski | this service can be used only by he decision engine (if a given strategy wants to use a scoring engine) | 13:23 |
| tkaczynski | there can be many scoring engines, they are pluggable and might use different libraries, frameworks or even third party cloud services | 13:24 |
| *** vtech has joined #openstack-watcher | 13:24 | |
| tkaczynski | what decision engine needs is to get a reference to a specific scoring engine (possibly by using some lookup method and a unique key) and then use it | 13:25 |
| openstackgerrit | Alexander Chadin proposed openstack/watcher-specs: Add Overload standard deviation strategy spec https://review.openstack.org/286153 | 13:26 |
| tkaczynski | scoring engines might have a similar API in a sense that they might have similar input/output, which I'm proposing to be (in both cases) an array of double values | 13:26 |
| tkaczynski | but other than that, they are completely different. the number of input/output doubles might (and will) be different, the algorithms will be different, interpretation of results will be different (they solve different problems) | 13:28 |
| tkaczynski | if we decide to wrap it up with the service, it means that we will have the following situation: | 13:28 |
| tkaczynski | 1. decision engine want's to use a scoring engine with some ID, let's call it SE_ID | 13:29 |
| tkaczynski | 2. every time decision engine needs to use scoring engine, it calls watcher scoring service with parameters like: SE_ID, [input array of doubles] | 13:30 |
| *** xenogear has joined #openstack-watcher | 13:30 | |
| tkaczynski | 3. scoring service also provides some additional API for scoring engine metadata, so decision engine might get some additional information about a particular scoring engine or how to interpret the results | 13:31 |
| tkaczynski | my question is: is it a good idea to have such a service | 13:32 |
| tkaczynski | the alternative would be to define a scoring module as a library (or a package), which offers a similar functionality, doesn't have dependencies on other components, but it's not a centralized service | 13:33 |
| tkaczynski | the first implementation would be fairly simple, like a scoring engine base class, some implementations of the scoring engines and some factory to get a handle (or a "client") to a specific scoring engine (using some ID) | 13:35 |
| *** vmahe1 has joined #openstack-watcher | 13:35 | |
| tkaczynski | in the future it this doesn't scale, it could be re-implemented using some parallel framework or whatever, but there wouldn't be any need for the change from decision engine point of view | 13:36 |
| tkaczynski | whether if we have a service it might be more difficult to scale | 13:37 |
| *** vmahe has quit IRC | 13:38 | |
| *** acabot has quit IRC | 13:38 | |
| *** vincentfrancoise has quit IRC | 13:38 | |
| *** vmahe has joined #openstack-watcher | 13:38 | |
| tkaczynski | also, because the only "user" of the scoring module is decision engine, I don't see any benefit of making it a service. nobody else will ever use it | 13:38 |
| tkaczynski | your toughts? | 13:39 |
| tkaczynski | *thoughts? | 13:39 |
| *** vmahe1 has quit IRC | 13:40 | |
| *** vmahe1 has joined #openstack-watcher | 13:41 | |
| *** vmahe has quit IRC | 13:43 | |
| *** vmahe has joined #openstack-watcher | 13:46 | |
| *** vmahe1 has quit IRC | 13:46 | |
| *** vmahe1 has joined #openstack-watcher | 13:49 | |
| *** acabot has joined #openstack-watcher | 13:50 | |
| *** vmahe has quit IRC | 13:50 | |
| *** vincentfrancoise has joined #openstack-watcher | 13:51 | |
| dtardivel | tkaczynski: first, I think you can propose both alternatives into your spec to be able to comment your proposition and find a consensus. jed56 and vmahe should be able to participate to the debate | 13:53 |
| *** alexchadin has quit IRC | 13:54 | |
| dtardivel | Is your library a kind of pluggable library you will add into decision engine service, through entry points definition ? | 13:55 |
| vincentfrancoise | tkaczynski: the idea of having a separate library to do this is maybe to much for now. Just like dtardivel said, this should probably look like the watcher applier with a separate process/service with a base class and an entry point system to make it pluggable via stevedore. | 13:56 |
| tkaczynski | dtardival: this library is not yet implemented, but I don't think so. this module is optional and up to the strategy to use it. it should be pluggable in a way that there should be an easy way to add third party scoring engines to the system (similarly as we want to add new strategies) | 13:58 |
| *** alexchadin has joined #openstack-watcher | 14:00 | |
| tkaczynski | vincentfrancoise: by "library" I really meant simply a package (directory) within a decision engine. making this a separate process/service like applier is exactly what I want to avoid | 14:00 |
| tkaczynski | this means serialization/deserialization and inter-process communication for every call to the scoring engine. for continuous audits this might become a problem, because there might be many strategies working in parallel, ceilometer might gather a lot of data from hundreds of VMs every 5 seconds or so, so we might hit very quickly a limit | 14:03 |
| *** openstackgerrit_ has joined #openstack-watcher | 14:03 | |
| *** openstackgerrit has quit IRC | 14:03 | |
| *** jwcroppe has quit IRC | 14:04 | |
| *** openstackgerrit_ is now known as openstackgerrit | 14:04 | |
| *** openstackgerrit has quit IRC | 14:04 | |
| *** openstackgerrit_ has joined #openstack-watcher | 14:04 | |
| *** jwcroppe has joined #openstack-watcher | 14:04 | |
| vincentfrancoise | tkaczynski: I understand your point. I was wondering about the case where people are not using an external service like TAP but would directly implement some Python code. This means it would run within the same process as the decision engine and that can be a problem too | 14:05 |
| *** openstackgerrit_ is now known as openstackgerrit | 14:05 | |
| *** openstackgerrit has quit IRC | 14:05 | |
| *** openstackgerrit_ has joined #openstack-watcher | 14:05 | |
| tkaczynski | this is something we don't know. the scoring engines can be in the cloud (like TAP), but also using some frameworks, not necessarily written in python | 14:06 |
| *** openstackgerrit_ is now known as openstackgerrit | 14:06 | |
| *** openstackgerrit has quit IRC | 14:07 | |
| *** openstackgerrit_ has joined #openstack-watcher | 14:07 | |
| tkaczynski | but of course it can be simply a python code, so it might affect the decision engine process of course | 14:07 |
| dtardivel | I think workload and workflow for scoring processing and strategy processing are completely different. That implies, from my point of view, a dedicated service. | 14:07 |
| *** openstackgerrit_ is now known as openstackgerrit | 14:08 | |
| *** openstackgerrit has quit IRC | 14:08 | |
| *** openstackgerrit_ has joined #openstack-watcher | 14:08 | |
| vincentfrancoise | I know, but in my case having a separate process would be better off whereas in the case of an external framework, your option seems better | 14:08 |
| vincentfrancoise | so I guess we really need to lay both alternatives down into the specs so that everyone can have a say on this | 14:09 |
| *** openstackgerrit_ is now known as openstackgerrit | 14:09 | |
| *** jwcroppe has quit IRC | 14:09 | |
| *** openstackgerrit has quit IRC | 14:09 | |
| *** openstackgerrit_ has joined #openstack-watcher | 14:09 | |
| *** openstackgerrit_ is now known as openstackgerrit | 14:10 | |
| *** Guest41345 has joined #openstack-watcher | 14:10 | |
| tkaczynski | what I'm proposing is really an abstraction layer. internally this might call an external service if this is needed. but the use case from strategy code would be as simple as: 1. call some factory method to get a scoring engine "object" 2. use this object in the strategy | 14:10 |
| tkaczynski | having that abstraction we can do what we want | 14:11 |
| vincentfrancoise | I agree | 14:13 |
| dtardivel | +1 | 14:13 |
| tkaczynski | dtardivel: I like your idea about discussing the alternatives in the BP. I will update the spec | 14:15 |
| vincentfrancoise | so can you clearly explain this in the spec as well? | 14:15 |
| dtardivel | I think you have a section about alternatives in the spec template | 14:15 |
| tkaczynski | yes, I will explain it. and I will propose the abstraction layer I was talking about. | 14:16 |
| dtardivel | tkaczynski: please try to add pro/cons arguments for both alternatives | 14:17 |
| tkaczynski | sounds good | 14:18 |
| tkaczynski | thanks guys for your attention :) | 14:20 |
| *** alexchadin has quit IRC | 14:25 | |
| openstackgerrit | Bruno Grazioli proposed openstack/watcher: Integrated consolidation strategy with watcher https://review.openstack.org/289259 | 14:46 |
| openstackgerrit | Antoine Cabot proposed openstack/watcher-specs: Select destination filter for live migration https://review.openstack.org/291214 | 14:51 |
| *** hvprash_ has quit IRC | 14:58 | |
| *** wootehfoot has quit IRC | 15:00 | |
| openstackgerrit | Vincent Françoise proposed openstack/watcher: Added purge script for soft deleted objects https://review.openstack.org/286049 | 15:11 |
| *** wootehfoot has joined #openstack-watcher | 15:12 | |
| openstackgerrit | Bruno Grazioli proposed openstack/watcher: Integrated consolidation strategy with watcher https://review.openstack.org/289259 | 15:18 |
| *** hvprash has joined #openstack-watcher | 15:20 | |
| *** hvprash has quit IRC | 15:20 | |
| *** jwcroppe has joined #openstack-watcher | 15:39 | |
| openstackgerrit | Bruno Grazioli proposed openstack/watcher: Integrated consolidation strategy with watcher https://review.openstack.org/289259 | 15:42 |
| openstackgerrit | Merged openstack/watcher-dashboard: Renamed 'TRIGGERED' state to 'PENDING' https://review.openstack.org/291080 | 15:50 |
| *** wootehfoot has quit IRC | 15:52 | |
| openstackgerrit | Vincent Françoise proposed openstack/watcher: Documentation on purge command https://review.openstack.org/286111 | 15:59 |
| openstackgerrit | Vincent Françoise proposed openstack/watcher: Documentation on purge command https://review.openstack.org/286111 | 16:05 |
| openstackgerrit | Vincent Françoise proposed openstack/watcher: Added purge script for soft deleted objects https://review.openstack.org/286049 | 16:05 |
| openstackgerrit | Bruno Grazioli proposed openstack/watcher: Integrated consolidation strategy with watcher https://review.openstack.org/289259 | 16:06 |
| brunograz | hi guys, apologizes for submiting multiple PS for our strategie with minor changes - we had an issue with flake8 and pep8 which were not in the latest version and not catching errors in the code | 16:24 |
| brunograz | that happened when we were running tests locally | 16:26 |
| *** vtech has quit IRC | 16:31 | |
| *** vtech has joined #openstack-watcher | 16:32 | |
| *** jwcroppe has quit IRC | 16:58 | |
| *** jwcroppe has joined #openstack-watcher | 16:59 | |
| *** jwcroppe has quit IRC | 17:03 | |
| *** vtech has quit IRC | 17:18 | |
| *** vincentfrancoise has quit IRC | 17:37 | |
| *** openstackgerrit_ has joined #openstack-watcher | 18:29 | |
| *** xenogear has quit IRC | 18:34 | |
| *** vtech has joined #openstack-watcher | 19:16 | |
| *** hvprash has joined #openstack-watcher | 19:59 | |
| *** hvprash has quit IRC | 20:03 | |
| *** hvprash has joined #openstack-watcher | 20:17 | |
| *** hvprash has quit IRC | 20:21 | |
| *** wootehfoot has joined #openstack-watcher | 20:30 | |
| *** hvprash has joined #openstack-watcher | 20:45 | |
| *** hvprash has quit IRC | 20:50 | |
| *** dtardivel has quit IRC | 21:08 | |
| *** thorst_ has quit IRC | 21:56 | |
| *** thorst_ has joined #openstack-watcher | 22:02 | |
| *** thorst__ has joined #openstack-watcher | 22:04 | |
| *** thorst_ has quit IRC | 22:06 | |
| *** thorst__ has quit IRC | 22:08 | |
| *** thorst_ has joined #openstack-watcher | 22:22 | |
| *** thorst__ has joined #openstack-watcher | 22:26 | |
| *** thorst_ has quit IRC | 22:29 | |
| *** jwcroppe has joined #openstack-watcher | 23:24 | |
| *** vtech has quit IRC | 23:25 | |
| *** jwcroppe has quit IRC | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
