| *** https_GK1wmSU has joined #openstack-monasca | 00:03 | |
| *** https_GK1wmSU has left #openstack-monasca | 00:04 | |
| *** vishwana_ has joined #openstack-monasca | 00:18 | |
| *** vishwanathj has quit IRC | 00:22 | |
| *** bobh has joined #openstack-monasca | 02:26 | |
| *** https_GK1wmSU has joined #openstack-monasca | 02:28 | |
| *** https_GK1wmSU has left #openstack-monasca | 02:29 | |
| *** ddyer has quit IRC | 02:46 | |
| *** raddaoui has quit IRC | 02:54 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/monasca-api master: Updated from global requirements https://review.openstack.org/489449 | 02:56 |
|---|---|---|
| *** bobh has quit IRC | 03:23 | |
| *** nicolasbock has joined #openstack-monasca | 03:45 | |
| openstackgerrit | Haruki Yamanashi proposed openstack/monasca-agent master: Libvirt monitoring default configuration parameter https://review.openstack.org/489139 | 05:23 |
| openstackgerrit | Haruki Yamanashi proposed openstack/monasca-agent master: Template of Libvirt monitoring configuration https://review.openstack.org/489471 | 05:39 |
| openstackgerrit | Haruki Yamanashi proposed openstack/monasca-agent master: Template of Libvirt monitoring configuration https://review.openstack.org/489471 | 05:46 |
| openstackgerrit | Artur Basiak proposed openstack/monasca-ui master: Fix monasca-ui forms https://review.openstack.org/486926 | 06:04 |
| openstackgerrit | Artur Basiak proposed openstack/monasca-events-api master: Provide oslo.policies https://review.openstack.org/482837 | 06:20 |
| openstackgerrit | Artur Basiak proposed openstack/monasca-ui master: Fix monasca-ui forms https://review.openstack.org/486926 | 06:26 |
| openstackgerrit | Artur Basiak proposed openstack/monasca-ui master: Fix monasca-ui forms https://review.openstack.org/486926 | 06:38 |
| openstackgerrit | Jakub Wachowski proposed openstack/monasca-persister master: [WIP] Add support for events https://review.openstack.org/485113 | 07:04 |
| *** pcaruana has joined #openstack-monasca | 07:06 | |
| openstackgerrit | Artur Basiak proposed openstack/monasca-events-api master: Provide oslo.policies https://review.openstack.org/482837 | 07:19 |
| *** ukaynar has quit IRC | 07:47 | |
| *** ukaynar has joined #openstack-monasca | 07:47 | |
| *** ukaynar has quit IRC | 07:52 | |
| *** efoley has joined #openstack-monasca | 08:17 | |
| *** efoley has quit IRC | 08:18 | |
| *** efoley has joined #openstack-monasca | 08:18 | |
| openstackgerrit | Jakub Wachowski proposed openstack/monasca-persister master: Use ujson instead of json for parsing messages. https://review.openstack.org/489526 | 08:20 |
| openstackgerrit | Haruki Yamanashi proposed openstack/monasca-agent master: Change template/example of Libvirt config file https://review.openstack.org/489471 | 08:39 |
| openstackgerrit | Artur Basiak proposed openstack/monasca-events-api master: Provide oslo.policies https://review.openstack.org/482837 | 09:13 |
| openstackgerrit | Merged openstack/monasca-api master: Updated from global requirements https://review.openstack.org/489449 | 09:19 |
| openstackgerrit | Artur Basiak proposed openstack/monasca-events-api master: Provide oslo.policies https://review.openstack.org/482837 | 10:43 |
| openstackgerrit | Jakub Wachowski proposed openstack/monasca-persister master: [WIP] Add support for events https://review.openstack.org/485113 | 10:57 |
| openstackgerrit | Jakub Wachowski proposed openstack/monasca-events-api master: Create devstack plugin for monasca-events https://review.openstack.org/482899 | 10:58 |
| *** jromero has joined #openstack-monasca | 10:59 | |
| jromero | hej | 11:00 |
| jromero | i have a question regarding the monasca api | 11:00 |
| openstackgerrit | Artur Basiak proposed openstack/monasca-events-api master: Provide oslo.policies https://review.openstack.org/482837 | 11:00 |
| openstackgerrit | Artur Basiak proposed openstack/monasca-ui master: Fix monasca-ui forms https://review.openstack.org/486926 | 11:02 |
| arturb | jromero: what question? | 11:18 |
| openstackgerrit | Artur Basiak proposed openstack/monasca-events-api master: Provide oslo.policies https://review.openstack.org/482837 | 11:23 |
| jromero | hej arturb | 11:24 |
| jromero | sorry i was on the phone | 11:24 |
| arturb | no problem | 11:24 |
| jromero | we have seen a weird behavior when we load balance 16 api nodes | 11:24 |
| jromero | then we start getting 503 | 11:24 |
| jromero | the specs of the api servers are 8c 8m | 11:24 |
| jromero | as load balancer we use nginx | 11:24 |
| arturb | 8c 8m? | 11:25 |
| arturb | sorry but probably I won't be able to help you with load balancing | 11:26 |
| jromero | 8 cores and 8G of ram | 11:26 |
| jromero | i think the load balancer works fine | 11:26 |
| jromero | i think is a problem witht he api | 11:26 |
| jromero | we use the monasga peformance test tool | 11:26 |
| jromero | and then we get 503 or unavailable service | 11:27 |
| arturb | I will check the error code in api from where it comes from | 11:27 |
| jromero | specially when we start stressing the system | 11:27 |
| jromero | i think is an http error | 11:27 |
| jromero | brb phone | 11:27 |
| arturb | kk | 11:27 |
| openstackgerrit | Merged openstack/monasca-ui master: Fix monasca-ui forms https://review.openstack.org/486926 | 11:27 |
| arturb | what it the query/request are trying to send to api? healthcheck? | 11:28 |
| openstackgerrit | Artur Basiak proposed openstack/monasca-ui master: Updated from global requirements https://review.openstack.org/485616 | 11:29 |
| openstackgerrit | Artur Basiak proposed openstack/monasca-ui master: Extended grafana dashboard to display dimensions. https://review.openstack.org/485684 | 11:29 |
| openstackgerrit | Artur Basiak proposed openstack/monasca-ui master: Imported Translations from Zanata https://review.openstack.org/487747 | 11:29 |
| jromero | add metric | 11:34 |
| jromero | http://10.5.0.55:8070/v2.0/metrics | 11:35 |
| arturb | is there anything in api logs? | 11:35 |
| jromero | we are suspecting some problem with keystone and heavy load | 11:36 |
| jromero | but not sure | 11:36 |
| arturb | also check if kafka is alive | 11:36 |
| jromero | we are trying to hit 1 million metrics inside | 11:36 |
| jromero | arturb: kafka is not even swetting | 11:36 |
| jromero | we measure the inputs in kafka mostly but the api has this 503 | 11:37 |
| arturb | hmm | 11:37 |
| jromero | and we calculated a rate of 80 requests /s per core | 11:37 |
| arturb | is api connection to the database is valid? mariadb, influxdb | 11:37 |
| arturb | it might be a problem | 11:37 |
| openstackgerrit | Merged openstack/monasca-events-api master: Provide oslo.policies https://review.openstack.org/482837 | 11:37 |
| jromero | around 8000 messages since each request packs 100 metrics | 11:37 |
| jromero | arturb: we are not getting that far we are just stressing the api and understanding how much nodes we need to get in the system around 1M messages | 11:38 |
| jromero | if we connect one stress node to one api node we have been able to scale to half million easy | 11:39 |
| jromero | but each stress node is 2 cores and each api node is 8 cores | 11:39 |
| jromero | performance wise the api workes are unefficient | 11:39 |
| jromero | i think | 11:39 |
| jromero | but the problem comes when we use the load balancer then you start seeing issues on service not avaiable | 11:40 |
| jromero | 503 | 11:40 |
| jromero | and i wanted to know how you did in previous stress tests | 11:40 |
| jromero | maybe we are doing something wrong on our setup | 11:40 |
| jromero | by our calculations we will need 121 cores to reach 1M messages per second in kafka | 11:41 |
| jromero | and with the load balancer services starts to die | 11:41 |
| arturb | so maybe the problem is in ngnix configuration as a load balancer | 11:43 |
| jromero | does not seem to be there really | 11:44 |
| jromero | the messages are returned by the api | 11:44 |
| jromero | not by the balancer | 11:44 |
| arturb | is there anything in api logs? | 11:44 |
| arturb | and which version of api are you using? | 11:45 |
| jromero | latest in the repo | 11:47 |
| jromero | performance wise | 11:47 |
| openstackgerrit | Jakub Wachowski proposed openstack/monasca-persister master: Use ujson instead of json for parsing messages. https://review.openstack.org/489526 | 11:47 |
| jromero | 80 connections per second to the api per core or worker is a number you recognice as good? | 11:48 |
| openstackgerrit | Tomasz Trębski proposed openstack/monasca-api master: Composite paste for monasca-api https://review.openstack.org/485446 | 11:52 |
| jromero | hallo? | 12:03 |
| jromero | arturb: there? 80 connections /sec seems to be a good average per core or worker? | 12:03 |
| arturb | jromero sorry I was busy | 12:04 |
| arturb | about connections per sec per core I need to connact to the system test. They made performance tests | 12:05 |
| arturb | team | 12:05 |
| arturb | adrianc any ideas about this problem? | 12:06 |
| openstackgerrit | Artur Basiak proposed openstack/monasca-events-api master: Monasca-events-api spec documentation https://review.openstack.org/482443 | 12:15 |
| openstackgerrit | Artur Basiak proposed openstack/monasca-ui master: [WIP] Investigate problem with tests https://review.openstack.org/489585 | 12:30 |
| jromero | we think is related to the keystone token release | 12:38 |
| jromero | for the performance | 12:38 |
| jromero | we have some ideas to avoid this | 12:38 |
| jromero | we are going to start the 16 stressers with a delay of 30 seconds to make sure does not collide witht he keystone key retrieval | 12:39 |
| jromero | we are using a devstack single machine as keystone server | 12:39 |
| jromero | but still we not see keystone outperforming we only see 50% utilization on that machine all times | 12:39 |
| jromero | but if they the stressers are getting tokens at same time the 503 is assured so seems to be fair to think has something to do with the keystone release of tokens | 12:40 |
| *** ukaynar has joined #openstack-monasca | 12:41 | |
| openstackgerrit | Artur Basiak proposed openstack/monasca-ui master: Fix tags order in test https://review.openstack.org/489588 | 12:49 |
| *** ddyer has joined #openstack-monasca | 13:09 | |
| openstackgerrit | Artur Basiak proposed openstack/monasca-events-api master: Add sample configuration to documenatation https://review.openstack.org/489608 | 13:41 |
| openstackgerrit | Artur Basiak proposed openstack/monasca-events-api master: Add sample configuration files to documentation https://review.openstack.org/489608 | 13:41 |
| *** jromero has quit IRC | 13:57 | |
| *** ukaynar_ has joined #openstack-monasca | 14:05 | |
| *** ukaynar has quit IRC | 14:08 | |
| *** ukaynar_ has quit IRC | 14:13 | |
| *** rbrndt has joined #openstack-monasca | 14:20 | |
| *** bobh has joined #openstack-monasca | 15:06 | |
| *** pcaruana has quit IRC | 15:22 | |
| openstackgerrit | Merged openstack/monasca-events-api master: Add sample configuration files to documentation https://review.openstack.org/489608 | 15:30 |
| openstackgerrit | Kaiyan Sheng proposed openstack/monasca-notification master: Modify Notification Engine to allow inhibit, silence, and group https://review.openstack.org/434537 | 15:31 |
| openstackgerrit | Merged openstack/monasca-ui master: Fix tags order in test https://review.openstack.org/489588 | 15:37 |
| *** pcaruana has joined #openstack-monasca | 15:40 | |
| *** vishwana_ has quit IRC | 16:18 | |
| *** vishwanathj has joined #openstack-monasca | 16:19 | |
| openstackgerrit | Artur Basiak proposed openstack/monasca-ui master: Updated from global requirements https://review.openstack.org/485616 | 16:37 |
| openstackgerrit | Artur Basiak proposed openstack/monasca-ui master: Imported Translations from Zanata https://review.openstack.org/487747 | 16:37 |
| openstackgerrit | Artur Basiak proposed openstack/monasca-ui master: Extended grafana dashboard to display dimensions. https://review.openstack.org/485684 | 16:37 |
| iurygregory | Hello, does anyone know if there is a table for vm metrics (libvirt) with the semantics information like this one https://github.com/openstack/monasca-agent/blob/5e07c43e5edeacbb378e90b2dcfa33114148c97d/docs/Plugins.md#cpu ? | 16:43 |
| rbrndt | https://github.com/openstack/monasca-agent/blob/5e07c43e5edeacbb378e90b2dcfa33114148c97d/docs/Libvirt.md#per-instance-metrics | 16:46 |
| *** bobh has quit IRC | 16:47 | |
| iurygregory | ty rbrndt :D | 16:49 |
| *** bobh has joined #openstack-monasca | 16:58 | |
| *** pcaruana has quit IRC | 16:58 | |
| openstackgerrit | Kaiyan Sheng proposed openstack/monasca-notification master: Modify Notification Engine to allow inhibit, silence, and group https://review.openstack.org/434537 | 17:06 |
| *** efoley has quit IRC | 17:07 | |
| *** rbrndt has quit IRC | 17:19 | |
| *** rbrndt has joined #openstack-monasca | 18:21 | |
| *** bobh has quit IRC | 18:58 | |
| *** nicolasbock has quit IRC | 19:45 | |
| *** bobh has joined #openstack-monasca | 20:00 | |
| *** aagate has joined #openstack-monasca | 21:35 | |
| *** bobh has quit IRC | 22:16 | |
| openstackgerrit | Tim Buckley proposed openstack/monasca-agent master: Fix slash handling in KubernetesConnector's get_request https://review.openstack.org/489769 | 22:31 |
| *** rbrndt has quit IRC | 22:50 | |
| *** rbrndt has joined #openstack-monasca | 22:50 | |
| -openstackstatus- NOTICE: osic nodes have been removed from nodepool due to a problem with the mirror host beginning around 22:20 UTC. please recheck any jobs with failures installing packages. | 23:47 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!