| *** yujunz has joined #openstack-performance | 00:16 | |
| *** catintheroof has quit IRC | 00:19 | |
| *** tovin07_ has joined #openstack-performance | 01:20 | |
| *** dixiaoli has joined #openstack-performance | 01:54 | |
| *** dixiaoli has quit IRC | 01:58 | |
| *** dixiaoli has joined #openstack-performance | 02:02 | |
| *** catintheroof has joined #openstack-performance | 02:16 | |
| *** catintheroof has quit IRC | 02:34 | |
| *** catintheroof has joined #openstack-performance | 02:35 | |
| *** catintheroof has quit IRC | 02:40 | |
| *** dixiaoli has quit IRC | 03:00 | |
| *** dixiaoli has joined #openstack-performance | 03:01 | |
| *** chyka has quit IRC | 03:14 | |
| *** dimtruck is now known as zz_dimtruck | 03:38 | |
| *** chyka has joined #openstack-performance | 04:06 | |
| *** Dinesh_Bhor has quit IRC | 04:07 | |
| *** markvoelker has quit IRC | 04:09 | |
| *** chyka has quit IRC | 04:11 | |
| *** dixiaoli has quit IRC | 04:17 | |
| *** dixiaoli has joined #openstack-performance | 04:19 | |
| *** dixiaoli has quit IRC | 04:23 | |
| *** Dinesh_Bhor has joined #openstack-performance | 05:11 | |
| *** klindgren_ has quit IRC | 05:13 | |
| *** dixiaoli has joined #openstack-performance | 05:26 | |
| *** klindgren has joined #openstack-performance | 05:28 | |
| *** chyka has joined #openstack-performance | 06:01 | |
| *** chyka has quit IRC | 06:05 | |
| *** aojea has joined #openstack-performance | 06:12 | |
| *** mgagne has quit IRC | 06:52 | |
| *** chyka has joined #openstack-performance | 07:01 | |
| *** chyka has quit IRC | 07:06 | |
| *** pcaruana has joined #openstack-performance | 07:06 | |
| *** aojea has quit IRC | 07:16 | |
| *** aojea has joined #openstack-performance | 07:19 | |
| *** hieulq has quit IRC | 07:27 | |
| *** hieulq has joined #openstack-performance | 07:27 | |
| *** hieulq has quit IRC | 07:29 | |
| *** hieulq has joined #openstack-performance | 07:29 | |
| *** hieulq has quit IRC | 07:29 | |
| *** hieulq has joined #openstack-performance | 07:29 | |
| *** hieulq is now known as Guest33220 | 07:30 | |
| *** hieulq_ has joined #openstack-performance | 07:30 | |
| *** hieulq_ has quit IRC | 07:30 | |
| *** hieulq_ has joined #openstack-performance | 07:41 | |
| *** rpodolyaka_afk is now known as rpodolyaka | 07:47 | |
| *** hieulq_ has quit IRC | 07:54 | |
| *** hieulq_ has joined #openstack-performance | 07:56 | |
| *** hieulq_ has quit IRC | 08:01 | |
| *** chyka has joined #openstack-performance | 08:02 | |
| *** aojea has quit IRC | 08:04 | |
| *** aojea has joined #openstack-performance | 08:06 | |
| *** chyka has quit IRC | 08:07 | |
| *** aojea has quit IRC | 08:07 | |
| *** aojea has joined #openstack-performance | 08:25 | |
| *** aojea has quit IRC | 08:29 | |
| *** rmart04 has joined #openstack-performance | 08:43 | |
| *** abalutoiu has quit IRC | 09:08 | |
| *** aojea has joined #openstack-performance | 09:26 | |
| *** aojea has quit IRC | 09:32 | |
| *** tovin07_ has quit IRC | 09:55 | |
| *** chyka has joined #openstack-performance | 10:14 | |
| *** yujunz has quit IRC | 10:17 | |
| *** yujunz has joined #openstack-performance | 10:18 | |
| *** chyka has quit IRC | 10:19 | |
| *** yujunz has quit IRC | 10:23 | |
| *** yujunz has joined #openstack-performance | 10:26 | |
| *** aojea has joined #openstack-performance | 10:28 | |
| *** yujunz has quit IRC | 10:31 | |
| *** aojea has quit IRC | 10:33 | |
| *** yujunz has joined #openstack-performance | 10:36 | |
| *** jkilpatr has quit IRC | 10:43 | |
| *** jkilpatr has joined #openstack-performance | 10:59 | |
| *** tovin07 has joined #openstack-performance | 11:16 | |
| *** tovin07 has quit IRC | 11:16 | |
| *** dixiaoli has quit IRC | 11:21 | |
| *** dixiaoli has joined #openstack-performance | 11:21 | |
| *** dixiaoli has quit IRC | 11:25 | |
| *** yujunz has quit IRC | 11:29 | |
| *** aojea has joined #openstack-performance | 11:29 | |
| *** yujunz has joined #openstack-performance | 11:31 | |
| *** aojea has quit IRC | 11:33 | |
| *** zz_dimtruck is now known as dimtruck | 11:43 | |
| *** markvoelker has joined #openstack-performance | 11:56 | |
| *** chyka has joined #openstack-performance | 12:03 | |
| *** markvoelker has quit IRC | 12:03 | |
| *** chyka has quit IRC | 12:07 | |
| *** rmart04 has quit IRC | 12:09 | |
| *** yujunz is now known as yujunz[away] | 12:14 | |
| *** yujunz[away] is now known as yujunz | 12:17 | |
| *** yujunz has quit IRC | 12:19 | |
| *** aojea has joined #openstack-performance | 12:30 | |
| *** aojea has quit IRC | 12:34 | |
| *** chyka has joined #openstack-performance | 13:03 | |
| *** chyka has quit IRC | 13:08 | |
| *** rfolco has joined #openstack-performance | 13:15 | |
| *** yujunz has joined #openstack-performance | 13:17 | |
| *** yujunz has quit IRC | 13:18 | |
| *** Dinesh_Bhor has quit IRC | 13:22 | |
| *** Dinesh_Bhor has joined #openstack-performance | 13:24 | |
| *** aojea has joined #openstack-performance | 13:31 | |
| *** aojea has quit IRC | 13:36 | |
| *** jkilpatr has quit IRC | 13:46 | |
| *** jkilpatr has joined #openstack-performance | 13:49 | |
| *** dimtruck is now known as zz_dimtruck | 13:53 | |
| *** markvoelker has joined #openstack-performance | 14:12 | |
| *** markvoelker has quit IRC | 14:12 | |
| *** markvoelker has joined #openstack-performance | 14:13 | |
| *** abalutoiu has joined #openstack-performance | 14:16 | |
| *** yujunz has joined #openstack-performance | 14:23 | |
| *** abalutoiu_ has joined #openstack-performance | 14:24 | |
| *** jkilpatr has quit IRC | 14:26 | |
| *** abalutoiu has quit IRC | 14:27 | |
| *** aojea has joined #openstack-performance | 14:31 | |
| *** aojea has quit IRC | 14:36 | |
| *** jkilpatr has joined #openstack-performance | 14:40 | |
| *** chyka has joined #openstack-performance | 14:51 | |
| *** yujunz is now known as yujunz[away] | 14:53 | |
| *** yujunz[away] is now known as yujunz | 14:54 | |
| *** chyka has quit IRC | 14:56 | |
| *** zz_dimtruck is now known as dimtruck | 15:01 | |
| *** yujunz has quit IRC | 15:15 | |
| *** tovin07 has joined #openstack-performance | 15:23 | |
| *** openstackgerrit has joined #openstack-performance | 15:28 | |
| openstackgerrit | Merged openstack/performance-docs master: Add logs to ceph rbd test results https://review.openstack.org/454232 | 15:28 |
|---|---|---|
| *** rcherrueau has joined #openstack-performance | 15:29 | |
| openstackgerrit | Merged openstack/performance-docs master: Adding Test plan for ML2/ODL https://review.openstack.org/424689 | 15:29 |
| DinaBelova | #startmeeting Performance Team | 15:30 |
| openstack | Meeting started Tue Apr 11 15:30:00 2017 UTC and is due to finish in 60 minutes. The chair is DinaBelova. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:30 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:30 |
| openstackgerrit | Merged openstack/performance-docs master: Ceph RBD test plan https://review.openstack.org/453600 | 15:30 |
| openstack | The meeting name has been set to 'performance_team' | 15:30 |
| openstackgerrit | Merged openstack/performance-docs master: Kube-proxy performance test plan and report https://review.openstack.org/452257 | 15:30 |
| DinaBelova | hello folks! | 15:30 |
| DinaBelova | rcherrueau tovin07 akrzos o/ | 15:30 |
| rcherrueau | o/ | 15:30 |
| tovin07 | o/ | 15:30 |
| akrzos | DinaBelova: o/ | 15:30 |
| DinaBelova | #info today I have only 30 minutes - have to run to the next meeting >_< | 15:31 |
| DinaBelova | so let's get started | 15:31 |
| DinaBelova | #topic Action Items | 15:31 |
| DinaBelova | last time we had two action items, both on rcherrueau :) | 15:31 |
| DinaBelova | I see you have no objections to tovin07 's commit to osprofiler | 15:32 |
| rcherrueau | Yep. my first priority is finalising wan experiments, but I make a WIP test plan for OpenStack under WAN. | 15:32 |
| DinaBelova | rcherrueau ack | 15:32 |
| openstackgerrit | Merged openstack/performance-docs master: Add Fuel CCP upgrade testing https://review.openstack.org/451419 | 15:32 |
| DinaBelova | so lemme keep this action item for now | 15:32 |
| openstackgerrit | Merged openstack/performance-docs master: Fuel-CCP OpenStack reliability testing report https://review.openstack.org/449087 | 15:32 |
| rcherrueau | #link https://github.com/rcherrueau/performance-docs/tree/openstack-wan | 15:32 |
| rcherrueau | I will iterate on that at the same time I iterate on experiments. | 15:32 |
| *** aojea has joined #openstack-performance | 15:32 | |
| rcherrueau | You know, the protocol become more and more precise as I deep into experiments. | 15:32 |
| DinaBelova | #action rcherrueau add OpenStack testing under networking delays (e.g. multisite deployment) methodology to performance docs (openstack under WAN) | 15:32 |
| DinaBelova | rcherrueau true | 15:32 |
| DinaBelova | #topic Current progress on the planned tests | 15:33 |
| rcherrueau | Anyway, at some point, I expect to provide in the document the `tc` program that helps you set up traffic shaping. | 15:33 |
| DinaBelova | rcherrueau I guess it makes sense for you to share current progress now :) | 15:33 |
| rcherrueau | For the WAN experiments, we have the felling that high latency and traffic shaping will be a problem for REST call and oslo_messaging. | 15:33 |
| rcherrueau | To investigate this, we need a good knowledge of API/RPC workflow and OSProfiler is a good friend here :) | 15:33 |
| rcherrueau | So we decided to go a bit further and develop a tool that automatically generate a Sequence Diagram based on a Trace. | 15:33 |
| DinaBelova | yay :) | 15:33 |
| rcherrueau | #link https://github.com/rcherrueau/osp-utils | 15:33 |
| rcherrueau | You can find an sequence diagram of `openstack server create ...` in test/rsc/out.svg | 15:34 |
| rcherrueau | #link https://github.com/rcherrueau/osp-utils/blob/1af8c4e0a0c3e5f5722fe7cf33109eafb77c2c74/tests/rsc/out.svg | 15:34 |
| rcherrueau | From now, it's not really readable. The diagram is really huge because of tons of calls to the DB. | 15:34 |
| rcherrueau | But I'm writing a folding function that let you aggregate similar traces based on a predicate. | 15:34 |
| rcherrueau | Maybe one last remark, the tool has been written in Haskell. | 15:34 |
| rcherrueau | I know this is not OpenStack compliant, but I always say to my students: the right language for the right tool ;) | 15:34 |
| DinaBelova | rcherrueau yeah, I noticed this already :) | 15:35 |
| rcherrueau | And I was afraid to go with python regarding the huge size of traces | 15:35 |
| rcherrueau | In addition, depending how fast I go on WAN experiments, I hope I can add a Query Language to filter traces. | 15:35 |
| rcherrueau | For instance, to only keep RPC calls. | 15:35 |
| rcherrueau | that's all for this week on my side. | 15:35 |
| DinaBelova | rcherrueau ack, thanks | 15:35 |
| DinaBelova | btw | 15:35 |
| DinaBelova | if I'm not mistaken, tovin07, we were talking one time about adding filtration to the osprofiler report? | 15:36 |
| DinaBelova | to filter rpc / db / other types of calls | 15:36 |
| tovin07 | yes | 15:36 |
| tovin07 | we were talking about that | 15:37 |
| DinaBelova | tovin07 did you have a chance to look at this already? | 15:37 |
| tovin07 | a very long time ago | 15:37 |
| DinaBelova | :D | 15:37 |
| *** aojea has quit IRC | 15:37 | |
| tovin07 | maybe :D | 15:37 |
| tovin07 | I will investigate | 15:37 |
| DinaBelova | okay, I just think that this thing becomes even more actual | 15:37 |
| DinaBelova | thank you | 15:37 |
| rcherrueau | tovin07: thanks | 15:37 |
| tovin07 | btw | 15:37 |
| DinaBelova | tovin07 yes? | 15:38 |
| tovin07 | I think that’s a necessary feature | 15:38 |
| DinaBelova | it is :) | 15:38 |
| tovin07 | sometimes, we’ve got alot of db trace :v | 15:38 |
| rcherrueau | tovin07: too much db traces ... | 15:39 |
| tovin07 | yup | 15:39 |
| DinaBelova | true :) OpenStack communicates with DB in an extensive manner :) | 15:39 |
| tovin07 | okay, we should move on to the next topic :D | 15:39 |
| DinaBelova | ack :) | 15:39 |
| DinaBelova | akrzos sir, how's your summit preparation is going on? | 15:39 |
| akrzos | running telemetry with Ocata now | 15:40 |
| akrzos | also removing the collector | 15:40 |
| akrzos | i expect some solid results on this by this time next week | 15:40 |
| akrzos | also | 15:40 |
| akrzos | we identified a big issue with gnocchi and ceph | 15:41 |
| akrzos | especially evident when gnocchi is untuned | 15:41 |
| DinaBelova | akrzos oh, can you please share it? | 15:41 |
| akrzos | a single object in ceph is used to hold a list of keys | 15:41 |
| akrzos | each key is a small object in ceph for an unprocessed measure | 15:41 |
| akrzos | so two issues | 15:41 |
| akrzos | the list of keys | 15:42 |
| akrzos | can grow too large | 15:42 |
| akrzos | and this can cause ceph to fail | 15:42 |
| akrzos | whatever your replica is | 15:42 |
| *** chyka has joined #openstack-performance | 15:42 | |
| akrzos | those osds will fail | 15:42 |
| DinaBelova | oh, unpleasant :D | 15:42 |
| akrzos | so if you have replica 3 on the metrics pool | 15:42 |
| akrzos | 3 osds will go out | 15:42 |
| akrzos | so this occurs when the backlog grows | 15:42 |
| DinaBelova | okay, so this definitely looks as issues #1 and issue #2 ? | 15:43 |
| akrzos | so if you don't have sufficent capacity to process all your measures | 15:43 |
| akrzos | that can occur | 15:43 |
| akrzos | issue #2 | 15:43 |
| akrzos | is ceph as a filestore is bad for small objects | 15:43 |
| akrzos | resolution basiclly won't come until ceph bluestore is shipping | 15:43 |
| akrzos | and integrated | 15:43 |
| akrzos | so pretty much if you scale up and don't tune gnocchi | 15:44 |
| akrzos | you hit ceph problems | 15:44 |
| akrzos | if you tune gnocchi | 15:44 |
| akrzos | and then scale up | 15:44 |
| DinaBelova | yeah, I remember concerns about it when gnocchi was starting | 15:44 |
| akrzos | you'll hit ceilometer collector issues | 15:44 |
| akrzos | so need to test with ceilometer collector removed | 15:44 |
| DinaBelova | akrzos ack, thank you sir for the update | 15:45 |
| akrzos | basiclly in ocata and > you can change the publisher to gnocchi so agent-notification can publish directly to gnocchi | 15:45 |
| akrzos | so thats what i am working on | 15:45 |
| akrzos | also | 15:45 |
| akrzos | i got WIP gnocchi benchmarks in rally | 15:45 |
| akrzos | needs a bit more touch up | 15:45 |
| DinaBelova | do you have some links to them? | 15:45 |
| DinaBelova | akrzos ^^ | 15:47 |
| akrzos | yes | 15:47 |
| akrzos | #link https://review.openstack.org/#/c/453861/ | 15:48 |
| DinaBelova | akrzos thanks! | 15:49 |
| DinaBelova | from Mirantis side - we're still working on the next cycle of testing scope, as product testing priorities strike back :) so last week was mostly around planning and some more time is required there.... from what I see right now we'll be even more k8s centric this quarter in terms of RnD | 15:49 |
| DinaBelova | but it's not defined yet, sadly | 15:49 |
| DinaBelova | hope to give more details next time | 15:49 |
| DinaBelova | #topic Open Discussion | 15:49 |
| DinaBelova | tovin07 I see items from you sir :) | 15:49 |
| DinaBelova | please proceed | 15:50 |
| tovin07 | yup | 15:50 |
| tovin07 | (tovin07) OSprofiler show sql/function results (continue) | 15:50 |
| tovin07 | rcherrueau: do you have any comment? | 15:50 |
| rcherrueau | yep | 15:50 |
| rcherrueau | I wanna try your patch, going with a devstack AIO deployment and `enable_plugin osprofiler https://git.openstack.org/openstack/osprofiler refs/changes/72/450072/2` but devstack always install osprofiler master. | 15:51 |
| rcherrueau | Don't know why! | 15:51 |
| rcherrueau | By the way, I wrote a review based on the demo you provided in your commit message. | 15:51 |
| tovin07 | yes, I see | 15:52 |
| DinaBelova | rcherrueau shouldn't that be some flag in devstack? to set up specific branch for the library? | 15:52 |
| tovin07 | thank you rcherrueau, I will re-recheck it | 15:52 |
| rcherrueau | DinaBelova: no idea, I rarely use devstack | 15:53 |
| rcherrueau | tovin07: cool | 15:53 |
| tovin07 | okay | 15:53 |
| tovin07 | next item | 15:53 |
| tovin07 | (tovin07) About Rally + OSprofiler for OpenStack services (rcherrueau had discussed with Rally PTL about this) | 15:53 |
| tovin07 | this one is related to rcherrueau (too) | 15:53 |
| tovin07 | #link Rally + OSprofiler http://eavesdrop.openstack.org/meetings/rally/2017/rally.2017-04-10-14.05.log.html#l-39 | 15:54 |
| tovin07 | as I discussed in Rally IRC meeting yesterday | 15:54 |
| tovin07 | <andreykurilin> Ok, I'll start a spec about it. | 15:54 |
| rcherrueau | tovin07: nice | 15:55 |
| tovin07 | so, I think that I can help a little bit on that work | 15:55 |
| DinaBelova | tovin07 thank you sir :) | 15:56 |
| rcherrueau | Speaking about OSProfiler, I finally managed to get it works into kolla-ansible. Briefly, osprofiler dependency was missing in nova/keystone/glance/... containers. | 15:56 |
| rcherrueau | So the import of osprofiler always failed and services never called osprofiler to put trace events into oslo_messaging | 15:56 |
| tovin07 | rcherrueau: cool | 15:56 |
| rcherrueau | This error was really silent! | 15:56 |
| tovin07 | yes | 15:57 |
| DinaBelova | rcherrueau ah, so that was about right configuration in the end :) | 15:57 |
| DinaBelova | 95% of all issues :D | 15:57 |
| tovin07 | osprofiler is not a required dependency | 15:57 |
| tovin07 | it’s optional | 15:57 |
| rcherrueau | DinaBelova: yep :D | 15:57 |
| DinaBelova | ok, folks, I have to run to other meeting... tovin07 rcherrueau akrzos anything to share at last minutes? | 15:58 |
| tovin07 | nope | 15:58 |
| rcherrueau | no | 15:58 |
| DinaBelova | thank you folks! have a nice day / evening! | 15:58 |
| DinaBelova | #endmeeting | 15:58 |
| openstack | Meeting ended Tue Apr 11 15:58:37 2017 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:58 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/performance_team/2017/performance_team.2017-04-11-15.30.html | 15:58 |
| rcherrueau | thanks | 15:58 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/performance_team/2017/performance_team.2017-04-11-15.30.txt | 15:58 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/performance_team/2017/performance_team.2017-04-11-15.30.log.html | 15:58 |
| akrzos | Thanks! | 15:58 |
| tovin07 | thanks! | 15:58 |
| tovin07 | :D | 15:58 |
| *** tovin07 has quit IRC | 16:14 | |
| *** rcherrueau has quit IRC | 16:22 | |
| *** pcaruana has quit IRC | 16:29 | |
| *** aojea has joined #openstack-performance | 16:33 | |
| *** aojea has quit IRC | 16:38 | |
| *** mgagne has joined #openstack-performance | 17:00 | |
| *** abalutoiu_ is now known as abalutoiu | 17:08 | |
| *** aojea has joined #openstack-performance | 17:34 | |
| *** aojea has quit IRC | 17:38 | |
| *** sai is now known as sai-afk | 18:08 | |
| *** sai-afk is now known as sai | 18:22 | |
| *** abalutoiu has quit IRC | 18:25 | |
| *** abalutoiu has joined #openstack-performance | 18:27 | |
| *** abalutoiu_ has joined #openstack-performance | 18:31 | |
| *** abalutoiu has quit IRC | 18:34 | |
| *** aojea has joined #openstack-performance | 18:35 | |
| *** mgagne_ has joined #openstack-performance | 18:35 | |
| *** mgagne_ is now known as Guest89317 | 18:35 | |
| *** aojea has quit IRC | 18:39 | |
| *** Guest89317 has quit IRC | 18:46 | |
| *** abalutoiu has joined #openstack-performance | 18:56 | |
| *** abalutoiu_ has quit IRC | 19:00 | |
| *** abalutoiu_ has joined #openstack-performance | 19:08 | |
| *** abalutoiu__ has joined #openstack-performance | 19:09 | |
| *** abalutoiu has quit IRC | 19:12 | |
| *** abalutoiu_ has quit IRC | 19:12 | |
| *** mgagne_ has joined #openstack-performance | 19:18 | |
| *** mgagne_ is now known as Guest4366 | 19:19 | |
| *** mgagne has quit IRC | 19:24 | |
| *** Guest4366 is now known as mgagne | 19:25 | |
| *** mgagne has quit IRC | 19:25 | |
| *** mgagne has joined #openstack-performance | 19:25 | |
| *** jkilpatr_ has joined #openstack-performance | 19:45 | |
| *** jkilpatr has quit IRC | 19:47 | |
| *** markvoelker has quit IRC | 19:56 | |
| *** dimtruck is now known as zz_dimtruck | 20:35 | |
| *** rfolco has quit IRC | 21:01 | |
| *** zz_dimtruck is now known as dimtruck | 21:02 | |
| *** catintheroof has joined #openstack-performance | 21:19 | |
| *** catintheroof has quit IRC | 21:21 | |
| *** catintheroof has joined #openstack-performance | 21:21 | |
| *** abalutoiu_ has joined #openstack-performance | 21:23 | |
| *** abalutoiu__ has quit IRC | 21:27 | |
| *** jkilpatr_ has quit IRC | 21:30 | |
| *** jkilpatr_ has joined #openstack-performance | 22:08 | |
| *** dixiaoli has joined #openstack-performance | 22:43 | |
| *** meteorfox has left #openstack-performance | 22:59 | |
| *** chyka has quit IRC | 23:05 | |
| *** chyka has joined #openstack-performance | 23:05 | |
| *** catintheroof has quit IRC | 23:29 | |
| *** catintheroof has joined #openstack-performance | 23:30 | |
| *** catintheroof has quit IRC | 23:30 | |
| *** chyka has quit IRC | 23:30 | |
| *** dixiaoli has quit IRC | 23:36 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
