| *** Paco has joined #openstack-performance | 02:21 | |
| *** Paco has left #openstack-performance | 02:25 | |
| *** Paco has joined #openstack-performance | 02:31 | |
| *** arnoldje has joined #openstack-performance | 03:24 | |
| *** sw3 has joined #openstack-performance | 03:52 | |
| *** badari has quit IRC | 04:17 | |
| *** badari has joined #openstack-performance | 04:22 | |
| *** pengzh has joined #openstack-performance | 05:31 | |
| *** pengzh has quit IRC | 05:32 | |
| *** arnoldje has quit IRC | 05:38 | |
| *** david-bravo23 has quit IRC | 06:00 | |
| *** Paco has quit IRC | 08:42 | |
| DinaBelova | harlowja_, kun_huang, SpamapS - hehe, I felt asleep last night too early :) good discussion :) | 08:45 |
|---|---|---|
| DinaBelova | folks, btw I'm really encouraging you to add your content to the https://wiki.openstack.org/wiki/Meetings/Performance next meeting agenda :) | 08:47 |
| DinaBelova | it was a complain that it was too short timeframe last time :) | 08:47 |
| *** Paco has joined #openstack-performance | 08:49 | |
| *** amaretskiy has joined #openstack-performance | 09:32 | |
| boris-42 | DinaBelova: btw | 10:02 |
| boris-42 | DinaBelova: I am not sure that we should have separated chat for osprofiler | 10:02 |
| boris-42 | DinaBelova: however I created it already | 10:02 |
| boris-42 | openstack-osprofiler | 10:02 |
| boris-42 | 1 year ago=) | 10:02 |
| DinaBelova | boris-42 :D | 10:03 |
| DinaBelova | well, personally I'll prefer to chat here about it | 10:03 |
| DinaBelova | as interested people will be here :) | 10:03 |
| DinaBelova | I suppose that at least :) | 10:03 |
| boris-42 | DinaBelova: btw https://review.openstack.org/#/c/242168/ | 10:03 |
| DinaBelova | boris-42 - yep, saw that | 10:04 |
| boris-42 | DinaBelova: I would put on title: "Dead of scalability" | 10:04 |
| DinaBelova | most probably issues will be faced for sure... although that was logical step that would have been done for sure. The thing is to fix all issues | 10:04 |
| DinaBelova | not try to write something else once that will be clear there are bugs :) | 10:05 |
| DinaBelova | boris-42 - I've started working on https://github.com/openstack/osprofiler/blob/master/doc/specs/in-progress/multi_backend_support.rst | 10:07 |
| DinaBelova | and fixed your comment on https://review.openstack.org/#/c/244084/ | 10:07 |
| DinaBelova | kun_huang, SpamapS, harlowja_ - btw, performance team inside Mirantis is currently working on creating the template to publish our performance test plans. I hope something will be available in human-readable way next week and I'll share it | 10:09 |
| DinaBelova | and I | 10:11 |
| DinaBelova | will ask all of you guys here to review them, propose your own ideas, etc. | 10:11 |
| DinaBelova | we'll try to perform them in life as well | 10:11 |
| boris-42 | DinaBelova: that one spec is quite hard | 10:30 |
| DinaBelova | @boris-42 - yep, I see that.. I think I'll start with drivers directory creation and adding Ceilometer there | 10:31 |
| DinaBelova | well, at least I may start :) | 10:31 |
| boris-42 | DinaBelova: btw current ceilometer backend is broken | 10:32 |
| DinaBelova | boris-42 - sh*t, I forgot about it | 10:32 |
| DinaBelova | boris-42 - well, I'll start with its fixing I guess | 10:33 |
| boris-42 | DinaBelova: ok =) | 10:35 |
| boris-42 | DinaBelova: I will cut now 0.3.1 version | 10:36 |
| DinaBelova | ack | 10:36 |
| DinaBelova | @boris-42 - although I suppose after ceilo backend fixing that should be done once more time? | 10:38 |
| boris-42 | DinaBelova: we can cut releases even every day =) | 10:41 |
| DinaBelova | :D | 10:41 |
| boris-42 | DinaBelova: so I will cut 0.3.1 | 10:41 |
| DinaBelova | ack | 10:41 |
| boris-42 | DinaBelova: and I will make patch in global requriemnts | 10:41 |
| boris-42 | DinaBelova: to force everywhere osprofiler >= 0.3.1 | 10:42 |
| DinaBelova | ok, cool | 10:42 |
| DinaBelova | after that I can make changes to Cinder, Trove, etc. | 10:42 |
| boris-42 | DinaBelova: after that you or me or somebody else will be able to fix everything | 10:42 |
| boris-42 | DinaBelova: yep | 10:42 |
| DinaBelova | ok, cool | 10:42 |
| boris-42 | DinaBelova: new release is here https://github.com/openstack/osprofiler/releases | 10:45 |
| DinaBelova | ok, give me please a link to the requirements as well | 10:45 |
| DinaBelova | @boris-42 ^^ | 10:45 |
| boris-42 | DinaBelova: https://review.openstack.org/244543 | 10:48 |
| DinaBelova | thank | 10:48 |
| *** Paco has quit IRC | 11:11 | |
| *** mwagner has quit IRC | 11:34 | |
| *** ozamiatin has joined #openstack-performance | 11:38 | |
| *** badari has quit IRC | 11:45 | |
| *** ozamiatin has quit IRC | 11:49 | |
| *** ozamiatin has joined #openstack-performance | 12:44 | |
| *** mwagner has joined #openstack-performance | 13:09 | |
| *** rvasilets has joined #openstack-performance | 13:20 | |
| *** ozamiatin has quit IRC | 13:25 | |
| *** ozamiatin has joined #openstack-performance | 13:27 | |
| *** ozamiatin has quit IRC | 13:37 | |
| *** ozamiatin has joined #openstack-performance | 13:41 | |
| *** boris-42 has quit IRC | 13:48 | |
| *** andreykurilin has left #openstack-performance | 14:01 | |
| *** andreykurilin has joined #openstack-performance | 14:02 | |
| *** arnoldje has joined #openstack-performance | 14:21 | |
| *** regXboi has joined #openstack-performance | 14:30 | |
| *** badari has joined #openstack-performance | 14:43 | |
| *** arnoldje has quit IRC | 15:03 | |
| *** arnoldje has joined #openstack-performance | 15:22 | |
| *** rvasilets has quit IRC | 15:40 | |
| *** mdorman has joined #openstack-performance | 15:57 | |
| *** mdorman has quit IRC | 16:38 | |
| *** ozamiatin has quit IRC | 16:38 | |
| *** mwagner has quit IRC | 16:41 | |
| *** mdorman has joined #openstack-performance | 16:42 | |
| *** rmart04 has joined #openstack-performance | 17:16 | |
| *** mwagner has joined #openstack-performance | 17:20 | |
| *** rmart04 has quit IRC | 17:21 | |
| *** rmart04 has joined #openstack-performance | 17:21 | |
| *** rmart04_ has joined #openstack-performance | 17:26 | |
| *** rmart04 has quit IRC | 17:28 | |
| *** rmart04_ is now known as rmart04 | 17:28 | |
| *** amaretskiy has quit IRC | 17:50 | |
| *** serverascode has joined #openstack-performance | 18:43 | |
| *** markvoelker has joined #openstack-performance | 18:45 | |
| *** rmart04 has quit IRC | 18:47 | |
| *** klindgren__ has joined #openstack-performance | 18:48 | |
| *** klindgren__ is now known as klindgren | 18:48 | |
| klindgren | DinaBelova, has Miratnis been able to duplicate remote conductor melting cores in medium sized envs (200+ HV's)? | 18:49 |
| *** jproulx1 has joined #openstack-performance | 18:49 | |
| harlowja_ | melt some marshmallows on the melting cores | 18:50 |
| harlowja_ | and make smores | 18:51 |
| *** mgagne has joined #openstack-performance | 18:53 | |
| *** mdorman has quit IRC | 18:58 | |
| *** mdorman has joined #openstack-performance | 18:59 | |
| harlowja_ | klindgren perhaps we should start making an etherpad (or something) just about this issue? | 19:01 |
| harlowja_ | centralizing all the different etherpads where it has been discussed ---> one etherpad | 19:01 |
| harlowja_ | (less scattered info ftw) | 19:02 |
| harlowja_ | SpamapS maybe falling over isn't the right word, 'melting cores' might be better, lol | 19:16 |
| harlowja_ | klindgren loves the melting cores, lol | 19:16 |
| SpamapS | if it really is just pegging CPUs, that definitely points at unoptimized encapsulation. | 19:17 |
| SpamapS | Talking to rabbit is not CPU bound | 19:17 |
| SpamapS | so....? | 19:17 |
| klindgren | we run conductor with 20 workers, typically they are running between 8-55% cpu per Conductor worker | 19:19 |
| klindgren | typically seems to be ~20% cpu per conductor worker | 19:20 |
| klindgren | same box also runs rabbitmq - typically consuming ~200-300% cpu | 19:21 |
| klindgren | and thats basically on an idle system with ~250 hv's | 19:22 |
| klindgren | start doing deployments and conductor use jumps pretty high | 19:22 |
| klindgren | We also have 3 of these servers running conductor - all will peg out cpu | 19:23 |
| *** wootehfoot has joined #openstack-performance | 19:29 | |
| *** wootehfoot has quit IRC | 19:29 | |
| harlowja_ | :-/ | 19:41 |
| harlowja_ | so 3 servers running conductor * 20 == 60 conductor processes | 19:42 |
| harlowja_ | klindgren thats accurate right? | 19:42 |
| *** rmart04 has joined #openstack-performance | 19:51 | |
| *** ozamiatin has joined #openstack-performance | 19:57 | |
| *** arnoldje has quit IRC | 20:03 | |
| klindgren | harlowja_, yep | 20:03 |
| klindgren | it use to be 40 conductor processes each | 20:04 |
| klindgren | but at we kept tripping 100% of all cores utilized for a period > 4 minutes alarms | 20:04 |
| klindgren | so we dropped it down a little bit to ease the alarms we were seeing for those hosts | 20:05 |
| klindgren | because we like sleep | 20:05 |
| klindgren | but I am noticing frequent Model server went away errors across HV's so I am tempted to increase the number of workers to see if that drops off | 20:06 |
| *** rmart04 has quit IRC | 20:06 | |
| *** ozamiatin has quit IRC | 20:06 | |
| *** arnoldje has joined #openstack-performance | 20:19 | |
| DinaBelova | klindgren - nope, we did not have the opportunity to spend time on his particular issues yet. although it is in our plans due to the importance of this problem | 20:23 |
| DinaBelova | klindgren - may you please create the etherpad with all details of your particular situation? this will be *VERY* helpful | 20:26 |
| DinaBelova | klindgren - this point will be one of the topic on the next meeting | 20:34 |
| DinaBelova | I suppose I'll check when Mirantis can work on this before Tuesday | 20:34 |
| *** arnoldje has quit IRC | 20:42 | |
| *** arnoldje has joined #openstack-performance | 21:07 | |
| *** mikal has joined #openstack-performance | 21:31 | |
| *** HenryG has joined #openstack-performance | 21:36 | |
| *** harshs has joined #openstack-performance | 22:05 | |
| *** mdorman has quit IRC | 22:06 | |
| *** regXboi has quit IRC | 22:15 | |
| *** boris-42 has joined #openstack-performance | 22:19 | |
| klindgren | DinaBelova, https://etherpad.openstack.org/p/remote-conductor-performance - Unsure what you else you need. Like we said aside from adding some more hardware and tuning some workers - we haven't really dug in. | 22:45 |
| *** arnoldje has quit IRC | 22:54 | |
| *** redixin has quit IRC | 22:54 | |
| *** mdorman has joined #openstack-performance | 22:58 | |
| harlowja_ | thx klindgren | 23:07 |
| *** harshs has quit IRC | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!