| *** derekh has joined #openstack-operators | 00:13 | |
| *** harshs has quit IRC | 00:17 | |
| *** sanjayu has joined #openstack-operators | 00:20 | |
| *** derekh has quit IRC | 00:20 | |
| *** harshs has joined #openstack-operators | 00:31 | |
| *** dminer has quit IRC | 00:31 | |
| *** harshs_ has joined #openstack-operators | 00:34 | |
| *** harshs has quit IRC | 00:36 | |
| *** harshs_ is now known as harshs | 00:36 | |
| *** harshs has quit IRC | 00:37 | |
| *** mdorman has quit IRC | 00:40 | |
| *** harshs has joined #openstack-operators | 00:44 | |
| *** harshs has quit IRC | 00:44 | |
| *** shimura_ken has joined #openstack-operators | 00:52 | |
| *** armax has joined #openstack-operators | 00:57 | |
| *** gyee has quit IRC | 00:57 | |
| *** krobzaur_ has quit IRC | 01:00 | |
| *** SimonChung has joined #openstack-operators | 01:01 | |
| *** SimonChung2 has quit IRC | 01:01 | |
| *** SimonChung1 has joined #openstack-operators | 01:02 | |
| *** SimonChung has quit IRC | 01:02 | |
| *** SimonChung has joined #openstack-operators | 01:03 | |
| *** SimonChung1 has quit IRC | 01:03 | |
| *** mriedem has joined #openstack-operators | 01:05 | |
| *** alop has quit IRC | 01:05 | |
| *** zhangjn has joined #openstack-operators | 01:06 | |
| *** SimonChung has quit IRC | 01:14 | |
| *** kencjohnston has joined #openstack-operators | 01:26 | |
| *** VW has joined #openstack-operators | 01:52 | |
| *** harshs has joined #openstack-operators | 01:52 | |
| *** harshs has quit IRC | 02:02 | |
| *** Marga_ has quit IRC | 02:11 | |
| *** VW has quit IRC | 02:16 | |
| *** sanjayu has quit IRC | 02:23 | |
| *** ccarmack has joined #openstack-operators | 02:33 | |
| *** fawadkhaliq has joined #openstack-operators | 02:42 | |
| *** lhcheng has quit IRC | 02:49 | |
| *** lhcheng has joined #openstack-operators | 02:50 | |
| *** lhcheng has quit IRC | 02:50 | |
| *** shimura_ken has quit IRC | 02:51 | |
| *** shimura_ken has joined #openstack-operators | 02:52 | |
| *** shimura_ken has quit IRC | 02:53 | |
| *** yantarou____ has joined #openstack-operators | 02:53 | |
| *** rick_ has joined #openstack-operators | 03:10 | |
| *** maishsk has quit IRC | 03:10 | |
| *** yantarou____ has quit IRC | 03:17 | |
| *** bvandenh has joined #openstack-operators | 03:19 | |
| *** lhcheng has joined #openstack-operators | 03:23 | |
| *** bvandenh has quit IRC | 03:25 | |
| *** yantarou has joined #openstack-operators | 03:36 | |
| *** mriedem has quit IRC | 03:55 | |
| *** fawadkhaliq has quit IRC | 04:06 | |
| *** VW has joined #openstack-operators | 04:16 | |
| *** SimonChung has joined #openstack-operators | 04:17 | |
| *** SimonChung1 has joined #openstack-operators | 04:19 | |
| *** VW has quit IRC | 04:20 | |
| *** SimonChung has quit IRC | 04:21 | |
| *** fawadkhaliq has joined #openstack-operators | 04:27 | |
| *** gfa is now known as gfa_ | 04:35 | |
| *** zhangjn has quit IRC | 04:37 | |
| *** zhangjn has joined #openstack-operators | 04:38 | |
| *** markvoelker_ has quit IRC | 04:40 | |
| *** harshs has joined #openstack-operators | 04:52 | |
| *** harshs_ has joined #openstack-operators | 04:52 | |
| *** harshs has quit IRC | 04:56 | |
| *** harshs_ is now known as harshs | 04:56 | |
| *** rick_ has quit IRC | 05:09 | |
| *** zerda has joined #openstack-operators | 05:14 | |
| *** Marga_ has joined #openstack-operators | 05:20 | |
| *** Marga_ has quit IRC | 05:20 | |
| *** Marga_ has joined #openstack-operators | 05:21 | |
| *** rick_ has joined #openstack-operators | 05:22 | |
| *** zhangjn_ has joined #openstack-operators | 05:26 | |
| *** zhangjn has quit IRC | 05:27 | |
| *** lhcheng_ has joined #openstack-operators | 05:29 | |
| *** lhcheng has quit IRC | 05:32 | |
| *** zhangjn_ has quit IRC | 05:37 | |
| *** zhangjn has joined #openstack-operators | 05:38 | |
| *** harshs_ has joined #openstack-operators | 05:43 | |
| *** harshs has quit IRC | 05:44 | |
| *** harshs_ is now known as harshs | 05:44 | |
| *** sanjayu has joined #openstack-operators | 05:52 | |
| *** clayton has quit IRC | 06:22 | |
| *** clayton has joined #openstack-operators | 06:23 | |
| *** rcernin has joined #openstack-operators | 06:37 | |
| *** lhcheng has joined #openstack-operators | 06:39 | |
| *** markvoelker has joined #openstack-operators | 06:40 | |
| *** lhcheng_ has quit IRC | 06:42 | |
| *** markvoelker has quit IRC | 06:45 | |
| *** elemoine has joined #openstack-operators | 06:56 | |
| *** Marga_ has quit IRC | 07:03 | |
| *** Marga_ has joined #openstack-operators | 07:05 | |
| *** armax has quit IRC | 07:16 | |
| *** subscope has joined #openstack-operators | 07:24 | |
| *** harshs has quit IRC | 07:36 | |
| *** markvoelker has joined #openstack-operators | 07:41 | |
| *** markvoelker has quit IRC | 07:46 | |
| *** jmccrory has quit IRC | 07:49 | |
| *** jmccrory has joined #openstack-operators | 07:54 | |
| *** belmoreira has joined #openstack-operators | 07:57 | |
| *** matrohon has joined #openstack-operators | 08:05 | |
| *** liverpooler has joined #openstack-operators | 08:07 | |
| *** maishsk has joined #openstack-operators | 08:14 | |
| *** egonzalez has joined #openstack-operators | 08:30 | |
| *** rick_ has quit IRC | 08:33 | |
| *** fawadkhaliq has quit IRC | 08:41 | |
| *** fawadkhaliq has joined #openstack-operators | 08:42 | |
| *** lhcheng has quit IRC | 08:45 | |
| *** ferest has joined #openstack-operators | 08:48 | |
| *** ferest has quit IRC | 08:53 | |
| *** fawadkhaliq has quit IRC | 08:57 | |
| *** egonzalez has quit IRC | 09:20 | |
| *** Piet has quit IRC | 09:21 | |
| *** cbrown2_ocf has joined #openstack-operators | 09:29 | |
| *** ahonda has quit IRC | 09:32 | |
| *** markvoelker has joined #openstack-operators | 09:42 | |
| *** markvoelker has quit IRC | 09:46 | |
| *** cbrown2_ocf has quit IRC | 10:12 | |
| *** cbrown2_ocf has joined #openstack-operators | 10:22 | |
| *** zhangjn has quit IRC | 10:28 | |
| *** fawadkhaliq has joined #openstack-operators | 10:36 | |
| *** marekd has joined #openstack-operators | 10:38 | |
| *** zhangjn has joined #openstack-operators | 10:51 | |
| *** zhangjn has quit IRC | 11:11 | |
| *** VW has joined #openstack-operators | 11:18 | |
| *** VW has quit IRC | 11:23 | |
| *** aswadr has joined #openstack-operators | 11:32 | |
| *** subscope has quit IRC | 11:41 | |
| *** subscope has joined #openstack-operators | 11:42 | |
| *** markvoelker has joined #openstack-operators | 11:43 | |
| *** markvoelker has quit IRC | 11:47 | |
| *** bvandenh has joined #openstack-operators | 11:50 | |
| *** egonzalez has joined #openstack-operators | 12:03 | |
| *** bvandenh has quit IRC | 12:05 | |
| *** zerda has quit IRC | 12:08 | |
| *** zhangjn has joined #openstack-operators | 12:09 | |
| *** zhangjn has quit IRC | 12:09 | |
| *** zhangjn has joined #openstack-operators | 12:10 | |
| *** zhangjn has quit IRC | 12:10 | |
| *** zhangjn has joined #openstack-operators | 12:10 | |
| *** aswadr has quit IRC | 12:15 | |
| *** subscope has quit IRC | 12:20 | |
| *** subscope has joined #openstack-operators | 12:21 | |
| *** subscope has quit IRC | 12:21 | |
| *** egonzalez has quit IRC | 12:22 | |
| *** alejandrito has joined #openstack-operators | 12:23 | |
| *** jkraj has joined #openstack-operators | 12:28 | |
| *** berendt has joined #openstack-operators | 12:35 | |
| *** fawadkhaliq has quit IRC | 12:42 | |
| *** markvoelker has joined #openstack-operators | 12:43 | |
| *** markvoelker has quit IRC | 12:48 | |
| *** electrofelix has joined #openstack-operators | 12:57 | |
| *** markvoelker has joined #openstack-operators | 13:09 | |
| *** cbrown2_ocf has quit IRC | 13:27 | |
| *** bvandenh has joined #openstack-operators | 13:28 | |
| *** sanjayu has quit IRC | 13:35 | |
| *** ccarmack has left #openstack-operators | 13:45 | |
| *** elemoine has quit IRC | 13:56 | |
| *** krobzaur_ has joined #openstack-operators | 14:07 | |
| *** bapalm has joined #openstack-operators | 14:08 | |
| *** bvandenh has quit IRC | 14:30 | |
| *** openstackgerrit has quit IRC | 14:31 | |
| *** openstackgerrit has joined #openstack-operators | 14:32 | |
| *** cbrown2_ocf has joined #openstack-operators | 14:32 | |
| *** cbrown2_ocf has quit IRC | 14:33 | |
| *** jkraj has quit IRC | 14:37 | |
| *** bapalm has quit IRC | 14:43 | |
| *** bapalm has joined #openstack-operators | 14:43 | |
| *** fawadkhaliq has joined #openstack-operators | 14:44 | |
| *** mriedem has joined #openstack-operators | 14:45 | |
| *** cbrown2_ocf has joined #openstack-operators | 14:49 | |
| *** ToMiles has joined #openstack-operators | 14:51 | |
| *** cbrown2_ocf has quit IRC | 14:52 | |
| *** laron has joined #openstack-operators | 14:54 | |
| *** harshs has joined #openstack-operators | 14:58 | |
| *** kencjohnston has quit IRC | 14:58 | |
| *** krobzaur has joined #openstack-operators | 15:01 | |
| *** krobzaur_ has quit IRC | 15:03 | |
| *** signed8bit has joined #openstack-operators | 15:04 | |
| *** subscope has joined #openstack-operators | 15:08 | |
| *** harshs has quit IRC | 15:10 | |
| *** liverpooler has quit IRC | 15:16 | |
| *** jeh has quit IRC | 15:17 | |
| *** csoukup has joined #openstack-operators | 15:27 | |
| *** berendt has quit IRC | 15:30 | |
| *** krobzaur has quit IRC | 15:32 | |
| *** kencjohnston has joined #openstack-operators | 15:39 | |
| *** krobzaur has joined #openstack-operators | 15:40 | |
| *** kencjohnston has quit IRC | 15:41 | |
| *** kencjohnston has joined #openstack-operators | 15:47 | |
| *** mdorman has joined #openstack-operators | 15:49 | |
| *** mriedem is now known as mriedem_meeting | 15:56 | |
| *** rcernin has quit IRC | 16:09 | |
| *** maishsk has quit IRC | 16:10 | |
| *** maishsk has joined #openstack-operators | 16:12 | |
| *** kencjohnston has quit IRC | 16:13 | |
| *** elemoine has joined #openstack-operators | 16:13 | |
| *** kencjohnston has joined #openstack-operators | 16:14 | |
| *** maishsk_ has joined #openstack-operators | 16:17 | |
| *** maishsk has quit IRC | 16:20 | |
| *** maishsk_ is now known as maishsk | 16:20 | |
| *** fawadkhaliq has quit IRC | 16:20 | |
| *** alop has joined #openstack-operators | 16:21 | |
| *** belmoreira has quit IRC | 16:24 | |
| *** dminer has joined #openstack-operators | 16:25 | |
| *** Piet has joined #openstack-operators | 16:38 | |
| *** Marga_ has quit IRC | 16:40 | |
| *** Marga_ has joined #openstack-operators | 16:42 | |
| openstackgerrit | Merged openstack/osops-tools-contrib: Adding the README https://review.openstack.org/245316 | 16:42 |
|---|---|---|
| *** SimonChung1 has quit IRC | 16:43 | |
| *** britthouser has quit IRC | 16:46 | |
| *** maishsk has quit IRC | 16:47 | |
| mdorman | @j^2 my calendar says there’s an osops irc meeting today. is that true? | 16:52 |
| *** britthouser has joined #openstack-operators | 16:54 | |
| *** fawadkhaliq has joined #openstack-operators | 16:54 | |
| *** fawadkhaliq has quit IRC | 16:55 | |
| *** fawadkhaliq has joined #openstack-operators | 16:55 | |
| *** cbrown2_ocf has joined #openstack-operators | 16:57 | |
| j^2 | Mdorman: I'm planning on being there. I don't control the agenda though raginbajin does | 16:58 |
| *** belmoreira has joined #openstack-operators | 16:58 | |
| raginbajin | mdorman: There should be. We didn't have previous one because it was right after the summit and people were probably a bit tired. | 16:59 |
| j^2 | I'd like to start the code stds today too | 17:00 |
| *** lhcheng has joined #openstack-operators | 17:01 | |
| *** lhcheng_ has joined #openstack-operators | 17:02 | |
| *** lhcheng has quit IRC | 17:02 | |
| raginbajin | That seems good to me. I think I understood as that when the summit was over the ideas and work would start up around then since we'd have more feedback from the summit. | 17:04 |
| mdorman | j^2 / raginbajin sounds good | 17:05 |
| *** lhcheng_ is now known as lhcheng | 17:05 | |
| balajin | mdorman: j^2: raginbajin: where can i find further details of the meeting | 17:06 |
| balajin | would like to add to my calendar and if free will join the meeting | 17:06 |
| balajin | thanks in advance | 17:06 |
| *** matrohon has quit IRC | 17:07 | |
| *** mriedem_meeting is now known as mriedem | 17:08 | |
| raginbajin | http://eavesdrop.openstack.org/#Operators_Monitoring/Ops_Tools_Working_Group | 17:08 |
| raginbajin | That has the current schedule as well as an iCal that you can add for the meeting. | 17:08 |
| raginbajin | balajin you can also see all the logs from our meetings here: http://eavesdrop.openstack.org/meetings/operators_ops_tools_monitoring/2015/ | 17:09 |
| balajin | raginbajin: thank you | 17:10 |
| balajin | http://eavesdrop.openstack.org/#OpenStack_Operators_Team_Meeting - is this the same as the LDT operators meeting? | 17:13 |
| raginbajin | Those are all different meetings | 17:17 |
| raginbajin | I believe anyways | 17:18 |
| raginbajin | maybe klindgren knows | 17:18 |
| klindgren | LDT is an odd schedule | 17:19 |
| j^2 | klindgren: we should remove the "old" meeting then | 17:19 |
| klindgren | its every month - alternating timezones | 17:19 |
| klindgren | I dont see an LDT meeting in the eavesdrop link above? | 17:20 |
| klindgren | the OPS one isn't LDT | 17:20 |
| *** gyee has joined #openstack-operators | 17:20 | |
| j^2 | ah cool | 17:21 |
| *** belmoreira has quit IRC | 17:25 | |
| *** krobzaur has quit IRC | 17:25 | |
| *** maishsk has joined #openstack-operators | 17:27 | |
| mdorman | so is the “OpenStack Operators Team Meeting” the “old” meeting that’s no longer in place? | 17:32 |
| mdorman | confused about all this | 17:32 |
| balajin | haha | 17:33 |
| balajin | so we do not have an ops meeting in 27 mins from now | 17:33 |
| mdorman | well the one 27 mins from now is “Operators Monitoring/Ops Tools Working Group” right? | 17:34 |
| mdorman | er | 17:35 |
| balajin | mdorman: nope, that is 87 mins from now | 17:35 |
| balajin | :) | 17:35 |
| balajin | or rather 85 | 17:35 |
| mdorman | yeah you’re right. | 17:35 |
| *** elemoine has quit IRC | 17:39 | |
| j^2 | mdorman, balajin: wait what? I thought everything falls under the "Tools Working group" | 17:39 |
| j^2 | that's the one that i've put OSOps in | 17:40 |
| balajin | j^2: there is a 10a PST meeting which is not happening any more | 17:40 |
| balajin | the 11a PST is the tools working group meeting | 17:40 |
| *** armax has joined #openstack-operators | 17:41 | |
| j^2 | balajin: so we've condensed everything into: http://eavesdrop.openstack.org/#Operators_Monitoring/Ops_Tools_Working_Group right? | 17:42 |
| j^2 | or am i missign something? | 17:42 |
| balajin | j^2: i think that is hte conclusion that i am hearing so far | 17:44 |
| j^2 | awesome, good, and to confirm that is 1300CST or 1100PST? | 17:45 |
| balajin | yup | 17:45 |
| j^2 | rockon, ok i'm in the clear now, but raginbajin do we have an etherpad with the agenda on it? or anything? | 17:45 |
| j^2 | if not, we should start putting them together for at least the next meeting | 17:46 |
| *** SimonChung has joined #openstack-operators | 17:47 | |
| *** cbrown2_ocf has quit IRC | 17:51 | |
| mdorman | ok so sticking with the 11am PST one then. got it. | 17:52 |
| mdorman | how do we get that other one off that wiki? | 17:52 |
| mriedem | nova could use some ops feedback on an issue related to nova-manage db archive_deleted_instances, spelled out in this thread http://lists.openstack.org/pipermail/openstack-dev/2015-November/079778.html | 17:52 |
| mdorman | not a wiki, i guess. | 17:52 |
| j^2 | mdorman: i think it's just a remove in infra, i can take that if you don't want to figure it out | 17:54 |
| mdorman | sure that’d be great :) | 17:54 |
| *** verdurin has quit IRC | 17:55 | |
| j^2 | :rockon: on it | 17:55 |
| *** ToMiles has quit IRC | 17:56 | |
| j^2 | i think i'll propose to convert the tools working group to OSOps or something like that so we all know where to meet from now on | 17:59 |
| j^2 | so we don't have this confusion againg | 17:59 |
| mdorman | yeah that sounds good | 18:02 |
| balajin | thanks j^2 | 18:03 |
| j^2 | :D i'm here to whip us into shape :D | 18:03 |
| j^2 | yall don't like chef, and i want to be part of the community so fuckit, i'll get OSOps off the ground :D | 18:03 |
| j^2 | balajin: well you do ;) | 18:04 |
| balajin | ha ha | 18:04 |
| *** belmoreira has joined #openstack-operators | 18:04 | |
| balajin | j^2: i have taken over 6+ months to start liking chef | 18:05 |
| *** belmoreira has quit IRC | 18:05 | |
| raginbajin | So it's called the Monitoring and Tools workign group because at all the Summits there have been sessions and large discussions around it. | 18:05 |
| balajin | and now that i understand how to use chef, i am slowly trying to move everything that we want to do for config management and deployment to chef | 18:05 |
| raginbajin | I just don't want anyone to go and create another one because we renamed it to OSOps | 18:06 |
| j^2 | raginbajin: could we add OSOps in the title? | 18:06 |
| raginbajin | The Operators group meeting was an old one that was just for Operators to meet. It was very generic. | 18:06 |
| raginbajin | Absolutely | 18:06 |
| *** alop has quit IRC | 18:06 | |
| j^2 | ok, lets table this and we can talk about in the meeting | 18:06 |
| j^2 | balajin: yeah it takes a while, but as soon as you get your head wrapped around it it's crazy powerful :D | 18:07 |
| j^2 | balajin, mdorman, klindgren, raginbajin: https://review.openstack.org/247070 | 18:08 |
| balajin | agreed | 18:11 |
| j^2 | raginbajin: https://review.openstack.org/247075 With the wording it seemed correct to put OSOps where i did. Totally open to suggestions | 18:12 |
| *** Marga_ has quit IRC | 18:13 | |
| raginbajin | That seems to work for me.. We could drop the Monitoring and Tools part in the future once everyone is comfortable and just keep the it in the description so people can see what it is about. | 18:14 |
| raginbajin | But for now I think that works great. | 18:14 |
| j^2 | perfect :D | 18:15 |
| *** signed8b_ has joined #openstack-operators | 18:17 | |
| j^2 | added to the OSOps wiki too: https://wiki.openstack.org/wiki/Osops#Contents need the agendas, eventually, but it's a start | 18:19 |
| *** signed8bit has quit IRC | 18:19 | |
| *** regXboi has joined #openstack-operators | 18:20 | |
| *** regXboi has quit IRC | 18:21 | |
| *** fawadkhaliq has quit IRC | 18:21 | |
| *** verdurin has joined #openstack-operators | 18:22 | |
| *** harshs has joined #openstack-operators | 18:43 | |
| *** signed8b_ is now known as signed8bit_ZZZzz | 18:44 | |
| *** signed8bit_ZZZzz is now known as signed8b_ | 18:47 | |
| *** bvandenh has joined #openstack-operators | 18:49 | |
| *** kencjohnston_ has joined #openstack-operators | 18:51 | |
| *** kencjohnston has quit IRC | 18:54 | |
| *** signed8bit has joined #openstack-operators | 19:03 | |
| *** signed8b_ has quit IRC | 19:04 | |
| *** Guest12059 is now known as med_ | 19:04 | |
| *** med_ has quit IRC | 19:04 | |
| *** med_ has joined #openstack-operators | 19:04 | |
| *** harlowja has quit IRC | 19:05 | |
| *** VW has joined #openstack-operators | 19:07 | |
| *** VW has quit IRC | 19:07 | |
| *** harlowja has joined #openstack-operators | 19:08 | |
| *** Marga_ has joined #openstack-operators | 19:19 | |
| *** mriedem has quit IRC | 19:20 | |
| *** bapalm has quit IRC | 19:23 | |
| *** elemoine has joined #openstack-operators | 19:24 | |
| *** bapalm has joined #openstack-operators | 19:26 | |
| *** mriedem has joined #openstack-operators | 19:30 | |
| *** Marga_ has quit IRC | 19:47 | |
| *** laron has quit IRC | 19:54 | |
| *** Rockyg has joined #openstack-operators | 20:01 | |
| *** laron_ has joined #openstack-operators | 20:02 | |
| *** subscope has quit IRC | 20:06 | |
| *** subscope has joined #openstack-operators | 20:06 | |
| *** laron_ has quit IRC | 20:06 | |
| *** laron has joined #openstack-operators | 20:07 | |
| *** subscope has quit IRC | 20:09 | |
| *** subscope has joined #openstack-operators | 20:10 | |
| *** Marga_ has joined #openstack-operators | 20:14 | |
| *** bvandenh has quit IRC | 20:15 | |
| *** subscope has quit IRC | 20:17 | |
| *** lhcheng has quit IRC | 20:35 | |
| *** harlowja has quit IRC | 20:36 | |
| *** harlowja has joined #openstack-operators | 20:41 | |
| *** Marga_ has quit IRC | 20:45 | |
| *** verdurin has quit IRC | 20:49 | |
| *** vinsh_ has joined #openstack-operators | 20:51 | |
| *** rebase has joined #openstack-operators | 20:53 | |
| balajin | klindgren: when you are done with the openstack-conductor meeting, mind sending a summary to the ops mail? | 20:54 |
| *** vinsh has quit IRC | 20:54 | |
| balajin | i am guessing there is lot of interest there given the recent post about deprecating local conductor in the next major release | 20:54 |
| klindgren | yea - I dunno how long that will take though? | 20:55 |
| *** vinsh has joined #openstack-operators | 21:00 | |
| *** vinsh_ has quit IRC | 21:00 | |
| *** laron has quit IRC | 21:01 | |
| *** openstack has joined #openstack-operators | 21:04 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 21:04 | |
| *** alejandrito has quit IRC | 21:04 | |
| *** elemoine has quit IRC | 21:12 | |
| *** verdurin has joined #openstack-operators | 21:18 | |
| *** laron has joined #openstack-operators | 21:18 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 21:19 | |
| *** lhcheng has joined #openstack-operators | 21:24 | |
| *** verdurin has quit IRC | 21:31 | |
| *** Rockyg has quit IRC | 21:31 | |
| *** maishsk_ has joined #openstack-operators | 21:34 | |
| *** Marga_ has joined #openstack-operators | 21:34 | |
| *** maishsk has quit IRC | 21:35 | |
| *** maishsk_ is now known as maishsk | 21:35 | |
| *** Marga_ has quit IRC | 21:39 | |
| *** Marga_ has joined #openstack-operators | 21:40 | |
| *** elo has quit IRC | 21:44 | |
| *** kencjohnston_ has quit IRC | 21:45 | |
| drwahl | anyone have any scripts to help find orphaned neutron ports? | 21:47 |
| drwahl | was about to write something up and figured someone else had to have had this done already | 21:47 |
| drwahl | but, i don't see it in the osops-tools-generic repo | 21:47 |
| mdorman | klindgren may have something, i’m not sure. | 21:48 |
| drwahl | i thought i saw something floating around at one time, but can't seem to find it now | 21:48 |
| klindgren | only did some one liners around fips | 21:49 |
| klindgren | nothing around ports | 21:49 |
| drwahl | bleh, useless ;) | 21:49 |
| drwahl | ok, i'll try to come up with something and get it up for review | 21:50 |
| drwahl | thanks guys | 21:50 |
| klindgren | drwahl, if you also want to do routers - that would be awesome | 21:53 |
| drwahl | orphaned routers? | 21:54 |
| drwahl | or orphaned ports on routers? | 21:54 |
| klindgren | both? | 21:54 |
| *** VW has joined #openstack-operators | 21:54 | |
| *** VW has quit IRC | 21:55 | |
| drwahl | wfm | 21:55 |
| *** VW has joined #openstack-operators | 21:55 | |
| klindgren | I think orphaned ports should find orphaned ports on routers | 21:56 |
| drwahl | ya, i think so | 21:57 |
| *** verdurin has joined #openstack-operators | 21:59 | |
| *** verdurin has quit IRC | 22:03 | |
| *** lhcheng has quit IRC | 22:04 | |
| *** lhcheng has joined #openstack-operators | 22:15 | |
| *** harshs has quit IRC | 22:26 | |
| klindgren | Those people following remote conductor work.... | 22:28 |
| klindgren | Turned off SSL to rabbitmq *MASSIVE* drop in cpu consumption by conductor | 22:28 |
| dansmith | klindgren: so I expect you were paying a lot for SSLing the heartbeats | 22:29 |
| dansmith | based on the number of calls | 22:29 |
| klindgren | we disabled heartbeats and it didn't really make a change | 22:29 |
| dansmith | klindgren: okay, but the call counts don't match (even close) between those | 22:30 |
| dansmith | well, maybe that's how it's being used, I dunno | 22:30 |
| dansmith | klindgren: but if conductor is idle, I dunno how else to explain the drop when not doing things | 22:30 |
| dansmith | but whatever | 22:30 |
| klindgren | well went went from being 9-10% cpu usage on conductor when not really doing anything with it using 20-30% cpu per worker when doing stuff | 22:31 |
| *** verdurin has joined #openstack-operators | 22:31 | |
| klindgren | to bascially never higher than 1.3% | 22:31 |
| klindgren | most of the time around .7% | 22:31 |
| mgagne | wow | 22:32 |
| dansmith | mgagne: are you using ssl to rabbit? | 22:32 |
| mgagne | dansmith I'm not yet | 22:32 |
| dansmith | klindgren: out of curiosity, what is the thinking about why ssl was a good idea in the first place? | 22:32 |
| klindgren | dansmith, security req | 22:32 |
| mgagne | dansmith OpenStack Security Guide recommendations? =) | 22:32 |
| dansmith | klindgren: presumably your management network is private enough to not sorry about anything right? | 22:32 |
| klindgren | dont pass stuff in plain text | 22:33 |
| dansmith | mgagne: oh, does it recommend that? | 22:33 |
| mgagne | dansmith yea, I heard that one once | 22:33 |
| mgagne | dansmith of course | 22:33 |
| dansmith | the reason I say that is that there's really no auth on the other end | 22:33 |
| serverascode | we run rabbit with ssl too, though not sure how common it is | 22:33 |
| klindgren | dansmith, that doesn't stop the security folks from being like - zomg plaintext stuff on the netwokr | 22:33 |
| klindgren | death | 22:33 |
| dansmith | so if it's really just snooping, then whatever, but doesn't seem worth it too me, especially given today's revelation | 22:34 |
| *** maishsk_ has joined #openstack-operators | 22:34 | |
| mgagne | klindgren can't you configure client cert auth too? | 22:34 |
| dansmith | one compute node can trivially impersonate another, so if you're concerned about something untrusted on that networ, | 22:34 |
| dansmith | you're pretty much screwed anyway | 22:34 |
| dansmith | the credentials for rabbit are on every node, so it's really not even worth sniffing the network for an attacker | 22:35 |
| *** rick_ has joined #openstack-operators | 22:35 | |
| balajin | drwahl: http://git.openstack.org/cgit/openstack/osops-tools-generic/tree/neutron/listorphans.py | 22:35 |
| klindgren | dansmith, not disagreeing but - I dont think I am going to be able to get the sign-off from another team | 22:36 |
| dansmith | klindgren: okay, but you'll attribute the capital expense PO to SSL not conductor, right? | 22:37 |
| *** maishsk has quit IRC | 22:37 | |
| *** maishsk_ is now known as maishsk | 22:37 | |
| drwahl | balajin: thx, that might get me what i'm looking for | 22:37 |
| dansmith | you can now very clearly document the cost of "security" | 22:37 |
| dansmith | or an SSL accelerator | 22:37 |
| klindgren | I am curious though - we have other python things that are listening to fairly high traffic rabbitmq sources and processing a large number of requests and dont see this same load | 22:37 |
| klindgren | IE notifications.info | 22:38 |
| balajin | klindgren: the reason why i want ssl is because we do not have a separate data plane + control plane and i dont want control plane data going in plain text | 22:38 |
| dansmith | balajin: scary :( | 22:38 |
| dansmith | klindgren: how long did your profiler run for that graph? | 22:38 |
| mgagne | klindgren was SSL used before kilo? | 22:38 |
| klindgren | erm ~2minutes | 22:38 |
| klindgren | mgagne, since havana | 22:39 |
| dansmith | klindgren: so that looks to me like 165 nova messages | 22:39 |
| dansmith | klindgren: over the course of two minutes | 22:39 |
| balajin | dansmith: :( yeah but that is how it is | 22:39 |
| mgagne | klindgren so although SSL is taxing a lot of CPU, it doesn't fully explain the increase in CPU usage | 22:39 |
| dansmith | klindgren: so if it's not lower level traffic, I dunno what it is | 22:39 |
| dansmith | balajin: please tell me what cloud you run so I can stay away :) | 22:39 |
| balajin | private cloud, so unless you work for us you wont use it | 22:40 |
| dansmith | balajin: okay good :) | 22:40 |
| balajin | and there is valid reason why we do that - cost | 22:40 |
| dansmith | oh sure I know why :) | 22:40 |
| dansmith | the cost of the first breach will be fun too | 22:41 |
| balajin | dansmith: may be or may be not | 22:46 |
| *** sanjayu has joined #openstack-operators | 22:47 | |
| *** csoukup has quit IRC | 22:48 | |
| *** laron has quit IRC | 22:51 | |
| *** arcimboldo has joined #openstack-operators | 22:53 | |
| *** mriedem has quit IRC | 22:54 | |
| harlowja | dansmith balajin whats the percantage of companies that do seperate it, i'd be curious to know, especially for large companies | 22:59 |
| harlowja | (if anyone knows) | 22:59 |
| harlowja | can't be cheap to install and built-out datacenters with 2 isolated networks in the same DC | 22:59 |
| dansmith | harlowja: really? | 23:00 |
| harlowja | really, i'm curious to know | 23:00 |
| dansmith | harlowja: most everyone has two nics to two TOR switches, no? | 23:00 |
| dansmith | at a minimum | 23:00 |
| harlowja | idk, i have 1 company experience/knowledge, i don't know others | 23:00 |
| harlowja | (which is why i ask) | 23:00 |
| dansmith | I have less, I guess, but still.. | 23:00 |
| dansmith | I would expect at least vlan separation | 23:00 |
| harlowja | ya, we have that at y! afaik | 23:01 |
| harlowja | but does someone like google have 2 nics to two TOR switches? | 23:01 |
| dansmith | but I know people will have like two 1G nics on the box, bonded for HA, an then two 10G nics bonded for data path | 23:01 |
| harlowja | or microsoft, or rackspace or ... | 23:01 |
| dansmith | I would surely expect they do | 23:01 |
| harlowja | got me :-P | 23:01 |
| dansmith | bonded 1G nics for control path I meant | 23:02 |
| andyhky | right, at a minimum VLANs, most rackspace setups have > 1 NIC and > 1 TOR | 23:02 |
| dansmith | yeah | 23:02 |
| balajin | dansmith: vlan separation is obvious | 23:02 |
| balajin | but vlan happing to | 23:02 |
| harlowja | k, so we have 1 company > 1 NIC and > 1 TOR and 1 company 1 nice with 1 tor + vlan | 23:02 |
| harlowja | *nic not nice | 23:02 |
| harlowja | nice nic | 23:02 |
| harlowja | lol | 23:02 |
| harlowja | balajin i think knows more of the reasons than i do though | 23:03 |
| harlowja | (or other network archictects) | 23:03 |
| balajin | fyi, we do the same thing to - multi-vlan | 23:03 |
| dansmith | I don't know of any vlan exploits, so I'm not sure why vlan justifies needing SSL | 23:03 |
| balajin | dansmith: vlan hopping? | 23:03 |
| dansmith | balajin: is that a thing? how does that happen? | 23:04 |
| dansmith | wikipedia tells me it's usually due to improper configuration :) | 23:05 |
| dansmith | anyway | 23:05 |
| serverascode | dansmith: what's wrong with incremental improvements? ie. using ssl with rabbit while we wait for all the other things we need to | 23:05 |
| serverascode | *too | 23:05 |
| *** Piet has quit IRC | 23:05 | |
| dansmith | serverascode: nothing other than that I just don't think it buys you much | 23:05 |
| balajin | dansmith: if you allow native vlans it is very much possible | 23:05 |
| dansmith | serverascode: certainly not if it requires you to run 2-4x the number of workers | 23:05 |
| klindgren | sooooo turns out I screwed up the change to non-ssl | 23:06 |
| klindgren | and it wasn't actually connecting to rabbitmq | 23:06 |
| dansmith | balajin: mkay, well, anyway.. even just for isolation of the impact of downloading an image on the paying customers, I'd expect two paths | 23:06 |
| mgagne | so even when conductor does nothing, it eats up to 7% cpu :D | 23:06 |
| dansmith | mgagne: no, 0.7% he said | 23:07 |
| klindgren | cpu usage dropped a bit | 23:07 |
| klindgren | but waiting for a while to figure out what the overhead of ssl is | 23:07 |
| mgagne | oh =( | 23:07 |
| balajin | dansmith: aboslutely but for private i dont see a need | 23:07 |
| dansmith | klindgren: okay "a bit" is much more in line with what I'd expect for no heartbeats | 23:07 |
| dansmith | klindgren: still seems like you have something else going on, given you had 165 messages over the course of two minutes | 23:08 |
| klindgren | well considering that computes are hardset to report in usage for vm's every minute or something? | 23:08 |
| klindgren | that plus metadata | 23:09 |
| dansmith | klindgren: remind me why you have 30k metadata requests per hour? | 23:09 |
| dansmith | klindgren: and what is the "some" value of actually not using ssl? | 23:10 |
| klindgren | people run puppet every minute in their vm's to talk to a metadata service | 23:11 |
| klindgren | puppet by default has the ec2 facter fact enabled | 23:11 |
| klindgren | which hits metadata for things like az, hostname, region | 23:11 |
| klindgren | ect ect | 23:11 |
| dansmith | okay | 23:12 |
| dansmith | klindgren: instance_get_by_uuid() is 3% of the total workload in that graph | 23:13 |
| dansmith | klindgren: do you have debug logging turned on such that you can see if any lazy-loading is happening? | 23:14 |
| dansmith | in compute | 23:14 |
| klindgren | I will get a cProfiler run with ssl enabled as well | 23:14 |
| dansmith | or metadata or whatever | 23:14 |
| klindgren | we don't have debug enabled | 23:14 |
| klindgren | I can enable it on a few servers though | 23:14 |
| klindgren | anything you specifically want me to look for | 23:14 |
| klindgren | ? | 23:14 |
| dansmith | klindgren: https://github.com/openstack/nova/blob/master/nova/objects/instance.py#L832-L836 | 23:16 |
| *** rick_ has quit IRC | 23:16 | |
| klindgren | root@p3plcldhv003-09 ~]# grep "Lazy-loading" /var/log/nova/nova* | 23:18 |
| klindgren | [root@p3plcldhv003-09 ~]# | 23:18 |
| klindgren | nothing on the box I enabled debug on | 23:18 |
| dansmith | klindgren: but it was enabled for like 30 seconds? | 23:19 |
| dansmith | regardless, I don't really see any evidence of that happening a lot on in the call graph either | 23:19 |
| klindgren | right - jsut saying so far I see nothing | 23:19 |
| klindgren | its still enabled | 23:20 |
| dansmith | the 100% timer thing is something I can't explain, but it might be something related to eventlet, I dunno | 23:20 |
| *** signed8bit is now known as signed8bit_ZZZzz | 23:20 | |
| klindgren | I think that might be somethign that happens during startup | 23:20 |
| klindgren | I noticed tht for a few seconds it takes 100% cpu? | 23:20 |
| *** ducttape_ has joined #openstack-operators | 23:20 | |
| klindgren | then forks the children and cpu drops | 23:20 |
| dansmith | but according to the docs on that profiler, it means that 100% of execution time was spent there | 23:20 |
| dansmith | it's percentage of execution time, not percent of a core or something | 23:21 |
| klindgren | but it also didn't actually do anything? I dont really have an answer on it either, I just passwed it off as catchign hte startup bit that did consume a bunch of cpu *shrug* | 23:23 |
| *** harshs has joined #openstack-operators | 23:23 | |
| dansmith | well, | 23:23 |
| dansmith | there's also poll() a ton | 23:23 |
| dansmith | so what I'm saying is, | 23:24 |
| dansmith | if there is some eventlet thread that is tight looping, | 23:24 |
| dansmith | you'd see that, and a bunch of cpu usage, basically any time the process wasn't busy being blocked on a mysql call | 23:24 |
| klindgren | so SSL disabled is good for a 4% reduction in cpu across all cpu's | 23:25 |
| klindgren | IE util went from average of 48% to 44% | 23:25 |
| *** VW has quit IRC | 23:26 | |
| dansmith | okay | 23:27 |
| dansmith | I wonder if that means that the bulk of the CPU is not actually passing (and encrypting) messages | 23:27 |
| dansmith | to clarify my point earlier, | 23:27 |
| dansmith | instance_get_by_uuid() is the target of both a lazy-load and of a metadata hit | 23:28 |
| dansmith | so that being 3% of the total rules out those two activities in my mind | 23:28 |
| *** ducttape_ has quit IRC | 23:28 | |
| dansmith | and really, | 23:30 |
| dansmith | dispatch_and_reply is 10% total | 23:30 |
| dansmith | which is the parent of every rpc inbound call | 23:30 |
| dansmith | so I think that means 90% of the work is being spent elsewhere | 23:30 |
| klindgren | let me run profile with ssl disabled | 23:31 |
| klindgren | cProfile* | 23:31 |
| *** dminer has quit IRC | 23:34 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 23:34 | |
| klindgren | http://tempsend.com/AD0217D818/AEA4/conductor1workernossl.svg | 23:40 |
| dansmith | still <10% handling RPC requests | 23:41 |
| dansmith | 56% in "impl_rabbit:ensure" -- whatever that is | 23:41 |
| *** ahonda has joined #openstack-operators | 23:42 | |
| *** VW has joined #openstack-operators | 23:42 | |
| dansmith | maybe the 96% in poll:wait means it's spending all that time waiting on a socket, but it surely doesn't seem like it | 23:42 |
| *** VW has quit IRC | 23:43 | |
| *** krobzaur has joined #openstack-operators | 23:46 | |
| dansmith | klindgren: that looks like ~900 rpc messages in that trace, fwiw | 23:47 |
| klindgren | it was almost exactly 4 minutes btw | 23:48 |
| dansmith | okay | 23:48 |
| *** signed8bit is now known as signed8bit_ZZZzz | 23:50 | |
| harlowja | btw, https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L581 (ensure) i think is the main dispatch call, so thats probably why its at the top there | 23:53 |
| harlowja | basically the thing that retries and handles errors... | 23:53 |
| harlowja | pretty sure its just a wrapper though | 23:54 |
| harlowja | (which is just in this case wrapping execute_method ) | 23:55 |
| harlowja | and/or _object_dispatch (?) | 23:55 |
| dansmith | harlowja: but 50% for ensure and 10% for do_dispatch()? | 23:56 |
| dansmith | 57% and 8% in this case | 23:57 |
| dansmith | where is the rest of that time going/ | 23:57 |
| harlowja | ya, thats the question | 23:57 |
| dansmith | it's also confusing that our receive/dispatch method is called "ensure" but whatever :) | 23:57 |
| harlowja | def | 23:57 |
| harlowja | ensure provided 'method' that is being wrapped works | 23:58 |
| harlowja | pretty sure thats why its called that | 23:58 |
| harlowja | kombu itself has similar code (that i think post-existed oslo.messaging ensure) | 23:58 |
| dansmith | yeah, it's just weird | 23:58 |
| harlowja | https://github.com/celery/kombu/blob/master/kombu/connection.py#L388 .. | 23:59 |
| harlowja | 'Ensure operation completes, regardless of any channel/connection | 23:59 |
| harlowja | errors occurring.' | 23:59 |
| harlowja | the oslo.messaging ensure is similar/equivalent(?) | 23:59 |
| harlowja | probably don't even need the oslo.messaging one anymore imho | 23:59 |
| harlowja | but medhi would know best | 23:59 |
| harlowja | ' | 23:59 |
| harlowja | pyamqp:90:drain_events | 23:59 |
| harlowja | 56.79%' | 23:59 |
| dansmith | I'm really curious about where that time is going | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
