| dansmith | because if we're spinning on a socket or something that would explain a lot | 00:00 |
|---|---|---|
| harlowja | which is dropping into | 00:00 |
| harlowja | method_framing:184:read_method | 00:00 |
| harlowja | 56.29% | 00:00 |
| harlowja | :-/ | 00:00 |
| harlowja | and then its gone, lol | 00:00 |
| harlowja | https://github.com/celery/py-amqp/blob/master/amqp/method_framing.py | 00:00 |
| harlowja | perhaps thats method_framing | 00:01 |
| *** mriedem has joined #openstack-operators | 00:01 | |
| harlowja | * https://github.com/celery/py-amqp/blob/1.4/amqp/method_framing.py#L184 | 00:01 |
| harlowja | ^ read_method | 00:01 |
| harlowja | that method is oddly gone in newer pyamqp | 00:01 |
| harlowja | (on master) | 00:01 |
| klindgren | I can run a different profiler or something else - or if shown how - I can profile a separate section. Like I mentioned earlier I am just starting python with the cProfiler module and saving the output - then running that call tracing thingy against it | 00:02 |
| *** krobzaur has quit IRC | 00:02 | |
| harlowja | ya, klindgren np | 00:02 |
| harlowja | its dropping off into https://github.com/celery/py-amqp/blob/1.4/amqp/method_framing.py#L104 | 00:02 |
| harlowja | which might be dropping into c code | 00:03 |
| harlowja | not sure | 00:03 |
| harlowja | which i guess is (maybe) https://github.com/celery/py-amqp/blob/1.4/amqp/transport.py#L150 | 00:03 |
| * harlowja wonders if kombu can buffer messages better | 00:04 | |
| harlowja | and its lack of buffering things makes this suck | 00:04 |
| dansmith | I guess that I think we're learning that the hotspot is not where we thought | 00:06 |
| dansmith | or I thought at least | 00:06 |
| harlowja | possibly | 00:06 |
| harlowja | shit ton of read_frame (aka, no buffering?) | 00:06 |
| dansmith | I figured it would be mostly in the ORM bits, jsonification, etc | 00:06 |
| *** zhangjn has quit IRC | 00:07 | |
| dansmith | but it seems like we've got good data on all that and it's a small part of the total | 00:07 |
| harlowja | maybe | 00:08 |
| klindgren | dansmith, checkign back to that box we enabled debug on | 00:08 |
| *** zhangjn has joined #openstack-operators | 00:08 | |
| *** zhangjn has quit IRC | 00:08 | |
| klindgren | I do see a bunch of hits for Lazy-loading | 00:08 |
| dansmith | klindgren: bunch like..? | 00:09 |
| *** zhangjn has joined #openstack-operators | 00:09 | |
| *** zhangjn has quit IRC | 00:09 | |
| klindgren | on this box grouped into 2 requests - 3-4 a minute | 00:09 |
| klindgren | sometimes 8 | 00:10 |
| dansmith | yeah, so .. that's nothing | 00:10 |
| dansmith | what are they loading? | 00:10 |
| dansmith | pci_crapolium_somethingorother/ | 00:10 |
| klindgren | metadata system_metadata | 00:10 |
| *** zhangjn has joined #openstack-operators | 00:10 | |
| klindgren | info_cache | 00:10 |
| klindgren | mainly metadata | 00:11 |
| dansmith | ah, | 00:11 |
| dansmith | that's the periodic task that doesn't pre-fetch that stuff until it determines it needs it | 00:11 |
| dansmith | so, doing what it's supposed to | 00:11 |
| klindgren | also - we run metadata on every compute node | 00:11 |
| klindgren | so I would imagine they are all doing it | 00:11 |
| klindgren | ls | 00:11 |
| dansmith | that's not because of the metadata service, especially if you're seeing it in compute log | 00:12 |
| klindgren | 1 hit on info_cache which was compute | 00:12 |
| klindgren | everything else came from metadata.log | 00:13 |
| dansmith | okay, well, still not even close to the problem | 00:13 |
| j^2 | I just posted this: bit.ly/successful-chatroom-meetings, i'm going to try to stick to it | 00:13 |
| dansmith | klindgren: can you try starting up a conductor worker with the topic mangled so it gets no traffic/ | 00:19 |
| klindgren | sure | 00:20 |
| dansmith | klindgren: if it sits there, connected but with no traffic, and burns cpu, then that means it's being really stupid | 00:20 |
| dansmith | I don't expect this to yield anything, but if it does, it'll be much more telling | 00:20 |
| *** lhcheng has quit IRC | 00:20 | |
| *** zhangjn has quit IRC | 00:21 | |
| *** lhcheng has joined #openstack-operators | 00:21 | |
| klindgren | topic=donothing in the conductor section | 00:22 |
| klindgren | one process is doing like 0.7-1% cpu everything else is a solid .3 | 00:23 |
| dansmith | yeah | 00:23 |
| klindgren | so effectively idle | 00:23 |
| dansmith | but we get multiple connections to rabbit over time right? | 00:23 |
| dansmith | I mean, not with this, but when there is real load | 00:24 |
| klindgren | right | 00:24 |
| dansmith | I'm not sure why it's not zero in this config, I mean | 00:24 |
| dansmith | and when you start up a conductor cold, it goes straight to 48% or whatever number right? | 00:25 |
| dansmith | it doesn't take a few minutes to warm up or anything? | 00:25 |
| klindgren | it usually burns 100% cpu for a few seconds | 00:26 |
| klindgren | then pretty pquickly the processes start taking cpu | 00:26 |
| dansmith | hmm, that doesn't really make any sense to me either | 00:26 |
| klindgren | which bounce around but usually look like: | 00:26 |
| klindgren | http://paste.ubuntu.com/13243560/ | 00:26 |
| dansmith | klindgren: what does a nova-compute look like? | 00:27 |
| dansmith | any similar behavior? | 00:27 |
| dansmith | all the underlying machinery is pretty much the same | 00:27 |
| klindgren | yea nova compute burns 100% for a few seconds | 00:27 |
| klindgren | and drops off to say 2-3% | 00:27 |
| dansmith | hmm | 00:28 |
| klindgren | (just did a restart test) | 00:28 |
| klindgren | 2015-11-18 17:21:48.154 4043 INFO nova.service [-] Starting donothing node (version 2015.1.0) - didn't realize that was using the topic name | 00:29 |
| *** arcimboldo has quit IRC | 00:30 | |
| *** krobzaur has joined #openstack-operators | 00:30 | |
| dansmith | klindgren: can you measure actual throughput of traffic between rabbit and a conductor? | 00:31 |
| klindgren | with conductor conenct to the correct topic - we typical see a bunch of: http://paste.ubuntu.com/13337010/ | 00:31 |
| dansmith | right | 00:31 |
| klindgren | dansmith, how would I do that? | 00:31 |
| dansmith | klindgren: iftop, iptraf, etc? | 00:31 |
| *** rebase has quit IRC | 00:32 | |
| klindgren | so in this particular case rabbitmq and conductor is on the same host | 00:33 |
| klindgren | right now | 00:33 |
| dansmith | oh really | 00:33 |
| klindgren | yea | 00:33 |
| klindgren | each cell server is running rabbitmq for the cell as well | 00:34 |
| klindgren | with each cell member preferring its local rabbit in the list of hosts (IE its first) | 00:34 |
| dansmith | so iftop will work on lo, and will display ports so you can look at just those connections | 00:35 |
| dansmith | I'm just curious if we're actually getting a lot of traffic from rabbit and having to do something with it, | 00:36 |
| dansmith | or if we're spinning on a lack of traffic | 00:36 |
| dansmith | it won't likely lead anywhere, I'm just trying to think of next steps | 00:36 |
| dansmith | I'm guessing that getting some o.msg person to look is what we really should be doing | 00:36 |
| dansmith | that, and me getting dinner | 00:37 |
| j^2 | yay! | 00:37 |
| j^2 | http://eavesdrop.openstack.org/#OpenStack_OSOps/Monitoring_and_Tools_Working_Group | 00:37 |
| harlowja | ya, i start to wonder if oslo.messaging should buffer better ,idk | 00:41 |
| harlowja | *and/or kombu | 00:41 |
| harlowja | but this starts to get into using a message queue as a queue vs an RPC thing | 00:42 |
| *** mdorman has quit IRC | 00:44 | |
| dansmith | I don't see why this starts to get into rpc-over-queue, but okay :) | 00:44 |
| harlowja | some semantics might be lost if we start buffering things more into memory and memory goes poof | 00:46 |
| harlowja | but i guess memory can already go poof due to other problems, and we don't handle it anyway, lol | 00:46 |
| *** Piet has joined #openstack-operators | 00:49 | |
| *** sanjayu has quit IRC | 00:52 | |
| *** krobzaur has quit IRC | 00:55 | |
| *** krobzaur has joined #openstack-operators | 00:55 | |
| *** rebase has joined #openstack-operators | 00:59 | |
| *** zhangjn has joined #openstack-operators | 01:07 | |
| *** zhangjn_ has joined #openstack-operators | 01:08 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 01:11 | |
| *** zhangjn has quit IRC | 01:12 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 01:12 | |
| *** SimonChung has quit IRC | 01:24 | |
| *** fawadkhaliq has joined #openstack-operators | 01:26 | |
| *** sgordon has quit IRC | 01:47 | |
| *** sgordon has joined #openstack-operators | 01:49 | |
| *** krobzaur has quit IRC | 01:50 | |
| *** dims_ has joined #openstack-operators | 01:54 | |
| *** gyee has quit IRC | 01:55 | |
| *** fawadkhaliq has quit IRC | 01:57 | |
| *** krobzaur has joined #openstack-operators | 02:05 | |
| *** dims has joined #openstack-operators | 02:09 | |
| *** dims_ has quit IRC | 02:13 | |
| *** krobzaur has quit IRC | 02:16 | |
| *** ducttape_ has joined #openstack-operators | 02:17 | |
| *** openstack has joined #openstack-operators | 02:26 | |
| -cameron.freenode.net- [freenode-info] if you're at a conference and other people are having trouble connecting, please mention it to staff: http://freenode.net/faq.shtml#gettinghelp | 02:26 | |
| *** SimonChung has joined #openstack-operators | 02:30 | |
| *** kencjohnston has joined #openstack-operators | 02:33 | |
| *** mriedem has quit IRC | 02:36 | |
| *** armax has quit IRC | 02:41 | |
| *** ducttape_ has quit IRC | 02:45 | |
| *** krobzaur has joined #openstack-operators | 02:58 | |
| *** rebase has quit IRC | 03:14 | |
| *** bapalm has quit IRC | 03:15 | |
| *** kencjohnston has quit IRC | 03:21 | |
| *** bapalm has joined #openstack-operators | 03:21 | |
| *** krobzaur has quit IRC | 03:23 | |
| *** kencjohnston has joined #openstack-operators | 03:23 | |
| *** harshs has quit IRC | 03:27 | |
| *** csoukup has joined #openstack-operators | 03:32 | |
| *** dims has quit IRC | 03:32 | |
| *** kencjohnston has quit IRC | 03:37 | |
| *** gfa_ is now known as gfa | 03:48 | |
| *** signed8bit_ZZZzz has quit IRC | 03:51 | |
| *** armax has joined #openstack-operators | 03:54 | |
| *** lhcheng has quit IRC | 04:03 | |
| *** Marga_ has quit IRC | 04:27 | |
| *** fawadkhaliq has joined #openstack-operators | 04:28 | |
| *** csoukup has quit IRC | 04:32 | |
| *** VW has joined #openstack-operators | 04:43 | |
| *** VW has quit IRC | 04:48 | |
| *** kencjohnston has joined #openstack-operators | 04:52 | |
| *** lhcheng has joined #openstack-operators | 04:53 | |
| *** britthouser has quit IRC | 05:20 | |
| *** britthouser has joined #openstack-operators | 05:21 | |
| *** kencjohnston has quit IRC | 05:38 | |
| *** jkraj has joined #openstack-operators | 05:57 | |
| *** britthouser has quit IRC | 06:01 | |
| *** subscope has joined #openstack-operators | 06:21 | |
| *** rcernin has joined #openstack-operators | 06:23 | |
| *** lhcheng has quit IRC | 06:35 | |
| *** SimonChung1 has joined #openstack-operators | 06:45 | |
| *** SimonChung has quit IRC | 06:48 | |
| *** elemoine has joined #openstack-operators | 06:48 | |
| *** elemoine__ has joined #openstack-operators | 06:57 | |
| *** elemoine__ has left #openstack-operators | 07:05 | |
| *** sanjayu has joined #openstack-operators | 07:12 | |
| *** liverpooler has joined #openstack-operators | 07:27 | |
| *** elemoine has quit IRC | 07:27 | |
| *** elemoine has joined #openstack-operators | 07:43 | |
| *** zhangjn_ has quit IRC | 07:54 | |
| *** zhangjn has joined #openstack-operators | 07:56 | |
| *** belmoreira has joined #openstack-operators | 08:00 | |
| *** markvoelker has quit IRC | 08:00 | |
| *** maishsk has quit IRC | 08:03 | |
| *** matrohon has joined #openstack-operators | 08:09 | |
| *** egonzalez has joined #openstack-operators | 08:26 | |
| *** fawadkhaliq has quit IRC | 08:26 | |
| *** armax has quit IRC | 08:32 | |
| *** vinsh_ has joined #openstack-operators | 08:33 | |
| *** vinsh has quit IRC | 08:35 | |
| *** clayton has quit IRC | 08:35 | |
| *** phchoi1 has joined #openstack-operators | 08:42 | |
| *** clayton has joined #openstack-operators | 08:44 | |
| *** VW has joined #openstack-operators | 08:45 | |
| *** VW has quit IRC | 08:49 | |
| *** markvoelker has joined #openstack-operators | 09:01 | |
| *** markvoelker has quit IRC | 09:06 | |
| *** egonzalez has quit IRC | 09:09 | |
| *** maishsk has joined #openstack-operators | 09:30 | |
| *** egonzalez has joined #openstack-operators | 09:36 | |
| *** cbrown2_ocf has joined #openstack-operators | 09:37 | |
| *** elemoine has quit IRC | 09:44 | |
| *** elemoine_ has joined #openstack-operators | 09:49 | |
| *** openstackgerrit has quit IRC | 10:01 | |
| *** openstackgerrit has joined #openstack-operators | 10:02 | |
| *** maishsk has quit IRC | 10:04 | |
| *** subscope has quit IRC | 10:07 | |
| *** subscope has joined #openstack-operators | 10:08 | |
| *** subscope has quit IRC | 10:08 | |
| *** gfa is now known as gfa_ | 10:08 | |
| *** arcimboldo has joined #openstack-operators | 10:12 | |
| *** fawadkhaliq has joined #openstack-operators | 10:14 | |
| *** openstack has joined #openstack-operators | 10:19 | |
| *** subscope has joined #openstack-operators | 10:25 | |
| *** dims has joined #openstack-operators | 10:45 | |
| *** logan2 has quit IRC | 11:01 | |
| *** logan2 has joined #openstack-operators | 11:01 | |
| *** markvoelker has joined #openstack-operators | 11:02 | |
| *** markvoelker has quit IRC | 11:07 | |
| *** zhangjn has quit IRC | 11:07 | |
| *** jmccrory has quit IRC | 11:20 | |
| *** jmccrory has joined #openstack-operators | 11:22 | |
| *** jmccrory has quit IRC | 11:26 | |
| *** egonzalez has quit IRC | 11:27 | |
| *** jmccrory has joined #openstack-operators | 11:30 | |
| *** zhangjn has joined #openstack-operators | 11:33 | |
| *** subscope has quit IRC | 11:39 | |
| *** subscope has joined #openstack-operators | 11:40 | |
| *** subscope has quit IRC | 11:40 | |
| *** jkraj has quit IRC | 12:09 | |
| *** dims has quit IRC | 12:16 | |
| *** markvoelker has joined #openstack-operators | 12:33 | |
| raginbajin | Has anyone every ran into an issue where Neutron Gateway's start showing a lot of "BUG: soft lockup - CPU#0 stuck for 22s" | 12:37 |
| *** markvoelker has quit IRC | 12:38 | |
| clayton | I've seen that in other situations | 12:50 |
| *** egonzalez has joined #openstack-operators | 12:56 | |
| *** subscope has joined #openstack-operators | 12:57 | |
| *** bvandenh has joined #openstack-operators | 13:11 | |
| *** signed8bit has joined #openstack-operators | 13:13 | |
| *** ducttape_ has joined #openstack-operators | 13:14 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 13:16 | |
| *** liverpooler has quit IRC | 13:18 | |
| *** fawadkhaliq has quit IRC | 13:22 | |
| *** signed8bit_ZZZzz has quit IRC | 13:32 | |
| *** signed8bit has joined #openstack-operators | 13:32 | |
| *** markvoelker has joined #openstack-operators | 13:35 | |
| *** bvandenh has quit IRC | 13:36 | |
| *** ducttape_ has quit IRC | 13:36 | |
| *** regXboi has joined #openstack-operators | 13:36 | |
| *** laron has joined #openstack-operators | 13:37 | |
| *** VW has joined #openstack-operators | 13:46 | |
| *** VW has quit IRC | 13:47 | |
| *** ccarmack has joined #openstack-operators | 13:56 | |
| *** ccarmack has left #openstack-operators | 13:57 | |
| *** arcimboldo has quit IRC | 13:59 | |
| *** sanjayu has quit IRC | 14:02 | |
| *** ccarmack has joined #openstack-operators | 14:10 | |
| *** ccarmack has left #openstack-operators | 14:10 | |
| *** arcimboldo has joined #openstack-operators | 14:20 | |
| *** dims has joined #openstack-operators | 14:27 | |
| *** mriedem has joined #openstack-operators | 14:28 | |
| *** arcimboldo has quit IRC | 14:34 | |
| *** kencjohnston has joined #openstack-operators | 14:35 | |
| *** britthouser has joined #openstack-operators | 14:37 | |
| *** k_stev has joined #openstack-operators | 14:37 | |
| *** phchoi1 has quit IRC | 14:48 | |
| *** arcimboldo has joined #openstack-operators | 14:51 | |
| *** dminer has joined #openstack-operators | 14:51 | |
| *** k_stev has quit IRC | 14:55 | |
| *** kencjohnston has quit IRC | 14:56 | |
| *** DarthVigil has joined #openstack-operators | 14:59 | |
| *** kencjohnston has joined #openstack-operators | 14:59 | |
| *** kencjohnston has quit IRC | 15:00 | |
| *** kencjohnston has joined #openstack-operators | 15:01 | |
| *** rcernin has quit IRC | 15:03 | |
| *** phchoi1 has joined #openstack-operators | 15:05 | |
| *** ducttape_ has joined #openstack-operators | 15:08 | |
| *** ducttape_ has quit IRC | 15:09 | |
| *** ducttape_ has joined #openstack-operators | 15:09 | |
| *** dminer has quit IRC | 15:16 | |
| *** DarthVigil has quit IRC | 15:20 | |
| *** laron has quit IRC | 15:21 | |
| *** Piet has joined #openstack-operators | 15:25 | |
| *** k_stev has joined #openstack-operators | 15:25 | |
| *** ggillies has quit IRC | 15:29 | |
| *** dminer has joined #openstack-operators | 15:30 | |
| *** ggillies has joined #openstack-operators | 15:31 | |
| *** laron has joined #openstack-operators | 15:42 | |
| *** mdorman has joined #openstack-operators | 15:46 | |
| *** laron has quit IRC | 15:46 | |
| *** fawadkhaliq has joined #openstack-operators | 15:47 | |
| *** harshs has joined #openstack-operators | 15:47 | |
| *** krobzaur has joined #openstack-operators | 15:55 | |
| *** zul has quit IRC | 16:02 | |
| *** laron has joined #openstack-operators | 16:02 | |
| *** k_stev has quit IRC | 16:02 | |
| *** zul has joined #openstack-operators | 16:04 | |
| *** SimonChung1 has quit IRC | 16:10 | |
| *** csoukup has joined #openstack-operators | 16:12 | |
| *** elo has joined #openstack-operators | 16:17 | |
| *** armax has joined #openstack-operators | 16:18 | |
| *** fawadkhaliq has quit IRC | 16:19 | |
| *** fawadkhaliq has joined #openstack-operators | 16:20 | |
| *** subscope has quit IRC | 16:24 | |
| *** subscope has joined #openstack-operators | 16:24 | |
| *** subscope has quit IRC | 16:25 | |
| *** arcimboldo has quit IRC | 16:26 | |
| *** gfa_ is now known as gfa | 16:28 | |
| *** fawadkhaliq has quit IRC | 16:29 | |
| *** harshs has quit IRC | 16:33 | |
| *** ducttape_ has quit IRC | 16:33 | |
| *** ducttape_ has joined #openstack-operators | 16:34 | |
| *** subscope has joined #openstack-operators | 16:35 | |
| *** subscope has quit IRC | 16:46 | |
| *** DarthVigil has joined #openstack-operators | 16:49 | |
| *** alejandrito has joined #openstack-operators | 16:49 | |
| *** elo has quit IRC | 16:50 | |
| *** fawadkhaliq has joined #openstack-operators | 16:50 | |
| *** belmoreira has quit IRC | 16:51 | |
| *** jlk has quit IRC | 16:51 | |
| *** jlk has joined #openstack-operators | 16:51 | |
| *** gyee has joined #openstack-operators | 16:51 | |
| *** vinsh_ is now known as Vinsh | 16:51 | |
| *** jlk has quit IRC | 16:51 | |
| *** jlk has joined #openstack-operators | 16:51 | |
| *** krobzaur has quit IRC | 16:56 | |
| *** elo has joined #openstack-operators | 17:07 | |
| *** alop has joined #openstack-operators | 17:12 | |
| *** laron has quit IRC | 17:13 | |
| *** krobzaur has joined #openstack-operators | 17:14 | |
| *** elo has quit IRC | 17:18 | |
| *** elo has joined #openstack-operators | 17:18 | |
| *** SimonChung has joined #openstack-operators | 17:20 | |
| *** laron has joined #openstack-operators | 17:21 | |
| *** mdorman has quit IRC | 17:21 | |
| *** mdorman has joined #openstack-operators | 17:25 | |
| *** dims has quit IRC | 17:36 | |
| *** matrohon has quit IRC | 17:43 | |
| *** egonzalez has quit IRC | 17:44 | |
| *** DarthVigil has quit IRC | 17:48 | |
| *** krobzaur has quit IRC | 17:48 | |
| *** krobzaur has joined #openstack-operators | 17:57 | |
| *** dims has joined #openstack-operators | 18:06 | |
| *** lhcheng has joined #openstack-operators | 18:06 | |
| *** phchoi1 has quit IRC | 18:07 | |
| *** lhcheng_ has joined #openstack-operators | 18:08 | |
| *** harshs has joined #openstack-operators | 18:09 | |
| *** lhcheng has quit IRC | 18:11 | |
| *** armax has quit IRC | 18:14 | |
| *** gyee has quit IRC | 18:20 | |
| *** dims has quit IRC | 18:26 | |
| *** ducttape_ has quit IRC | 18:33 | |
| *** ducttape_ has joined #openstack-operators | 18:34 | |
| *** stanchan has joined #openstack-operators | 18:35 | |
| *** lhcheng_ has quit IRC | 18:41 | |
| *** VW has joined #openstack-operators | 18:48 | |
| *** david-lyle has quit IRC | 18:48 | |
| *** phchoi1 has joined #openstack-operators | 18:48 | |
| *** VW has quit IRC | 18:52 | |
| *** dims has joined #openstack-operators | 18:52 | |
| *** fawadkhaliq has quit IRC | 18:54 | |
| *** ducttape_ has quit IRC | 18:57 | |
| *** VW has joined #openstack-operators | 18:58 | |
| *** VW has quit IRC | 18:59 | |
| *** VW has joined #openstack-operators | 19:00 | |
| *** armax has joined #openstack-operators | 19:01 | |
| xavpaice | uh, yep... followed by a watchdog timeout and the box crashing | 19:02 |
| xavpaice | raginbajin: ^^ | 19:03 |
| *** ducttape_ has joined #openstack-operators | 19:06 | |
| *** cbrown2_ocf has quit IRC | 19:08 | |
| *** gfa is now known as gfa_ | 19:14 | |
| *** mriedem has quit IRC | 19:19 | |
| *** lhcheng has joined #openstack-operators | 19:22 | |
| *** VW has quit IRC | 19:28 | |
| *** electrofelix has quit IRC | 19:29 | |
| *** mriedem has joined #openstack-operators | 19:31 | |
| *** Piet has quit IRC | 19:31 | |
| *** krobzaur has quit IRC | 19:34 | |
| *** dims has quit IRC | 19:36 | |
| *** david-lyle has joined #openstack-operators | 19:43 | |
| *** elo has quit IRC | 19:46 | |
| *** openstackgerrit has quit IRC | 19:46 | |
| *** openstackgerrit has joined #openstack-operators | 19:47 | |
| *** elo has joined #openstack-operators | 19:57 | |
| *** elo has quit IRC | 19:57 | |
| *** laron_ has joined #openstack-operators | 19:57 | |
| *** maishsk has joined #openstack-operators | 19:58 | |
| *** laron_ has quit IRC | 19:59 | |
| *** laron has quit IRC | 19:59 | |
| *** laron has joined #openstack-operators | 20:02 | |
| *** maishsk_ has joined #openstack-operators | 20:11 | |
| *** laron has quit IRC | 20:12 | |
| *** krobzaur has joined #openstack-operators | 20:12 | |
| *** maishsk has quit IRC | 20:14 | |
| *** maishsk_ is now known as maishsk | 20:14 | |
| *** Piet has joined #openstack-operators | 20:17 | |
| *** cbrown2_ocf has joined #openstack-operators | 20:23 | |
| *** Piet has quit IRC | 20:24 | |
| *** elo has joined #openstack-operators | 20:25 | |
| *** rebase has joined #openstack-operators | 20:29 | |
| *** rebase has quit IRC | 20:31 | |
| *** elo has quit IRC | 20:31 | |
| *** elo has joined #openstack-operators | 20:31 | |
| *** britthouser has quit IRC | 20:37 | |
| *** britthouser_ has joined #openstack-operators | 20:37 | |
| *** britthouser_ is now known as britthouser | 20:41 | |
| *** harshs has quit IRC | 20:43 | |
| *** elo has quit IRC | 20:44 | |
| *** elo has joined #openstack-operators | 20:48 | |
| *** harshs has joined #openstack-operators | 20:50 | |
| *** B_Smith__ has quit IRC | 20:54 | |
| *** maishsk has quit IRC | 20:55 | |
| *** maishsk has joined #openstack-operators | 20:56 | |
| *** B_Smith_ has joined #openstack-operators | 20:56 | |
| *** armax has quit IRC | 21:00 | |
| *** Piet has joined #openstack-operators | 21:13 | |
| *** krobzaur has quit IRC | 21:23 | |
| darrenc | hi everyone, the ops guide meeting will start in 8 minutes in #openstack-meeting-3 | 21:23 |
| *** maishsk has quit IRC | 21:24 | |
| *** ducttape_ has quit IRC | 21:31 | |
| *** tpeoples has joined #openstack-operators | 21:32 | |
| *** maishsk has joined #openstack-operators | 21:34 | |
| mriedem | j^2: which is the operators meeting in http://eavesdrop.openstack.org/ ? | 21:44 |
| mriedem | like general ops | 21:45 |
| j^2 | mriedem: one sec | 21:45 |
| j^2 | mriedem: http://eavesdrop.openstack.org/#OpenStack_OSOps/Monitoring_and_Tools_Working_Group we've absorbed this one | 21:45 |
| mriedem | ah | 21:47 |
| j^2 | mriedem: how can i help? | 21:47 |
| mriedem | metacloud guys aren't even here! | 21:47 |
| mriedem | j^2: nic from metacloud was asking when that meeting was | 21:49 |
| mriedem | i have an agenda item for next time | 21:49 |
| mriedem | prereading: http://lists.openstack.org/pipermail/openstack-dev/2015-November/079778.html | 21:49 |
| j^2 | mriedem: nice, we have a posted agenda now, so please don't hesitate to put things on it | 21:50 |
| mriedem | where is the agenda? | 21:50 |
| mriedem | it's not in that link | 21:50 |
| j^2 | https://wiki.openstack.org/wiki/Osops | 21:51 |
| j^2 | it's on that page, i guess i need to update that post, my bad | 21:51 |
| j^2 | https://wiki.openstack.org/wiki/Osops/Osops-agenda | 21:51 |
| mriedem | cool, thanks | 21:52 |
| j^2 | and it sends you to an etherpad, I've found more success with etherpad then wiki pages for agendas | 21:52 |
| *** harshs has quit IRC | 21:54 | |
| *** ducttape_ has joined #openstack-operators | 21:55 | |
| mriedem | ok, i noticed the agenda was crickets | 21:57 |
| j^2 | yeah it was just created yesterday ;) | 21:58 |
| j^2 | hey, we're trying to get things in shape, baby steps | 21:58 |
| mriedem | i saw the ML thread | 21:58 |
| j^2 | i posted about it on the ML :) | 21:58 |
| j^2 | yeah | 21:58 |
| mriedem | 'we're terrible, let's do better!' | 21:58 |
| j^2 | mriedem: in a way, fuck yes. | 21:58 |
| mriedem | always a good iea | 21:58 |
| mriedem | *idea | 21:58 |
| j^2 | raginbajin: those notes? *poke* *poke* | 21:58 |
| j^2 | mriedem: i have faith that we can do better, make it a operators community that the foundation can acknologe, then we can all get ATC and then we can enjoy the same privs that devs do | 21:59 |
| j^2 | I'm going to make this happen come hell or high water | 22:00 |
| j^2 | even if yall still don't like chef ;) | 22:00 |
| raginbajin | j^2 Yes Yes Yes.. Sorry. Just been a tad busy and hopefully should be able to put them together tonight and send them out tonight or in the morning. | 22:00 |
| *** harshs has joined #openstack-operators | 22:00 | |
| *** ctrath has joined #openstack-operators | 22:00 | |
| j^2 | raginbajin: rock on, i'll keep ontop of you for em ;) | 22:00 |
| raginbajin | j^2 That's be great.. Attention Span is **stares into space** | 22:01 |
| mriedem | j^2: oh yes, ATC privileges, you wouldn't imagine the VIP status we get at the design summits :) | 22:01 |
| mriedem | there might be like, an urn with coffee in it that's not in the marketing orgy hall | 22:01 |
| j^2 | raginbajin: :D | 22:02 |
| j^2 | mriedem: and that's what i'm trying to fix, we run these damn things, it's not achedemic anymore, it's time they realize that Operators have to create tools and develop fixes to make customers happy | 22:03 |
| *** dims has joined #openstack-operators | 22:03 | |
| j^2 | anyway, you get what i'm trying to say, we've all said it, and i've taken the sisyphean task to wake people up | 22:03 |
| j^2 | all i need is input, to make sure this is what _we_ want i refuse to be a dictator in this thing | 22:04 |
| *** jamespd has joined #openstack-operators | 22:05 | |
| mriedem | alright, item added https://etherpad.openstack.org/p/osops-irc-meeting-template | 22:05 |
| *** kencjohnston has quit IRC | 22:05 | |
| *** cburgess has joined #openstack-operators | 22:06 | |
| j^2 | wrong one ;) | 22:07 |
| j^2 | that's the template | 22:07 |
| j^2 | the link below it is the link to the agenda for the next meeting | 22:08 |
| mriedem | gah | 22:11 |
| j^2 | heh | 22:11 |
| *** mdorman has quit IRC | 22:12 | |
| xavpaice | raginbajin: got any logs from the soft lockup issues in a paste? | 22:12 |
| *** elo has quit IRC | 22:12 | |
| j^2 | https://etherpad.openstack.org/p/osops-irc-meeting-20151202 | 22:14 |
| *** maishsk_ has joined #openstack-operators | 22:14 | |
| mriedem | might want to rename that etherpad link https://wiki.openstack.org/wiki/Osops/Osops-agenda | 22:14 |
| mriedem | since it says chef | 22:14 |
| *** maishsk has quit IRC | 22:16 | |
| *** maishsk_ is now known as maishsk | 22:16 | |
| j^2 | shit, i cargo culted it ;) | 22:16 |
| j^2 | fixed | 22:17 |
| *** krobzaur has joined #openstack-operators | 22:26 | |
| *** elo has joined #openstack-operators | 22:27 | |
| *** harshs has quit IRC | 22:28 | |
| *** lhcheng has quit IRC | 22:29 | |
| *** elo has quit IRC | 22:31 | |
| *** lhcheng has joined #openstack-operators | 22:31 | |
| j^2 | mriedem: https://review.openstack.org/247818 | 22:34 |
| j^2 | to add to the eavesdrop.openstack.org | 22:34 |
| *** harshs has joined #openstack-operators | 22:36 | |
| openstackgerrit | David Wahlstrom proposed openstack/osops-tools-contrib: neutron/orphaned_ports.py: initial commit https://review.openstack.org/247823 | 22:44 |
| drwahl | ^^ that script i mentioned yesterday | 22:45 |
| drwahl | i played with the 'listorphans.py' script, but it didn't do as thorough as a job as i would have liked | 22:45 |
| drwahl | the listorphans.py script relies on the APIs being able to report "everything" correctly, which i've learned is not usually a reliable way to list "everything" | 22:46 |
| drwahl | using listorphans.py, i found zero orphaned ports | 22:46 |
| drwahl | using my script, i found 2 orphaned ports (in our cluster) | 22:47 |
| *** mriedem has quit IRC | 22:53 | |
| balajin | drwahl: modify and PR please | 22:55 |
| balajin | should we need a complete new script? | 22:55 |
| drwahl | they are radically different approaches... | 22:56 |
| drwahl | one is asking the openstack APIs for what they know and determining what is orphaned from that | 22:57 |
| drwahl | mine is asking the database what is knows and comparing outputs | 22:57 |
| drwahl | mine isn't even touching the APIs. it's all just db queries | 22:57 |
| drwahl | i mean, i guess i could try to update listorphans.py, but it seems like a *completely* different code path | 22:58 |
| drwahl | there would be almost nothing shared | 22:58 |
| *** regXboi has quit IRC | 22:59 | |
| *** lhcheng has quit IRC | 23:00 | |
| *** mdorman has joined #openstack-operators | 23:01 | |
| *** krobzaur has quit IRC | 23:01 | |
| *** csoukup has quit IRC | 23:02 | |
| *** k_stev has joined #openstack-operators | 23:06 | |
| *** stanchan has quit IRC | 23:08 | |
| *** ctrath has quit IRC | 23:10 | |
| *** logan2 has quit IRC | 23:10 | |
| *** logan2 has joined #openstack-operators | 23:10 | |
| *** alejandrito has quit IRC | 23:11 | |
| *** lhcheng has joined #openstack-operators | 23:13 | |
| *** alop has quit IRC | 23:15 | |
| *** ctrath has joined #openstack-operators | 23:15 | |
| *** sanjayu has joined #openstack-operators | 23:16 | |
| *** k_stev has quit IRC | 23:19 | |
| *** alop has joined #openstack-operators | 23:20 | |
| *** dims_ has joined #openstack-operators | 23:22 | |
| *** ducttape_ has quit IRC | 23:24 | |
| *** dims has quit IRC | 23:24 | |
| *** zhangjn has quit IRC | 23:33 | |
| *** darrenc is now known as darrenc_afk | 23:33 | |
| *** zhangjn has joined #openstack-operators | 23:38 | |
| *** zhangjn has quit IRC | 23:38 | |
| *** zhangjn has joined #openstack-operators | 23:39 | |
| *** zhangjn has quit IRC | 23:39 | |
| *** dminer has quit IRC | 23:40 | |
| *** zhangjn has joined #openstack-operators | 23:40 | |
| *** mdorman has quit IRC | 23:41 | |
| *** k_stev has joined #openstack-operators | 23:42 | |
| *** gfa_ is now known as gfa | 23:53 | |
| *** darrenc_afk is now known as darrenc | 23:56 | |
| *** alop_ has joined #openstack-operators | 23:59 | |
| *** alop has quit IRC | 23:59 | |
| *** alop_ is now known as alop | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!