| *** ducttape_ has quit IRC | 00:01 | |
| *** TxGirlGeek has quit IRC | 00:21 | |
| *** ducttape_ has joined #openstack-operators | 00:32 | |
| *** emagana has quit IRC | 00:43 | |
| *** VW has quit IRC | 00:43 | |
| mriedem | erhudy: isn't that a bug in nova? | 00:50 |
|---|---|---|
| *** VW has joined #openstack-operators | 00:57 | |
| *** eloy__ has joined #openstack-operators | 01:01 | |
| *** eloy__ is now known as Eloy | 01:05 | |
| *** harlowja has joined #openstack-operators | 01:06 | |
| *** Eloy is now known as Eloy_ | 01:09 | |
| *** ducttape_ has quit IRC | 01:10 | |
| *** ducttape_ has joined #openstack-operators | 01:11 | |
| *** vinsh has joined #openstack-operators | 01:12 | |
| *** ducttape_ has quit IRC | 01:15 | |
| *** VW has quit IRC | 01:20 | |
| *** haplo37 has quit IRC | 01:22 | |
| *** VW has joined #openstack-operators | 01:26 | |
| *** haplo37 has joined #openstack-operators | 01:30 | |
| *** ducttape_ has joined #openstack-operators | 01:37 | |
| *** haplo37 has quit IRC | 01:39 | |
| *** haplo37_ has joined #openstack-operators | 01:41 | |
| *** harlowja has quit IRC | 01:44 | |
| *** eloy__ has joined #openstack-operators | 01:50 | |
| *** Eloy_ has quit IRC | 01:53 | |
| *** ducttape_ has quit IRC | 01:56 | |
| *** ducttape_ has joined #openstack-operators | 01:57 | |
| *** VW has quit IRC | 02:00 | |
| *** ducttape_ has quit IRC | 02:17 | |
| *** eloy__ has quit IRC | 02:27 | |
| *** jamesdenton has joined #openstack-operators | 03:12 | |
| *** ducttape_ has joined #openstack-operators | 03:18 | |
| *** ducttape_ has quit IRC | 03:23 | |
| *** jamesdenton has quit IRC | 03:24 | |
| *** fragatina has joined #openstack-operators | 03:56 | |
| *** VW has joined #openstack-operators | 03:58 | |
| *** ducttape_ has joined #openstack-operators | 04:49 | |
| *** ducttape_ has quit IRC | 04:53 | |
| *** udesale has joined #openstack-operators | 05:04 | |
| *** flaviodsr has quit IRC | 05:20 | |
| *** VW has quit IRC | 05:30 | |
| *** twiggy has joined #openstack-operators | 05:45 | |
| *** ducttape_ has joined #openstack-operators | 06:19 | |
| *** Miouge has joined #openstack-operators | 06:23 | |
| *** ducttape_ has quit IRC | 06:24 | |
| *** VW has joined #openstack-operators | 06:31 | |
| *** emccormickva has joined #openstack-operators | 06:35 | |
| *** VW has quit IRC | 06:36 | |
| *** emccormick has quit IRC | 06:38 | |
| *** armax has joined #openstack-operators | 06:39 | |
| *** mriedem has quit IRC | 06:49 | |
| *** tesseract has joined #openstack-operators | 07:08 | |
| *** tesseract is now known as Guest29114 | 07:09 | |
| *** Oku_OS-away is now known as Oku_OS | 07:19 | |
| *** bvandenh has quit IRC | 07:33 | |
| *** armax has quit IRC | 07:34 | |
| *** pcaruana has joined #openstack-operators | 07:34 | |
| *** armax has joined #openstack-operators | 07:36 | |
| *** armax has quit IRC | 07:41 | |
| *** ducttape_ has joined #openstack-operators | 07:50 | |
| *** ducttape_ has quit IRC | 07:54 | |
| *** pontusf1 has joined #openstack-operators | 08:05 | |
| *** rcernin has joined #openstack-operators | 08:06 | |
| *** pontusf has quit IRC | 08:09 | |
| *** VW has joined #openstack-operators | 08:33 | |
| *** VW has quit IRC | 08:37 | |
| *** ducttape_ has joined #openstack-operators | 08:50 | |
| *** bvandenh has joined #openstack-operators | 08:54 | |
| *** ducttape_ has quit IRC | 08:55 | |
| *** rarcea has joined #openstack-operators | 08:55 | |
| *** ducttape_ has joined #openstack-operators | 09:00 | |
| *** zioproto has joined #openstack-operators | 09:01 | |
| *** ducttape_ has quit IRC | 09:05 | |
| *** rcernin has quit IRC | 09:11 | |
| *** chlong has quit IRC | 09:34 | |
| *** dbecker has quit IRC | 09:35 | |
| *** VW has joined #openstack-operators | 09:35 | |
| *** dbecker has joined #openstack-operators | 09:36 | |
| dewanee | erhudy, yes we do | 09:40 |
| *** VW has quit IRC | 09:40 | |
| dewanee | that's exactly what we tried | 09:40 |
| dewanee | but it's really hacky | 09:40 |
| dewanee | how many open files have you allowed? | 09:41 |
| dewanee | our situation is slightly different though | 09:41 |
| *** derekh has joined #openstack-operators | 09:41 | |
| dewanee | we think we had the problem you describe on liberty | 09:42 |
| dewanee | on mitaka though the connection to the rabbit are preserved | 09:42 |
| dewanee | so the nova-compute appears to be up on the service list | 09:42 |
| dewanee | there's an open bug \anyhow if someone is interested: | 09:43 |
| dewanee | https://bugs.launchpad.net/nova/+bug/1651526 | 09:43 |
| openstack | Launchpad bug 1651526 in OpenStack Compute (nova) "nova-compute reports OSError: [Errno 24] Too many open files" [Undecided,New] | 09:43 |
| *** mriedem has joined #openstack-operators | 09:44 | |
| *** chatter has joined #openstack-operators | 09:56 | |
| chatter | hey guys | 09:56 |
| *** openstackgerrit has joined #openstack-operators | 09:56 | |
| openstackgerrit | Saverio Proto proposed openstack/osops-tools-contrib: l3-agent-evacuate.py: evacuate a network node https://review.openstack.org/401409 | 09:56 |
| *** ducttape_ has joined #openstack-operators | 10:02 | |
| *** slaweq_ has quit IRC | 10:02 | |
| *** chatter has quit IRC | 10:04 | |
| yankcrime | klindgren_: i've a bunch that are down but i've got a problem with a dead ovs-agent on one network node that's confusing things | 10:04 |
| *** chatter has joined #openstack-operators | 10:04 | |
| *** chatter has quit IRC | 10:06 | |
| yankcrime | (which of course i can't restart because of neutron-ovs-cleanup being called) | 10:06 |
| *** chatter has joined #openstack-operators | 10:06 | |
| *** ducttape_ has quit IRC | 10:07 | |
| *** chatter has quit IRC | 10:08 | |
| *** chatter has joined #openstack-operators | 10:08 | |
| *** slaweq_ has joined #openstack-operators | 10:08 | |
| *** electrofelix has joined #openstack-operators | 10:10 | |
| *** chatter has quit IRC | 10:13 | |
| *** zioproto has quit IRC | 10:25 | |
| *** simon-AS559 has joined #openstack-operators | 10:32 | |
| *** ducttape_ has joined #openstack-operators | 11:02 | |
| *** udesale has quit IRC | 11:06 | |
| *** ducttape_ has quit IRC | 11:07 | |
| *** bvandenh has quit IRC | 11:13 | |
| *** slaweq__ has joined #openstack-operators | 11:32 | |
| *** VW has joined #openstack-operators | 11:37 | |
| *** dbecker has quit IRC | 11:42 | |
| *** VW has quit IRC | 11:42 | |
| *** dbecker has joined #openstack-operators | 11:46 | |
| *** ducttape_ has joined #openstack-operators | 12:03 | |
| *** ducttape_ has quit IRC | 12:08 | |
| *** rcernin has joined #openstack-operators | 12:26 | |
| *** rmart04 has joined #openstack-operators | 12:32 | |
| *** VW has joined #openstack-operators | 12:41 | |
| *** bvandenh has joined #openstack-operators | 12:49 | |
| *** pcaruana has quit IRC | 12:49 | |
| *** vinsh has quit IRC | 12:57 | |
| *** matrohon has joined #openstack-operators | 12:59 | |
| *** ducttape_ has joined #openstack-operators | 13:04 | |
| *** ducttape_ has quit IRC | 13:06 | |
| *** ducttape_ has joined #openstack-operators | 13:06 | |
| *** ducttape_ has quit IRC | 13:07 | |
| *** ducttape_ has joined #openstack-operators | 13:23 | |
| *** VW has quit IRC | 13:27 | |
| *** pcaruana has joined #openstack-operators | 13:32 | |
| *** armax has joined #openstack-operators | 13:36 | |
| *** armax has quit IRC | 13:49 | |
| *** ducttape_ has quit IRC | 13:51 | |
| *** mriedem has quit IRC | 14:01 | |
| *** udesale has joined #openstack-operators | 14:02 | |
| *** matt_jarvis has joined #openstack-operators | 14:05 | |
| *** dminer has joined #openstack-operators | 14:10 | |
| *** matt_jarvis has quit IRC | 14:15 | |
| *** jamesdenton has joined #openstack-operators | 14:17 | |
| *** liverpooler has joined #openstack-operators | 14:21 | |
| *** liverpooler has quit IRC | 14:21 | |
| *** liverpooler has joined #openstack-operators | 14:21 | |
| *** jamesdenton has quit IRC | 14:35 | |
| *** jamesdenton has joined #openstack-operators | 14:35 | |
| *** pcaruana has quit IRC | 14:38 | |
| *** vinsh has joined #openstack-operators | 14:41 | |
| *** rmart04 has quit IRC | 14:42 | |
| *** rmart04 has joined #openstack-operators | 14:43 | |
| *** rmart04_ has joined #openstack-operators | 14:44 | |
| *** rmart04 has quit IRC | 14:47 | |
| *** rmart04_ is now known as rmart04 | 14:47 | |
| *** vinsh has quit IRC | 14:48 | |
| *** pcaruana has joined #openstack-operators | 14:51 | |
| *** twiggy has quit IRC | 15:04 | |
| *** mriedem has joined #openstack-operators | 15:05 | |
| *** mriedem has quit IRC | 15:06 | |
| *** mriedem has joined #openstack-operators | 15:06 | |
| *** rmart04 has quit IRC | 15:16 | |
| *** VW has joined #openstack-operators | 15:28 | |
| *** markvoelker has quit IRC | 15:29 | |
| *** TxGirlGeek has joined #openstack-operators | 15:29 | |
| *** markvoelker has joined #openstack-operators | 15:31 | |
| *** VW has quit IRC | 15:32 | |
| *** twiggy has joined #openstack-operators | 15:33 | |
| *** zioproto has joined #openstack-operators | 15:39 | |
| *** openstackgerrit_ has joined #openstack-operators | 15:47 | |
| *** openstackgerrit_ has quit IRC | 15:49 | |
| *** chlong has joined #openstack-operators | 15:52 | |
| *** fragatina has quit IRC | 15:57 | |
| *** fragatina has joined #openstack-operators | 15:57 | |
| *** dtrainor has quit IRC | 15:59 | |
| erhudy | mriedem: we delegated looking into it to canonical for the time being because we're in EOY lockdown and can't really monkey with production clusters | 16:02 |
| erhudy | dewanee: i changed it to 4096/4096 from 1024/4096 | 16:03 |
| erhudy | in a cluster with 700 OSDs, i think it started happening sporadically when we got to somewhere around 600-650 OSDs | 16:04 |
| *** dtrainor has joined #openstack-operators | 16:04 | |
| erhudy | nova-compute itself would never log anything once it died, i only figured out what was going on via strace | 16:05 |
| mriedem | erhudy: ok, thanks. i'm honestly not sure what in nova would do that, it could be something in rbd itself or how nova is using it, not sure. it would be interesting to see if others like cburgess or nic have seen the same thing in their ceph deployments. | 16:07 |
| *** udesale has quit IRC | 16:07 | |
| erhudy | yeah, part of the reason i haven't filed an LP bug is the affected cluster is still on kilo and it could be in the RBD bindings or librbd itself as well | 16:07 |
| *** pcaruana has quit IRC | 16:13 | |
| *** Guest29114 has quit IRC | 16:15 | |
| *** ducttape_ has joined #openstack-operators | 16:23 | |
| *** ducttape_ has quit IRC | 16:27 | |
| *** TxGirlGeek has quit IRC | 16:31 | |
| *** Miouge has quit IRC | 16:45 | |
| *** VW has joined #openstack-operators | 16:46 | |
| *** VW has quit IRC | 16:46 | |
| *** Miouge has joined #openstack-operators | 16:50 | |
| *** karad has quit IRC | 17:01 | |
| *** openstack has quit IRC | 17:02 | |
| *** openstack has joined #openstack-operators | 17:04 | |
| *** mriedem has quit IRC | 17:07 | |
| *** mriedem has joined #openstack-operators | 17:08 | |
| *** derekh has quit IRC | 17:09 | |
| *** rcernin has quit IRC | 17:10 | |
| *** Oku_OS is now known as Oku_OS-away | 17:12 | |
| *** flaviodsr has joined #openstack-operators | 17:15 | |
| *** ducttape_ has joined #openstack-operators | 17:23 | |
| *** zioproto has quit IRC | 17:23 | |
| *** ducttape_ has quit IRC | 17:36 | |
| *** karad has joined #openstack-operators | 17:41 | |
| *** ducttape_ has joined #openstack-operators | 17:44 | |
| *** TxGirlGeek has joined #openstack-operators | 17:47 | |
| *** simon-AS559 has quit IRC | 17:52 | |
| *** ducttape_ has quit IRC | 17:57 | |
| *** ducttape_ has joined #openstack-operators | 18:00 | |
| *** slaweq_ has quit IRC | 18:11 | |
| *** slaweq__ has quit IRC | 18:12 | |
| *** slaweq has joined #openstack-operators | 18:13 | |
| *** slaweq_ has joined #openstack-operators | 18:13 | |
| *** karad has quit IRC | 18:14 | |
| *** sterdnotshaken has joined #openstack-operators | 18:14 | |
| sterdnotshaken | What is the most unicast vxlan tunnels you have seen openstack build using ovs on a compute nodes? 10,000? More? | 18:15 |
| *** ducttape_ has quit IRC | 18:18 | |
| *** slaweq_ has quit IRC | 18:21 | |
| *** slaweq has quit IRC | 18:22 | |
| *** ducttape_ has joined #openstack-operators | 18:30 | |
| *** armax has joined #openstack-operators | 18:33 | |
| *** pcaruana has joined #openstack-operators | 18:34 | |
| *** Miouge has quit IRC | 18:36 | |
| *** simon-AS559 has joined #openstack-operators | 18:42 | |
| *** simon-AS559 has quit IRC | 18:46 | |
| *** VW has joined #openstack-operators | 18:47 | |
| *** slaweq has joined #openstack-operators | 18:49 | |
| *** slaweq_ has joined #openstack-operators | 18:49 | |
| *** VW has quit IRC | 18:51 | |
| *** fragatina has quit IRC | 18:51 | |
| *** armax has quit IRC | 19:00 | |
| *** ducttape_ has quit IRC | 19:00 | |
| *** simon-AS559 has joined #openstack-operators | 19:13 | |
| raddaoui | sterdnotshaken: good question I am using 1000 in our cloud but in this blog http://www.opencloudblog.com/?p=300 they are using 69999. I guess you can use any number to 16 million (2 power 24) whcih is the size of vnid | 19:15 |
| *** electrofelix has quit IRC | 19:23 | |
| sterdnotshaken | raddaoui: Thanks for your answer! So you currently have one compute node with 1000 unicast vxlan tunnels (going east\west to other compute nodes) then? | 19:23 |
| sterdnotshaken | I looked at your link, but I could only see they were using the VNI ranges between 65537:69999… did I miss something? I'm just curious if anyone has ever tried to actually stress-test the tenant unicast vxlan setup to see if compute nodes can actually run thousands or 10's of thousands of vxlan tunnels…? | 19:25 |
| *** twiggy has quit IRC | 19:34 | |
| *** ducttape_ has joined #openstack-operators | 19:35 | |
| raddaoui | sterdnotshaken: no It is just configured to use 1:1000 range but not actually using them. yeah I am curious to see a benchmarking test on that | 19:37 |
| *** sterdnotshaken has quit IRC | 19:37 | |
| *** ducttape_ has quit IRC | 19:48 | |
| *** ducttape_ has joined #openstack-operators | 19:52 | |
| *** pcaruana has quit IRC | 20:11 | |
| *** ducttape_ has quit IRC | 20:12 | |
| *** dtrainor has quit IRC | 20:21 | |
| *** chlong has quit IRC | 20:36 | |
| *** sterdnotshaken has joined #openstack-operators | 20:37 | |
| *** chlong has joined #openstack-operators | 20:52 | |
| *** ducttape_ has joined #openstack-operators | 21:12 | |
| *** liverpooler has quit IRC | 21:14 | |
| *** sterdnotshaken has quit IRC | 21:17 | |
| *** ducttape_ has quit IRC | 21:17 | |
| *** jamesdenton has quit IRC | 21:20 | |
| *** rarcea has quit IRC | 21:30 | |
| *** VW has joined #openstack-operators | 21:39 | |
| *** matrohon has quit IRC | 21:52 | |
| *** sterdnotshaken has joined #openstack-operators | 21:53 | |
| *** matrohon has joined #openstack-operators | 21:56 | |
| dewanee | erhudy, our nova still logs stuff: I think that was a big change since nova 13.1.2 | 22:00 |
| dewanee | if you take a look at: https://bugs.launchpad.net/nova/+bug/1607461 | 22:00 |
| openstack | Launchpad bug 1607461 in OpenStack Compute (nova) "nova-compute hangs while executing a blocking call to librbd" [Medium,Fix released] - Assigned to Roman Podoliaka (rpodolyaka) | 22:00 |
| dewanee | "To make sure nova-compute remains responsive we need to wrap all the calls in tpool.Proxy() eventlet helper, that switches the execution context back to the event loop, while the call is executed in a native OS thread from a pool. | 22:01 |
| dewanee | " | 22:02 |
| *** sterdnotshaken has quit IRC | 22:04 | |
| *** chlong has quit IRC | 22:04 | |
| *** Miouge has joined #openstack-operators | 22:04 | |
| *** sterdnotshaken has joined #openstack-operators | 22:04 | |
| *** sterdnotshaken1 has joined #openstack-operators | 22:05 | |
| *** sterdnotshaken has quit IRC | 22:09 | |
| *** VW has quit IRC | 22:11 | |
| *** dminer has quit IRC | 22:14 | |
| *** jamesdenton has joined #openstack-operators | 22:22 | |
| *** mriedem has quit IRC | 22:24 | |
| *** kstev has joined #openstack-operators | 22:25 | |
| *** sterdnotshaken1 has quit IRC | 22:41 | |
| *** ducttape_ has joined #openstack-operators | 22:43 | |
| *** ducttape_ has quit IRC | 22:48 | |
| *** Miouge has quit IRC | 22:48 | |
| *** TxGirlGeek has quit IRC | 22:48 | |
| *** kstev has quit IRC | 22:52 | |
| *** sterdnotshaken has joined #openstack-operators | 23:00 | |
| *** flaviodsr has quit IRC | 23:00 | |
| *** slaweq has quit IRC | 23:04 | |
| *** slaweq_ has quit IRC | 23:04 | |
| *** saneax-_-|AFK is now known as saneax | 23:09 | |
| *** slaweq has joined #openstack-operators | 23:12 | |
| *** sterdnotshaken has quit IRC | 23:18 | |
| *** simon-AS559 has quit IRC | 23:22 | |
| *** matrohon has quit IRC | 23:34 | |
| *** sterdnotshaken has joined #openstack-operators | 23:43 | |
| *** sterdnotshaken has quit IRC | 23:44 | |
| *** catintheroof has joined #openstack-operators | 23:48 | |
| *** sterdnotshaken has joined #openstack-operators | 23:49 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!