*** miyagishi_t has joined #openstack-operators | 00:01 | |
*** SimonChung1 has quit IRC | 00:03 | |
*** SimonChung has joined #openstack-operators | 00:04 | |
*** SimonChung has quit IRC | 00:04 | |
mfisch | klindgren_: I've been pretty disappoined in kilo so far, sounds like you're having issues too | 00:17 |
---|---|---|
klindgren_ | mfisch, nothing terrible - yet | 00:18 |
klindgren_ | ipset race conditions seems to be the only thing so far | 00:18 |
mfisch | we're having dhcp issues | 00:18 |
mfisch | above and beyond what we presented in PA | 00:19 |
klindgren_ | hrm what specifically re: dhcp? | 00:19 |
mfisch | dhcp agents losing their brain and tenant VMs losing connectivity when the lease times out | 00:19 |
klindgren_ | to be honest -we use config-drive to statically IP that stuff... | 00:20 |
klindgren_ | so | 00:20 |
klindgren_ | dhcp might be broken - but we don't notice ^_^ | 00:20 |
mfisch | good choice | 00:21 |
klindgren_ | Curious - as after mfisch talked about it.... anyone else here have issues with dhcp_agent getting moved between nodes for apparently no good reason? | 00:24 |
klindgren_ | we've staitcally assigned dhcp_agents to specifical networks since havana - under kilo the assigments started randomly getting switched around | 00:25 |
mfisch | my honest advice would be wait on kilo too | 00:25 |
klindgren_ | mfisch, want to do a ML post? I can chime in on what we have seen - see if we can get more people on it. | 00:26 |
mfisch | I'd need to work with eric on it | 00:27 |
mfisch | he did more digging | 00:27 |
mfisch | we did file a bug | 00:27 |
klindgren_ | got the bug refrence? | 00:27 |
*** VW_ has joined #openstack-operators | 00:37 | |
*** VW_ has quit IRC | 00:39 | |
*** VW_ has joined #openstack-operators | 00:39 | |
*** VW_ has quit IRC | 00:44 | |
*** zhangjn has joined #openstack-operators | 00:44 | |
*** mpopow_ has joined #openstack-operators | 01:25 | |
*** mpopow has quit IRC | 01:29 | |
*** SimonChung has joined #openstack-operators | 01:31 | |
*** SimonChung1 has joined #openstack-operators | 01:33 | |
*** SimonChung has quit IRC | 01:36 | |
*** resker has joined #openstack-operators | 01:48 | |
*** esker has quit IRC | 01:52 | |
gfa | mfisch: our dhcp lease time is like a year long. we don't have problems with that. on havana we have issues when the dhcp-agent don't notify dnsmasq about the lease being released and dnsmasq refusing to give that ip again. but since icehouse is ok | 01:55 |
mfisch | I suggested that as a fix | 01:56 |
mfisch | but it just masks stuff | 01:56 |
*** zhangjn_ has joined #openstack-operators | 01:57 | |
*** zhangjn has quit IRC | 01:58 | |
*** harshs has quit IRC | 02:08 | |
*** kencjohn_ has joined #openstack-operators | 02:15 | |
*** kencjohn_ has quit IRC | 02:15 | |
*** kencjohn_ has joined #openstack-operators | 02:16 | |
*** klindgren_ has quit IRC | 02:25 | |
*** resker has quit IRC | 02:41 | |
*** gfa is now known as gfa_ | 02:46 | |
*** hakimo_ has joined #openstack-operators | 02:52 | |
*** hakimo has quit IRC | 02:54 | |
*** gfa_ is now known as gfa | 02:59 | |
*** kencjohn_ has quit IRC | 03:18 | |
*** harshs has joined #openstack-operators | 03:27 | |
*** esker has joined #openstack-operators | 03:35 | |
*** esker has quit IRC | 03:40 | |
*** stanchan has joined #openstack-operators | 03:41 | |
*** maishsk has quit IRC | 03:47 | |
*** zhangjn has joined #openstack-operators | 04:00 | |
*** zhangjn_ has quit IRC | 04:02 | |
*** csoukup has joined #openstack-operators | 04:04 | |
*** csoukup has quit IRC | 04:05 | |
*** bhunter71 has quit IRC | 04:11 | |
*** esker has joined #openstack-operators | 04:30 | |
*** esker has quit IRC | 04:34 | |
*** gfa is now known as gfa_ | 04:36 | |
*** gfa_ is now known as gfa | 04:36 | |
*** esker has joined #openstack-operators | 04:43 | |
*** harshs has quit IRC | 04:45 | |
*** esker has quit IRC | 04:49 | |
*** harshs has joined #openstack-operators | 05:00 | |
*** mpopow__ has joined #openstack-operators | 05:03 | |
*** markvoelker has joined #openstack-operators | 05:06 | |
*** mpopow_ has quit IRC | 05:07 | |
*** markvoelker has quit IRC | 05:11 | |
*** markvoelker has joined #openstack-operators | 05:12 | |
*** harshs has quit IRC | 05:23 | |
*** vsedelnik has joined #openstack-operators | 05:45 | |
*** gfa is now known as gfa_ | 05:47 | |
*** gfa_ is now known as gfa | 05:57 | |
*** priteau has joined #openstack-operators | 06:21 | |
*** markvoelker has quit IRC | 06:23 | |
*** sdake has joined #openstack-operators | 06:24 | |
*** priteau has quit IRC | 06:26 | |
*** mpopow__ has quit IRC | 06:47 | |
*** maishsk has joined #openstack-operators | 07:09 | |
*** vinsh has quit IRC | 07:34 | |
*** vinsh has joined #openstack-operators | 07:35 | |
*** esker has joined #openstack-operators | 07:48 | |
*** mpopow__ has joined #openstack-operators | 07:48 | |
*** mpopow__ has quit IRC | 07:52 | |
*** esker has quit IRC | 07:52 | |
gfa | anyone knows how to improve the neutron's metadata cache? is possible to use memcached? instead of "memory" | 07:56 |
*** matrohon has quit IRC | 08:35 | |
*** miyagishi_t has quit IRC | 08:43 | |
*** Marga_ has joined #openstack-operators | 08:44 | |
*** Marga_ has quit IRC | 08:48 | |
*** Marga_ has joined #openstack-operators | 08:49 | |
*** derekh has joined #openstack-operators | 08:50 | |
*** markvoelker has joined #openstack-operators | 08:57 | |
*** matrohon has joined #openstack-operators | 09:01 | |
*** markvoelker has quit IRC | 09:02 | |
*** Marga_ has quit IRC | 09:05 | |
*** sanjayu has joined #openstack-operators | 09:17 | |
*** zhangjn has quit IRC | 09:17 | |
*** sanjayu_ has quit IRC | 09:21 | |
*** esker has joined #openstack-operators | 09:36 | |
*** esker has quit IRC | 09:41 | |
*** regXboi has joined #openstack-operators | 10:01 | |
*** gparaskevas___ has joined #openstack-operators | 10:09 | |
gparaskevas___ | hello | 10:09 |
gparaskevas___ | there | 10:09 |
gparaskevas___ | i am having a bit of an issue with instance launching | 10:10 |
gparaskevas___ | Juno on phisical machines/Ubuntu 14.04 | 10:10 |
gparaskevas___ | neutron/ml2/linux bridge | 10:10 |
gparaskevas___ | Single nic machines | 10:10 |
*** esker has joined #openstack-operators | 10:14 | |
*** esker has quit IRC | 10:18 | |
*** sanjayu_ has joined #openstack-operators | 10:20 | |
*** zhangjn has joined #openstack-operators | 10:23 | |
*** sanjayu has quit IRC | 10:24 | |
*** zhangjn has quit IRC | 10:26 | |
*** esker has joined #openstack-operators | 10:50 | |
*** vsedelnik has quit IRC | 11:00 | |
*** vsedelnik has joined #openstack-operators | 11:01 | |
*** purp_2 has joined #openstack-operators | 11:02 | |
*** purp has quit IRC | 11:02 | |
*** gparaskevas___ has quit IRC | 11:03 | |
*** esker has quit IRC | 11:26 | |
*** esker has joined #openstack-operators | 11:26 | |
*** matrohon has quit IRC | 11:29 | |
*** bapalm has quit IRC | 11:43 | |
*** bapalm has joined #openstack-operators | 11:49 | |
*** esker has quit IRC | 11:52 | |
*** matrohon has joined #openstack-operators | 12:02 | |
*** vsedelnik has quit IRC | 12:04 | |
*** jaypipes has joined #openstack-operators | 12:08 | |
*** delattec has quit IRC | 12:15 | |
*** cdelatte has quit IRC | 12:15 | |
*** vsedelnik has joined #openstack-operators | 12:20 | |
*** purp_2 is now known as purp | 12:27 | |
*** david-lyle has quit IRC | 12:27 | |
*** david-lyle has joined #openstack-operators | 12:28 | |
*** zhangjn has joined #openstack-operators | 12:47 | |
*** zhangjn has quit IRC | 12:54 | |
*** sanjayu has joined #openstack-operators | 12:55 | |
*** vsedelnik has quit IRC | 12:56 | |
*** vsedelnik has joined #openstack-operators | 12:58 | |
*** sanjayu_ has quit IRC | 12:58 | |
*** vsedelnik has quit IRC | 12:59 | |
*** vsedelnik has joined #openstack-operators | 12:59 | |
*** zhangjn has joined #openstack-operators | 13:04 | |
*** zhangjn has quit IRC | 13:04 | |
*** sanjayu_ has joined #openstack-operators | 13:04 | |
*** zhangjn has joined #openstack-operators | 13:05 | |
*** sanjayu has quit IRC | 13:07 | |
*** vsedelnik has quit IRC | 13:16 | |
*** vsedelnik has joined #openstack-operators | 13:17 | |
*** esker has joined #openstack-operators | 13:18 | |
*** kencjohn_ has joined #openstack-operators | 13:19 | |
*** esker has quit IRC | 13:20 | |
*** vsedelnik has quit IRC | 13:22 | |
*** vsedelnik has joined #openstack-operators | 13:23 | |
*** esker has joined #openstack-operators | 13:25 | |
*** esker has quit IRC | 13:26 | |
*** esker has joined #openstack-operators | 13:27 | |
*** zul has quit IRC | 13:29 | |
*** VW_ has joined #openstack-operators | 13:30 | |
*** zul has joined #openstack-operators | 13:32 | |
*** markvoelker has joined #openstack-operators | 14:01 | |
*** markvoelker has quit IRC | 14:06 | |
*** Piet has quit IRC | 14:13 | |
*** cdelatte has joined #openstack-operators | 14:25 | |
*** delattec has joined #openstack-operators | 14:25 | |
*** esker has quit IRC | 14:27 | |
*** sdake has quit IRC | 14:28 | |
*** sdake has joined #openstack-operators | 14:32 | |
*** kencjohn_ has quit IRC | 14:32 | |
*** mpopow__ has joined #openstack-operators | 14:34 | |
*** kencjohn_ has joined #openstack-operators | 14:38 | |
*** mpopow__ has quit IRC | 14:38 | |
*** esker has joined #openstack-operators | 14:39 | |
*** kencjoh__ has joined #openstack-operators | 14:41 | |
*** markvoelker has joined #openstack-operators | 14:42 | |
*** markvoelker has quit IRC | 14:42 | |
*** markvoelker has joined #openstack-operators | 14:43 | |
*** kencjohn_ has quit IRC | 14:43 | |
*** zhangjn has quit IRC | 14:45 | |
*** zhangjn has joined #openstack-operators | 14:45 | |
*** zhangjn has quit IRC | 14:45 | |
*** gtt116__ has joined #openstack-operators | 14:46 | |
*** zhangjn has joined #openstack-operators | 14:46 | |
*** zhangjn has quit IRC | 14:46 | |
*** cdelatte has quit IRC | 14:47 | |
*** delattec has quit IRC | 14:47 | |
*** cdelatte has joined #openstack-operators | 14:47 | |
*** delattec has joined #openstack-operators | 14:47 | |
*** zhangjn has joined #openstack-operators | 14:47 | |
*** zhangjn has quit IRC | 14:49 | |
*** gtt116_ has quit IRC | 14:49 | |
*** zhangjn has joined #openstack-operators | 14:50 | |
*** zhangjn has quit IRC | 14:50 | |
*** esker has quit IRC | 14:53 | |
*** klindgren has joined #openstack-operators | 14:57 | |
*** maishsk has quit IRC | 14:59 | |
*** krotscheck is now known as kro_afk | 14:59 | |
*** matrohon has quit IRC | 15:00 | |
*** bhunter71 has joined #openstack-operators | 15:02 | |
*** kro_afk is now known as krotscheck | 15:04 | |
*** SimonChung1 has quit IRC | 15:11 | |
*** harshs has joined #openstack-operators | 15:11 | |
*** priteau has joined #openstack-operators | 15:12 | |
*** gtt116_ has joined #openstack-operators | 15:16 | |
*** gtt116__ has quit IRC | 15:19 | |
*** emagana has joined #openstack-operators | 15:23 | |
*** britthouser has quit IRC | 15:34 | |
*** geoffarnold has joined #openstack-operators | 15:36 | |
*** geoffarnold has quit IRC | 15:38 | |
*** geoffarnold has joined #openstack-operators | 15:39 | |
*** vsedelnik has quit IRC | 15:49 | |
*** mpopow__ has joined #openstack-operators | 15:50 | |
*** vsedelnik has joined #openstack-operators | 15:52 | |
*** mpopow__ has quit IRC | 15:54 | |
*** regXboi has quit IRC | 15:55 | |
*** esker has joined #openstack-operators | 15:57 | |
*** gyee has joined #openstack-operators | 15:57 | |
*** esker has quit IRC | 15:59 | |
*** markvoelker has quit IRC | 16:03 | |
*** esker has joined #openstack-operators | 16:14 | |
*** SimonChung has joined #openstack-operators | 16:15 | |
*** devlaps has joined #openstack-operators | 16:16 | |
*** esker has quit IRC | 16:23 | |
*** harshs has quit IRC | 16:27 | |
klindgren | mfisch, *ugh* looks like we've had some neutron security group stuff running for almost 20 days looping trying to do the same work over and over and over again on some hosts | 16:28 |
*** Piet has joined #openstack-operators | 16:28 | |
mfisch | whats the path back to Juno ;) | 16:29 |
mfisch | a talk I'd ike to see in Tokyo | 16:29 |
*** mpopow__ has joined #openstack-operators | 16:32 | |
klindgren | mfisch, you use security groups no? | 16:43 |
mfisch | yes | 16:43 |
*** harshs has joined #openstack-operators | 16:43 | |
klindgren | can you check to see if you have "port_unbound(): net_uuid None not in local_vlan_map" repeating in your openvswitch-agent logs | 16:44 |
* jlk is hoping these are problems specific to OVS | 16:44 | |
klindgren | since 8/12 I have ~1.5 million entires for that in logstash | 16:44 |
klindgren | each server repeating that log message with a req-id thats the same - but seems to be unique per server | 16:45 |
mfisch | openvswitch-agent.log.1:2015-09-01 18:30:44.780 11624 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-110fb0b3-dbbd-41c8-a3e4-a6c21f0a225d ] port_unbound(): net_uuid None not in local_vlan_map | 16:45 |
mfisch | openvswitch-agent.log.1:2015-09-01 18:33:06.832 11624 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-110fb0b3-dbbd-41c8-a3e4-a6c21f0a225d ] port_unbound(): net_uuid None not in local_vlan_map | 16:45 |
mfisch | openvswitch-agent.log.1:2015-09-01 18:34:34.907 11624 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-110fb0b3-dbbd-41c8-a3e4-a6c21f0a225d ] port_unbound(): net_uuid None not in local_vlan_map | 16:45 |
klindgren | its an INFO level message btw | 16:45 |
*** carl_baldwin has joined #openstack-operators | 16:46 | |
klindgren | WE did our kilo upgrade July 31'st in this env | 16:46 |
klindgren | so wtf | 16:46 |
klindgren | our started ~12 days after the upgrade | 16:47 |
*** mpopow__ has quit IRC | 16:53 | |
*** derekh has quit IRC | 16:57 | |
klindgren | ugh - we had to nuke some logstash data due to the fact that neutron ipset looped every second dumping a full iptables-restore output at trace level. So its probably been doing that for longer than that. | 17:00 |
j^2 | I’ve started the process to make the OSOps group from github moved into Stackforge | 17:09 |
j^2 | https://review.openstack.org/#/c/219760/ | 17:09 |
j^2 | I’ve announced it on the mailing list too, i think we shouldn’t have any resistance making this happen | 17:10 |
*** kencjoh__ has quit IRC | 17:19 | |
*** VW_ has quit IRC | 17:22 | |
*** stanchan has quit IRC | 17:22 | |
*** VW_ has joined #openstack-operators | 17:22 | |
*** maishsk has joined #openstack-operators | 17:23 | |
*** VW_ has quit IRC | 17:27 | |
*** VW_ has joined #openstack-operators | 17:28 | |
*** vsedelnik has quit IRC | 17:32 | |
*** markvoelker has joined #openstack-operators | 17:35 | |
*** markvoelker has quit IRC | 17:35 | |
*** markvoelker has joined #openstack-operators | 17:36 | |
*** vsedelnik has joined #openstack-operators | 17:38 | |
*** vsedelnik has quit IRC | 17:38 | |
*** vsedelnik has joined #openstack-operators | 17:39 | |
*** emagana has quit IRC | 17:43 | |
*** snk has quit IRC | 17:45 | |
*** snk has joined #openstack-operators | 17:45 | |
*** snk has quit IRC | 17:45 | |
*** snk has joined #openstack-operators | 17:46 | |
*** rockyg has joined #openstack-operators | 17:50 | |
*** harshs has quit IRC | 17:58 | |
*** mpopow__ has joined #openstack-operators | 17:58 | |
*** maishsk has quit IRC | 17:59 | |
*** vsedelnik has quit IRC | 18:04 | |
*** maishsk has joined #openstack-operators | 18:05 | |
*** markvoelker has quit IRC | 18:05 | |
*** harshs has joined #openstack-operators | 18:10 | |
*** mpopow__ has quit IRC | 18:13 | |
*** sdake_ has joined #openstack-operators | 18:14 | |
*** csoukup has joined #openstack-operators | 18:16 | |
*** rockyg has quit IRC | 18:16 | |
*** kencjohn_ has joined #openstack-operators | 18:16 | |
*** sdake has quit IRC | 18:17 | |
*** maishsk has quit IRC | 18:17 | |
*** sdake has joined #openstack-operators | 18:21 | |
*** maishsk has joined #openstack-operators | 18:22 | |
*** sdake_ has quit IRC | 18:25 | |
*** esker has joined #openstack-operators | 18:34 | |
*** esker has quit IRC | 18:35 | |
*** esker has joined #openstack-operators | 18:35 | |
*** sdake_ has joined #openstack-operators | 18:41 | |
*** bvandenh has joined #openstack-operators | 18:43 | |
*** sdake has quit IRC | 18:45 | |
*** ybabenko has joined #openstack-operators | 18:47 | |
*** bvandenh has quit IRC | 18:51 | |
*** harlowja has quit IRC | 18:54 | |
j^2 | I’ve created a wiki page for this project: https://wiki.openstack.org/wiki/Osops | 18:55 |
*** harlowja has joined #openstack-operators | 18:58 | |
*** Rockyg has joined #openstack-operators | 18:58 | |
*** markvoelker has joined #openstack-operators | 19:00 | |
*** ybabenko has quit IRC | 19:02 | |
*** klindgren_ has joined #openstack-operators | 19:09 | |
*** matrohon has joined #openstack-operators | 19:10 | |
*** vsedelnik has joined #openstack-operators | 19:10 | |
*** klindgren has quit IRC | 19:12 | |
*** bvandenh has joined #openstack-operators | 19:28 | |
*** gyee has quit IRC | 19:44 | |
*** esker has quit IRC | 19:48 | |
*** vsedelnik has quit IRC | 19:51 | |
*** geoffarnold is now known as geoffarnoldX | 20:03 | |
*** jmckind has joined #openstack-operators | 20:04 | |
*** gyee has joined #openstack-operators | 20:06 | |
*** alop has joined #openstack-operators | 20:10 | |
*** ybabenko has joined #openstack-operators | 20:10 | |
*** sdake_ is now known as sdake | 20:15 | |
*** vsedelnik has joined #openstack-operators | 20:17 | |
*** geoffarnoldX is now known as geoffarnold | 20:19 | |
*** Marga_ has joined #openstack-operators | 20:20 | |
*** markvoelker has quit IRC | 20:21 | |
*** stanchan has joined #openstack-operators | 20:22 | |
*** stanchan has quit IRC | 20:27 | |
*** bhunter71 has quit IRC | 20:30 | |
*** vsedelnik has quit IRC | 20:31 | |
*** markvoelker has joined #openstack-operators | 20:37 | |
*** matrohon has quit IRC | 20:44 | |
*** Piet has quit IRC | 20:45 | |
*** harlowja has quit IRC | 20:45 | |
*** signed8bit has joined #openstack-operators | 20:47 | |
*** markvoelker has quit IRC | 20:48 | |
*** vsedelnik has joined #openstack-operators | 21:00 | |
*** Rockyg has quit IRC | 21:08 | |
*** Marga_ has quit IRC | 21:15 | |
*** Marga_ has joined #openstack-operators | 21:16 | |
*** vsedelnik has quit IRC | 21:21 | |
*** harlowja has joined #openstack-operators | 21:21 | |
*** SimonChung has quit IRC | 21:27 | |
*** derekh has joined #openstack-operators | 21:33 | |
*** priteau has quit IRC | 21:34 | |
*** priteau has joined #openstack-operators | 21:38 | |
*** carl_baldwin has quit IRC | 21:41 | |
*** priteau has quit IRC | 21:43 | |
*** priteau has joined #openstack-operators | 21:44 | |
*** kencjohn_ has quit IRC | 21:47 | |
*** SimonChung has joined #openstack-operators | 21:49 | |
*** jmckind has quit IRC | 21:54 | |
*** harshs has quit IRC | 22:04 | |
*** signed8bit has quit IRC | 22:08 | |
*** harshs has joined #openstack-operators | 22:09 | |
*** jaypipes has quit IRC | 22:10 | |
*** Rockyg has joined #openstack-operators | 22:14 | |
*** bvandenh has quit IRC | 22:16 | |
*** emagana has joined #openstack-operators | 22:26 | |
*** Marga_ has quit IRC | 22:27 | |
*** zhangjn has joined #openstack-operators | 22:32 | |
*** VW__ has joined #openstack-operators | 22:35 | |
*** zhangjn has quit IRC | 22:36 | |
*** VW_ has quit IRC | 22:39 | |
*** VW__ has quit IRC | 22:40 | |
*** csoukup has quit IRC | 22:47 | |
*** Rockyg has quit IRC | 22:49 | |
*** derekh has quit IRC | 22:50 | |
*** ybabenko has quit IRC | 22:55 | |
*** SimonChung has quit IRC | 23:09 | |
*** sdake has quit IRC | 23:15 | |
*** `mjr_ has quit IRC | 23:17 | |
*** Piet has joined #openstack-operators | 23:17 | |
*** sdake has joined #openstack-operators | 23:28 | |
*** SimonChung has joined #openstack-operators | 23:28 | |
*** alop has quit IRC | 23:30 | |
*** krotscheck is now known as kro_paternity | 23:34 | |
*** bhunter71 has joined #openstack-operators | 23:35 | |
*** harlowja has quit IRC | 23:42 | |
*** harlowja has joined #openstack-operators | 23:43 | |
*** markvoelker has joined #openstack-operators | 23:46 | |
*** harshs has quit IRC | 23:57 | |
*** markvoelker has quit IRC | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!