Wednesday, 2015-09-02

*** miyagishi_t has joined #openstack-operators00:01
*** SimonChung1 has quit IRC00:03
*** SimonChung has joined #openstack-operators00:04
*** SimonChung has quit IRC00:04
mfischklindgren_: I've been pretty disappoined in kilo so far, sounds like you're having issues too00:17
klindgren_mfisch, nothing terrible - yet00:18
klindgren_ipset race conditions seems to be the only thing so far00:18
mfischwe're having dhcp issues00:18
mfischabove and beyond what we presented in PA00:19
klindgren_hrm what specifically re: dhcp?00:19
mfischdhcp agents losing their brain and tenant VMs losing connectivity when the lease times out00:19
klindgren_to be honest -we use config-drive to statically IP that stuff...00:20
klindgren_so00:20
klindgren_dhcp might be broken - but we don't notice ^_^00:20
mfischgood choice00:21
klindgren_Curious - as after mfisch talked about it.... anyone else here have issues with dhcp_agent getting moved between nodes for apparently no good reason?00:24
klindgren_we've staitcally assigned dhcp_agents to specifical networks since havana - under kilo the assigments started randomly getting switched around00:25
mfischmy honest advice would be wait on kilo too00:25
klindgren_mfisch, want to do a ML post?  I can chime in on what we have seen - see if we can get more people on it.00:26
mfischI'd need to work with eric on it00:27
mfischhe did more digging00:27
mfischwe did file a bug00:27
klindgren_got the bug refrence?00:27
*** VW_ has joined #openstack-operators00:37
*** VW_ has quit IRC00:39
*** VW_ has joined #openstack-operators00:39
*** VW_ has quit IRC00:44
*** zhangjn has joined #openstack-operators00:44
*** mpopow_ has joined #openstack-operators01:25
*** mpopow has quit IRC01:29
*** SimonChung has joined #openstack-operators01:31
*** SimonChung1 has joined #openstack-operators01:33
*** SimonChung has quit IRC01:36
*** resker has joined #openstack-operators01:48
*** esker has quit IRC01:52
gfamfisch: our dhcp lease time is like a year long. we don't have problems with that. on havana we have issues when the dhcp-agent don't notify dnsmasq about the lease being released and dnsmasq refusing to give that ip again. but since icehouse is ok01:55
mfischI suggested that as a fix01:56
mfischbut it just masks stuff01:56
*** zhangjn_ has joined #openstack-operators01:57
*** zhangjn has quit IRC01:58
*** harshs has quit IRC02:08
*** kencjohn_ has joined #openstack-operators02:15
*** kencjohn_ has quit IRC02:15
*** kencjohn_ has joined #openstack-operators02:16
*** klindgren_ has quit IRC02:25
*** resker has quit IRC02:41
*** gfa is now known as gfa_02:46
*** hakimo_ has joined #openstack-operators02:52
*** hakimo has quit IRC02:54
*** gfa_ is now known as gfa02:59
*** kencjohn_ has quit IRC03:18
*** harshs has joined #openstack-operators03:27
*** esker has joined #openstack-operators03:35
*** esker has quit IRC03:40
*** stanchan has joined #openstack-operators03:41
*** maishsk has quit IRC03:47
*** zhangjn has joined #openstack-operators04:00
*** zhangjn_ has quit IRC04:02
*** csoukup has joined #openstack-operators04:04
*** csoukup has quit IRC04:05
*** bhunter71 has quit IRC04:11
*** esker has joined #openstack-operators04:30
*** esker has quit IRC04:34
*** gfa is now known as gfa_04:36
*** gfa_ is now known as gfa04:36
*** esker has joined #openstack-operators04:43
*** harshs has quit IRC04:45
*** esker has quit IRC04:49
*** harshs has joined #openstack-operators05:00
*** mpopow__ has joined #openstack-operators05:03
*** markvoelker has joined #openstack-operators05:06
*** mpopow_ has quit IRC05:07
*** markvoelker has quit IRC05:11
*** markvoelker has joined #openstack-operators05:12
*** harshs has quit IRC05:23
*** vsedelnik has joined #openstack-operators05:45
*** gfa is now known as gfa_05:47
*** gfa_ is now known as gfa05:57
*** priteau has joined #openstack-operators06:21
*** markvoelker has quit IRC06:23
*** sdake has joined #openstack-operators06:24
*** priteau has quit IRC06:26
*** mpopow__ has quit IRC06:47
*** maishsk has joined #openstack-operators07:09
*** vinsh has quit IRC07:34
*** vinsh has joined #openstack-operators07:35
*** esker has joined #openstack-operators07:48
*** mpopow__ has joined #openstack-operators07:48
*** mpopow__ has quit IRC07:52
*** esker has quit IRC07:52
gfaanyone knows how to improve the neutron's metadata cache? is possible to use memcached? instead of "memory"07:56
*** matrohon has quit IRC08:35
*** miyagishi_t has quit IRC08:43
*** Marga_ has joined #openstack-operators08:44
*** Marga_ has quit IRC08:48
*** Marga_ has joined #openstack-operators08:49
*** derekh has joined #openstack-operators08:50
*** markvoelker has joined #openstack-operators08:57
*** matrohon has joined #openstack-operators09:01
*** markvoelker has quit IRC09:02
*** Marga_ has quit IRC09:05
*** sanjayu has joined #openstack-operators09:17
*** zhangjn has quit IRC09:17
*** sanjayu_ has quit IRC09:21
*** esker has joined #openstack-operators09:36
*** esker has quit IRC09:41
*** regXboi has joined #openstack-operators10:01
*** gparaskevas___ has joined #openstack-operators10:09
gparaskevas___hello10:09
gparaskevas___there10:09
gparaskevas___i am having a bit of an issue with instance launching10:10
gparaskevas___Juno on phisical machines/Ubuntu 14.0410:10
gparaskevas___neutron/ml2/linux bridge10:10
gparaskevas___Single nic machines10:10
*** esker has joined #openstack-operators10:14
*** esker has quit IRC10:18
*** sanjayu_ has joined #openstack-operators10:20
*** zhangjn has joined #openstack-operators10:23
*** sanjayu has quit IRC10:24
*** zhangjn has quit IRC10:26
*** esker has joined #openstack-operators10:50
*** vsedelnik has quit IRC11:00
*** vsedelnik has joined #openstack-operators11:01
*** purp_2 has joined #openstack-operators11:02
*** purp has quit IRC11:02
*** gparaskevas___ has quit IRC11:03
*** esker has quit IRC11:26
*** esker has joined #openstack-operators11:26
*** matrohon has quit IRC11:29
*** bapalm has quit IRC11:43
*** bapalm has joined #openstack-operators11:49
*** esker has quit IRC11:52
*** matrohon has joined #openstack-operators12:02
*** vsedelnik has quit IRC12:04
*** jaypipes has joined #openstack-operators12:08
*** delattec has quit IRC12:15
*** cdelatte has quit IRC12:15
*** vsedelnik has joined #openstack-operators12:20
*** purp_2 is now known as purp12:27
*** david-lyle has quit IRC12:27
*** david-lyle has joined #openstack-operators12:28
*** zhangjn has joined #openstack-operators12:47
*** zhangjn has quit IRC12:54
*** sanjayu has joined #openstack-operators12:55
*** vsedelnik has quit IRC12:56
*** vsedelnik has joined #openstack-operators12:58
*** sanjayu_ has quit IRC12:58
*** vsedelnik has quit IRC12:59
*** vsedelnik has joined #openstack-operators12:59
*** zhangjn has joined #openstack-operators13:04
*** zhangjn has quit IRC13:04
*** sanjayu_ has joined #openstack-operators13:04
*** zhangjn has joined #openstack-operators13:05
*** sanjayu has quit IRC13:07
*** vsedelnik has quit IRC13:16
*** vsedelnik has joined #openstack-operators13:17
*** esker has joined #openstack-operators13:18
*** kencjohn_ has joined #openstack-operators13:19
*** esker has quit IRC13:20
*** vsedelnik has quit IRC13:22
*** vsedelnik has joined #openstack-operators13:23
*** esker has joined #openstack-operators13:25
*** esker has quit IRC13:26
*** esker has joined #openstack-operators13:27
*** zul has quit IRC13:29
*** VW_ has joined #openstack-operators13:30
*** zul has joined #openstack-operators13:32
*** markvoelker has joined #openstack-operators14:01
*** markvoelker has quit IRC14:06
*** Piet has quit IRC14:13
*** cdelatte has joined #openstack-operators14:25
*** delattec has joined #openstack-operators14:25
*** esker has quit IRC14:27
*** sdake has quit IRC14:28
*** sdake has joined #openstack-operators14:32
*** kencjohn_ has quit IRC14:32
*** mpopow__ has joined #openstack-operators14:34
*** kencjohn_ has joined #openstack-operators14:38
*** mpopow__ has quit IRC14:38
*** esker has joined #openstack-operators14:39
*** kencjoh__ has joined #openstack-operators14:41
*** markvoelker has joined #openstack-operators14:42
*** markvoelker has quit IRC14:42
*** markvoelker has joined #openstack-operators14:43
*** kencjohn_ has quit IRC14:43
*** zhangjn has quit IRC14:45
*** zhangjn has joined #openstack-operators14:45
*** zhangjn has quit IRC14:45
*** gtt116__ has joined #openstack-operators14:46
*** zhangjn has joined #openstack-operators14:46
*** zhangjn has quit IRC14:46
*** cdelatte has quit IRC14:47
*** delattec has quit IRC14:47
*** cdelatte has joined #openstack-operators14:47
*** delattec has joined #openstack-operators14:47
*** zhangjn has joined #openstack-operators14:47
*** zhangjn has quit IRC14:49
*** gtt116_ has quit IRC14:49
*** zhangjn has joined #openstack-operators14:50
*** zhangjn has quit IRC14:50
*** esker has quit IRC14:53
*** klindgren has joined #openstack-operators14:57
*** maishsk has quit IRC14:59
*** krotscheck is now known as kro_afk14:59
*** matrohon has quit IRC15:00
*** bhunter71 has joined #openstack-operators15:02
*** kro_afk is now known as krotscheck15:04
*** SimonChung1 has quit IRC15:11
*** harshs has joined #openstack-operators15:11
*** priteau has joined #openstack-operators15:12
*** gtt116_ has joined #openstack-operators15:16
*** gtt116__ has quit IRC15:19
*** emagana has joined #openstack-operators15:23
*** britthouser has quit IRC15:34
*** geoffarnold has joined #openstack-operators15:36
*** geoffarnold has quit IRC15:38
*** geoffarnold has joined #openstack-operators15:39
*** vsedelnik has quit IRC15:49
*** mpopow__ has joined #openstack-operators15:50
*** vsedelnik has joined #openstack-operators15:52
*** mpopow__ has quit IRC15:54
*** regXboi has quit IRC15:55
*** esker has joined #openstack-operators15:57
*** gyee has joined #openstack-operators15:57
*** esker has quit IRC15:59
*** markvoelker has quit IRC16:03
*** esker has joined #openstack-operators16:14
*** SimonChung has joined #openstack-operators16:15
*** devlaps has joined #openstack-operators16:16
*** esker has quit IRC16:23
*** harshs has quit IRC16:27
klindgrenmfisch, *ugh* looks like we've had some neutron security group stuff running for almost 20 days looping trying to do the same work over and over and over again on some hosts16:28
*** Piet has joined #openstack-operators16:28
mfischwhats the path back to Juno ;)16:29
mfischa talk I'd ike to see in Tokyo16:29
*** mpopow__ has joined #openstack-operators16:32
klindgrenmfisch, you use security groups no?16:43
mfischyes16:43
*** harshs has joined #openstack-operators16:43
klindgrencan you check to see if you have "port_unbound(): net_uuid None not in local_vlan_map" repeating in your openvswitch-agent logs16:44
* jlk is hoping these are problems specific to OVS16:44
klindgrensince 8/12 I have ~1.5 million entires for that in logstash16:44
klindgreneach server repeating that log message with a req-id thats the same - but seems to be unique per server16:45
mfischopenvswitch-agent.log.1:2015-09-01 18:30:44.780 11624 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-110fb0b3-dbbd-41c8-a3e4-a6c21f0a225d ] port_unbound(): net_uuid None not in local_vlan_map16:45
mfischopenvswitch-agent.log.1:2015-09-01 18:33:06.832 11624 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-110fb0b3-dbbd-41c8-a3e4-a6c21f0a225d ] port_unbound(): net_uuid None not in local_vlan_map16:45
mfischopenvswitch-agent.log.1:2015-09-01 18:34:34.907 11624 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-110fb0b3-dbbd-41c8-a3e4-a6c21f0a225d ] port_unbound(): net_uuid None not in local_vlan_map16:45
klindgrenits an INFO level message btw16:45
*** carl_baldwin has joined #openstack-operators16:46
klindgrenWE did our kilo upgrade July 31'st in this env16:46
klindgrenso wtf16:46
klindgrenour started ~12 days after the upgrade16:47
*** mpopow__ has quit IRC16:53
*** derekh has quit IRC16:57
klindgrenugh - we had to nuke some logstash data due to the fact that neutron ipset looped every second dumping a full iptables-restore output at trace level.  So its probably been doing that for longer than that.17:00
j^2I’ve started the process to make the OSOps group from github moved into Stackforge17:09
j^2https://review.openstack.org/#/c/219760/17:09
j^2I’ve announced it on the mailing list too, i think we shouldn’t have any resistance making this happen17:10
*** kencjoh__ has quit IRC17:19
*** VW_ has quit IRC17:22
*** stanchan has quit IRC17:22
*** VW_ has joined #openstack-operators17:22
*** maishsk has joined #openstack-operators17:23
*** VW_ has quit IRC17:27
*** VW_ has joined #openstack-operators17:28
*** vsedelnik has quit IRC17:32
*** markvoelker has joined #openstack-operators17:35
*** markvoelker has quit IRC17:35
*** markvoelker has joined #openstack-operators17:36
*** vsedelnik has joined #openstack-operators17:38
*** vsedelnik has quit IRC17:38
*** vsedelnik has joined #openstack-operators17:39
*** emagana has quit IRC17:43
*** snk has quit IRC17:45
*** snk has joined #openstack-operators17:45
*** snk has quit IRC17:45
*** snk has joined #openstack-operators17:46
*** rockyg has joined #openstack-operators17:50
*** harshs has quit IRC17:58
*** mpopow__ has joined #openstack-operators17:58
*** maishsk has quit IRC17:59
*** vsedelnik has quit IRC18:04
*** maishsk has joined #openstack-operators18:05
*** markvoelker has quit IRC18:05
*** harshs has joined #openstack-operators18:10
*** mpopow__ has quit IRC18:13
*** sdake_ has joined #openstack-operators18:14
*** csoukup has joined #openstack-operators18:16
*** rockyg has quit IRC18:16
*** kencjohn_ has joined #openstack-operators18:16
*** sdake has quit IRC18:17
*** maishsk has quit IRC18:17
*** sdake has joined #openstack-operators18:21
*** maishsk has joined #openstack-operators18:22
*** sdake_ has quit IRC18:25
*** esker has joined #openstack-operators18:34
*** esker has quit IRC18:35
*** esker has joined #openstack-operators18:35
*** sdake_ has joined #openstack-operators18:41
*** bvandenh has joined #openstack-operators18:43
*** sdake has quit IRC18:45
*** ybabenko has joined #openstack-operators18:47
*** bvandenh has quit IRC18:51
*** harlowja has quit IRC18:54
j^2I’ve created a wiki page for this project: https://wiki.openstack.org/wiki/Osops18:55
*** harlowja has joined #openstack-operators18:58
*** Rockyg has joined #openstack-operators18:58
*** markvoelker has joined #openstack-operators19:00
*** ybabenko has quit IRC19:02
*** klindgren_ has joined #openstack-operators19:09
*** matrohon has joined #openstack-operators19:10
*** vsedelnik has joined #openstack-operators19:10
*** klindgren has quit IRC19:12
*** bvandenh has joined #openstack-operators19:28
*** gyee has quit IRC19:44
*** esker has quit IRC19:48
*** vsedelnik has quit IRC19:51
*** geoffarnold is now known as geoffarnoldX20:03
*** jmckind has joined #openstack-operators20:04
*** gyee has joined #openstack-operators20:06
*** alop has joined #openstack-operators20:10
*** ybabenko has joined #openstack-operators20:10
*** sdake_ is now known as sdake20:15
*** vsedelnik has joined #openstack-operators20:17
*** geoffarnoldX is now known as geoffarnold20:19
*** Marga_ has joined #openstack-operators20:20
*** markvoelker has quit IRC20:21
*** stanchan has joined #openstack-operators20:22
*** stanchan has quit IRC20:27
*** bhunter71 has quit IRC20:30
*** vsedelnik has quit IRC20:31
*** markvoelker has joined #openstack-operators20:37
*** matrohon has quit IRC20:44
*** Piet has quit IRC20:45
*** harlowja has quit IRC20:45
*** signed8bit has joined #openstack-operators20:47
*** markvoelker has quit IRC20:48
*** vsedelnik has joined #openstack-operators21:00
*** Rockyg has quit IRC21:08
*** Marga_ has quit IRC21:15
*** Marga_ has joined #openstack-operators21:16
*** vsedelnik has quit IRC21:21
*** harlowja has joined #openstack-operators21:21
*** SimonChung has quit IRC21:27
*** derekh has joined #openstack-operators21:33
*** priteau has quit IRC21:34
*** priteau has joined #openstack-operators21:38
*** carl_baldwin has quit IRC21:41
*** priteau has quit IRC21:43
*** priteau has joined #openstack-operators21:44
*** kencjohn_ has quit IRC21:47
*** SimonChung has joined #openstack-operators21:49
*** jmckind has quit IRC21:54
*** harshs has quit IRC22:04
*** signed8bit has quit IRC22:08
*** harshs has joined #openstack-operators22:09
*** jaypipes has quit IRC22:10
*** Rockyg has joined #openstack-operators22:14
*** bvandenh has quit IRC22:16
*** emagana has joined #openstack-operators22:26
*** Marga_ has quit IRC22:27
*** zhangjn has joined #openstack-operators22:32
*** VW__ has joined #openstack-operators22:35
*** zhangjn has quit IRC22:36
*** VW_ has quit IRC22:39
*** VW__ has quit IRC22:40
*** csoukup has quit IRC22:47
*** Rockyg has quit IRC22:49
*** derekh has quit IRC22:50
*** ybabenko has quit IRC22:55
*** SimonChung has quit IRC23:09
*** sdake has quit IRC23:15
*** `mjr_ has quit IRC23:17
*** Piet has joined #openstack-operators23:17
*** sdake has joined #openstack-operators23:28
*** SimonChung has joined #openstack-operators23:28
*** alop has quit IRC23:30
*** krotscheck is now known as kro_paternity23:34
*** bhunter71 has joined #openstack-operators23:35
*** harlowja has quit IRC23:42
*** harlowja has joined #openstack-operators23:43
*** markvoelker has joined #openstack-operators23:46
*** harshs has quit IRC23:57
*** markvoelker has quit IRC23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!