| *** sdake has joined #openstack-operators | 00:19 | |
| *** geoffarnold is now known as geoffarnoldX | 00:27 | |
| *** esker has quit IRC | 00:42 | |
| *** esker has joined #openstack-operators | 00:50 | |
| *** kencjohn_ has quit IRC | 00:57 | |
| *** Marga_ has joined #openstack-operators | 01:19 | |
| *** zhangjn has joined #openstack-operators | 01:19 | |
| *** Marga_ has quit IRC | 01:20 | |
| *** Marga_ has joined #openstack-operators | 01:21 | |
| *** esker has quit IRC | 01:32 | |
| *** esker has joined #openstack-operators | 01:33 | |
| *** Marga_ has quit IRC | 01:35 | |
| egon | that max_count parameter is for booting multuple instances in a single request, not for quota or rate-limiting. | 01:45 |
|---|---|---|
| *** sorrison has quit IRC | 01:47 | |
| *** sorrison has joined #openstack-operators | 01:50 | |
| *** david-ly_ has joined #openstack-operators | 01:51 | |
| *** david-lyle has quit IRC | 01:54 | |
| *** esker has quit IRC | 01:57 | |
| *** prometheanfire has left #openstack-operators | 01:58 | |
| *** esker has joined #openstack-operators | 02:06 | |
| *** esker has quit IRC | 02:31 | |
| *** esker has joined #openstack-operators | 02:38 | |
| *** esker has quit IRC | 03:05 | |
| *** esker has joined #openstack-operators | 03:10 | |
| *** VW_ has joined #openstack-operators | 03:20 | |
| *** VW_ has quit IRC | 03:22 | |
| *** VW_ has joined #openstack-operators | 03:23 | |
| *** esker has quit IRC | 03:26 | |
| *** VW_ has quit IRC | 03:28 | |
| *** csoukup has joined #openstack-operators | 03:34 | |
| *** esker has joined #openstack-operators | 03:35 | |
| *** klindgren has joined #openstack-operators | 03:37 | |
| *** zhangjn_ has joined #openstack-operators | 03:52 | |
| *** esker has quit IRC | 03:54 | |
| *** zhangjn has quit IRC | 03:56 | |
| *** esker has joined #openstack-operators | 04:01 | |
| *** vsedelnik has joined #openstack-operators | 04:22 | |
| *** resker has joined #openstack-operators | 04:23 | |
| *** esker has quit IRC | 04:24 | |
| *** vsedelnik has quit IRC | 04:26 | |
| *** zhangjn_ has quit IRC | 04:27 | |
| *** vsedelnik has joined #openstack-operators | 04:34 | |
| *** resker has quit IRC | 04:44 | |
| *** esker has joined #openstack-operators | 04:48 | |
| *** csoukup has quit IRC | 04:59 | |
| *** vsedelnik has quit IRC | 05:08 | |
| *** csoukup has joined #openstack-operators | 05:15 | |
| *** esker has quit IRC | 05:16 | |
| *** csoukup has quit IRC | 05:19 | |
| *** esker has joined #openstack-operators | 05:22 | |
| *** esker has quit IRC | 05:45 | |
| *** esker has joined #openstack-operators | 05:51 | |
| *** vsedelnik has joined #openstack-operators | 06:08 | |
| *** matrohon has quit IRC | 06:14 | |
| *** vsedelni_ has joined #openstack-operators | 06:22 | |
| *** esker has quit IRC | 06:25 | |
| *** vsedelnik has quit IRC | 06:26 | |
| *** vsedelnik has joined #openstack-operators | 06:27 | |
| *** esker has joined #openstack-operators | 06:27 | |
| *** vsedelni_ has quit IRC | 06:29 | |
| *** esker has quit IRC | 06:45 | |
| *** esker has joined #openstack-operators | 06:50 | |
| *** fifieldt has joined #openstack-operators | 07:13 | |
| *** esker has quit IRC | 07:17 | |
| *** zhangjn has joined #openstack-operators | 07:19 | |
| *** esker has joined #openstack-operators | 07:23 | |
| *** zhangjn_ has joined #openstack-operators | 07:30 | |
| *** Guest32267 is now known as d0ugal | 07:32 | |
| *** d0ugal has joined #openstack-operators | 07:32 | |
| *** zhangjn has quit IRC | 07:34 | |
| *** zhangjn_ has quit IRC | 07:35 | |
| *** esker has quit IRC | 07:52 | |
| *** chlong has quit IRC | 07:56 | |
| *** esker has joined #openstack-operators | 07:59 | |
| *** matrohon has joined #openstack-operators | 08:09 | |
| *** vsedelni_ has joined #openstack-operators | 08:15 | |
| *** vsedelnik has quit IRC | 08:18 | |
| *** esker has quit IRC | 08:24 | |
| *** esker has joined #openstack-operators | 08:29 | |
| *** derekh has joined #openstack-operators | 08:46 | |
| *** vsedelnik has joined #openstack-operators | 08:49 | |
| *** vsedelni_ has quit IRC | 08:52 | |
| *** esker has quit IRC | 08:54 | |
| *** esker has joined #openstack-operators | 09:05 | |
| *** esker has quit IRC | 09:25 | |
| *** esker has joined #openstack-operators | 09:33 | |
| *** vsedelnik has quit IRC | 09:36 | |
| *** vsedelnik has joined #openstack-operators | 09:37 | |
| *** miyagishi_t has quit IRC | 09:39 | |
| *** Marga_ has joined #openstack-operators | 09:41 | |
| *** sdake has quit IRC | 09:42 | |
| *** matrohon has quit IRC | 09:51 | |
| *** esker has quit IRC | 10:05 | |
| *** esker has joined #openstack-operators | 10:11 | |
| *** priteau has joined #openstack-operators | 10:14 | |
| *** vsedelnik has quit IRC | 10:22 | |
| *** vsedelnik has joined #openstack-operators | 10:29 | |
| *** esker has quit IRC | 10:33 | |
| *** esker has joined #openstack-operators | 10:39 | |
| *** Marga_ has quit IRC | 10:46 | |
| *** vsedelnik has quit IRC | 10:51 | |
| *** Marga_ has joined #openstack-operators | 10:58 | |
| *** vsedelnik has joined #openstack-operators | 10:59 | |
| *** esker has quit IRC | 11:10 | |
| *** esker has joined #openstack-operators | 11:18 | |
| *** vsedelnik has quit IRC | 11:19 | |
| *** Marga_ has quit IRC | 11:20 | |
| *** markd_ has quit IRC | 11:30 | |
| *** esker has quit IRC | 11:36 | |
| *** vsedelnik has joined #openstack-operators | 11:39 | |
| *** esker has joined #openstack-operators | 11:45 | |
| *** esker has quit IRC | 12:07 | |
| *** zhangjn has joined #openstack-operators | 12:15 | |
| *** zhangjn has quit IRC | 12:17 | |
| *** esker has joined #openstack-operators | 12:17 | |
| *** markvoelker has joined #openstack-operators | 12:19 | |
| *** markvoelker has quit IRC | 12:20 | |
| *** markvoelker has joined #openstack-operators | 12:20 | |
| *** Marga_ has joined #openstack-operators | 12:21 | |
| *** chlong has joined #openstack-operators | 12:22 | |
| *** Marga_ has quit IRC | 12:26 | |
| *** esker has quit IRC | 12:38 | |
| *** vsedelnik has quit IRC | 12:43 | |
| *** esker has joined #openstack-operators | 12:44 | |
| *** vsedelnik has joined #openstack-operators | 12:44 | |
| *** matrohon has joined #openstack-operators | 12:46 | |
| *** mdavidson has joined #openstack-operators | 12:48 | |
| *** esker has quit IRC | 12:52 | |
| *** esker has joined #openstack-operators | 12:52 | |
| *** esker has quit IRC | 12:54 | |
| *** esker has joined #openstack-operators | 12:56 | |
| *** esker has quit IRC | 12:56 | |
| *** zhangjn has joined #openstack-operators | 13:01 | |
| *** Piet has quit IRC | 13:07 | |
| *** geoffarnoldX is now known as geoffarnold | 13:11 | |
| *** pilgrimstack1 has left #openstack-operators | 13:18 | |
| *** zhangjn has quit IRC | 13:20 | |
| *** Marga_ has joined #openstack-operators | 13:20 | |
| *** geoffarnold has quit IRC | 13:22 | |
| *** Marga_ has quit IRC | 13:24 | |
| *** Marga_ has joined #openstack-operators | 13:37 | |
| *** Marga_ has quit IRC | 13:37 | |
| *** Marga_ has joined #openstack-operators | 13:37 | |
| *** VW_ has joined #openstack-operators | 13:46 | |
| *** emagana has joined #openstack-operators | 14:00 | |
| *** Marga_ has quit IRC | 14:03 | |
| *** dminer has joined #openstack-operators | 14:04 | |
| *** vsedelnik has quit IRC | 14:12 | |
| *** Marga_ has joined #openstack-operators | 14:13 | |
| *** vsedelnik has joined #openstack-operators | 14:17 | |
| *** Marga_ has quit IRC | 14:17 | |
| *** carl_baldwin has joined #openstack-operators | 14:17 | |
| *** csoukup has joined #openstack-operators | 14:21 | |
| *** zhangjn has joined #openstack-operators | 14:26 | |
| *** zhangjn has quit IRC | 14:26 | |
| *** zhangjn has joined #openstack-operators | 14:37 | |
| *** Marga_ has joined #openstack-operators | 14:38 | |
| *** Marga_ has quit IRC | 14:38 | |
| *** Marga_ has joined #openstack-operators | 14:38 | |
| *** zhangjn has quit IRC | 14:39 | |
| *** Marga_ has quit IRC | 14:39 | |
| *** Marga_ has joined #openstack-operators | 14:42 | |
| klindgren | @jlk re: the ipset delete issue, sorrison is able to reproduce the issue on linuxbridge as well. Seems fairly simple to repo. Create a security group that refrences members of a porject and spinup/remove 80 vm's at a time. Eventually you will get ipsets to start churning. | 14:44 |
| jlk | awesome :/ | 14:44 |
| klindgren | For us we don't have projects with nearly that many vm's (afaik) however we still run into that bug on occasion. | 14:45 |
| jlk | we might run into it with a rally run | 14:45 |
| klindgren | yea - I wish that on gate we left instances stood up longer and ran a longer suite of tests. Like a few rally tests and the like | 14:47 |
| klindgren | here is to hoping that the venture between intel and rackspace turns up some more of these issues | 14:48 |
| klindgren | with the few thousand compute node clusters | 14:48 |
| *** laron has joined #openstack-operators | 14:49 | |
| *** david-ly_ is now known as david-lyle | 14:53 | |
| *** geoffarnold has joined #openstack-operators | 14:55 | |
| *** mdorman has joined #openstack-operators | 14:59 | |
| *** fifieldt has quit IRC | 15:00 | |
| *** laron_ has joined #openstack-operators | 15:02 | |
| *** geoffarnold is now known as geoffarnoldX | 15:04 | |
| *** geoffarnoldX is now known as geoffarnold | 15:04 | |
| *** stanchan has quit IRC | 15:04 | |
| *** laron has quit IRC | 15:06 | |
| *** Piet has joined #openstack-operators | 15:10 | |
| *** carl_baldwin has quit IRC | 15:11 | |
| *** geoffarnold is now known as geoffarnoldX | 15:12 | |
| *** geoffarnoldX is now known as geoffarnold | 15:12 | |
| *** vsedelnik has quit IRC | 15:12 | |
| *** vsedelnik has joined #openstack-operators | 15:13 | |
| *** kencjohn_ has joined #openstack-operators | 15:16 | |
| *** regXboi has joined #openstack-operators | 15:26 | |
| *** bradjones has joined #openstack-operators | 15:28 | |
| *** bradjones has quit IRC | 15:28 | |
| *** bradjones has joined #openstack-operators | 15:28 | |
| *** laron_ has quit IRC | 15:33 | |
| *** carl_baldwin has joined #openstack-operators | 15:37 | |
| *** devlaps has joined #openstack-operators | 15:43 | |
| *** laron has joined #openstack-operators | 15:48 | |
| *** alop has joined #openstack-operators | 15:48 | |
| *** jproulx has joined #openstack-operators | 15:50 | |
| jproulx | since kilo upgrade nova-metadata service seems to be unhappy with '/latest/' urls which we don't really use but is making puppet facts unhappy...is this gone or did I miss a config option somewhere? | 15:51 |
| *** matrohon has quit IRC | 15:52 | |
| jproulx | never mind it's apparently only on "some' nodes, clearly it's my doing somehow... | 15:53 |
| *** laron has quit IRC | 15:55 | |
| *** Marga_ has quit IRC | 16:00 | |
| *** gyee has joined #openstack-operators | 16:03 | |
| *** sdake_ has joined #openstack-operators | 16:06 | |
| *** Guest100 has joined #openstack-operators | 16:09 | |
| *** vsedelnik has quit IRC | 16:15 | |
| *** esker has joined #openstack-operators | 16:16 | |
| *** vsedelnik has joined #openstack-operators | 16:22 | |
| *** Marga_ has joined #openstack-operators | 16:23 | |
| *** SimonChung has joined #openstack-operators | 16:25 | |
| *** Guest100 has quit IRC | 16:38 | |
| *** zhangjn has joined #openstack-operators | 16:38 | |
| *** Guest60469 is now known as mgagne | 16:40 | |
| *** mgagne has joined #openstack-operators | 16:40 | |
| *** kencjohn_ has quit IRC | 16:41 | |
| *** derekh has quit IRC | 16:42 | |
| *** kencjohn_ has joined #openstack-operators | 16:48 | |
| *** VW_ has quit IRC | 16:59 | |
| *** VW_ has joined #openstack-operators | 17:00 | |
| *** SimonChung1 has joined #openstack-operators | 17:00 | |
| *** VW_ has quit IRC | 17:00 | |
| *** SimonChung has quit IRC | 17:00 | |
| *** VW_ has joined #openstack-operators | 17:00 | |
| *** SimonChung has joined #openstack-operators | 17:02 | |
| *** SimonChung1 has quit IRC | 17:02 | |
| *** harlowja has joined #openstack-operators | 17:03 | |
| *** SimonChung1 has joined #openstack-operators | 17:06 | |
| *** SimonChung has quit IRC | 17:06 | |
| *** Marga_ has quit IRC | 17:07 | |
| *** Marga_ has joined #openstack-operators | 17:07 | |
| *** Marga_ has quit IRC | 17:08 | |
| *** Marga_ has joined #openstack-operators | 17:09 | |
| *** Marga_ has quit IRC | 17:11 | |
| *** Marga_ has joined #openstack-operators | 17:15 | |
| *** sdake_ has quit IRC | 17:19 | |
| *** laron has joined #openstack-operators | 17:20 | |
| *** zhangjn has quit IRC | 17:21 | |
| *** laron_ has joined #openstack-operators | 17:23 | |
| *** laron_ has joined #openstack-operators | 17:24 | |
| *** laron has quit IRC | 17:24 | |
| *** sdake has joined #openstack-operators | 17:44 | |
| *** laron_ has quit IRC | 17:49 | |
| *** priteau has quit IRC | 17:49 | |
| *** laron has joined #openstack-operators | 18:06 | |
| *** SimonChung1 has quit IRC | 18:10 | |
| *** SimonChung has joined #openstack-operators | 18:10 | |
| *** SimonChung has quit IRC | 18:10 | |
| *** SimonChung1 has joined #openstack-operators | 18:10 | |
| *** carl_baldwin has quit IRC | 18:12 | |
| mgagne | klindgren: we did encounter this issue (ipset + lot of vms) | 18:18 |
| mgagne | klindgren: one idea I got was to optimize this so there is a common chain (with all VMs IPs) so all individual chains can refer to so you don't end up with a huge number of rules. | 18:19 |
| klindgren | mgagne, Ican you update/add yourself to the bug? | 18:20 |
| mgagne | klindgren: was reading backlog and didn't see bug #. what is it? | 18:20 |
| klindgren | on sec | 18:21 |
| klindgren | https://bugs.launchpad.net/neutron/+bug/1491131 | 18:21 |
| openstack | Launchpad bug 1491131 in neutron kilo "Ipset race condition" [High,New] | 18:21 |
| mgagne | thanks | 18:21 |
| mgagne | will see if it's the same issue or another | 18:21 |
| klindgren | on purposed fix is to create an ipset for the members of a project | 18:21 |
| mgagne | +1 | 18:21 |
| klindgren | and then only update the ipset when adding/removing a member | 18:22 |
| klindgren | however- I would still like to see some sanity put into place around the ipset operation | 18:22 |
| klindgren | infact in general churning forever trying to do some thing - is badness | 18:22 |
| mgagne | yep | 18:22 |
| mgagne | oh, we have also see SG with duplicated rules | 18:22 |
| mgagne | especially the one where another SG is allowed | 18:23 |
| mgagne | imagine the madness | 18:23 |
| klindgren | yea - the security group stuff has me kinda concerned | 18:23 |
| klindgren | i've heard about scalabiltiy issues with it - but no one really quantified that | 18:24 |
| klindgren | though really - duplicated rules within reason are not really an issue | 18:24 |
| mgagne | well, we need operators with real use cases otherwise devstack will tell you it's fine =) | 18:24 |
| klindgren | they jsut take up memory | 18:24 |
| klindgren | so we have 3 people urnning neutron in prod saying - hey ipsets are blowing up - awesome | 18:25 |
| mgagne | klindgren: well, allowing twice a SG with 100x VMs, it adds up quickly since each VM/tap will have the same set of rules (duplicated) | 18:25 |
| *** sdake_ has joined #openstack-operators | 18:26 | |
| klindgren | ohhhhh I see what you are saying | 18:26 |
| *** laron has quit IRC | 18:26 | |
| mgagne | 100 VM with 2 nics: 100 rules per tap per VM * 100. 20k rules, yea | 18:26 |
| klindgren | and how much of that is getting stuffed into an ipset? | 18:27 |
| klindgren | because thats - terrible | 18:27 |
| mgagne | yep, it's terrible | 18:27 |
| * klindgren hopes that they are looking to going towards the revamped netfilter stuff | 18:28 | |
| mgagne | I got a hint about the situation when I got a binding_failed with "Failed to apply iptables rules on line 24503". I was like: wut? 24k?! | 18:28 |
| *** carl_baldwin has joined #openstack-operators | 18:28 | |
| klindgren | ipset are nice improvement - but the new netfilter stuff is much much improved around being able to do a nice filtering level in a single rule | 18:29 |
| *** sdake has quit IRC | 18:29 | |
| *** laron has joined #openstack-operators | 18:30 | |
| *** laron has quit IRC | 18:30 | |
| *** laron has joined #openstack-operators | 18:30 | |
| mgagne | are you running kilo? | 18:32 |
| klindgren | indeed | 18:33 |
| klindgren | we have been running ipsets since juno though | 18:33 |
| klindgren | only started having issues in kilo | 18:33 |
| mgagne | hmmm... | 18:33 |
| mgagne | now you got me concerned | 18:33 |
| *** vsedelnik has quit IRC | 18:36 | |
| *** sdake_ is now known as sdake | 18:40 | |
| *** SimonChung has joined #openstack-operators | 18:40 | |
| *** SimonChung1 has quit IRC | 18:40 | |
| *** SimonChung has quit IRC | 18:41 | |
| *** SimonChung has joined #openstack-operators | 18:41 | |
| *** laron has quit IRC | 18:41 | |
| *** laron has joined #openstack-operators | 18:43 | |
| klindgren | +1 on concerned :-) | 18:44 |
| klindgren | we are ML2 + OVS, sorrinson is ML2 + Linuxbridge | 18:44 |
| klindgren | both appear to be equally impacted | 18:44 |
| *** sdake_ has joined #openstack-operators | 18:47 | |
| *** sdake has quit IRC | 18:49 | |
| *** derekh has joined #openstack-operators | 19:00 | |
| *** Marga_ has quit IRC | 19:06 | |
| jlk | Have y'all seen https://review.openstack.org/#/c/181393/ ? Standardization of the catalog entries. | 19:14 |
| *** SimonChung1 has joined #openstack-operators | 19:28 | |
| *** SimonChung has quit IRC | 19:28 | |
| *** vsedelnik has joined #openstack-operators | 19:45 | |
| *** maishsk has joined #openstack-operators | 19:48 | |
| *** sdake_ is now known as sdake | 19:49 | |
| *** jmckind has joined #openstack-operators | 19:57 | |
| *** derekh is now known as derekh_afk | 19:58 | |
| *** geoffarnold has quit IRC | 20:04 | |
| *** kencjohn_ has quit IRC | 20:34 | |
| *** tiny-hands has joined #openstack-operators | 20:35 | |
| *** Marga_ has joined #openstack-operators | 20:46 | |
| *** derekh_afk has quit IRC | 20:48 | |
| *** VW__ has joined #openstack-operators | 20:54 | |
| *** cdelatte has quit IRC | 20:56 | |
| *** VW_ has quit IRC | 20:57 | |
| *** geoffarnold has joined #openstack-operators | 20:58 | |
| *** SimonChung1 has quit IRC | 21:00 | |
| *** SimonChung has joined #openstack-operators | 21:00 | |
| *** kencjohn_ has joined #openstack-operators | 21:03 | |
| *** harshs has joined #openstack-operators | 21:04 | |
| *** harshs has left #openstack-operators | 21:04 | |
| *** harshs has joined #openstack-operators | 21:07 | |
| *** maishsk has quit IRC | 21:10 | |
| *** maishsk has joined #openstack-operators | 21:15 | |
| *** sdake_ has joined #openstack-operators | 21:16 | |
| *** sdake has quit IRC | 21:20 | |
| *** tiny-hands has quit IRC | 21:31 | |
| *** kencjohn_ has quit IRC | 21:35 | |
| *** devlaps has quit IRC | 21:38 | |
| *** priteau has joined #openstack-operators | 21:49 | |
| *** priteau has quit IRC | 21:54 | |
| *** sdake_ is now known as sdake | 21:55 | |
| *** VW_ has joined #openstack-operators | 21:56 | |
| *** laron has quit IRC | 21:58 | |
| *** VW__ has quit IRC | 22:00 | |
| *** jmckind has quit IRC | 22:00 | |
| *** VW_ has quit IRC | 22:01 | |
| *** sdake_ has joined #openstack-operators | 22:02 | |
| *** derekh has joined #openstack-operators | 22:03 | |
| *** SimonChung1 has joined #openstack-operators | 22:05 | |
| *** SimonChung has quit IRC | 22:05 | |
| *** sdake has quit IRC | 22:05 | |
| *** Marga_ has quit IRC | 22:07 | |
| *** Marga_ has joined #openstack-operators | 22:07 | |
| *** regXboi has quit IRC | 22:08 | |
| *** vsedelnik has quit IRC | 22:08 | |
| *** Piet has quit IRC | 22:11 | |
| *** derekh has quit IRC | 22:24 | |
| *** dminer has quit IRC | 22:41 | |
| *** mjr_ has joined #openstack-operators | 22:46 | |
| *** Marga__ has joined #openstack-operators | 22:49 | |
| *** Marga_ has quit IRC | 22:51 | |
| *** Piet has joined #openstack-operators | 23:00 | |
| *** vsedelnik has joined #openstack-operators | 23:00 | |
| *** vsedelnik has quit IRC | 23:00 | |
| *** mdorman has quit IRC | 23:04 | |
| *** csoukup has quit IRC | 23:08 | |
| *** carl_baldwin has quit IRC | 23:17 | |
| *** chlong has quit IRC | 23:27 | |
| *** chlong has joined #openstack-operators | 23:28 | |
| *** cdelatte has joined #openstack-operators | 23:34 | |
| *** emagana has quit IRC | 23:37 | |
| *** cdelatte has quit IRC | 23:37 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!