Friday, 2017-03-24

*** armax has joined #openstack-operators00:30
*** armax has quit IRC00:38
*** armax has joined #openstack-operators01:02
*** rebase has joined #openstack-operators01:09
*** Rodrigo_BR has joined #openstack-operators01:14
*** armax has quit IRC01:22
*** VW has quit IRC01:41
*** VW has joined #openstack-operators01:42
*** zul has quit IRC01:43
*** zul has joined #openstack-operators01:44
*** Rodrigo_BR has quit IRC01:45
*** Apoorva_ has quit IRC01:47
*** rebase has quit IRC01:54
*** VW_ has joined #openstack-operators02:18
*** kukacz has quit IRC02:20
*** VW has quit IRC02:21
*** karthikpr has joined #openstack-operators02:29
*** shewless_ has joined #openstack-operators02:33
*** karthikpr has quit IRC02:33
*** shashank_t_ has joined #openstack-operators02:46
*** raginbajin has quit IRC02:47
*** raginbajin has joined #openstack-operators02:52
shewless_Is the firewall_driver in openvswitchagent.ini just a compute option?02:54
shewless_I'm trying to figure out which part gets configured on the network node02:54
shewless_I see two relavent files:02:55
shewless_neutron/plugins/ml2/ml2_conf.ini and neutron/plugins/ml2/openvswitch_agent.ini02:56
shewless_anyone know what the significance of firewall_driver is on the network node and those two files?02:57
*** cemason1 has joined #openstack-operators03:03
*** cemason has quit IRC03:03
*** shewless_ has quit IRC03:04
*** fandi has joined #openstack-operators03:53
*** karthikpr has joined #openstack-operators04:09
*** marst has joined #openstack-operators04:31
*** haplo37 has quit IRC04:59
*** fragatin_ has joined #openstack-operators05:01
*** fragati__ has joined #openstack-operators05:03
*** fragatina has quit IRC05:05
*** fragatin_ has quit IRC05:06
*** fandi has quit IRC05:07
*** fragati__ has quit IRC05:07
*** haplo37 has joined #openstack-operators05:08
*** armax has joined #openstack-operators05:11
*** arnewiebalck_ has joined #openstack-operators05:12
*** karthikpr has quit IRC05:20
*** fragatina has joined #openstack-operators05:21
*** fragatina has quit IRC05:25
*** fragatina has joined #openstack-operators05:33
*** fragatina has quit IRC05:33
*** fragatina has joined #openstack-operators05:34
*** shashank_t_ has quit IRC05:53
*** yprokule has joined #openstack-operators05:53
*** karthikpr has joined #openstack-operators05:55
*** shashank_t_ has joined #openstack-operators05:59
*** karthikpr has quit IRC05:59
*** shashank_t_ has quit IRC06:03
*** arnewiebalck_ has quit IRC06:09
*** aojea has joined #openstack-operators06:12
*** aojea has quit IRC06:18
*** armax has quit IRC06:29
*** manheim has joined #openstack-operators07:22
*** manheim has quit IRC07:22
*** manheim has joined #openstack-operators07:23
*** tesseract has joined #openstack-operators07:31
*** simon-AS559 has joined #openstack-operators07:45
*** belmoreira has joined #openstack-operators07:59
*** simon-AS559 has quit IRC08:03
*** simon-AS559 has joined #openstack-operators08:16
*** mriedem has quit IRC08:16
*** aojea has joined #openstack-operators08:18
*** aojea has quit IRC08:18
*** aojea has joined #openstack-operators08:19
*** manheim has quit IRC08:33
*** treiz has joined #openstack-operators09:02
*** manheim has joined #openstack-operators09:03
*** aojea_ has joined #openstack-operators09:10
*** aojea has quit IRC09:12
*** racedo has joined #openstack-operators09:21
*** dbecker has joined #openstack-operators09:30
*** derekh has joined #openstack-operators09:33
yankcrimeshewless: possibly related to fwaas?09:40
*** electrofelix has joined #openstack-operators09:41
*** pcaruana has joined #openstack-operators10:05
*** karthik__ has joined #openstack-operators10:16
*** shewless has quit IRC10:18
*** karthik__ has quit IRC10:21
*** manheim has quit IRC10:24
*** manheim has joined #openstack-operators10:26
*** karthikpr has joined #openstack-operators10:33
*** manheim has quit IRC10:35
*** manheim has joined #openstack-operators10:36
*** karthikpr has quit IRC10:38
*** karthikpr has joined #openstack-operators10:50
*** karthikpr has quit IRC10:54
*** armax has joined #openstack-operators11:02
*** karthikpr has joined #openstack-operators11:07
*** Dinesh_Bhor has quit IRC11:07
*** karthikpr has quit IRC11:11
*** armax has quit IRC11:16
*** Dinesh_Bhor has joined #openstack-operators11:20
*** alexpilotti has quit IRC11:40
*** alexpilotti has joined #openstack-operators11:41
*** alexpilotti has quit IRC11:45
*** alexpilotti has joined #openstack-operators12:13
*** karthikpr has joined #openstack-operators12:17
*** karthikpr has quit IRC12:18
*** manheim has quit IRC12:20
*** manheim has joined #openstack-operators12:21
*** d0ugal has quit IRC12:24
*** liverpooler has quit IRC12:26
*** liverpooler has joined #openstack-operators12:26
*** racedo has quit IRC12:34
*** shewless has joined #openstack-operators13:01
*** karthik__ has joined #openstack-operators13:03
*** karthik__ has quit IRC13:04
*** mriedem has joined #openstack-operators13:06
*** karthikpr has joined #openstack-operators13:07
*** karthikpr has quit IRC13:11
*** cemason1 has quit IRC13:11
*** cemason has joined #openstack-operators13:15
*** manheim has quit IRC13:21
*** karthikpr has joined #openstack-operators13:34
*** karthikpr has quit IRC13:38
*** Dinesh_Bhor has quit IRC13:39
*** manheim has joined #openstack-operators13:42
*** jamesdenton has joined #openstack-operators13:43
*** d0ugal has joined #openstack-operators13:45
*** dansmith is now known as superdan13:49
*** rebase has joined #openstack-operators13:51
*** karthikpr has joined #openstack-operators13:53
*** rebase has quit IRC13:55
*** VW_ has quit IRC13:57
*** karthikpr has quit IRC13:58
*** alexpilotti has quit IRC13:58
*** alexpilotti has joined #openstack-operators13:59
*** kstev has joined #openstack-operators13:59
*** alexpilotti has quit IRC14:03
*** racedo has joined #openstack-operators14:17
*** jbadiapa has quit IRC14:18
*** alexpilotti has joined #openstack-operators14:20
*** electrofelix has quit IRC14:21
*** electrofelix has joined #openstack-operators14:21
*** shashank_t_ has joined #openstack-operators14:22
*** VW has joined #openstack-operators14:25
*** chlong has joined #openstack-operators14:25
*** yprokule has quit IRC14:30
*** jbadiapa has joined #openstack-operators14:33
zioprotohello all. I have a Openstack Heat problem. I want as an admin list all the heat stacks for all the tenants. If I try to do the command "openstack stack list --all-projects" I get a funny error "ERROR: You are not authorized to use global_index."14:34
zioprotowith the legacy "heat" client I dont see any command line option to list all tenants14:34
*** jsheeren has joined #openstack-operators14:36
zioprotomaybe I found the problem ... policy.json14:37
*** electrofelix has quit IRC14:38
*** electrofelix has joined #openstack-operators14:38
*** electrofelix has quit IRC14:41
*** electrofelix has joined #openstack-operators14:41
*** marst has quit IRC14:41
zioprotoI fixed it ! "stacks:global_index": "rule:context_is_admin", in the policy.json  where context_is_admin is defined as "context_is_admin":  "role:admin",14:45
zioprotobut I had to figure it out from https://ask.openstack.org/en/question/91180/magnum-bay-create-timeout-you-are-not-authorized-to-use-global_index/ that talks about something else14:46
*** fragatina has quit IRC14:47
*** marst has joined #openstack-operators14:49
*** erhudy has quit IRC14:50
*** fragatina has joined #openstack-operators14:53
*** karthikpr has joined #openstack-operators14:55
*** haplo37 has quit IRC14:55
*** jbadiapa has quit IRC14:55
*** jbadiapa has joined #openstack-operators14:56
*** alexpilo_ has joined #openstack-operators14:57
*** Vivek__ is now known as Vivek14:58
*** Vivek has quit IRC14:58
*** Vivek has joined #openstack-operators14:58
*** alexpilotti has quit IRC14:58
*** alexpilo_ has quit IRC15:02
*** haplo37 has joined #openstack-operators15:05
*** alexpilotti has joined #openstack-operators15:06
*** karthikpr has quit IRC15:08
*** jsheeren has quit IRC15:10
*** rebase has joined #openstack-operators15:13
*** rebase has quit IRC15:18
zioprotoyankcrime: Now I know why I hit every upgrade this collate issues. The puppet modules are changing my settings: https://github.com/openstack/puppet-openstacklib/blob/master/manifests/db/mysql.pp#L5915:29
klindgrenIs anyone here configuring neutron-openvswitch-agent to use ovsdb vs's using rootwrap for ovs configuration options?15:45
*** gyee has joined #openstack-operators15:45
klindgrenTHe documentation on how to do this is......... lacking...... to say the least.15:45
*** rebase has joined #openstack-operators15:48
*** d0ugal has quit IRC15:52
*** manheim has quit IRC15:56
*** manheim has joined #openstack-operators15:56
*** rebase has quit IRC15:59
*** rebase has joined #openstack-operators15:59
*** manheim has quit IRC16:01
*** pcaruana has quit IRC16:03
*** manheim has joined #openstack-operators16:13
*** d0ugal has joined #openstack-operators16:14
*** d0ugal has quit IRC16:14
*** d0ugal has joined #openstack-operators16:14
*** Oku_OS is now known as Oku_OS-away16:15
*** belmoreira has quit IRC16:21
*** manheim has quit IRC16:22
*** d0ugal has quit IRC16:22
*** manheim has joined #openstack-operators16:22
*** manheim has quit IRC16:27
*** shashank_t_ has quit IRC16:35
*** shashank_t_ has joined #openstack-operators16:35
*** kstev has quit IRC16:38
*** Apoorva has joined #openstack-operators16:42
*** Apoorva has quit IRC16:42
*** Apoorva has joined #openstack-operators16:42
*** fragatina has quit IRC16:46
*** simon-AS559 has quit IRC16:52
*** kstev has joined #openstack-operators16:54
*** aojea_ has quit IRC16:56
yankcrimelol zioproto16:56
yankcrime(╯°□°)╯︵ ┻━┻16:56
*** racedo has quit IRC16:57
*** makowals has quit IRC17:00
shewlessHi guys. Have you ever seen conntrack go out of control on your compute nodes? To the point where networking fails?17:08
shewlessIf I'm using hybridiptables for my firewall_driver can I blacklist the conntrac?17:08
*** electrofelix has quit IRC17:13
*** catintheroof has joined #openstack-operators17:14
*** catintheroof has quit IRC17:15
*** catintheroof has joined #openstack-operators17:15
*** makowals has joined #openstack-operators17:17
*** d0ugal has joined #openstack-operators17:17
*** racedo has joined #openstack-operators17:20
*** simon-AS559 has joined #openstack-operators17:21
shewlessanyone? I'm seeing over 500,000 conntrack connections on each compute.. getting crazy!17:23
*** simon-AS5591 has joined #openstack-operators17:23
*** simon-AS559 has quit IRC17:26
admin0just ?17:26
admin0shewless: our alerting is at 2 mil conntrack connections17:27
shewlessadmin0: that info helps! so it's acceptable to bump the max I guess :D17:27
shewless(in your experience)17:27
admin0yes17:31
admin0but if its out of the blue, you have an abuser, or a victim17:31
admin0of ddos17:31
admin0we graph all conntrack per compute nodes and then we can isolate from where its coming from17:31
admin0or going to17:31
*** marst has quit IRC17:37
yankcrimeshewless: yes17:40
yankcrimeit's usually, as admin0 says, the sign of a compromised vm or abusive behaviour (i.e a spammer)17:40
*** derekh has quit IRC17:40
admin0or a new customer who is running a new blog that has links to some most-wanted HD videos, or someone running his haproxy17:41
shewlessyankcrime, admin0: any recommendations to isolate the abuser? We can isolate to compute node already but not sure how to further drill down to instance17:42
admin0you can look into the conntrack table17:42
admin0and do some sort, awk magic17:42
shewlessare you using linux bridge networking?17:42
admin0conntrack -L  | bash-magic :)17:43
shewlessadmin0: thanks. Would I be looking to isolate by mac or instance id or something?17:43
shewlessAm I just looking for the thing that has the most entries?17:44
admin0things that has most entries17:44
yankcrimeyeah we've some messy bit of python that narrows down the abuser17:44
shewlessokay let me have a look17:45
shewlessbut you are both using linux bridge I take it?17:45
shewlessno option to "blacklist" conntrack?17:45
yankcrimeOVS in our case17:45
shewlessyankcrime: hmm. I didn't think you needed conntrack with OVS17:45
yankcrimewell, we still have linux bridges17:45
shewlessare you using the hybridiptables firewall_driver?17:46
yankcrimein order to apply iptables rules for security groups17:46
shewlessAh.. so that's what I know as the hybridiptables firewall_driver17:46
shewlesswhy not just use the openvswitch to do that? it should work with newer versions of OVS right?17:46
admin0i am linux-bridge17:47
yankcrimethere's been some progress in that area, but the version of openstack and ovs we're on means you still need linux bridges in the mix in order to be able to apply rules to vm's tap devices17:48
shewlessyankcrime: thanks.. that's what we're doing too.. though I want to move to straight ovs17:48
shewlessadmin0: thanks for the info.17:49
*** manheim has joined #openstack-operators17:49
*** manheim has quit IRC17:49
*** tesseract has quit IRC17:50
*** manheim has joined #openstack-operators17:50
*** alexpilotti has quit IRC17:52
*** alexpilotti has joined #openstack-operators17:52
shewlessyankcrime, admin0: do you guys know: if I create a network with port security disabled AND I'm using OVS + linux bridge for firewall: will this traffic go through conntrack?17:56
*** rebase has quit IRC17:59
*** rebase has joined #openstack-operators17:59
*** marst has joined #openstack-operators18:01
shewlessyankcrime, admin0: also. my conntrack -L is showing me a bunch of IP addresses that seem to be local to the instance.  Since my stacks use the same ip ranges for the most part I'm not sure how I can track down the offender. Are there any other optoins to use with -L?18:02
*** alexpilotti has quit IRC18:10
*** manheim has quit IRC18:14
*** manheim has joined #openstack-operators18:15
*** d0ugal has quit IRC18:17
yankcrimeshewless: pretty sure iptables is still involved, it just disables the anti-spoofing rules for that particular port18:17
shewlessyankcrime: darn. So I expect users to send a lot of traffic18:18
shewlessso I guess just increase conntrack18:18
yankcrimeand in our case for the tests we perform we actually parse /proc/sys/net/netfilter/nf_conntrack_max directly18:18
shewlessdo you mean /proc/net/ip_conntrack?18:19
yankcrimeand then the conntrack output we use is 'conntrack -L -o xml'18:19
yankcrimeiirc there's not that many options to userland conntrack tool18:20
*** dtrainor has quit IRC18:21
shewlessyankcrime: do you know if you need the firewall_driver set on the controller/network node or just the compute nodes?18:21
*** manheim has quit IRC18:24
*** chlong has quit IRC18:24
shewlessadmin0, yankcrime: does "a lot of traffic" always equal a lot of conntrack connections?18:29
shewlessIs there any case where that is not true?18:29
admin0it does not :)18:30
admin0lots of connects = not lots of traffic18:30
admin0i can use iperf, 5 -10 connections and choke the traffic out18:31
shewlessadmin0: ah I see.. lots of connections.. got it18:31
admin0in a while true loop ..18:31
yankcrimeyup, the problem here is not throughput - it's the number of connections and their states that need to be tracked18:31
yankcrime"a lot of traffic" can mean a variety of things ;)18:31
yankcrimedunno about that firewall driver offhand, i'd have to check18:32
shewlessyankcrime, admin0: I see this:18:35
shewlesstcp      6 428035 ESTABLISHED src=10.0.3.88 dst=10.0.6.148 sport=42730 dport=1723 src=10.0.6.148 dst=10.0.3.88 sport=1723 dport=42730 [ASSURED] mark=0 use=1 tcp      6 427137 ESTABLISHED src=10.0.3.3 dst=10.0.4.185 sport=27763 dport=56649 src=10.0.4.185 dst=10.0.3.3 sport=56649 dport=27763 [ASSURED] mark=0 use=1 tcp      6 426106 ESTABLISHED src=10.0.1.156 dst=10.0.2.246 sport=52167 dport=1723 src=10.0.2.246 dst=10.0.1.118:35
shewlessany idea how I can track down what's using 10.0.X.X?18:35
*** dtrainor has joined #openstack-operators18:38
*** shashank_t_ has quit IRC18:40
*** racedo has quit IRC18:41
*** arnewiebalck_ has joined #openstack-operators18:45
*** dtrainor has quit IRC18:46
*** shashank_t_ has joined #openstack-operators18:50
*** racedo has joined #openstack-operators18:51
*** dtrainor has joined #openstack-operators18:52
shewlessyankcrime, admin0: do you know what the repercussions would be if I completely disable/blacklist conntrack?18:55
*** rebase has quit IRC18:58
shewlessalso. is it possible to flag a network in openstack to use the NOTRACK option?19:14
shewlesssomething like iptables -t raw -A PREROUTING -d 22.33.44.55 -p tcp --dport 80 -j NOTRACK19:14
shewlesscould I manually add some raw PREROUTING entries to my computes?19:17
shewlesssomething like19:17
shewlessiptables -t raw -A PREROUTING -s 172.20.0.0/16 -d 172.20.0.0/16 -j NOTRACK19:18
*** arnewiebalck_ has quit IRC19:20
*** erhudy has joined #openstack-operators19:20
*** racedo has quit IRC19:22
*** racedo has joined #openstack-operators19:25
*** shashank_t_ has quit IRC19:27
*** shashank_t_ has joined #openstack-operators19:27
*** shashank_t_ has quit IRC19:31
klindgrenSo the issue with using notrack is that you have to portit both the inbound and outbound (related connection) via notrack19:31
klindgrenmost firewall rules permit either the inbound or the outbound part then uses --state established, related19:32
klindgrenor something similar.  But we notrack ssh, and other important for the compute node to work connections19:32
klindgrenYou can also adjust the number of tables and the size of the top level hash in real time. as well.19:33
klindgrenyou have to specifically allow intbound/outbound.19:34
*** arnewiebalck_ has joined #openstack-operators19:34
klindgrenIE doing: iptables -t raw -A PREROUTING -s 172.20.0.0/16 -d 172.20.0.0/16 -j NOTRACK will notrack the connections19:34
admin0back .. was driving19:34
admin0you can opt to not track, but you will then be unable to detect abusers until too late .. or you will suddenly find a lot of cpu usage, but no visible proces, because the CPUs are busy with interrupts and context switches19:35
klindgrenbut if you are relying on established,related, such as iptables -I INPUT -s 172.20.0.0./16 --dport 22 -j ACCEPT, iirc y also need to add the iptables -O OUTPUT -d 172.20.0.0/16 --sport 22 -j ACCEPT19:38
shewlesshmm.19:38
klindgrenbecause connection tracking job is to make it easier to allow both sides of connections through.19:39
shewlessMy problem is that I am providing stacks that utilize traffic simulation tools to send traffic over a private network to other instances19:40
shewlessthe private network has port security disabled completely19:40
shewlessthe traffic tools may simulate hundreds of connections..19:40
klindgrenpretty sure it still has anti-spoofing rules enabled?19:40
shewlessklindgren: probably.. not sure exactly what that means but it seems to be going through conntrack19:41
klindgreniptables-save | grep magic19:41
klindgrenlook for neutron-s(some-id)19:41
klindgrenis anti spoofing19:42
klindgrenneutron-o(some-id) is outbound19:42
klindgrenneutron-i(some-id) is inbound19:42
klindgrenalso tap(some-id-with-one-more-digit) is the vm's tap device19:44
shewlesslots of "neutron-openvswi-" entries.. no neutron-s or neutron-i19:44
shewlessklindgren: lots of neutron-openvswi-o, -s, and -i19:46
klindgrenah19:46
shewlessnot sure what that means though.. I guess it means antispoof?19:46
klindgrenso if you grep for the vm ip19:46
klindgrenyou should find -s rules19:47
klindgrenwhich try to make sure it has the correct mac address19:47
shewlessis that why conntrack is needed?19:47
shewlessklindgren: actually there are no rules in iptables-save for my "dataplane" Ip range19:50
*** slaweq has quit IRC19:52
klindgrenyea thts why contrack is needed.  because of the established,related allows in the -i rules for vm's19:52
klindgrenonce its enabled all connections by default are tracked.19:53
*** slaweq has joined #openstack-operators19:53
shewlessso in my case.. where I don't really need anything special on this private network, should i just increase the conntrack limit to a huge number or should I try and notrack certain ranges?19:54
*** liverpooler has quit IRC19:55
*** slaweq has quit IRC19:58
*** racedo has quit IRC20:05
*** slaweq has joined #openstack-operators20:06
*** aojea has joined #openstack-operators20:18
*** arnewiebalck_ has quit IRC20:26
*** armax has joined #openstack-operators20:29
*** kstev has quit IRC20:38
*** jamesdenton has quit IRC21:01
klindgrensorry - just coming back to this.  Thats what I would do.21:03
klindgrenYou can also adjust the connection tracking tables timeouts for things as well21:04
klindgrenso that some rules stick around for less time.21:04
klindgrenlike syn sent and the like21:04
*** aojea has quit IRC21:06
*** rebase has joined #openstack-operators21:07
*** beddari has quit IRC21:14
*** shasha_t_ has quit IRC21:21
*** aojea has joined #openstack-operators21:22
*** fragatina has joined #openstack-operators21:29
*** rebase has quit IRC22:11
*** rebase has joined #openstack-operators22:12
*** emerson has joined #openstack-operators22:37
*** simon-AS5591 has quit IRC22:38
*** arnewiebalck_ has joined #openstack-operators22:52
*** aojea has quit IRC22:59
*** erhudy has quit IRC23:00
*** arnewiebalck_ has quit IRC23:11
*** VW_ has joined #openstack-operators23:11
*** VW has quit IRC23:14
*** VW_ has quit IRC23:15
zioprotoshewless: just to check the obvious things. Read about conntrack kernel tunables here https://wiki.openstack.org/wiki/Documentation/HypervisorTuningGuide - do you have reasonable settings ?23:23
*** markvoelker has quit IRC23:24
*** jamesden_ has joined #openstack-operators23:35
*** racedo has joined #openstack-operators23:37
*** racedo has quit IRC23:42
*** jamesden_ has quit IRC23:53

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!