Wireless Access

Reply
New Contributor
Posts: 4
Registered: 3 weeks ago

Massive volume of GRE traffic between AP and controller crippling network

First post here, I apologize if it's not in the correct forum section.  We have been seeing an issue for the last 3 weeks in a row that we can't remedy and haven't gotten anywhere with either our firewall support (palo alto) or wirelss vendor support (aruba).  Every Thursday around the same time (noonish) for the last 3 weeks in a row we've had an incident that has crippled our network for 1/2 hour-1.5 hours.  Our network monitoring (solarwinds) will start alerting that many of our sites have gone down, and when we start investigating we see an incredible amount of bandwidth in our firewall logs between an access point and it's wireless controller.  The traffic is identified as GRE traffic with a source of the access point, destination of the wireless controller.  We have seen 20+TB in under 15 minutes!  The last 2 times it was isolated to 1 site, 1-2 AP's and 1 controller.  Today, we saw it on 10-15 APs at one site and one at another.  We arent' able to get much more insight on the traffic as it's GRE encapsulated and we're not able to get on the wireless controller at the time of the issue, eseentially the network is brought to a crippling halt.  We've attempted to examine the traffic/threat logs from the palo alto firewall with vendor support and haven't been able to track down the issue.  We've worked with aruba support to no avail, provided them full logs of the events during the issue and they have no idea what's happaning.  I can tell you it's not passing from/to the internet as the monitoring of our perimeter firewall shows no abnormal bandwidth during the attacks.  The network monitoring on the firewall ports that bring our WAN (where the APs are) into our LAN (where the wireless controllers are) actually seems to show a marked decline in through put during these incidents, which is mind boggling.

 

Has anyone seen anything like this before? 

 

Any ideas of where to look, or tips to try? 

 

We have seen DDoS attacks in the past, but it was clearly inbound from the internet - this seems to originate on the WLAN and stay within our LAN.

Guru Elite
Posts: 21,291
Registered: ‎03-29-2007

Re: Massive volume of GRE traffic between AP and controller crippling network

User traffic is typically sent over GRE tunnels.  I would:

 

- Make sure broadcast and Multicast is dropped at the Virtual AP  to ensure that it is not wired multicast that is bringing your network to a halt:



Colin Joseph
Aruba Customer Engineering

Looking for an Answer? Search the Community Knowledge Base Here: Community Knowledge Base

New Contributor
Posts: 4
Registered: 3 weeks ago

Re: Massive volume of GRE traffic between AP and controller crippling network

I appreciate the quick reply Colin.  I'm new to this wireless controller, where would I find those settings?  I looked around the AP config and found a 'broadcast' setting that was checked...not sure that is what you are referring to though?

Guru Elite
Posts: 21,291
Registered: ‎03-29-2007

Re: Massive volume of GRE traffic between AP and controller crippling network

You should go to Network > IP > IP Interface.  Edit the VLAN with the problem.  Enable BCMC Optimization.



Colin Joseph
Aruba Customer Engineering

Looking for an Answer? Search the Community Knowledge Base Here: Community Knowledge Base

New Contributor
Posts: 4
Registered: 3 weeks ago

Re: Massive volume of GRE traffic between AP and controller crippling network

Thanks again!  We'll give this a try and post back if it fixed it or not...seems to happen every thursday for some reason so will post back in a week

New Contributor
Posts: 4
Registered: 3 weeks ago

Re: Massive volume of GRE traffic between AP and controller crippling network

This appears to have fixed our issue, thanks for the help Colin!

Search Airheads
Showing results for 
Search instead for 
Did you mean: