Wireless Access

First post here, I apologize if it's not in the correct forum section.  We have been seeing an issue for the last 3 weeks in a row that we can't remedy and haven't gotten anywhere with either our firewall support (palo alto) or wirelss vendor support (aruba).  Every Thursday around the same time (noonish) for the last 3 weeks in a row we've had an incident that has crippled our network for 1/2 hour-1.5 hours.  Our network monitoring (solarwinds) will start alerting that many of our sites have gone down, and when we start investigating we see an incredible amount of bandwidth in our firewall logs between an access point and it's wireless controller.  The traffic is identified as GRE traffic with a source of the access point, destination of the wireless controller.  We have seen 20+TB in under 15 minutes!  The last 2 times it was isolated to 1 site, 1-2 AP's and 1 controller.  Today, we saw it on 10-15 APs at one site and one at another.  We arent' able to get much more insight on the traffic as it's GRE encapsulated and we're not able to get on the wireless controller at the time of the issue, eseentially the network is brought to a crippling halt.  We've attempted to examine the traffic/threat logs from the palo alto firewall with vendor support and haven't been able to track down the issue.  We've worked with aruba support to no avail, provided them full logs of the events during the issue and they have no idea what's happaning.  I can tell you it's not passing from/to the internet as the monitoring of our perimeter firewall shows no abnormal bandwidth during the attacks.  The network monitoring on the firewall ports that bring our WAN (where the APs are) into our LAN (where the wireless controllers are) actually seems to show a marked decline in through put during these incidents, which is mind boggling.

Has anyone seen anything like this before? 

Any ideas of where to look, or tips to try? 

We have seen DDoS attacks in the past, but it was clearly inbound from the internet - this seems to originate on the WLAN and stay within our LAN.

User traffic is typically sent over GRE tunnels.  I would:

- Make sure broadcast and Multicast is dropped at the Virtual AP  to ensure that it is not wired multicast that is bringing your network to a halt:

I appreciate the quick reply Colin.  I'm new to this wireless controller, where would I find those settings?  I looked around the AP config and found a 'broadcast' setting that was checked...not sure that is what you are referring to though?

You should go to Network > IP > IP Interface.  Edit the VLAN with the problem.  Enable BCMC Optimization.

Thanks again!  We'll give this a try and post back if it fixed it or not...seems to happen every thursday for some reason so will post back in a week

This appears to have fixed our issue, thanks for the help Colin!