Wireless Access

Hope someone can help me shed some light on an issue we currently suffer from. We have an Aruba3200 controller running 5.0.3.0 that was inherited from the previous admins with a total of 29 APs connected to it. On one of the remote sites we started seeing hourly network dropouts due to congestions on the network, which started coming after we added 4 new APs to the site. Further investigaions shows that the same traffic pattern is related to all APs connected to this controller. Below is a dump from on of the ports which an AP hangs on. As you can see, every hour there is a small peak. The total bandwidth will vary somewhat but the same pattern is visible for all ports which an AP is connected to.

 

So the question then becomes, what is it that the Aruba controller is doing every hour and what can we do to minimize the impact?

 

 

Regards,

Thomas

#3200

You can enable "Drop Broadcast and Multicast" at the virtual AP level. It is entirely possible that someone on your wired or wireless network that is shared with your wireless clients is sensing a high volume of broadcasts or multicast at that time. Those broadcasts get replicated to all access points that have users on them.

If the previous suggestion doesn't shed any light...

 

The graph you have provided, seems to indicate "in bps" on the switch port (which attaches to the example AP). Correct?

 

If that's the case, the traffic level shown is sourced from the AP, not the controller necessarily. Of course, it could be reply traffic of some sort responding to a controller request (unlikely). Much more likely, it's some real ingress traffic from a device/user coming up the GRE tunnel.

 

What does the output bps graph look like for the same port (which will be controller to AP traffic)? That would be good to see.

 

Also, does the monitoring platform you're using allow you to tell what type of traffic is represented (via sflow/netflow or similar)? That might greatly help narrow things down. If not, you could try a wireshark capture of a mirrored AP-to-switch port physically near to you? Is that practical?

 

 

Disabling Broadcast/Multicast is already planned for the weekend. Let's hope its that easy.

 

The previous graph was from a AP125 unit. The graph below is from the controller port.

 

I don't have anything set up for netflow management at the memoment but can be done if needed. Wireshark dumps is also easily done. If disabling the broadcast/multicast option doesn't work I'll look at these as a next option.

 

Thanks guys!

- Thomas

 

Disabling broadcasts/multicast did not help. Today we're seeing even more traffic from the APs...Traffic is definitely sent from the AP to the controller. Graph below is form an AP125 - blue marks outbound traffic. Problem is when we have 10 of these in one remote location on a 10 Mbit link... The traffic is not client traffic either since most of  these APs on the site have a very low usage frequency. I'm at a loss as to what this might be caused by.

 

 

Regards,

- Thomas

At this point, we are just guessing.  Please tell us :

 

- What version of ArubaOS

- How many access points you have

- What SSIDs you are broadcasting at these sites, with their encryption types

- Are you using the second port on the AP125 for wired access, as well.

 

Traffic from the access point could be management traffic, client traffic or both.  We need to figure out if this coincides with a specific event in your network.

 

 

 

Thank

- ArubaOS 5.0.3.0

- 30 APs in total. Mostly AP125/121

- 2 to 3 SSIDs being broadcasted depending on site/AP group.

- WPA2/802.1x (AES-CCMP) for PCs, WPA2/PSK (AES-CCMP) for PDAs and None/Open if the 3rd - Guest SSID is provisioned.

- Only 4 APs are being used with the second wired port (remote home offices)

 

Regards,

- Thomas

 

 

Thank you.  Do you have "Drop Broadcast and Multicast" enabled in ALL virtual APs in that AP-Group?  If not, broadcasts can be propagated from other SSIDs.

 

If you have wired access at those sites, you also need to enable broadcast/multicast optimization (same as Drop Broadcast and Multicast, but for the wired) at the VLAN level to prevent broadcast propagation.  Whatever vlan is assigned to that wired port:

 

(192.168.1.3) #configure t
Enter Configuration commands, one per line. End with CNTL/Z

(192.168.1.3) (config) #interface vlan 1000
(192.168.1.3) (config-subif)#bcmc-optimization 

 If those do not work, we need to look at other things.

Thanks for your help so far cjoseph! Much appreciated.

 

I have enabled bcmc-optimiization on the vlan interfaces. Drop Broadcast/Multicast is already disabled for all the AP groups.

 

However, would we not see more data coming in to the APs if this was caused by broadcast/mulitcast? As it is now the traffic is originating from the APs and not to the APs.

 

Thomas

 

thnilsen,

 

If it is from a single access point at a time, it could be a single user with a large download.  User traffic appears to come from the same port/ip address of the access point.  If it is multiple access points at a time, we did what we need to with broadcast and multicast suppression.  Do you have anything that is doing historical monitoring in your network, like Airwave?

We don't have anything that logs history at traffic level apart from RRD graphs of the traffic on switch ports.  TCPdump can be done for AP or controller if needed.

 

Should know within the next 12 minutes if the problem still exists or not since the issue mostly occurs between xx:18 and xx.20,

 

Thanks.

 

Thomas

No joy - APs are still peeking as before.

 

Next step I guess would be to do a packet capture of the traffic during the peek?

 

- Thomas

Yes.  At the controller port would probably be the best thing.

 

A full packet capture on one of the APs revealed the issue. There was indeed multicast traffic hitting the APs. The traffic originated from one of the VLANs which I have not enabled bcmc-optimization on, since it is the same VLAN the APs connect to the controller via ADP. I guess I'll just have to create access list for the wired interfaces on our controller to drop this particular multicast group traffic.

 

Thanks to anyone who offered to help!

 

Regards,

Thomas

 

 

Thomas,

 

What version of ArubaOS are you running?  The controller is designed to pass ADP (Aruba Discovery Protocol) traffic, even if bcmc-optimization is enabled on the VLAN.  There is a bug, however, where the controller blocks it and it is resolved by upgrading to ArubaOS 6.3.1.3.

We're om 5.0.3.0. Upgrading to newer firmware is not an option at the moment as the support contract has expired.

 

- Thomas