Controllerless Networks

I have 13 IAP 215s spread across 3 locations that will randomly all have their utilization go through the roof and then they will stop passing traffic and the whole wifi network will go down for 5 minutes or so. Then it will come back up and work for a random amount of time before it happens again.  Sometimes it starts with high noise on the AP's sometimes not.  I opened a case with Aruba support, they gathered the logs and said they would call me back and that has been two days ago so I'm hoping someone here can help me figure something out.  

#AP125

What version of instant code is this?
Which utilization are you talking about specifically?
Do you have Airwave monitoring this installation?

It is version 6.4.2.6-4.1.1.11_52666. 

 

The utilization meter on the dashboard is what I'm talking about.  Noise meter as well.  

 

I do not have Airwave monitoring this install.  I'm new to Aruba, only had these about a year and they worked great until about a week ago.  

Could you provide a screenshot of the dashboard?

It's only happening to one right now, but that's what it looks like.  It will start with one, spread to the others and take the network down, or sometimes the noise will go up on one spread to all and then cause the utilization to go up.  If I reboot all the AP's and the Firewall, it will come back up and stay for longer than if I just let it come back up by itself.  

 

 

High utilization can be brought about by legitimate high usage or too much management traffic.  Please click on the access point heading so we can see the details of your access points (power/channel,etc).

It shouldn't be legitimate high user usage because it doesn't seem like all the AP's would be afftected at once.  

 

Here is the AP setup but it's not happening right now.  

We can't tell anything from that chart.  You need to click on the 13 access points heading so we can see the channel and power of all of your access points.

 

Question:  Are the clients and the access points and wired clients all on the same subnet?

Question:  Do you have "Broadcast Filter ARP" enabled on all of your SSIDs in the advanced dialog?

 

Sorry, here is the correct picture.  

 

The wired and wireless are on seperate subnets. 

 

Broadcast Filtering is set to ARP. 

All of your access points are at maximum power, which would create if not add to utilization issues and create roaming issues.  You should go to the ARM maximum transmit power and change it to 18, to reduce the effects of that.  You should even lower it to 15 if you have dense coverage.  If you can see 3 or more access points at -65 or stronger when standing in the same spot, you could turn it down to 15 and see if the utilization will go down with it.

Thank you very much for your help, I will try that.  

Do I have to reboot the APs for ti to take affect? 

No, a power change should not require a reboot.  The changes I recommended are not particularly intrusive.  Typically if the APs are seeing alot of broadcast traffic on their management subnet, that could affect how they are performing.  How big is the AP management subnet?

The AP management subnet is a /28

Okay.  That does not seem too big.  Please monitor and see if the utilization is lower.  If not, more steps will need to be taken.

Stragely enough, it may be something with the firewall.  Today I noticed it was terrible until I rebooted the firewall first and then the AP's......it has worked all day since that.   I did a quick survey and nothing is above -65, but I turned them all down to 18.  I don't have a dense ap population, what I love about the Arubas is that I was able to use 1/3 of the crap Sonicpoints I was using.  However the Arubas are still connected to the Sonicwall as their gateway/dhcp server.  Would there be anything there?  

Are you using Manual Channel and power or are you relying on the ARM Max and Minimum to manage the power?

 

Relying on the ARM max and min.  

Ok.

 

I do not know if your firewall could be the cause or contribute the issue.  We need to monitor the utilization after the power has been lowered to see if it makes a difference.  It could be one of several steps we need to take to understand what could be going on here.

 

 

It happened again.  When it happened one of the APs dropped off the list and I went down to 12.  Do you think it could be a bad AP? 

 

Can you please PM me your Tech-Support?

I mean PM me your TAC ticket #

done. 

I changed out the firewall to see if that was it, but I'm having the same problems today.  How can I look at the logs to find something out on this. 

 

 

 

If you run the below via the CLI it will give you all the log options.

 

# show log ?

Lokean,

 

I hope you are continuing to work with TAC, because they have all of your data and can look at this live.  Please call their number instead of email if this is continuing to cause customer-facing issues.

I called them back in and they want me to put a packet capture on the network and see if they can tell from that.  I do notice this in the logs when it happens. 

 

SAPCP: Unknown frame type

 

Showing over and over again. 

I also see this when the utilization goes high, but no access points around me move then it will go back down and none will be interfering. 

 

Hi!

 

Have you seen any specific time of day this seems to be happening? High load times only or also on low or no load times?

 

You said you APs, get their IP-address on a /28 subnet. Is it only APs residing on that subnet and the gateway is the Sonicwall? Do you have an special configuration we should know about? Perhaps you can export your config, erase sensitive information and post it to us so we can look at your configuration?

 

Cheers,

I may have actually narrowed this down to my EAP network.  Everything else I chased around kept cropping back up, I even replaced the Sonicwall with a Fortigate firewall, but I disabled the EAP network this morning and it has been ok since. I do still get this error quite a bit:

 

The AP cannot receive data from this client because the integrity check of the received message (MIC) has failed. Recommend checking the encryption setting on the client and on the AP. 

 

To answer your questions, I was actually wrong and the AP addresses are on a /24 along with the wireless clients, there is no seperate mgmt network yet.  I don't have a special configuration that I know of.  How would I export my config? 

in the GUI, maintenance (right top) and then configuration, make backup.

 

but restoring that will also restore changes on your interfaces so im not sure if that is what you want.

First thing I would do is to create a new VLAN for your wireless clients, I asume this is your EAP network that ends up on the same subnet as your APs. Put that network on the new VLAN instead to separate them from your APs and see if it helps.

 

Cheers,

i did try that and it didn't work.  I've been searching for loops in the network because I recently have found that the network will get IGMP flooded mostly by apple products and that's what's causing the utilization to go through the roof.   I tried blocking the IGMP packets per SSID with a rule, but it doesn't seem to help and as much as I search the couple wifi switches, I don't see a loop.  Is there any way to stop these floods by policy on the arubas?