Controllerless Networks

I was hoping to pick some brains out there on where to begin trouble shooting a problem I am facing with clients dropping suddenly on our Aruba Instant 225 deployment at a school site.

For reference here is our setup:

-Airwave 8.0.10  - located at Datacenter at District Office

-Instant IAP 225s running firmware 6.4.2.6-4.1.1.11_52666 - located at a school site connected over WAN

-We have 44 IAPs deployed on this campus. One in every classroom.

-We are using a full deployment of Aruba MAS 2500/3500 switches and an Alcatel Lucent OS6900 Fiber router on the site.

-Mix of client devices, but primarily Acer netbooks running Ubermix (Ubuntu mod). Roughly 600 deployed on the campus.

Our problem:

We recently noticed that 90-100% of our clients are showing as dropping connection a few times a day. We have heard complaints from a couple of teachers that they are seeing the laptops drop their connections. I have attached an image of the "Clients graph" from airwave for this school site.

What I know:

-We have 23 school sites with full Aruba deployment and this is the only one showing these kinds of client drops.

-The uptime of the Access Points is 12hours + (as seen in the image) and so there is no suggestion that the AP's are rebooting for any reason.

-We are not losing packets to any of switches/routers out at the school site, so we are confident this is not a network outage scenario or flapping issue.

-There hasn't been any pattern or common factor, that we can identify, as to why the clients are dropping.

Anyways, I'm not entirely comfortable with Airwave and was hoping someone could point me in a good starting direction for finding the root cause of these MASS drops.

Thanks for any insight you can give!

In the SSID profiles, under Advanced, do you have Broadcast Filter ARP enabled?

Broadcast filter is set to 'Disabled.'

wlan ssid-profile PBVUSD-PSK
 enable
 index 2
 type employee
 essid PBVUSD-PSK
 wpa-passphrase 82895e8f696905660370fe24536864050c2eac882b1370d2
 opmode wpa2-psk-aes
 max-authentication-failures 0
 vlan 20
 rf-band all
 captive-portal disable
 hide-ssid
 dtim-period 1
 inactivity-timeout 86400
 broadcast-filter none
 dmo-channel-utilization-threshold 90
 local-probe-req-thresh 0
 max-clients-threshold 64

I would try setting that to ARP....on ALL your SSIDs.   Random broadcasts from clients is exactly that, random, so the client mix at that location might be more prone to broadcasts or you have a single client sending out sustained broadcasts which destroys wireless traffic.  If you have your wireless clients also sharing a Vlan with wired clients, those wired clients send out broadcasts at line rate and punish your client traffic.  Try setting broadcast filter to ARP and see if it continues.

I will give that a try and let you know how it goes! Thank you for the detailed response :-)

Unfortunately, we still saw a complete drop at about 11:00pm last night.

We do have a very vague error message on our controller though.

18:64:72:c9:aa:92# show ver
Aruba Operating System Software.
ArubaOS (MODEL: 225), Version 6.4.2.6-4.1.1.11
Website: http://www.arubanetworks.com
Copyright (c) 2002-2015, Aruba Networks, Inc.
Compiled on 2015-11-22 at 16:41:06 PST (build 52666) by p4build

AP uptime is 9 hours 5 minutes 44 seconds
Reboot Time and Cause: Reboot caused by kernel panic: Fatal exception in interrupt
18:64:72:c9:aa:92# 

Is this something I need to pursue further through support or is there a known cause for this error message?

that error is to general for some to say what happened. to me it seems your network or at least parts of it just randomly crash / reboot. i would contact TAC asap.

Just as an update, TAC was unable to help and our IAPs continue to randomly reboot. 

TAC forwarded the issue onto engineering on Monday and we have not heard anything.

What is the TAC ticket #?

Case number: 1825652

Just another update. As of Today, we have made zero progress on the problem. Engineering is still having me send logs in as often as I can, but we cannot pinpoint what is causing the random reboots with the "Fatal exception in interrupt" message.

State testing for the students starts tomorrow, so it should be an interesting couple of months. 

Got the same issue at a customer during a POC :|

Only 3 APs but one of them keeps 'rebooting' with that same error. Also running 6.4.2.6.

Slight difference with your case is we are using AP-215.

For us the AP uptime is resetting though.

Another thing I notices is that this AP continiously has a higer CPU utilization even though it is serving practicaly no clients and isn't the VC.  The VC with 10-15 clients is runnign at ~10% cpu util while this rebooting AP hovers around 25-35%.

Did you ever get a response from engineering?

We never found a complete resolution. We are a school district and during the summer when there are little-to-no clients connecting, we have noticed that the problem stops. We have closed the ticket but will reopen if the problem continues when school starts and the students return.

Here is what we did throughout the ticket:

• You mention few IAP’s 205 in the cluster are rebooting frequently.
• Checked the tech support log of the slaves we found there were no crash info.
• On the tech support log we saw IAP was rebooting due to kernel panic.
• IAP’s were running on 4.2.1.2. Total IAP’s pm cluster were 45.
• Issue was still observed on the above code. Then cluster was moved to 4.1.1.11 which was under general availability.
• After checking internally we suspected it to be an AMPDU issue with following changes :
rf dot11g-radio-profile
legacy-mode
rf dot11a-radio-profile
legacy-mode

The last change ( setting the radio profiles to legacy-mode) HELPED. We had far fewer reboots, but we did not see a complete resolution to the problem as they would still reboot 2-3 times a week.

This was filed as a bug in the IAP code and has been resolved in the 6.4.4.8-4.2.4.2 early-deployment firmware!!! The specific problem we were having is called out in the "fixed" section of the release notes.

Our IAPs that were rebooting every 30minutes - 1 hour have now been up for 20+ days :-) 

Kudos for reporting back with an update!