Wireless Access

I'm running a cluster of 24 IAP-105s.  Last night I updated them from firmware revision 6.2.1.0-3.3.0.0 to revision 6.2.1.0-3.3.0.3, and this morning three of the APs began rebooting every 10-20 minutes.  'show version' shows "Reboot Time and Cause: unknown".  

The APs that are having the issue are in the parts of our building where we have the most devices on the network.  

Can anyone offer any assistance in troubleshooting this?  I'm not sure how to tell whether this is just due to load, or if there's a specific device causing the issue (the three APs that I've seen rebooting are on different floors of our building in the same corner, so it's conceivable that the same device is hitting all three of them).  We do mobile QA, and we have a lot of devices running various older versions of iOS and Android that have known breakage in their network and wireless implementations, so it's quite possible that one of them is sending some evil malformed auth request or something like that.

I'm running two SSIDs, one using WPA-2 Enterprise authenticaion with an external RADIUS server and the other using WPA-2 Personal (again, due to broken devices that are incapable of connecting WPA-2 Enterprise).

Thanks,

-- monroe

Just so you're not waiting for someone, you should open a TAC case so they can get the logs. If you think it's code related, you can roll back to older Instant code, assuming you can take the outage.

Thanks, I've sumbmitted a ticket (#1450199).

Since the problem began I've set up a syslog server to capture logs from the cluster.

As far as I can tell, there's no indication of trouble in the logs before the APs reboot, the first thing I see when one reboots is an entry from the elected controller saying "Find AP-xxx.xxx.xxx.xxx fail." 

I am planning to roll back to 6.2.1.0-3.3.0.0 tonight (6PM U.S. Pacific time) if I can't find some way to mitigate the issue before then.  If there's anything I can do to troubleshoot before the rollback, I'd love to find out about it ASAP. :)

It looks like the rollback to 6.2.1.0-3.3.0.0 did stop the reboots -- yesterday I had 67 reboots between 8AM and 1PM, and today I have zero.

Is there an easy way to rollback the code? I have never down graded firmware before and I am having the same issue when I upraded from 3.4.0.6 to 4.0.0.6.  Only remote locations seem to have problems, anything behind the controller is fine with the new code level.

I have a support ticket open and disabled client match which has seemed to help but our users are getting booted off the 5ghz band very frequently. 

Thank you

trekowski,

You need to download the code from the support site to downgrade.

We are not clear on your issue.  You say you are having the same issue, but that users are getting booted off the 5ghz.  The issue is that access points are rebooting.  How do you know, in addition that users are getting booted off the 5ghz?

When I upgraded to the new code level the AP's were rebooting at all remote locations like crazy. Airwave would see them drop I could also see the ports go down in the switches powering the AP's. 

I opened a case and the first thing they had me do was remove client match and that stopped them from rebooting. Now at the 4.0.0.6 code level when a user is connected they will lose connection for a split second and reconnect. This is happening at all remote locations since the upgrade and is the most noticable when a user is trying to stream.

trekowski,

You are doing the right thing contacting support.  You could have a hardware issue.  Downgrading could lead to other issues that are fixed, so it is risky.  I am not sure that we can do more than what TAC is doing here.  Please report back on the result.