Wireless Access

We seem to have a lot of AP103H units that reboot for no obvious reason. This is happening enough that it's going to be impacting users. Not all of them do this, but a lot of them do. Wondered if anyone else has seen this.

Here's an example - the airwave log:

Mon Nov 27 05:15:22 2017 System Configuration verification succeeded; configuration is good
Mon Nov 27 02:07:05 2017 System Configuration verification succeeded; configuration is good
Mon Nov 27 02:06:47 2017 System Up
Mon Nov 27 02:06:47 2017 System Device Up: Device: coheaap14: Device Type is Access Point and Device Type is not Remote AP (Normal)
Mon Nov 27 02:06:47 2017 System Down
Mon Nov 27 02:06:47 2017 System Device has rebooted: Device uptime value changed (current: 4 mins 39 secs, calculated: 9 days 0 hrs 49 mins 52 secs) and Change in
number of reboots detected (got 1, expected 0)
Sun Nov 26 05:14:26 2017 System Configuration verification succeeded; configuration is good
...omitted 3 duplicate messages...
Wed Nov 22 08:18:36 2017 System Configuration verification succeeded; configuration is good
Wed Nov 22 08:10:22 2017 System Status changed to 'OK'
Wed Nov 22 08:10:22 2017 System Up
Wed Nov 22 08:07:40 2017 System AP's associated controller is changed from Arubal3 to Arubal1
Wed Nov 22 08:06:10 2017 System Status changed to 'AP is no longer associated with controller'
Wed Nov 22 08:06:10 2017 System Down
Wed Nov 22 04:47:56 2017 System Configuration verification succeeded; configuration is good
...omitted 4 duplicate messages...
Sat Nov 18 01:21:33 2017 System Configuration verification succeeded; configuration is good
Sat Nov 18 01:21:21 2017 System Up
Sat Nov 18 01:21:20 2017 System Down
Sat Nov 18 01:21:20 2017 System Device has rebooted: Device uptime value changed (current: 4 mins 30 secs, calculated: 3 hrs 15 mins 18 secs) and Change in number
of reboots detected (got 3, expected 2)
Fri Nov 17 22:08:18 2017 System Configuration verification succeeded; configuration is good
Fri Nov 17 22:08:01 2017 System Up
Fri Nov 17 22:08:01 2017 System Device Up: Device: coheaap14: Device Type is Access Point and Device Type is not Remote AP (Normal)
Fri Nov 17 22:08:01 2017 System Down
Fri Nov 17 22:08:01 2017 System Device has rebooted: Device uptime value changed (current: 2 mins 0 secs, calculated: 23 days 16 hrs 43 mins 39 secs) and Change
in number of reboots detected (got 2, expected 1)

#show ap debug system-status ap-name coheaap14

Reboot Information
------------------
(none found)
------------

Rebootstrap Information
-----------------------
Date Time Reason (Latest 10)
--------------------------------------
(none found)

HA Failover Information
-----------------------
Date Time Reason (Latest 10)
--------------------------------------
(none found)

Recent Control Messages from AP to Controller
---------------------------------------------
Date Time Message Description
-------------------------------------------------
Mon Nov 27 08:53:05 2017(248 secs ago): SENT REQ type=KEEPALIVE len=45 peer=144.32.64.34 seq_num=44 num_attempts=1 rtt=0 secs 04000000280400000002050A04E13104386D43C0040000002C045A1BD27105FFFFFC00050A04E0010000000000
Mon Nov 27 08:43:05 2017(848 secs ago): SENT REQ type=KEEPALIVE len=45 peer=144.32.64.34 seq_num=43 num_attempts=1 rtt=0 secs 04000000280400000002050A04E13104386D43C0040000002B045A1BD01905FFFFFC00050A04E0010000000000
Mon Nov 27 08:33:05 2017(1448 secs ago): SENT REQ type=KEEPALIVE len=45 peer=144.32.64.34 seq_num=42 num_attempts=1 rtt=0 secs 04000000280400000002050A04E13104386D43C0040000002A045A1BCDC105FFFFFC00050A04E0010000000000

Rebootstrap LMS
---------------
(none found)
------------

Crash Information
-----------------
(none found)
------------

How often do they reboot or is it a random occurrence? Is there any errors on the switch ports at the time of reboot as well?

Any chance you can run the below on the controller and provide the output?

#show log all | include XXXX (where XXX is MAC of the AP)

#show ap debug counters ap-name XXX (where XXX is name of the AP)

The reboot is fairly random, typically see several reboot over a 24 hour period, but not all the same units. This particular example rebooted in the early hours of this morning, the 18th and the 17th.

There's nothing in the log.

Here are the debug counters:

AP Counters
-----------
Name Group IP Address Configs Sent Configs Acked AP Boots Sent AP Boots Acked Bootstraps (Total) Reboots Crash Current License counter Global License counter GSM Info for AP
---- ----- ---------- ------------ ------------- ------------- -------------- ------------------ ------- ----- ----------------------- ---------------------- ---------------
coheaap14 Constantine 10.4.225.49 3 2 0 0 1 (2 ) 1 N 1/0/1/0/0/0/0 2/1/2/1/0/0/0 40/4/0/0

Should add that I've checked the switches for errors and all is good.

You might need to speak to TAC then if there is nothing in the logs/debug. Only other option I can suggest is to maybe attach a console cable to an AP, wait for it to reboot and see if you see any errors that way.

Yeah, it's going to be a TAC case. We have another problem with these APs anyway, where the CPU clock goes slow and the AP is rebooted for that reason. This is a "fix" for the slow clock bug, which isn't a fix at all imho but an ugly hack to resolve broken hardware. I'm attempting to reopen that case. This may end up being the same issue not being logged correctly. I've noticed that sometimes when the AP103H reboots because of a slow clock this isn't logged by the controller. It should be, but isn't always.

Recording the console log is a good idea, unfortunately these APs are in student bedrooms, so that isn't an option. We have several hundred of these and the rebooting is random so there's no guarantee any AP I choose to connect to will go wrong over several weeks.