Wireless Access

Reply
MVP
Posts: 1,418
Registered: ‎10-25-2011

150 AP93s offline for some reason, reboot controller, they come back up

[ Edited ]

Hi,

 

One of our 3600 controllers experienced some weirdness early this morning where 150 out of 500 APs associated to the controller went offline, a reboot of the controller brought them all back up.

 

Any particular issue that anyone can think of that can cause this?

 

Our graveyard guy saved the logs, but looking at them, I do not see anything related to this.

 

Calling Aruba TAC since the issue is now resolved probably wont do much since they want to see the issue live and so would I.

 

Any thoughts?

 

 

Firmware: 6.1.3.0_32142

 

Pasquale Monardo | Senior Network Solutions Consultant
ACDX #420 | ACMP
[If you found my post helpful, please give kudos!]
Aruba Employee
Posts: 664
Registered: ‎04-15-2009

Re: 150 AP93s offline for some reason, reboot controller, they come back up

Since you have 500 APs on a 3600, I assume these are all RAPs, right?  Could it have been an issue with a specific ISP?  Are all 150 that bounced in a certain geo. location?  When they came back up, did you get a log message saying why they bounced?  If so, can you post it for 1-2 or them?

MVP
Posts: 1,418
Registered: ‎10-25-2011

Re: 150 AP93s offline for some reason, reboot controller, they come back up

They are all RAPs in bridged mode.

They are spread all over Canada.

 

Do not think it was an ISP issue.

 

Unfortunately all I got is what my agent saved as logs. He rebooted the controller therefore all of the logs are gone prior reboot.

I can't seem to attach it to my post.

 

Any commands I could run currently to validate?

I ran show ap debug system-status ap-name <>

 

 

 

 

Pasquale Monardo | Senior Network Solutions Consultant
ACDX #420 | ACMP
[If you found my post helpful, please give kudos!]
Aruba Employee
Posts: 664
Registered: ‎04-15-2009

Re: 150 AP93s offline for some reason, reboot controller, they come back up

Unfortunately, nothing jumps out of the logs (at least to me).

 

When you ran "show ap debug system-status ap-name <name of a RAP that booted>", what did you see for the reboots/bootstraps?

MVP
Posts: 1,418
Registered: ‎10-25-2011

Re: 150 AP93s offline for some reason, reboot controller, they come back up

[ Edited ]

That is what I saw. Weird that the date/time is WAY off.

The clock on the controller is valid.

 
Reboot Information
------------------
AP rebooted Mon Jul 23 01:21:56 UTC 2012; Process /usr/sbin/dnsmasq has too many open files (770)
-------------------------------------------------------------------------------------------------

Rebootstrap Information
-----------------------
Date       Time     Reason (Latest 10)
--------------------------------------
1999-12-31 16:02:03 Changing to LMS #0 (38.108.87.62)
2012-07-23 02:01:17 Missed heartbeats
2012-07-23 02:05:32 Changing to LMS #0 (38.108.87.62)
2012-07-23 04:01:31 Missed heartbeats
2012-07-23 04:05:12 Changing to LMS #0 (38.108.87.62)


Rebootstrap LMS
---------------
2012-07-23 04:05:12 Changing to LMS #0 (38.108.87.62)
-----------------------------------------------------

 

edit: sorry pasted wrong information

Pasquale Monardo | Senior Network Solutions Consultant
ACDX #420 | ACMP
[If you found my post helpful, please give kudos!]
Aruba Employee
Posts: 664
Registered: ‎04-15-2009

Re: 150 AP93s offline for some reason, reboot controller, they come back up

When an AP reboots, they report their time like that.  There is no battery for the clock, so when they power off, they "forget" the time and the first log message or two will be their factory default time.  As soon as they sync with a controller, they update their time, though.

 

I would open a TAC case on this.  The message below is the culprit:

 

Reboot Information
------------------
AP rebooted Mon Jul 23 01:21:56 UTC 2012; Process /usr/sbin/dnsmasq has too many open files (770)
--------------------------------------------------

-----------------------------------------------

 

TAC may be able to get the crash information from the AP and open a bug (or add your case to an existing bug to get it fixed quicker).

MVP
Posts: 1,418
Registered: ‎10-25-2011

Re: 150 AP93s offline for some reason, reboot controller, they come back up

Perfect thanks.

 

Any idea what the reboot msg means?

Pasquale Monardo | Senior Network Solutions Consultant
ACDX #420 | ACMP
[If you found my post helpful, please give kudos!]
Aruba Employee
Posts: 664
Registered: ‎04-15-2009

Re: 150 AP93s offline for some reason, reboot controller, they come back up

Seems to be a bug, but I am not 100% sure.  TAC can dig into it further and verify it's a known bug or open a new one if they haven't seen it before.

MVP
Posts: 1,418
Registered: ‎10-25-2011

Re: 150 AP93s offline for some reason, reboot controller, they come back up

On the phone currently opening a case.

 

Thanks for all your help.

Pasquale Monardo | Senior Network Solutions Consultant
ACDX #420 | ACMP
[If you found my post helpful, please give kudos!]
MVP
Posts: 1,418
Registered: ‎10-25-2011

Re: 150 AP93s offline for some reason, reboot controller, they come back up

Shipped off logs, crash info and output of 5 'show ap debug system-status ap-name <ap-name>.

 

 

Pasquale Monardo | Senior Network Solutions Consultant
ACDX #420 | ACMP
[If you found my post helpful, please give kudos!]
Search Airheads
Showing results for 
Search instead for 
Did you mean: