Wireless Access

Reply
Occasional Contributor II

APs randomly dropping offline with no config status

Master controller: 3600 running 6.2.1.3, in a datacenter here in Vancouver, WA

Local controller: 3200 running 6.2.1.3, onsite in Seattle, WA

Local switch: 3500-48p

Access points: 8x 105

Connectivity between locations is a private WAN connection and is 100% reliable as far as I can tell. Wired computers are reporting no issues. Wireless connectivity drops campus-wide 4-5 times a day.

I have a new 7010 that will replace the 3200 soon, but I'm not sure this will resolve the issue.

 

This setup was reliable up until this week. Suddenly, we have outages a few times a day. All of the access points will disappear from the local, and show up on the master controller in ID status (Inactive, Drity or no config). Re-provisioning them from the master causes them to come back up quickly. Doing nothing, they come back up on their own after a while (not sure how much of a delay). The access points, switch, and controller are not reporting any reboots - they all show uptime from before the outage.

 

I have found no evidence of connectivity loss locally, and the wireless controllers, APs, and Adtran router (for connectivity back to the datacenter) are all on the same 3500-48p switch. It seems that the APs are sporadically unable to find the local controller, and falling back to the master, but I can't find any evidence to support why. I've moved the controller to a new port on the switch, and neither port has shown any downtime in SolarWinds. No device shows downtime in SolarWinds either. During outages, I am able to access the master, local, switch, and APs with no issues.

 

Attached is a screenshot of client totals in the last 24 hours. The biggest trough (between 12:00 and 15:00) was especially long because someone accidentally unplugged power to the local controller. The others are the outages I am referring to.

 

Any thoughts on cause or next troubleshooting steps? I will be replacing the local 3200 controller with the 7010 on the 12th, though I'm not confident that will provide any relief.

Guru Elite

Re: APs randomly dropping offline with no config status

If you haven't, please open a tac case so that they can get all the details that you cannot reveal here. We would only be guessing with limited information on this forum, but please have TAC also work on it in the background.


Colin Joseph
Aruba Customer Engineering

Looking for an Answer? Search the Community Knowledge Base Here: Community Knowledge Base

Guru Elite

Re: APs randomly dropping offline with no config status

With that being said, you should type "show log system all" on your controllers to see if you can get a hint of why your devices are dropping off...



Colin Joseph
Aruba Customer Engineering

Looking for an Answer? Search the Community Knowledge Base Here: Community Knowledge Base

Occasional Contributor II

Re: APs randomly dropping offline with no config status

I'm copying the crash.tar now, and will attach it to my support case. Log output, with MACs and IPs removed:

 

Apr 8 13:29:20 :303073:  <ERRS> |nanny|  Process /mswitch/bin/stm [pid 1621] died: got signal SIGSEGV

Apr 8 13:29:23 :303029:  <ERRS> |nanny|  Process /mswitch/bin/stm [pid 1621]: crash data saved in dir /flash/crash/process/4-8-2016@13-29-20/stm

Apr 8 13:29:27 :303079:  <ERRS> |nanny|  Restarted process /mswitch/bin/stm, new pid 7757

Apr 8 13:29:27 :303025:  <ERRS> |nanny|  Found core file /tmp/core.1621.stm.A6xxx_39155, 20033536 bytes, compressing...

Apr 8 13:29:45 :311004:  <WARN> |AP 084-1 sapd|  Missed 8 heartbeats; rebootstrapping

Apr 8 13:29:45 :311004:  <WARN> |AP 084-2 sapd|  Missed 8 heartbeats; rebootstrapping

Apr 8 13:29:45 :311004:  <WARN> |AP 084-3 sapd|  Missed 8 heartbeats; rebootstrapping

Apr 8 13:29:45 :311004:  <WARN> |AP 084-4 sapd|  Missed 8 heartbeats; rebootstrapping

Apr 8 13:29:45 :311004:  <WARN> |AP 084-5 sapd|  Missed 8 heartbeats; rebootstrapping

Apr 8 13:29:45 :311004:  <WARN> |AP 084-6 sapd|  Missed 8 heartbeats; rebootstrapping

Apr 8 13:29:46 :311004:  <WARN> |AP 084-7 sapd|  Missed 8 heartbeats; rebootstrapping

Apr 8 13:29:46 :311004:  <WARN> |AP 084-8 sapd|  Missed 8 heartbeats; rebootstrapping

Apr 8 13:29:51 :303080:  <ERRS> |nanny|  Please tar and email the file crash.tar to support@arubanetworks.com

Guru Elite

Re: APs randomly dropping offline with no config status

Your switch is crashing and your access points are losing connectivity to it as a result.  You need to contact TAC to see if an upgrade can help you.



Colin Joseph
Aruba Customer Engineering

Looking for an Answer? Search the Community Knowledge Base Here: Community Knowledge Base

Occasional Contributor II

Re: APs randomly dropping offline with no config status

We swapped out the controller with a 7010 as planned, and have not had the issue occur again since. At this point I'm guessing it was maybe a hardware problem with the interface on the controller, but I'll likely never know. Thanks for your help!

Search Airheads
cancel
Showing results for 
Search instead for 
Did you mean: