Wireless Access

So, we have several access points in five different buildings throughout the school district that I work for. They are broken up into clusters with 8-14 access points in each cluster, being managed by virtual controllers. The middle school is giving us some serious problems, and now the elementary and the high school are starting to show the same symptoms. The access point that was designated as the master keeps randomly rebooting and forcing all of the access points to reboot or to just drop. This is getting rather crazy. We talked to TAC and they were sorta helpful, but we are still working on them. I have a bunch of logs and such, we are getting tons of PAPI errors

 

" PAPI_Send failed: papi_send_status_callback, 2879: Connection timed out; from 8451"

 

I was hoping that someone has seen this before and could possibly offer some insight. Thanks!

That message is not very deterministic, in itself.  It could be cosmetic in some cases.  You need to look for the reboot reason in the IAP, by typing "show version" on the commandline or use the "show tech-support" in the diagnostics tab:

 

show version
Aruba Operating System Software.
ArubaOS (MODEL: 225), Version 6.4.2.6-4.1.1.8
Website: http://www.arubanetworks.com
Copyright (c) 2002-2015, Aruba Networks, Inc.
Compiled on 2015-07-25 at 04:49:45 PDT (build 50989) by p4build

AP uptime is 3 weeks 5 days 21 hours 14 minutes 25 seconds
Reboot Time and Cause: AP rebooted Thu Oct 1 15:23:16 CDT 2015; Image Upgrade Successful
end of show version

Ok, I've done that for a few of the access points in the area where we are having the most issues, and the results are as follows:

 

MS-Lab15-205-C# sho ver
Aruba Operating System Software.
ArubaOS (MODEL: 205), Version 6.4.2.6-4.1.1.10
Website: http://www.arubanetworks.com
Copyright (c) 2002-2015, Aruba Networks, Inc.
Compiled on 2015-09-23 at 00:11:51 PDT (build 51810) by p4build

AP uptime is 5 minutes 8 seconds
Reboot Time and Cause: unknown

 

MS-CafeNorth-105# sho ver
Aruba Operating System Software.
ArubaOS (MODEL: 105), Version 6.4.2.6-4.1.1.10
Website: http://www.arubanetworks.com
Copyright (c) 2002-2015, Aruba Networks, Inc.
Compiled on 2015-09-23 at 00:58:16 PDT (build 51810) by p4build

AP uptime is 5 minutes 7 seconds
Reboot Time and Cause: AP rebooted Thu Oct 29 13:34:33 PDT 2015; Gateway unreachable

 

Aruba Operating System Software.
ArubaOS (MODEL: 205), Version 6.4.2.6-4.1.1.10
Website: http://www.arubanetworks.com
Copyright (c) 2002-2015, Aruba Networks, Inc.
Compiled on 2015-09-23 at 00:11:51 PDT (build 51810) by p4build

AP uptime is 7 minutes 7 seconds
Reboot Time and Cause: unknown

 

MS-Wrestling-105# sho ver
Aruba Operating System Software.
ArubaOS (MODEL: 105), Version 6.4.2.6-4.1.1.10
Website: http://www.arubanetworks.com
Copyright (c) 2002-2015, Aruba Networks, Inc.
Compiled on 2015-09-23 at 00:58:16 PDT (build 51810) by p4build

AP uptime is 3 hours 20 minutes 41 seconds
Reboot Time and Cause: AP rebooted Thu Oct 29 10:20:06 PDT 2015; Image Upgrade S uccessful

 

MS-Staff2-105# sho ver
Aruba Operating System Software.
ArubaOS (MODEL: 105), Version 6.4.2.6-4.1.1.10
Website: http://www.arubanetworks.com
Copyright (c) 2002-2015, Aruba Networks, Inc.
Compiled on 2015-09-23 at 00:58:16 PDT (build 51810) by p4build

AP uptime is 7 minutes 23 seconds
Reboot Time and Cause: AP rebooted Thu Oct 29 13:34:21 PDT 2015; Gateway unreachable

 

 

 

 

For the unknown, I would check whatever is powering the access points to see if the power is being interrupted.  For the gateway unreachable, I would check to make sure that there are no duplicate ip addresses in your network.

 

Those are two general things.  The specifics on how you would check that depends on how your network is setup and what you are using for DHCP.

 

Power is not being interrupted by anything other that us trying to reboot the AP's to get them to work. They are all on Cisco POE catalyst switches. I intentionally gave configs for access points that are all on different switches too, and the IP addresses are reserved and set out of the DHCP scope. 

Okay.  The unknown is probably you cutting the power to reboot.  That leaves the gateway unreachable.  How many APs do you have in each cluster?  Are there any different clusters that share a subnet? (two clusters to a subnet)

This particular cluster has 12 IAP's. One has 10, another has 9. No less than 8 and no more than 12. All of the different clusters are on different subnets. 

So how long has this been happening? Did you deploy clusters successfully and then this started happening?

This has been going on for a week now and getting progressively worse. This system has been in place for about a year, and this is the first time we have experienced anything like this. This have been working great, and we have been extremely satisfied on the performance of the access points... until last Thursday when this started happening for no apparent reason. We have upgraded firmware, rolled back firmware, all kinds of different things to no avail. We took all of the access points down last night and brought them back up one by one. They held solidly for around 12 hours, and at 7:41 this morning, all of the symptoms started coming back. Without anyone touching anything, I was the only one in the office when it started happening. 

 

Around every half hour, (all on the same version of code) the master will reboot. Just the master, then all of the PAPI messages will start. We started syslogging for troubleshooting. The access points completely freak out, and all without prompting from anyone. 

What shares the management subnet with the access points at that location?  If the default gateway is being spoofed or taken over, you might want to take a look at other devices on that same subnet to see if they are having issues, as well.

 

Again, this is your infrastructure, so I can only guess what is happening.

Dear Team,

 

I read all the above messages, i am getting the same issue however my deployment is only based on three IAP-105. It start rebooting it again and again after some time or 10hours but it rebooted at the end. 

 

We have assigned the static IPs for the APs and gateway and power is solid, other IP phones are also connected on the same switch so power is not the issue.

 

Please advise,

I would type "show log system" to see if that tells you what happened. Hopefully the cabling to your access points is good, and there are no errors on your switch ports.

Yes, there are no errors on the switch port but still rebooted, please is the logs.

 

Jan 1 03:00:44 syslog: <341005> <ERRS> |AP ITCCAP1-TO: | terminal access is enabled but dropbear is not running, starting
Jan 1 03:00:49 nanny[1528]: <303022> <WARN> |AP ITCCAP1-TO: | Reboot Reason: AP rebooted Thu Feb 4 16:15:24 UTC 2016; master transitioned to local (new master 192.168.131.243, uplink flaps 0, max beacon miss 37)
Feb 4 16:17:12 cli[1604]: <341317> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| Regulatory table file init at version 1.0_50775, build Jul 9,2015.
Feb 4 16:17:12 cli[1604]: <341004> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| Extended ssid enabled
Feb 4 16:17:12 cli[1604]: <341207> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| AP support up to 8 SSID.
Feb 4 16:17:13 cli[1604]: <341172> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| Find enet0 name bond0.
Feb 4 16:17:13 cli[1604]: <341004> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| cli starting, clearing any dpimgr
Feb 4 16:17:13 cli[1604]: <341004> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| cli_choose_eth_uplink:eth current uplink set to DHCP
Feb 4 16:17:13 cli[1604]: <341169> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| Add uplink Ethernet 0.
Feb 4 16:17:13 cli[1604]: <341167> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| Uplink bond0 type Ethernet, state INIT->LOAD.
Feb 4 16:17:13 cli[1604]: <341169> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| Add uplink Wifi-sta 6.
Feb 4 16:17:13 cli[1604]: <341169> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| Add uplink 3G/4G 7.
Feb 4 16:17:13 cli[1604]: <341167> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| Uplink type Dummy, state INIT->LOAD.
Feb 4 16:17:13 cli[1604]: <341004> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| receive sapd hello for process recovery
Feb 4 16:17:14 cli[1604]: <341174> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| No current uplink, pick the highest one - Ethernet bond0.
Feb 4 16:17:14 cli[1604]: <341263> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| enable uplink bond0.
Feb 4 16:17:14 cli[1604]: <341167> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| Uplink bond0 type Ethernet, state LOAD->PROBE.
Feb 4 16:17:14 cli[1604]: <341265> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| enable ethernet uplink bond0.
Feb 4 16:17:14 cli[1604]: <341004> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| the ethernet uplink inuse in first time, get ip address.
Feb 4 16:17:14 cli[1604]: <341167> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| Uplink bond0 type Ethernet, state PROBE->UP.
Feb 4 16:17:14 cli[1604]: <341185> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| Retrieving ip address from br0, ip 192.168.131.244, mask 255.255.255.0.
Feb 4 16:17:14 cli[1604]: <341274> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| Update election ip from br0, election ip 192.168.131.244/255.255.255.0.
Feb 4 16:17:14 cli[1604]: <341004> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| build_my_ip_address: old ip and new ip same; skipping
Feb 4 16:17:14 cli[1604]: <341006> <CRIT> |AP ITCCAP1-TO@192.168.131.244 cli| Sending uplink_up msg to sapd
Feb 4 16:17:14 cli[1604]: <341006> <CRIT> |AP ITCCAP1-TO@192.168.131.244 cli| trap uplink info.
Feb 4 16:17:14 cli[1604]: <341004> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| last uplink is null, it's the first uplink, ignore this trap.
Feb 4 16:17:14 cli[1604]: <341173> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| Current uplink set to Ethernet, state UP.
Feb 4 16:17:14 cli[1604]: <341135> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| Master Changed - new 192.168.131.243 old 0.0.0.0 current swarm state 0.
Feb 4 16:17:15 cli[1604]: <341135> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| Master Changed - new 192.168.131.243 old 0.0.0.0 current swarm state 1.
Feb 4 16:17:18 cli[1604]: <341004> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| Swarm State Change from Found to Image_ok
Feb 4 16:17:26 cli[1604]: <341307> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| recv_config_init: receive config from master 192.168.131.243, using url 0, auto save disable 0.

Feb 4 16:17:26 cli[1604]: <341004> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| Extended ssid enabled
Feb 4 16:17:26 cli[1604]: <341207> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| AP support up to 8 SSID.
Feb 4 16:17:27 cli[1604]: <341172> <WARN> |AP ITCCAP1-TO@192.168.131.244 cli| Find enet0 name bond0.

 

 

We are using the below AP version.

 

ArubaOS (MODEL: 105), Version 6.4.2.6-4.1.1.8

Do you have a device set to the preferred master? It seems like this switch was the master, but another device, the .243 all of a sudden said that it was the master of the VC. We should get the system lig from that device..

Dear,

 

Thank you for your kind support, i make .244 as prefered master. before i check the logs it was rebooted due to master to local transition. I hope after prefered master it would not be rebooted again and again.

 

Also i would be so obliged if you could check that the installed code is ok?

You can download the .11 update and there are a few fixes if you are willing to try.

Dear,

 

I just checked the release note for .11, it didnt mentioned any issue i am faceing right now. I am hopping that prefered master will solve my problem if not then i can check other OS.

Dears,

 

So far my all IAP's are up from last 24 hours after setting the prefered master. I hope this is the solution.

Regards,

 

 

Dear rkeeler,

 

You mean that i should use different vlan for managment the APs? as of now all three aps are in same VLAN.? This is what you mean?

What we ended up doing was separating our vlans so they couldn't talk to
each other across schools. We also managed to get a trial of Airwave so we
could isolate the problem. I'd recommend that as well.

--
*Regards,*

*Robyn Keeler*

*Network Systems Specialist*
*rkeeler@mlsd.org *
*(509) 565-3127*