Wireless Access

We have a hotel here with two Aruba WiFi controllers and 200+ access points. Whenever an AP gets restarted - no matter the reason - there is about 10% chance it will not appear on the controller again. Since all APs are powered by PoE from HP access switches, it's relatively easy to power-cycle them remotely. After that some APs make the connection again. Others are more resistant and does not "wake up" no matter how many times they are restarted. But after I leave them alone for couple days they somehow magically make the connection again.

During this "down" period they can be pinged from the controller and shows up in the ARP table with correct MAC address. However no WiFi signals are transmitted - which kind of makes sense, but creates a problem of "holes" in signal coverage.

Logs on the controller are full of all kinds of warning messages, but nothing seems to be related to "unable establish a connection" or anything like that. (Or I just don't know what to look for.)

Last week there was some kind of software upgrade or something and AP's started to restart on their own. After a few hours of isolating the problematic APs and restarting them, we were left with about 25 APs in this diminished working state. We tried a configuration reset on a couple of them - that worked. I was able to put them back in the correct group and after that they worked fine. However I'd say that it can be used only as last resort since some APs are tucked in so deep into the ceiling that reaching one would take hours. And if it's in a populated guest room - it's just plain not possible.

I've searched the internet as well as any articles/discussions here for couple months now with no luck.

Recently I just found the option to enable telnet on the APs, but I haven't had a chance to find an AP which has received this new config and can't connect to the controller.

- Is this a known problem somewhere? Or perhaps an expected behaviour?

- What commands can I run on an AP to diagnose this? (Most of what I've tried so far returns "permission denied".)

Controllers are Aruba7210 with software version 6.4.3.4.

APs are all model 205.

"permission denied" is when you have a limited access account to the controller (network admin vs. root).  You should try to get a root account so that you can run the full range of commands.

What is the discovery mechanism that the APs use to find the controller?  Also, do both controllers have the same version of code?  (type "show version" on each to find out).  This is important because APs could be caught in an endless upgrade/downgrade cycle if not.

If you have a support account you should open a TAC case in parallel to this post to get the best help possible.  Again, you most likely will need a root account to the controllers to get to the bottom of all of this...

You can type "whoami" at the commandline to determine what your role is:

(Aruba7640-US) #whoami
user admin - role root 

I get "permission denied" on an AP (via telnet) not on the controller. Controller CLI access does not have any issues.

Sorry - forgot to mention the discovery. About half of all APs use DHCP with ADP discovery and the other half uses static IPs with DNS based discovery. About 20 "disappeared" APs was on the static/DNS side, but DHCP/ADP also had a few so I would guess that discovery protocol is not at fault here.

Unfortunately I don't have a support account and legally there are a bunch of companies between myself and HP which has resold these APs and controllers further that it will be a while until I dig through them and get a proper support access.

Both controllers are running the same ArubaOS 6.4.3.4 with the same build 51619 on both partitions.

Okay.   By default telnet is not turned on in the APs, but it is not very useful, so don't worry about that.

If I had to guess, I would say that your problem is your DHCP server might be giving out ip addresses that statically addressed access points or other devices already have, and that is creating your communications issue.  You should choose static or DHCP with DHCP being the preferred mechanism that has much less administrative overhead.  I would type "show ap database" on the commandline of the controller to get a full list of all of your APs and corresponding ip addresses.  I would then go down the line and type:

show ap provisioning ip-addr <ip adddres of ap> | include IP

Do this for each access point to determine which are statically addressed and which are not.  The strategy would be to find out what APs are statically addressed and then reprovision them as DHCP.  Statically addressed APs are harder to troubleshoot and not inflexibile if you wanted to change the underlying network for any reason.

Again, this is all assuming your problem is rooted in mixing DHCP and statically addressed APS.

Thank you for the idea - I hadn't thought of that before.

At the moment I have one AP in this diminished state, so I did the usual ping:

Then I switched off PoE going to that port, and:

So apparently there are no other devices with this IP address on the network. I even tried pinging from a few other devices and all show either the same MAC address or no MAC address at all (that was after I switched off the AP).

And a few minutes after I switched the AP back on:

I'd say that this is pretty much what I expected because I have been careful with static IPs and DHCP ranges and am sure that they do not overlap. There are some devices on the network that are controlled by other companies so in theory it's possible that they might commandeer some IPs for some reason, but as we can see above - this is not the case.

Half of the APs have static IPs and other half are working with DHCP is because I'm in process of reconfiguring them to static version (which also presents the problem of disappearing APs after reboot) because hotel has a requirement for APs to be statically addressed.

Would be easier to monitor them with external monitoring systems as well.. but that's beside the point.

 Any other thoughts?

For an AP that is exhibiting that issue, type "show datapath session table <ip address of ap>" to see what traffic it is sending the controller.  Do you know if you have the controllers configured as master/backup master or master/local?  Do you know what controller APS should be on when they are in operation?

You can type "show log system 50" to possibly get an idea what is happening with your access points.

When AP 

When AP is not connecting to the controller, it's datapath session table is empty. Today 10 APs disappeared from the controller and every single one shows nothing in it.

Controllers are tied together in a cluster and all APs should connect only to the primary.

As I wrote in the first post - I've read the logs quite thoroughly but have not found anything that could be related to this..

I tried to peruse the logs filtering by AP name, IP address and MAC address and this is what I got:

(Aruba7210pri) #show log all | include AP-R0.3-39R

Aug 28 09:00:58  authmgr[3899]: <132093> <ERRS> |authmgr|  WPA2 Key message 2 from Station a0:cb:fd:d1:03:ce 84:d4:7e:1f:17:22 AP-R0.3-39R did not match the replay counter 01 vs 03
Aug 28 09:00:58  authmgr[3899]: <132093> <ERRS> |authmgr|  WPA2 Key message 2 from Station a0:cb:fd:d1:03:ce 84:d4:7e:1f:17:22 AP-R0.3-39R did not match the replay counter 01 vs 03
Aug 28 09:00:58  authmgr[3899]: <132093> <ERRS> |authmgr|  WPA2 Key message 2 from Station a0:cb:fd:d1:03:ce 84:d4:7e:1f:17:22 AP-R0.3-39R did not match the replay counter 03 vs 04
Aug 28 09:00:58  authmgr[3899]: <132093> <ERRS> |authmgr|  WPA2 Key message 2 from Station a0:cb:fd:d1:03:ce 84:d4:7e:1f:17:22 AP-R0.3-39R did not match the replay counter 03 vs 04
Aug 29 12:47:31  sapd[925]: <404069> <WARN> |AP AP-R0.3-39R@10.155.4.19 sapd|  AM 84:d4:7e:1f:17:20: ARM Channel Interference Trigger new 13-60 old 3-89 new_rra 13/7 TCI 10
Aug 29 12:53:46  sapd[925]: <404070> <WARN> |AP AP-R0.3-39R@10.155.4.19 sapd|  AM 84:d4:7e:1f:17:20: ARM Empty Channel Trigger new 10-0/0 old 13-27/1 new_rra 10/7

(Aruba7210pri) #show log all | include 10.155.4.19

Aug 21 11:28:14  nanny[981]: <303086> <ERRS> |AP 84:d4:7e:c9:f2:4a@10.155.4.197 nanny| Process Manager (nanny) shutting down - AP will reboot!
Aug 21 11:31:07  nanny[1217]: <303086> <ERRS> |AP 84:d4:7e:c9:f2:4a@10.155.4.197 nanny| Process Manager (nanny) shutting down - AP will reboot!
Aug 21 11:42:55  nanny[981]: <303086> <ERRS> |AP 84:d4:7e:c9:f2:44@10.155.4.190 nanny| Process Manager (nanny) shutting down - AP will reboot!
Aug 21 11:46:09  nanny[1386]: <303086> <ERRS> |AP 84:d4:7e:c9:f2:44@10.155.4.190 nanny| Process Manager (nanny) shutting down - AP will reboot!
Aug 29 12:47:31  sapd[925]: <404069> <WARN> |AP AP-R0.3-39R@10.155.4.19 sapd|  AM 84:d4:7e:1f:17:20: ARM Channel Interference Trigger new 13-60 old 3-89 new_rra 13/7 TCI 10
Aug 29 12:53:46  sapd[925]: <404070> <WARN> |AP AP-R0.3-39R@10.155.4.19 sapd|  AM 84:d4:7e:1f:17:20: ARM Empty Channel Trigger new 10-0/0 old 13-27/1 new_rra 10/7

(Aruba7210pri) #show log all | include 84:d4:7e:c9:f1:72

(Aruba7210pri) #

(This is one of the APs that disappeared today.)

So I am still baffled. AP seems to disappear without a single message. Or could those channel messages have something to do with it?

The reason for its reboot is also unknown. Could there be some kind of automatic updates going on?

I've tried to connect to APs that disconnected today via telnet, but there I also get 'Connection refused'. Which is weird, because I'm sure that those APs have received the new config where telnet is enabled.

"Last week there was some kind of software upgrade or something and AP's started to restart on their own. After a few hours of isolating the problematic APs and restarting them, we were left with about 25 APs in this diminished working state. We tried a configuration reset on a couple of them - that worked. I was able to put them back in the correct group and after that they worked fine. However I'd say that it can be used only as last resort since some APs are tucked in so deep into the ceiling that reaching one would take hours. And if it's in a populated guest room - it's just plain not possible."

Who upgraded the software and why?

The bit about software upgrade was just a guess (hence the "or something" part) - I actually have no idea what happened there. There are no log entries about it so I have no way to know for sure.

Nobody has upgraded the controller - I'm sure of that. However I have no idea how is AP software upgrade handled as I have not found any options about that on the controller.

Please PM me your email address.