Wireless Access

We have some APs that were switched to backup controller(switch in their terminology) and refuse to go back to LMS.

We rebooted them several times, but now luck.

What would be the known reasons for that?

What commands we can run to troubleshoot this problem?

There's a couple of things you need to check :

- can you ping the other controller from the AP console ?
- do you have enough licenses ?
- Do you CPSec enabled ?
- did you write mem or save config on the primary controller

Sent from Outlook for iPhone

> can you ping the other controller from the AP console ?

I am not able to test this, because everything is remote for me and I access the network through the VPN

What I can see that I can ping AP from the LMS itself.

 

> do you have enough licenses ?

#show license limits
License Limits
--------------
Limit Value
----- -----
24 Access Points
24 RF Protect
0 xSec Module
0 120abg Upgrade
0 121abg Upgrade
0 124abg Upgrade
0 125abg Upgrade
24 Next Generation Policy Enforcement Firewall Module
0 Advanced Cryptography
0 Service provider AP
 

 

All 24 AP are licensed. 15 APs are connected to the LMS, 9 APs are connected to the backup controller.

 

>- Do you CPSec enabled ?

  #show running-config | include cpsec
Building Configuration...
no cpsec-enable

 

> did you write mem or save config on the primary controller

 

Not sure how it related to my problem...
 

 

 

On the controller where the AP is now do a "show ap config ap-name xxxxx"  See it it has listed as the primary and backup controller.   Also do both controllers have the same AOS?  I had issues once caused by mismatched AOS versions.

I run ""show ap config ap-name xxxxx"  on both (LMS and Backup LMS controller) for one of the AP which is switched to backup LMS and one which is on primary LMS, configuration is identical (I have removed real IPs and changed profiel name for security reasons):

 

....
Parameter 802.11g 802.11a Source
--------- ------- ------- ------
.....
LMS IP "IP1" "IP1" ap system-profile "Profile1"
Backup "IP2" "IP2" ap system-profile "Profile1"
LMS IPv6 N/A N/A ap system-profile "Profile1"
Backup LMS IPv6 N/A N/A ap system-profile "Profile1"
LMS Preemption Enabled Enabled ap system-profile "Profile1"
LMS Hold-down Period 600 sec 600 sec ap system-profile "Profile1"
LMS ping interval 20 20 ap system-profile "Profile1"
....

AOS is the same on both controllers, bootstrap is different but it never were the problem before.

Also didn't mentioned that Backup LMS is the Master controller for all LMS controllers, but I am not sure it is relevant.

Backup LMS is:

 

ArubaOS (MODEL: Aruba3600), Version 6.4.2.4
Website: http://www.arubanetworks.com
Copyright (c) 2002-2015, Aruba Networks, Inc.
Compiled on 2015-01-15 at 18:50:32 PST (build 48122) by p4build

ROM: System Bootstrap, Version CPBoot 1.3.0.1 (build 28907)

LMS is

ArubaOS (MODEL: Aruba3400), Version 6.4.2.4
Website: http://www.arubanetworks.com
Copyright (c) 2002-2015, Aruba Networks, Inc.
Compiled on 2015-01-15 at 18:50:32 PST (build 48122) by p4build

ROM: System Bootstrap, Version CPBoot 1.3.0.3 (build 34552)

 

Just throwing out ideas but you should be able to manually reprovision the APs to the correct controller from the backup controller.  If the system profiles are correct then how about DHCP options, or ADP turned on? 

 

Are the APs on the same subnet as the backup controller?  Can you share the matching system profile settings for us?

I tried one, doesn't look it helps. 

All APs are receiving IP address from DHCP. ADP is on.

Subnet for AP and LMS is the same.

SUbnets for LMS  and backup LMS are different.

Sorry, what do you mean by "matching system profile settings"? All configuration is identical for all APs. Also everything was working before fine.

The only difference is that 9 APs that are in question sit on the different physical switch. Another 15 APs that are working fine are sitting on another physical switch. 

When LMS losts powere or  rebooted all APs are switching to Backup LMS. In this case we reboot the switch where APs are connected, AP are losing power, reboot and go back to LMS. Those 9 APs do not want to switch back, in spite of the fact we rebooted their switch numerous times already.

I have a feeling that all this is related to the switch where those APs are connected. I can login to the switch and ping LMS and Backup LMS . 

Still lost where to look ...

OK so you are saying that all 24 APs are in the same AP group which means they have the same system profile and 15 are working as expect but the 9 that are not are on a different switch AND that even though they are on a different switch all 24 are on the same subnet as the LMS?

 

If by chance the second switch has a different subnet do extended pings from the vlan source IP instead of a standard ping which would likely use the connected interface as the source for the ping.  "ping IP" enter and follow the prompts choose "yes" for extended pings and enter the default gateway IP of the APs for the source.

 

Also are the 9 APs on the backup funtioning correctly and in the same AP group as the 15 on the LMS?  Any flags on the backup controller for these APs when you do a "show ap database long"

 

1) All 24 access points are in the same AP Group and have identical configuration.

2) All 24 access points are on the same subnet and have the same default gateway

The only difference is the physical connection (15 AP on one switch, 9 ap on another switch).

 

I run "show ap database long" on the backup controller (which is the  master controller). Nine access points that are switched to backup controller are showing "D" flag. 

Below is the result for two APs (one is switched and one is not)

AP Database
-----------
Name               Group         AP Type  IP Address      Status              Flags  Switch IP       Standby IP  Wired MAC Address  Serial #   Port  FQLN  Outer IP  User
----               -----         -------  ----------      ------              -----  ---------       ----------  -----------------  --------   ----  ----  --------  ----
...
6c:f3:7f:ca:30:b0  Group1           105      AP-IP-Address1  Up 9h:31m:11s       D      BACKUP-LMS-IP-ADDRESS      0.0.0.0     6c:f3:7f:ca:30:b0  BT0251796  N/A   N/A   N/A
6c:f3:7f:ca:30:b5  Group1           105      AP-IP-Address2  Up 11h:35m:27s             LMS-IP-ADDRESS  0.0.0.0     6c:f3:7f:ca:30:b5  BT0251801        N/A   N/A
....

Also, you can note that uptime for the AP which is switched is less, because we were rebooting it, trying to fix the issue.

 

I would imagine that 9 APs that are not working are not seeing the LMS correctly, but I can ping LMS from their switch no problem. Unfortunately, I do not see any troubleshooting opportunities "fom inside "the AP itself.

 

On the AP with the issue, please execute "show ap debug-log ap-name <name of ap>" to see how it came up....

If it were me and I was seeing the D flag I would try reprovisioning the APs on the backup to the LMS controller.  May not work but I would start there because it will allow the AP to redownload the configs.  Lots of reasons for a D flag.  I am guessing your APs got partitial config download somehow.  I can understand wanting to find the cause but maybe reprovision one AP on that switch and then test it to see if it fails properly after the reprovision.  If it does just do the others and be done with it.

 

 

 

 I have tried to re-provison 1 AP. May be I didn't do it properly, but I haven't noticed any changes on that AP. 

 

show ap debug-log ap-name.. doesn't show any relevant information (1 old warning from December last year).

 

Probably shouldn't mention this because it is a little off topic but I have had dirty flags where the only thing that would resolve them (even with TAC engaged) was converting the APs to RAPs.  Basically the MTU is handled differently from a Campus AP with a GRE tunnel versus an IPSEC tunnel in a RAP.  I doubt you have this issue mine was over LAN to LAN VPN connection.  Any chance your backup is located across a VPN tunnel?

 

 

Backup controller is located over MPLS or VPN network. But what difference doest it make? APs are not able to see LMS which is on the same network with them, but switch to Backup with not problem (which is on different physical network and subnet).

I was just throwing it out there for discusion based on a past issue I had.  Because you have the dirty D flag and they are stuck on the controller across the "MPLS or VPN".  If it is a VPN connection specifically your AP could have gotten dirty config due to MTU issues (maybe).

 

Earlier you said you tried to reprovision them from the backup controller.  I am guessing that they did not reboot.  If this is the case you are going to need to console into them.  From there purge them and start over.  IF you still get a D flag then I bet converting them to a RAP will fix your issue.

I went to Configuration-AP Installation-

 

chose one of the AP and provisioned it as Remote AP.

 

I did not notice anything happened and AP uptime is still the same.

 

 

I have a guide we created on how to do this it is just a few steps but you have to have console access to the AP to convert it.  I will send you the guide if you want it I do not want to post it here.  send me a mail.  alan.scott@fluor.com

 

I assume you are working with TAC.  You likely have a different issue from me but it doesnt hurt to learn these steps just in case you need them one day.  regardless it sounds like you are going to need console access to the APs one way or another as they seem to not be responding to request for reboots.

 

 

Alan,

 

Thanks for instruction. I do not see the point to convert APs to Remote, because they are sitting in the same subnet with their LMS and there is no WAN between AP and Master Controller.

Why make things more complicated than they are?

Well in my case the APs were remote to the controller and it is best practice to deploy APs as RAPs if they are not on the local LAN. I have lots of APs configured as CAPs that are remote to the controller and in my case I ran into an issue at one site were CAPs stopped working with the D flag and TAC could not resolve. They recommended we convert to RAPs (best practice anyways) and it worked.   For you I think you just need to console in to the AP purge it and start fresh as the APs got a partial config download (my guess).

I have read about D flag. AP is not able to communicate with controller is most reason for that. Also our APs are local to their LMS they using some WAN infrastructure to access MAster Controller (MPLS or whatever). The whole point to make them Remote is to create a tunnel for  communication messages with Master which doesn't affected by the MTU size on all these middle routers.Which makes sense.

What I do not understand why AP can stuck in this "D" condition even if you reboot it.  Why reboot is not whiping out all configuration?

Finally, we had to physically disconnect all problem APs, reset them with the button and reconnect and provision one by one.

Physical reset had fixed this issue

Also I checked MTU on both switches. The same 1514 bytes...