ArubaOS and Controllers

Reply
Occasional Contributor II
Posts: 44
Registered: ‎04-02-2007

APs failing onto master instead of backup-lms?

Hi all,

We've got a set of A6000's acting as locals, with a 3200 as master. The A6000's have about 200 APs each; the 3200 is licensed for 8 APs for the purpose of provisioning. We're running ArubaOS 5.0.3.0. The APs are provisioned with a master-ip of the master controller, server-ip of the local controller, and have an AP system profile with an LMS IP of their primary local, and a backup-LMS IP of their secondary local (another of the A6000s).

We recently experienced an LMS failure and saw some interesting behavior. Approximately 25% of the APs on the failed LMS did not connect to their backup. Instead, they connected to the master, coming up in an IL state since it's got insufficient licenses. The master doesn't even have our client VLANs terminating on it, so APs on it are a no-no.

I figured this was a thundering-herd issue, and suspected these APs might have connected to the master while waiting for their holddown period again. I waited 11 minutes (our holddown is 600 seconds), but they stayed on the master. I gave them another 5 minutes and they still remained on the master.

At that point, I issued an "apboot all local" on the master, and about 20% of the affected APs failed to their backup LMS. The remaining 80% came back up on the master again. I had to issue this command about 5 times before they'd all failed over.

This week, we replaced the failed LMS and failed back. The same thing happened.

Issuing a show ap debug system-status on an affected ap shows the following (keep in mind we're failing back in this instance):

LMS Information
---------------
Item Value
---- -----
Primary LMS
Backup LMS
Using Primary
Preemption Disabled
VRRP No


Other APs, which failed over correctly, show:

LMS Information
---------------
Item Value
---- -----
Primary LMS
Backup LMS
Using Primary
Preemption Enabled
Hold-down period 600
VRRP No

I've got a TAC case open on this, but I thought I'd ask the community if they've seen anything similar while I wait.
Guru Elite
Posts: 20,761
Registered: ‎03-29-2007

Re: APs failing onto master instead of backup-lms?

what does the AP use to initially discover the master controller, DNS, DHCP option, or do you have it hardcoded into the AP?


Colin Joseph
Aruba Customer Engineering

Looking for an Answer? Search the Community Knowledge Base Here: Community Knowledge Base

Occasional Contributor II
Posts: 44
Registered: ‎04-02-2007

Re: APs failing onto master instead of backup-lms?




Master controller is hardcoded into the AP by ip address.

Guru Elite
Posts: 20,761
Registered: ‎03-29-2007

Re: APs failing onto master instead of backup-lms?

You have quite a few variables in your setup, currently, not limited to putting both a masterip and a server ip in both, not enough licenses and failover between three controllers. With limited information, it could take some time to unravel here, and support will probably have it resolved by then.

Let me state some ideas, instead to see if we can shed some light on this based on what you mentioned below

- You should use some form of discovery for your access points like a DNS A-record with the size of your deployment, because it is more flexible and easier to change if you had to re-ip address your controller.
- If you do NOT use discovery, you should ONLY have a master-ip address defined in your access point (not a server-ip, as well).
- What AP-groups do you have the AP System profile applied to, and how many ap-groups do you have. How many access points are in each?
- An access point on a controller that does not have enough licensing will not send it to its backup lms. Please contact your local sales team if you want to request a feature to do this
- You should only engage preemption if you have your primary fault tolerance tested, because that will just add another variable.
- What is the capacity of the master, and the two locals and how many access points do you have in total?
- What version of code is this?


Colin Joseph
Aruba Customer Engineering

Looking for an Answer? Search the Community Knowledge Base Here: Community Knowledge Base

Occasional Contributor II
Posts: 44
Registered: ‎04-02-2007

Re: APs failing onto master instead of backup-lms?

Colin,

Thanks for the valuable input. Here are some answers.




We do have the A-record in place. It's my understanding that the APs will use DNS if their hardcoded master is unreachable. My way of thinking is that we prefer the master to be predictable (and the APs not dependent on DNS), but want the capability to transition master IPs via DNS if absolutely necessary. Does that hold water?




This is good to know. We've done the server-ip since the inception of our original network, presumably at the behest of our SE at the time. That said, we were running 2.4 back then, I know a lot has changed.




We have 12 AP groups. 9 of them are building-specific, the rest are generic per-controller groups. There are only three AP system-profiles, identical except for the LMS IPs, and they are applied to the appropriate groups. The affected APs are in multiple groups, including the generic per-controller groups. Other APs in the same groups are not exhibiting this behavior.



We tried failovers during rollout of these controllers and they worked as intended, at which point we enabled preemption. That was a year ago, though.


What is the capacity of the master, and the two locals and how many access points do you have in total?
- What version of code is this?



Master is a 3200 with 8 AP capacity. Three locals, all M3's, 512 AP capacity each. We have 447 APs deployed, more or less evenly across the locals. Code is 5.0.3.0 all around.
Guru Elite
Posts: 20,761
Registered: ‎03-29-2007

Re: APs failing onto master instead of backup-lms?

If there is a configured master ip, no DNS discovery will take place, at all.

The most economical way to do this (and the way we know this works is):

Leave the master discovery DNS name to the default "aruba-master"

Make sure Aruba-Master points to the master controller

In the AP system profile the lms-ip should be the primary local and the backup lms-ip should be the secondary local.

In this scenario you should not use preemption. You should just let the access points reboot if they cannot find the second controller and everything starts from scratch and is more deterministic.


Colin Joseph
Aruba Customer Engineering

Looking for an Answer? Search the Community Knowledge Base Here: Community Knowledge Base

Occasional Contributor II
Posts: 44
Registered: ‎04-02-2007

Re: APs failing onto master instead of backup-lms?

Thanks Colin. We'll implement those changes at the next opportunity, good to know the BCP. Do you know if that's documented somewhere I could've looked?
Guru Elite
Posts: 20,761
Registered: ‎03-29-2007

Re: APs failing onto master instead of backup-lms?

I found this on page 57 of the ArubaOS 6 user guide. It might provide some perspective. The lines that start with "If the master provisioning parameter is not set" means that these methods cannot be used with the master ip address or DNS name configured:

Locating the Controller
An AP can discover the IP address of the controller in the following ways:
␣ From a DNS server
␣ From a DHCP server
␣ Using the Aruba Discovery Protocol(ADP)

At boot time, the AP builds a list of controller IP addresses and then tries these addresses in order until a controller is reached successfully. The list of controller addresses is constructed as follows:

1. If the master provisioning parameter is set to a DNS name, that name is resolved and all resulting addresses are put on the list. If master is set to an IP address, that address is put on the list.
2. If the master provisioning parameter is not set and a controller address was received in DHCP Option 43, that address is put on the list.
3. If the master provisioning parameter is not set and no address was received via DHCP option 43, ADP is used to discover a controller address and that address is put on the list.
4. Controller addresses derived from the server-name and server-ip provisioning parameters and the default controller name aruba-master are added to the list. Note that if a DNS name resolves to multiple addresses, all addresses are added to the list.

This list of controller IP addresses provides an enhanced redundancy scheme for controllers that are located in multiple data centers separated across Layer-3 networks.
From a DNS Server APs are factory-configured to use the host name aruba-master for the master controller. For the DNS server to resolve this host name to the IP address of the master controller, you must configure an entry on the DNS server for the name aruba-master.

For information on how to configure a host name entry on the DNS server, refer to the vendor documentation for your server.

Aruba recommends using a DNS server to provide APs with the IP address of the master controller because it involves minimal changes to the network and provides the greatest flexibility in the placement of APs.

When using DNS, the AP can learn multiple IP addresses to associate with a controller. If the primary controller is unavailable or does not respond, the AP continues through the list of learned IP addresses until it establishes a connection with an available controller. This takes approximately 3.5 minutes per LMS.

From a DHCP Server
You can configure a DHCP server to provide the master controller’s IP address. You must configure the DHCP server to send the controller’s IP address using the DHCP vendor-specific attribute option 43. APs identify themselves with a vendor class identifier set to Aruba AP in their DHCP request. When the DHCP server responds to the request, it will send the controller’s IP address as the value of option 43.
When using DHCP option 43, the AP accepts only one IP address. If the IP address of the controller provided by DHCP is not available, the AP can use the other IP addresses provisioned or learned by DNS to establish a connection.


Colin Joseph
Aruba Customer Engineering

Looking for an Answer? Search the Community Knowledge Base Here: Community Knowledge Base

Search Airheads
Showing results for 
Search instead for 
Did you mean: