Wireless Access

At what point does the "full config" of managed devices get updated or written to the mobility master? While troubleshooting an issue last week with CCM, noticed the full-folder had the configuration files along with Config-ID - and the controllers IDs were older. This ended up causing an issue during the L3 MM fail-over where the managed devices reverted back to older versions of our validuser acl. TAC case is currently open.

n=md=Campus=<local-controller-1-395>.cfg

n=md=Campus=<local-controller-2-302>.cfg

n=md=Campus-405.cfg

All you had to say is L3 MM failover.  I am not sure that is supported as of yet.

EDIT: Yes it is supported.

L3 MM failover is supported. If a change was made between the replication window of the L3 MMs (this is configured or configurable) then the two MMs could have different config IDs. 

Anytime the MM makes a config change and then a write mem is ussued, then a new config ID is generated and the diff pushed to the controllers. I don't know if there is a set specific window where the CFGs are merged. So long as the config ID is the same for the active MM and the MDs, that's what matters. If your L3 MMs are not syncing correctly, or the sync is disabled, please open a TAC case and have them check it. But the smallest window for L3 MM sync is two hours. SO there IS a rare chance that if you have an active MM and just pushed a config change and then lost the MM, that the MDs would revert to the L3 MM after 15min and may pull an older ocnfig. 

@cjoseph wrote:

All you had to say is L3 MM failover.  I am not sure that is supported as of yet.

EDIT: Yes it is supported.

@cjoseph, funny thing. You helped answer one of the the 3 issues we ran into during the L3 MM Fail-Over when you prompted that support question. :-) I went looking for one of the original L3 MM posts for support verification, and remembered the lack of "configuration ability" from the standby L3 MM was by design - https://community.arubanetworks.com/t5/Wireless-Access/Layer-3-Redundancy-for-Mobility-Master-on-8-2/m-p/313229#M75784 -- from jhoward - "There will not be any configuration abaility (ability to manage configurations of the MCs) *unless* you manually promote the Standby MM to Primary. This is to prevent split brain issues and you would only promote the standby if you know the primary MM is down for good or for awhile."

@jhoward wrote:

L3 MM failover is supported. If a change was made between the replication window of the L3 MMs (this is configured or configurable) then the two MMs could have different config IDs. 

Anytime the MM makes a config change and then a write mem is ussued, then a new config ID is generated and the diff pushed to the controllers. I don't know if there is a set specific window where the CFGs are merged. So long as the config ID is the same for the active MM and the MDs, that's what matters. If your L3 MMs are not syncing correctly, or the sync is disabled, please open a TAC case and have them check it. But the smallest window for L3 MM sync is two hours. SO there IS a rare chance that if you have an active MM and just pushed a config change and then lost the MM, that the MDs would revert to the L3 MM after 15min and may pull an older ocnfig. 

 

@ @jhoward, thanks for the additional information (and the other post about lack of configuration ability from a Standby L3 MM Failover unless it's manually promoted to Primary). I'm continuing to work with TAC on the issue - a bit more info I should have included - I saw this behavior about 8 weeks ago (thought it had been a fluke):

I was clearing up a "config commit error" related to a syslog server setting on the MDs:

1. Removed errored out syslog config setting
2. Issued a "ccm-debug full-config-sync" to perform a full sync per another post
3. Sync completed successfully after performing "show switches"
4. Started receiving "connectivity issues" - upon reviewing the configuration of the "validuser" acl - discovered the "any any any deny" had moved up to the top.

What it almost looked like happened in both cases. When a "Full Sync" occurred, it loaded a version of the "validuser" acl from the "full folder", but none of the partial configs since then (based on obervation -> I gave TAC serveral screenshots of my observation from the flashbackups) - and referenced if this could be related to Defect 182049 - which is resolved in ArubaOS 8.2.2.2.

Sorry for all the complex observations. :-) I am still working with TAC.

While on topic - how does one promote a Standby L3 MM to a Primary? I've asked TAC about this for future reference, but haven't heard back yet.

---

@cbjohns wrote:

@cjoseph wrote:

All you had to say is L3 MM failover.  I am not sure that is supported as of yet.

EDIT: Yes it is supported.

@cjoseph, funny thing. You helped answer one of the the 3 issues we ran into during the L3 MM Fail-Over when you prompted that support question. :-) I went looking for one of the original L3 MM posts for support verification, and remembered the lack of "configuration ability" from the standby L3 MM was by design - https://community.arubanetworks.com/t5/Wireless-Access/Layer-3-Redundancy-for-Mobility-Master-on-8-2/m-p/313229#M75784 -- from jhoward - "There will not be any configuration abaility (ability to manage configurations of the MCs) *unless* you manually promote the Standby MM to Primary. This is to prevent split brain issues and you would only promote the standby if you know the primary MM is down for good or for awhile."

---

@jhoward wrote:

L3 MM failover is supported. If a change was made between the replication window of the L3 MMs (this is configured or configurable) then the two MMs could have different config IDs. 

Anytime the MM makes a config change and then a write mem is ussued, then a new config ID is generated and the diff pushed to the controllers. I don't know if there is a set specific window where the CFGs are merged. So long as the config ID is the same for the active MM and the MDs, that's what matters. If your L3 MMs are not syncing correctly, or the sync is disabled, please open a TAC case and have them check it. But the smallest window for L3 MM sync is two hours. SO there IS a rare chance that if you have an active MM and just pushed a config change and then lost the MM, that the MDs would revert to the L3 MM after 15min and may pull an older ocnfig. 

 

---

@ @jhoward, thanks for the additional information (and the other post about lack of configuration ability from a Standby L3 MM Failover unless it's manually promoted to Primary). I'm continuing to work with TAC on the issue - a bit more info I should have included - I saw this behavior about 8 weeks ago (thought it had been a fluke):

 

I was clearing up a "config commit error" related to a syslog server setting on the MDs:

1. Removed errored out syslog config setting
2. Issued a "ccm-debug full-config-sync" to perform a full sync per another post
3. Sync completed successfully after performing "show switches"
4. Started receiving "connectivity issues" - upon reviewing the configuration of the "validuser" acl - discovered the "any any any deny" had moved up to the top.

What it almost looked like happened in both cases. When a "Full Sync" occurred, it loaded a version of the "validuser" acl from the "full folder", but none of the partial configs since then (based on obervation -> I gave TAC serveral screenshots of my observation from the flashbackups) - and referenced if this could be related to Defect 182049 - which is resolved in ArubaOS 8.2.2.2.

Sorry for all the complex observations. :-) I am still working with TAC.

 

While on topic - how does one promote a Standby L3 MM to a Primary? I've asked TAC about this for future reference, but haven't heard back yet.

 

---

Updating my original post even though resolved several months ago. Engineering identified our issue as 177162 - Resolved in 8.3.0.2 - https://www.arubanetworks.com/techdocs/ArubaOS/8.3.x.x/Content/ReleaseNotes/ResolvedIssues/Resolved8.3.0.x/Resolved8.3.0.x.htm

One other thing to remember is the synchronization of the MM databases. L2 synchronization is every 60 minutes between the MMs by default and L3 synchronization is every 120 minutes to the L3 redundant MM(s), as shown in the output below.

(MM2) [mynode] #show database synchronize

Last L2 synchronization time: Fri May 31 17:04:22 2019
Last L3 synchronization time: Fri May 31 17:04:23 2019
To Master Switch at 10.254.1.101: succeeded
To Secondary Master Switch at 192.168.240.100: succeeded
WMS Database backup file size: 47118 bytes
Upgrademgr Database backup file size: 3392 bytes
Cluster upgrademgr Database backup file size: 3885 bytes
Local User Database backup file size: 35912 bytes
Global AP Database backup file size: 24270 bytes
IAP Database backup file size: 3769 bytes
Airgroup Database backup file size: 3069 bytes
License Database backup file size: 3248 bytes
CPSec Database backup file size: 3224 bytes
Bocmgr Database backup file size: 6047 bytes
L2 Synchronization took 3 second
L3 Synchronization took less than one second

199 L2 synchronization attempted
0 L2 synchronization have failed

101 L3 synchronization attempted
2 L3 synchronization have failed

L2 Periodic synchronization is enabled and runs every 60 minutes

L3 Periodic synchronization is enabled and runs every 120 minutes

Synchronization doesn't include Captive Portal Custom data
Airmatch database gets synchronized periodically. Last synchronization time : 2019-05-31 16:48:29

I hope this is helpful,

I forgot to mention you can easily force a synchronization.

To see the status, use the following command

show database synchronize

To force a synchronization (of both L2 and L3 MMs) use the following command from the MM that is the VRRP master

database-synchronize

The following output shows the synchronization status, then performs a synchronization, and then shows the status again.

(MM2) [mynode] #show database synchronize

Last L2 synchronization time: Fri May 31 18:04:28 2019
Last L3 synchronization time: Fri May 31 17:04:23 2019
To Master Switch at 10.254.1.101: succeeded
To Secondary Master Switch at 192.168.240.100: succeeded
WMS Database backup file size: 45968 bytes
Upgrademgr Database backup file size: 3392 bytes
Cluster upgrademgr Database backup file size: 3885 bytes
Local User Database backup file size: 35910 bytes
Global AP Database backup file size: 24270 bytes
IAP Database backup file size: 3769 bytes
Airgroup Database backup file size: 3069 bytes
License Database backup file size: 5599 bytes
CPSec Database backup file size: 3224 bytes
Bocmgr Database backup file size: 6047 bytes
L2 Synchronization took 4 second
L3 Synchronization took less than one second

200 L2 synchronization attempted
0 L2 synchronization have failed

101 L3 synchronization attempted
2 L3 synchronization have failed

L2 Periodic synchronization is enabled and runs every 60 minutes

L3 Periodic synchronization is enabled and runs every 120 minutes

Synchronization doesn't include Captive Portal Custom data
Airmatch database gets synchronized periodically. Last synchronization time : 2019-05-31 17:49:35

(MM2) [mynode] #database-synchronize
Database synchronize has been initiated.
Please check the status using 'show database synchronize' command.

(MM2) [mynode] #show database synchronize

Last L2 manual synchronization time: Fri May 31 18:09:36 2019
Last L3 manual synchronization time: Fri May 31 18:09:37 2019
To Master Switch at 10.254.1.101: succeeded
To Secondary Master Switch at 192.168.240.100: succeeded
WMS Database backup file size: 46124 bytes
Upgrademgr Database backup file size: 3392 bytes
Cluster upgrademgr Database backup file size: 3885 bytes
Local User Database backup file size: 35910 bytes
Global AP Database backup file size: 24270 bytes
IAP Database backup file size: 3769 bytes
Airgroup Database backup file size: 3069 bytes
License Database backup file size: 3248 bytes
CPSec Database backup file size: 3224 bytes
Bocmgr Database backup file size: 6047 bytes
L2 Synchronization took 3 second
L3 Synchronization took 1 second

201 L2 synchronization attempted
0 L2 synchronization have failed

102 L3 synchronization attempted
2 L3 synchronization have failed

L2 Periodic synchronization is enabled and runs every 60 minutes

L3 Periodic synchronization is enabled and runs every 120 minutes

Synchronization doesn't include Captive Portal Custom data
Airmatch database gets synchronized periodically. Last synchronization time : 2019-05-31 17:49:35