Wireless Access

I am testing Master-Master, Master-Standby VRRP shown below:

[Master-Master]  ArubaOS 5.0.4.6  Aruba 3400 

interface vlan 1
        ip address 172.200.1.206 255.255.252.0

master-redundancy
  master-vrrp 10
  peer-ip-address 172.200.1.207 ipsec ********

vrrp 10
  priority 110
  ip address 172.200.1.208
  vlan 1
  preempt
  tracking master-up-time 30 add 20
  no shutdown

ip default-gateway 172.200.1.1

[Master-Standby]  ArubaOS 5.0.4.6  Aruba 3400 

interface vlan 1
        ip address 172.200.1.207 255.255.252.0

master-redundancy
  master-vrrp 10
  peer-ip-address 172.200.1.206 ipsec ********

vrrp 10
  priority 110
  ip address 172.200.1.208
  vlan 1
  preempt
  tracking master-up-time 30 add 20
  no shutdown

ip default-gateway 172.200.1.1

When I configured above, WebUI for 172.200.1.208 has a problem. After typed userid and password in WebUI,

it takes so long (more than 30mins) to show first screen. In the show log,

May  3 03:37:50  webui[1314]: PAPI_Send: To: 7f000001:8224 Type:0x4 Timed out.

Also, when I tried to perform "show running-config" in Master-Master controller using serial connection,

Module STM busy is displayed.

(Master-Master) #show run
Building Configuration...
Module STM is busy. Please try later

There is a entry in show log:

May  3 03:24:46  stm[1432]: <304001> <ERRS> |stm|  Unexpected stm (Station managment) runtime error at data_path_handler, 649, data_path_handler: recv - Network is down

Off cource, Master-Master and Master-Standby communicates each other in Layer2 switch.

I left above over night and WebUI showed normally and show running-config responded normally. I do not know how long did it take for timeout.

From my experience in past, when I used VLAN1 for Master-Local redundancy, same symptom occured.

Since then, I avoid using VLAN1 for Master-Master or Master-Local controller ipsec communication.

If I use VLAN10 instead of VLAN1, everything works fine. (I posted Master-Master and Master-Standby VRRP redundancy successful case using VLAN10 in previous post)

I am going to attach show log below. It seems that PAPI communication fails, even though two controllers are connected each other through VLAN1.

Aruba's sample configuration never use VLAN1 for actual controller-controller communication. Aruba sample config uses VLAN99, not VLAN1. Also Loopback should be defined as a endpoint of controller cimmunication.

I am fine how to fix this problem, just avoid using VLAN1. But I want to see Aruba's document why VLAN1 should not be used.

[Master-Master's show log all - from initial boot time]

May  3 03:23:49  ike[1338]: <103062> <INFO> |ike|  Starting cryptoPOST
May  3 03:23:51  fpapps[1406]: <315382> <CRIT> |fpapps|  Reboot Cause: User reboot.
May  3 03:23:51  fpapps[1406]: <315382> <CRIT> |fpapps|  Reboot Cause: User reboot.
May  3 03:23:52  isakmpd[1422]: <103061> <ERRS> |ike|  udp_make: setsockopt (-1, 65535, 4, 0x7f8aab40, 4)
May  3 03:23:52  isakmpd[1422]: <103061> <ERRS> |ike|  udp_make: setsockopt (-1, 65535, 4, 0x7f8aab40, 4)
May  3 03:23:52  isakmpd[1422]: <103061> <ERRS> |ike|  virtual_bind_ADDR_ANY: could not allocate default IPv6 ISAKMP port(s)
May  3 03:23:52  isakmpd[1422]: <103061> <ERRS> |ike|  virtual_bind_ADDR_ANY: could not allocate default IPv6 ISAKMP port(s)
May  3 03:23:55  KERNEL:     max_val[0] = a2c2a, max_val[others] = 3640e,
/////// Omit several KERNEL lines ///////

May  3 03:23:55  KERNEL: Aruba Networks
May  3 03:23:55  KERNEL: ArubaOS Version 5.0.4.6 (build 33270 / label #33270)
May  3 03:23:55  KERNEL: Built by p4build@corsica on 2012-04-13 at 07:05:48 PDT (gcc version 3.4.3)
May  3 03:23:55  KERNEL: ility-core init in kernel
May  3 03:23:55  KERNEL: klogd started: BusyBox v1.01 (2012.04.13-14:01+0000)
May  3 03:24:04  nanny[1262]: PAPI_Send: To: 7f000001:8407 Type:0x4 Timed out.
May  3 03:24:04  nanny[1262]: PAPI_Send: To: 7f000001:8407 Type:0x4 Timed out.
May  3 03:24:06  cts[1463]: PAPI_Send: To: 7f000001:8226 Type:0x4 Timed out.
May  3 03:24:06  cts[1463]: PAPI_Send: To: 7f000001:8226 Type:0x4 Timed out.
May  3 03:24:09  mobileip[1436]: PAPI_Send: To: 7f000001:8226 Type:0x4 Timed out.
May  3 03:24:09  mobileip[1436]: PAPI_Send: To: 7f000001:8226 Type:0x4 Timed out.
May  3 03:24:10  nanny[1262]: PAPI_Send: To: 7f000001:8407 Type:0x4 Timed out.
May  3 03:24:12  dbsync[1445]: PAPI_Send: To: 7f000001:8226 Type:0x4 Timed out.
May  3 03:24:12  dbsync[1445]: PAPI_Send: To: 7f000001:8226 Type:0x4 Timed out.
May  3 03:24:12  snmp[1439]: PAPI_Send: To: 7f000001:8212 Type:0x4 Timed out.
May  3 03:24:12  snmp[1439]: PAPI_Send: To: 7f000001:8212 Type:0x4 Timed out.
May  3 03:24:12  snmp[1441]: PAPI_Send: To: 7f000001:8212 Type:0x4 Timed out.
May  3 03:24:12  snmp[1441]: PAPI_Send: To: 7f000001:8212 Type:0x4 Timed out.
May  3 03:24:13  authmgr[1431]: PAPI RxPacket: ACK to invalid packet type 0x00000022
May  3 03:24:13  authmgr[1431]: PAPI RxPacket: ACK to invalid packet type 0x00000022
May  3 03:24:13  syslogdwrap[1401]: PAPI_Send: To: 7f000001:8226 Type:0x4 Timed out.
May  3 03:24:13  syslogdwrap[1401]: PAPI_Send: To: 7f000001:8226 Type:0x4 Timed out.
May  3 03:24:16  nanny[1262]: PAPI_Send: To: 7f000001:8407 Type:0x4 Timed out.
May  3 03:24:18  cli[1321]: PAPI_Send: To: 7f000001:8372 Type:0x4 Timed out.
May  3 03:24:18  fpapps[1406]: <313256> <WARN> |fpapps| Route resolve returned an Error
May  3 03:24:18  fpapps[1505]: <313328> <WARN> |fpapps|  vrrp: vrid "10" - VRRP state transitioned from INIT to BACKUP
May  3 03:24:18  httpd[1466]: PAPI_Send: To: 7f000001:8214 Type:0x4 Timed out.
May  3 03:24:18  httpd[1466]: PAPI_Send: To: 7f000001:8214 Type:0x4 Timed out.
May  3 03:24:20  publisher[1399]: <306510> <WARN> |publisher|  Dropping message from 8214 for service 'aaa-idle-user-timeout (service not found)'
May  3 03:24:22  KERNEL: 3:<4>process `snmpd' is using obsolete setsockopt SO_BSDCOMPAT
May  3 03:24:25  KERNEL: 1:<4>process `trapd' is using obsolete setsockopt SO_BSDCOMPAT
May  3 03:24:25  snmp[1441]: IP addr 0xa8001ce port 0
May  3 03:24:25  snmp[1441]: trapSrcIp 0x0
May  3 03:24:28  ads[1449]: PAPI_Send: To: 7f000001:8214 Type:0x4 Timed out.
May  3 03:24:28  ads[1449]: PAPI_Send: To: 7f000001:8214 Type:0x4 Timed out.
May  3 03:24:34  authmgr[1431]: <325022> <ERRS> |authmgr|  Bogus VLAN ID:4095 received.
May  3 03:24:34  authmgr[1431]: <325022> <ERRS> |authmgr|  Bogus VLAN ID:4095 received.
May  3 03:24:46  stm[1432]: <304001> <ERRS> |stm|  Unexpected stm (Station managment) runtime error at data_path_handler, 649, data_path_handler: recv - Network is down
May  3 03:24:46  stm[1432]: <304001> <ERRS> |stm|  Unexpected stm (Station managment) runtime error at data_path_handler, 649, data_path_handler: recv - Network is down
May  3 03:24:48  fpapps[1505]: <313328> <WARN> |fpapps|  vrrp: vrid "10" - VRRP state transitioned from BACKUP to MASTER
May  3 03:24:48  fpapps[1505]: <313331> <WARN> |fpapps|  VRRP: vrid "10" - Missed 3 Hello Advertisements from VRRP Master 172.200.1.206
May  3 03:25:22  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:25:22  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:25:58  publisher[1399]: <306514> <ERRS> |publisher|  Pubsub send message code 0 source port 8378 to destination port 8224 failed, errno Connection timed out
May  3 03:25:58  publisher[1399]: <306514> <ERRS> |publisher|  Pubsub send message code 0 source port 8378 to destination port 8224 failed, errno Connection timed out
May  3 03:26:50  publisher[1399]: <306514> <ERRS> |publisher|  Pubsub send message code 0 source port 8378 to destination port 8224 failed, errno Connection timed out
May  3 03:26:50  publisher[1399]: <306514> <ERRS> |publisher|  Pubsub send message code 0 source port 8378 to destination port 8224 failed, errno Connection timed out
May  3 03:26:54  publisher[1399]: <306514> <ERRS> |publisher|  Pubsub send message code 0 source port 8378 to destination port 8345 failed, errno Connection timed out
May  3 03:26:54  publisher[1399]: <306514> <ERRS> |publisher|  Pubsub send message code 0 source port 8378 to destination port 8345 failed, errno Connection timed out
May  3 03:26:54  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:26:54  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:28:14  cli[1321]: USER: admin has logged in using serial.
May  3 03:28:26  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:28:26  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:28:49  webui[1314]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:28:49  webui[1314]: USER: Error Executing the Command
May  3 03:28:53  cli[1321]: PAPI_Send: To: 7f000001:8222 Type:0x4 Timed out.
May  3 03:28:53  cli[1321]: USER:admin Error Executing the Command
May  3 03:29:49  webui[1314]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:29:49  webui[1314]: USER: Error Executing the Command
May  3 03:29:58  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:29:58  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:30:49  webui[1314]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:30:49  webui[1314]: USER: Error Executing the Command
May  3 03:31:30  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:31:30  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:31:49  webui[1314]: PAPI_Send: To: 7f000001:8224 Type:0x4 Timed out.
May  3 03:32:49  webui[1314]: PAPI_Send: To: 7f000001:8222 Type:0x4 Timed out.
May  3 03:32:49  webui[1314]: USER: Error Executing the Command
May  3 03:33:02  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:33:02  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:33:49  webui[1314]: PAPI_Send: To: 7f000001:8224 Type:0x4 Timed out.
May  3 03:34:34  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:34:34  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:34:50  webui[1314]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:34:50  webui[1314]: USER: Error Executing the Command
May  3 03:35:50  webui[1314]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:36:06  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:36:06  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:36:50  webui[1314]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:36:50  webui[1314]: USER: Error Executing the Command
May  3 03:37:38  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:37:38  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:37:50  webui[1314]: PAPI_Send: To: 7f000001:8224 Type:0x4 Timed out.
May  3 03:38:50  webui[1314]: PAPI_Send: To: 7f000001:8222 Type:0x4 Timed out.
May  3 03:38:50  webui[1314]: USER: Error Executing the Command
May  3 03:39:10  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:39:10  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:39:48  cli[1321]: PAPI_Send: To: 7f000001:8222 Type:0x4 Timed out.
May  3 03:39:48  cli[1321]: USER:admin Error Executing the Command
May  3 03:39:50  webui[1314]: PAPI_Send: To: 7f000001:8224 Type:0x4 Timed out.
May  3 03:39:50  webui[1314]: USER: Error Executing the Command
May  3 03:40:42  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:40:42  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:42:14  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:42:14  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:43:12  webui[1314]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 03:43:12  webui[1314]: USER: Error Executing the Command

In the log,

May  3 03:24:48  fpapps[1505]: <313331> <WARN> |fpapps|  VRRP: vrid "10" - Missed 3 Hello Advertisements from VRRP Master 172.200.1.206

is really strange. Because Master-Master's VLAN1 interface IP is 172.200.1.206. Therefore this Master-Master(perhaps VRRP IP 172.200.1.208) tried to receive VRRP Hello from himself, 172.200.1.206 .... I do not know why this only happens with VLAN1.

Turn authentication on, on the VRRP on both sides to make sure if you have a mystery VRRP or HSRP on your network, it is not bothering it.

I added authentication for each VRRP 10 definition:

   Auth type PASSWORD, Auth data: ********

then rebooted Master-Master and Master-Standby.

Symptom was slightly changed.

WebUI through VRRP IP 172.200.1.208 - Became OK. (Previously NG)

WebUI to 172.200.1.206 (Master-Master) - Became OK. (Previously NG)

WebUI to 172.200.1.207 (Master-Standby) - Became NH (Previously OK)

show run at Master-Master became OK (Previously NG - Module STM is busy)

show run at Master-Standby became NG (Previously OK)

I still believe that VLAN1 is special and should not be used for Controller communication.

If I keep VLAN1 having initial IP 172.21.x.x. assigned and disabled,

and assign VLAN10 for controller-controller communication, everything worked fine.

Do you know why VLAN1 is so special?

I use VLAN 1 for VRRP in my production master-standy and my lab master-standy setup and never saw a problem.

There's nothing wrong with VLAN 1 conceptually in this scenario. A lot of security purists don't like the use of VLAN 1 on any network gear, but that's subjective.

Anyway, can you post your physical interface config, "show port status" outputs, "show arp" outputs, "show vlan" outputs please?

I've used VLAN 1 tons for VRRP. I've hit some bugs with it in advanced setups that the support guys resolved with updates. Need to see your outputs first to work out what's going on.

Thanks.

You're right about the purists, Racking Monkey.  I think it all goes back to Cisco always telling people to avoid using VLAN 1 on their switches, because Cisco  uses VLAN 1 for control plane traffic on their switches.  I have no idea if Aruba does that though.

Bl**dy Cisco. LOL. Yeah, that's where it comes from. Even then though, it's nothing that worries me once you understand why they tow that line. Anyway....

AFAIK, Aruba is VLAN agnostic as it were in terms of specific numbers (quite right too). Can't remember any specific gotchas.

There's two typical stumbling blocks I've seen people hit.

1. Leaving the ports untrusted where the VRRP hellos come in/out. That will break it unless you put an ACL on the port (I had to write one once).

2. Overlapping VRRP IDs or IP conflicts. That tends to wreck it really quickly.

Hi The.racking.monkey and mike.j.gallagher,

I am going to attach outputs here.

I think this issue is "Master-Master redundancy" issue. In the beninning, I configured VRRP10 in VLAN1 and that was fine.

Afterwords, when I configured Master-Master redundancy over VRRP10, this symptom started.

Switch is Cisco 2960G and all port is default (VLAN1)

(Master-Master) #show run
Building Configuration...
Module STM is busy. Please try later ===> This is the symtom (1)

Another symptom is - 172.200.1.206 and 172.200.1.208 WebUI does not respond - process is very slow.

                                        172.200.1.207 WebUI is fine.

(Master-Master) #show port status

Port Status
-----------
Slot-Port  PortType  adminstate  operstate  poe      Trusted  SpanningTree  PortMode
---------  --------  ----------  ---------  ---      -------  ------------  --------
1/0        GE        Enabled     Up         Enabled  Yes      Forwarding    Access
1/1        GE        Enabled     Down       Enabled  Yes      Disabled      Access
1/2        GE        Enabled     Down       Enabled  Yes      Disabled      Access
1/3        GE        Enabled     Down       Enabled  Yes      Disabled      Access

(Master-Master) #show arp

Protocol        Address         Hardware Address        Interface
Internet        172.200.1.33     68:B5:99:F4:15:5D       vlan1
Internet        172.200.1.207    00:0B:86:6D:A5:EC       vlan1

(Master-Master) #show vlan

VLAN CONFIGURATION
------------------
VLAN  Description  Ports
----  -----------  -----
1     Default      GE1/0-3 Pc0-7

(Master-Master) #show crypto ipsec sa

IPSEC SA Active Session Information
-----------------------------------
Initiator IP     Responder IP      InitiatorID   ResponderID  Flags    Start Time      Inner IP
------------     ------------      -----------   -----------  ----------   ----------      --------
172.200.1.206     172.200.1.207       172.200.1.206/32 172.200.1.207/32  T    May  3 08:00:33     -
172.200.1.207     172.200.1.206       172.200.1.207/32 172.200.1.206/32  T    May  3 08:00:26     -

Flags: T = Tunnel Mode; E = Transport Mode; U = UDP Encap
       L = L2TP Tunnel; N = Nortel Client; C = Client

Total IPSEC SAs: 2

<<Comment - I think two ipsec tunnels between Master-Master and Master-Standby is not normal .. I think there should be only one ipsec sa>>

(Master-Master) #show vrrp

Virtual Router 10:
    Description
    Admin State UP, VR State MASTER
    IP Address 172.200.1.208, MAC Address 00:00:5e:00:01:0a, vlan 1
    Priority 110, Advertisement 1 sec, Preemption Enable
    Auth type NONE
    tracking type is master-up-time, duration 30 minutes, value 20
    tracked priority 130

(Master-Master) show log all

I think - key to resolve this problem is the log message:

May  3 08:00:05  fpapps[1505]: <313331> <WARN> |fpapps|  VRRP: vrid "10" - Missed 3 Hello Advertisements from VRRP Master 172.200.1.206

This is Master-Master, and Master-Master's VLAN1 interface is 172.200.1.206. Log above tried to receive Hello Advertisements from 172.200.1.206 - that is itself. At this time, Master-Master has VRRP IP Address 172.200.1.208. If Master-Master sends Hello Advertisement using IP 172.200.1.208 (VRRP IP) but expecting to receive Hello Advertisement from 172.200.1.206, this never happen.....

May  3 07:59:12  KERNEL: klogd started: BusyBox v1.01 (2012.04.13-14:01+0000)
May  3 07:59:21  nanny[1262]: PAPI_Send: To: 7f000001:8407 Type:0x4 Timed out.
May  3 07:59:21  nanny[1262]: PAPI_Send: To: 7f000001:8407 Type:0x4 Timed out.
May  3 07:59:22  syslogdwrap[1401]: PAPI_Send: To: 7f000001:8226 Type:0x4 Timed out.
May  3 07:59:22  syslogdwrap[1401]: PAPI_Send: To: 7f000001:8226 Type:0x4 Timed out.
May  3 07:59:23  cts[1462]: PAPI_Send: To: 7f000001:8226 Type:0x4 Timed out.
May  3 07:59:23  cts[1462]: PAPI_Send: To: 7f000001:8226 Type:0x4 Timed out.
May  3 07:59:27  mobileip[1436]: PAPI_Send: To: 7f000001:8226 Type:0x4 Timed out.
May  3 07:59:27  mobileip[1436]: PAPI_Send: To: 7f000001:8226 Type:0x4 Timed out.
May  3 07:59:27  nanny[1262]: PAPI_Send: To: 7f000001:8407 Type:0x4 Timed out.
May  3 07:59:29  dbsync[1444]: PAPI_Send: To: 7f000001:8226 Type:0x4 Timed out.
May  3 07:59:29  dbsync[1444]: PAPI_Send: To: 7f000001:8226 Type:0x4 Timed out.
May  3 07:59:29  snmp[1441]: PAPI_Send: To: 7f000001:8212 Type:0x4 Timed out.
May  3 07:59:29  snmp[1441]: PAPI_Send: To: 7f000001:8212 Type:0x4 Timed out.
May  3 07:59:30  authmgr[1431]: PAPI RxPacket: ACK to invalid packet type 0x00000022
May  3 07:59:30  authmgr[1431]: PAPI RxPacket: ACK to invalid packet type 0x00000022
May  3 07:59:30  snmp[1440]: PAPI_Send: To: 7f000001:8212 Type:0x4 Timed out.
May  3 07:59:30  snmp[1440]: PAPI_Send: To: 7f000001:8212 Type:0x4 Timed out.
May  3 07:59:33  nanny[1262]: PAPI_Send: To: 7f000001:8407 Type:0x4 Timed out.
May  3 07:59:34  httpd[1466]: PAPI_Send: To: 7f000001:8214 Type:0x4 Timed out.
May  3 07:59:34  httpd[1466]: PAPI_Send: To: 7f000001:8214 Type:0x4 Timed out.
May  3 07:59:35  fpapps[1406]: <313256> <WARN> |fpapps| Route resolve returned an Error
May  3 07:59:35  fpapps[1505]: <313328> <WARN> |fpapps|  vrrp: vrid "10" - VRRP state transitioned from INIT to BACKUP
May  3 07:59:37  cli[1321]: PAPI_Send: To: 7f000001:8372 Type:0x4 Timed out.
May  3 07:59:37  publisher[1399]: <306510> <WARN> |publisher|  Dropping message from 8214 for service 'aaa-idle-user-timeout (service not found)'
May  3 07:59:40  KERNEL: 0:<4>process `snmpd' is using obsolete setsockopt SO_BSDCOMPAT
May  3 07:59:44  KERNEL: 3:<4>process `trapd' is using obsolete setsockopt SO_BSDCOMPAT
May  3 07:59:44  snmp[1441]: IP addr 0xa8001ce port 0
May  3 07:59:44  snmp[1441]: trapSrcIp 0x0
May  3 07:59:45  ads[1450]: PAPI_Send: To: 7f000001:8214 Type:0x4 Timed out.
May  3 07:59:45  ads[1450]: PAPI_Send: To: 7f000001:8214 Type:0x4 Timed out.
May  3 07:59:46  cli[1321]: PAPI_Send: To: 7f000001:8372 Type:0x4 Timed out.
May  3 07:59:50  authmgr[1431]: <325022> <ERRS> |authmgr|  Bogus VLAN ID:4095 received.
May  3 07:59:50  authmgr[1431]: <325022> <ERRS> |authmgr|  Bogus VLAN ID:4095 received.
May  3 08:00:03  stm[1432]: <304001> <ERRS> |stm|  Unexpected stm (Station managment) runtime error at data_path_handler, 649, data_path_handler: recv - Network is down
May  3 08:00:03  stm[1432]: <304001> <ERRS> |stm|  Unexpected stm (Station managment) runtime error at data_path_handler, 649, data_path_handler: recv - Network is down
May  3 08:00:05  fpapps[1406]: PAPI RxPacket: Timer already removed - could be a duplicate ACK
May  3 08:00:05  fpapps[1406]: PAPI RxPacket: Timer already removed - could be a duplicate ACK
May  3 08:00:05  fpapps[1505]: <313328> <WARN> |fpapps|  vrrp: vrid "10" - VRRP state transitioned from BACKUP to MASTER
May  3 08:00:05  fpapps[1505]: <313331> <WARN> |fpapps|  VRRP: vrid "10" - Missed 3 Hello Advertisements from VRRP Master 172.200.1.206
May  3 08:00:39  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 08:00:39  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 08:01:15  publisher[1399]: <306514> <ERRS> |publisher|  Pubsub send message code 0 source port 8378 to destination port 8224 failed, errno Connection timed out
May  3 08:01:15  publisher[1399]: <306514> <ERRS> |publisher|  Pubsub send message code 0 source port 8378 to destination port 8224 failed, errno Connection timed out
May  3 08:01:15  publisher[1399]: <306514> <ERRS> |publisher|  Pubsub send message code 0 source port 8378 to destination port 8345 failed, errno Connection timed out
May  3 08:01:15  publisher[1399]: <306514> <ERRS> |publisher|  Pubsub send message code 0 source port 8378 to destination port 8345 failed, errno Connection timed out
May  3 08:02:07  publisher[1399]: <306514> <ERRS> |publisher|  Pubsub send message code 0 source port 8378 to destination port 8224 failed, errno Connection timed out
May  3 08:02:07  publisher[1399]: <306514> <ERRS> |publisher|  Pubsub send message code 0 source port 8378 to destination port 8224 failed, errno Connection timed out
May  3 08:02:10  publisher[1399]: <306514> <ERRS> |publisher|  Pubsub send message code 0 source port 8378 to destination port 8345 failed, errno Connection timed out
May  3 08:02:10  publisher[1399]: <306514> <ERRS> |publisher|  Pubsub send message code 0 source port 8378 to destination port 8345 failed, errno Connection timed out
May  3 08:02:11  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 08:02:11  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 08:46:39  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
/// Omitted same wms messages ///

May  3 08:48:11  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 08:48:11  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 08:49:41  webui[1312]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 08:49:41  webui[1312]: USER: Error Executing the Command
May  3 08:49:43  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 08:49:43  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 08:50:41  webui[1312]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 08:50:41  webui[1312]: USER: Error Executing the Command
May  3 08:51:15  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 08:51:15  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 08:51:41  webui[1312]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 08:52:41  webui[1312]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 08:52:41  webui[1312]: USER: Error Executing the Command
May  3 08:52:47  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 08:52:47  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 08:53:41  webui[1312]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 08:53:43  cli[1321]: USER: admin has logged in using serial.
May  3 08:54:19  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 08:54:19  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 08:54:23  cli[1321]: PAPI_Send: To: 7f000001:8222 Type:0x4 Timed out.
May  3 08:54:23  cli[1321]: USER:admin Error Executing the Command
May  3 08:54:41  webui[1312]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 08:54:41  webui[1312]: USER: Error Executing the Command
May  3 08:55:41  webui[1312]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 08:55:41  webui[1312]: USER: Error Executing the Command
May  3 08:55:51  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 08:55:51  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 08:56:41  webui[1312]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 08:57:23  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May  3 08:57:23  wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.

(Master-Standby) #show port status

Port Status
-----------
Slot-Port  PortType  adminstate  operstate  poe      Trusted  SpanningTree  PortMode
---------  --------  ----------  ---------  ---      -------  ------------  --------
1/0        GE        Enabled     Up         Enabled  Yes      Disabled      Access
1/1        GE        Enabled     Down       Enabled  Yes      Disabled      Access
1/2        GE        Enabled     Down       Enabled  Yes      Disabled      Access
1/3        GE        Enabled     Down       Enabled  Yes      Disabled      Access

(Master-Standby) #show arp

Protocol        Address         Hardware Address        Interface
Internet        172.200.1.206    00:0B:86:6D:A6:98       vlan1
Internet        172.200.1.33     68:B5:99:F4:15:5D       vlan1

(Master-Standby) #show vlan

VLAN CONFIGURATION
------------------
VLAN  Description  Ports
----  -----------  -----
1     Default      GE1/0-3 Pc0-7

(Master-Standby) #show vrrp

Virtual Router 10:
    Description
    Admin State UP, VR State BACKUP
    IP Address 172.200.1.208, MAC Address 00:00:5e:00:01:0a, vlan 1
    Priority 100, Advertisement 1 sec, Preemption Enable
    Auth type NONE
    tracking type is master-up-time, duration 30 minutes, value 20
    tracked priority 100

(Master-Standby) #show crypto ipsec sa

IPSEC SA Active Session Information
-----------------------------------
Initiator IP     Responder IP      InitiatorID   ResponderID  Flags    Start Time      Inner IP
------------     ------------      -----------   -----------  ----------   ----------      --------
172.200.1.206     172.200.1.207       172.200.1.206/32 172.200.1.207/32  T    May  3 10:41:33     -
172.200.1.207     172.200.1.206       172.200.1.207/32 172.200.1.206/32  T    May  3 10:41:26     -

Flags: T = Tunnel Mode; E = Transport Mode; U = UDP Encap
       L = L2TP Tunnel; N = Nortel Client; C = Client

Total IPSEC SAs: 2

For comparison, I will change VLAN from VLAN1 to VLAN10 and post the same command outputs.

Please post your physical interface commands/running config too.

Full config for both boxes excluding ACLs/roles etc. Starting at first physical port etc.

Hello,

I have been trying to isolate this symptom and found some facts.

First of all, this issue is not related to VLAN1. I could reproduce the same symptom with VLAN10.

1. Symptom

In Master-Master and Master-Standby VRRP redundancy environment, such as

Master-Master VLAN1 172.200.1.206     VRRP priority 110

Master-Standby VLAN1 172.200.1.207   VRRP priority 100

VRRP IP 172.200.1.208

If I powered off then power on Master-Master and Master-Standby at the same time,

symptom (1) With serial console, show running-config command times out with

(Master-Master) #show run
Building Configuration...
Module STM is busy. Please try later

Symptom (2) WebUI did not show all screen. In the left borrom corner, "READ xxxxx" is displayed.

2. Problem isolation

When these symptom is occuring, there are two ipsec sas established between Master-Master and Master-Local.

(Master-Master) # show crypto ipsec sa

IPSEC SA Active Session Information
-----------------------------------
Initiator IP     Responder IP      InitiatorID   ResponderID  Flags    Start Ti e      Inner IP
------------     ------------      -----------   -----------  ----------   ---- -----      --------
172.200.1.206     172.200.1.207       172.200.1.206/32 172.200.1.207/32  T    May    05:55:42     -
172.200.1.207     172.200.1.206       172.200.1.207/32 172.200.1.206/32  T    May    05:55:46     -

Flags: T = Tunnel Mode; E = Transport Mode; U = UDP Encap
       L = L2TP Tunnel; N = Nortel Client; C = Client

Total IPSEC SAs: 2

When this symprom is occuring, reboot the controller which cannot execute show run or WebUI.

(Only one controller at time)

After controller becomes up, there is only one sa exist. I believe that this is normal indication.

Also show running-config and WebUI works fine on both controllers.

(Master-Master) #show crypto ipsec sa

IPSEC SA Active Session Information
-----------------------------------
Initiator IP     Responder IP      InitiatorID   ResponderID  Flags    Start Ti e      Inner IP
------------     ------------      -----------   -----------  ----------   ---- -----      --------
172.200.1.207     172.200.1.206       172.200.1.207/32 172.200.1.206/32  T    May    13:49:04     -

Flags: T = Tunnel Mode; E = Transport Mode; U = UDP Encap
       L = L2TP Tunnel; N = Nortel Client; C = Client

Total IPSEC SAs: 1

3. Assumption

I assume that the reason why this symptom occurs is - when Master-Master or Master-Standby controller power on at the same time, both controller tries to establish ipsec session. If two ipsec sessions are established in error, this symptom might occur.

Also, if we newly configure Master-Master and Master-Standby configuration, we should follow these steps to avoid this symptom:

1. Configure VRRP for Master-Master and Master-Standby

2. Configure master-redundancy only at Master-Master side only, then reboot Master-Master. (When I configure master-redundncy on Master-Master and Master-Standby at the same time and rebooted both controllers, never be successful)

3. After Master-Master comes up, configure master-redundancy on Master-Standby. This is important - because - If Master-Standby already became a pair of master-redundancy, I could not change the configuration. Only if I keep Master-Standby as Master controller, I could configure master-redundancy.

4. Reboot Master-Standby

5. If show run does not work or WebUI does not work, reboot the controller which the symptom resides in.

Everyone, have you experienced such a symptom before?

Not seen it before to do with code. Please post your whole configs as I asked previously.

Any controller that has a master-backup master or master-local relationship will say STM busy until it establishes connectivity with its parent controller.  Those controllers need to reach steady state before anything is done on the commandline.  

The console does come up and allow you to login before it is ready to execute any commands.  If the controller says STM busy after it reaches steady state, you need to open a case so your configuration can be troubleshot in detail.

Hi cjoseph,

When this symptom happens, "STM Busy" as a response of "show run" more than 1 hour.

Also, if this symptom does not happen, "show run" responded after 3 minutes of boot completion.

Therefore, I had to admit that there was something abnormal going on.

My assumtion is - two IPSEC tunnels between Master-Master and Master-Standby controller

might cause routing problem inside of the controller, therefore internal application's communication

such as task-task communication (semaphore) cannot be performed. STM might be trying to do

something, but because of abnormal condition, STM had to wait.

My focus today is to try to determine internal tcp communication and route. If there is a command

to display STM's socket number and peer ip address, it i much helpful, but I do not know what that command

is. I will try to find by myself.

If you know a command to display internal application's communication status (port number and peer ip address)

it is much helpful.

Hi The.racking.monkey,

I am going to paste Master-Master and Master-Standby configuration.

Apologies that I had to mask admin's password and ike secret.

I just configured VLAN1 IP address, VRRP 10, and master-redundancy.

(Master-Master) #show run
Building Configuration...

version 5.0
enable secret "******"
hostname "Master-Master"
clock timezone EST -5
masterip 0.0.0.0
location "Building1.floor1"
mms config 0
controller config 0
ip access-list eth validuserethacl
  permit any
!
netservice svc-snmp-trap udp 162
netservice svc-netbios-dgm udp 138
netservice svc-dhcp udp 67 68
netservice svc-smb-tcp tcp 445
netservice svc-https tcp 443
netservice svc-ike udp 500
netservice svc-l2tp udp 1701
netservice svc-syslog udp 514
netservice svc-pptp tcp 1723
netservice svc-telnet tcp 23
netservice svc-http-accl tcp 88
netservice svc-sccp tcp 2000
netservice svc-sec-papi udp 8209
netservice svc-tftp udp 69
netservice svc-kerberos udp 88
netservice svc-sip-tcp tcp 5060
netservice svc-netbios-ssn tcp 139
netservice svc-pop3 tcp 110
netservice svc-adp udp 8200
netservice svc-cfgm-tcp tcp 8211
netservice svc-noe udp 32512
netservice svc-http-proxy3 tcp 8888
netservice svc-lpd-tcp tcp 631
netservice svc-dns udp 53
netservice svc-msrpc-tcp tcp 135 139
netservice svc-rtsp tcp 554
netservice svc-http tcp 80
netservice svc-vocera udp 5002
netservice svc-h323-tcp tcp 1720
netservice svc-h323-udp udp 1718 1719
netservice svc-nterm tcp 1026 1028
netservice svc-sip-udp udp 5060
netservice svc-http-proxy2 tcp 8080
netservice svc-papi udp 8211
netservice svc-noe-oxo udp 5000 alg noe
netservice svc-ftp tcp 21
netservice svc-natt udp 4500
netservice svc-svp 119
netservice svc-microsoft-ds tcp 445
netservice svc-gre 47
netservice svc-smtp tcp 25
netservice svc-smb-udp udp 445
netservice svc-sips tcp 5061
netservice svc-netbios-ns udp 137
netservice svc-esp 50
netservice svc-cups tcp 515
netservice svc-bootp udp 67 69
netservice svc-snmp udp 161
netservice svc-v6-dhcp udp 546 547
netservice svc-icmp 1
netservice svc-ntp udp 123
netservice svc-msrpc-udp udp 135 139
netservice svc-ssh tcp 22
netservice svc-http-proxy1 tcp 3128
netservice svc-v6-icmp 58
netservice svc-lpd-udp udp 631
ip access-list session allow-diskservices
  any any svc-netbios-dgm permit
  any any svc-netbios-ssn permit
  any any svc-microsoft-ds permit
  any any svc-netbios-ns permit
!
ip access-list session control
  user any udp 68 deny
  any any svc-icmp permit
  any any svc-dns permit
  any any svc-papi permit
  any any svc-sec-papi permit
  any any svc-cfgm-tcp permit
  any any svc-adp permit
  any any svc-tftp permit
  any any svc-dhcp permit
  any any svc-natt permit
!
ip access-list session validuser
  any any any permit
  any any any permit
!
ip access-list session vocera-acl
  any any svc-vocera permit queue high
!
ip access-list session icmp-acl
  any any svc-icmp permit
!
ip access-list session captiveportal
  user   alias controller svc-https dst-nat 8081
  user any svc-http dst-nat 8080
  user any svc-https dst-nat 8081
  user any svc-http-proxy1 dst-nat 8088
  user any svc-http-proxy2 dst-nat 8088
  user any svc-http-proxy3 dst-nat 8088
!
ip access-list session allowall
  any any any permit
!
ip access-list session sip-acl
  any any svc-sip-udp permit queue high
  any any svc-sip-tcp permit queue high
!
ip access-list session https-acl
  any any svc-https permit
!
ip access-list session dns-acl
  any any svc-dns permit
!
ip access-list session allow-printservices
  any any svc-cups permit
  any any svc-lpd-tcp permit
  any any svc-lpd-udp permit
!
ip access-list session logon-control
  user any udp 68 deny
  any any svc-icmp permit
  any any svc-dns permit
  any any svc-dhcp permit
  any any svc-natt permit
!
ip access-list session vpnlogon
  user any svc-ike permit
  user any svc-esp permit
  any any svc-l2tp permit
  any any svc-pptp permit
  any any svc-gre permit
!
ip access-list session srcnat
  user any any src-nat
!
ip access-list session skinny-acl
  any any svc-sccp permit queue high
!
ip access-list session tftp-acl
  any any svc-tftp permit
!
ip access-list session cplogout
  user   alias controller svc-https dst-nat 8081
!
ip access-list session dhcp-acl
  any any svc-dhcp permit
!
ip access-list session http-acl
  any any svc-http permit
!
ip access-list session ap-uplink-acl
  any any udp 68 permit
  any any svc-icmp permit
  any host 224.0.0.251 udp 5353 permit
!
ip access-list session ap-acl
  any any svc-gre permit
  any any svc-syslog permit
  any user svc-snmp permit
  user any svc-http permit
  user any svc-http-accl permit
  user any svc-smb-tcp permit
  user any svc-msrpc-tcp permit
  user any svc-snmp-trap permit
  user any svc-ntp permit
  user   alias controller svc-ftp permit
!
ip access-list session svp-acl
  any any svc-svp permit queue high
  user host 224.0.1.116 any permit
!
ip access-list session noe-acl
  any any svc-noe permit queue high
!
ip access-list session h323-acl
  any any svc-h323-tcp permit queue high
  any any svc-h323-udp permit queue high
!
ipv6 access-list session v6-icmp-acl
  any any svc-v6-icmp permit
!
ipv6 access-list session v6-https-acl
  any any svc-https permit
!
ipv6 access-list session v6-dhcp-acl
  any any svc-v6-dhcp permit
!
ipv6 access-list session v6-dns-acl
  any any svc-dns permit
!
ipv6 access-list session v6-allowall
  any any any permit
!
ipv6 access-list session v6-http-acl
  any any svc-http permit
!
ipv6 access-list session v6-logon-control
  user any udp 68 deny
  any any svc-v6-icmp permit
  any any svc-v6-dhcp permit
  any any svc-dns permit
!
vpn-dialer default-dialer
  ike authentication PRE-SHARE ******
!
user-role ap-role
 session-acl control
 session-acl ap-acl
!
user-role default-vpn-role
 session-acl allowall
 ipv6 session-acl v6-allowall
!
user-role voice
 session-acl sip-acl
 session-acl noe-acl
 session-acl svp-acl
 session-acl vocera-acl
 session-acl skinny-acl
 session-acl h323-acl
 session-acl dhcp-acl
 session-acl tftp-acl
 session-acl dns-acl
 session-acl icmp-acl
!
user-role default-via-role
 session-acl allowall
 ipv6 session-acl v6-allowall
!
user-role guest-logon
 captive-portal "default"
 session-acl logon-control
 session-acl captiveportal
!
user-role guest
 session-acl http-acl
 session-acl https-acl
 session-acl dhcp-acl
 session-acl icmp-acl
 session-acl dns-acl
 ipv6 session-acl v6-http-acl
 ipv6 session-acl v6-https-acl
 ipv6 session-acl v6-dhcp-acl
 ipv6 session-acl v6-icmp-acl
 ipv6 session-acl v6-dns-acl
!
user-role stateful-dot1x
!
user-role authenticated
 session-acl allowall
 ipv6 session-acl v6-allowall
!
user-role logon
 session-acl logon-control
 session-acl captiveportal
 session-acl vpnlogon
 ipv6 session-acl v6-logon-control
!
!

no spanning-tree
interface mgmt
        shutdown
!

dialer group evdo_us
  init-string ATQ0V1E0
  dial-string ATDT#777
!

dialer group gsm_us
  init-string AT+CGDCONT=1,"IP","ISP.CINGULAR"
  dial-string ATD*99#
!

dialer group vivo_br
  init-string AT+CGDCONT=1,"IP","zap.vivo.com.br"
  dial-string ATD*99#
!

interface gigabitethernet  1/0
        description "GE1/0"
        trusted
        trusted vlan 1-4094
!

interface gigabitethernet  1/1
        description "GE1/1"
        trusted
        trusted vlan 1-4094
!

interface gigabitethernet  1/2
        description "GE1/2"
        trusted
        trusted vlan 1-4094
!

interface gigabitethernet  1/3
        description "GE1/3"
        trusted
        trusted vlan 1-4094
!

interface vlan 1
        ip address 172.200.1.206 255.255.252.0
!

master-redundancy
  master-vrrp 10
  peer-ip-address 172.200.1.207 ipsec **********************************************
!
vrrp 10
  priority 150
  ip address 172.200.1.208
  vlan 1
  preempt
  tracking master-up-time 30 add 20
  no shutdown
!
ip default-gateway 172.200.0.1

ap mesh-recovery-profile cluster RecoveryiWFjpGRjGG7S0s6q wpa-hexkey 625880ae9103527d34c679f3e598c50a12ea8de08dc33bdd2a2cf2d8da7ec9619cf38e81fe54b69357a819d18e6aaff6a3036ca347159c13d78ba9f21e8bfb0e640da8b6185b285a5645b3008ad89ffa
wms
 general poll-interval 60000
 general poll-retries 3
 general ap-ageout-interval 30
 general sta-ageout-interval 30
 general learn-ap disable
 general persistent-known-interfering enable
 general propagate-wired-macs enable
 general stat-update enable
 general collect-stats disable
!
crypto isakmp policy 20
  encryption aes256
!

crypto ipsec transform-set default-aes esp-aes256 esp-sha-hmac
crypto dynamic-map default-dynamicmap 10000
  set transform-set default-transform default-aes
!

vpdn group l2tp
!

ip dhcp default-pool private

!

vpdn group pptp
!

mux-address 0.0.0.0

adp discovery enable
adp igmp-join enable
adp igmp-vlan 0

voip prioritization disable
voip rtcp-inactivity disable
voip sip-midcall-req-timeout disable

ssh mgmt-auth username/password
mgmt-user admin root ****************************************

database synchronize period 60
database synchronize rf-plan-data

ip mobile domain default
!

ip igmp
!

no firewall attack-rate cp 1024

!
firewall cp

!
firewall cp

no acceleration cifs caching
no acceleration cifs chattiness
no acceleration cifs read-ahead
no acceleration cifs write-behind
no acceleration http authentication
no acceleration http caching
no acceleration http deduplication
no acceleration http post
no acceleration http sharepoint
no acceleration mapi aggregation
no acceleration mapi caching
no acceleration mapi prefetching
!

packet-capture-defaults tcp disable udp disable sysmsg disable other disable
!
ip domain lookup
!
country US
aaa authentication mac "default"
!
aaa authentication dot1x "default"
!
aaa server-group "default"
 auth-server Internal
 set role condition role value-of
!
aaa authentication via connection-profile "default"
!
aaa authentication via web-auth "default"
!
aaa authentication via global-config
!
aaa profile "default"
!
aaa authentication captive-portal "default"
!
aaa authentication wispr "default"
!
aaa authentication vpn "default"
!
aaa authentication vpn "default-rap"
!
aaa authentication mgmt
!
aaa authentication stateful-ntlm "default"
!
aaa authentication stateful-kerberos "default"
!
aaa authentication stateful-dot1x
!
aaa authentication via auth-profile "default"
!
aaa authentication wired
!
web-server
!
papi-security
!
guest-access-email
!
control-plane-security
!
voice dialplan-profile "default"
!
voice sip
!
aaa password-policy mgmt
!
ap system-profile "default"
!
ap regulatory-domain-profile "default"
   country-code US
   valid-11g-channel 1
   valid-11g-channel 6
   valid-11g-channel 11
   valid-11a-channel 36
   valid-11a-channel 40
   valid-11a-channel 44
   valid-11a-channel 48
   valid-11a-channel 149
   valid-11a-channel 153
   valid-11a-channel 157
   valid-11a-channel 161
   valid-11a-channel 165
   valid-11g-40mhz-channel-pair 1-5
   valid-11g-40mhz-channel-pair 7-11
   valid-11a-40mhz-channel-pair 36-40
   valid-11a-40mhz-channel-pair 44-48
   valid-11a-40mhz-channel-pair 149-153
   valid-11a-40mhz-channel-pair 157-161
!
ap wired-ap-profile "default"
!
ap enet-link-profile "default"
!
ap mesh-ht-ssid-profile "default"
!
ap mesh-cluster-profile "default"
!
ap wired-port-profile "default"
!
ap mesh-radio-profile "default"
!
ids general-profile "default"
!
ids unauthorized-device-profile "default"
!
ids profile "default"
!
rf arm-profile "default"
!
rf optimization-profile "default"
!
rf event-thresholds-profile "default"
!
rf dot11a-radio-profile "default"
!
rf dot11g-radio-profile "default"
!
wlan dot11k-profile "default"
!
wlan voip-cac-profile "default"
!
wlan ht-ssid-profile "default"
!
valid-network-oui-profile
!
wlan edca-parameters-profile station "default"
!
wlan edca-parameters-profile ap "default"
!
wlan ssid-profile "default"
!
wlan virtual-ap "default"
!
ap provisioning-profile "default"
!
ap-group "default"
   virtual-ap "default"
!

snmp-server enable trap

process monitor log
end

(Master-Master) #

Here is Master-Standby configuration.

(Master-Standby) #show run
Building Configuration...

version 5.0
enable secret "******"
hostname "Master-Standby"
clock timezone EST -5
masterip 0.0.0.0
location "Building1.floor1"
mms config 0
controller config 0
ip access-list eth validuserethacl
  permit any
!
netservice svc-snmp-trap udp 162
netservice svc-netbios-dgm udp 138
netservice svc-dhcp udp 67 68
netservice svc-smb-tcp tcp 445
netservice svc-https tcp 443
netservice svc-ike udp 500
netservice svc-l2tp udp 1701
netservice svc-syslog udp 514
netservice svc-pptp tcp 1723
netservice svc-telnet tcp 23
netservice svc-http-accl tcp 88
netservice svc-sccp tcp 2000
netservice svc-sec-papi udp 8209
netservice svc-tftp udp 69
netservice svc-kerberos udp 88
netservice svc-sip-tcp tcp 5060
netservice svc-netbios-ssn tcp 139
netservice svc-pop3 tcp 110
netservice svc-adp udp 8200
netservice svc-cfgm-tcp tcp 8211
netservice svc-noe udp 32512
netservice svc-http-proxy3 tcp 8888
netservice svc-lpd-tcp tcp 631
netservice svc-dns udp 53
netservice svc-msrpc-tcp tcp 135 139
netservice svc-rtsp tcp 554
netservice svc-http tcp 80
netservice svc-vocera udp 5002
netservice svc-h323-tcp tcp 1720
netservice svc-h323-udp udp 1718 1719
netservice svc-nterm tcp 1026 1028
netservice svc-sip-udp udp 5060
netservice svc-http-proxy2 tcp 8080
netservice svc-papi udp 8211
netservice svc-noe-oxo udp 5000 alg noe
netservice svc-ftp tcp 21
netservice svc-natt udp 4500
netservice svc-svp 119
netservice svc-microsoft-ds tcp 445
netservice svc-gre 47
netservice svc-smtp tcp 25
netservice svc-smb-udp udp 445
netservice svc-sips tcp 5061
netservice svc-netbios-ns udp 137
netservice svc-esp 50
netservice svc-cups tcp 515
netservice svc-bootp udp 67 69
netservice svc-snmp udp 161
netservice svc-v6-dhcp udp 546 547
netservice svc-icmp 1
netservice svc-ntp udp 123
netservice svc-msrpc-udp udp 135 139
netservice svc-ssh tcp 22
netservice svc-http-proxy1 tcp 3128
netservice svc-v6-icmp 58
netservice svc-lpd-udp udp 631
ip access-list session allow-diskservices
  any any svc-netbios-dgm permit
  any any svc-netbios-ssn permit
  any any svc-microsoft-ds permit
  any any svc-netbios-ns permit
!
ip access-list session control
  user any udp 68 deny
  any any svc-icmp permit
  any any svc-dns permit
  any any svc-papi permit
  any any svc-sec-papi permit
  any any svc-cfgm-tcp permit
  any any svc-adp permit
  any any svc-tftp permit
  any any svc-dhcp permit
  any any svc-natt permit
!
ip access-list session validuser
  any any any permit
  any any any permit
!
ip access-list session vocera-acl
  any any svc-vocera permit queue high
!
ip access-list session icmp-acl
  any any svc-icmp permit
!
ip access-list session captiveportal
  user   alias controller svc-https dst-nat 8081
  user any svc-http dst-nat 8080
  user any svc-https dst-nat 8081
  user any svc-http-proxy1 dst-nat 8088
  user any svc-http-proxy2 dst-nat 8088
  user any svc-http-proxy3 dst-nat 8088
!
ip access-list session allowall
  any any any permit
!
ip access-list session sip-acl
  any any svc-sip-udp permit queue high
  any any svc-sip-tcp permit queue high
!
ip access-list session https-acl
  any any svc-https permit
!
ip access-list session dns-acl
  any any svc-dns permit
!
ip access-list session allow-printservices
  any any svc-cups permit
  any any svc-lpd-tcp permit
  any any svc-lpd-udp permit
!
ip access-list session logon-control
  user any udp 68 deny
  any any svc-icmp permit
  any any svc-dns permit
  any any svc-dhcp permit
  any any svc-natt permit
!
ip access-list session vpnlogon
  user any svc-ike permit
  user any svc-esp permit
  any any svc-l2tp permit
  any any svc-pptp permit
  any any svc-gre permit
!
ip access-list session srcnat
  user any any src-nat
!
ip access-list session skinny-acl
  any any svc-sccp permit queue high
!
ip access-list session tftp-acl
  any any svc-tftp permit
!
ip access-list session cplogout
  user   alias controller svc-https dst-nat 8081
!
ip access-list session dhcp-acl
  any any svc-dhcp permit
!
ip access-list session http-acl
  any any svc-http permit
!
ip access-list session ap-uplink-acl
  any any udp 68 permit
  any any svc-icmp permit
  any host 224.0.0.251 udp 5353 permit
!
ip access-list session ap-acl
  any any svc-gre permit
  any any svc-syslog permit
  any user svc-snmp permit
  user any svc-http permit
  user any svc-http-accl permit
  user any svc-smb-tcp permit
  user any svc-msrpc-tcp permit
  user any svc-snmp-trap permit
  user any svc-ntp permit
  user   alias controller svc-ftp permit
!
ip access-list session svp-acl
  any any svc-svp permit queue high
  user host 224.0.1.116 any permit
!
ip access-list session noe-acl
  any any svc-noe permit queue high
!
ip access-list session h323-acl
  any any svc-h323-tcp permit queue high
  any any svc-h323-udp permit queue high
!
ipv6 access-list session v6-icmp-acl
  any any svc-v6-icmp permit
!
ipv6 access-list session v6-https-acl
  any any svc-https permit
!
ipv6 access-list session v6-dhcp-acl
  any any svc-v6-dhcp permit
!
ipv6 access-list session v6-dns-acl
  any any svc-dns permit
!
ipv6 access-list session v6-allowall
  any any any permit
!
ipv6 access-list session v6-http-acl
  any any svc-http permit
!
ipv6 access-list session v6-logon-control
  user any udp 68 deny
  any any svc-v6-icmp permit
  any any svc-v6-dhcp permit
  any any svc-dns permit
!
vpn-dialer default-dialer
  ike authentication PRE-SHARE ******
!
user-role ap-role
 session-acl control
 session-acl ap-acl
!
user-role default-vpn-role
 session-acl allowall
 ipv6 session-acl v6-allowall
!
user-role voice
 session-acl sip-acl
 session-acl noe-acl
 session-acl svp-acl
 session-acl vocera-acl
 session-acl skinny-acl
 session-acl h323-acl
 session-acl dhcp-acl
 session-acl tftp-acl
 session-acl dns-acl
 session-acl icmp-acl
!
user-role default-via-role
 session-acl allowall
 ipv6 session-acl v6-allowall
!
user-role guest-logon
 captive-portal "default"
 session-acl logon-control
 session-acl captiveportal
!
user-role guest
 session-acl http-acl
 session-acl https-acl
 session-acl dhcp-acl
 session-acl icmp-acl
 session-acl dns-acl
 ipv6 session-acl v6-http-acl
 ipv6 session-acl v6-https-acl
 ipv6 session-acl v6-dhcp-acl
 ipv6 session-acl v6-icmp-acl
 ipv6 session-acl v6-dns-acl
!
user-role stateful-dot1x
!
user-role authenticated
 session-acl allowall
 ipv6 session-acl v6-allowall
!
user-role logon
 session-acl logon-control
 session-acl captiveportal
 session-acl vpnlogon
 ipv6 session-acl v6-logon-control
!
!

no spanning-tree
interface mgmt
        shutdown
!

dialer group evdo_us
  init-string ATQ0V1E0
  dial-string ATDT#777
!

dialer group gsm_us
  init-string AT+CGDCONT=1,"IP","ISP.CINGULAR"
  dial-string ATD*99#
!

dialer group vivo_br
  init-string AT+CGDCONT=1,"IP","zap.vivo.com.br"
  dial-string ATD*99#
!

interface gigabitethernet  1/0
        description "GE1/0"
        trusted
        trusted vlan 1-4094
!

interface gigabitethernet  1/1
        description "GE1/1"
        trusted
        trusted vlan 1-4094
!

interface gigabitethernet  1/2
        description "GE1/2"
        trusted
        trusted vlan 1-4094
!

interface gigabitethernet  1/3
        description "GE1/3"
        trusted
        trusted vlan 1-4094
!

interface vlan 1
        ip address 172.200.1.207 255.255.252.0
!

master-redundancy
  master-vrrp 10
  peer-ip-address 172.200.1.206 ipsec ***************************************************
!
vrrp 10
  ip address 172.200.1.208
  vlan 1
  preempt
  tracking master-up-time 30 add 20
  no shutdown
!
ip default-gateway 172.200.0.1

ap mesh-recovery-profile cluster RecoveryiWFjpGRjGG7S0s6q wpa-hexkey 34286188d9209a09db797329a81e293ab0b7fb70c23d2ab13b3438372f56f9ed8d26ca9f6125932da1aa4e82221b4e928702c736dcaef15881cf3524d8739ccdf880b6d73199d00530c115c144298b68
wms
 general poll-interval 60000
 general poll-retries 3
 general ap-ageout-interval 30
 general sta-ageout-interval 30
 general learn-ap disable
 general persistent-known-interfering enable
 general propagate-wired-macs enable
 general stat-update enable
 general collect-stats disable
!
crypto isakmp policy 20
  encryption aes256
!

crypto ipsec transform-set default-aes esp-aes256 esp-sha-hmac
crypto dynamic-map default-dynamicmap 10000
  set transform-set default-transform default-aes
!

vpdn group l2tp
!

ip dhcp default-pool private

!

vpdn group pptp
!

mux-address 0.0.0.0

adp discovery enable
adp igmp-join enable
adp igmp-vlan 0

voip prioritization disable
voip rtcp-inactivity disable
voip sip-midcall-req-timeout disable

ssh mgmt-auth username/password
mgmt-user admin root *******************************************************************

no database synchronize
database synchronize rf-plan-data

ip mobile domain default
!

ip igmp
!

no firewall attack-rate cp 1024

!
firewall cp

!
firewall cp

no acceleration cifs caching
no acceleration cifs chattiness
no acceleration cifs read-ahead
no acceleration cifs write-behind
no acceleration http authentication
no acceleration http caching
no acceleration http deduplication
no acceleration http post
no acceleration http sharepoint
no acceleration mapi aggregation
no acceleration mapi caching
no acceleration mapi prefetching
!

packet-capture-defaults tcp disable udp disable sysmsg disable other disable
!
ip domain lookup
!
country US
aaa authentication mac "default"
!
aaa authentication dot1x "default"
!
aaa server-group "default"
 auth-server Internal
 set role condition role value-of
!
aaa authentication via connection-profile "default"
!
aaa authentication via web-auth "default"
!
aaa authentication via global-config
!
aaa profile "default"
!
aaa authentication captive-portal "default"
!
aaa authentication wispr "default"
!
aaa authentication vpn "default"
!
aaa authentication vpn "default-rap"
!
aaa authentication mgmt
!
aaa authentication stateful-ntlm "default"
!
aaa authentication stateful-kerberos "default"
!
aaa authentication stateful-dot1x
!
aaa authentication via auth-profile "default"
!
aaa authentication wired
!
web-server
!
papi-security
!
guest-access-email
!
control-plane-security
!
voice dialplan-profile "default"
!
voice sip
!
aaa password-policy mgmt
!
ap system-profile "default"
!
ap regulatory-domain-profile "default"
   country-code US
   valid-11g-channel 1
   valid-11g-channel 6
   valid-11g-channel 11
   valid-11a-channel 36
   valid-11a-channel 40
   valid-11a-channel 44
   valid-11a-channel 48
   valid-11a-channel 149
   valid-11a-channel 153
   valid-11a-channel 157
   valid-11a-channel 161
   valid-11a-channel 165
   valid-11g-40mhz-channel-pair 1-5
   valid-11g-40mhz-channel-pair 7-11
   valid-11a-40mhz-channel-pair 36-40
   valid-11a-40mhz-channel-pair 44-48
   valid-11a-40mhz-channel-pair 149-153
   valid-11a-40mhz-channel-pair 157-161
!
ap wired-ap-profile "default"
!
ap enet-link-profile "default"
!
ap mesh-ht-ssid-profile "default"
!
ap mesh-cluster-profile "default"
!
ap wired-port-profile "default"
!
ap mesh-radio-profile "default"
!
ids general-profile "default"
!
ids unauthorized-device-profile "default"
!
ids profile "default"
!
rf arm-profile "default"
!
rf optimization-profile "default"
!
rf event-thresholds-profile "default"
!
rf dot11a-radio-profile "default"
!
rf dot11g-radio-profile "default"
!
wlan dot11k-profile "default"
!
wlan voip-cac-profile "default"
!
wlan ht-ssid-profile "default"
!
valid-network-oui-profile
!
wlan edca-parameters-profile station "default"
!
wlan edca-parameters-profile ap "default"
!
wlan ssid-profile "default"
!
wlan virtual-ap "default"
!
ap provisioning-profile "default"
!
ap-group "default"
   virtual-ap "default"
!

snmp-server enable trap

process monitor log
end

(Master-Standby) #

Mike8877,

Unfortunately, the config only shows how the device is configured and does not show the device's response to the the environment with that configuration.

On another note, to see the sockets, you would do a "show netstat'

I changed boot image from 5.0.4.6 to 6.1.1.0(Native image came with 3400 controller)

and found that this symptom does not occur in 6.1.1.0

It seems that 6.1.1.0 code does not create ipsec connection between Master-Master and Master-Standby, if there is already established.

BUT - I found 6.1.1.0 has serious problem.

With this Master-Master and Master-Standby configuration, 6.1.1.0 code does not display Config - Controller.

Config - Controller is very important to configure IKE shared secret between Master and Local controller.

Therefore I gave up 6.1.1.0 code.

Because I have to stay at 5.0.4.6 code, I am going to find a best practice how to power on Master-Master and Master-Local.

in show crypto ipsec sa screen, Initiator seems to be the controller which first came up.

So, best practice would be (1) Power on Master-Master and wait for boot completion. Wait additional 5 minutes (2) Power on Master-Standby  (3) Check show crypto ipsec sa and confirm if there is only one ipsec between Master-Master and Master-Standby.

To be clear 6.1.1.0 should not be used....  Please use the latest 6.1.3.x 

Also, please open a case in parallel to get all your questions answered, specific to your use case.

I tested 6.1.3.1 but Configure - Controller was not displayed. Even I went through Controller Wizard then rebooted, Configure-Controller was not displayed. Therefore I am going back to 5.0.4.6. With 5.0.4.6 I will find a good method to boot Master-Master and Master-Standby controllers.

---

@mikek8877 wrote:

I tested 6.1.3.1 but Configure - Controller was not displayed. Even I went through Controller Wizard then rebooted, Configure-Controller was not displayed. Therefore I am going back to 5.0.4.6. With 5.0.4.6 I will find a good method to boot Master-Master and Master-Standby controllers.

---

A few points:

When you upgrade/downgrade, you need to clear your browser cache to make sure that everything displays correctly.

Next, if you are configuring a controller through Configuration> Controller and pointing it at an ip address, that forms a master/local pair and not a master/backup master pair.  Master redundancy is configured in Configuration> Advanced Services> Redundancy.

The STM is definitely when a local controller has not found its master controller or has not connected with it yet.

Please contact support, if you have not done this already to walk you through what you need to do.