Hi The.racking.monkey and mike.j.gallagher,
I am going to attach outputs here.
I think this issue is "Master-Master redundancy" issue. In the beninning, I configured VRRP10 in VLAN1 and that was fine.
Afterwords, when I configured Master-Master redundancy over VRRP10, this symptom started.
Switch is Cisco 2960G and all port is default (VLAN1)
(Master-Master) #show run
Building Configuration...
Module STM is busy. Please try later ===> This is the symtom (1)
Another symptom is - 172.200.1.206 and 172.200.1.208 WebUI does not respond - process is very slow.
172.200.1.207 WebUI is fine.
(Master-Master) #show port status
Port Status
-----------
Slot-Port PortType adminstate operstate poe Trusted SpanningTree PortMode
--------- -------- ---------- --------- --- ------- ------------ --------
1/0 GE Enabled Up Enabled Yes Forwarding Access
1/1 GE Enabled Down Enabled Yes Disabled Access
1/2 GE Enabled Down Enabled Yes Disabled Access
1/3 GE Enabled Down Enabled Yes Disabled Access
(Master-Master) #show arp
Protocol Address Hardware Address Interface
Internet 172.200.1.33 68:B5:99:F4:15:5D vlan1
Internet 172.200.1.207 00:0B:86:6D:A5:EC vlan1
(Master-Master) #show vlan
VLAN CONFIGURATION
------------------
VLAN Description Ports
---- ----------- -----
1 Default GE1/0-3 Pc0-7
(Master-Master) #show crypto ipsec sa
IPSEC SA Active Session Information
-----------------------------------
Initiator IP Responder IP InitiatorID ResponderID Flags Start Time Inner IP
------------ ------------ ----------- ----------- ---------- ---------- --------
172.200.1.206 172.200.1.207 172.200.1.206/32 172.200.1.207/32 T May 3 08:00:33 -
172.200.1.207 172.200.1.206 172.200.1.207/32 172.200.1.206/32 T May 3 08:00:26 -
Flags: T = Tunnel Mode; E = Transport Mode; U = UDP Encap
L = L2TP Tunnel; N = Nortel Client; C = Client
Total IPSEC SAs: 2
<<Comment - I think two ipsec tunnels between Master-Master and Master-Standby is not normal .. I think there should be only one ipsec sa>>
(Master-Master) #show vrrp
Virtual Router 10:
Description
Admin State UP, VR State MASTER
IP Address 172.200.1.208, MAC Address 00:00:5e:00:01:0a, vlan 1
Priority 110, Advertisement 1 sec, Preemption Enable
Auth type NONE
tracking type is master-up-time, duration 30 minutes, value 20
tracked priority 130
(Master-Master) show log all
I think - key to resolve this problem is the log message:
May 3 08:00:05 fpapps[1505]: <313331> <WARN> |fpapps| VRRP: vrid "10" - Missed 3 Hello Advertisements from VRRP Master 172.200.1.206
This is Master-Master, and Master-Master's VLAN1 interface is 172.200.1.206. Log above tried to receive Hello Advertisements from 172.200.1.206 - that is itself. At this time, Master-Master has VRRP IP Address 172.200.1.208. If Master-Master sends Hello Advertisement using IP 172.200.1.208 (VRRP IP) but expecting to receive Hello Advertisement from 172.200.1.206, this never happen.....
May 3 07:59:12 KERNEL: klogd started: BusyBox v1.01 (2012.04.13-14:01+0000)
May 3 07:59:21 nanny[1262]: PAPI_Send: To: 7f000001:8407 Type:0x4 Timed out.
May 3 07:59:21 nanny[1262]: PAPI_Send: To: 7f000001:8407 Type:0x4 Timed out.
May 3 07:59:22 syslogdwrap[1401]: PAPI_Send: To: 7f000001:8226 Type:0x4 Timed out.
May 3 07:59:22 syslogdwrap[1401]: PAPI_Send: To: 7f000001:8226 Type:0x4 Timed out.
May 3 07:59:23 cts[1462]: PAPI_Send: To: 7f000001:8226 Type:0x4 Timed out.
May 3 07:59:23 cts[1462]: PAPI_Send: To: 7f000001:8226 Type:0x4 Timed out.
May 3 07:59:27 mobileip[1436]: PAPI_Send: To: 7f000001:8226 Type:0x4 Timed out.
May 3 07:59:27 mobileip[1436]: PAPI_Send: To: 7f000001:8226 Type:0x4 Timed out.
May 3 07:59:27 nanny[1262]: PAPI_Send: To: 7f000001:8407 Type:0x4 Timed out.
May 3 07:59:29 dbsync[1444]: PAPI_Send: To: 7f000001:8226 Type:0x4 Timed out.
May 3 07:59:29 dbsync[1444]: PAPI_Send: To: 7f000001:8226 Type:0x4 Timed out.
May 3 07:59:29 snmp[1441]: PAPI_Send: To: 7f000001:8212 Type:0x4 Timed out.
May 3 07:59:29 snmp[1441]: PAPI_Send: To: 7f000001:8212 Type:0x4 Timed out.
May 3 07:59:30 authmgr[1431]: PAPI RxPacket: ACK to invalid packet type 0x00000022
May 3 07:59:30 authmgr[1431]: PAPI RxPacket: ACK to invalid packet type 0x00000022
May 3 07:59:30 snmp[1440]: PAPI_Send: To: 7f000001:8212 Type:0x4 Timed out.
May 3 07:59:30 snmp[1440]: PAPI_Send: To: 7f000001:8212 Type:0x4 Timed out.
May 3 07:59:33 nanny[1262]: PAPI_Send: To: 7f000001:8407 Type:0x4 Timed out.
May 3 07:59:34 httpd[1466]: PAPI_Send: To: 7f000001:8214 Type:0x4 Timed out.
May 3 07:59:34 httpd[1466]: PAPI_Send: To: 7f000001:8214 Type:0x4 Timed out.
May 3 07:59:35 fpapps[1406]: <313256> <WARN> |fpapps| Route resolve returned an Error
May 3 07:59:35 fpapps[1505]: <313328> <WARN> |fpapps| vrrp: vrid "10" - VRRP state transitioned from INIT to BACKUP
May 3 07:59:37 cli[1321]: PAPI_Send: To: 7f000001:8372 Type:0x4 Timed out.
May 3 07:59:37 publisher[1399]: <306510> <WARN> |publisher| Dropping message from 8214 for service 'aaa-idle-user-timeout (service not found)'
May 3 07:59:40 KERNEL: 0:<4>process `snmpd' is using obsolete setsockopt SO_BSDCOMPAT
May 3 07:59:44 KERNEL: 3:<4>process `trapd' is using obsolete setsockopt SO_BSDCOMPAT
May 3 07:59:44 snmp[1441]: IP addr 0xa8001ce port 0
May 3 07:59:44 snmp[1441]: trapSrcIp 0x0
May 3 07:59:45 ads[1450]: PAPI_Send: To: 7f000001:8214 Type:0x4 Timed out.
May 3 07:59:45 ads[1450]: PAPI_Send: To: 7f000001:8214 Type:0x4 Timed out.
May 3 07:59:46 cli[1321]: PAPI_Send: To: 7f000001:8372 Type:0x4 Timed out.
May 3 07:59:50 authmgr[1431]: <325022> <ERRS> |authmgr| Bogus VLAN ID:4095 received.
May 3 07:59:50 authmgr[1431]: <325022> <ERRS> |authmgr| Bogus VLAN ID:4095 received.
May 3 08:00:03 stm[1432]: <304001> <ERRS> |stm| Unexpected stm (Station managment) runtime error at data_path_handler, 649, data_path_handler: recv - Network is down
May 3 08:00:03 stm[1432]: <304001> <ERRS> |stm| Unexpected stm (Station managment) runtime error at data_path_handler, 649, data_path_handler: recv - Network is down
May 3 08:00:05 fpapps[1406]: PAPI RxPacket: Timer already removed - could be a duplicate ACK
May 3 08:00:05 fpapps[1406]: PAPI RxPacket: Timer already removed - could be a duplicate ACK
May 3 08:00:05 fpapps[1505]: <313328> <WARN> |fpapps| vrrp: vrid "10" - VRRP state transitioned from BACKUP to MASTER
May 3 08:00:05 fpapps[1505]: <313331> <WARN> |fpapps| VRRP: vrid "10" - Missed 3 Hello Advertisements from VRRP Master 172.200.1.206
May 3 08:00:39 wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May 3 08:00:39 wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May 3 08:01:15 publisher[1399]: <306514> <ERRS> |publisher| Pubsub send message code 0 source port 8378 to destination port 8224 failed, errno Connection timed out
May 3 08:01:15 publisher[1399]: <306514> <ERRS> |publisher| Pubsub send message code 0 source port 8378 to destination port 8224 failed, errno Connection timed out
May 3 08:01:15 publisher[1399]: <306514> <ERRS> |publisher| Pubsub send message code 0 source port 8378 to destination port 8345 failed, errno Connection timed out
May 3 08:01:15 publisher[1399]: <306514> <ERRS> |publisher| Pubsub send message code 0 source port 8378 to destination port 8345 failed, errno Connection timed out
May 3 08:02:07 publisher[1399]: <306514> <ERRS> |publisher| Pubsub send message code 0 source port 8378 to destination port 8224 failed, errno Connection timed out
May 3 08:02:07 publisher[1399]: <306514> <ERRS> |publisher| Pubsub send message code 0 source port 8378 to destination port 8224 failed, errno Connection timed out
May 3 08:02:10 publisher[1399]: <306514> <ERRS> |publisher| Pubsub send message code 0 source port 8378 to destination port 8345 failed, errno Connection timed out
May 3 08:02:10 publisher[1399]: <306514> <ERRS> |publisher| Pubsub send message code 0 source port 8378 to destination port 8345 failed, errno Connection timed out
May 3 08:02:11 wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May 3 08:02:11 wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May 3 08:46:39 wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
/// Omitted same wms messages ///
May 3 08:48:11 wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May 3 08:48:11 wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May 3 08:49:41 webui[1312]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May 3 08:49:41 webui[1312]: USER: Error Executing the Command
May 3 08:49:43 wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May 3 08:49:43 wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May 3 08:50:41 webui[1312]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May 3 08:50:41 webui[1312]: USER: Error Executing the Command
May 3 08:51:15 wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May 3 08:51:15 wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May 3 08:51:41 webui[1312]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May 3 08:52:41 webui[1312]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May 3 08:52:41 webui[1312]: USER: Error Executing the Command
May 3 08:52:47 wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May 3 08:52:47 wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May 3 08:53:41 webui[1312]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May 3 08:53:43 cli[1321]: USER: admin has logged in using serial.
May 3 08:54:19 wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May 3 08:54:19 wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May 3 08:54:23 cli[1321]: PAPI_Send: To: 7f000001:8222 Type:0x4 Timed out.
May 3 08:54:23 cli[1321]: USER:admin Error Executing the Command
May 3 08:54:41 webui[1312]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May 3 08:54:41 webui[1312]: USER: Error Executing the Command
May 3 08:55:41 webui[1312]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May 3 08:55:41 webui[1312]: USER: Error Executing the Command
May 3 08:55:51 wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May 3 08:55:51 wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May 3 08:56:41 webui[1312]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May 3 08:57:23 wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
May 3 08:57:23 wms[1427]: PAPI_Send: To: 7f000001:8345 Type:0x4 Timed out.
(Master-Standby) #show port status
Port Status
-----------
Slot-Port PortType adminstate operstate poe Trusted SpanningTree PortMode
--------- -------- ---------- --------- --- ------- ------------ --------
1/0 GE Enabled Up Enabled Yes Disabled Access
1/1 GE Enabled Down Enabled Yes Disabled Access
1/2 GE Enabled Down Enabled Yes Disabled Access
1/3 GE Enabled Down Enabled Yes Disabled Access
(Master-Standby) #show arp
Protocol Address Hardware Address Interface
Internet 172.200.1.206 00:0B:86:6D:A6:98 vlan1
Internet 172.200.1.33 68:B5:99:F4:15:5D vlan1
(Master-Standby) #show vlan
VLAN CONFIGURATION
------------------
VLAN Description Ports
---- ----------- -----
1 Default GE1/0-3 Pc0-7
(Master-Standby) #show vrrp
Virtual Router 10:
Description
Admin State UP, VR State BACKUP
IP Address 172.200.1.208, MAC Address 00:00:5e:00:01:0a, vlan 1
Priority 100, Advertisement 1 sec, Preemption Enable
Auth type NONE
tracking type is master-up-time, duration 30 minutes, value 20
tracked priority 100
(Master-Standby) #show crypto ipsec sa
IPSEC SA Active Session Information
-----------------------------------
Initiator IP Responder IP InitiatorID ResponderID Flags Start Time Inner IP
------------ ------------ ----------- ----------- ---------- ---------- --------
172.200.1.206 172.200.1.207 172.200.1.206/32 172.200.1.207/32 T May 3 10:41:33 -
172.200.1.207 172.200.1.206 172.200.1.207/32 172.200.1.206/32 T May 3 10:41:26 -
Flags: T = Tunnel Mode; E = Transport Mode; U = UDP Encap
L = L2TP Tunnel; N = Nortel Client; C = Client
Total IPSEC SAs: 2
For comparison, I will change VLAN from VLAN1 to VLAN10 and post the same command outputs.