One of our sites with an MD on it has a site to site VPN connection that isn't the most stable i.e. drops once a week.
The MD is local on site but when the site to site goes down and the MM sees the MD as "down" the actual site loses internet connectivity. All I can see on the MD logs is:
Sep 27 13:35:07 isakmpd[3445]: <103103> <3445> <WARN> |ike| IKE SA Deletion: IKE2_delSa peer:xx.xx.xx.2:500 id:3652689924 errcode:ERR_IKESA_EXPIRED saflags:0x1 arflags:0x0
Sep 27 13:35:37 fpapps[3410]: <399838> <3417> <WARN> |fpapps| Received TUN_DOWN from IKE for default-local-master-ipsecmap
Sep 27 13:35:37 isakmpd[3445]: <103103> <3445> <WARN> |ike| DPD PEER DEAD: peer xx.xx.xx port:4500
Sep 27 13:35:37 isakmpd[3445]: <103103> <3445> <WARN> |ike| IKE SA Deletion: IKE2_delSa peer:xx.xx.xx:4500 id:3652687756 errcode:ERR_IKESA_CLEARED saflags:0x1100002d arflags:0x200
Sep 27 13:35:37 isakmpd[3445]: <103103> <3445> <WARN> |ike| IPSec SA Deletion: IPSEC_delSa SPI:72396100 OppSPI:7bd81800 Dst:xx.xx.xx Src:xx.xx.xx.3 flags:19 dstPort:0 srcPort:0
Sep 27 13:35:43 HCM[3944]: <359000> <3944> <ERRS> |HCM| Unexpected HCM runtime error at hcm_keepalive_timer_handler 514 State changed to Down probeip xx.xx.xx src-intf 9
Sep 27 13:35:43 fpapps[3410]: <399838> <4092> <WARN> |fpapps| ipMapDeleteUplinkDefaultGateway: Deleting static vlan 9 gateway xx.xx.xx.1
Sep 27 13:35:43 fpapps[3410]: <399838> <4092> <WARN> |fpapps| updateUplinkReachState: Wired uplink vlan 9 (bkp NO) reachability (to xx.xx.xx) changed from 1(UP) to 2(DOWN). mode 0. lb-state: ENABLED
Sep 27 13:35:45 snmp[3611]: <399803> <3611> <ERRS> |snmp| An internal system error has occurred at file ../unix/aruba_main.c function snmpRequestProcessing line 704 error Cannot send snmp response.
Sep 27 13:35:46 cfgm[3380]: <399838> <3380> <WARN> |cfgm| LmsHeartBeatResultAction: State(READY:UPDATE SUCCESSFUL:CFGID-494:PEND-488:INITCFGID:473) FD=33:Cannot heartbeat with the master.
Sep 27 13:35:50 snmp[3611]: <399803> <3611> <ERRS> |snmp| An internal system error has occurred at file ../unix/aruba_main.c function snmpRequestProcessing line 704 error Cannot send snmp response.
Sep 27 13:35:53 ntpwrap: ntpdPollingTimer:512 Upstream servers not reachable via local interface.
Sep 27 13:35:54 ntpwrap: getNtpSrvRouteAddr:304:connect() failed
Sep 27 13:35:56 cfgm[3380]: <399838> <3380> <WARN> |cfgm| LmsHeartBeatResultAction: State(READY:UPDATE SUCCESSFUL:CFGID-494:PEND-488:INITCFGID:473) FD=33:Cannot heartbeat with the master.
Sep 27 13:36:00 snmp[3611]: <399803> <3611> <ERRS> |snmp| An internal system error has occurred at file ../unix/aruba_main.c function snmpRequestProcessing line 704 error Cannot send snmp response.
Sep 27 13:36:06 cfgm[3380]: <399838> <3380> <WARN> |cfgm| LmsHeartBeatResultAction: State(READY:UPDATE SUCCESSFUL:CFGID-494:PEND-488:INITCFGID:473) FD=33:Cannot heartbeat with the master.
Sep 27 13:36:07 snmp[3618]: <301250> <3618> <ERRS> |snmp| Sendto failed, unable to send trap to manager xx.xx.xx80:161.
Sep 27 13:36:07 snmp[3618]: <301250> <3618> <ERRS> |snmp| Sendto failed, unable to send trap to manager xx.xx.xx80:162.
Sep 27 13:36:10 authmgr[3478]: <124006> <3478> <WARN> |authmgr| {580} TCP srcip=172.21.201.101 srcport=54323 dstip=192.168.168.34 dstport=7680, action=deny, role=Kainos-GDN-kWireless, policy=kainos-gdn-kwireless
Sep 27 13:36:10 dhcpdwrap[3553]: <202078> <3553> <ERRS> |dhcpdwrap| Relay ERROR: Network is unreachable DISCOVER server=172.21.0.29 giaddr=192.168.160.3 MAC=38:f9:d3:94:04:8e
Sep 27 13:36:10 dhcpdwrap[3553]: <202078> <3553> <ERRS> |dhcpdwrap| Relay ERROR: Network is unreachable DISCOVER server=172.21.0.31 giaddr=192.168.160.3 MAC=38:f9:d3:94:04:8e
Sep 27 13:36:10 isakmpd[3445]: <103103> <3445> <WARN> |ike| IKE SA Deletion: IKE2_delSa peer:xx.xx.xx.2:500 id:3652689925 errcode:ERR_IKESA_EXPIRED saflags:0x1 arflags:0x0
Sep 27 13:36:10 snmp[3611]: <399803> <3611> <ERRS> |snmp| An internal system error has occurred at file ../unix/aruba_main.c function snmpRequestProcessing line 704 error Cannot send snmp response.
Sep 27 13:36:10 snmp[3618]: <301250> <3618> <ERRS> |snmp| Sendto failed, unable to send trap to manager xx.xx.xx80:161.
Sep 27 13:36:10 snmp[3618]: <301250> <3618> <ERRS> |snmp| Sendto failed, unable to send trap to manager xx.xx.xx80:162.
Sep 27 13:36:12 dhcpdwrap[3553]: <202078> <3553> <ERRS> |dhcpdwrap| Relay ERROR: Network is unreachable DISCOVER server=172.21.0.29 giaddr=192.168.160.3 MAC=38:f9:d3:94:04:8e
Sep 27 13:36:12 dhcpdwrap[3553]: <202078> <3553> <ERRS> |dhcpdwrap| Relay ERROR: Network is unreachable DISCOVER server=172.21.0.31 giaddr=192.168.160.3 MAC=38:f9:d3:94:04:8e
Sep 27 13:36:12 snmp[3611]: <399803> <3611> <ERRS> |snmp| An internal system error has occurred at file ../unix/aruba_main.c function snmpRequestProcessing line 704 error Cannot send snmp response.
Sep 27 13:36:14 dhcpdwrap[3553]: <202078> <3553> <ERRS> |dhcpdwrap| Relay ERROR: Network is unreachable DISCOVER server=172.21.0.29 giaddr=192.168.160.3 MAC=38:f9:d3:94:04:8e
Sep 27 13:36:14 dhcpdwrap[3553]: <202078> <3553> <ERRS> |dhcpdwrap| Relay ERROR: Network is unreachable DISCOVER server=172.21.0.31 giaddr=192.168.160.3 MAC=38:f9:d3:94:04:8e
Sep 27 13:36:16 cfgm[3380]: <399838> <3380> <WARN> |cfgm| LmsHeartBeatResultAction: State(READY:UPDATE SUCCESSFUL:CFGID-494:PEND-488:INITCFGID:473) FD=33:Cannot heartbeat with the master.
Sep 27 13:36:16 snmp[3611]: <399803> <3611> <ERRS> |snmp| An internal system error has occurred at file ../unix/aruba_main.c function snmpRequestProcessing line 704 error Cannot send snmp response.
Sep 27 13:36:18 dot1x-proc:2[4061]: <121010> <4061> <ERRS> |dot1x-proc:2| |aaa| Error 128,Network is unreachable sending 215 bytes on radius socket 41
Sep 27 13:36:18 dot1x-proc:2[4061]: <138057> <4061> <ERRS> |dot1x-proc:2| Failed to send the radius request for Station 38:f9:d3:94:04:8e 80:8d:b7:61:74:31
Sep 27 13:36:19 isakmpd[3445]: <103103> <3445> <WARN> |ike| IKE SA Deletion: IKE2_delSa peer:xx.xx.xx:4500 id:3652687756 errcode:OK saflags:0x1110002d arflags:0x200
Sep 27 13:36:19 isakmpd[3445]: <103103> <3445> <WARN> |ike| IKE SA Deletion: IKE2_delSa peer:xx.xx.xx:4500 id:3652689926 errcode:ERR_IKESA_EXPIRED saflags:0x41000005 arflags:0x20
Sep 27 13:36:23 dot1x-proc:2[4061]: <121010> <4061> <ERRS> |dot1x-proc:2| |aaa| Error 128,Network is unreachable sending 227 bytes on radius socket 41
Sep 27 13:36:23 dot1x-proc:2[4061]: <138057> <4061> <ERRS> |dot1x-proc:2| Failed to send the radius request for Station 38:f9:d3:94:04:8e 80:8d:b7:61:74:31
Sep 27 13:36:25 authmgr[3478]: <124006> <3478> <WARN> |authmgr| {581} TCP srcip=172.21.201.101 srcport=54323 dstip=192.168.168.34 dstport=7680, action=deny, role=Kainos-GDN-kWireless, policy=kainos-gdn-kwireless
Sep 27 13:36:26 cfgm[3380]: <399838> <3380> <WARN> |cfgm| LmsHeartBeatResultAction: State(READY:UPDATE SUCCESSFUL:CFGID-494:PEND-488:INITCFGID:473) FD=33:Cannot heartbeat with the master.
Sep 27 13:36:27 dot1x-proc:1[4058]: <121010> <4058> <ERRS> |dot1x-proc:1| |aaa| Error 128,Network is unreachable sending 223 bytes on radius socket 42
Sep 27 13:36:27 dot1x-proc:1[4058]: <138057> <4058> <ERRS> |dot1x-proc:1| Failed to send the radius request for Station 98:01:a7:9b:53:71 80:8d:b7:61:74:31
Sep 27 13:36:27 snmp[3611]: <399803> <3611> <ERRS> |snmp| An internal system error has occurred at file ../unix/aruba_main.c function snmpRequestProcessing line 704 error Cannot send snmp response.
Sep 27 13:36:27 snmp[3618]: <301250> <3618> <ERRS> |snmp| Sendto failed, unable to send trap to manager xx.xx.xx80:161.
Sep 27 13:36:27 snmp[3618]: <301250> <3618> <ERRS> |snmp| Sendto failed, unable to send trap to manager xx.xx.xx80:162.
Sep 27 13:36:28 dot1x-proc:2[4061]: <121010> <4061> <ERRS> |dot1x-proc:2| |aaa| Error 128,Network is unreachable sending 227 bytes on radius socket 41
Sep 27 13:36:28 dot1x-proc:2[4061]: <138057> <4061> <ERRS> |dot1x-proc:2| Failed to send the radius request for Station 38:f9:d3:94:04:8e 80:8d:b7:61:74:31
Sep 27 13:36:28 dot1x-proc:2[4061]: <138094> <4061> <WARN> |dot1x-proc:2| MIC failed in WPA2 Key Message 2 from Station 64:a2:f9:a9:33:b6 80:8d:b7:61:13:a0 GDN-AP-08
Sep 27 13:36:29 dot1x-proc:2[4061]: <138094> <4061> <WARN> |dot1x-proc:2| MIC failed in WPA2 Key Message 2 from Station 64:a2:f9:a9:33:b6 80:8d:b7:61:13:a0 GDN-AP-08
Sep 27 13:36:32 dot1x-proc:1[4058]: <121010> <4058> <ERRS> |dot1x-proc:1| |aaa| Error 128,Network is unreachable sending 223 bytes on radius socket 42
Sep 27 13:36:32 dot1x-proc:1[4058]: <138057> <4058> <ERRS> |dot1x-proc:1| Failed to send the radius request for Station 98:01:a7:9b:53:71 80:8d:b7:61:74:31
Sep 27 13:36:33 dot1x-proc:2[4061]: <121010> <4061> <ERRS> |dot1x-proc:2| |aaa| Error 128,Network is unreachable sending 227 bytes on radius socket 41
Sep 27 13:36:33 dot1x-proc:2[4061]: <138057> <4061> <ERRS> |dot1x-proc:2| Failed to send the radius request for Station 38:f9:d3:94:04:8e 80:8d:b7:61:74:31
Sep 27 13:36:34 dot1x-proc:1[4058]: <138094> <4058> <WARN> |dot1x-proc:1| MIC failed in WPA2 Key Message 2 from Station 64:a2:f9:a9:33:b6 80:8d:b7:61:13:b0 GDN-AP-08
Sep 27 13:36:36 cfgm[3380]: <399838> <3380> <WARN> |cfgm| LmsHeartBeatResultAction: State(READY:UPDATE SUCCESSFUL:CFGID-494:PEND-488:INITCFGID:473) FD=33:Cannot heartbeat with the master.
Sep 27 13:36:37 dot1x-proc:1[4058]: <121010> <4058> <ERRS> |dot1x-proc:1| |aaa| Error 128,Network is unreachable sending 235 bytes on radius socket 42
Sep 27 13:36:37 dot1x-proc:1[4058]: <138057> <4058> <ERRS> |dot1x-proc:1| Failed to send the radius request for Station 98:01:a7:9b:53:71 80:8d:b7:61:74:31
Sep 27 13:36:38 dot1x-proc:2[4061]: <121010> <4061> <ERRS> |dot1x-proc:2| |aaa| Error 128,Network is unreachable sending 227 bytes on radius socket 41
Sep 27 13:36:38 dot1x-proc:2[4061]: <138057> <4061> <ERRS> |dot1x-proc:2| Failed to send the radius request for Station 38:f9:d3:94:04:8e 80:8d:b7:61:74:31
Sep 27 13:36:38 snmp[3618]: <301250> <3618> <ERRS> |snmp| Sendto failed, unable to send trap to manager xx.xx.xx80:161.
Sep 27 13:36:38 snmp[3618]: <301250> <3618> <ERRS> |snmp| Sendto failed, unable to send trap to manager xx.xx.xx80:162.
Sep 27 13:36:39 ntpwrap: getNtpSrvRouteAddr:304:connect() failed
Sep 27 13:36:40 snmp[3611]: <399803> <3611> <ERRS> |snmp| An internal system error has occurred at file ../unix/aruba_main.c function snmpRequestProcessing line 704 error Cannot send snmp response.
Sep 27 13:36:42 dot1x-proc:1[4058]: <121010> <4058> <ERRS> |dot1x-proc:1| |aaa| Error 128,Network is unreachable sending 235 bytes on radius socket 42
Sep 27 13:36:42 dot1x-proc:1[4058]: <138057> <4058> <ERRS> |dot1x-proc:1| Failed to send the radius request for Station 98:01:a7:9b:53:71 80:8d:b7:61:74:31
Sep 27 13:36:43 dot1x-proc:2[4061]: <121010> <4061> <ERRS> |dot1x-proc:2| |aaa| Error 128,Network is unreachable sending 227 bytes on radius socket 41
Sep 27 13:36:43 dot1x-proc:2[4061]: <138057> <4061> <ERRS> |dot1x-proc:2| Failed to send the radius request for Station 38:f9:d3:94:04:8e 80:8d:b7:61:74:31
Sep 27 13:36:46 cfgm[3380]: <399838> <3380> <WARN> |cfgm| LmsHeartBeatResultAction: State(READY:UPDATE SUCCESSFUL:CFGID-494:PEND-488:INITCFGID:473) FD=33:Cannot heartbeat with the master.
Sep 27 13:36:46 snmp[3611]: <399803> <3611> <ERRS> |snmp| An internal system error has occurred at file ../unix/aruba_main.c function snmpRequestProcessing line 704 error Cannot send snmp response.
Sep 27 13:36:47 dot1x-proc:1[4058]: <121010> <4058> <ERRS> |dot1x-proc:1| |aaa| Error 128,Network is unreachable sending 235 bytes on radius socket 42