Wireless Access

Reply
Contributor I

APs in reboot loop

first of all, these are all CAPs. I have seen some things about this happening with RAPs, but these are all CAPs. As far as I know, nothing happened on the controller, but 34 of our 64 135s started going into a reboot loop one evening. I have a ticket open with support, but as I said, all of their recommendations are based on RAPs. 

 

Here is the startup log on an affected AP:

 

APBoot 1.2.8.1 (build 34939)
Built: 2012-08-17 at 12:54:18

Model: AP-13x
CPU: 88F6560 A0 (DDR3)
Clock: CPU 1600MHz, L2 533MHz, SysClock 533MHz, TClock 200MHz
DRAM: 256MB
POST1: passed
Flash: 16 MB
Power: 802.3af POE
LAN: done
PHY: done
PEX 0: RC, link up, x1
bus.dev fn venID devID class rev MBAR0 MBAR1 MBAR2 MBAR3
00.00 00 11ab 6560 00005 02 f1000000 00000000 00000000 00000000
00.01 00 168c 0030 00002 01 90000000 00000000 00000000 00000000
PEX 1: RC, link up, x1
bus.dev fn venID devID class rev MBAR0 MBAR1 MBAR2 MBAR3
01.00 00 11ab 6500 00005 02 f1000000 00000000 00000000 00000000
01.01 00 168c 0030 00002 01 94000000 00000000 00000000 00000000
Net: eth0, eth1
Radio: ar9390#0, ar9390#1

Hit <Enter> to stop autoboot: 0
Checking image @ 0xf6100000
Invalid image format version: 0xffffffff
Checking image @ 0xf6800000
Invalid image format version: 0xc
eth0: up, 1 Gb/s, full duplex
DHCP broadcast 1
DHCP broadcast 2
DHCP IP address: 10.25.0.73
DHCP subnet mask: 255.255.255.0
DHCP def gateway: 10.25.0.1
DHCP DNS server: 10.7.0.112
DHCP DNS domain: hmcorp.local
ADP multicast 1
Controller address: 10.7.0.162
Using eth0 device
TFTP from server 10.7.0.162; our IP address is 10.25.0.73; sending through gateway 10.25.0.1
Filename 'armv5te.ari'.
Load address: 0x2000000
Loading: #################################################################
##############
done
Bytes transferred = 5168276 (4edc94 hex)

Image is signed; verifying checksum... passed
Signer Cert OK
Policy Cert OK
RSA signature verified.
Automatic boot of image at addr 0x02000000 ...
## Booting image at 02000200 ...
Uncompressing... done

Aruba Networks
ArubaOS Version 6.3.1.1 (build 40563 / label #40563)
Built by p4build@port-royal on 2013-10-24 at 17:29:27 PDT (gcc version 4.3.3)
Memory: 256MB = 256MB total
Memory: 243840KB available (2816K code, 9021K data, 3936K init, 0K highmem)
Delay calibration in progress:
Calibrating delay loop... 1597.44 BogoMIPS (lpj=1597440)
PEX0 interface detected Link X1
PEX1 interface detected Link X1
bio: create slab <bio-0> at 0
wdt: registered with refresh
Enabling Watchdog
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
i2c /dev entries driver
i2c-arran: using default base 0xf1018180

Starting Kernel SHA1 KAT ...Completed Kernel SHA1 KAT
Starting Kernel HMAC-SHA1 KAT ...Completed Kernel HMAC-SHA1 KAT
Starting Kernel DES KAT ...Completed Kernel DES KAT
Starting Kernel AES KAT ...Completed Kernel AES KAT

Starting Kernel AESGCM KAT ...Completed Kernel AESGCM KAT
Domain Name: arubanetworks.com
No panic info available
init mv_cesa ok
Ethernet port 1 mode: active-staEthernet Channel Bonding Driver: v3.5.0 (November 4, 2008)
ndby
eth0: link up, full duplex, speed 1 Gbps
ADDRCONF(NETDEV_UP): bond0: link is not ready
eth0: link up, full duplex, speed 1 Gbps
bonding: bond0: making interface eth0 the new active one.
bonding: bond0: first active interface up!
bonding: bond0: enslaving eth0 as an active interface with an up link.
ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
eth1: link down
ADDRCONF(NETDEV_UP): eth1: link is not ready
eth1: link down
bonding: bond0: enslaving eth1 as a backup interface with a down link.
AP xml model 51, num_radios 2 (jiffies 2617)
init_asap_mod: installation:0
radio 0: band 1 ant 0 max_ssid 16
radio 1: band 0 ant 0 max_ssid 16
ethernet_device_event: dev eth0 is up
eth0: link up, full duplex, speed 1 Gbps
eth0: link up, full duplex, speed 1 Gbps
Shutting down eth1 due to insufficient POE voltage [power profile 2]
Starting watchdog process...
Getting an IP address...
10.25.0.73 255.255.255.0 10.25.0.1
Running ADP...Done. Master is 10.7.0.162
ath_hal: 0.9.17.1 (AR5416, AR9380, REGOPS_FUNC, PRIVATE_DIAG, WRITE_EEPROM, 11D)
ath_rate_atheros: Copyright (c) 2001-2005 Atheros Communications, Inc, All Rights Reserved
ath_rate_atheros: Aruba Networks Rate Control Algorithm
ath_dfs: Version 2.0.0
Copyright (c) 2005-2006 Atheros Communications, Inc. All Rights Reserved
ath_spectrum: Version 2.0.0
Copyright (c) 2005-2006 Atheros Communications, Inc. All Rights Reserved
ath_dev: Copyright (c) 2001-2007 Atheros Communications, Inc, All Rights Reserved
ath_pci: 0.9.4.5 (Atheros/multi-bss)
ath_attach: scn cd3c0320 sc cd6c0000 ah cd340000
wifi0: Base BSSID 9c:1c:12:a3:25:b0, 16 available BSSID(s)
eth0 address=9c:1c:12:c2:32:5a
br0 address=9c:1c:12:c2:32:5a
wifi0: AP type AP-135, radio 0, max_bssids 16
wifi0: Atheros 9380: mem=0xf3000000, irq=25 hw_base=0xd19c0000
ath_attach: scn cc820320 sc cc840000 ah cc880000
wifi1: Base BSSID 9c:1c:12:a3:25:a0, 16 available BSSID(s)
eth0 address=9c:1c:12:c2:32:5a
br0 address=9c:1c:12:c2:32:5a
wifi1: AP type AP-135, radio 1, max_bssids 16
wifi1: Atheros 9380: mem=0xf4000000, irq=24 hw_base=0xd1dc0000
ath_ahb: 0.9.4.5 (Atheros/multi-bss)
ath_ahb: No devices found, driver not installed.

Starting FIPS KAT ... Completed FIPS KAT

AP rebooted Fri Dec 31 16:04:22 PST 1999; SAPD: Unable to contact switch: HELLO-TIMEOUT. Last Ctrl msg: HELLO len=920 dest=10.7.0.162 tries=10 seq=0
shutting down watchdog process (nanny will restart it)...

<<<<< Welcome to the Access Point >>>>>

~ # Enabling eth1 due to LLDP power grant [power profile 1]

 

 

Here is the AP pinging the controller:

 

~ # ping 10.7.0.162
PING 10.7.0.162 (10.7.0.162): 56 data bytes
64 bytes from 10.7.0.162: icmp_seq=0 ttl=63 time=0.8 ms
64 bytes from 10.7.0.162: icmp_seq=1 ttl=63 time=0.7 ms
64 bytes from 10.7.0.162: icmp_seq=2 ttl=63 time=0.7 ms
64 bytes from 10.7.0.162: icmp_seq=3 ttl=63 time=0.7 ms
64 bytes from 10.7.0.162: icmp_seq=4 ttl=63 time=0.7 ms
64 bytes from 10.7.0.162: icmp_seq=5 ttl=63 time=0.7 ms
64 bytes from 10.7.0.162: icmp_seq=6 ttl=63 time=0.7 ms
64 bytes from 10.7.0.162: icmp_seq=7 ttl=63 time=0.7 ms
64 bytes from 10.7.0.162: icmp_seq=8 ttl=63 time=0.7 ms
64 bytes from 10.7.0.162: icmp_seq=9 ttl=63 time=0.9 ms
64 bytes from 10.7.0.162: icmp_seq=10 ttl=63 time=0.7 ms
64 bytes from 10.7.0.162: icmp_seq=11 ttl=63 time=1.0 ms
64 bytes from 10.7.0.162: icmp_seq=12 ttl=63 time=0.7 ms
64 bytes from 10.7.0.162: icmp_seq=13 ttl=63 time=0.7 ms
64 bytes from 10.7.0.162: icmp_seq=14 ttl=63 time=12.0 ms
64 bytes from 10.7.0.162: icmp_seq=15 ttl=63 time=0.9 ms
64 bytes from 10.7.0.162: icmp_seq=16 ttl=63 time=0.7 ms
64 bytes from 10.7.0.162: icmp_seq=17 ttl=63 time=0.8 ms
64 bytes from 10.7.0.162: icmp_seq=18 ttl=63 time=0.9 ms
^C
--- 10.7.0.162 ping statistics ---
19 packets transmitted, 19 packets received, 0% packet loss
round-trip min/avg/max = 0.7/1.3/12.0 ms

~ #

 

and finally:

 

(SPIAARUBA01) # show datapath session table 10.25.0.73


Datapath Session Table Entries
------------------------------

Flags: F - fast age, S - src NAT, N - dest NAT
D - deny, R - redirect, Y - no syn
H - high prio, P - set prio, T - set ToS
C - client, M - mirror, V - VOIP
Q - Real-Time Quality analysis
I - Deep inspect, U - Locally destined
E - Media Deep Inspect, G - media signal

Source IP Destination IP Prot SPort DPort Cntr Prio ToS Age Destination TAge Packets Bytes Flags
-------------- -------------- ---- ----- ----- ---- ---- --- --- ----------- ---- --------- --------- -----
10.25.0.73 10.7.0.162 17 8211 8222 0/0 0 0 1 1/2 f 0 0 FYCI
10.25.0.73 10.7.0.162 17 8211 8211 0/0 0 0 1 1/2 f 0 0 FCI
10.7.0.162 10.25.0.73 17 8211 8211 0/0 0 0 1 1/2 f 0 0 FYI
10.7.0.162 10.25.0.73 17 8222 8211 0/0 0 0 1 1/2 f 0 0 FYI

 

Any ideas what I can do?

 

Russell

 

Guru Elite

Re: APs in reboot loop

1.  Find out what ap-group that access point is in.

2.  Find out if in the AP system profile in that AP-Group if there is an LMS-ip.

3.  If that LMS-IP does not point to a controller that is active, that access point is being orphaned.  To solve, you either need to remove that ip address or change it to one that points to an active controller.

 



Colin Joseph
Aruba Customer Engineering

Looking for an Answer? Search the Community Knowledge Base Here: Community Knowledge Base

Contributor I

Re: APs in reboot loop

Thanks, for the response. LMS-ip not being used. All of the affected APs are in one of three ap-groups:

 

ap-group "HMN-RAL-WIDS"
virtual-ap "RAL_CLAIMS_GUEST-vap_prof"
virtual-ap "RAL_INTERNAL-vap_prof"
virtual-ap "RAL_EMPLOYEE-vap"
dot11a-radio-profile "Spectrum-Mode-a"
dot11g-radio-profile "Spectrum-Mode-bg"
ids-profile "RAL-WIPS"
!
ap-group "HMN-SPI-LC-WIDS"
virtual-ap "SPI_INTERNAL_VAP"
virtual-ap "SPI_EMPLOYEE_VAP"
virtual-ap "QA-TESTER_VAP"
virtual-ap "SPI_GUEST_VAP"
virtual-ap "SPI_PHONE_VAP"
dot11a-radio-profile "Spectrum-Mode-a"
dot11g-radio-profile "Spectrum-Mode-bg"
ids-profile "HMN-SPI-WIPS"
!
ap-group "HMN-SPI-WIDS"
virtual-ap "SPI_INTERNAL_VAP"
virtual-ap "SPI_EMPLOYEE_VAP"
virtual-ap "QA-TESTER_VAP"
virtual-ap "SPI_GUEST_VAP"
virtual-ap "SPI_PHONE_VAP"
virtual-ap "SPI_EXT_TEST_VAP"
dot11a-radio-profile "Spectrum-Mode-a"
dot11g-radio-profile "Spectrum-Mode-bg"
ids-profile "HMN-SPI-WIPS"

 

which use the default system-profile:

 

ap system-profile "default"
!

 

what next?

Guru Elite

Re: APs in reboot loop

I would get the output of "show log system 100" to see if you get any clues.



Colin Joseph
Aruba Customer Engineering

Looking for an Answer? Search the Community Knowledge Base Here: Community Knowledge Base

Contributor I

Re: APs in reboot loop

(SPIAARUBA01) # show log system 100

Apr 7 11:17:36 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:17:37 :303022: <WARN> |AP GLN-PortN-123-9c:1c:12:c2:04:72@10.25.0.207 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:16:51 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:17:37 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:17:38 :303022: <WARN> |AP GLSSW-PortS289-9c:1c:12:c2:32:5e@10.25.0.58 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:16:52 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:17:38 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:18:10 :303022: <WARN> |AP 3W-PortN191-9c:1c:12:c2:2f:d2@10.25.0.30 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:17:00 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:18:18 :303022: <WARN> |AP GLSSE-PortS292-9c:1c:12:c2:32:78@10.25.0.52 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:17:32 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:18:20 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:18:23 :303022: <WARN> |AP 2S-PortS174-9c:1c:12:c2:32:1e@10.25.0.61 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:17:12 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:18:27 :303022: <WARN> |AP RAL-WEST-d8:c7:c8:cc:6e:92@10.70.21.146 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:17:17 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:18:27 :303022: <WARN> |AP RAL-EAST-d8:c7:c8:cc:6e:e8@10.70.21.96 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:17:15 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:18:33 :303022: <WARN> |AP 6N-MID-PortN147-9c:1c:12:c2:04:82@10.25.0.23 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:17:48 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:18:38 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:18:46 :303022: <WARN> |AP GLS-Port291-9c:1c:12:c2:32:6c@10.25.0.51 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:18:01 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:18:50 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:18:53 :303022: <WARN> |AP RAL-EAST-d8:c7:c8:cc:6e:ce@10.70.21.151 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:17:42 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:18:53 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:18:54 :303022: <WARN> |AP RAL-WEST-d8:c7:c8:cc:6e:84@10.70.21.140 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:17:42 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:18:55 :303022: <WARN> |AP LC-Port42-9c:1c:12:c2:04:aa@10.25.0.60 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:18:10 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:18:56 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:18:58 :303022: <WARN> |AP LC-Port41-9c:1c:12:c2:04:b0@10.25.0.20 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:18:13 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:19:00 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:19:10 :303022: <WARN> |AP GLS-Port293-9c:1c:12:c2:32:66@10.25.0.41 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:18:25 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:19:14 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:19:35 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:18:25 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:19:35 :303022: <WARN> |AP 6NE-PortN146-9c:1c:12:c2:32:02@10.25.0.26 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:18:50 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:19:35 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:19:56 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.43 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:19:10 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:20:03 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:20:07 :303022: <WARN> |AP PC01-9c:1c:12:c2:32:32@10.25.0.62 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:19:21 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:20:07 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:20:34 :303022: <WARN> |AP 2W-PortN182-9c:1c:12:c2:04:66@10.25.0.65 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:19:24 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:20:35 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:20:40 :303022: <WARN> |AP 6NW-PortN149-9c:1c:12:c2:31:e2@10.25.0.59 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:19:55 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:20:42 :303022: <WARN> |AP PC-Port170-9c:1c:12:c2:04:7a@10.25.0.40 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:19:57 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:20:43 :303022: <WARN> |AP 4N-Port165-9c:1c:12:c2:32:2e@10.25.0.37 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:19:33 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:20:50 :303022: <WARN> |AP GLN-Port122-9c:1c:12:c2:32:60@10.25.0.25 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:20:05 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:20:51 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:21:03 :303022: <WARN> |AP LC-Port21-9c:1c:12:c2:04:bc@10.25.0.63 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:20:17 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:21:05 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:21:07 :303022: <WARN> |AP 4N-Port164-9c:1c:12:c2:32:36@10.25.0.32 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:19:57 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:21:08 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:21:08 :303022: <WARN> |AP 3S-Port175-9c:1c:12:c2:32:3a@10.25.0.42 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:20:23 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:21:09 :303022: <WARN> |AP 6W-PortN148-9c:1c:12:c2:04:9c@10.25.0.66 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:20:23 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:21:09 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:21:34 :303022: <WARN> |AP 5S-PortS159-9c:1c:12:c2:04:70@10.25.0.27 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:20:23 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:21:35 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:21:41 :303022: <WARN> |AP GLN-PortN-123-9c:1c:12:c2:04:72@10.25.0.22 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:20:55 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:21:42 :303022: <WARN> |AP GLSSW-PortS289-9c:1c:12:c2:32:5e@10.25.0.58 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:20:56 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:21:50 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:22:21 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:22:22 :303022: <WARN> |AP GLSSE-PortS292-9c:1c:12:c2:32:78@10.25.0.52 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:21:36 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:22:22 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:22:37 :303022: <WARN> |AP 6N-MID-PortN147-9c:1c:12:c2:04:82@10.25.0.23 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:21:52 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:22:40 :303022: <WARN> |AP 3W-PortN191-9c:1c:12:c2:2f:d2@10.25.0.30 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:21:30 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:22:50 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:22:50 :303022: <WARN> |AP GLS-Port291-9c:1c:12:c2:32:6c@10.25.0.51 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:22:05 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:22:50 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:22:53 :303022: <WARN> |AP 2S-PortS174-9c:1c:12:c2:32:1e@10.25.0.61 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:21:42 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:22:57 :303022: <WARN> |AP RAL-WEST-d8:c7:c8:cc:6e:92@10.70.21.146 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:21:47 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:22:58 :303022: <WARN> |AP RAL-EAST-d8:c7:c8:cc:6e:e8@10.70.21.96 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:21:47 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:23:00 :303022: <WARN> |AP LC-Port42-9c:1c:12:c2:04:aa@10.25.0.60 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:22:14 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:23:02 :303022: <WARN> |AP LC-Port41-9c:1c:12:c2:04:b0@10.25.0.20 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:22:17 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:23:04 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:23:22 :303022: <WARN> |AP RAL-EAST-d8:c7:c8:cc:6e:ce@10.70.21.151 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:22:12 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:23:22 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:23:24 :303022: <WARN> |AP RAL-WEST-d8:c7:c8:cc:6e:84@10.70.21.140 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:22:14 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:23:26 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:23:27 :303022: <WARN> |AP GLS-Port293-9c:1c:12:c2:32:66@10.25.0.41 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:22:41 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:23:27 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:23:40 :303022: <WARN> |AP 6NE-PortN146-9c:1c:12:c2:32:02@10.25.0.26 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:22:54 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:23:40 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:24:00 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.43 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:23:14 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:24:05 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:22:55 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:24:06 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:24:12 :303022: <WARN> |AP PC01-9c:1c:12:c2:32:32@10.25.0.62 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:23:26 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:24:20 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:24:45 :303022: <WARN> |AP 6NW-PortN149-9c:1c:12:c2:31:e2@10.25.0.59 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:23:59 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:24:46 :304055: <ERRS> |AP DAL-WN-d8:c7:c8:cc:6e:bc@10.70.20.146 stm| |ap| Unexpected stm (Station management) runtime error at bridge_user_del, 11004, bridge_user_del: bridge user doesn't exist(1) ip:10.70.20.84
Apr 7 11:24:46 :304055: <ERRS> |AP DAL-WN-d8:c7:c8:cc:6e:bc@10.70.20.146 stm| |ap| Unexpected stm (Station management) runtime error at bridge_user_del_from_hash, 11036, bridge_user_del_from_hash: bridge_user_del returned err
Apr 7 11:24:46 :304055: <ERRS> |AP DAL-WN-d8:c7:c8:cc:6e:bc@10.70.20.146 stm| |ap| Unexpected stm (Station management) runtime error at bridge_user_del, 11004, bridge_user_del: bridge user doesn't exist(1) ip:10.70.20.84
Apr 7 11:24:46 :303022: <WARN> |AP PC-Port170-9c:1c:12:c2:04:7a@10.25.0.40 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:24:01 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:24:46 :304055: <ERRS> |AP DAL-WN-d8:c7:c8:cc:6e:bc@10.70.20.146 stm| |ap| Unexpected stm (Station management) runtime error at bridge_user_del_from_hash, 11036, bridge_user_del_from_hash: bridge_user_del returned err
Apr 7 11:24:50 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:24:55 :303022: <WARN> |AP GLN-Port122-9c:1c:12:c2:32:60@10.25.0.25 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:24:09 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:25:04 :303022: <WARN> |AP 2W-PortN182-9c:1c:12:c2:04:66@10.25.0.65 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:23:54 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:25:07 :303022: <WARN> |AP LC-Port21-9c:1c:12:c2:04:bc@10.25.0.63 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:24:22 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:25:12 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:25:13 :303022: <WARN> |AP 6W-PortN148-9c:1c:12:c2:04:9c@10.25.0.66 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:24:27 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:25:13 :303022: <WARN> |AP 4N-Port165-9c:1c:12:c2:32:2e@10.25.0.37 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:24:03 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:25:14 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:25:25 :303022: <WARN> |AP 3S-Port175-9c:1c:12:c2:32:3a@10.25.0.42 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:24:40 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:25:32 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:25:37 :303022: <WARN> |AP 4N-Port164-9c:1c:12:c2:32:36@10.25.0.32 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:24:27 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:25:37 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:25:45 :303022: <WARN> |AP GLN-PortN-123-9c:1c:12:c2:04:72@10.25.0.22 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:25:00 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:25:47 :303022: <WARN> |AP GLSSW-PortS289-9c:1c:12:c2:32:5e@10.25.0.58 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:25:01 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:25:53 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
Apr 7 11:26:04 :303022: <WARN> |AP 5S-PortS159-9c:1c:12:c2:04:70@10.25.0.27 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:24:53 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 7 11:26:06 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663

 

unable to set up IPSec tunnel , but why all of a sudden? and only on some? And on the same switch stack, some APs are fine and some are rebooting?

Moderator

Re: APs in reboot loop

Hi regibbons,

 

Possible causes of such a log may be

>> problem with isakmpd at controller

>> ipsec rekeying issue

>> routing problem

>> dupe IP on network

 

some thoughts on what to check

 

>> go into the AP shell and take a look in /tmp/rapper.txt - this is the ipsec client on the APs log file (it's name is a hangover from RAP days before CPSEC)

>> make sure the isakmpd is healthy on the controller (check number of SAs, cpu load, check stats)

>> make sure that the def gw, AP IP and controller IP are not being duped by another host

 

Questions

>> does every AP always come up then at some time later it fails ?

>> once an AP reboots, does it come up straight away, or takes some time ?

 


regards

-jeff

Contributor I

Re: APs in reboot loop

Thanks for the reply, I will try to adress everything you mentioned.

I am not not sure how to check isakmpd or ipsec rekeying. As far as a routing problem, in at least one case I have two APs on the same switch stack where one is fine and the other is rebooting. IPsec is disabled according to the controller on all of the ap-groups in question. As far as I can tell, there is no dupe of the controller IP on the network.

 

I'm going to get a failing AP out of the ceiling and see what I can learn from it. Again, not sure how to chek number of SAs, or stats, but the cpu load is 4%

 

there are ~34 APs being affected by this and ~30 that are not. the 34 that are affected reboot in the pattern below. This is one example, but they all follow this pattern:

 

Apr 8 12:13:11 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:12:26 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:14:47 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:14:02 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:16:24 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:15:38 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:18:00 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:17:14 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:19:36 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:18:50 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:21:12 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:20:27 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:22:48 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:22:03 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:24:24 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:23:39 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:26:01 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:25:15 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:27:37 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:26:51 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:29:13 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:28:27 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:30:49 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:30:03 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:32:25 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:31:39 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:34:01 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:33:16 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:35:37 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:34:52 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:37:14 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:36:28 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:38:50 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:38:04 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:40:26 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:39:40 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:42:02 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:41:17 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:43:38 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:42:53 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:45:14 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:44:29 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:46:51 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:46:05 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:48:27 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:47:41 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:50:03 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:49:17 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats

 

I am adding a second example, because it is throwing different errors, which may lead to something else. Note that I changed the ipsec retry to 0, which the manual suggests will cause the AP not to reboot if an IPsec tunnel cannot come up. However, they still are rebooting, and as I mentioned above, IPsec is showing disabled on the controller for all APs.

 

(SPIAARUBA01) #show log system all | i 2N-PortN181-9c:1c:12:c2:04:a2
Apr 8 12:12:37 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:11:27 CDT 2014; Unable to set up IPSec tunnel to saved lms, Error:Missed heartbeats
Apr 8 12:13:16 :311020: <ERRS> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 sapd| An internal system error has occurred at file sapd_redun.c function redun_retry_tunnel line 4446 error redun_retry_tunnel: Switching to clear. Error:RC_ERROR_IKEV2_TIMEOUT. Ipsec not successful after reboot.
Apr 8 12:13:36 :311002: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 sapd| Rebooting: SAPD: Rebooting after setting cert_cap=1. Need to open a secure channel(IPSEC)
Apr 8 12:13:37 :303086: <ERRS> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Process Manager (nanny) shutting down - AP will reboot!
Apr 8 12:14:47 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:13:37 CDT 2014; SAPD: Rebooting after setting cert_cap=1. Need to open a secure channel(IPSEC)
Apr 8 12:16:49 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:15:39 CDT 2014; Unable to set up IPSec tunnel to saved lms, Error:Missed heartbeats
Apr 8 12:18:51 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:17:40 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:20:52 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:19:42 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:22:54 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:21:43 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:24:55 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:23:45 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:27:27 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:26:17 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:28:06 :311020: <ERRS> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 sapd| An internal system error has occurred at file sapd_redun.c function redun_retry_tunnel line 4446 error redun_retry_tunnel: Switching to clear. Error:RC_ERROR_IKEP2_PKT1. Ipsec not successful after reboot.
Apr 8 12:29:11 :311002: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 sapd| Rebooting: SAPD: Rebooting after setting cert_cap=1. Need to open a secure channel(IPSEC)
Apr 8 12:29:12 :303086: <ERRS> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Process Manager (nanny) shutting down - AP will reboot!
Apr 8 12:30:22 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:29:12 CDT 2014; SAPD: Rebooting after setting cert_cap=1. Need to open a secure channel(IPSEC)
Apr 8 12:32:25 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:31:14 CDT 2014; Unable to set up IPSec tunnel to saved lms, Error:Missed heartbeats
Apr 8 12:33:14 :311020: <ERRS> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 sapd| An internal system error has occurred at file sapd_redun.c function redun_retry_tunnel line 4446 error redun_retry_tunnel: Switching to clear. Error:RC_ERROR_IKEP2_PKT1. Ipsec not successful after reboot.
Apr 8 12:34:19 :311002: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 sapd| Rebooting: SAPD: Rebooting after setting cert_cap=1. Need to open a secure channel(IPSEC)
Apr 8 12:34:19 :303086: <ERRS> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Process Manager (nanny) shutting down - AP will reboot!
Apr 8 12:35:30 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:34:19 CDT 2014; SAPD: Rebooting after setting cert_cap=1. Need to open a secure channel(IPSEC)
Apr 8 12:37:32 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:36:22 CDT 2014; Unable to set up IPSec tunnel to saved lms, Error:Missed heartbeats
Apr 8 12:39:45 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:38:34 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:40:24 :311020: <ERRS> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 sapd| An internal system error has occurred at file sapd_redun.c function redun_retry_tunnel line 4446 error redun_retry_tunnel: Switching to clear. Error:RC_ERROR_IKEV2_TIMEOUT. Ipsec not successful after reboot.
Apr 8 12:40:44 :311002: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 sapd| Rebooting: SAPD: Rebooting after setting cert_cap=1. Need to open a secure channel(IPSEC)
Apr 8 12:40:44 :303086: <ERRS> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Process Manager (nanny) shutting down - AP will reboot!
Apr 8 12:41:55 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:40:44 CDT 2014; SAPD: Rebooting after setting cert_cap=1. Need to open a secure channel(IPSEC)
Apr 8 12:42:34 :311020: <ERRS> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 sapd| An internal system error has occurred at file sapd_redun.c function redun_retry_tunnel line 4441 error redun_retry_tunnel: Ipsec not successful to saved lms. Error:RC_ERROR_IKEP2_PKT1. rebooting.
Apr 8 12:42:35 :311002: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 sapd| Rebooting: Unable to set up IPSec tunnel to saved lms, Error:RC_ERROR_IKEP2_PKT1
Apr 8 12:42:36 :303086: <ERRS> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Process Manager (nanny) shutting down - AP will reboot!
Apr 8 12:43:51 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Fri Dec 31 16:01:18 PST 1999; Unable to set up IPSec tunnel to saved lms, Error:RC_ERROR_IKEP2_PKT1
Apr 8 12:44:51 :311020: <ERRS> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 sapd| An internal system error has occurred at file sapd_redun.c function redun_retry_tunnel line 4446 error redun_retry_tunnel: Switching to clear. Error:RC_ERROR_IKEP2_PKT1. Ipsec not successful after reboot.
Apr 8 12:45:55 :311002: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 sapd| Rebooting: SAPD: Rebooting after setting cert_cap=1. Need to open a secure channel(IPSEC)
Apr 8 12:45:56 :303086: <ERRS> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Process Manager (nanny) shutting down - AP will reboot!
Apr 8 12:47:07 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:45:56 CDT 2014; SAPD: Rebooting after setting cert_cap=1. Need to open a secure channel(IPSEC)
Apr 8 12:49:09 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:47:58 CDT 2014; Unable to set up IPSec tunnel to saved lms, Error:Missed heartbeats
Apr 8 12:51:12 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:50:01 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:53:14 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:52:03 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
Apr 8 12:55:16 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:54:06 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats

 

Thanks again,

 

Russell

 

 

Contributor I

Re: APs in reboot loop

I seem to have found a common issue. all of the APs that are working are in eth0, all of those rebooting are in eth1. Is there a setting in the controller that could make that happen?

Frequent Contributor II

Re: APs in reboot loop

Interesting... semms to be old and no answer...

 

Any news?

 

IAP/CAP? LLDP things? Uplink switches firmware?

 

Best regards.

Contributor I

Re: APs in reboot loop

ultimately, I physically moved all of the cables from Eth1 to Eth0. That was a resolution until I recently upgraded to 10.4.4.9, and had two groups of APs start rebooting again. One was a remote group over an MPLS cloud, and I added the following to the ap-group:

 

ap system-profile <name>
rap-dhcp-server-id <IP>
rap-dhcp-dns-server <IP>
shell-passwd <pwd>
bkup-passwords <pwd>
!

ap-group "<name>-WIDS"
ap-system-profile <name>

 

The other 'group' of APs were simply connected to a switch in my main building that had a slightly slower link than the rest - a 1Gbps link rather than 2 10Gbps links in an LACP. For those, I increased the bootstrap threshhold, which apparently allows for more missed heartbeats. All APs have been stable since then.

Search Airheads
cancel
Showing results for 
Search instead for 
Did you mean: