Wireless Access

last person joined: yesterday 

Access network design for branch, remote, outdoor, and campus locations with HPE Aruba Networking access points and mobility controllers.
Back to discussions
Expand all | Collapse all

APs in reboot loop

This thread has been viewed 10 times

  • 1.  APs in reboot loop

    Posted Apr 07, 2014 11:30 AM

    first of all, these are all CAPs. I have seen some things about this happening with RAPs, but these are all CAPs. As far as I know, nothing happened on the controller, but 34 of our 64 135s started going into a reboot loop one evening. I have a ticket open with support, but as I said, all of their recommendations are based on RAPs. 

     

    Here is the startup log on an affected AP:

     

    APBoot 1.2.8.1 (build 34939)
    Built: 2012-08-17 at 12:54:18

    Model: AP-13x
    CPU: 88F6560 A0 (DDR3)
    Clock: CPU 1600MHz, L2 533MHz, SysClock 533MHz, TClock 200MHz
    DRAM: 256MB
    POST1: passed
    Flash: 16 MB
    Power: 802.3af POE
    LAN: done
    PHY: done
    PEX 0: RC, link up, x1
    bus.dev fn venID devID class rev MBAR0 MBAR1 MBAR2 MBAR3
    00.00 00 11ab 6560 00005 02 f1000000 00000000 00000000 00000000
    00.01 00 168c 0030 00002 01 90000000 00000000 00000000 00000000
    PEX 1: RC, link up, x1
    bus.dev fn venID devID class rev MBAR0 MBAR1 MBAR2 MBAR3
    01.00 00 11ab 6500 00005 02 f1000000 00000000 00000000 00000000
    01.01 00 168c 0030 00002 01 94000000 00000000 00000000 00000000
    Net: eth0, eth1
    Radio: ar9390#0, ar9390#1

    Hit <Enter> to stop autoboot: 0
    Checking image @ 0xf6100000
    Invalid image format version: 0xffffffff
    Checking image @ 0xf6800000
    Invalid image format version: 0xc
    eth0: up, 1 Gb/s, full duplex
    DHCP broadcast 1
    DHCP broadcast 2
    DHCP IP address: 10.25.0.73
    DHCP subnet mask: 255.255.255.0
    DHCP def gateway: 10.25.0.1
    DHCP DNS server: 10.7.0.112
    DHCP DNS domain: hmcorp.local
    ADP multicast 1
    Controller address: 10.7.0.162
    Using eth0 device
    TFTP from server 10.7.0.162; our IP address is 10.25.0.73; sending through gateway 10.25.0.1
    Filename 'armv5te.ari'.
    Load address: 0x2000000
    Loading: #################################################################
    ##############
    done
    Bytes transferred = 5168276 (4edc94 hex)

    Image is signed; verifying checksum... passed
    Signer Cert OK
    Policy Cert OK
    RSA signature verified.
    Automatic boot of image at addr 0x02000000 ...
    ## Booting image at 02000200 ...
    Uncompressing... done

    Aruba Networks
    ArubaOS Version 6.3.1.1 (build 40563 / label #40563)
    Built by p4build@port-royal on 2013-10-24 at 17:29:27 PDT (gcc version 4.3.3)
    Memory: 256MB = 256MB total
    Memory: 243840KB available (2816K code, 9021K data, 3936K init, 0K highmem)
    Delay calibration in progress:
    Calibrating delay loop... 1597.44 BogoMIPS (lpj=1597440)
    PEX0 interface detected Link X1
    PEX1 interface detected Link X1
    bio: create slab <bio-0> at 0
    wdt: registered with refresh
    Enabling Watchdog
    Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
    i2c /dev entries driver
    i2c-arran: using default base 0xf1018180

    Starting Kernel SHA1 KAT ...Completed Kernel SHA1 KAT
    Starting Kernel HMAC-SHA1 KAT ...Completed Kernel HMAC-SHA1 KAT
    Starting Kernel DES KAT ...Completed Kernel DES KAT
    Starting Kernel AES KAT ...Completed Kernel AES KAT

    Starting Kernel AESGCM KAT ...Completed Kernel AESGCM KAT
    Domain Name: arubanetworks.com
    No panic info available
    init mv_cesa ok
    Ethernet port 1 mode: active-staEthernet Channel Bonding Driver: v3.5.0 (November 4, 2008)
    ndby
    eth0: link up, full duplex, speed 1 Gbps
    ADDRCONF(NETDEV_UP): bond0: link is not ready
    eth0: link up, full duplex, speed 1 Gbps
    bonding: bond0: making interface eth0 the new active one.
    bonding: bond0: first active interface up!
    bonding: bond0: enslaving eth0 as an active interface with an up link.
    ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
    eth1: link down
    ADDRCONF(NETDEV_UP): eth1: link is not ready
    eth1: link down
    bonding: bond0: enslaving eth1 as a backup interface with a down link.
    AP xml model 51, num_radios 2 (jiffies 2617)
    init_asap_mod: installation:0
    radio 0: band 1 ant 0 max_ssid 16
    radio 1: band 0 ant 0 max_ssid 16
    ethernet_device_event: dev eth0 is up
    eth0: link up, full duplex, speed 1 Gbps
    eth0: link up, full duplex, speed 1 Gbps
    Shutting down eth1 due to insufficient POE voltage [power profile 2]
    Starting watchdog process...
    Getting an IP address...
    10.25.0.73 255.255.255.0 10.25.0.1
    Running ADP...Done. Master is 10.7.0.162
    ath_hal: 0.9.17.1 (AR5416, AR9380, REGOPS_FUNC, PRIVATE_DIAG, WRITE_EEPROM, 11D)
    ath_rate_atheros: Copyright (c) 2001-2005 Atheros Communications, Inc, All Rights Reserved
    ath_rate_atheros: Aruba Networks Rate Control Algorithm
    ath_dfs: Version 2.0.0
    Copyright (c) 2005-2006 Atheros Communications, Inc. All Rights Reserved
    ath_spectrum: Version 2.0.0
    Copyright (c) 2005-2006 Atheros Communications, Inc. All Rights Reserved
    ath_dev: Copyright (c) 2001-2007 Atheros Communications, Inc, All Rights Reserved
    ath_pci: 0.9.4.5 (Atheros/multi-bss)
    ath_attach: scn cd3c0320 sc cd6c0000 ah cd340000
    wifi0: Base BSSID 9c:1c:12:a3:25:b0, 16 available BSSID(s)
    eth0 address=9c:1c:12:c2:32:5a
    br0 address=9c:1c:12:c2:32:5a
    wifi0: AP type AP-135, radio 0, max_bssids 16
    wifi0: Atheros 9380: mem=0xf3000000, irq=25 hw_base=0xd19c0000
    ath_attach: scn cc820320 sc cc840000 ah cc880000
    wifi1: Base BSSID 9c:1c:12:a3:25:a0, 16 available BSSID(s)
    eth0 address=9c:1c:12:c2:32:5a
    br0 address=9c:1c:12:c2:32:5a
    wifi1: AP type AP-135, radio 1, max_bssids 16
    wifi1: Atheros 9380: mem=0xf4000000, irq=24 hw_base=0xd1dc0000
    ath_ahb: 0.9.4.5 (Atheros/multi-bss)
    ath_ahb: No devices found, driver not installed.

    Starting FIPS KAT ... Completed FIPS KAT

    AP rebooted Fri Dec 31 16:04:22 PST 1999; SAPD: Unable to contact switch: HELLO-TIMEOUT. Last Ctrl msg: HELLO len=920 dest=10.7.0.162 tries=10 seq=0
    shutting down watchdog process (nanny will restart it)...

    <<<<< Welcome to the Access Point >>>>>

    ~ # Enabling eth1 due to LLDP power grant [power profile 1]

     

     

    Here is the AP pinging the controller:

     

    ~ # ping 10.7.0.162
    PING 10.7.0.162 (10.7.0.162): 56 data bytes
    64 bytes from 10.7.0.162: icmp_seq=0 ttl=63 time=0.8 ms
    64 bytes from 10.7.0.162: icmp_seq=1 ttl=63 time=0.7 ms
    64 bytes from 10.7.0.162: icmp_seq=2 ttl=63 time=0.7 ms
    64 bytes from 10.7.0.162: icmp_seq=3 ttl=63 time=0.7 ms
    64 bytes from 10.7.0.162: icmp_seq=4 ttl=63 time=0.7 ms
    64 bytes from 10.7.0.162: icmp_seq=5 ttl=63 time=0.7 ms
    64 bytes from 10.7.0.162: icmp_seq=6 ttl=63 time=0.7 ms
    64 bytes from 10.7.0.162: icmp_seq=7 ttl=63 time=0.7 ms
    64 bytes from 10.7.0.162: icmp_seq=8 ttl=63 time=0.7 ms
    64 bytes from 10.7.0.162: icmp_seq=9 ttl=63 time=0.9 ms
    64 bytes from 10.7.0.162: icmp_seq=10 ttl=63 time=0.7 ms
    64 bytes from 10.7.0.162: icmp_seq=11 ttl=63 time=1.0 ms
    64 bytes from 10.7.0.162: icmp_seq=12 ttl=63 time=0.7 ms
    64 bytes from 10.7.0.162: icmp_seq=13 ttl=63 time=0.7 ms
    64 bytes from 10.7.0.162: icmp_seq=14 ttl=63 time=12.0 ms
    64 bytes from 10.7.0.162: icmp_seq=15 ttl=63 time=0.9 ms
    64 bytes from 10.7.0.162: icmp_seq=16 ttl=63 time=0.7 ms
    64 bytes from 10.7.0.162: icmp_seq=17 ttl=63 time=0.8 ms
    64 bytes from 10.7.0.162: icmp_seq=18 ttl=63 time=0.9 ms
    ^C
    --- 10.7.0.162 ping statistics ---
    19 packets transmitted, 19 packets received, 0% packet loss
    round-trip min/avg/max = 0.7/1.3/12.0 ms

    ~ #

     

    and finally:

     

    (SPIAARUBA01) # show datapath session table 10.25.0.73


    Datapath Session Table Entries
    ------------------------------

    Flags: F - fast age, S - src NAT, N - dest NAT
    D - deny, R - redirect, Y - no syn
    H - high prio, P - set prio, T - set ToS
    C - client, M - mirror, V - VOIP
    Q - Real-Time Quality analysis
    I - Deep inspect, U - Locally destined
    E - Media Deep Inspect, G - media signal

    Source IP Destination IP Prot SPort DPort Cntr Prio ToS Age Destination TAge Packets Bytes Flags
    -------------- -------------- ---- ----- ----- ---- ---- --- --- ----------- ---- --------- --------- -----
    10.25.0.73 10.7.0.162 17 8211 8222 0/0 0 0 1 1/2 f 0 0 FYCI
    10.25.0.73 10.7.0.162 17 8211 8211 0/0 0 0 1 1/2 f 0 0 FCI
    10.7.0.162 10.25.0.73 17 8211 8211 0/0 0 0 1 1/2 f 0 0 FYI
    10.7.0.162 10.25.0.73 17 8222 8211 0/0 0 0 1 1/2 f 0 0 FYI

     

    Any ideas what I can do?

     

    Russell

     



  • 2.  RE: APs in reboot loop

    EMPLOYEE
    Posted Apr 07, 2014 11:37 AM

    1.  Find out what ap-group that access point is in.

    2.  Find out if in the AP system profile in that AP-Group if there is an LMS-ip.

    3.  If that LMS-IP does not point to a controller that is active, that access point is being orphaned.  To solve, you either need to remove that ip address or change it to one that points to an active controller.

     



  • 3.  RE: APs in reboot loop

    Posted Apr 07, 2014 12:00 PM

    Thanks, for the response. LMS-ip not being used. All of the affected APs are in one of three ap-groups:

     

    ap-group "HMN-RAL-WIDS"
    virtual-ap "RAL_CLAIMS_GUEST-vap_prof"
    virtual-ap "RAL_INTERNAL-vap_prof"
    virtual-ap "RAL_EMPLOYEE-vap"
    dot11a-radio-profile "Spectrum-Mode-a"
    dot11g-radio-profile "Spectrum-Mode-bg"
    ids-profile "RAL-WIPS"
    !
    ap-group "HMN-SPI-LC-WIDS"
    virtual-ap "SPI_INTERNAL_VAP"
    virtual-ap "SPI_EMPLOYEE_VAP"
    virtual-ap "QA-TESTER_VAP"
    virtual-ap "SPI_GUEST_VAP"
    virtual-ap "SPI_PHONE_VAP"
    dot11a-radio-profile "Spectrum-Mode-a"
    dot11g-radio-profile "Spectrum-Mode-bg"
    ids-profile "HMN-SPI-WIPS"
    !
    ap-group "HMN-SPI-WIDS"
    virtual-ap "SPI_INTERNAL_VAP"
    virtual-ap "SPI_EMPLOYEE_VAP"
    virtual-ap "QA-TESTER_VAP"
    virtual-ap "SPI_GUEST_VAP"
    virtual-ap "SPI_PHONE_VAP"
    virtual-ap "SPI_EXT_TEST_VAP"
    dot11a-radio-profile "Spectrum-Mode-a"
    dot11g-radio-profile "Spectrum-Mode-bg"
    ids-profile "HMN-SPI-WIPS"

     

    which use the default system-profile:

     

    ap system-profile "default"
    !

     

    what next?



  • 4.  RE: APs in reboot loop

    EMPLOYEE
    Posted Apr 07, 2014 12:05 PM

    I would get the output of "show log system 100" to see if you get any clues.



  • 5.  RE: APs in reboot loop

    Posted Apr 07, 2014 12:29 PM

    (SPIAARUBA01) # show log system 100

    Apr 7 11:17:36 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:17:37 :303022: <WARN> |AP GLN-PortN-123-9c:1c:12:c2:04:72@10.25.0.207 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:16:51 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:17:37 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:17:38 :303022: <WARN> |AP GLSSW-PortS289-9c:1c:12:c2:32:5e@10.25.0.58 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:16:52 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:17:38 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:18:10 :303022: <WARN> |AP 3W-PortN191-9c:1c:12:c2:2f:d2@10.25.0.30 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:17:00 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:18:18 :303022: <WARN> |AP GLSSE-PortS292-9c:1c:12:c2:32:78@10.25.0.52 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:17:32 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:18:20 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:18:23 :303022: <WARN> |AP 2S-PortS174-9c:1c:12:c2:32:1e@10.25.0.61 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:17:12 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:18:27 :303022: <WARN> |AP RAL-WEST-d8:c7:c8:cc:6e:92@10.70.21.146 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:17:17 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:18:27 :303022: <WARN> |AP RAL-EAST-d8:c7:c8:cc:6e:e8@10.70.21.96 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:17:15 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:18:33 :303022: <WARN> |AP 6N-MID-PortN147-9c:1c:12:c2:04:82@10.25.0.23 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:17:48 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:18:38 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:18:46 :303022: <WARN> |AP GLS-Port291-9c:1c:12:c2:32:6c@10.25.0.51 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:18:01 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:18:50 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:18:53 :303022: <WARN> |AP RAL-EAST-d8:c7:c8:cc:6e:ce@10.70.21.151 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:17:42 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:18:53 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:18:54 :303022: <WARN> |AP RAL-WEST-d8:c7:c8:cc:6e:84@10.70.21.140 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:17:42 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:18:55 :303022: <WARN> |AP LC-Port42-9c:1c:12:c2:04:aa@10.25.0.60 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:18:10 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:18:56 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:18:58 :303022: <WARN> |AP LC-Port41-9c:1c:12:c2:04:b0@10.25.0.20 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:18:13 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:19:00 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:19:10 :303022: <WARN> |AP GLS-Port293-9c:1c:12:c2:32:66@10.25.0.41 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:18:25 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:19:14 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:19:35 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:18:25 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:19:35 :303022: <WARN> |AP 6NE-PortN146-9c:1c:12:c2:32:02@10.25.0.26 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:18:50 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:19:35 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:19:56 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.43 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:19:10 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:20:03 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:20:07 :303022: <WARN> |AP PC01-9c:1c:12:c2:32:32@10.25.0.62 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:19:21 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:20:07 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:20:34 :303022: <WARN> |AP 2W-PortN182-9c:1c:12:c2:04:66@10.25.0.65 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:19:24 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:20:35 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:20:40 :303022: <WARN> |AP 6NW-PortN149-9c:1c:12:c2:31:e2@10.25.0.59 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:19:55 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:20:42 :303022: <WARN> |AP PC-Port170-9c:1c:12:c2:04:7a@10.25.0.40 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:19:57 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:20:43 :303022: <WARN> |AP 4N-Port165-9c:1c:12:c2:32:2e@10.25.0.37 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:19:33 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:20:50 :303022: <WARN> |AP GLN-Port122-9c:1c:12:c2:32:60@10.25.0.25 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:20:05 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:20:51 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:21:03 :303022: <WARN> |AP LC-Port21-9c:1c:12:c2:04:bc@10.25.0.63 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:20:17 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:21:05 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:21:07 :303022: <WARN> |AP 4N-Port164-9c:1c:12:c2:32:36@10.25.0.32 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:19:57 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:21:08 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:21:08 :303022: <WARN> |AP 3S-Port175-9c:1c:12:c2:32:3a@10.25.0.42 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:20:23 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:21:09 :303022: <WARN> |AP 6W-PortN148-9c:1c:12:c2:04:9c@10.25.0.66 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:20:23 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:21:09 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:21:34 :303022: <WARN> |AP 5S-PortS159-9c:1c:12:c2:04:70@10.25.0.27 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:20:23 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:21:35 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:21:41 :303022: <WARN> |AP GLN-PortN-123-9c:1c:12:c2:04:72@10.25.0.22 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:20:55 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:21:42 :303022: <WARN> |AP GLSSW-PortS289-9c:1c:12:c2:32:5e@10.25.0.58 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:20:56 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:21:50 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:22:21 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:22:22 :303022: <WARN> |AP GLSSE-PortS292-9c:1c:12:c2:32:78@10.25.0.52 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:21:36 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:22:22 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:22:37 :303022: <WARN> |AP 6N-MID-PortN147-9c:1c:12:c2:04:82@10.25.0.23 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:21:52 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:22:40 :303022: <WARN> |AP 3W-PortN191-9c:1c:12:c2:2f:d2@10.25.0.30 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:21:30 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:22:50 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:22:50 :303022: <WARN> |AP GLS-Port291-9c:1c:12:c2:32:6c@10.25.0.51 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:22:05 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:22:50 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:22:53 :303022: <WARN> |AP 2S-PortS174-9c:1c:12:c2:32:1e@10.25.0.61 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:21:42 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:22:57 :303022: <WARN> |AP RAL-WEST-d8:c7:c8:cc:6e:92@10.70.21.146 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:21:47 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:22:58 :303022: <WARN> |AP RAL-EAST-d8:c7:c8:cc:6e:e8@10.70.21.96 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:21:47 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:23:00 :303022: <WARN> |AP LC-Port42-9c:1c:12:c2:04:aa@10.25.0.60 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:22:14 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:23:02 :303022: <WARN> |AP LC-Port41-9c:1c:12:c2:04:b0@10.25.0.20 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:22:17 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:23:04 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:23:22 :303022: <WARN> |AP RAL-EAST-d8:c7:c8:cc:6e:ce@10.70.21.151 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:22:12 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:23:22 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:23:24 :303022: <WARN> |AP RAL-WEST-d8:c7:c8:cc:6e:84@10.70.21.140 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:22:14 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:23:26 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:23:27 :303022: <WARN> |AP GLS-Port293-9c:1c:12:c2:32:66@10.25.0.41 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:22:41 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:23:27 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:23:40 :303022: <WARN> |AP 6NE-PortN146-9c:1c:12:c2:32:02@10.25.0.26 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:22:54 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:23:40 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:24:00 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.43 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:23:14 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:24:05 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:22:55 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:24:06 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:24:12 :303022: <WARN> |AP PC01-9c:1c:12:c2:32:32@10.25.0.62 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:23:26 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:24:20 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:24:45 :303022: <WARN> |AP 6NW-PortN149-9c:1c:12:c2:31:e2@10.25.0.59 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:23:59 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:24:46 :304055: <ERRS> |AP DAL-WN-d8:c7:c8:cc:6e:bc@10.70.20.146 stm| |ap| Unexpected stm (Station management) runtime error at bridge_user_del, 11004, bridge_user_del: bridge user doesn't exist(1) ip:10.70.20.84
    Apr 7 11:24:46 :304055: <ERRS> |AP DAL-WN-d8:c7:c8:cc:6e:bc@10.70.20.146 stm| |ap| Unexpected stm (Station management) runtime error at bridge_user_del_from_hash, 11036, bridge_user_del_from_hash: bridge_user_del returned err
    Apr 7 11:24:46 :304055: <ERRS> |AP DAL-WN-d8:c7:c8:cc:6e:bc@10.70.20.146 stm| |ap| Unexpected stm (Station management) runtime error at bridge_user_del, 11004, bridge_user_del: bridge user doesn't exist(1) ip:10.70.20.84
    Apr 7 11:24:46 :303022: <WARN> |AP PC-Port170-9c:1c:12:c2:04:7a@10.25.0.40 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:24:01 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:24:46 :304055: <ERRS> |AP DAL-WN-d8:c7:c8:cc:6e:bc@10.70.20.146 stm| |ap| Unexpected stm (Station management) runtime error at bridge_user_del_from_hash, 11036, bridge_user_del_from_hash: bridge_user_del returned err
    Apr 7 11:24:50 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:24:55 :303022: <WARN> |AP GLN-Port122-9c:1c:12:c2:32:60@10.25.0.25 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:24:09 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:25:04 :303022: <WARN> |AP 2W-PortN182-9c:1c:12:c2:04:66@10.25.0.65 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:23:54 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:25:07 :303022: <WARN> |AP LC-Port21-9c:1c:12:c2:04:bc@10.25.0.63 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:24:22 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:25:12 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:25:13 :303022: <WARN> |AP 6W-PortN148-9c:1c:12:c2:04:9c@10.25.0.66 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:24:27 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:25:13 :303022: <WARN> |AP 4N-Port165-9c:1c:12:c2:32:2e@10.25.0.37 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:24:03 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:25:14 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:25:25 :303022: <WARN> |AP 3S-Port175-9c:1c:12:c2:32:3a@10.25.0.42 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:24:40 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:25:32 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:25:37 :303022: <WARN> |AP 4N-Port164-9c:1c:12:c2:32:36@10.25.0.32 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:24:27 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:25:37 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:25:45 :303022: <WARN> |AP GLN-PortN-123-9c:1c:12:c2:04:72@10.25.0.22 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:25:00 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:25:47 :303022: <WARN> |AP GLSSW-PortS289-9c:1c:12:c2:32:5e@10.25.0.58 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:25:01 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:25:53 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663
    Apr 7 11:26:04 :303022: <WARN> |AP 5S-PortS159-9c:1c:12:c2:04:70@10.25.0.27 nanny| Reboot Reason: AP rebooted Mon Apr 7 11:24:53 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 7 11:26:06 :301134: <WARN> |snmp| SNMP V3 Message parse error: Not in life time window failure: Possible Privacy password mismatch. 663

     

    unable to set up IPSec tunnel , but why all of a sudden? and only on some? And on the same switch stack, some APs are fine and some are rebooting?



  • 6.  RE: APs in reboot loop

    EMPLOYEE
    Posted Apr 07, 2014 10:38 PM

    Hi regibbons,

     

    Possible causes of such a log may be

    >> problem with isakmpd at controller

    >> ipsec rekeying issue

    >> routing problem

    >> dupe IP on network

     

    some thoughts on what to check

     

    >> go into the AP shell and take a look in /tmp/rapper.txt - this is the ipsec client on the APs log file (it's name is a hangover from RAP days before CPSEC)

    >> make sure the isakmpd is healthy on the controller (check number of SAs, cpu load, check stats)

    >> make sure that the def gw, AP IP and controller IP are not being duped by another host

     

    Questions

    >> does every AP always come up then at some time later it fails ?

    >> once an AP reboots, does it come up straight away, or takes some time ?

     


    regards

    -jeff



  • 7.  RE: APs in reboot loop

    Posted Apr 08, 2014 02:01 PM

    Thanks for the reply, I will try to adress everything you mentioned.

    I am not not sure how to check isakmpd or ipsec rekeying. As far as a routing problem, in at least one case I have two APs on the same switch stack where one is fine and the other is rebooting. IPsec is disabled according to the controller on all of the ap-groups in question. As far as I can tell, there is no dupe of the controller IP on the network.

     

    I'm going to get a failing AP out of the ceiling and see what I can learn from it. Again, not sure how to chek number of SAs, or stats, but the cpu load is 4%

     

    there are ~34 APs being affected by this and ~30 that are not. the 34 that are affected reboot in the pattern below. This is one example, but they all follow this pattern:

     

    Apr 8 12:13:11 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:12:26 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:14:47 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:14:02 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:16:24 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:15:38 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:18:00 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:17:14 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:19:36 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:18:50 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:21:12 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:20:27 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:22:48 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:22:03 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:24:24 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:23:39 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:26:01 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:25:15 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:27:37 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:26:51 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:29:13 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:28:27 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:30:49 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:30:03 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:32:25 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:31:39 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:34:01 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:33:16 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:35:37 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:34:52 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:37:14 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:36:28 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:38:50 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:38:04 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:40:26 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:39:40 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:42:02 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:41:17 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:43:38 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:42:53 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:45:14 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:44:29 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:46:51 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:46:05 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:48:27 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:47:41 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:50:03 :303022: <WARN> |AP GLS-Port290-9c:1c:12:c2:32:72@10.25.0.56 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:49:17 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats

     

    I am adding a second example, because it is throwing different errors, which may lead to something else. Note that I changed the ipsec retry to 0, which the manual suggests will cause the AP not to reboot if an IPsec tunnel cannot come up. However, they still are rebooting, and as I mentioned above, IPsec is showing disabled on the controller for all APs.

     

    (SPIAARUBA01) #show log system all | i 2N-PortN181-9c:1c:12:c2:04:a2
    Apr 8 12:12:37 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:11:27 CDT 2014; Unable to set up IPSec tunnel to saved lms, Error:Missed heartbeats
    Apr 8 12:13:16 :311020: <ERRS> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 sapd| An internal system error has occurred at file sapd_redun.c function redun_retry_tunnel line 4446 error redun_retry_tunnel: Switching to clear. Error:RC_ERROR_IKEV2_TIMEOUT. Ipsec not successful after reboot.
    Apr 8 12:13:36 :311002: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 sapd| Rebooting: SAPD: Rebooting after setting cert_cap=1. Need to open a secure channel(IPSEC)
    Apr 8 12:13:37 :303086: <ERRS> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Process Manager (nanny) shutting down - AP will reboot!
    Apr 8 12:14:47 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:13:37 CDT 2014; SAPD: Rebooting after setting cert_cap=1. Need to open a secure channel(IPSEC)
    Apr 8 12:16:49 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:15:39 CDT 2014; Unable to set up IPSec tunnel to saved lms, Error:Missed heartbeats
    Apr 8 12:18:51 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:17:40 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:20:52 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:19:42 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:22:54 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:21:43 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:24:55 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:23:45 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:27:27 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:26:17 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:28:06 :311020: <ERRS> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 sapd| An internal system error has occurred at file sapd_redun.c function redun_retry_tunnel line 4446 error redun_retry_tunnel: Switching to clear. Error:RC_ERROR_IKEP2_PKT1. Ipsec not successful after reboot.
    Apr 8 12:29:11 :311002: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 sapd| Rebooting: SAPD: Rebooting after setting cert_cap=1. Need to open a secure channel(IPSEC)
    Apr 8 12:29:12 :303086: <ERRS> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Process Manager (nanny) shutting down - AP will reboot!
    Apr 8 12:30:22 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:29:12 CDT 2014; SAPD: Rebooting after setting cert_cap=1. Need to open a secure channel(IPSEC)
    Apr 8 12:32:25 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:31:14 CDT 2014; Unable to set up IPSec tunnel to saved lms, Error:Missed heartbeats
    Apr 8 12:33:14 :311020: <ERRS> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 sapd| An internal system error has occurred at file sapd_redun.c function redun_retry_tunnel line 4446 error redun_retry_tunnel: Switching to clear. Error:RC_ERROR_IKEP2_PKT1. Ipsec not successful after reboot.
    Apr 8 12:34:19 :311002: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 sapd| Rebooting: SAPD: Rebooting after setting cert_cap=1. Need to open a secure channel(IPSEC)
    Apr 8 12:34:19 :303086: <ERRS> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Process Manager (nanny) shutting down - AP will reboot!
    Apr 8 12:35:30 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:34:19 CDT 2014; SAPD: Rebooting after setting cert_cap=1. Need to open a secure channel(IPSEC)
    Apr 8 12:37:32 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:36:22 CDT 2014; Unable to set up IPSec tunnel to saved lms, Error:Missed heartbeats
    Apr 8 12:39:45 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:38:34 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:40:24 :311020: <ERRS> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 sapd| An internal system error has occurred at file sapd_redun.c function redun_retry_tunnel line 4446 error redun_retry_tunnel: Switching to clear. Error:RC_ERROR_IKEV2_TIMEOUT. Ipsec not successful after reboot.
    Apr 8 12:40:44 :311002: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 sapd| Rebooting: SAPD: Rebooting after setting cert_cap=1. Need to open a secure channel(IPSEC)
    Apr 8 12:40:44 :303086: <ERRS> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Process Manager (nanny) shutting down - AP will reboot!
    Apr 8 12:41:55 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:40:44 CDT 2014; SAPD: Rebooting after setting cert_cap=1. Need to open a secure channel(IPSEC)
    Apr 8 12:42:34 :311020: <ERRS> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 sapd| An internal system error has occurred at file sapd_redun.c function redun_retry_tunnel line 4441 error redun_retry_tunnel: Ipsec not successful to saved lms. Error:RC_ERROR_IKEP2_PKT1. rebooting.
    Apr 8 12:42:35 :311002: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 sapd| Rebooting: Unable to set up IPSec tunnel to saved lms, Error:RC_ERROR_IKEP2_PKT1
    Apr 8 12:42:36 :303086: <ERRS> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Process Manager (nanny) shutting down - AP will reboot!
    Apr 8 12:43:51 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Fri Dec 31 16:01:18 PST 1999; Unable to set up IPSec tunnel to saved lms, Error:RC_ERROR_IKEP2_PKT1
    Apr 8 12:44:51 :311020: <ERRS> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 sapd| An internal system error has occurred at file sapd_redun.c function redun_retry_tunnel line 4446 error redun_retry_tunnel: Switching to clear. Error:RC_ERROR_IKEP2_PKT1. Ipsec not successful after reboot.
    Apr 8 12:45:55 :311002: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 sapd| Rebooting: SAPD: Rebooting after setting cert_cap=1. Need to open a secure channel(IPSEC)
    Apr 8 12:45:56 :303086: <ERRS> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Process Manager (nanny) shutting down - AP will reboot!
    Apr 8 12:47:07 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:45:56 CDT 2014; SAPD: Rebooting after setting cert_cap=1. Need to open a secure channel(IPSEC)
    Apr 8 12:49:09 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:47:58 CDT 2014; Unable to set up IPSec tunnel to saved lms, Error:Missed heartbeats
    Apr 8 12:51:12 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:50:01 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:53:14 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:52:03 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats
    Apr 8 12:55:16 :303022: <WARN> |AP 2N-PortN181-9c:1c:12:c2:04:a2@10.25.0.55 nanny| Reboot Reason: AP rebooted Tue Apr 8 12:54:06 CDT 2014; Unable to set up IPSec tunnel, Error:Missed heartbeats

     

    Thanks again,

     

    Russell

     

     



  • 8.  RE: APs in reboot loop

    Posted Apr 08, 2014 03:48 PM

    I seem to have found a common issue. all of the APs that are working are in eth0, all of those rebooting are in eth1. Is there a setting in the controller that could make that happen?



  • 9.  RE: APs in reboot loop

    Posted Sep 12, 2016 03:11 PM

    Interesting... semms to be old and no answer...

     

    Any news?

     

    IAP/CAP? LLDP things? Uplink switches firmware?

     

    Best regards.



  • 10.  RE: APs in reboot loop

    Posted Sep 12, 2016 04:12 PM

    ultimately, I physically moved all of the cables from Eth1 to Eth0. That was a resolution until I recently upgraded to 10.4.4.9, and had two groups of APs start rebooting again. One was a remote group over an MPLS cloud, and I added the following to the ap-group:

     

    ap system-profile <name>
    rap-dhcp-server-id <IP>
    rap-dhcp-dns-server <IP>
    shell-passwd <pwd>
    bkup-passwords <pwd>
    !

    ap-group "<name>-WIDS"
    ap-system-profile <name>

     

    The other 'group' of APs were simply connected to a switch in my main building that had a slightly slower link than the rest - a 1Gbps link rather than 2 10Gbps links in an LACP. For those, I increased the bootstrap threshhold, which apparently allows for more missed heartbeats. All APs have been stable since then.