Wireless Access

 View Only
Expand all | Collapse all

7205 Mobility Controller stuck in a rebooting loop - Reboot Cause: Nanny rebooted machine - fpapps process died

This thread has been viewed 18 times
  • 1.  7205 Mobility Controller stuck in a rebooting loop - Reboot Cause: Nanny rebooted machine - fpapps process died

    Posted Dec 16, 2021 12:09 AM
    Edited by Greg_W Dec 17, 2021 10:13 AM
    Background
    Inherited a single mobility controller deployement with 20 APs. Currently attempting to resolve licensing/support contract transfer, not looking like it will happen quickly so posting here for any ideas in the meantime. Any help is appreciated, thank you.

    Cause
    Changed the Controller Management IP. Prompted to reboot and accepted. The system is now in an endless reboot cycle. Unable to connect to any IP ports, only console.

    Console Output
    Stage 1 Loader 1.0.5.0-FIPS (build 54487)
    Built: 2016-04-05 at 06:52:00

    Secure Boot Enabled on the Processor

    Bank: Primary
    CPLD:  rev: 3.0 (0x10:002c)
    PRID: 000C1104
    Initialized I2C0 Controller.
    Initialized I2C1 Controller.
    SPD Rev:0x13 DIMM:0 Type:2 Speed:666MHz #Rank:1
    DDR3: Node 0 Channel 0 Mem size = 4096 MB UDIMM
    SPD Rev:0x13 DIMM:0 Type:2 Speed:666MHz #Rank:1
    DDR3: Node 0 Channel 1 Mem size = 4096 MB UDIMM
    Ref Clk @133MHz
    DDR3: Node 0 DRAM frequency 666 MHz
    DDR3: Node 0 CPU frequency 1200 MHz
    mtb_ps:125 clock:1500 trc:33 trcd:9 trp:9
    AP3:A0CFFF0 ODTP1:10000
    Board DDR VDD set to 1.5V.
    N:0 CH:0 additional rdlvl rdly:1
    N:0 Ch:0 m:32 s:32 R OK.
    Rank:0 HW WLVL Passed Mask:1FF
    AP3:A0CFFF0 ODTP1:10000
    n:0 ch:0 RTT WR:0200
    ===N:0 Ch:0 m:32 s:32 RW OK.
    Node:0 Ch:0 TGE Set Memory:4096 MB value:FF FF -- PASS
    Node:0 Ch:0 TGE Set Memory:4096 MB value:FF 00 -- PASS
    Node:0 Ch:0 TGE Set Memory:4096 MB value:AA 55 -- PASS
    Node:0 Ch:0 TGE Set Memory:4096 MB value:00 00 -- PASS
    mtb_ps:125 clock:1500 trc:33 trcd:9 trp:9
    AP3:A0CFFF0 ODTP1:10000
    Board DDR VDD set to 1.5V.
    N:0 CH:1 additional rdlvl rdly:1
    N:0 Ch:1 m:32 s:32 R OK.
    Rank:0 HW WLVL Passed Mask:1FF
    AP3:A0CFFF0 ODTP1:10000
    n:0 ch:1 RTT WR:0200
    ===N:0 Ch:1 m:32 s:32 RW OK.
    Node:0 Ch:1 TGE Set Memory:4096 MB value:FF FF -- PASS
    Node:0 Ch:1 TGE Set Memory:4096 MB value:FF 00 -- PASS
    Node:0 Ch:1 TGE Set Memory:4096 MB value:AA 55 -- PASS
    Node:0 Ch:1 TGE Set Memory:4096 MB value:00 00 -- PASS

    DDR3 Initialization Passed.
    NBU0 DRAM BAR0 base: 00000000 limit: 0013f000 xlate: 0000000b node: 00000000 (                                                                                                                         0 MB ->   320 MB, size:   320 MB)
    NBU0 DRAM BAR1 base: 001d0000 limit: 00bff000 xlate: 0009000b node: 00000000 (                                                                                                                       464 MB ->  3072 MB, size:  2608 MB)
    NBU0 DRAM BAR2 base: 00e00000 limit: 0228f000 xlate: 0029000b node: 00000000 ( 3                                                                                                                     584 MB ->  8848 MB, size:  5264 MB)


    CPBoot 1.0.5.0-FIPS (build 54487)
    Built: 2016-04-05 at 06:39:01

    DRAM:  8 GB
    Detected [XLP316 Rev B2 (Secure Boot) ]
    CPLD:      rev: 3.0
    ispPAC:    rev: 0.9
    SW CPLD:   rev: 1.d
    SW ispPAC: rev: 0.7
    Flash: 32 MB
    PCIE (B0:D01:F0) : Link up (Gen(1))
    PCIE (B0:D01:F1) : No Link.
    PCIE (B0:D01:F2) : Link up (Gen(1))
    PCIE (B0:D01:F3) : Link up (Gen(2))
    Bank:  Primary
    Board: A7205
    CPU:   XLP316 Rev B2 (Secure Boot)
    Clock: Core 1200 MHz / SoC 2500 MHz (43f91fe2)
    Reboot code: 0:2:16:54:39 (002c)
    Net:   ge-0, ge-1, ge-2
    FIPS POST: PASS
    Inventory Verification: PASS
    Hit Ctrl + X key to stop autoboot:  0
    USB(0):   2 USB Device(s) found
    1 Storage Device(s) found
    Loading image 0:0########################################
    Image is signed; verifying checksum...
    passed
    Signer Cert OK
    Policy Cert OK
    RSA signature verified.
    [    0.000000]   0:xlp_napi_vc_mask 0xf
    [    0.000000]   0:sae frequency is 500
    [    0.000000]   0:-- SAE Frequency set to 500
    [    0.000000]   0:SAE Frequency set to 500MHz
    [    0.000000]   0:MSGRING_NAPI: Initializing NLM NAPI subsystem
    [16:55:09]:...Starting rcS...

    Aruba Networks
    ArubaOS Version 8.6.0.7-FIPS (build 78216 / label #78216)
    Built by p4build@pr-hpn-build08 on 2020-12-11 at 14:22:21 UTC (gcc version 4.4.5                                                                                                                     )
    (c) Copyright 2020 Hewlett Packard Enterprise Development LP.

               <<<<<    Welcome to Aruba Networks - Aruba A7205    >>>>>

    [16:55:09]:Probing for EEPROM devices                 [ OK ]
    [16:55:09]:Probing for real-time clock                [ OK ]
    [16:55:09]:Initializing LCD module                    [ OK ]
    [16:55:09]:Uncompressing core image files             [ OK ]
    [16:55:30]:Extracting corefs                          [ OK ]

    [16:55:31]:Enabling watchdog                          [ OK ]
    [16:55:31]:Starting device manager

    Performing eUSB Flash fast test...                    [ DONE ]
        [ OK ]
    [16:55:43]:Mounting flash                             [ OK ]
    [16:55:44]:Initializing 1GB as swap on zRam0          [ OK ]
    [16:55:47]:Turning swap ON on zRAM0                   [ OK ]
    [16:55:48]:Checking system inventory                  [ OK ]
    [16:55:48]:Installing ancillary FS                    [ OK ]
    Performing integrity check on ancillary partition 0   [ OK ]
     Extracting Webui files../

    [16:56:09]:Reboot Cause: Nanny rebooted machine - fpapps process died (Intent:ca                                                                                                                     use:register 34:86:0:2c)
    [16:56:09]:Crash information available.
    [16:56:09]:Starting syslog service                    [ OK ]
    [16:56:11]:Restoring the database                     [ OK ]
    [16:56:18]:Generating SSH keys                        [ OK ]
    [16:56:19]:Initializing TPM and certificates          [ OK ]
    [16:56:50]:Checking for configuration upgrade         [ OK ]
    [16:56:52]:Installing crash kernel                    [ OK ]


    [16:56:52]:rcS Done(103 sec)

    [16:56:52]:Starting OS services                       [ OK ]



    [16:56:57]:Initializing GSM                           [ DONE ]
    [16:56:59]:Initializing CCM                           Starting FIPS Aruba Crypto                                                                                                                     graphic KAT test
    Completed FIPS Aruba Cryptographic KAT test successfully.
    [ DONE ]
    [16:57:43]:Initializing FPAPPs                        Starting OpenSSL FIPS KAT                                                                                                                      test
    Completed OpenSSL FIPS KAT test successfully.
    Successfully started XLP FIPS KAT test.


    [16:58:32]:Starting rebootme

    [16:58:32]:Shutdown processing started
    [16:58:37]:Starting database backup
    [16:58:37]:Syncing data...
    .........................

    [16:59:04]:done.
    [16:59:04]:Shutting down database server
    [16:59:10]:Starting Time sync
    [16:59:10]:Time sync [Done]
    [16:59:10]:kill all process
    [16:59:12]:Sending SIGSTP to all processes, except init process
    [16:59:12]:Sending SIGKILL to all processes, except init process
    [16:59:13]:kill all process again
    [16:59:13]:Running fsck  fsck.ext3 -n /dev/sda3 ...Flash
    Please stand by while rebooting the system.
    OS Info
    cpboot> osinfo
    Default boot @ device:0 partion:0
    USB(0):   2 USB Device(s) found
    1 Storage Device(s) found

    Partition 0:
     Reading image...........................................done
        image type: 0
      machine type: 18
              size: 89080768
           version: 8.6.0.7-FIPS
      build string: ArubaOS version 8.6.0.7-FIPS for A72xx (p4build@pr-hpn-build08) (gcc version 4.4.5) #78216 SMP PREEMPT Fri Dec 11 14:22:21 UTC 2020
             flags:
               oem: aruba

    Image is signed; verifying checksum...
    passed
    Signer Cert OK
    Policy Cert OK
    RSA signature verified.
      image verify: PASS

    Partition 1:
     Reading image.................................done
        image type: 0
      machine type: 18
              size: 68959764
           version: 6.4.4.16-FIPS
      build string: ArubaOS version 6.4.4.16-FIPS for A72xx (p4build@chios) (gcc version 4.4.5) #61810 SMP PREEMPT Sat Oct 7 00:46:57 PDT 2017
             flags:
               oem: aruba

    Image is signed; verifying checksum...
    passed
    Signer Cert OK
    Policy Cert OK
    RSA signature verified.
      image verify: PASS
    PrintEnv
    cpboot> printenv
    bootargs=quiet
    bootcmd=bootf
    bootdelay=2
    baudrate=9600
    netretry=no
    ethact=ge-0
    cfgfile=1
    stdin=serial
    stdout=serial
    stderr=serial
    fdtaddr=fffffff... <redacted>
    ethaddr= <redacted>
    eth1addr= <redacted>

    Change cfgfile - Same Result
    cpboot> setenv cfgfile default1.cfg
    cpboot> saveenv
    Saving Environment to Flash...
    Un-Protected 1 sectors
    Erasing Flash...
    . done
    Writing to Flash... done
    Protected 1 sectors
    cpboot> reset
    Boot into Aruba 6 Partition - Enters factory state but instantly starts rebooting when configuration prompt appears - Same reeason for reboot
    cpboot> bootf 0:1
    Loading image 0:1###############################
    Image is signed; verifying checksum...
    passed
    Signer Cert OK
    Policy Cert OK
    RSA signature verified.
    [    0.000000]   0:xlp_napi_vc_mask 0xf
    [    0.000000]   0:sae frequency is 500
    [    0.000000]   0:-- SAE Frequency set to 500
    [    0.000000]   0:SAE Frequency set to 500MHz
    [    0.000000]   0:MSGRING_NAPI: Initializing NLM NAPI subsystem


    Aruba Networks
    ArubaOS Version 6.4.4.16-FIPS (build 61810 / label #61810)
    Built by p4build@chios on 2017-10-07 at 00:46:57 PDT (gcc version 4.4.5)
    Copyright (c) 2002-2017, Aruba Networks, an HP company.

               <<<<<    Welcome to Aruba Networks - Aruba A7205    >>>>>

    Probing for EEPROM devices                            [ OK ]
    Probing for real-time clock                           [ OK ]
    Initializing LCD module                               [ OK ]
    Uncompressing core image files                        [ OK ]
    Extracting corefs                                     [ OK ]

    Enabling watchdog                                     [ OK ]
    Starting device manager                               [ OK ]
    Performing eUSB Flash fast test...                    [ DONE ]
        [ OK ]
    Mounting flash                                        [ OK ]
    Checking system inventory                             [ OK ]
    Installing ancillary FS                               [ OK ]
    Performing integrity check on ancillary partition 0   [ FAIL : Ancillary image stored on flash is not for this release]

    Reboot Cause: Nanny rebooted machine - fpapps process died (Intent:cause:register 34:86:0:2c)
    Crash information available.

    Starting syslog service                               [ OK ]
    Restoring the database                                [ OK ]
    Generating SSH keys                                   [ OK ]
    Initializing TPM and certificates                     [ OK ]
    Checking for configuration upgrade                    [ OK ]
    Starting OS services                                  [ OK ]

    Starting FIPS Aruba Cryptographic KAT test
    Completed FIPS Aruba Cryptographic KAT test successfully.
    Starting OpenSSL FIPS KAT test
    Completed OpenSSL FIPS KAT test successfully.
    Reading configuration from factory-default.cfg

    ***************** Welcome to the Aruba7205 setup dialog *****************
    This dialog will help you to set the basic configuration for the switch.
    These settings, except for the Country Code, can later be changed from the
    Command Line Interface or Graphical User Interface.


    Commands: <Enter> Submit input or use [default value], <ctrl-I> Help
    <ctrl-B> Back, <ctrl-F> Forward, <ctrl-A> Line begin, <ctrl-E> Line end
    <ctrl-D> Delete, <BackSpace> Delete back, <ctrl-K> Delete to end of line
    <ctrl-P> Previous question <ctrl-X> Restart beginning


    Enter System name [Aruba7205]:
    Shutdown processing started
    Syncing data........done.
    Time sync [Done]
    Please stand by while rebooting the system.





    ------------------------------
    ------------------------------


  • 2.  RE: 7205 Mobility Controller stuck in a rebooting loop - Reboot Cause: Nanny rebooted machine - fpapps process died

    Posted Dec 17, 2021 10:37 AM
    Escalate the process with TAC, and if it is clear that you are eligible and there is just issues with processing and getting the right entitlements, ask them to process the case already. Or ask (through your partner) your local Aruba SE to assist in here.

    Was this controller running 8.6 already? As when you move from 6.x (in the secondary partition, that's why I'm asking) you need to do a complete wipe (write erase all). If that didn't happen, you can try to boot in the other partition and do the 'write erase all' again. Please be aware that everything is wiped, configuration, databases, but also licenses.

    Is this controller running stand-alone? Or under an MM/MCR?

    I checked a few cases with this message, and it looks like a rare condition, and in a few cases that I checked, I see a replacement of the hardware happen. That's why escalating/speeding up the processing of your support entitlement may be good.

    ------------------------------
    Herman Robers
    ------------------------
    If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

    In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.
    ------------------------------