Wireless Access

last person joined: yesterday 

Access network design for branch, remote, outdoor, and campus locations with HPE Aruba Networking access points and mobility controllers.
Back to discussions
Expand all | Collapse all

Why(How) would a controller update itself?

This thread has been viewed 0 times

  • 1.  Why(How) would a controller update itself?

    Posted Jun 22, 2016 04:48 PM

    7010 local controller

    3060 master controller running 6.4.2.0

    S3500-48P running 7.4.0.2

    Assorted AP 105s

    Airwave 8.2.0.2

     

    Yesterday morning we got a call that wireless was down at one of our sites. The helpdesk was unable to log in to the controller and instructed onsite staff to power cycle the controller and switch. When it did not come back up immediately, they contacted me. I found that all the APs were attached to the master controller, and then the local controller came back up. The APs started upgrading and rebooting. I logged in to the local and found that it was running 6.4.2.3. All of our controllers run 6.4.2.0 - no exceptions. This one in particular was a new 7010 I deployed in March, and I have documentation of the work I did downgrading it to 6.4.2.0 (it shipped with 6.4.3.4).

     

    Fun facts (edited from an email to management about the outage)

    1. An analysis of the logs show that nobody logged in to this device during this time period. I can see log ins from Airwave to check config, and my login afterwards.

     

    1. The specific firmware in question (ArubaOS 6.4.2.3) is not in use on any device on our network. The update file does not exist in the Airwave repository, or any other location that update files are usually stored.

     

    1. Standard practice here is to update only one of the two partitions on the device, so that the other partition can be made active as a rollback plan. Both partitions on this device were updated to the same version.

     

    1. The master controllers deployed at the colo are capable of storing update files and applying updates. This feature is disabled, and no update files exist on the master controllers.

     

    1. The wireless access points onsite rebooted and upgraded – this upgrade process was visible during troubleshooting, and further verifies that an update occurred during this time period.
    2. Airwave is configured for monitor-only, and there are no logs to indicate that it attempted to send any confguration/update commands.

     

    I have a case open with TAC, who is understandably stumped, as am I. Anyone have any ideas?



  • 2.  RE: Why(How) would a controller update itself?

    EMPLOYEE
    Posted Jun 22, 2016 04:50 PM
    Someone uploaded but did not reboot the controller in the past. The reboot of the controller then triggered the upgrade.


  • 3.  RE: Why(How) would a controller update itself?

    Posted Jun 22, 2016 05:26 PM

    I know from a technical perspective that's the only apparent answer.

     

    Non-technically, it makes no sense. I am the only person who touches any of our Aruba gear outside of basic helpdesk troubleshooting. There are two people who have managed it before, and both were glad to hand it off to me and never touch it again. I have an issue with the idea that one of the two of them (the only other people in the company who have access to download firmware) randomly decided to download newer firmware, upload it to both partitions and reboot in the middle of the day, and then lie to me about it.



  • 4.  RE: Why(How) would a controller update itself?

    EMPLOYEE
    Posted Jun 22, 2016 05:34 PM

    Do you have a syslog server so that you can inspect the audit trail? Did you change the admin/root password to the controller once you took it over so that ONLY you have access?

     

    Controllers don't have the capability to upgrade themselves, there's just no commands or logic for it, as it requires someone to upload new code (via local file in the GUI, TFTP, FTP, SCP, etc). So the only way what you describes could happen is if someone uploaded new code but didn't ugprade, or loaded new code and upgraded. 



  • 5.  RE: Why(How) would a controller update itself?

    Posted Jun 22, 2016 05:42 PM

    I only have Airwave, no separate logging server. I haven't changed the password yet, but plan on changing it for all our devices.



  • 6.  RE: Why(How) would a controller update itself?

    MVP EXPERT
    Posted Jun 23, 2016 06:18 AM

    Is it possible the boot statement was changed or not saved after the upgrade? What does the output of the below commands say?

     

    #show image version

    #show audit-trail

    #show upgrade status 

    #show upgrade configuration



  • 7.  RE: Why(How) would a controller update itself?

    Posted Jun 23, 2016 11:23 AM

    Issue occured June 21st at about 11:45am.

     

     

    (Local Controller) #show image version

    ----------------------------------

    Partition               : 0:0 (/dev/usb/flash1) **Default boot**

    Software Version        : ArubaOS 6.4.2.3 (Digitally Signed - Production Build)

    Build number            : 47524

    Label                   : 47524

    Built on                : Sat Dec 6 11:24:05 PST 2014

    ----------------------------------

    Partition               : 0:1 (/dev/usb/flash2)

    Software Version        : ArubaOS 6.4.2.3 (Digitally Signed - Production Build)

    Build number            : 47524

    Label                   : 47524

    Built on                : Sat Dec 6 11:24:05 PST 2014

     

    (Local Controller) #show audit-trail

    Jun 21 09:54:14  cli[3252]: USER:admin@<Airwave> COMMAND:<no paging > -- comm                                                                                                                     and executed successfully

    Jun 21 09:54:14  cli[3252]: USER:admin@<Airwave> COMMAND:<encrypt disable > -                                                                                                                     - command executed successfully

    Jun 21 09:54:18  cli[3252]: USER:admin@<Airwave> COMMAND:<no paging > -- comm                                                                                                                     and executed successfully

    Jun 21 09:54:19  cli[3252]: USER:admin@<Airwave> COMMAND:<encrypt disable > -                                                                                                                     - command executed successfully

    Jun 21 09:55:44  cli[3252]: USER:admin@<Airwave> COMMAND:<no paging > -- comm                                                                                                                     and executed successfully

    Jun 21 09:55:44  cli[3252]: USER:admin@<Airwave> COMMAND:<encrypt disable > -                                                                                                                     - command executed successfully

    Jun 21 11:11:59  fpcli: USER:admin@<Airwave> COMMAND:<no paging > -- command                                                                                                                      executed successfully

    Jun 21 11:11:59  fpcli: USER:admin@<Airwave> COMMAND:<encrypt disable > -- co                                                                                                                     mmand executed successfully

    Jun 21 11:12:30  fpcli: USER:admin@<Airwave> COMMAND:<encrypt disable > -- co                                                                                                                     mmand executed successfully

    Jun 21 11:12:45  fpcli: USER:admin@<Airwave> COMMAND:<no paging > -- command                                                                                                                      executed successfully

    Jun 21 11:12:45  fpcli: USER:admin@<Airwave> COMMAND:<encrypt disable > -- co                                                                                                                     mmand executed successfully

    Jun 21 11:26:59  fpcli: USER:admin@<Airwave> COMMAND:<no paging > -- command                                                                                                                      executed successfully

    Jun 21 11:26:59  fpcli: USER:admin@<Airwave> COMMAND:<encrypt disable > -- co                                                                                                                     mmand executed successfully

    Jun 22 02:21:01  fpcli: USER:admin@<Airwave> COMMAND:<no paging > -- command                                                                                                                      executed successfully

    Jun 22 02:21:01  fpcli: USER:admin@<Airwave> COMMAND:<encrypt disable > -- co                                                                                                                     mmand executed successfully

    Jun 22 02:22:37  fpcli: USER:admin@<Airwave> COMMAND:<backup flash > -- comma                                                                                                                     nd executed successfully

    Jun 22 02:22:39  fpcli: USER:admin@<Airwave> COMMAND:<copy flash: "flashbacku                                                                                                                     p.tar.gz" scp: "<Airwave>" "scpuser10618" "/tmp/flashbackup_6423_1106_1466595                                                                                                                     459.tar.gz" > -- command executed successfully

    Jun 22 02:22:41  fpcli: USER:admin@<Airwave> COMMAND:<encrypt disable > -- co                                                                                                                     mmand executed successfully

    Jun 22 02:23:13  fpcli: USER:admin@<Airwave> COMMAND:<encrypt disable > -- co                                                                                                                     mmand executed successfully

    Jun 22 11:23:32  fpcli: USER:admin@<Airwave> COMMAND:<no paging > -- command                                                                                                                      executed successfully

    Jun 22 11:23:32  fpcli: USER:admin@<Airwave> COMMAND:<encrypt disable > -- co                                                                                                                     mmand executed successfully

    Jun 22 11:24:03  fpcli: USER:admin@<Airwave> COMMAND:<encrypt disable > -- co                                                                                                                     mmand executed successfully

    Jun 22 11:24:17  fpcli: USER:admin@<Airwave> COMMAND:<no paging > -- command                                                                                                                      executed successfully

    Jun 22 11:24:17  fpcli: USER:admin@<Airwave> COMMAND:<encrypt disable > -- co                                                                                                                     mmand executed successfully

    Jun 22 14:14:03  fpcli: USER:admin@<Airwave> COMMAND:<no paging > -- command                                                                                                                      executed successfully

    Jun 22 14:14:03  fpcli: USER:admin@<Airwave> COMMAND:<encrypt disable > -- co                                                                                                                     mmand executed successfully

    Jun 23 02:22:19  fpcli: USER:admin@<Airwave> COMMAND:<no paging > -- command                                                                                                                      executed successfully

    Jun 23 02:22:19  fpcli: USER:admin@<Airwave> COMMAND:<encrypt disable > -- co                                                                                                                     mmand executed successfully

    Jun 23 02:23:56  fpcli: USER:admin@<Airwave> COMMAND:<backup flash > -- comma                                                                                                                     nd executed successfully

    Jun 23 02:23:58  fpcli: USER:admin@<Airwave> COMMAND:<copy flash: "flashbacku                                                                                                                     p.tar.gz" scp: "<Airwave>" "scpuser11270" "/tmp/flashbackup_6423_1106_1466681                                                                                                                     936.tar.gz" > -- command executed successfully

    Jun 23 02:24:07  fpcli: USER:admin@<Airwave> COMMAND:<encrypt disable > -- co                                                                                                                     mmand executed successfully

    Jun 23 02:24:40  fpcli: USER:admin@<Airwave> COMMAND:<encrypt disable > -- co                                                                                                                     mmand executed successfully

     

    (Local Controller) #show upgrade status

    This command is applicable only on Master Controller.

     

     

    (Local Controller) #show upgrade configuration

     

    Upgrade is disabled.

     

     

    (Master Controller) #show upgrade status

     

    Upgrade profile is not enabled.

     

    (Master Controller) #show upgrade configuration

     

    Upgrade is disabled.

     

    Upgrade target is not configured.



  • 8.  RE: Why(How) would a controller update itself?

    Posted Jun 23, 2016 11:25 AM

    Airwave logs for the controller:

     

     

    Wed Jun 22 04:37:42 2016SystemBackup Operation Success
    Wed Jun 22 04:37:39 2016SystemBackup In Progress
    Wed Jun 22 04:37:39 2016SystemConfiguration verification: configuration on device does not match desired configuration
    Tue Jun 21 13:18:29 2016SystemError in SNMP polling: Timeout(thin_ap_interface_monitor)
    Tue Jun 21 13:18:11 2016SystemError in SNMP polling: Timeout(thin_ap_client)
    Tue Jun 21 13:17:21 2016SystemError in SNMP polling: Timeout(interface_monitor)
    Tue Jun 21 13:17:20 2016SystemError in SNMP polling: Timeout(thin_ap_mesh)
    Tue Jun 21 13:16:57 2016SystemError in SNMP polling: Timeout(thin_ap_status)
    Tue Jun 21 13:16:54 2016SystemError in SNMP polling: Timeout(device_resources)
    Tue Jun 21 12:11:52 2016SystemConfiguration verification: configuration on device does not match desired configuration
    Tue Jun 21 12:10:26 2016SystemDesired config changed while verifying, flagging for re-verify
    Tue Jun 21 12:10:26 2016SystemConfiguration verification: configuration on device does not match desired configuration
    Tue Jun 21 12:09:16 2016SystemStatus changed to 'OK'
    Tue Jun 21 12:09:16 2016SystemUp
    Tue Jun 21 12:09:12 2016SystemDevice rebooted while marked as down: Device uptime value changed (current: 5 mins 49 secs, calculated: 71 days 2 hrs 3 mins 58
    secs)
    Tue Jun 21 12:03:52 2016SystemStatus changed to 'ICMP ping failed (after SNMP get failed)'
    Tue Jun 21 12:03:52 2016SystemDown
    Tue Jun 21 12:01:27 2016SystemError in SNMP polling: Timeout(interface_monitor)
    Tue Jun 21 12:01:25 2016SystemError in SNMP polling: Timeout(thin_ap_mesh)
    Tue Jun 21 04:36:45 2016SystemBackup Operation Success
    Tue Jun 21 04:36:42 2016SystemBackup In Progress


  • 9.  RE: Why(How) would a controller update itself?

    EMPLOYEE
    Posted Jun 23, 2016 12:04 PM
    The controller could be up for a year, but if someone uploaded an image a year ago, whenever it rebooted, it would upgrade the controller.

    The evidence of that might be longer than your logging history. The controller cannot upgrade to an image that was not uploaded to it at some time or another.


  • 10.  RE: Why(How) would a controller update itself?

    Posted Jun 23, 2016 12:09 PM

    I get that - so how do I find out how/when the upgrade occurred?



  • 11.  RE: Why(How) would a controller update itself?

    EMPLOYEE
    Posted Jun 23, 2016 03:10 PM
    A real upgrade only occurs when the controller is rebooted.... Uploading the file to the controller or setting the boot partition to a different one is not recorded in the audit trail unless you do a "config t audit-trail all"