Wireless Access

Reply
Occasional Contributor II
Posts: 11
Registered: ‎02-24-2016

Why(How) would a controller update itself?

7010 local controller

3060 master controller running 6.4.2.0

S3500-48P running 7.4.0.2

Assorted AP 105s

Airwave 8.2.0.2

 

Yesterday morning we got a call that wireless was down at one of our sites. The helpdesk was unable to log in to the controller and instructed onsite staff to power cycle the controller and switch. When it did not come back up immediately, they contacted me. I found that all the APs were attached to the master controller, and then the local controller came back up. The APs started upgrading and rebooting. I logged in to the local and found that it was running 6.4.2.3. All of our controllers run 6.4.2.0 - no exceptions. This one in particular was a new 7010 I deployed in March, and I have documentation of the work I did downgrading it to 6.4.2.0 (it shipped with 6.4.3.4).

 

Fun facts (edited from an email to management about the outage)

  1. An analysis of the logs show that nobody logged in to this device during this time period. I can see log ins from Airwave to check config, and my login afterwards.

 

  1. The specific firmware in question (ArubaOS 6.4.2.3) is not in use on any device on our network. The update file does not exist in the Airwave repository, or any other location that update files are usually stored.

 

  1. Standard practice here is to update only one of the two partitions on the device, so that the other partition can be made active as a rollback plan. Both partitions on this device were updated to the same version.

 

  1. The master controllers deployed at the colo are capable of storing update files and applying updates. This feature is disabled, and no update files exist on the master controllers.

 

  1. The wireless access points onsite rebooted and upgraded – this upgrade process was visible during troubleshooting, and further verifies that an update occurred during this time period.
  2. Airwave is configured for monitor-only, and there are no logs to indicate that it attempted to send any confguration/update commands.

 

I have a case open with TAC, who is understandably stumped, as am I. Anyone have any ideas?

Guru Elite
Posts: 20,810
Registered: ‎03-29-2007

Re: Why(How) would a controller update itself?

Someone uploaded but did not reboot the controller in the past. The reboot of the controller then triggered the upgrade.


Colin Joseph
Aruba Customer Engineering

Looking for an Answer? Search the Community Knowledge Base Here: Community Knowledge Base

Occasional Contributor II
Posts: 11
Registered: ‎02-24-2016

Re: Why(How) would a controller update itself?

I know from a technical perspective that's the only apparent answer.

 

Non-technically, it makes no sense. I am the only person who touches any of our Aruba gear outside of basic helpdesk troubleshooting. There are two people who have managed it before, and both were glad to hand it off to me and never touch it again. I have an issue with the idea that one of the two of them (the only other people in the company who have access to download firmware) randomly decided to download newer firmware, upload it to both partitions and reboot in the middle of the day, and then lie to me about it.

MVP
Posts: 1,302
Registered: ‎11-07-2008

Re: Why(How) would a controller update itself?

Do you have a syslog server so that you can inspect the audit trail? Did you change the admin/root password to the controller once you took it over so that ONLY you have access?

 

Controllers don't have the capability to upgrade themselves, there's just no commands or logic for it, as it requires someone to upload new code (via local file in the GUI, TFTP, FTP, SCP, etc). So the only way what you describes could happen is if someone uploaded new code but didn't ugprade, or loaded new code and upgraded. 

Jerrod Howard
Sr. Techical Marketing Engineer
Occasional Contributor II
Posts: 11
Registered: ‎02-24-2016

Re: Why(How) would a controller update itself?

I only have Airwave, no separate logging server. I haven't changed the password yet, but plan on changing it for all our devices.

MVP
Posts: 342
Registered: ‎07-26-2011

Re: Why(How) would a controller update itself?

Is it possible the boot statement was changed or not saved after the upgrade? What does the output of the below commands say?

 

#show image version

#show audit-trail

#show upgrade status 

#show upgrade configuration

ACMA, ACMP
If my post addresses your query, give kudos:)
Occasional Contributor II
Posts: 11
Registered: ‎02-24-2016

Re: Why(How) would a controller update itself?

Issue occured June 21st at about 11:45am.

 

 

(Local Controller) #show image version

----------------------------------

Partition               : 0:0 (/dev/usb/flash1) **Default boot**

Software Version        : ArubaOS 6.4.2.3 (Digitally Signed - Production Build)

Build number            : 47524

Label                   : 47524

Built on                : Sat Dec 6 11:24:05 PST 2014

----------------------------------

Partition               : 0:1 (/dev/usb/flash2)

Software Version        : ArubaOS 6.4.2.3 (Digitally Signed - Production Build)

Build number            : 47524

Label                   : 47524

Built on                : Sat Dec 6 11:24:05 PST 2014

 

(Local Controller) #show audit-trail

Jun 21 09:54:14  cli[3252]: USER:admin@<Airwave> COMMAND:<no paging > -- comm                                                                                                                     and executed successfully

Jun 21 09:54:14  cli[3252]: USER:admin@<Airwave> COMMAND:<encrypt disable > -                                                                                                                     - command executed successfully

Jun 21 09:54:18  cli[3252]: USER:admin@<Airwave> COMMAND:<no paging > -- comm                                                                                                                     and executed successfully

Jun 21 09:54:19  cli[3252]: USER:admin@<Airwave> COMMAND:<encrypt disable > -                                                                                                                     - command executed successfully

Jun 21 09:55:44  cli[3252]: USER:admin@<Airwave> COMMAND:<no paging > -- comm                                                                                                                     and executed successfully

Jun 21 09:55:44  cli[3252]: USER:admin@<Airwave> COMMAND:<encrypt disable > -                                                                                                                     - command executed successfully

Jun 21 11:11:59  fpcli: USER:admin@<Airwave> COMMAND:<no paging > -- command                                                                                                                      executed successfully

Jun 21 11:11:59  fpcli: USER:admin@<Airwave> COMMAND:<encrypt disable > -- co                                                                                                                     mmand executed successfully

Jun 21 11:12:30  fpcli: USER:admin@<Airwave> COMMAND:<encrypt disable > -- co                                                                                                                     mmand executed successfully

Jun 21 11:12:45  fpcli: USER:admin@<Airwave> COMMAND:<no paging > -- command                                                                                                                      executed successfully

Jun 21 11:12:45  fpcli: USER:admin@<Airwave> COMMAND:<encrypt disable > -- co                                                                                                                     mmand executed successfully

Jun 21 11:26:59  fpcli: USER:admin@<Airwave> COMMAND:<no paging > -- command                                                                                                                      executed successfully

Jun 21 11:26:59  fpcli: USER:admin@<Airwave> COMMAND:<encrypt disable > -- co                                                                                                                     mmand executed successfully

Jun 22 02:21:01  fpcli: USER:admin@<Airwave> COMMAND:<no paging > -- command                                                                                                                      executed successfully

Jun 22 02:21:01  fpcli: USER:admin@<Airwave> COMMAND:<encrypt disable > -- co                                                                                                                     mmand executed successfully

Jun 22 02:22:37  fpcli: USER:admin@<Airwave> COMMAND:<backup flash > -- comma                                                                                                                     nd executed successfully

Jun 22 02:22:39  fpcli: USER:admin@<Airwave> COMMAND:<copy flash: "flashbacku                                                                                                                     p.tar.gz" scp: "<Airwave>" "scpuser10618" "/tmp/flashbackup_6423_1106_1466595                                                                                                                     459.tar.gz" > -- command executed successfully

Jun 22 02:22:41  fpcli: USER:admin@<Airwave> COMMAND:<encrypt disable > -- co                                                                                                                     mmand executed successfully

Jun 22 02:23:13  fpcli: USER:admin@<Airwave> COMMAND:<encrypt disable > -- co                                                                                                                     mmand executed successfully

Jun 22 11:23:32  fpcli: USER:admin@<Airwave> COMMAND:<no paging > -- command                                                                                                                      executed successfully

Jun 22 11:23:32  fpcli: USER:admin@<Airwave> COMMAND:<encrypt disable > -- co                                                                                                                     mmand executed successfully

Jun 22 11:24:03  fpcli: USER:admin@<Airwave> COMMAND:<encrypt disable > -- co                                                                                                                     mmand executed successfully

Jun 22 11:24:17  fpcli: USER:admin@<Airwave> COMMAND:<no paging > -- command                                                                                                                      executed successfully

Jun 22 11:24:17  fpcli: USER:admin@<Airwave> COMMAND:<encrypt disable > -- co                                                                                                                     mmand executed successfully

Jun 22 14:14:03  fpcli: USER:admin@<Airwave> COMMAND:<no paging > -- command                                                                                                                      executed successfully

Jun 22 14:14:03  fpcli: USER:admin@<Airwave> COMMAND:<encrypt disable > -- co                                                                                                                     mmand executed successfully

Jun 23 02:22:19  fpcli: USER:admin@<Airwave> COMMAND:<no paging > -- command                                                                                                                      executed successfully

Jun 23 02:22:19  fpcli: USER:admin@<Airwave> COMMAND:<encrypt disable > -- co                                                                                                                     mmand executed successfully

Jun 23 02:23:56  fpcli: USER:admin@<Airwave> COMMAND:<backup flash > -- comma                                                                                                                     nd executed successfully

Jun 23 02:23:58  fpcli: USER:admin@<Airwave> COMMAND:<copy flash: "flashbacku                                                                                                                     p.tar.gz" scp: "<Airwave>" "scpuser11270" "/tmp/flashbackup_6423_1106_1466681                                                                                                                     936.tar.gz" > -- command executed successfully

Jun 23 02:24:07  fpcli: USER:admin@<Airwave> COMMAND:<encrypt disable > -- co                                                                                                                     mmand executed successfully

Jun 23 02:24:40  fpcli: USER:admin@<Airwave> COMMAND:<encrypt disable > -- co                                                                                                                     mmand executed successfully

 

(Local Controller) #show upgrade status

This command is applicable only on Master Controller.

 

 

(Local Controller) #show upgrade configuration

 

Upgrade is disabled.

 

 

(Master Controller) #show upgrade status

 

Upgrade profile is not enabled.

 

(Master Controller) #show upgrade configuration

 

Upgrade is disabled.

 

Upgrade target is not configured.

Occasional Contributor II
Posts: 11
Registered: ‎02-24-2016

Re: Why(How) would a controller update itself?

Airwave logs for the controller:

 

 

Wed Jun 22 04:37:42 2016SystemBackup Operation Success
Wed Jun 22 04:37:39 2016SystemBackup In Progress
Wed Jun 22 04:37:39 2016SystemConfiguration verification: configuration on device does not match desired configuration
Tue Jun 21 13:18:29 2016SystemError in SNMP polling: Timeout(thin_ap_interface_monitor)
Tue Jun 21 13:18:11 2016SystemError in SNMP polling: Timeout(thin_ap_client)
Tue Jun 21 13:17:21 2016SystemError in SNMP polling: Timeout(interface_monitor)
Tue Jun 21 13:17:20 2016SystemError in SNMP polling: Timeout(thin_ap_mesh)
Tue Jun 21 13:16:57 2016SystemError in SNMP polling: Timeout(thin_ap_status)
Tue Jun 21 13:16:54 2016SystemError in SNMP polling: Timeout(device_resources)
Tue Jun 21 12:11:52 2016SystemConfiguration verification: configuration on device does not match desired configuration
Tue Jun 21 12:10:26 2016SystemDesired config changed while verifying, flagging for re-verify
Tue Jun 21 12:10:26 2016SystemConfiguration verification: configuration on device does not match desired configuration
Tue Jun 21 12:09:16 2016SystemStatus changed to 'OK'
Tue Jun 21 12:09:16 2016SystemUp
Tue Jun 21 12:09:12 2016SystemDevice rebooted while marked as down: Device uptime value changed (current: 5 mins 49 secs, calculated: 71 days 2 hrs 3 mins 58
secs)
Tue Jun 21 12:03:52 2016SystemStatus changed to 'ICMP ping failed (after SNMP get failed)'
Tue Jun 21 12:03:52 2016SystemDown
Tue Jun 21 12:01:27 2016SystemError in SNMP polling: Timeout(interface_monitor)
Tue Jun 21 12:01:25 2016SystemError in SNMP polling: Timeout(thin_ap_mesh)
Tue Jun 21 04:36:45 2016SystemBackup Operation Success
Tue Jun 21 04:36:42 2016SystemBackup In Progress
Guru Elite
Posts: 20,810
Registered: ‎03-29-2007

Re: Why(How) would a controller update itself?

The controller could be up for a year, but if someone uploaded an image a year ago, whenever it rebooted, it would upgrade the controller.

The evidence of that might be longer than your logging history. The controller cannot upgrade to an image that was not uploaded to it at some time or another.


Colin Joseph
Aruba Customer Engineering

Looking for an Answer? Search the Community Knowledge Base Here: Community Knowledge Base

Occasional Contributor II
Posts: 11
Registered: ‎02-24-2016

Re: Why(How) would a controller update itself?

I get that - so how do I find out how/when the upgrade occurred?

Search Airheads
Showing results for 
Search instead for 
Did you mean: