Network Management

Hi,

We have recently upgraded to Airwave 8.2.10 to allow us to manage our switches (we have motd banner set).  Upon trying to upgrade the firmware (16.08 to 16.09) on an ArubaOS-Switch device (2930) the upgrade fails when it's awaiting reboot.

Thanks to TACACS, I can see that the following commands are run by Airwave:

configure
console local-terminal none
session interactive-mode disable
terminal width 1920
include-credentials
show flash
copy tftp flash X.X.X.X WC_16_09_0004_swi_0.bin primary
show flash
write memory
boot system flash primary

Now, all looks fine here, but the switch doesn't actually reboot, and as such the fw upgrade job fails.

A show flash indicates that the firmware upload was a success, it was just the reboot that didn't happen.

I can log into the switch (with putty) as the same user, and perform the same command, and the switch reboots fine - on the new version.

Any ideas?

Thanks,

Ben.

We need to look in to switch telnet log in Airwave to see whether server pushed reboot command or not and also check if reboot option is enabled under Group>Firmware page?

Hi Pavan - the commands in my initial post were what was authorized by ClearPass - so Airwave definately sent all of them.

The device is Monitor Only + Firmware Updates, and reboots are enabled under AMP Settings.

The only setting I see in the group is:

Force Switch Reboot:
This is valid for switches with FW version 16.05 and above. If option is set to 'Yes' and if the configuration requires reboot to be effective, switch will be rebooted after configuration is pushed. However, if option is set to 'No' and if the configuration requires reboot to be effective, no configuration is pushed to the switch.

This is set to No, but how I read it, it only applies to pushing configs?

Thanks,

Ben.

Hi,

do You have any update about this issue ?

best regards

Does reboot option is enabled under Groups>Firmware option, we get this page option after mapping device model firmware and click save and upgrade button.

Hi option is enable

This looks to be a bug, can you please open a TAC case to look at the AirWave and Switch logs if the reboot command is initiated from AirWave and the switch able to execute the command.

Apologies all, I should've kept the community up to date. 

We logged a case (The Airwave case is #5342489004, the switch case is #5342625152) at the time and it took a few weeks of the blame being passed between ClearPass, switch and Airwave teams, but we got there in the end - and you're right Gowri, it is a bug.

Airwave is, as I suspected from the start, disconnecting from the SSH session prior to the switch receving approval from ClearPass for the reboot command (all happens in a matter of a few milliseconds).  This causes the switch to discard the TACACS approval and not perform the reboot.

We have been informed (mid December) that a fix will be released in an Airwave update due to be dropped on Jan 15.

Just another update - the fix that was meant to be released on January the 15th has been rolled up into 8.2.11 which is due end of April.  I had to probe for this information, it was not supplied automatically which was disappointing considering I had an open case waiting on the fix release.

I have since received a private patch/diff file for this which has been applied today by tech support.  Unfortunately this has not fixed the issue and appears to have actually made it worse (sometimes airwave doesn't even log on to the switch now).

*sigh*

Hi Pavan - there's a couple;

#5342489004 - Airwave team (orignal ticket, from 2019-10-23)

#5342625152 - Switch team (logged by Airwave team as they believed it was a switch issue).  This is where most of the progress was made, although the fix has been an Airwave fix.

I am looking in to it, will try to expedite and get it fixed.

Hi,

It looks issue mainly related to switch not Airwave, already defect is field on switch side.

Currently in case of reboot during firmware upgrade, Airwave does not wait for any response and closes the session and moves on to the next task. Due to changes at switch end due in TACACS authentication, the reboot fails if there is delay in TACACS response.

Patch applied is to make airwave to wait for more time before it timeout and move to next task, error which noticed after applying patch is totally different

Thu Jan 30 16:25:35 2020:

>> copy tftp flash 10.10.8.10 YA_16_08_0005_swi_0.bin primary

<< The primary image will be deleted.

<< Continue (y/n)? y

<< 000M Transport error.

Case is already escalated , Airwave/switch tickets been montiored by escalation engineers.

It's an interesting scenario, that's for sure.  The 'delay' you talk about with TACACS auth is 50-100ms - so no excessive.  It's just that Airwave disconnects the exact millisecond that it has sent the reboot command.

The Airwave Team claim the switch should honour the command as it was issued to the switch, and clearpass subsequently authorized it.

The Switch Team believe Airwave should be 'better behaved' and wait for the command prompt to return prior to disconnecting the SSH session - by which time the command will have been authorized and executed.

I can see both sides of the argument - but I side with the Switch team - Airwave should correctly consume the CLI, not just 'fire and forget' the commands.

In regards to the issue as it stands - we have tried TFTP from many sources including putting a device on the local subnet of the Airwave box and it appears that somehow (possibly?) the patch has killed the TFTP service.  It's back with HP at the moment, no doubt another multi-hour remote session is coming...

the internal discussion between the different Aruba teams (switching, airwave, clearpass) shows once again what the priority is for arubanetworks, and that does not seem to be customer interest.

We know root cause of the issue and escalation team working on priority!

@Dirk- If you have any case pending for long time with TAC, I am happy to look in to it.

Well, we got there.

To summarise for the benefit of people after a solution;

Aruba made a private patch to resolve an issue whereby Airwave would disconnect prematurely from the switch prior to ClearPass authorization meaning the reboot command wasn't performed (I'm sure Aruba will share this patch if you log a TAC case).  I'm told this fix will be rolled out in 8.2.11

Further failures after this were caused by yet another defect/bug which saw existing TFTP files to become unusable (permissions issue I think?) due to the patch/make operation..  The fix was simply to remove and re-upload the files.

Thanks for your attention to the issue Pavan.

For any others following this message, I also experienced the issue even after I upgraded to 8.2.11.0.  I have two different TAC cases open now.  One on the Airwave side (5346795386) and now a new one on the switching side (5347231419).

The workaround for us was to switch the transport method from tftp to sftp and we're were able to upgrade a couple of switches without issue.  the two I've tested so far were both 2930M's.  I'm going to try a 3810 today.

Rick

Today's update... We found a workaround to Airwave 8.2.11.0.  Changing the permissions on the /var/tftpboot directory fixed it.  See below a snippet from Aruba TAC:

I got confirmation from our engineering team that this issue is fixed in Airwave version 8.2.11.1.  As a workaround we were asked to change the file permission for tftpboot directory.

Please find the below command:

root@localhost CentOS7 / # cd /

root@localhost CentOS7 / # ls -lthr

root@localhost CentOS7 / chmod 775 /var/tftpboot

This fixed it for us on 8.2.11.0.  I already had root access because of some custom scripts we run on our site.  Others may need to contact TAC to get them to change the permissions.

Rick

Sorry, I should have added this was for the tftp method of upgrade.