02-07-2013 06:13 PM - edited 02-07-2013 07:01 PM
Is there a way to test if this actually works?
My understanding is the internal FTP server will only perform one transfer at a time. My hope was to speed up the process, so I've configured an external FTP server to upgrade the firmware on my controllers.
Problem is that the firmware is getting upgraded only 1 at a time.
I'm also running into problems where the transfer is taking a really long time.....as in 1-2 hours....either that or the process is hanging between controllers.
Currently, I can into my FTP server and see the first 2 controllers were successful in connecting and getting the firmware file. Airwave is reporting that the 3rd controller is "in progress," but I have nothing on the FTP server indicating a connection from the controller.
I need to upgrade 170 controllers, and while I'll do these in batches of 10 (to make sure no issues with APs falling offline, etc), I can't do this if I takes hours. I started a batch of 10 at 5:30pm, and 3 1/2 hours later I have 2 completed and 1 in progress, and no idea how to tell if there are any problems. My branch offices have good bandwidth (10-15Mbps) so wondering why it is taking so long to transfer a 50Mb file.
Is there a way to troubleshoot? I assumed at some point if the connection/transfer was unsuccessful it would fail and move on to the next controller (well, I actually assumed it would attempt to upgrade more than 1 controller at a time.....but you know what I mean....I hope).
02-07-2013 06:52 PM
02-08-2013 08:14 AM
The logging is fairly light on this, so it might be a better idea to open a support case to add some further investigation into this process.
Senior QA Engineer - Network Services
Aruba Networks, a Hewlett Packard Enterprise Company
02-08-2013 03:27 PM
I may have to give that a try. Only issue is our support contract expired in October - not sure what happened, but looks like we no longer have a relationship with the reseller, so we didn't get a renewal notice. Our RAM put us in contact with another reseller and the new invoice has been sitting on management's desk since December. I still have inventory of controllers, APs, and licenses, so not a big concern with any hardware failures at this point.
Canceled my upgrades last night, rebooted Airwave and created a smaller batch (3 controllers). They all downloaded from the FTP server and upgraded within 10 minutes.
Queued up 7 more and they just sat there. Looked at the controller info in Airwave and it had messages about errors with Telnet/SSH, along with confirmation of file transfer initiated. Confirmed settings (username and PW in Airwave), ability to SSH into controller (from laptop client, as well as from Airwave server via command line), FTP server up and accepting connections, and bandwidth to/from the site. Everything seemed fine. Upped the Telnet/SSH Timeout from 10 to 20 seconds and canceled first "stuck" controller from the job.
Went to bed.
Came in today and found they had finished (1 failed), but seemed to be almost exactly 2 hours between each controller (1:04, 1:06, 3:06, 5:07, and 5:08).
Seems like it works fine unless it hits a problem, then maybe it continues to try and doesn't fail and move on to the next for 2 hours? Can't seem to find any settings that would indicate any sort of wait time.
Tried again this afternoon - 3 controllers with no users. Less than 2 minutes per transfer and controllers upgraded without any issue. Again queued up 7 more. Have time stamps on those of 2:42, 2:45, 4:45, 4:47, 4:49, and 4:51pm.
The 4:45 is still "in progress" but have yet to see it hit the FTP server and I see this listed in the Recent AMP Device Events:
|Fri Feb 8 16:45:11 2013||System||Attempted firmware upgrade to 184.108.40.206 (220.127.116.11 for 6xx controllers)|
|Fri Feb 8 16:45:11 2013||System||Firmware Upgrade: Error transfering the file.|
|Fri Feb 8 16:45:11 2013||System||Telnet/SSH Error: command timed-out|
|Fri Feb 8 15:45:10 2013||System||Telnet/SSH Error: command timed-out|
|Fri Feb 8 14:45:09 2013||System||Firmware Upgrade: Initiating file transfer.|
|Fri Feb 8 14:45:08 2013||System||Needs firmware update|
The 7th controller says "in progress" but no time stamp, and nothing on the FTP server. Waiting to see if this will be like the others and complete 2 hours after the last.....guess I will know at 6:45.
I suppose I could do 3 at a time, but have a feeling this is not working as intended.....just not sure if it's something on our side. Can't find anything to lead me to believe it's an issue with Airwave, but just stumped on where to look.
Also looks like it doesn't matter if I use external FTP, Airwave will only process 1 transfer at a time? I guess the only reason to use external FTP would be to reduce traffic from the server - currently monitoring 540 devices at the moment......we don't use Airwave for configuration changes - mainly just for VisualRF. Everything is in Monitor-only.
Nothing critical with the upgrades, just trying to get up to the latest for the newer features (and have a box of APs that aren't compatible with 5.x.x code). If anything, this is giving me a reason to finally read the manual!
Will play around with it some more and see what happens. Will delete FTP server from group settings and just use AMP and see if it helps. If not I'll schedule some time to call support next week, or see if our SE is available to come in for some training.
02-08-2013 05:08 PM
Just an update.....
Uploaded the firmware to Airwave and now using the built in functionality.
Created a batch of 26 controllers and kicked it off at 7:23pm. As of 8:06 it is on controller #9 and haven't had any issues.
Not sure if the external FTP server was the problem? Doesn't explain the Telnet/SSH errors that were being reported, but at this point it looks to be working fine.
May kick off another job with the external server and see if the problem pops up again......now I'm just curious......
02-11-2013 07:24 AM
Still issues, but all seem to be whenever there is a Telnet/SSH error.
Job failed after 3 failed tasks.
Job: Firmware upgrade for multiple devices
Started: 2/9/2013 8:07 PM
3 tasks remain in the queue.
Device name: PC7850-Aruba620
Current version: 18.104.22.168
Desired version: 22.214.171.124
Failed: 2/10/2013 2:18 AM
Device name: PC6480-Aruba620 Current version: 126.96.36.199 Desired version: 188.8.131.52
Failed: 2/10/2013 4:21 AM
Device name: PC118s-Aruba620 Current version: 184.108.40.206 Desired version: 220.127.116.11
Failed: 2/10/2013 6:21 AM
Now to figure out why these Telnet/SSH errors are occuring. All controllers have the same credentials.
Works nicely when you schedule 15-20 and see the first 6-7 cruise through and upgrade every 3-4 minutes......not so cool when you go to bed and find out that 1 failed and the job is held up at that point. Even less cool when it finally gives up on the "problem" controller and moves on to the rest in the queue.....2 days later, in the middle of the day......and you aren't sitting around to check that the APs aren't defaulting to AM mode or rolling back to the master and need to be reprovisioned back to get them back to the local.