Wireless Access

How long should it take for an AP to upgrade firmware once the controller has been upgraded?

I'm in the process of testing out 6.3.1.6.  Upgraded local 620controller from 6.1.3.10.  Controller came back up with 0 APs.  Saw the APs on the master with a status of upgrading.  After 2 hours the APs were still no showing on the local, and both showing "down" on the master.  2 hours later still having the same issue.  

Finally decided to reboot the controller and controller came back up with both APs provisioned.

Tried this again yesterday and had the same issue, although this time the APs did not show up as "upgrading" on the master - they simply showed up as down.   I remembered that I could check the activity on the controller ports, and did see that the ports were passing traffic, although it was very slow.  

What is a reasonable amount of time for APs to upgrade their firmware?  If the testing goes well I'd like to roll out the 6.3 firmware, but can't do that if it's going to take this long, or if I'm going to need to manually check all 190 controllers to make sure all the APs come back up.

It really depends on the link speed between the APs and the controllers and how many APs.

You can check the status of the AP buy doing a 'show datapath session table <ap-ip>'.  If you see a port 22, that means it is downloading its image via ftp.  Once you see protocol 47, that is the GRE tunnel from the AP, though it may take another 30-60 secs to show as up on the controller.  Normally I would allow a good ten mins for this to happen.

If the APs don't come back after an extended time, this is normally resolved by rebooting the AP, but unplugging or bouncing the switchport.

Did you upgrade the master as well?  If not, what would be happening is that the APs upgrade from the local, and then when they reboot and come up firstly to the master, they downgrade, then upgrade, then downgrade.....and get stuck in a loop.

To avoid this for your testing, reprovision those aps with a static master-ip to be that of your upgraded local controller.

Thanks, it must be the loop thing.

Unfortunately I can't upgrade the master until I do some testing.  I have a 620 in a lab that I normally use for testing, but I've been having issues with an office and my SE suggested I try a firmware upgrade to see if it fixed anything. 

I typically upgrade the master first, so hopefully won't have that issue in the future - other than the increased time to upgrade over the WAN link when they hit the master.

you have quitea task ahead of you with 190 controllers.

Are most of the aps on the same subnet as their local controller?  How do the aps find the master?  If it is with DNS and they are (mostly) on the same subnet as their local controller, you could disable that DNS entry during the upgrade so that they get their image locally rather than across the wan from the master.

---

@COLE1 wrote:

Thanks, it must be the loop thing.

 

Unfortunately I can't upgrade the master until I do some testing.  I have a 620 in a lab that I normally use for testing, but I've been having issues with an office and my SE suggested I try a firmware upgrade to see if it fixed anything. 

 

I typically upgrade the master first, so hopefully won't have that issue in the future - other than the increased time to upgrade over the WAN link when they hit the master.

---

COLE1

The 600 series controllers take the longest to boot out of all of the controllers (8 minutes depending), so you would factor that in.

I would plan this upgrade out with my SE.  If you have been managing the WLAN the whole time, you need to be aware of how all of your access points discover their local controller.  If they discover their local controller THROUGH the master (using DNS), the master and all the locals need to be upgraded at the same time.  If they discover their local controller through a DHCP option or a local broadcast, it is possible that local controller can be upgraded individually.

The reason why you need to upgrade a master AND local at the same time if the local is discovered through the master is that if the master is on one version of code and the local is on a different version, the AP will do this:

- Discover the master through DNS

- Upgrade its code to match the master

- Reboot

- Discovery the master through DNS

- AP Code Matches master, so master redirects to local

- AP Code does not match local, so AP downgrades and reboots

- Go back to beginning

If the local is discovered by the access point through a DHCP option, it will only be upgraded or downgraded by that controller.  The only gotcha with access points that discover their master through a local broadcast, is if the controller is rebooted and the access point cannot find the controller through a broadcast, because it is rebooting, it may fail back to DNS and find the master, which might have the wrong version of code.  If a local site relies on broacasts to find the master, see if you can configure  a DHCP option instead to point to that local controller, so that their reboot sequence will be more deterministic.

Those are just the tip of the iceberg when it comes to upgrading controllers, and depending on the types of clients you have, there is probably much more investigation that needs to be done, and that is where your SE or maybe a consultant will come in.  Access points taking long to come back in the lab gives you a golden opportunity to understand more about your network and probably correct some historical issues you never knew you had..

Thanks for the insight.  Really touches on the ongoing issue I've had with these code upgrades.

We do use master discovery via DNS.

I have 2 masters (3600 w/ VRRF) and 190 (or so) local controllers.  Upgrading via Airwave has proven spotty at best.  Jobs will seem to hang on a certain controller and retry multiple times before either failing and moving on, or eventually work and reboot controller in the middle of the day.

In the past I've just done the upgrades manually.  Log in to each controller, kick off the FTP process, reboot and check to make sure everything is back up.   I can usually do about 30-40 a night.   10 at a time - by the time you finish kicking off the upgrade on the 10th the 1st one is usually back up and running.

I could probably just manually FTP the image to each controller and then use Airwave to schedule a simultaneous reboot of all the controllers - or I guess in that case I should probably do the master controller last so that any APs that can't reach the local will not start pulling down firmware from the master over the WAN links.   I guess my only gotcha in that event would be any offices that have power issues that reboot controllers before I'm ready.

I'll ping my SE before my next upgrade.

Upload the image onto the controllers in advance, but make sure you change the boot statement back.

On the night, just log in and change the boot statement to the partition with the new version and then roboot.

You should disconnect the locals from the master controller though, by changing the localip.  Command syntax can change between versions, so if they are not on the same version, the master can push config that is not valid.

Personally I would do as follows (assuming the image is already loaded onto the controllers)

• Disconnect locals from the master.
• Shutdown the master-vrrp on the master-backup.  Upgrade the master-backup.
• Upgrade the master.
• Enable the master-vrrp on the master-backup.  Make sure they are both up and running fine without issue with master-redundancy etc.
• If your APs are mostly on the same subnet as the locals, disable that DNS entry for aruba-master, to ensure the APs upgrade their image from their local instead of across the wan to the master.
• Upgrade your locals.
• Ensure they come up fine and APs are upgraded and back up.
• When you are done, put back in the original localips on the master, so the locals can communicate again with the master.
• Reenable the DNS entry for aruba-master so that any APs not on the same subnet as a local can then upgrade on the master.

Certainly do a few first to make sure everything is ok, then get airwave to reboot all the other controllers.

And most importantly, make sure you release notes and in particular the upgrade instructions.

Good luck