COTD: How to Upgrade
03-14-2010 06:44 PM - edited 04-07-2016 04:12 AM
Below is an ad-hoc guide to making your upgrade very predictable
Some may think that upgrading your Aruba infrastructure can be a confusing thing, but there are things that you can do to make it straightforward. Here is a list of things you can do to make it easier:
- Find out what version of ArubaOS all of your controllers are running (Do a "show version" on the commandline).
- Take a flashbackup of your controllers and optionally a logs-tar with tech support of each, as well.
- Enumerate what services you are using for each controller (Employee Wireless, Guest Access, Remote AP, Wireless Voice, etc).
- Find out exactly how many access points you have on each controller (the logs.tar above will give you that info).
- Find out exactly how each access point discovers each controller (DNS, DHCP Option, broadcast), and make sure those methods are working (you might want to reboot an AP to make sure).
- Determine if you are having any existing consistent or intermittent issues and seek to resolve them with the help of TAC (problems don't go away magically with an upgrade--but you already knew that, right?).
- Enumerate the devices in your infrastructure that are used to provide your wireless users with connectivity (Core switches, radius servers, DHCP servers, firewalls?).
- Get Airwave http://www.airwave.com to baseline all your inventory, wired AND wireless, and your user population (Your local Sales team would certainly help you to get this installed on an evaulation basis).
- IMPORTANT*** Backup all your flash and configs offline and submit your tech support to TAC for a recommendation on which version of code to upgrade to .
Technical Upgrading Advice:
- Know your topology. The most important path is the connectivity between your access points and the controller. If you have any issues with connectivity, this will keep you from completing your upgrade.
- If you don't know your topology, have someone during the upgrade that does and can test and possibly make changes to routing, switching, DHCP, authentication, etc to ensure everything in the path of traffic is functioning correctly.
- Avoid combining a controller upgrade with other upgrades---it is still tough to find another job ;)
- Plan your upgrade when there is the least amount of traffic in your network, so that if there are any issues, you do not have to wade through other stuff to get information.
- Inform your community and even send out a meeting invite so that nobody freaks out when they cannot download previews from the new "Game of Thrones".
- Make sure that all of your controllers are running the same version of code in a master-local relationship. This makes behavior in a multiple controller environment consistent; and a mismatch could keep your APs from coming up.
- Try to download code from the support website with Firefox or Chrome instead of Internet Explorer. Internet Explorer adds parenthesis in filenames which will cause the controller to reject the image. If you do have parentheses, regardless of the browser, remove them before uploading the image to the controller.
- If you cannot upload directly from your management PC, use FTP, NOT tftp to upload images to the controller. FTP is much faster when sending a 80 to 100 megabyte image to a controller and also offers more resilience over slower links. Some tftp servers cannot send over 30 megs and will error out inexplicably. There are a number of freeware ftp servers that can be used for this.
- 802.11n does not support any ciphers besides Open and AES. If you have any clients that are doing WEP or WPA-TKIP for example, those clients will be limited to only 54 megabit performance on the wireless even when connected to a 802.11n or 802.11ac access point.
- Always upgrade the partition opposite from the one the controller booted from. If anything happens and you need to revert, you could just restore the flashbackup, and switch back to the old partition. If you overwrite the partition the controller booted from, you have to look for the old version of code to upload later.
- The upgrade sequence play by play usually goes like this:
(a) Upload the same version of new images to all controllers
(b) Reboot all controllers at the same time
(c) do a Ping -t to find out when the controllers come back up
(d) SSH into the Master Controller
(e) Do a "show ap database" constantly to see if any access points are upgrading. Keep in mind that if you are upgrading between major versions of code, the messaging probably has changed and you might not see any access points upgrading (but then, you might, as well). Do a "show datapath session table" on the commandline and see if there is any traffic on port 21 (ftp) to see if any APs are upgrading.
(f) Do a "show log system all" to see if any access points are "rebooting after image upgrade"
(g) Do a "show ap database" to see if any access points are up and ready to accept clients. Carefully observe if there are any flags in the "flags" column like "ILGN or U" that would keep them from coming up.
(h) Do a "show ap active" to see how many APs are actually up and running
(i) Cycle between steps (e) and (h) until a sufficient amount of access points are up
(j) "show ap database" will show all of the access points, up or down. If some access points are down, do a "show datapath session table <ip address of that access point>" to see if they are sending the controller any traffic. If not, attempt to ping them. If they still do not respond, do a "show ap database long" so you can get the wired mac address so you can find it in your infrastructure.
(k) Airwave automates much of steps "a" through "j" and saves alot of time. It also allows you to easily compile all of those time-consuming "Wow!" reports that your higher ups like to see and take credit for.
(l) Check to make sure the number of access points and clients are what you would expect
(m) Test a different type of client for each access method (802.1x, VPN, Remote AP, Captive Portal, Voice). Test in different locations, if you can.
(n) Use the time you saved with Airwave to watch that episode of Family Guy.
In even an average world with gigabit connections, a single controller upgrading 200 access points would take about 15 to 20 minutes total from controller up to steady state and 90% of the time you can ignore all the steps above, because the upgrade will be uneventful. Testing your clients and ensuring performance and connectivity is probably the most time-consuming part of the upgrade, so enlist users in different locations to help ahead of time.
*Answers and views expressed by me on this forum are my own and not necessarily the position of Aruba Networks or Hewlett Packard Enterprise.*
ArubaOS 8.5 User Guide
InstantOS 8.5 User Guide
Airheads Video Knowledge Base
Remote Access Point Solution Guide
ArubaOS Consolidated Release Notes
ArubaOS 8 ViA VPN Solution Guide
what happens if controllers aren't rebooted simultaneously?
10-14-2010 08:29 AM
At the main location (which also has the master controller) we have the shortest maintenance window and need to get the main SSID working in that facility as quickly as possible.
I had planned to upgrade that controller, reprogram and get that sites main SSID going before upgrading any of the others. Is this ill-advised?
I could also simultaneously reboot/upgrade all the controllers...I just figured it would take a little longer before all sites reach a "clean" state for when I could do the write erase and reprogramming. The master controller actually terminates the most AP's of any site (18), so perhaps we wouldn't lose any time upgrading them simultaneously.
Great COTD, btw.