Wireless Access

last person joined: 4 hours ago 

Access network design for branch, remote, outdoor and campus locations with Aruba access points, and mobility controllers.
Expand all | Collapse all

Managing AOS8 upgrades

  • 1.  Managing AOS8 upgrades

    Posted Jul 29, 2019 12:01 PM

    Knowing that I'm not the only one to hit the problem of wayward APs refusing to upgrade in a timely fashion, I wondered how folk are handling upgrading AOS8 controllers with large networks.

     

    A recent upgrade to 8.4.0.4 took about 11 hours, mainly because in most partitions of APs being upgraded at least one didn't play ball. 

     

    If we were unlucky enough to have belligerent APs in every partition it would take more than 17 hours to upgrade our network. For some reason these times are not seen as acceptable maintenance windows.... 

     

    So how are you handling this?



  • 2.  RE: Managing AOS8 upgrades

    Posted Jul 29, 2019 05:04 PM

    Declare an outage Window.  Upgrade everything at the same time (no live/cluster upgrade).

     

    Fight me.



  • 3.  RE: Managing AOS8 upgrades

    Posted Jul 31, 2019 04:23 PM

    Hahaha,.. I do love the way Aruba launch a fancy new feature and everyone in the know seems to say "Yeah, don't use that..."

     

    I'm with you on this. The rolling upgrade is a nice feature, and if it can be made more reliable and considerably less prone to doing nothing for hours I think it's a good option for campus sites that never really have total downtime... But next time I'll go back to the one hit upgrade I think.



  • 4.  RE: Managing AOS8 upgrades

    Posted Jul 31, 2019 04:34 PM

    @MatthewSeymour wrote:

    Hahaha,.. I do love the way Aruba launch a fancy new feature and everyone in the know seems to say "Yeah, don't use that..."

     

    I'm with you on this. The rolling upgrade is a nice feature, and if it can be made more reliable and considerably less prone to doing nothing for hours I think it's a good option for campus sites that never really have total downtime... But next time I'll go back to the one hit upgrade I think.


    That is not what I said.  If you are concerned about the time it takes, use a traditional upgrade.  Live Upgrade is for users who have no time for downtime.  It will take longer if you have sparse coverage.



  • 5.  RE: Managing AOS8 upgrades

    Posted Jul 31, 2019 05:40 PM

    Much of the upgrade wait times were addressed in AOS 8.4. However, some reports from the field indicate there are some still seeing long wait times. PLM and engineering are investigatin, but at the end of the day, if an AP image pre-load fails for some reason (interrupted file transfer, etc), it's better to wait then take another group of APs down as it could introduce a significant coverage gap for some clients. So slow-and-caution instead of fast-and-furious was the decided path, but it is being looked at and continually refined. 



  • 6.  RE: Managing AOS8 upgrades

    Posted Aug 01, 2019 04:14 PM
    Last time I did the upgrade I fell asleep and when I woke up it was done. So success?


    Sent from VMWare Boxer.


  • 7.  RE: Managing AOS8 upgrades

    Posted Aug 02, 2019 04:51 AM

    Ok, so jolity aside.... The issue is a large number of APs failed to preload. We have no pointers as to why. The AP303h and the AP203h appeared to be disproportionately represented. AOS created partitions of approx 120 APs at a time. Almost every partition of APs had at least one that didn't preload. This holds up the entire process for a good 50 minutes. 

     

    The rolling upgrade process is definitely our preferred method. We aim to communicate what's going on to users, but know the vast majority of students living on campus won't actually get the message for one reason or another. The design in many of our accommodation buildings should allow for hitless upgrade. 

     

    I went to bed. Sadly when I got up the upgrade had not completed. 

     

    I agree the aim should be to avoid downtime but it seems clear that once an AP has failed to preload that's it... It doesn't appear to matter how long you wait and how many times you retry, the AP is in a state where it's not going to work so it would be much better to move on if a preload fails.

     

    In our case 67 APs failed to preload, the upgrade took over 11 hours and those APs ultimately still just had to be rebooted. But now that's happening in the morning when it's more user affecting. If the system had just continued without 50 minutes of retries those APs would have been kicked in the middle of the night... much better.

     

    I like this upgrade approach... I want to use it.... I'm trying to be constructive here people.



  • 8.  RE: Managing AOS8 upgrades

    Posted Aug 02, 2019 07:44 AM

    Do you have a TAC case #?  If not, please open a case when the next upgrade fails so they can determine what your issue is and see if it is a new or existing issue.  A cluster upgrade should work and if it doesn't a TAC case should be opened to determine why.



  • 9.  RE: Managing AOS8 upgrades

    Posted Aug 02, 2019 08:03 AM

    That's a fair response and something I'll look to do... I'm currently dealing with TAC on the upgrade from 8.4.0.2 to 8.4.0.4 resulting in broken config on the MDs and the bucketmap not working... so I have bigger problems rn ;)