Wired Intelligent Edge

 View Only
last person joined: 2 days ago 

Bring performance and reliability to your network with the HPE Aruba Networking Core, Aggregation, and Access layer switches. Discuss the latest features and functionality of your switching devices, and find ways to improve security across your network to bring together a mobile-first solution
Back to discussions
Expand all | Collapse all

Product Bulletin: CX Storage Improvement Software Releases - November 2020

This thread has been viewed 78 times

  • 1.  Product Bulletin: CX Storage Improvement Software Releases - November 2020

    MVP GURU
    Posted Nov 19, 2020 09:21 AM
    Hi all!

    reading the Product Bulletin "CX Storage Improvement Software Releases - November 2020" released just few days ago (last Saturday, if I'm not mistaken) I can't understand what exactly Aruba CX series are susceptible of reported "storage exhaustion affecting the usable life of the non-volatile memory" and what are not.

    The bulletin reports all Aruba CX Switch series as affected - minus the Aruba 8360 (probably because it is running only on AOS-CX 10.6 or newer) - saying that all Aruba CX Switch Series are currently susceptible to storage exhaustion ("CX 6200, CX 6300 stack members and CX 6400 or 8400 management modules running as standby")...thus...given that annotation, it seems to me that impacted series are mostly 6200, 6300, 6400 and 8400 and only in particular configurations while others are not, totally ("Standalone 6200/6300/83xx, 6200/6300 stack commanders and 6400/8400 with single management module/active management module are not susceptible to storage exhaustion.").

    The point is that the Important Information message available on 10.05.0021/10.04.3031 Release Notes:
    is totally absent on Aruba 8320 and 8325 Switch Series' Release Notes for those builds.

    Is the absence of that informative message reasonable or not on Aruba 8320/8325?

    In other terms it would be nice to understand if, as example, a standalone Aruba 8320 switch (running any build of ArubaOS-CX 10.03, 10.4 or 10.5) shall be upgraded to suggested versions (10.04.3031 or newer, 10.05.0021 or newer) no matter what, for the reason listed above as well as for having important security reports like the CVE-2020-7121, CVE-2020-7122 fixed or simply it's not mandatory at all (because Aruba 8320 and 8325 are de-facto not susceptible of the storage exhaustion issue).

    That's to understand the urgency behind recommended update and to understand if the issue (as its fix) has a software origin...if so...maybe...some older software versions (such as 10.03) are currently immune while others (10.4 or 10.5 are affected and are affecting the underlying hardware on particular Switch series only).

    ------------------------------
    Davide Poletto
    ------------------------------


  • 2.  RE: Product Bulletin: CX Storage Improvement Software Releases - November 2020

    Posted Nov 21, 2020 05:46 PM
    From my experience, it seems to affect 6200 and 6300 running as VSF.
    All my secondaries and members currently have wear > 90% (imminent failure) and 80-90% . Masters stay normal (0-10%).

    ------------------------------
    Ricardo Duarte
    ------------------------------

    Original Message


  • 3.  RE: Product Bulletin: CX Storage Improvement Software Releases - November 2020

    Posted Nov 22, 2020 07:57 AM
    Where can I see the wear level?

    Also is 10.05.0011 new enough? It's not very clear on the release notes what is new enough version
    Original Message


  • 4.  RE: Product Bulletin: CX Storage Improvement Software Releases - November 2020

    Posted Nov 22, 2020 10:07 AM 
    show system resource-utilization all | include Endurance​


    ------------------------------
    Ricardo Duarte
    ------------------------------

    Original Message


  • 5.  RE: Product Bulletin: CX Storage Improvement Software Releases - November 2020

    Posted Nov 22, 2020 10:11 AM
    Thanks. Couple VSF stacks I checked seem to all have 0-10% wear. So wonder if 10.05.0011 is considered new version enough or if I really need to get some downtime agreed for tomorrow evening :)
    Original Message


  • 6.  RE: Product Bulletin: CX Storage Improvement Software Releases - November 2020

    Posted Nov 22, 2020 10:22 AM
    I would not risk running older CX firmware.
    Also 10.05.0011 has some really ugly bugs that were fixed on later versions.

    ------------------------------
    Ricardo Duarte
    ------------------------------

    Original Message


  • 7.  RE: Product Bulletin: CX Storage Improvement Software Releases - November 2020

    MVP GURU
    Posted Nov 22, 2020 11:42 AM
    Hi @pubjohndoe,

    "Also is 10.05.0011 new enough? It's not very clear on the release notes what is new enough version"

    The suggested remediation action was to update to ArubaOS-CX 10.05.0021 or newer (thus ArubaOS-CX 10.05.0030 at present day) or to update to ArubaOS-CX 10.04.3031 or newer (thus ArubaOS-CX 10.04.3040 at present day) for those that have their Aruba switches on ArubaOS-CX 10.5/10.4 respectively​.

    Note 1: If I'm not wrong the ArubaOS-CX 10.05.0030 and 10.04.3040 were published just two days ago (IIRC 20th November); ironically for me, I upgraded our VSX to 10.05.0021 from 10.03.090 that very same day...but, given at least two bugs that are worrying me - one "cosmetic" about Amber LED when negotiated port speed is 1Gbps and the other about VSX sync to secondary - I plan for sure to update to very latest ArubaOS-CX 10.05.0030 as early as possible within two week at worst.

    Note 2: I checked the storage endurance on our two Aruba 8320 switches when they were running ArubaOS-CX 10.03.0090 (thus just before the upgrade to 10.05.0021) and it was OK on both (and it reported ssd_endurance(%) : 0.00) so, ironically again, the same CLI command used to understand the storage wearing (show system resources-utilization) produces no informative output on ArubaOS-CX 10.05.0021 about the wearing status of internal storage...I'm start thinking that, with that regard, the Aruba 8320 switch series is truly a lucky strike if they removed that information (the option "all" wasn't available neither before nor after the upgrade):

    vsx-1# show system resource-utilization 

System Resources:
Processes: 215       
CPU usage(%): 7         
Memory usage(%): 20
Open FD's: 1733​

    ​

    ------------------------------
    Davide Poletto
    ------------------------------

    Original Message


  • 8.  RE: Product Bulletin: CX Storage Improvement Software Releases - November 2020

    Posted Nov 22, 2020 11:49 AM
    I think I saw this first when someone on Reddit mentioned that 10.04.3031 mentioned this, this was 11 days or so. In that it is mentioned:

    " Software older than version 10.04.3031 writes to this memory at anunintended and accelerated pace. "

    So they just updated the text on the newer version and it wasn't really clear what is the actual version where this was first fixed.

    In any case we try to update the switches next week
    ​
    Original Message


  • 9.  RE: Product Bulletin: CX Storage Improvement Software Releases - November 2020

    MVP GURU
    Posted Nov 22, 2020 02:15 PM
    Wouldn't be really more simple (for the sake of completeness) to state what exact ArubaOS-CX (10.5/10.4) builds introduced the storage wearing issue? ...and, additionally, state also which switch series are really unaffected?



    Original Message


  • 10.  RE: Product Bulletin: CX Storage Improvement Software Releases - November 2020

    MVP GURU
    Posted Nov 22, 2020 06:35 PM
    Hi @pubjohndoe!

    "​I think I saw this first when someone on Reddit mentioned that 10.04.3031 mentioned this, this was 11 days or so"

    Yes the information you were referring to appeared on Reddit (this post on November, 10th) and contained a link to this article from the The Register technology news site, article published on November, 9th.


    ------------------------------
    Davide Poletto
    ------------------------------

    Original Message


  • 11.  RE: Product Bulletin: CX Storage Improvement Software Releases - November 2020

    Posted Nov 22, 2020 06:40 PM
    It's like you said previously that it would be a lot easier if the release notes would just have the exact information what version is affected and what is not. There are not that many CX versions released so far and at least Aruba should know which ones are affected :)
    Original Message


  • 12.  RE: Product Bulletin: CX Storage Improvement Software Releases - November 2020

    MVP GURU
    Posted Nov 22, 2020 06:41 PM
    Hi @ricardoduarte,

    "From my experience, it seems to affect 6200 and 6300 running as VSF."

    I fear...not only Aruba CX 6200/6300 deployed as VSF...at least from what we can understand by reading available documents.

    ------------------------------
    Davide Poletto
    ------------------------------

    Original Message


  • 13.  RE: Product Bulletin: CX Storage Improvement Software Releases - November 2020

    EMPLOYEE
    Posted Nov 23, 2020 01:59 AM
    Let me clear this up for everyone. 
    The Improvements to the Storage Usage was 1st implemented in the following releases: 
    For customers standardizing on 10.05, please upgrade to release 10.05.0021 or later  including moving up to 10.06 releases. 
               customers standardizing on 10.04, please upgrade to release 10.04.3031 or later  including moving up to 10.05 or 10.06 releases. 
                  customers standardizing on 10.03, please upgrade to release 10.03.0101 or later including moving up to 10.05 or 10.06 releases.

    8400, 8320  customers that are running 10.00, 10.01 or 10.02 -- you'll have to upgrade to 10.03.0101 as an intermediate step to move up toe 10.04 or above  (an update to the schema requires this intermediate step)  See the release notes for upgrade procedures. 

    Yes, most affected are 6200 or 6300 VSF stack Members (the Conductors are not as affected). 
           6400 & 8400 Management Modules running as Standby (the Active MM is not as affected)
              8320, 8325 are not as affected. 
    Nonetheless, we recommend upgrading to the 10.05.21 or 10.06 releases, across the board because of the improvements, which also include improved error warnings  when the Storage system reached 90%, 100% and beyond wear levels.  With the newer software, that should take years to reach on new systems. 

    All of the recent updates that are "later" than the above referenced numbers have these improvements already implemented. 

    Please be aware that ANY switch you receive from distributors, or resellers or even recent RMA replacements, most likely has not been built with the newer software.  PLEASE UPGRADE before deploying your switches. 

    DO NOT CONSIDER any release not mentioned above as 'immune' to this storage impact -- please upgrade (yes, even the 83xx switches). 



    ------------------------------
    Ed Kurata
    ------------------------------

    Original Message


  • 14.  RE: Product Bulletin: CX Storage Improvement Software Releases - November 2020

    MVP GURU
    Posted Nov 23, 2020 11:47 AM
    Hi @EdKurata, thank you for providing this clarification! for sure the Product Bulletin "CX Storage Improvement Software Releases - November 2020" released just few days ago [**] is not so clear (at least for me). I don't know if it was then updated or not.

    The downside is (for me) this tiny one: if I had known that the ArubaOS-CX 10.03 was not impacted by the problem (or, better, if I had known that the Aruba 8320 running on ArubaOS-CX 10.03 was a little bit more immune if compared to other newer Aruba CX switch series running on newer ArubaOS-CX software branches), I probably would not have upgraded to ArubaOS-CX 10.5 so quickly (preferring to have our Aruba 8320 VSX updated to ArubaOS-CX 10.03.0110 before anything else, postponing the entry into ArubaOS-CX 10.5 world for other few weeks [*]) although - and I agree 100% with you here - upgrading to ArubaOS-CX 10.05.0021 (or newer) was/is and will be indeed a very reasonable approach in terms of added Security Fixes, Features and stability improvements if compared to staying on older builds.

    [*] Indeed I will need to apply another update to move away from 10.05.0021 (for reasons described on my previous reply), waiting the next build after the 10.05.0030 very recently released.

    [**] For completeness and for future reference I found a post on the Airheads French Forum (published earlier - November 4th, 2020 - than the email from Aruba I was referring to) which has attached a "CX Storage Improvement Releases" document (basically similar to the email from Aruba we are discussing here).

    ------------------------------
    Davide Poletto
    ------------------------------

    Original Message


  • 15.  RE: Product Bulletin: CX Storage Improvement Software Releases - November 2020

    EMPLOYEE
    Posted Nov 23, 2020 01:24 PM
    Davide,  thanks for your continued diligence with our Aruba products. 

    Apologies for the communication not being clearer...  but yes, we DID apply the improvements to a 10.03 major version bug fix, that being 10.03.0101. 
    So, your upgrade could have been done to the 8320 up to the 10.03.0101 build.   Because none of the earlier releases had these improvements, and we wanted to ensure customers upgraded only to those designated builds, we removed any earlier code (the Release Notes are still posted, just the Software removed). 

    Let me assure you that 'later' builds have these same improvements in there, and we also continue to make other improvements as well over time, and they'll show up in even later 'dot releases' within each branch.   You can guess that we're slowing down on the dot releases for the older major versions (10.03, 10.04)  and will continue on the journey with our 10.06 and 10.05 major release versions.

    ------------------------------
    Ed Kurata, Aruba Sr. Switching Product Line Manager,
    Transceivers, Accessories, Power, and other things
    Roseville CA, USA
    ------------------------------

    Original Message