Wired Intelligent Edge

 View Only
Back to discussions
Expand all | Collapse all

AOS-CX Tech Tips: "allow-unsafe-updates"

This thread has been viewed 98 times

  • 1.  AOS-CX Tech Tips: "allow-unsafe-updates"

    Posted Aug 26, 2020 01:56 PM

    Greetings!

     

    What does the "allow-unsafe-updates" command do?

     

    Occasionally, AOS-CX switch software releases include firmware updates for switch components such as the ServiceOS, boot loader, power management controller, and switch port PHYs. Some of these components have a redundant backup, like ServiceOS; others, including the boot loader, do not. Updates to components in the latter category are considered "unsafe" as interrupting the update process may result in rendering that component unusable, requiring the switch to be serviced to recover the affected component; for this reason, "unsafe" updates are disabled by default.

     

    Nonetheless, these updates are often necessary when updating to a new switch software release for proper switch operation. Therefore, a configuration context command has been provided to temporarily allow "unsafe" updates before upgrading to a new AOS-CX version to ensure that all components are updated to support the new software.

     

    switch(config)# allow-unsafe-updates 10

     

    This command will enable non-failsafe updates of programmable devices for the next 10 minutes. You will first need to wait for all line and fabric modules to reach the ready state, and then reboot the switch to begin applying any needed updates. Ensure that the switch will not lose power, be rebooted again, or have any modules removed until all updates have finished and all line and fabric modules have returned to the ready state.

     

    WARNING: Interrupting these updates may make the product unusable!

     

    Continue (y/n)?

     

    The window for allowing these updates may be set within a range of 1 to 120 minutes (2 hours); setting it to 0 disables the feature.

     

    When upgrading the switch to a new version that includes component firmware updates, do not interrupt the boot process in any way to avoid rendering the switch unusable; these updates will generally result in the switch rebooting itself multiple times, which is normal. The update process may be monitored by connecting to the switch USB-C console port or via a USB-A to serial cable connecting the USB-A port to a terminal server (requires AOS-CX 10.05.0001 or later).



  • 2.  RE: AOS-CX Tech Tips: "allow-unsafe-updates"

    Posted Aug 27, 2020 04:05 AM

    Hi Matthew! does it imply that unsafe updates are instead managed automatically (without any user intervention) by the ArubaOS-CX update procedure when a particular ArubaOS-CX software version requires it (stating somewhere it requires it)? or it never happens (and it never happened before) if not manually and temporarily enabled before starting an update procedure? if so...how will be able to differentiate between a software release requiring it from a software release not requiring it before starting the update procedure?



  • 3.  RE: AOS-CX Tech Tips: "allow-unsafe-updates"

    Posted Aug 27, 2020 08:56 AM

    Hi Matthew,

     

    I have some questions about this:

    - Do you know if Netedit "change firmware" will enable "allow-unsafe-updates"?

    - What happens if we update the switch to a AOS-CX version that has some "unsafe-updates" but reload it without setting the "allow-unsafe-updates" setting?

    - Is there any way to know if anything is pending install?

     

    Thanks



  • 4.  RE: AOS-CX Tech Tips: "allow-unsafe-updates"

    Posted Aug 27, 2020 10:45 AM

    To answer questions in order:

     

    • Unsafe updates are not automatically installed when booting to a new AOS-CX software version, and will remain pending until the command is run and the switch is rebooted to allow the updates to install
    • If a new AOS-CX version includes firmware updates, including unsafe updates, this will be displayed in the CLI when boot system is run, and will prompt the user to run 'allow-unsafe-updates' if any unsafe updates are included
    • NetEdit does not currently enable 'allow-unsafe-updates' when changing firmware versions; this currently needs to be done directly from the switch CLI itself
    • Booting to a new AOS-CX version without allowing unsafe updates will not, as a general rule, result in any operational or stability issues in any current release; however, future releases may require these updates to be installed for full operational capability
    • To see any pending firmware updates, use the show needed-updates command


  • 5.  RE: AOS-CX Tech Tips: "allow-unsafe-updates"

    Posted Aug 27, 2020 12:08 PM

    Hi Matt! this is becoming very interesting.

     

    I did a show needed-updates against VSX Primary:

     

     

    vsx-1# show needed-updates           
  next-boot  Show needed updates of programmable devices after a reboot 
  <cr>       

vsx-1# show needed-updates 

Chassis :
    SKU ID            : JL479A

MODULE 'mc' :
    (present)

--------------------------------------------------------------------


Would have updated 0 device(s).

vsx-1# show needed-updates next-boot 

Chassis :
    SKU ID            : JL479A

MODULE 'mc' :
    (present)

--------------------------------------------------------------------


Would have updated 0 device(s).

     

     

    and, form what I see, my system doesn't need any unsafe update...but I'm curious to understand if having done a lot of updates from 10.01 to 10.02 and lately within 10.03 I eventually missed some (because of what you wrote on point 1). Is it ever possible? I have the logs of almost all the updates I did but only few where collected at Console level (to see the boot messages)...so I've a doubt I've ever seen a message asking me that a unsafe update was needed (and I simply missed to allow it).

     

    The point 4 recalls me a request I did about Global Status LED color change to Green (it's Amber, actually) on Aruba 8320 and Vincent Giles answered me that to fix it by software required an unsafe update (terms were: non fail safe). At that time, I simply thought it was a matter of just waiting Aruba to provide an ArubaOS-CX software version fixing it and then normally update the switch so the "responsibility" of doing such potentially dangerous update was de-facto automatically "embedded" into the software update procedure itself. Worth to note that still some documentation of Aruba 8320 reports Global Status LED in Amber associating it with hardware failure condition.



  • 6.  RE: AOS-CX Tech Tips: "allow-unsafe-updates"

    Posted Aug 27, 2020 01:10 PM

    +1 for parnassus question.

     

    I did some updates through Netedit, from 10.04 all the way to 10.05.0001, and now wonder if I'm missing some unsafe-updates. Show needed-updates tells me 0 updates are available.

     

    Can we see the actual firmware version of all the components and compare them to the latest versions?



  • 7.  RE: AOS-CX Tech Tips: "allow-unsafe-updates"

    Posted Aug 28, 2020 04:43 AM

    That's another interesting point: I'm not aware of any Release Notes pointing out that a particular ArubaOS-CX software version requires unsafe updates to be allowed in order to be "fully applied" in case of update.

     

    Maybe I'm wrong but ArubaOS-CX Release Notes, up to now, never reported what are ArubaOS-CX modules versions a software release is currently carrying nor I'm sure there is an ArubaOS-CX commands to query such ArubaOS-CX modules versions.



  • 8.  RE: AOS-CX Tech Tips: "allow-unsafe-updates"

    Posted Aug 29, 2020 11:09 AM

    I also missed that info, but the release notes do say to enable "allow-unsafe-updates".

     

     

    CAUTION: If upgrading from ML.10.04, this version contains new BootROM image and isrecommended to be updated along with a new SVOS. The BootROM update is a non-failsafeupdate. Do not interrupt power to the switch during the update process or the update couldpermanently damage the device.

Invoke the command to allow unsafe updates to proceed after a switch reboot. Proceed to step 3 withinthe configured time.

switch# config
switch(config)# allow-unsafe-updates 30

     



  • 9.  RE: AOS-CX Tech Tips: "allow-unsafe-updates"

    Posted Aug 29, 2020 01:35 PM

    What software version are you referring to? ...go figure...vsx automatic update procedure will become a little bit less automatic if we need to worry about that (by the way it's better than missing it at all).

     

    The point IMHO is having a clear trace about when is going to be necessary or convenient (if not mandatory) to allow unsafe updates before proceeding with any software update: this should be clearly documented or ArubaOS-CX should provide a command to let us verify that (sort of a pre-flight check...give me the software image you want to use to update and the switch will tell you what is going to be updated safely and unsafely, if allowed).



  • 10.  RE: AOS-CX Tech Tips: "allow-unsafe-updates"

    Posted Aug 29, 2020 01:38 PM

    10.05.0001 and 10.05.0010 both have that remark on release notes.

    At least for 6300/6400.



  • 11.  RE: AOS-CX Tech Tips: "allow-unsafe-updates"

    Posted Aug 29, 2020 02:17 PM

    Thanks, you see...it's also a matter of specifying the hardware platform we're referring to (I'm inclined to think that "modular" chassis - like 6400 and 8400X - are probably affected more...but it's just an "uneducated guess").

     



  • 12.  RE: AOS-CX Tech Tips: "allow-unsafe-updates"

    Posted Sep 01, 2020 01:13 PM

    We are investigating with AOS-CX engineering on the best way to present an update history for switch firmware components. In the meantime, most of these updates are logged in the switch event log, and can be viewed using the following command (with sample output from a 6300M):

     

    switch# show events -a -d hpe-isp

    ---------------------------------------------------

    Event logs from previous boots

    ---------------------------------------------------

    2020-08-23T05:57:18.291888+00:00 6300 hpe-isp[3153]: Event|7212|LOG_INFO|||Starting update for MODULE 'mc' DEVICE 'svos_primary' from version FL.01.05.0004 to version FL.01.06.0004

    2020-08-23T05:57:18.292607+00:00 6300 hpe-isp[3153]: Event|7213|LOG_INFO|||Update successful for MODULE 'mc' DEVICE 'svos_primary' from version FL.01.05.0004 to version FL.01.06.0004

    2020-08-23T05:57:18.293208+00:00 6300 hpe-isp[3153]: Event|7212|LOG_INFO|||Starting update for MODULE 'mc' DEVICE 'svos_secondary' from version FL.01.05.0004 to version FL.01.06.0004

    2020-08-23T05:57:18.294730+00:00 6300 hpe-isp[3153]: Event|7213|LOG_INFO|||Update successful for MODULE 'mc' DEVICE 'svos_secondary' from version FL.01.05.0004 to version FL.01.06.0004

    2020-08-23T05:57:18.301199+00:00 6300 hpe-isp[3153]: Event|7212|LOG_INFO|||Starting update for MODULE 'mc' DEVICE 'pmc_primary' from version 0x8 to version 0xC

    2020-08-23T05:57:18.303287+00:00 6300 hpe-isp[3153]: Event|7215|LOG_INFO|||Deferred update for MODULE 'mc' DEVICE 'pmc_primary' will be performed after an automatic module reset

    2020-08-23T05:57:18.305612+00:00 6300 hpe-isp[3153]: Event|7213|LOG_INFO|||Update successful for MODULE 'mc' DEVICE 'pmc_primary' from version 0x8 to version 0xC

    2020-08-23T05:57:18.308205+00:00 6300 hpe-isp[3153]: Event|7212|LOG_INFO|||Starting update for MODULE 'mc' DEVICE 'pmc_secondary' from version 0x8 to version 0xC

    2020-08-23T05:57:18.310339+00:00 6300 hpe-isp[3153]: Event|7213|LOG_INFO|||Update successful for MODULE 'mc' DEVICE 'pmc_secondary' from version 0x8 to version 0xC

    2020-08-23T05:57:18.312699+00:00 6300 hpe-isp[3153]: Event|7212|LOG_INFO|||Starting update for MODULE 'mc' DEVICE 'pmc_primary' from version 0x8 to version 0xC

    2020-08-23T05:57:18.317481+00:00 6300 hpe-isp[3153]: Event|7213|LOG_INFO|||Update successful for MODULE 'mc' DEVICE 'pmc_primary' from version 0x8 to version 0xC



  • 13.  RE: AOS-CX Tech Tips: "allow-unsafe-updates"

    Posted Feb 19, 2021 10:38 AM
    A little update about the allow-unsafe-updates approach:

    I noticed that ArubaOS-CX 10.6 Release Notes, since the initial ArubaOS-CX 10.06.0001 software release, started to report that is necessary to enable the allow-unsafe-updates feature for 30 minutes before starting the software update procedure (I mainly checked about Aruba 8320 but I believe it a generic suggestion added on every Aruba CX series)...what instead is something totally new to me is that the same recommendation kicked in on the latest ArubaOS-CX 10.05.0051 Release Notes (on earlier 10.5 releases there wasn't any reference about this suggested/required step). No reference on any ArubaOS-CX 10.4 Release Notes.

    So to recap:

    • ArubaOS-CX 10.6 -> the allow-unsafe-updates 30 step was added since 10.06.0001 software release and it is present on any latter Release Notes published
    • ArubaOS-CX 10.5 -> the allow-unsafe-updates 30 step was added since 10.05.005 software release only (basically yesterday)
    • ArubaOS-CX 10.4 -> the allow-unsafe-updates 30 step is still absent on 10.4, up to 10.04.3050 software release
    It's not totally clear to me if the allow-unsafe-updates 30 step should be applied on each and every software update within the same 10.x software branch or only between upgrades (say, as example, from 10.4/10.5 to 10.6 or from 10.4 to 10.5), the more I read it the more it's like it should be done always and in any case.

    ------------------------------
    Davide Poletto
    ------------------------------

    Original Message


  • 14.  RE: AOS-CX Tech Tips: "allow-unsafe-updates"

    Posted Jul 09, 2021 04:54 AM
    Hello Matthew, sorry to resurrect this thread but I want to ask if, with regard to "We are investigating with AOS-CX engineering on the best way to present an update history for switch firmware components", there were updates we could be aware of.

    Sorry to stress about this in general but I'm planning a VSX upgrade from 10.5 to latest 10.7 on Aruba 8320 VSX and I've some doubts on how to proceed: do I need to enable on both VSX nodes the "allow-unsafe-updates" (for, say, 30 minutes) and then proceed with the usual automated "vsx update-software" procedure or what? what are the potential interactions between the two?

    ------------------------------
    Davide Poletto
    ------------------------------

    Original Message


  • 15.  RE: AOS-CX Tech Tips: "allow-unsafe-updates"

    Posted Jul 12, 2021 01:26 PM
    Because VSX members maintain their own control planes, you would need to run the command on each member separately before initiating the software upgrade.

    Note that firmware updates that require this setting to be enabled are infrequent, and in most cases apply to major version upgrades (10.4 to 10.5, 10.5 to 10.6, etc). That said, it is not a bad idea to get in the habit of enabling this setting periodically for upgrades within the same major version, particularly if there have been multiple releases between upgrades of a particular switch (e.g. upgrading a 6300 from 10.06.0002 to 10.06.0130). If there are applicable firmware updates in the new software image, they will be applied during the boot process as long as the timer has not yet expired; if there are no applicable firmware updates or if the timer has expired, the switch will boot normally.

    At present, there are no changes planned for update history logging on AOS-CX; the command for showing all events generated by the hpe-isp daemon (show events -a -d hpe-isp) is currently the best option available.

    ------------------------------
    Matt Fern
    Sr. Technical Marketing Engineer, Aruba Switching
    Aruba, a Hewlett Packard Enterprise company
    ------------------------------

    Original Message


  • 16.  RE: AOS-CX Tech Tips: "allow-unsafe-updates"

    Posted Jul 13, 2021 03:29 AM
    Hello Matthew,

    Thanks for answering!

    "Because VSX members maintain their own control planes, you would need to run the command on each member separately before initiating the software upgrade."

    Yes, definitively.

    For the rest, understood.

    Thanks.

    ------------------------------
    Davide Poletto
    ------------------------------

    Original Message


  • 17.  RE: AOS-CX Tech Tips: "allow-unsafe-updates"

    Posted Jun 19, 2022 08:59 PM
    Hi Davide

    I understand you have done quite a few 8320 upgrades. I like to ask what timer do you specify for the 'allow-unsafe-updates'. Would 10 mins be sufficient for a standalone and VSX-pair? And does that mean I execute the 'vsx update-software' (for vsx-pair) and 'boot system primary' (for standalone) after 10 mins?

    Thanks
    Original Message


  • 18.  RE: AOS-CX Tech Tips: "allow-unsafe-updates"

    Posted Jun 20, 2022 10:03 AM
    Edited by parnassus Jun 20, 2022 10:17 AM
    Hi, to be honest I've left 30 minutes but, I guess, that also only 15 minutes should be a fine value (at least in my environment): the orchestrated VSX upgrade procedure is very fast (I would say less than 6 minutes for each VSX Member, consider I seldom saw them performing unsafe updates)​ and if you launch the automatic procedure (yes, what you posted above) immediately after the allow-unsafe-updates <minutes> CLI command you should not experience any particular issue; I believe the 30 minutes is very cautelative.

    Original Message


Related Content