Wireless Access

last person joined: 20 hours ago 

Access network design for branch, remote, outdoor, and campus locations with HPE Aruba Networking access points and mobility controllers.
Expand all | Collapse all

Global VIA outage

This thread has been viewed 0 times
  • 1.  Global VIA outage

    EMPLOYEE
    Posted Jun 15, 2012 10:25 AM

    VIA for Windows stopped working for all users on June 15.  The root cause is a code signing certificate that expired.  To fix the problem, a new VIA client download will be required.  An updated version will be posted to the support site within the next 30 minutes.

     

    As a temporary workaround to get clients back online, set the Windows system clock back by two days.

     

    Why did this happen?  Short and simple, we screwed up in not understanding that an expired code signing certificate would cause this amount of damage.  Nobody had seen a case before where Windows would stop running software that was previously installed because of something like this.  Needless to say, once the fire-fighting is done we're going to have a detailed investigation into how we got here.

     

    We realize this is going to cause huge headaches for every VIA administrator out there, and the only thing I can do right now is apologize on behalf of the company.

     

    I will update this thread as updates become available.



  • 2.  RE: Global VIA outage

    EMPLOYEE
    Posted Jun 15, 2012 10:50 AM

    Updated client software is now available on the support site.

     

    In addition, I am in the process of uploading VIA to the Tools section of the support site, where no login is required.  This will let you hand out direct download links to end users.



  • 3.  RE: Global VIA outage

    EMPLOYEE
    Posted Jun 15, 2012 11:04 AM

    [Removed - new links posted below]

     

    If you need Dell or Alcatel-Lucent branded installers, the root folder is here:

    [New link below]

     

    These do NOT require a support site login - you can give out these URLs to end users for direct download.



  • 4.  RE: Global VIA outage

    EMPLOYEE
    Posted Jun 15, 2012 11:49 AM

    Passing along some notes from our TAC engineers on doing this upgrade:

     

     

    •             For any Windows VIA 1.X users, they must upgrade to VIA 2.x.

     

    •             In AOS 5.X, VIA was packaged as part of the AOS image itself, so customers may not be able to upload the new binaries to the controller.  Hence, we must turn OFF the ‘Allow client to auto-upgrade’ to ensure all clients don’t try to automatically update.

     

    •             In AOS 6.X, VIA can be updated on AOS using ‘Advanced Services > VPN Services > VIA’

    If administrators don’t update the controller-hosted VIA image, then we must turn OFF the ‘Allow client to auto-upgrade’ to avoid downloading the old versions as the clients are already updated.

     

     

    VIA 2.0.x Instructions for Upgrade

    As upgrade was not supported in this release (allowing users to upgrade by double-clicking the MSI file), users must manually uninstall and install this new version.

         Note – Administrator privileges are required to do uninstallation and installation.  Also, we will need to reboot (User will be prompted) after uninstallation before proceeding to installation.

     

    Note2 - It is possible to upgrade VIA 2.0 from the command line or through a batch script using the following:

     

    > msiexec REINSTALL=ALL REINSTALLMODE=vamus /qr /i ansetup.msi

     

     

     

    VIA 2.1.X Instructions for Upgrade

    Upgrade is supported in this release.  User must double-click on the executable, or choose "Run" when downloading through a browser, to update to this version.

    Note – Administrator privileges are required to do installation



  • 5.  RE: Global VIA outage

    Posted Jun 15, 2012 12:27 PM
    what about the Mac i cant connect using mac also. I update the windows and its working now but my mac user still having issue they are working yesterday morning. Thanks a1rhead


  • 6.  RE: Global VIA outage

    Posted Jun 15, 2012 12:40 PM
    Sorry its working. due to changes i did on the controller to fix the windows VIA issue i did some changes that affect the mac os i restore it already since the windows is fix. thanks a1rhead


  • 7.  RE: Global VIA outage

    EMPLOYEE
    Posted Jun 15, 2012 12:49 PM

    Thanks.  I was worried for a minute.  Our QA testing indicated that MacOS and iOS versions are NOT affected, so I'm glad to hear you were able to resolve the problem.



  • 8.  RE: Global VIA outage

    EMPLOYEE
    Posted Jun 15, 2012 05:58 PM

    A few folks have been asking for additional technical details.  First, it turns out it wasn't the code signing certificate at all, as I said in my original post (I was posting before morning coffee kicked in).  Here are the details of what went wrong:

     

    When Windows software is issued, it is digitally signed by the manufacturer using a “code signing certificate”.  If the code signing certificate were to expire, software would stop functioning.  To counteract this problem, a “time stamping” service, provided by a public PKI vendor, is used.  The timestamping service allows software to continue functioning as long as the code signing certificate was valid at the time the software was signed.

     

    In this case, Aruba’s code signing certificate did not expire.  However, the public certificate of the timestamping service did expire on June 15.  Aruba software developers did not recognize the existence of the timestamping service certificate, failed to check the expiration date of this certificate, and were thus unprepared for the failure.

     

    The ironic twist is that the very technology that was designed to prevent this sort of thing from happening is what bit us.



  • 9.  RE: Global VIA outage

    EMPLOYEE
    Posted Jun 19, 2012 08:13 PM

    We have a "VIA updater" tool that can fix the VIA problem described in this thread, and does not require admin rights to do so.  The tool is considered 'beta' quality right now, and we are looking for anyone who wants to test it.  Please leave a reply here, and I'll private message you with a download location.  Our intention is to release the updater to everyone this week.



  • 10.  RE: Global VIA outage

    EMPLOYEE
    Posted Jun 20, 2012 09:57 AM

    Update on VIA:

     

    -          The "emergency" VIA binaries we provided on Friday work now, but have the same certificate expiration issue as the previous one – the clock runs out in December of this year.

    -          New VIA binaries are now available which permanently correct the problem.  We recommend that all customers upgrade.  Links below.

    -          Engineering is doing some final testing on a “VIA updater” utility that will fix broken VIA installations WITHOUT needing admin rights.  We hope to release this publicly today.

     

    Links to latest VIA binaries:

     

    VIA 2.0.1.1 32-bit:               http://bit.ly/LkcqfS

    VIA 2.0.1.1 64-bit:                http://bit.ly/Ps5361

     

    VIA 2.1.0.1 32-bit:                http://bit.ly/N8Ool2

    VIA 2.1.0.1 64-bit:                http://bit.ly/Ps5OMs

     

    If you need VIA binaries for OEM vendors (Alcatel-Lucent, Dell) you can find the root directory here:

    http://support.arubanetworks.com/TOOLSRESOURCES/tabid/76/DMXModule/514/EntryId/8195/Default.aspx

     



  • 11.  RE: Global VIA outage

    EMPLOYEE
    Posted Jun 21, 2012 08:54 AM

    The VIA updater tool has been released.  This will do the following:

    - Update VIA on all flavors of Windows, 32/64-bit, Windows XP, Vista, Win7

    - If users are running VIA 2.0.1 or earlier, this tool will install VIA 2.0.1.1.34125

    - If users are running VIA 2.1, this tool will install VIA 2.1.0.1.34126

     

    http://support.arubanetworks.com/TOOLSRESOURCES/tabid/76/DMXModule/514/Command/Core_Download/Default.aspx?EntryId=8228

     

    or, if you like shortened URLs better:

    http://bit.ly/MH3W1S

     

    Please let us know on this forum if you have any problems with the tool.  Out of 100+ test runs, we've had one failure (still unexplained) so we have decent confidence in the tool working. 



  • 12.  RE: Global VIA outage

    Posted Jul 12, 2012 05:50 AM

    Hello,

     

    Any chance the shortened URL http://bit.ly/MH3W1S  be activated for open/public download of the updater please?

     

    I have some client laptops out in the field that require the updater and the shortened url is no longer working but points to the support login portal.

     

    Please treat this as urgent.

     

    Ravi K

    HPA

    London - UK



  • 13.  RE: Global VIA outage

    EMPLOYEE
    Posted Jul 12, 2012 05:57 AM

    I'm not sure if I can update that shortened URL, but here's a link to the full URL:

     

    http://support.arubanetworks.com/TOOLSRESOURCES/tabid/76/DMXModule/514/Command/Core_Download/Default.aspx?EntryId=8318

     

    Here's a new shortened URL:

     

    http://bit.ly/MjP10L

     



  • 14.  RE: Global VIA outage

    Posted Jul 12, 2012 06:38 AM

    Thank you.

     

    For the speedy response.

     

    Ravi.