Wireless Access

Reply
Moderator
Posts: 243
Registered: ‎09-12-2007

Global VIA outage

VIA for Windows stopped working for all users on June 15.  The root cause is a code signing certificate that expired.  To fix the problem, a new VIA client download will be required.  An updated version will be posted to the support site within the next 30 minutes.

 

As a temporary workaround to get clients back online, set the Windows system clock back by two days.

 

Why did this happen?  Short and simple, we screwed up in not understanding that an expired code signing certificate would cause this amount of damage.  Nobody had seen a case before where Windows would stop running software that was previously installed because of something like this.  Needless to say, once the fire-fighting is done we're going to have a detailed investigation into how we got here.

 

We realize this is going to cause huge headaches for every VIA administrator out there, and the only thing I can do right now is apologize on behalf of the company.

 

I will update this thread as updates become available.

---
Jon Green, ACMX, CISSP
Security Guy
Moderator
Posts: 243
Registered: ‎09-12-2007

Re: Global VIA outage

Updated client software is now available on the support site.

 

In addition, I am in the process of uploading VIA to the Tools section of the support site, where no login is required.  This will let you hand out direct download links to end users.

---
Jon Green, ACMX, CISSP
Security Guy
Moderator
Posts: 243
Registered: ‎09-12-2007

Re: Global VIA outage

[ Edited ]

[Removed - new links posted below]

 

If you need Dell or Alcatel-Lucent branded installers, the root folder is here:

[New link below]

 

These do NOT require a support site login - you can give out these URLs to end users for direct download.

---
Jon Green, ACMX, CISSP
Security Guy
Moderator
Posts: 243
Registered: ‎09-12-2007

Re: Global VIA outage

Passing along some notes from our TAC engineers on doing this upgrade:

 

 

•             For any Windows VIA 1.X users, they must upgrade to VIA 2.x.

 

•             In AOS 5.X, VIA was packaged as part of the AOS image itself, so customers may not be able to upload the new binaries to the controller.  Hence, we must turn OFF the ‘Allow client to auto-upgrade’ to ensure all clients don’t try to automatically update.

 

•             In AOS 6.X, VIA can be updated on AOS using ‘Advanced Services > VPN Services > VIA’

If administrators don’t update the controller-hosted VIA image, then we must turn OFF the ‘Allow client to auto-upgrade’ to avoid downloading the old versions as the clients are already updated.

 

 

VIA 2.0.x Instructions for Upgrade

As upgrade was not supported in this release (allowing users to upgrade by double-clicking the MSI file), users must manually uninstall and install this new version.

     Note – Administrator privileges are required to do uninstallation and installation.  Also, we will need to reboot (User will be prompted) after uninstallation before proceeding to installation.

 

Note2 - It is possible to upgrade VIA 2.0 from the command line or through a batch script using the following:

 

> msiexec REINSTALL=ALL REINSTALLMODE=vamus /qr /i ansetup.msi

 

 

 

VIA 2.1.X Instructions for Upgrade

Upgrade is supported in this release.  User must double-click on the executable, or choose "Run" when downloading through a browser, to update to this version.

Note – Administrator privileges are required to do installation

---
Jon Green, ACMX, CISSP
Security Guy
New Contributor
Posts: 4
Registered: ‎01-12-2012

Re: Global VIA outage

what about the Mac i cant connect using mac also. I update the windows and its working now but my mac user still having issue they are working yesterday morning. Thanks a1rhead
New Contributor
Posts: 4
Registered: ‎01-12-2012

Re: Global VIA outage

Sorry its working. due to changes i did on the controller to fix the windows VIA issue i did some changes that affect the mac os i restore it already since the windows is fix. thanks a1rhead
Moderator
Posts: 243
Registered: ‎09-12-2007

Re: Global VIA outage

Thanks.  I was worried for a minute.  Our QA testing indicated that MacOS and iOS versions are NOT affected, so I'm glad to hear you were able to resolve the problem.

---
Jon Green, ACMX, CISSP
Security Guy
Moderator
Posts: 243
Registered: ‎09-12-2007

Re: Global VIA outage

A few folks have been asking for additional technical details.  First, it turns out it wasn't the code signing certificate at all, as I said in my original post (I was posting before morning coffee kicked in).  Here are the details of what went wrong:

 

When Windows software is issued, it is digitally signed by the manufacturer using a “code signing certificate”.  If the code signing certificate were to expire, software would stop functioning.  To counteract this problem, a “time stamping” service, provided by a public PKI vendor, is used.  The timestamping service allows software to continue functioning as long as the code signing certificate was valid at the time the software was signed.

 

In this case, Aruba’s code signing certificate did not expire.  However, the public certificate of the timestamping service did expire on June 15.  Aruba software developers did not recognize the existence of the timestamping service certificate, failed to check the expiration date of this certificate, and were thus unprepared for the failure.

 

The ironic twist is that the very technology that was designed to prevent this sort of thing from happening is what bit us.

---
Jon Green, ACMX, CISSP
Security Guy
Moderator
Posts: 243
Registered: ‎09-12-2007

Re: Global VIA outage

We have a "VIA updater" tool that can fix the VIA problem described in this thread, and does not require admin rights to do so.  The tool is considered 'beta' quality right now, and we are looking for anyone who wants to test it.  Please leave a reply here, and I'll private message you with a download location.  Our intention is to release the updater to everyone this week.

---
Jon Green, ACMX, CISSP
Security Guy
Moderator
Posts: 243
Registered: ‎09-12-2007

Re: Global VIA outage

Update on VIA:

 

-          The "emergency" VIA binaries we provided on Friday work now, but have the same certificate expiration issue as the previous one – the clock runs out in December of this year.

-          New VIA binaries are now available which permanently correct the problem.  We recommend that all customers upgrade.  Links below.

-          Engineering is doing some final testing on a “VIA updater” utility that will fix broken VIA installations WITHOUT needing admin rights.  We hope to release this publicly today.

 

Links to latest VIA binaries:

 

VIA 2.0.1.1 32-bit:               http://bit.ly/LkcqfS

VIA 2.0.1.1 64-bit:                http://bit.ly/Ps5361

 

VIA 2.1.0.1 32-bit:                http://bit.ly/N8Ool2

VIA 2.1.0.1 64-bit:                http://bit.ly/Ps5OMs

 

If you need VIA binaries for OEM vendors (Alcatel-Lucent, Dell) you can find the root directory here:

http://support.arubanetworks.com/TOOLSRESOURCES/tabid/76/DMXModule/514/EntryId/8195/Default.aspx

 

---
Jon Green, ACMX, CISSP
Security Guy
Search Airheads
Showing results for 
Search instead for 
Did you mean: