Wireless Access

We recently transitioned our network from WPA-TKIP to WPA2-AES.

After the transition we have received many complaints from Lync users that the call quality is basically unusable - choppy, only getting every other word, user cannot hear caller, caller cannot hear user.

My SE and I have looked at the settings from both networks and the only difference we can see is the encryption method.

I have looked at coverage and capacity as potential issues, and admittedly some of the offices do not have ideal coverage.  However, the problem is that we have turned the old TKIP network back on and the users report that the call quality on the old network is fine and "near perfect" - this is even with users that have poor health as reported on the controller dashboard - I'm talking health in the 2-10% range.  Speed tests from the user laptops show similar results regardless of which network they are connected.

Logically I can't find any reason why the encryption change would cause these issues, especially since these users can now connect at N speeds instead of the 54 that was limited by TKIP.

Our network was not designed (about 8 years ago!) with the idea that we would be supporting Lync or VoIP.  I do not have UCC monitoring, reporting, or any of those other features turned on.  I'm not sure what sort of load or other issues I would need to take into consideration if I were to do that - we have 230+ remote offices with local controllers, so I need to be careful with making any system wide changes.

Caveats and other issues:

I have not been able to duplicate these issues on our test network (configured exactly as our remote offices - same equipment, same outside WAN connection), using similar laptop hardware with the department image installed.  Of course that is making it very difficult to troubleshoot.

Is there anything that I should be looking at as to why this problem is happening with WPA2-AES and not with WPA-TKIP?

AOS - 6.4.2.3

Master Controllers:  3600

Local controllers: 620 and 7005

APs:  AP93, AP105, AP225

TKIP only supports up to 54 meg association rate (a/b/g).  AES supports up to any 802.11n speeds.  It could be that your clients have older drivers and have only been tuned or used to associating with TKIP.  Functioning better might require a driver upgrade.

What clients are these?

In addition, did you deploy client-side QOS on your Windows clients?

My on-site tech assured me that the wireless drivers were updated to the latest and greatest.

Part of the reason for the network change was to improve the speed/throughput and to finally move away from the outdated TKIP.

The clients are Windows 7 Pro.  To the best of my knowledge they are all Dell laptops - Latitude e6350, e5540, e5550 - possibly some others.  The account manager hasn't been able to give me a full inventory list of users and hardware.  I have seen one user with Dell 1707 wireless card (2.4 only), and other clients have dual-band capable cards.

I did send one user a dual-band USB adapter (StarTech USB867WAC22 - USB 3.0 to AC1200 802.11b/g/n/ac) to see if that made any difference.  Tech reported that it seemed to actually make it worse.

I'm not aware of client-side QoS on the clients, however we are not running QoS on the rest of the network.

I'm not familiar with any particular tuning for TKIP, however that is an interesting thing that I hadn't even thought about looking at.  The whole perplexing thing about this is that they claim phone quality is tons better on the older encryption and slower speeds, and none of that makes any logical sense.

I'll assume that would be inside the wireless adapter driver (usually in the advanced tab)?  I know that every driver is a little bit different and can have different names and default settings (especially for things like roaming aggressiveness).  Is there anything is particular I should be looking for?

My desktop adapter only has 3 settings, but the 2 laptops I am testing with have many others listed - just not familiar with them or sure which setting I should look at.

- What wireless adapters and driver dates/versions are we dealing with?  That would provide more insight.

- 802.11n provides more speed, but it also provides more opportunities for contention, where client's data traffic can easily starve other clients' VOIP traffic if QOS is not applied on the Client Side.

I would first look at the Lync VRD here: https://community.arubanetworks.com/aruba/attachments/aruba/Aruba-VRDs/4/1/Lync%20Over%20Aruba%20WiFi%20Validated%20Reference%20Design.pdf to see if you can manually apply Client-side QOS (page 77) and see if that makes a difference.  Microsoft asserts that client-side QOS is essential in any network with contention.  Wired networks have very little contention, but wireless networks do.

Thanks Colin, I'll see if I can get somebody on-site to try that client-side QoS stuff.

I was able to finally get the info for 4 of the users that seem to have the most problems/complaints with calls - (un)strangely enough they all seem to have the same hardware:

Latitude E5550

Microsoft Windows 7 Professional x64

Dell Wireless 1707 802.11b/g/n (2.4GHZ) Driver  DRVR     10.0.0.341

The user above is the one that we were able to work with the longest, and also the one that we upgraded drivers to latest, as well as tried the USB adapter. 

The other 3 users have same model laptop and wireless adapter, just look to be running a slightly older driver:

Dell Wireless 1707 802.11b/g/n (2.4GHZ) Driver  DRVR     10.0.0.293

When this all started it was a small list of users (4 to 6) in a handful of offices, and soon ballooned into "ALL users" in that department and the list grew to 20+ people and about 70 offices (the users are sales agents and travel to different branches).

I'm starting to think now that the final list was just a lazy grab of all users in that department in an effort to get the old network turned back on and avoid having to do any troubleshooting on the part of the account manager.

Unforunately I don't have access to a Latitude E5550 or anything with a Dell 1707 card.  The laptops I was given to play with are different models with different cards.

Now I'm beginning to wonder if it's something with that model or card. 

We were originally told that one of the users would be making a visit to us in HQ and we were hoping to be able to test, but that never happened.  Since we had many people out on leave,  we ended up just turning the old TKIP network back on (changed SSID to prevent all users from connecting) as a temporary fix and to get them working.   Obviously I can't leave that up and running as a permanent fix.

I may end up having to make the 2-4 hr drive to visit a user, but I'm not sure what I'm going to be able to do that a tech hasn't already done.

Of course the account manager is out this week, so I will have to wait a few more days before I have them try the client-side QoS thing.

I use Lync on my laptop and connect to the same AES network and have no problems, and like I said, I wasn't able to duplicate the problem with any other laptops, so more and more seems like an issue with a certain wireless adapter/laptop model.

Are the users only having problems on Lync?

What is your laptop adapter, driver and driver date?

Yes, seems to only be a problem with Lync.

My laptop is E6350 with Intel Centrino Ultimate-N 6300 AGN (15.7.0.3 dated 4/18/2013).

The other test laptop is E5540 with Intel Dual Band Wireless-AC 7260 (16.1.3.1 dated 8/1/2013)

The question is, are there more people where you are, or at the site where the problems are?  Lync is very delay and contention sensitive, and problems will occur first in those areas.  

You need to 100% need to apply the QOS on the client, otherwise all traffic leaving that client will be treated the same and step all over your Lync traffic.  There are quite a few organizations that we have gone through the client-side QOS issue with and applying it always makes it better.  Please see here:  http://community.arubanetworks.com/t5/Controllerless-Networks/IAP-Skype-For-Business-Audio-lag/m-p/301366  You might want to PM that user to get more info, possibly.

I haven't been able to pinpoint that as a possible issue.

I've had complaints in offices with only 2 other users (so 3 total), and also in offices with 8-9 connected users.

I was also able to test on a "sister" network in the same location - an affliated business using basically the same network (different name), but due to the way they operate, they have a seperate controller, AP, switch, router, and WAN connection.  That network only had 2 users connected - our problem user still had problems with the WPA2-AES connection.

In my test locally, there were 7 users, mostly help desk users, and I'm pretty sure they spend most of their time streaming video on their phones/tablets.  I was also about 100 feet away, connecting to an AP mounted on top of a metal wall-mounted rack (pointing down into the cabinet) and going through the wall of 2 offices.  My health was 5% on the dashboard, speed tests were under 1 Mbps.  I tried my best to create the worst possible scenario.   The only time I was able to experience any phone quality issues is when I moved another 12-15 feet away and was basically at 0% connection.

Since it's all coming from the same hardware, would the contention be the same regardless of the number of networks being broadcast (like would 9 users spread out over 2 networks cause the same contention as 9 users over 3 or 4 networks)?

The account manager has been out on vacation, but I was finally able to get information regarding our Lync servers, so I hope to be able to do more testing and work on getting the client-side QoS configured on those laptops.

I decided to do a little more looking around and see that one of our "problem" users has connected 30 times over the past 2 weeks.  22 of those times have been to the new WPA2/AES network, and 8 to the old TKIP network.

She is reporting that her Lync issues have gone away.

I also noticed that 24 of her connections are reported as 11b 2.4 GHz, 5 as 11n, and 1 as 11g.

I had read that it might help to disable certain connection speeds to basically prevent 11b connections and to provide better experiences for all users, but I have not done that as of yet.  I understand that if I disable lower connections, I may end up exposing coverage issues within my offices (230+).

I initially thought this might be a coverage issue until the number of users started to increase.  That also seemed to not be the issue after asking our tech to move closer and watching the user connection improve from 5% up to close to 100% (as per the controller dashboard).   The reports were always the same - choppy on AES, perfect on TKIP.

Of course seeing that the user has spent the majority of her time connected to the AES network seems to contradict everything that she is reporting.

I somehow could not see it in your post, but did you configure client-side QOS on the lady's device?  I don't understand what was done to make her issue go away.

No, we haven't yet been able to make any changes to the client.......which just baffles me even more.  I just received the Lync documentation on Wednesday, and the account manager just got back from vacation and has been busy catching up.  I can't make any changes to the client PCs unless it goes through the account manager.

He actually came to take back our test laptops yesterday and I asked if there had been anymore complaints from the users and was told "no, everybody is happy now."

I found it interesting because I had just looked at one of the users and saw that she was connected to the AES network.  I specifically asked about that user and got same reply "EVERYBODY is happy."

This particular client has been the most vocal about the issue and is the one that we did the most testing with - updated drivers, moved closer to AP, connected to similar network (different WAN connection, less users) and she always had the complaints about poor Lync quality unless she connected to TKIP.

Of course I haven't shared this information - I'm still trying to collect connection information on the other users and compile data on which offices, controller and AP types, if we've updated offices for coverage, number of users in the offices, and any other data that might be useful.  With a long list of users that constantly travel to many offices, it's been a nightmare trying to keep track of them all.

So, no, we haven't made the QoS change yet, but I will try to get to done for at least a couple of users next week to see if that helps.

If it doesn't, I imagine I'll be traveling 4-5 hrs to do some sort of on-site testing.

Unfortunately,  wifi traffic is very bursty so contention can occur at any time.  You need to make the QOS changes first (mandatory) to as many clients as possible.  You can the measure after.  Maybe their Lync server has a way to measure MOS scores before and after you made those changes to determine if things are indeed better or not.  Let that be your true litmus test.