Wireless Access

Hi All,

 

Please excuse the length of this post as we've been dealing with this problem for a month now. We've had Aruba engineers on the phone. The entire infrastructure team has been looking at this daily/hourly, and we still have no pattern to identify in order to fix the issue.

 

Problem: We've installed mostly 315's and 335's in a three story building. We had a professional heatmap done and installed 36 AP's and have about 140 clients connected at any given time. Since installing this system, we've dealt with an increasing number of "disconnects". These reports come from users being connected to "WePresent" devices and GoToMeeting presentations. They know, very quickly, that they've dropped wireless. The issue is, though, we do not see the user actually disconnecting from the AP. Their machine shows one of two errors:

1. Error 11004 Wireless security has stopped.

- This error will show up at the exact time the user reports their connection being interrupted. Our suspicion, and we've been able to confirm this on a few machines, is that they lose the ability to get to the default gateway.

2. WLAN AutoCOnfig detected limit connectivity, performing Reset/Recover.adapter

 

The user loses the ability to access anything, at this point. They have to wait a few moments for 1. the connectivity to return or 2. the adapter finally disconnects from the SSID.

 

Solutions we've tried:

1. The Aruba engineers have done the normal steps with our system. They've tweaked the power settings on the radio. We've monitored the VController to watch clients when they disconnect and have found they don't actually drop during this.

2. We've updated drivers to the latest, repeatedly. We've changed out Wireless NIC cards in machines. We thought we had a lock on a certain type of card, but over time, that has completely gone out the window. It seems to be affecting all types of cards and computers.

3. We turned off band steering temporarily.

4. We had an IDS system called Airtight on, but we turned that off in the beginning of troubleshooting.

5. And, of course, our Aruba system is as up-to-date as possible. Our computers are up-to-date running Windows 10.

 

I'm not sure what else to provide. Any help would be great. We're grasping at straws. So many straws that I'm starting to think it's rats running around the building cutting off wireless and then taping stuff back together. Any additional info you need from me, I can provide it. We've gathered a good amount of data.

 

Thanks in advance.

What version of code are you on?  Is this a new problem or has it always been there since installing?

Name:	Aruba Operating System Software
Type:	335
Build Time:	2017-08-04 21:26:21 UTC (build 60834) by p4build
Version:	6.5.4.0

And this problem has existed since we installed.

My first two questions without knowing enough about your configuration or environment:

1. How are you securing the connection?
2. Have you eliminated interference (both wifi and non-wifi sources?)

A quick google search turns up a fair number of folks with problems using the 3rd party "Wireless AutoConfig" drivers. The apparent fix from my limited research was to uninstall the AutoConfig drivers and let Windows install default drivers for the NIC.

Have you considered or tested that?

We have done a fair amount of testing on the drivers. We've used drivers from various websites as well as the default drivers through Windows. For the different types of NIC's, we're finding there are different versions for different manufacturers. We have primarily Dell drivers, but the ones on Dell's website are older than that on Intel, and different from that on HP's. We've installed the drivers from HP's site onto Dell's and so on in hopes we would hit pay dirt on that, but it hasn't made a difference.

If it was in my environment, I'd fire up Kismet or some other wireless sniffing tool on my laptop and try to watch a client as it failed.

 

That makes me think of two more things:

1. Is there any client which just works? (Linux laptop, Android tablet, iPhone - something not Win10??)
2. Can you set up Aruba's encapsulated packet capture from the AP to a Wireshark instance and watch a client as it fails?

 

might be a mpdu agg stall... set up a test SSID profile, something that can be used for testing by someone who is, er, still friendly. Create a fresh ssid and vap profile, use your existing aaa profile and use the same opmode etc. Call it "whatever-test" and ask someone to use that while you try to narrow it down.

 

Set the channel to 20Mhz and disable vht, ht and mpdu-aggregation for that test VAP that you made. It will require some hacking about of radio profiles, ht ssid profiles and the like but the idea here is get the connection back to a basic 11a or perhaps11n connection and remove features that can cause your symptom here (mpdu agg, qos) and see if it stabilises.

 

You could consider as a most basic test to just disable 2.4ghz on your test VAP and see if that helps at all, then you can rule in/out movement between bands as a trigger.

 

Assumptions made in the above response

a) your clients are dual band

b) bouncing the wifi at the client (reboot, driver stop start etc.) recovers it

 

Questions...

1. does it just affect random users on some random AP ?

2. ever any occurrence of multiple users on same AP ?

3. is the AP stable (show ap bss-table, e.g. if some AP1 has a client that got stuck, what's its uptime).

4. have you collected the 'show ap debug system-status' when the issue is happening, if not, get that (usefulness depends on 1, 2 and 3 above)

 

This problem seems very similar to our reported issues, and the ones also reported in this thread as a bug:

http://community.arubanetworks.com/t5/Wireless-Access/surface-device-disconnection-issues/td-p/305219/page/3

 

As described above, supposedly the bug was around mpdu agg, and fixed in 6.5.4.2.  However, we're now running 6.5.4.2 and aren't completely convinced it's fixed.  Quite frankly, this seems like a big problem in the 3xx / 6.5 train, and difficult to diagnose.

Hi Phillip,

 

I read that thread and it is very similar! Like eerily similar. I have marked that thread for discussion within our group. It looks like we're running an older version (by .2) and we can do an upgrade.

I can't tell, though, whether or not it is fully fixed by that thread or if it just allows us to do the "workaround" it is suggesting.

 

We might do both, separately, to see what happens. I will report back.

The bug fixes descriped in the release notes (about video calls having poor performance, etc).  are supposed to contain the bug fix in the .2 release, and our TAC engineer configured that it would apply to all 3xx series APs (but we've also seen this issue on a 277).

Since you are on 6.5.4.0, it's probably not a big deal to go to .2.  Again, we're not convinced the problem is solved, but having more customers upgrade and report back would be helpful.

Well, we've updated ourselves to 6.5.4.2 and, unfortunately, have already had our first failure.....so we're back to the drawing board. 

 

In addition, we started investigating the possiblity of this being related to the Anniversary edition and Creators edition wifi problems we've read about....and.....nope. We had a failure on a brand new machine, running Intel Killer Wireless, and Creators edition, and it failed yesterday. 

 

We couldn't be more lost right now. Still unsure what else to try, but we've come up with the following:

 

1. Set all power settings on the wireless to "Maximum Performance"

2. Uncheck "Allow the computer to turn off this device to save power."

3. Turn off the 5Ghz radio on all WAP's. 

Yuck, turning off 5ghz is no good. Are you just stuck with TAC? I'd ask for an escalation if you haven't already. I know our engineer was trying to sniff the air while a device was in-state, but we haven't been able to get to a device before it fixes itself (or the user moves or reconnects).

Yeah, I'm not convinced it's fully fixed either in 6542, but we are still waiting for a confirmed case ourselves.

For us, it's impossible to predict or replicate a client failure.  I can take my devices out and around our network (~1500 APs) and not have a problem.   We've seem the issue occur on Mac OSX, iPhone, Android, and Surface, so it's not limited to a particular client.   

 

---

@philliplyle wrote:

For us, it's impossible to predict or replicate a client failure.  I can take my devices out and around our network (~1500 APs) and not have a problem.   We've seem the issue occur on Mac OSX, iPhone, Android, and Surface, so it's not limited to a particular client.   

 

---

Disconnects have many reasons, but there are also many ways to diagnose and solve them.  That does not mean they are not real.  If disconnects happen across all devices, that typically means the infrastructure needs to be looked at.  Do you have Airwave to monitor your infrastructure so that you can correlate your disconnects to utilization, load or errors?

Colin, I posted our TAC case number in the other surface thread.  We already have had ACE on site and a senior engineer involved.   

Okay.  Based on the TAC ticket it does not seem to be a surface-specific issue, so the "surface disconnects" thread is not specific to your case.  If you would like, you can open up your own thread about your issue or stick to this thread.  Posting that the issue is not fixed in the surface disconnects thread causes confusion among people looking for a surface fix.

Sorry, the forums don't make replies too easy to follow.

I didn't reply in that thread that the issue wasn't fixed. I was replying to your comment that the patch was a 'surface-specific' fix. There's nothing in the public release notes that suggest that particular bug was affecting only surface devices.. And our tac engineer didn't make any such conclusions either.

As an update to our issue, we still haven't resolved it. Our environment went quiet for almost two weeks, but we discovered it was due to users being tired of complaining about it.

 

We've done the ugprade, we've disabled the computer from being able to turn the wireless NIC off to save power....I've NEVER seen a problem like this in my lifetime. It is the most elusive issue possible.

 

Our next steps is to enable management frame encryption and to disable IPv6 on all our windows clients.

Yes, we have the same issue with users not reporting the problem, which is why we're not confident things have improved.

 

Our engineer wanted us to try to "sniff the air" from a differnet device next to a client while it is in-state.  Do your clients often stay disconnected for many minutes at a time, or only a very short period?  If so, this is something you can try - a wireless packet capture while the device is having the problem.  We haven't successfully pulled it off.

I'm having a similar issue and I'm running 8.1.0.4. It is one of the weirdest things I have seen, too. Suddenly the machine just goes into a black hole. Disconnecting and reconnecting helps, or just sitting and waiting for a minute or two. I didn't have this problem on the old 6.5 software. At the same time we upgraded to 8.1, we added in 2.4 Ghz Microsoft Display adapters in our organization. We are trying to narrow down if either/both of these things are contributing to the issues or if it is something else.

 

iann miller

Any update on your issue? We've been having similar issues with the same models (315 - 335) since implementation accross various clients. We only have a few Surface devices and at first I thought it was limited to those, but now other Windows clients report similar behavior. Also we have Windows CE clients that are affected. We've upgraded the APs to firmware 6.5.4.7 and the issue is not resolved, but may subjectively be happening less frequently.

 

Do you have Airwave by chance? If so, I'm curious what it looks like in your AMP console when a client experiences this issue. 

1. We've secured the connection using WPA2 AES via a Radius server. We've changed the Radius server out altogether. Blew it away, rebuilt another one, etc.

2. We've looked at interference on all levels. We have some printers that are broadcasting, and we're working on disabling those. We also have some mifi devices broadcasting, but there's nothing wildly crazy out there. I would be very surprised if this was our culprit given the amount of broadcasts most other environments must have naturally.