Wireless Access

Last Sunday we upgraded our controllers from version 6.1.3.8-Airgroup to 6.3.0.2.  We liked the new licensing, and the ability to hook AP-225's into the network.

 

We currently run 6 M3 blades in two 6000 chassis.  Master, Backup, and 4 local controllers.

 

Since the upgrade, we've had numerous issues with random client disconnects, and overall inability to communicate anywhere from 5 seconds to 30 seconds.  I originally thought the problem was with Client Match, as it seemed that the clients were attempting to connect to a different AP several times per minute.  During these peroids, the clients would go offline for 5-20 seconds (unable to ping IP address).

 

I believe now the problem is more widespread and affects even less AP dense areas (our high schools have an AP about every 100 feet).  We have entire classes where half of the class cannot connect, or stay connected throughout the period.  

 

One of the most prominate places utilizes Dell laptops in a mobile lab.  The AP is directly outside of the room, and only half of the clients can connect (seemingly random amount).  I've seen the issue on OS X machines, Windows 7, and Androids.

 

We didn't have any of these issues before the upgrade to 6.3.0.2.

 

Is anyone else experiencing issues like these with 6.3.0.2?  I currently have an open ticket with Aruba Support, but wanted to see if others in the community are having these types of issues.

 

Thanks in advance!

What type of network(s) are being affected?

e.g.  OPEN?   802.1x?  WPA2?

 

JF

I've heard similar reports from Windows 7/8, OS X and iDevices. I'm currently running 6.3.0.0 on an 802.1x network.

If its a dot1x network... I would recommend running this command, and  checking the two parameters below.

 

The key here is to see how many transcations are going on and are queued when the issue hits.

 

JF

 

(Controller) #show dot1x counters 

 

       802.1x Counters

Throttling Counters

Dot1x high watermark..................150

Dot1x low watermark...................100

Dot1x auth pass count.................0

Dot1x auth fail count.................0

Double dot1x context init counts......0/0

Active dot1x station count............0                     <-----------  Check this

Max collisions in active table........0

Pending station count.................0                         <-----------  Check this

Max collisions in pending table...….0

Can you provide an example of acceptable and unacceptable numbers?

 

I'm currently seeing:

 

Active dot1x station count............147
Max collisions in active table........5 <----------------------what's this?
Pending station count.................0

I've seen the issue on Open and 802.1x networks.  The only report we've had on the Open network was from the Android where it was connected just fine, disconnected for about 30 seconds, and reconnected, all while the user was in the same location.

 

Here is a copy of our dot1x counters:

 

802.1x Counters
Throttling Counters
Dot1x high watermark..................70
Dot1x low watermark...................40
Dot1x auth pass count.................121647
Dot1x auth fail count.................0
Double dot1x context init counts......19337/19337
Active dot1x station count............22
Max collisions in active table........3
Pending station count.................0
Max collisions in pending table.......0

Here's the support troubleshooting.  I thought the commands were very useful, and wanted to share:

 

#show clock

 

Note down the exact time of the issue happened ( The client disconnected and unable to connect to the network)

 

#show ap remote debug mgmt-frames client-mac <wireless device's mac address> ap-name <name of the AP client associated>

 

This will list the 802.11 management packets (Association Request, Association Response, Disassociation, and Deauth) for the specified wireless device.

 

#show log system all | include “<wireless device's mac address>

 

This command will show if the problem client is attempting to associate.  Look for the problem client MAC address.  It will also show to which AP the client MAC is attempting to associate.  Note the BSSID.

 

#show ap association client-mac “<wireless device's mac address>

 

Use the AP BSSID taken from the output of the previous command and user MAC.  This command can be used to verify if there are other users currently associated to this same AP thus helping to rule out infrastructure issues vs a single-client issue.

 

 

#show log errorlog all | include “<wireless device's mac address>”

 

This command can be used to determine if the RADIUS server is responding to client requests.  Look for excessive RADIUS timeouts or instances of the Aruba switch taking a RADIUS server out of service for the server hold-down timer.  (This indicates possible RADIUS server connectivity or performance issues and should be investigated as needed.)

 

#Show ap debug client-table ap-name <AP name where the wireless device is associated to>

 

In part of this CLI output it will display the wireless device’s Last_Rx_SNR, Tx_Rate, and Rx_Rate.    If the SNR is 10 or lower, then the wireless device is far away from the AP. If the Tx_Rate or Rx_Rate are 1, 2, or 6 then the device may be experiencing interference or is far away from the AP

 

#show ip mobile trail <wirelessMACaddress>

 

This command displays the mobility history of a given client.  This can be used to check for the frequency of roaming especially if the user has been stationary.  If the client PC has been bouncing between APs due to driver issues, this can result in dropped WLAN connections.

 

#show user ip <ipaddr>

 

Investigate the following:  Channel Frame Retry Rate – 10-20% is normal, 30 is intermediate and 40% is critical.  This means 40% of the frames put to the air have been retransmitted.  This is a symptom of heavy interference or low signal strength.    Channel Noise – If channel noise is at a value of 65 or below then this is a critical interference level. 

 

#show ap debug radio-stats ap-name <name-of-ap client connected> radio <0>

#show ap debug radio-stats ap-name <name-of-ap client connected> radio <1>

 

From the above commands we can determine the current Noise Floor and the RSSI/SNR.

 

From these steps you can determine possible causes for disconnection such as roaming problems, low signal strength, roaming outside of the WLAN coverage area and most importantly, the presence or not of major sources of interference.

bwilson

 

Did you ever find out what caused this and how to solve this issue?

 

We have not had a fix yet.  We still see the issue occasionally on 8.3.1.1.  Support looked at it for several days, and never came up with a solution.

Is the environement running a mixed of 11a/g/an/gn/ac end-devices?

 

Able to isolate the common symptoms for affected laptops?

Yes, around 700 devices.  It wasn't just laptops seeing the issue, I've had reports of droid phones doing the same thing.  On the laptop side, it was reported more on the OSX platform, than the Win 7/8... but it may be due to we have more of those.

 

I have also reports of IOS7 devices with issues.

 

 

This sounds very much like an issue that we've been seeing.  Admittedly, it's hard to tell.  We have a roaming issue with Macs that existed pre-6.3, and Client Match made this issue worse by encouraging extra roaming (so we've turned that off).  We're also seeing non-Mac disconnects however we're not sure if they were around pre-6.3 either.  We did go from 6.1.3.8 (non Airgroup) to 6.3.0.2.  I would love to know if we're seeing the same issue you are.

---

@JohnKilpatrick wrote:

This sounds very much like an issue that we've been seeing.  Admittedly, it's hard to tell.  We have a roaming issue with Macs that existed pre-6.3, and Client Match made this issue worse by encouraging extra roaming (so we've turned that off).  We're also seeing non-Mac disconnects however we're not sure if they were around pre-6.3 either.  We did go from 6.1.3.8 (non Airgroup) to 6.3.0.2.  I would love to know if we're seeing the same issue you are.

---

JohnKilpatrick,

 

I suggest your open your own TAC case and report back to us.  We frequently see similar symptoms that have different root causes.  The worst thing that can happen is if you make changes that are not relevant to your own environment.

---

@cjoseph wrote:

JohnKilpatrick,

I suggest your open your own TAC case and report back to us.  We frequently see similar symptoms that have different root causes.  The worst thing that can happen is if you make changes that are not relevant to your own environment.

---

Trust me, I have multiple tac cases open.  if I had a nickle for every "tar logs tech-support" or "tar crash" I've sent in I wouldn't need to work.

---

@JohnKilpatrick wrote:

---

@cjoseph wrote:

JohnKilpatrick,

I suggest your open your own TAC case and report back to us.  We frequently see similar symptoms that have different root causes.  The worst thing that can happen is if you make changes that are not relevant to your own environment.

---

Trust me, I have multiple tac cases open.  if I had a nickle for every "tar logs tech-support" or "tar crash" I've sent in I wouldn't need to work.

---

Surely your circumstances are not identical to the OP, are they?  If so, are there any differences?  There are quite a few people who start out with complicated problems, escalate and at least make progress.  One of them is here:  http://community.arubanetworks.com/t5/MVP-Lounge/AOS-6-2-ARM-is-not-correctly-setting-power-levels-on-APs/gpm-p/115159

I can sympathize with Client Match excessively  moving clients.  I've sat here for about 20 minutes, not moving at all, and have been disconnected many times.  It takes about 30 seconds to reconnect, and failed reconnecting twice, where I needed to turn off the radio, and turn it back on.

---

@bwilson@lps.org wrote:

I can sympathize with Client Match excessively  moving clients.  I've sat here for about 20 minutes, not moving at all, and have been disconnected many times.  It takes about 30 seconds to reconnect, and failed reconnecting twice, where I needed to turn off the radio, and turn it back on.

---

bwilson@lps.org,

 

Edit:

 

Upon seeing your jpeg, you should open a case with support to look at your parameters to ensure they are optimal for your deployment.

I've opened two so far, and they've ensured me that it's optimal.  I've sent around 10 log files, and have worked with support for several days on the issue.  The problem is that it's not easily reproduceable.  It will work fine sometimes, and others will look like the picture.  Sometimes Clientmatch will go smoothly, and other times it will not.  It did seem somewhat more reliable after 6.3.1.1, but I don't have any metrics to support this.

 

Thanks for the discussion, and feedback!

Bwilson,

Thank you for the feedback.  Aruba relies on feedback from users to ensure we meet all of their needs.  This community allows others to also benefit from feedback from other users.  We really appreciate it.

---

@bwilson@lps.org wrote:

I've opened two so far, and they've ensured me that it's optimal.  I've sent around 10 log files, and have worked with support for several days on the issue.  The problem is that it's not easily reproduceable.  It will work fine sometimes, and others will look like the picture.  Sometimes Clientmatch will go smoothly, and other times it will not.  It did seem somewhat more reliable after 6.3.1.1, but I don't have any metrics to support this.

 

Thanks for the discussion, and feedback!

---

This is purely anecdotal but on 6.3.1.0 we had similar clientmatch issues mainly with Windows 8 clients where they'd flap between two access points and a reboot of the AP would fix them. TAC asked us to disable client match to see if it resolved the problem but it wasn't really something we wanted to do campus wide and since it was sporadic we couldn't recreate it but after upgrading to 6.3.1.1 we've had no new reported cases.

---

@bwilson@lps.org wrote:

Yes, around 700 devices.  It wasn't just laptops seeing the issue, I've had reports of droid phones doing the same thing.  On the laptop side, it was reported more on the OSX platform, than the Win 7/8... but it may be due to we have more of those.

 

I have also reports of IOS7 devices with issues.

 

 

---

bwilson,

 

Please continue to work with TAC and let us know if you make any progress.