Wireless Access

Before I open a support case, I trolled the forums for some answers. Our wireless network has exploded this year with access. I'm finding a lot of disconnects, or stationary clients switching aps. Digging into the ap stats, I found enormous CRC errors and "Config sent" not matching "Config acked" on one of the controllers (1master, 2 local). I'm hoping someone can point me to what stats I should be looking at so I can understand what's going before I talk to support.

--myron

 A couple of questions :

- Have you upgraded recently to a new OS ?

- You should take a look at the health of your RF environment : 

If you have AOS 6.x you can have an overall view of this information and determine if there's a lot of : Noise , Too much Broadcast or Multicast traffic , Channels changes , Power changes 

show ap arm history ap-name <ap name> -- Power / Channel changes over time

show ap arm rf-summary ap-name <ap name> -- Noise/ L2Retries 

show ap monitor ap-list ap-name <ap-name> Neighbor APs (Rogue or Valid)

- You should also check if the APs are bootstrapping :

show ap debug counters 

- Make sure that there's no wired layer 1  issues ( show port stats ) from the controller perspective 

- You will also see a lot of disconnects because most device in your wireless network are smartphones and those will be roaming a lot 

Thanks for getting back to me.

Alcatel came in in March and helped us upgrade, and set up a 1 master- 2 local setup. I just started working the wireless because of the anticpated ramp up. I'm sure we're running 6.

Here's the picture of one of the controllors from which the stats of the ap are below.

I'll give you the stats for an ap where we're having problems. When I said "disconnents," I didn't mean phones, or clients passing by. I meant students and teachers working in classrooms suddently losing connections.

show ap arm history ap-name HoS1-03

Interface :wifi0
ARM History
-----------
Time of Change Old Channel New Channel Old Power New Power Reason
-------------- ----------- ----------- --------- --------- ------
2013-10-30 11:32:07 6 6 21 18 P-
2013-10-30 11:26:04 6 6 18 21 P+
2013-10-30 10:56:24 6 6 24 18 P-
2013-10-30 10:51:14 6 6 21 24 P+
2013-10-30 10:45:37 6 6 18 21 P+
2013-10-30 10:37:47 6 6 24 18 P-
2013-10-30 10:30:30 6 6 21 24 P+
2013-10-30 10:23:33 6 6 18 21 P+
2013-10-30 10:12:25 6 6 24 18 P-
2013-10-30 10:02:22 6 6 21 24 P+
2013-10-30 09:58:09 6 6 18 21 P+
2013-10-30 09:50:50 6 6 24 18 P-
2013-10-30 09:39:03 6 6 21 24 P+
2013-10-30 09:34:38 6 6 18 21 P+
2013-10-30 04:01:21 6 6 24 18 P-
2013-10-30 03:49:37 6 6 21 24 P+
2013-10-30 03:42:34 6 6 18 21 P+
2013-10-30 03:36:02 6 6 24 18 P-
2013-10-30 03:24:35 6 6 21 24 P+
2013-10-30 03:17:09 6 6 18 21 P+
2013-10-30 03:09:54 6 6 24 18 P-
2013-10-30 03:01:33 6 6 21 24 P+
2013-10-30 02:57:20 6 6 18 21 P+
2013-10-30 02:52:36 6 6 24 18 P-
2013-10-30 02:41:55 6 6 21 24 P+
2013-10-30 02:35:16 6 6 18 21 P+
2013-10-29 22:29:06 6 6 21 18 P-
2013-10-29 22:22:49 6 6 18 21 P+
2013-10-29 14:29:29 6 6 24 18 P-
2013-10-29 14:23:49 6 6 21 24 P+
2013-10-29 14:19:35 6 6 18 21 P+
2013-10-29 14:05:09 6 6 24 18 P-

show ap arm rf-summary ap-name HoS1-03

Channel Summary
---------------
channel retry phy-err mac-err noise cov-idx intf_idx
------- ----- ------- ------- ----- ------- --------
1 0 0 0 99 4/0 24/8//2/1
6 0 0 17 94 6/0 3/15//3/1
11 0 0 0 97 9/2 49/1//12/0

Interface Name :wifi0
Current ARM Assignment :6/18
Covered channels a/g :0/3
Free channels a/g :0/0
ARM Edge State :enable
Last check channel/pwr :13m:37s/6m:59s
Last change channel/pwr :7d:18h:10m:56s/3h:14m:8s
Next Check channel/pwr :0s/28s

show ap monitor ap-list ap-name HoS1-03

Monitored AP Table
------------------
bssid essid chan ap-type phy-type dos dt/mt ut/it encr nstas avg-rssi curr-rssi wmacs ibss
----- ----- ---- ------- -------- --- ----- ----- ---- ----- -------- --------- ----- ----
00:0b:86:ac:17:40 moco 6 valid 80211b/g disable 3017806/3017806 0/0 wpa-8021x-tkip 1 30 30 245 no
00:0b:86:ac:17:42 MC Guest Wireless 6 valid 80211b/g disable 3017806/3017806 0/0 open 0 30 30 56 no
00:1a:1e:e6:ee:60 moco 6 valid 80211b/g disable 2858062/2837187 0/0 wpa-8021x-tkip 1 2 2 256 no
00:1a:1e:e6:ee:61 MOCO-2 6 valid 80211b/g disable 1619165/1619165 0/0 wpa2-8021x-aes 0 2 4 0 no
00:0b:86:ac:17:41 MOCO-2 6 valid 80211b/g disable 1619014/1619014 0/0 wpa2-8021x-aes 0 30 30 0 no
00:0b:86:ab:fc:40 moco 11 valid 80211b/g disable 254472/4357 232/2 wpa-8021x-tkip 0 0 17 5 no
00:0b:86:ab:fc:42 MC Guest Wireless 11 valid 80211b/g disable 254472/4357 232/2 open 0 0 19 0 no
00:0b:86:ab:fc:41 11 valid 80211b/g disable 254472/4357 49/2 wpa2-8021x-aes 0 0 18 0 no
c4:3d:c7:9a:61:60 D&B 2 interfering 80211b/g-HT-40 disable 95565/1585 88/0 wpa2-psk-aes 0 0 9 0 no
00:0b:86:a4:4f:11 MC Guest Wireless 11 valid 80211b/g disable 94700/1588 49/2 open 0 0 33 2 no
00:0b:86:a4:4f:10 moco 11 valid 80211b/g disable 94661/1588 232/2 wpa-8021x-tkip 0 0 32 2 no
00:0b:86:a4:4f:12 11 valid 80211b/g disable 94661/1588 49/2 wpa2-8021x-aes 0 0 32 0 no
00:1a:1e:e6:e2:e1 6 valid 80211b/g disable 39003/39003 10/9 wpa2-8021x-aes 0 1 1 0 no
00:1a:1e:f0:76:20 moco 1 valid 80211b/g disable 4684/55 452/3 wpa-8021x-tkip 0 0 3 2 no
00:1a:1e:f0:76:21 1 valid 80211b/g disable 3506/41 218/1 wpa2-8021x-aes 0 0 4 0 no
00:1a:1e:f0:7a:21 moco 1 valid 80211b/g disable 2411/27 218/1 wpa-8021x-tkip 0 0 20 0 no
00:1a:1e:f0:7a:20 1 valid 80211b/g disable 2401/26 218/1 wpa2-8021x-aes 0 0 20 0 no
00:1a:1e:e6:f4:41 1 valid 80211b/g disable 1551/17 350/2 wpa2-8021x-aes 0 0 1 0 no

Not so much bootstrapping and reboots for this controllor; however, I'll print out below where I see it a lot on the master. 

show ap debug counters

AP Counters
-----------
Name Group IP Address Configs Sent Configs Acked AP Boots Sent AP Boots Acked Bootstraps (Total) Reboots
---- ----- ---------- ------------ ------------- ------------- -------------- ------------------ -------

HoS1-01 MoravianPool 10.164.1.52 514 514 0 0 1 (1 ) 0
HoS1-02 MoravianPool 10.164.1.34 58 58 0 0 1 (1 ) 0
HoS1-03 MoravianPool 10.164.1.35 1789 1789 0 0 1 (1 ) 0
HoS1-04 MoravianPool 10.164.1.36 524 524 0 0 1 (1 ) 0
HoS1-05 MoravianPool 10.164.1.37 946 946 0 0 1 (1 ) 0
HoS2-01 MoravianPool 10.164.1.240 43 43 0 0 1 (1 ) 0
HoS2-02 MoravianPool 10.164.1.38 1744 1744 0 0 1 (1 ) 0
HoS2-03 MoravianPool 10.164.1.242 4 4 0 0 1 (1 ) 0

This is the master.

HILL2ANHall HILL 10.190.130.42 38 35 0 0 6 (7 ) 2
HILL2B-Hall HILL 10.190.130.35 5430 4182 0 0 10 (13 ) 4
HILL2B-Living HILL 10.190.130.37 6293 5565 0 0 4 (5 ) 1
HILL2C-Hall HILL 10.190.130.46 4833 2568 0 0 6 (6 ) 1
HILL2C-Living HILL 10.190.130.33 1532 840 0 0 6 (6 ) 1
HILL2D-Hall HILL 10.190.130.43 260 248 0 0 6 (7 ) 2
HILL2D-Living HILL 10.190.130.40 6534 3921 0 0 1 (2 ) 1
HILL2E-Hall HILL 10.190.130.49 5153 2760 0 0 4 (8 ) 5
HILL2E-LivingAM HILL 10.190.130.34 2 2 0 0 1 (1 ) 1
HILL2F-LivingAM HILL 10.190.130.44 3 3 0 0 1 (1 ) 1
HILL2FHall HILL 10.190.130.32 3342 2909 0 0 7 (14 ) 8
HILL2G-Hall HILL 10.190.130.41 2129 1369 0 0 9 (9 ) 1

I do see some issues on the port stats for the master.

show port stats

Port Statistics
---------------
Port PacketsIn PacketsOut BytesIn BytesOut                                                         InputErrorBytes OutputErrorBytes CRCErrors
---- --------- ---------- ------- -------- ---------------                                                             ---------------- ---------
GE 1/0 15083689456 19913284130 4766232584689 22022471413105     154425 0 0
GE 1/1 1432567261 9941983 98850364498 2039677119                               10044581 0 0
GE 1/2 16404840706 11586928943 19495594176326 3195362377268     10038327 0 0
GE 1/3 1022713714 540063949 860347182050 116701112630                   4832 0 0

I don't see any imput error bytes on the 2 controllors.

--myron

Based on the output you shared it seems overall things don't look that bad but should address some those radios that are showing a low noise value and the ones with a high channel utilization .

Are you guys experiencing performance issues ?

I wish I felt good about your comment, but my users would strongly disagree with you. The two issues you brought up: low noise, and high channel utilization is in student dorms with few ap's and too many students. We have plans to address that.

The performance issue is that they keep getting disconnected. So, for instance, a music teacher would be accessing some cloud iTune piece and then get disconnected and have to reestablish connection. We just (sort of) got the ov3600, and I see his history as being connected and then jumping to another ap.

Getting back to the data I sent you (I'm  learning to put it into a better format). You don't see the mismatch configs, and the bootstraps, and the reboots as a problem. From what I'm learning on the forums, these are not good numbers.

--myron

My comment was related to your RF environment based on the screenshots you shared.

But looking at the show ap debug counters output you provided , you may have a couple of issues :

- Network issues (L3/L2) , there might some packet loss between your controller and the APs and the PAPI/GRE are getting dropped along the path

- Layer 1 issue either on the edge switch or the connection from the controller and the uplink device.

What AOS version are you using ?

I suggested you open a ticket with TAC

I'm using version 6.1.3.7.

I will open a ticket when I get some feel for what's going on.

How did you come to the conclusion that there is a PAPI/GRE problem, and what commands can I use to show that?

--myron

The APs and the controller communicate using the following protocols:

PAPI: The controller sends its config to the AP using this protocol

GRE: It is use to carry the wireless traffic from the controller to AP , it uses a GRE tunnel for every SSID

If these packets don't make it to the AP then you will experience wireless issues (disconnects) because the SSIDs will not stay up consistently .

You should definitely consider opening a TAC case