Controllerless Networks

Hi,
I am running 12 units IAP-205 with software 6.4.2.6-4.1.1.11.
In random period of time, some groups of users connected to one AP suffer from packet loss.
From syslogs I see that every day radios (5Ghz or 2.4Ghz) on different APs rebooting with some error.
All APs not overloaded and in the moment of failure can have 2 clients.
This is typical message I catch from AP after problem:
show log driver

[8151765.330000] failed to get phy_rxiqest
[8151876.506000] stats > 100: ccastats_us: 74016, acc->statss_ms: 74
[8152001.944000] stats > 100: ccastats_us: 74268, acc->statss_ms: 74
[8152064.714000] stats > 100: ccastats_us: 74187, acc->statss_ms: 74
[8152142.786000] wl0 wlc_dotxstatus: consumes 0 frames!
[8152142.786000] wl0 raw txstatus 0063 3A09 | 077A 0000 | 00000000 00000000 || 00007195 00000000 | 00000000
[8152142.786000] pktpend: 0 1 0 0 0 ap: 1
[8152142.786000] frameid 0x3a09 queue 1
[8152142.786000] wl0: wlc_bmac_txstatus:: bad txstatus 3A090063 2000077A 00000000 00000000 || 00007195 00000000 00000000 C7B080C8
[8152142.786000] error 1 caught in wlc_bmac_txstatus
[8152142.786000] wl0: wlc_dpc: txstatus fatal error
[8152142.786000] wl0: wlc_reset: !!!BIG-hammer!!!
[8152142.786000] wl0: 1 tx fifo 1 underflows!
[8152142.806000] wl0: wlc_txbf_init_sta: scb aleady has user index 0
[8152147.590000] wl0: ampdu_resp_timeout: cleaning up resp tid 0 waiting for seq 0x6fb for 200 ms
[8152356.796000] wl0: ampdu_resp_timeout: cleaning up resp tid 0 waiting for seq 0x9a for 200 ms

show log kernel

54616.806000] wl0: PHYTX error
[8154616.806000] wl0: PHYTX error
[8154616.806000] wl0: PHYTX error
[8154616.806000] wl0: PHYTX error
....
[8154616.856000] wl0: PSM microcode watchdog fired at 8138192 (seconds). Resetting.
[8154616.856000] ucode revision 876.1 
[8154616.856000] wl0: PSM microcode watchdog fired at 8138192 (seconds)
[8154616.856000] wl0: psmdebug 0x00ff861c, phydebug 0x00000048, psm_brc 0x00b0 psm_brc_1 0x00a2
[8154616.856000] wepctl 0x0015 wepstat 0x0003 wep_hdrloc 0x0026 wep_psdulen 1454
[8154616.856000] RXE_RXCNT: 0x0014 DAGG: ctl: 0x0000, bleft: 0x095f
[8154616.856000] offs: 0x0034, stat: 0x13db, len: 0x0000
[8154616.856000] phydebug
[8154616.856000] 0x0048
...
[8154616.858000] PC :
[8154616.858000] 0xff861c
[8154616.858000] 0xff861c
[8154616.858000] wlc_dump_aggfifo:
[8154616.858000] framerdy 0x2 bmccmd 1 framecnt 0x2 
[8154616.858000] AQM agg params 0xfff maxlen hi/lo 0x4 0x2040 minlen 0xc8 adjlen 0x8
[8154616.858000] AQM agg results 0x8001 len hi/lo: 0x0 0x5b2 BAbitmap(0-3) 0 0 0 0
[8154616.858000] wlc_dpc:605 wl0 ASSERT 'PSM Watchdog' macintstatus:0x2800
[8154616.858000] wl0: wlc_dpc: macintstatus & MI_GP0 non-zero, HAMMERING
[8154616.858000] wl0: wlc_reset: !!!BIG-hammer!!!
[8154616.878000] wl0: wlc_txbf_init_sta: scb aleady has user index 0
[8154773.658000] wl0: ampdu_resp_timeout: cleaning up resp tid 1 waiting for seq 0x65c for 200 ms
[8154996.036000] stats > 100: ccastats_us: 74133, acc->statss_ms: 74
[8155028.930000] wl0: ampdu_resp_timeout: cleaning up resp tid 0 waiting for seq 0xd8a for 200 ms

Peak CPU Util in the last one hour

----------------------------------
Timestamp CPU Util(%) Memory Util(%)
--------- ----------- --------------
2016-05-08 14:36:21 37 67

I guess it is some driver problem but I`m not sure. If you have some thoughts, please share.

open a tac case, it has already been reported and under investigation.

Is this a known issue? Please provide additional information such as bug IDs or existing case numbers for this issue.

It is already resolved issue. Try latest stable firmware

Thanks for the suggestion, but we are running 6.5.0.3, which I believe is the latest firmware. 

We see this interface reset error on ~70 of our AP-225 units around campus (out of ~1,700).

The "wl1: fatal error, reinitializing > wl1: wlc_reset: !!!BIG-hammer!!! " error is generated by these APs every 2 seconds. It only affects the 2.4Ghz radio and prevents any clients from associating to that radio. The 5Ghz radio continues to work fine and serve clients. Rebooting the AP eliminates the issue and the 2.4Ghz radio begins serving clients agian. 

My Co-worker created a detailed post about this particular issue we are having:

https://community.arubanetworks.com/t5/Wireless-Access/Problem-with-radio-resets-quot-BIG-Hammer-quot/m-p/285430#M66902

 

Radio resets typically are due to errors due to interference. What is the arm min and max transmit power for your WLAN?

Our 2.4Ghz ARM profile is set to range from 15 to 9.

Our 5Ghz ARM profile is set to range from 18 to 12.

We see this issue only on our 2.4Ghz radios. The radio resets every 2 seconds, preventing any client associations. This behavior continues throughout the night when I'd imagine the RF environment should be less noisy. 

Daytime noise floor is typically in the mid to upper -90 range for the APs experiencing this issue.

If we reboot the AP, the problem goes away for at least a few days, but may show up on other APs in the building. 

 

Thanks for your thoughts, we also have an open case with TAC. They claim to have seen this before and development wants to collect some log data from us. 

I would stick with TAC on this one. You are pulling together threads that may or may not be relevant to your situation, and that might delay relief.

Noise is mainly an indicator for non-wifi interference. High utilization can be caused by wifi, non wifi interference or a combination. That could cause radio resets. If you have a dense deployment, I would lower the max TX for 2.4ghz to 12 and see if it continues. Since TX power is logarithmic, every 3 that you decrement is an effective halving of transmit power which can make a great difference.

If you type "show ap arm state ap-name <name of AP> how many APs can you see on the 2.4ghz band?

Thanks, we may try moving the 2.4Ghz max down to 12 and see if that changes the frequency of this problem. 

 

Trying your suggested command, some of the APs return no output besides the date. and time.

For the APs that do return ARM state info, the number of valid neighbors ranges from 10-20 with relevant neighbors in the 2-10 range. 

 

Try the max at 12.

I am talking about General Availability firmware.

I use 6.4.4.8-4.2.4.4_57496.

I have this on our APs. I did a debug:

"show ap tech-support ap-name AP225-G-frecuency-Issues Archivo-TechSupp-AP225-G-frecuency-Issues.txt" 

and in the file i found this (very cutted lines for short):

 

Toooooo many resets on radio 1:

show ap debug radio-stats ap-name "APAC-225-Issues.Level-3.Building-9F.HMO" radio 1 advanced

RADIO Stats
-----------
Parameter Value
--------- -----
------------------- General
Total Radio Resets 246575
Resets Beacon Fail 0
BB Check Positives 0
Resets BeacQ Stuck 0
Resets Fatal Intr 0
Resets RX Overrun 0
Resets RF Gain 0
Resets MTU Change 0
Resets TX Timeouts 0
Resets RX Timeouts 0
POE-Related Resets 0
External Reset 0
PCI Fatal Intr Reset 0
Chainmask Reset 0
TX stat Reset 0
TX Power Changes 5
Channel Changes 2
Failed Channel Changes 0
Radio Band Changes 0

 

======================= Radio 0 seems like no problem (resetes: 8 vs 246,575) ====

show ap debug radio-stats ap-name "APAC-225-Issues.Level-3.Building-9F.HMO" radio 0 advanced
RADIO Stats
-----------
Parameter Value
--------- -----
------------------- General
Total Radio Resets 8
Resets Beacon Fail 0
BB Check Positives 0
Resets BeacQ Stuck 0
Resets Fatal Intr 0
Resets RX Overrun 0
Resets RF Gain 0
Resets MTU Change 0
Resets TX Timeouts 0
Resets RX Timeouts 0
POE-Related Resets 0
External Reset 0
PCI Fatal Intr Reset 0
Chainmask Reset 0
TX stat Reset 0
TX Power Changes 3
Channel Changes 2
Failed Channel Changes 0
Radio Band Changes 0

 

======================  BIG HAMMER fatal error =====

show ap debug driver-log ap-name "APAC-225-Issues.Level-3.Building-9F.HMO"

reset: !!!BIG-hammer!!!
[564399.224855] wl1 bmac set short slot 1, wl status 1
[564399.225533] wl1 bmac set short slot 1, wl status 1
[564399.256318] wl1: ap watchdog handler error
[564401.170057] wl1: bcn inactivity detected HAMMERING
[564401.170069] wl1: txbcnfrm 0 prev txbcnfrm 0 txbcn inactivity 2 timeout 2
[564401.170081]
[564401.170083] chan dur ibss obss interf time
[564401.170093] wl1: fatal error, reinitializing
[564401.170104] wl1: wlc_reset: !!!BIG-hammer!!!
[564401.225479] wl1 bmac set short slot 1, wl status 1
[564401.226100] wl1 bmac set short slot 1, wl status 1
[564401.256888] wl1: ap watchdog handler error
[564403.168993] wl1: bcn inactivity detected HAMMERING
[564403.169005] wl1: txbcnfrm 0 prev txbcnfrm 0 txbcn inactivity 2 timeout 2
[564403.169017]
[564403.169020] chan dur ibss obss interf time
[564403.169030] wl1: fatal error, reinitializing
[564403.169040] wl1: wlc_reset: !!!BIG-hammer!!!
[564403.224947] wl1 bmac set short slot 1, wl status 1
[564403.225628] wl1 bmac set short slot 1, wl status 1
[564403.256532] wl1: ap watchdog handler error
[564405.169501] wl1: bcn inactivity detected HAMMERING
[564405.169513] wl1: txbcnfrm 0 prev txbcnfrm 0 txbcn inactivity 2 timeout 2
[564405.169525]
[564405.169527] chan dur ibss obss interf time
[564405.169537] wl1: fatal error, reinitializing
[564405.169548] wl1: wlc_reset: !!!BIG-hammer!!!
[564405.225434] wl1 bmac set short slot 1, wl status 1
[564405.226090] wl1 bmac set short slot 1, wl status 1
[564405.256978] wl1: ap watchdog handler error
[564406.223241] wl0: wlc_ampdu_tx_send_delba: tid 0 initiator 1 reason 39 dest 64:9a:be:38:33:62
[564407.171197] wl1: bcn inactivity detected HAMMERING
[564407.171208] wl1: txbcnfrm 0 prev txbcnfrm 0 txbcn inactivity 2 timeout 2
[564407.171228]
[564407.171230] chan dur ibss obss interf time
[564407.171241] wl1: fatal error, reinitializing
[564407.171252] wl1: wlc_reset: !!!BIG-hammer!!!
[564407.225774] wl0: wlc_ampdu_tx_send_delba: tid 6 initiator 1 reason 39 dest 24:f0:94:41:d6:d4
[564407.226696] wl1 bmac set short slot 1, wl status 1
[564407.227360] wl1 bmac set short slot 1, wl status 1
[564407.258005] wl1: ap watchdog handler error
[564409.169508] wl1: bcn inactivity detected HAMMERING
[564409.169519] wl1: txbcnfrm 0 prev txbcnfrm 0 txbcn inactivity 2 timeout 2
[564409.169531]
[564409.169533] chan dur ibss obss interf time
[564409.169543] wl1: fatal error, reinitializing
[564409.169554] wl1: wlc_reset: !!!BIG-hammer!!!
[564409.224961] wl1 bmac set short slot 1, wl status 1
[564409.225616] wl1 bmac set short slot 1, wl status 1
[564409.256352] wl1: ap watchdog handler error
[564410.223220] wl0: wlc_ampdu_tx_send_delba: tid 6 initiator 1 reason 39 dest 68:ae:20:0c:c2:6a
[564411.168992] wl1: bcn inactivity detected HAMMERING
[564411.169003] wl1: txbcnfrm 0 prev txbcnfrm 0 txbcn inactivity 2 timeout 2
[564411.169015]
[564411.169018] chan dur ibss obss interf time
[564411.169028] wl1: fatal error, reinitializing
[564411.169038] wl1: wlc_reset: !!!BIG-hammer!!!
[564411.224346] wl1 bmac set short slot 1, wl status 1
[564411.224949] wl1 bmac set short slot 1, wl status 1
[564411.255709] wl1: ap watchdog handler error
[564413.169486] wl1: bcn inactivity detected HAMMERING
[564413.169497] wl1: txbcnfrm 0 prev txbcnfrm 0 txbcn inactivity 2 timeout 2
[564413.169509]
[564413.169511] chan dur ibss obss interf time
[564413.169522] wl1: fatal error, reinitializing
[564413.169532] wl1: wlc_reset: !!!BIG-hammer!!!
[564413.224835] wl1 bmac set short slot 1, wl status 1
[564413.225468] wl1 bmac set short slot 1, wl status 1
[564413.256213] wl1: ap watchdog handler error
[564415.169489] wl1: bcn inactivity detected HAMMERING
[564415.169501] wl1: txbcnfrm 0 prev txbcnfrm 0 txbcn inactivity 2 timeout 2
[564415.169513]
[564415.169515] chan dur ibss obss interf time
[564415.169525] wl1: fatal error, reinitializing
[564415.169536] wl1: wlc_reset: !!!BIG-hammer!!!
[564415.224832] wl1 bmac set short slot 1, wl status 1
[564415.225492] wl1 bmac set short slot 1, wl status 1
[564415.256246] wl1: ap watchdog handler error
[564417.169498] wl1: bcn inactivity detected HAMMERING
[564417.169509] wl1: txbcnfrm 0 prev txbcnfrm 0 txbcn inactivity 2 timeout 2
[564417.169521]
[564417.169523] chan dur ibss obss interf time
[564417.169534] wl1: fatal error, reinitializing
[564417.169544] wl1: wlc_reset: !!!BIG-hammer!!!
[564417.224883] wl1 bmac set short slot 1, wl status 1
[564417.225536] wl1 bmac set short slot 1, wl status 1
[564417.256245] wl1: ap watchdog handler error
[564419.169505] wl1: bcn inactivity detected HAMMERING
[564419.169516] wl1: txbcnfrm 0 prev txbcnfrm 0 txbcn inactivity 2 timeout 2
[564419.169528]
[564419.169531] chan dur ibss obss interf time
[564419.169541] wl1: fatal error, reinitializing
[564419.169552] wl1: wlc_reset: !!!BIG-hammer!!!
[564419.225107] wl1 bmac set short slot 1, wl status 1
[564419.225717] wl1 bmac set short slot 1, wl status 1
[564419.229858] wl0: wlc_ampdu_tx_send_delba: tid 6 initiator 1 reason 39 dest 64:9a:be:38:33:62
[564419.256455] wl1: ap watchdog handler error
[564421.169551] wl1: bcn inactivity detected HAMMERING
[564421.169562] wl1: txbcnfrm 0 prev txbcnfrm 0 txbcn inactivity 2 timeout 2
[564421.169574]
[564421.169576] chan dur ibss obss interf time
[564421.169587] wl1: fatal error, reinitializing
[564421.169597] wl1: wlc_reset: !!!BIG-hammer!!!
[564421.225005] wl1 bmac set short slot 1, wl status 1
[564421.225646] wl1 bmac set short slot 1, wl status 1
[564421.256364] wl1: ap watchdog handler error

......

 

Any idea?

 

 

 

 Same pattern here... i did a debug and seems like something in the driver:

 

“show ap tech-support ap-name AP225-G-frecuency-Issues Archivo-TechSupp-AP225-G-frecuency-Issues.txt”

 

I extract few lines: 

show ap debug datapath ap-name "AP225-G-frecuency-Issues "

 

 

Datapath Parameters Table

-------------------------

essid     encr-alg        client-vlan-id  tunnel-id  gre-type  deny-bcast  num-clients

-----     --------        --------------  ---------  --------  ----------  -----------

Uni    WPA2 8021X AES  1128            0x10951    0x8210    disable     6

Mae  WPA2 8021X AES  1128            0x10608    0x8200    disable     1

Uni    WPA2 8021X AES  1128            0x105b1    0x8310    disable     0

Mae  WPA2 8021X AES  1128            0x10952    0x8300    disable     0

 

 

show ap debug client-table ap-name "AP225-G-frecuency-Issues "

 

 

Client Table

------------

MAC                ESSID     BSSID              Assoc_State    HT_State  AID  PS_State    UAPSD            Tx_Pkts  Rx_Pkts  PS_Qlen  Tx_Retries  Tx_Rate  Rx_Rate  Last_ACK_SNR  Last_Rx_SNR  TX_Chains  Tx_Timestamp              Rx_Timestamp              MFP Status (C,R)  Idle time  Client health (C/R)

---                -----     -----              -----------    --------  ---  --------    -----            -------  -------  -------  ----------  -------  -------  ------------  -----------  ---------  ------------              ------------              ----------------  ---------  -------------------

68:c4:4d:4c:43:cb  Mae  18:64:72:14:85:30  Associated     WSsb      0x2  Awake       (0,0,0,0,N/A,0)  222      69       0        154         45       19       11            9            3[0x7]     Thu Jan 26 12:05:34 2017  Thu Jan 26 12:05:34 2017  (0,0)             0          100/65

f4:09:d8:4f:6d:7b  Mae  18:64:72:14:85:30  Authenticated  AWvSsEe        Awake       (0,0,0,0,N/A,0)  289      45       0        9           866      780      50            48           3[0x7]     Thu Jan 26 12:05:28 2017  Thu Jan 26 12:05:28 2017  (0,0)             5          100/65

f4:09:d8:4f:6d:7b  Uni    18:64:72:14:85:31  Associated     AWvSsEe   0x6  Awake       (0,0,0,0,N/A,0)  59       20       0        21          866      780      44            47           3[0x7]     Thu Jan 26 12:05:34 2017  Thu Jan 26 12:05:34 2017  (0,0)             0          100/65

c0:bd:d1:f6:01:c6  Uni    18:64:72:14:85:31  Associated     AWvSsEe   0x4  Awake       (0,0,0,0,N/A,0)  1270     1003     0        124         195      87       10            15           3[0x7]     Thu Jan 26 12:05:34 2017  Thu Jan 26 12:05:34 2017  (0,0)             0          81/65

24:f0:94:41:d6:d4  Uni    18:64:72:14:85:31  Associated     AWvSsEe   0x5  Power-save  (0,0,0,0,N/A,0)  11       10       3        15          433      6        7             7            3[0x7]     Thu Jan 26 12:01:18 2017  Thu Jan 26 12:01:21 2017  (0,0)             253        100/65

68:ae:20:0c:c2:6a  Uni    18:64:72:14:85:31  Associated     WQSs      0x3  Awake       (0,0,0,0,N/A,0)  16627    15087    0        2981        14       13       9             7            3[0x7]     Thu Jan 26 12:05:31 2017  Thu Jan 26 12:04:44 2017  (0,0)             50         33/65

64:9a:be:38:33:62  Uni    18:64:72:14:85:31  Associated     AWvSsEQ   0x1  Awake       (0,0,0,0,N/A,0)  11362    10741    0        1388        7        6        11            6            3[0x7]     Thu Jan 26 12:05:30 2017  Thu Jan 26 12:02:09 2017  (0,0)             204        28/65

34:e6:ad:34:e0:18  Uni    18:64:72:14:85:31  Associated     WvSsEbQM  0x2  Awake       (0,0,0,0,N/A,0)  145521   96292    0        11670       433      351      34            27           3[0x7]     Thu Jan 26 12:05:34 2017  Thu Jan 26 12:05:34 2017  (0,0)             0          90/65

1c:7b:21:b4:71:3f  Mae  18:64:72:14:85:20  Authenticated  sb             Awake       (0,0,0,0,N/A,0)  33       43       0        35          11       19       34            31           1[0x1]     Fri Jan 20 19:05:55 2017  Thu Jan 26 10:58:56 2017  (0,0)             3999       90/65

9c:20:7b:0d:86:6a  Mae  18:64:72:14:85:20  Authenticated  Qb             Awake       (0,0,0,0,N/A,0)  98       234      0        50          39       26       17            25           1[0x1]     Fri Jan 20 19:04:18 2017  Thu Jan 26 12:05:00 2017  (0,0)             34         90/65

 

Num of associated clients: 7

 

UAPSD:(VO,VI,BK,BE,Max SP,Q Len)

HT  Flags: A - LDPC Coding; W - 40MHz; S - Short GI 40; s - Short GI 20

          D - Delayed BA; G - Greenfield; R - Dynamic SM PS

          Q - Static SM PS; N - A-MPDU disabled; B - TX STBC

          b - RX STBC; M - Max A-MSDU; I - HT40 Intolerant; t turbo-rates (256-QAM)

VHT Flags: C - 160MHz/80+80MHz; c - 80MHz;  V - Short GI 160; v - Short GI 80

                    E - Beamformee; e - Beamformer

HT_State shows client's original capabilities (not operational capabilities)

 

show ap debug driver-log ap-name "AP225-G-frecuency-Issues "

 

reset: !!!BIG-hammer!!!

[564399.224855] wl1 bmac set short slot 1, wl status 1

[564399.225533] wl1 bmac set short slot 1, wl status 1

[564399.256318] wl1: ap watchdog handler error

[564401.170057] wl1: bcn inactivity detected HAMMERING

[564401.170069] wl1: txbcnfrm 0 prev txbcnfrm 0 txbcn inactivity 2 timeout 2

[564401.170081]

[564401.170083] chan dur      ibss           obss           interf       time

[564401.170093] wl1: fatal error, reinitializing

[564401.170104] wl1: wlc_reset: !!!BIG-hammer!!!

[564401.225479] wl1 bmac set short slot 1, wl status 1

[564401.226100] wl1 bmac set short slot 1, wl status 1

[564401.256888] wl1: ap watchdog handler error

[564403.168993] wl1: bcn inactivity detected HAMMERING

[564403.169005] wl1: txbcnfrm 0 prev txbcnfrm 0 txbcn inactivity 2 timeout 2

[564403.169017]

 

.....

[564455.169713] chan dur      ibss           obss           interf       time

[564455.169723] wl1: fatal error, reinitializing

[564455.169733] wl1: wlc_reset: !!!BIG-hammer!!!

[564455.224983] wl1 bmac set short slot 1, wl status 1

[564455.225644] wl1 bmac set short slot 1, wl status 1

[564455.256365] wl1: ap watchdog handler error

[564457.169511] wl1: bcn inactivity detected HAMMERING

[564457.169522] wl1: txbcnfrm 0 prev txbcnfrm 0 txbcn inactivity 2 timeout 2

[564457.169535]

[564457.169537] chan dur      ibss           obss           interf       time

[564457.169547] wl1: fatal error, reinitializing

[564457.169558] wl1: wlc_reset: !!!BIG-hammer!!!

[564457.194509] stats > 100: ccastats_us: 4077, acc->statss_ms: 2

[564457.225173] wl1 bmac set short slot 1, wl status 1

[564457.225792] wl1 bmac set short slot 1, wl status 1

[564457.229810] wl0: wlc_ampdu_tx_send_delba: tid 6 initiator 1 reason 39 dest 24:f0:94:41:d6:d4

[564457.256587] wl1: ap watchdog handler error

 

Driver Log

--------------------------------------------------------

show ap debug radio-stats ap-name "AP225-G-frecuency-Issues" radio 0 advanced

 

RADIO Stats

-----------

Parameter                           Value

---------                           -----

-------------------                 General

Total Radio Resets                  8

Resets Beacon Fail                  0

BB Check Positives                  0

Resets BeacQ Stuck                  0

Resets Fatal Intr                   0

Resets RX Overrun                   0

Resets RF Gain                      0

Resets MTU Change                   0

Resets TX Timeouts                  0

Resets RX Timeouts                  0

POE-Related Resets                  0

External Reset                      0

PCI Fatal Intr Reset                0

Chainmask Reset                     0

TX stat Reset                       0

TX Power Changes                    3

Channel Changes                     2

Failed Channel Changes              0

Radio Band Changes                  0

Current Noise Floor                 92

Dummy NF pkts on home channel       0

Dummy NF pkts on scan channel       0

Avail TX Buffers                    2048

11g Protection                      0

Last TX Antenna                     0

Last RX Antenna                     0

Scan Requests                       54759

Scan Rejects                        15427

Load aware Scan Rejects             60

PS aware Scan Rejects               0

EAP Scan Rejects                    778

Voice aware Scan Rejects            14325

Video aware Scan Rejects            264

UAPSD Scan Rejects                  0

Post radar related scan Rejects     0

CABQ traffic Scan Rejects           0

Radio Reset Scan Rejects            0

Queue Drain Scan Rejects            0

Scan Success                        39198

EIRP                                18

MAX EIRP                            22

Dummy002                            19

Dummy007                            8

Dummy010                            1136

Dummy011                            82710

Dummy012                            0

Dummy013                            0

Dummy014                            0

Dummy015                            0

Dummy016                            0

Dummy017                            19

Dummy018                            0

Dummy019                            0

Dummy020                            0

Dummy021                            0

Dummy022                            22223

-------------------                 Transmit

Tx Time perct @ beacon intvl        1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Tx Frames Rcvd                      7124217

Tx Bcast Frames Rcvd                0

Tx Frames Dropped                   723350

Tx Bcast Frames Dropped             0

Tx Frames Transmitted               39305108

Tx Bytes Rcvd                       4123176852

Tx Bytes Transmitted                4200678483

Tx Time Frames Rcvd                 1107283613

Tx Time Frames Dropped              59540255

Tx Time Frames Transmitted          1107283613

Tx PS Unicast                       0

Tx DTIM Broadcast                   0

Tx Success With Retry               580417

Tx Multiple Retries                 48169

Tx Mgmt Frames                      1163162

Tx Mgmt Frames(PPS)                 0

Tx Beacons Transmitted              10998699

Tx Beacons Transmitted(PPS)         0

Tx Probe Responses                  64115

Tx Probe Responses(PPS)             0

Tx Data Transmitted Retried         5403232

Tx Data Transmitted                 6279400

Tx Data Frames                      7120523

Tx Broadcast Data Frames In         0

Tx Data Bytes Transmitted           4134319704

Tx Data Bytes                       4208380682

Tx Time Data Transmitted            661177657

Tx Time BC/MC Data                  6184

Tx Time Data dropped                59540255

Tx Time Data                        661177657

Tx Time Data (Ideal)                284675246

Client Health Samples               3944211

Tx Broadcast Data Frames Sent       0

Tx Broadcast Data Frames Sent(PPS)  0

Tx Multicast Data Frames            27651

Tx Multicast Data Frames(PPS)       0

Tx Workqueue Frames                 0

Tx Bypass Frames                    0

Tx DMO Multicast                    0

Tx DMO Invalid                      0

Tx DMO Converted                    0

Tx DMO Replicated                   0

Tx DMO Dropped                      0

Tx DMO No Client                    0

Tx DMO No BSSID                     0

Tx Unicast Data Frames              6294682

Tx RTS Frames                       9543605

Tx RTS Frames(PPS)                  0

Tx RTS Success                      3917024

Tx RTS Failed                       5626581

Tx CTS Frames                       1743003

Tx CTS Frames(PPS)                  0

Tx Powersave Queue Timeouts         0

Tx Dropped After Retry              0

Tx Dropped No Buffer                0

Tx Missed ACKs                      683721

Tx Failed Beacons                   22169

Tx Multi-Beacon Fail                0

Tx Long Preamble                    740603

Tx Short Preamble                   0

Tx Beacon Interrupts                11020870

Tx Interrupts                       0

Tx FIFO Underrun                    0

Tx Allocated Desc                   7124217

Tx Freed Desc                       0

Tx EAPOL Frames                     30465

TX STBC Frames                      844933

TX LDPC Frames                      2498708

Tx AGGR Good                        3820303

Tx AGGR Unaggr                      113946

Tx AGGR Retry                       485857

Tx WMM DATA [BK]                    367578

Tx Data Priority [BE]               2941642

Tx Data Priority [VI]               22

Tx Data Priority [VO]               2985521

Tx Data Frames  12 Mbps  (Mon)      161620

Tx Data Frames  54 Mbps  (Mon)      1432623

Tx Data Frames 108 Mbps  (Mon)      1516130

Tx Data Frames 300 Mbps  (Mon)      1540460

Tx Data Frames 450 Mbps  (Mon)      1573733

Tx Data Frames 1300 Mbps  (Mon)     70197

Tx Data Frames 1300 Mbps+ (Mon)     0

Tx Data Bytes   12 Mbps  (Mon)      77740156

Tx Data Bytes   54 Mbps  (Mon)      779547490

Tx Data Bytes  108 Mbps  (Mon)      1355673583

Tx Data Bytes  300 Mbps  (Mon)      1089933044

Tx Data Bytes  450 Mbps  (Mon)      798904944

Tx Data Bytes  1300 Mbps  (Mon)     32520487

Tx Data Bytes  1300 Mbps+ (Mon)     0

Tx 9 Mbps                           739976

Tx 12 Mbps                          8

Tx 24 Mbps                          63

Tx HT 7.2 Mbps                      131137

Tx HT 14.4 Mbps                     303377

Tx HT 21.7 Mbps                     2776

Tx HT 27 Mbps                       67

Tx HT 28.9 Mbps                     2738

Tx HT 30 Mbps                       586502

Tx HT 40.5 Mbps                     66

Tx HT 43.3 Mbps                     1676

Tx HT 45 Mbps                       535358

Tx HT 57.8 Mbps                     524

Tx HT 58.5 Mbps                     31

Tx HT 60 Mbps                       751752

Tx HT 65 Mbps                       181679

Tx HT 72.2 Mbps                     265

Tx HT 90 Mbps                       417256

Tx HT 97.5 Mbps                     164623

Tx HT 117 Mbps                      169

Tx HT 120 Mbps                      209081

Tx HT 130 Mbps                      500842

Tx HT 135 Mbps                      40911

Tx HT 150 Mbps                      46531

Tx HT 175.5 Mbps                    73

Tx HT 180 Mbps                      10952

Tx HT 195 Mbps                      380022

Tx HT 234 Mbps                      101

Tx HT 240 Mbps                      3198

Tx HT 260 Mbps                      230135

Tx HT 270 Mbps                      606

Tx HT 300 Mbps                      147

Tx VHT 292.5 Mbps                   117692

Tx VHT 325 Mbps                     171368

Tx VHT 351 Mbps                     26

Tx VHT 390 Mbps                     363961

Tx VHT 433.3 Mbps                   1038378

Tx VHT 468 Mbps                     12

Tx VHT 520 Mbps                     19140

Tx VHT 526.5 Mbps                   15

Tx VHT 585 Mbps                     10979

Tx VHT 650 Mbps                     6325

Tx VHT 702 Mbps                     16

Tx VHT 780 Mbps                     5243

Tx VHT 866.7 Mbps                   28399

Tx VHT 975 Mbps                     38

Tx VHT 1170 Mbps                    20

Tx VHT 1300 Mbps                    10

Tx WMM [BK]                         363008

Tx WMM [BE]                         2936847

Tx WMM [VI]                         22

Tx WMM [VO]                         2979523

Tx WMM [BK] Dropped                 878

Tx WMM [BE] Dropped                 31435

Tx WMM [VI] Dropped                 1

Tx WMM [VO] Dropped                 112849

Tx UAPSD OverflowDrop               0

TX Timeouts                         0

Lost Carrier Events                 0

Tx HT40 Hang Detected               0

Tx HT40 Hang Stuck                  0

Tx HT40 Hang Possible               0

Tx HT40 Dfs IMM WAR                 0

Tx HT40 Dfs HT20 WAR                0

Tx MAC/BB Hang Stuck                0

Tx Mgmt Bytes                       54925101

Tx Beacons Bytes                    0

Tx Managment Frames Dropped         571830

Tx Data Frames Dropped              151520

Tx AMSDU pkt count                  0

Tx EAPOL Frames Rcvd                30467

Tx EAPOL Frames Dropped             0

-------------------                 Receive

Rx Last SNR                         43

Rx Last SNR CTL0                    43

Rx Last SNR CTL1                    42

Rx Last SNR CTL2                    40

Rx Last SNR EXT0                    43

Rx Last SNR EXT1                    42

Rx Last SNR EXT2                    40

Rx Last ACK SNR                     43

Rx Frames Received                  17349116

Rx Good Frames                      16268094

Rx Bad Frames                       1081022

Rx Total Data Frames Recvd          4181985

Rx Total Mgmt Frames Recvd          1459967

Rx Total Control Frames Recvd       10626142

Rx Total Bytes Recvd                3408545096

Rx Total Data Bytes Recvd           2208072561

Rx Total RTS Frames Recvd           1772842

Rx Total CTS Frames Recvd           4358109

Rx Total ACK Frames                 376714

Rx Total Beacons Received           34204

Rx Total Probe Requests             266435

Rx Total Probe Responses            338

Rx Retry Frames                     813299

Channel Busy 1s                     2

Channel Busy 4s                     1

Channel Busy 64s                    3

Ch Busy perct @ beacon intvl        1 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2

Rx Time perct @ beacon intvl        0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Rx Discarded Events                 0

Rx ARM Scan Frames                  0

Rx Data Frames Retried              483511

Rx Data Frames                      5294491

Rx Data Frames(PPS)                 0

Rx Data Bytes                       2208072561

Rx Time Data                        542106938

Rx Duplicate Frames                 11724

Rx Broadcast Data Frames            71

Rx Multicast Data Frames            161

Rx Unicast Data Frames              4928038

Rx Null Data Frames                 1083011

Rx Mgmt Frames                      5048931

Rx Mgmt Frames(PPS)                 0

Rx Control Frames                   10626142

Rx Control Frames(PPS)              0

Rx Frames To Me                     15722659

Rx Bytes To Me                      3408545096

Rx Time To Me                       1489550426

Rx Broadcast Frames                 514942

Rx Probe Requests                   90593

Rx Probe Requests(PPS)              0

Rx RTS Frames                       1725089

Rx RTS Frames(PPS)                  0

Rx CTS Frames                       3917024

Rx CTS Frames(PPS)                  0

Rx PS Poll Frames                   28050

Rx CRC Errors                       1081017

Rx PLCP Errors                      57674

Rx Frames Dropped                   11438326

Rx PHY Events                       25048240

Rx RADAR Events                     0

RX Interrupts                       0

RX Overrun                          0

Rx Undecryptable                    0

Rx EAPOL Frames                     23599

Rx STBC Frames                      716111

Rx LDPC Frames                      1185771

Rx Data Priority [BK]               105965

Rx Data Priority [BE]               3770267

Rx Data Priority [VI]               202854

Rx Data Priority [VO]               102899

Rx Data Frames  12 Mbps  (Mon)      361350

Rx Data Frames  54 Mbps  (Mon)      1791647

Rx Data Frames 108 Mbps  (Mon)      955561

Rx Data Frames 300 Mbps  (Mon)      629169

Rx Data Frames 450 Mbps  (Mon)      431982

Rx Data Frames 1300 Mbps  (Mon)     12276

Rx Data Frames 1300 Mbps+ (Mon)     0

Rx Data Bytes   12 Mbps  (Mon)      51363930

Rx Data Bytes   54 Mbps  (Mon)      1575479703

Rx Data Bytes  108 Mbps  (Mon)      229712449

Rx Data Bytes  300 Mbps  (Mon)      264415827

Rx Data Bytes  450 Mbps  (Mon)      84631837

Rx Data Bytes  1300 Mbps  (Mon)     2468815

Rx Data Bytes  1300 Mbps+ (Mon)     0

Rx 6 Mbps                           128571

Rx 9 Mbps                           365491

Rx 12 Mbps                          188913

Rx 18 Mbps                          149337

Rx 24 Mbps                          100124

Rx 36 Mbps                          60159

Rx 48 Mbps                          207692

Rx HT 6.5 Mbps                      163653

Rx HT 7.2 Mbps                      4232

Rx HT 13 Mbps                       110149

Rx HT 13.5 Mbps                     78278

Rx HT 14.4 Mbps                     10721

Rx HT 19.5 Mbps                     39290

Rx HT 21.7 Mbps                     281

Rx HT 26 Mbps                       6307

Rx HT 27 Mbps                       352387

Rx HT 28.9 Mbps                     5571

Rx HT 30 Mbps                       13157

Rx HT 39 Mbps                       3847

Rx HT 40.5 Mbps                     850381

Rx HT 43.3 Mbps                     836

Rx HT 45 Mbps                       9346

Rx HT 52 Mbps                       1220

Rx HT 54 Mbps                       308260

Rx HT 57.8 Mbps                     4

Rx HT 58.5 Mbps                     171721

Rx HT 60 Mbps                       100193

Rx HT 65 Mbps                       35648

Rx HT 72.2 Mbps                     88

Rx HT 78 Mbps                       411

Rx HT 81 Mbps                       223908

Rx HT 86.7 Mbps                     115

Rx HT 90 Mbps                       48165

Rx HT 97.5 Mbps                     11496

Rx HT 104 Mbps                      1

Rx HT 108 Mbps                      250390

Rx HT 117 Mbps                      172322

Rx HT 120 Mbps                      4006

Rx HT 121.5 Mbps                    9786

Rx HT 130 Mbps                      4553

Rx HT 135 Mbps                      8933

Rx HT 150 Mbps                      48972

Rx HT 162 Mbps                      40249

Rx HT 175.5 Mbps                    125030

Rx HT 195 Mbps                      9913

Rx HT 216 Mbps                      22627

Rx HT 234 Mbps                      58665

Rx HT 243 Mbps                      691

Rx HT 260 Mbps                      5475

Rx HT 270 Mbps                      2

Rx VHT 29.3 Mbps                    12660

Rx VHT 32.5 Mbps                    3385

Rx VHT 87.8 Mbps                    136235

Rx VHT 263.3 Mbps                   58142

Rx VHT 292.5 Mbps                   68777

Rx VHT 325 Mbps                     25548

Rx VHT 351 Mbps                     84439

Rx VHT 390 Mbps                     213838

Rx VHT 433.3 Mbps                   108157

Rx VHT 468 Mbps                     3813

Rx VHT 526.5 Mbps                   704

Rx VHT 585 Mbps                     327

Rx VHT 702 Mbps                     39

Rx VHT 780 Mbps                     155

Rx VHT 866.7 Mbps                   7238

Rx WMM [BK]                         72538

Rx WMM [BE]                         3377971

Rx WMM [VI]                         201742

Rx WMM [VO]                         90953

RX bad length                       0

Rx Null Src MAC                     28539

-------------------                 SLB

SLB: Total Clients Found            0

....

SLB: Assoc Requests Refused         0

Rx Data Frames TxBF                 106

Throttle drops                      0

Stop all but Mgmt                   0

CCA percentage of our bss           1

CCA percentage of other bss         0

CCA percentage of interference      0

.....

MU-NDPA frames Tx failed            0

MU-NDP frames Tx failed             0

MU Sequences posted                 0

Client swap with pkt drop           0

Client swap pkt drop frames         0

 

 

...

..........................

...

 

MU Sequences posted                 0

Client swap with pkt drop           0

Client swap pkt drop frames         0

                                   

ANUL RADIO Stats

----------------

Parameter                   Value

---------                   -----

ANUL Assoc Rejects          0

ANUL Auth Rejects           0

ANUL Dos STA Rejects        0

ANUL Auth Throttle Rejects  0

ANUL Max STA Rejects        0

 

show ap debug radio-stats ap-name "AP225-G-frecuency-Issues" radio 1 advanced

 

 

RADIO Stats

-----------

Parameter                           Value

---------                           -----

-------------------                 General

Total Radio Resets                  246575

Resets Beacon Fail                  0

BB Check Positives                  0

 

....

External Reset                      0

PCI Fatal Intr Reset                0

Chainmask Reset                     0

TX stat Reset                       0

TX Power Changes                    5

Channel Changes                     2

Failed Channel Changes              0

Radio Band Changes                  0

Current Noise Floor                 96

Dummy NF pkts on home channel       0

Dummy NF pkts on scan channel       0

Avail TX Buffers                    2048

11g Protection                      1

Last TX Antenna                     0

Last RX Antenna                     0

Scan Requests                       318059

Scan Rejects                        5200

Load aware Scan Rejects             0

 

Any help is welcome.

Help for what?

 

Radio resets occur when there is severe interference, which happens every now and then.  If you think you have an issue, please open a case with TAC who will gather information and determine the best course of action.

A few resets from time to time can be caused by interference. The issue we are seeing results in 1 reset every 2 seconds. It happens only on the 2.4Ghz radio. 

Aside from verification through the logs, you can visit the AP location, switch off your 5Ghz client radio, and verify that there is no way to associate with the AP before the 2.4Ghz radio resets again. If you have something like splunk, you can find APs that generate thousands of radio resets every hour. 

The 2.4Ghz radio on the AP will be unusable until you perform a full reboot. After that point, it will work fine for days or weeks into the future (we haven't determined how long it is before or even if this problem will return or if it happens randomoly to APs). Aruba tech support is aware of this issue in certain trains of code and we are sending log information to assist the development team in diagnosing the problem.

 

Again though, if you see less than 1,000 resets per hour, you do not have this problem. This issue will generate well over 1,000 radio resets, per AP, per hour. 

 

Finally, the 5Ghz radio continues to work. As such, we have only a few examples of this actually affecting clients. Most use the 5Ghz radio or roam to another 2.4 nearby. 

 

---

@Lawson wrote:

A few resets from time to time can be caused by interference. The issue we are seeing results in 1 reset every 2 seconds. It happens only on the 2.4Ghz radio. 

Aside from verification through the logs, you can visit the AP location, switch off your 5Ghz client radio, and verify that there is no way to associate with the AP before the 2.4Ghz radio resets again. If you have something like splunk, you can find APs that generate thousands of radio resets every hour. 

The 2.4Ghz radio on the AP will be unusable until you perform a full reboot. After that point, it will work fine for days or weeks into the future (we haven't determined how long it is before or even if this problem will return or if it happens randomoly to APs). Aruba tech support is aware of this issue in certain trains of code and we are sending log information to assist the development team in diagnosing the problem.

 

Again though, if you see less than 1,000 resets per hour, you do not have this problem. This issue will generate well over 1,000 radio resets, per AP, per hour. 

 

Finally, the 5Ghz radio continues to work. As such, we have only a few examples of this actually affecting clients. Most use the 5Ghz radio or roam to another 2.4 nearby. 

 

---

Yes... same pattern... as i wrote (BTW: I have the same version: 6.5.0.3 and 80% are AP-225 model).

 

I just try to explain my analisys over the "sh ap tech-support ap-name APwiththisissue" output were I got 246,575 22.4GHz radio resets!!

 

and saw it directly (no splunk) the key lines:

#reset: !!!BIG-hammer!!! 

#wl1: fatal error, reinitializing

#wl1: ap watchdog handler error

 

Best regards.

 

 

---

@Homerodesepcionado wrote:

---

@Lawson wrote:

A few resets from time to time can be caused by interference. The issue we are seeing results in 1 reset every 2 seconds. It happens only on the 2.4Ghz radio. 

Aside from verification through the logs, you can visit the AP location, switch off your 5Ghz client radio, and verify that there is no way to associate with the AP before the 2.4Ghz radio resets again. If you have something like splunk, you can find APs that generate thousands of radio resets every hour. 

The 2.4Ghz radio on the AP will be unusable until you perform a full reboot. After that point, it will work fine for days or weeks into the future (we haven't determined how long it is before or even if this problem will return or if it happens randomoly to APs). Aruba tech support is aware of this issue in certain trains of code and we are sending log information to assist the development team in diagnosing the problem.

 

Again though, if you see less than 1,000 resets per hour, you do not have this problem. This issue will generate well over 1,000 radio resets, per AP, per hour. 

 

Finally, the 5Ghz radio continues to work. As such, we have only a few examples of this actually affecting clients. Most use the 5Ghz radio or roam to another 2.4 nearby. 

 

---

Yes... same pattern... as i wrote (BTW: I have the same version: 6.5.0.3 and 80% are AP-225 model).

 

I just try to explain my analisys over the "sh ap tech-support ap-name APwiththisissue" output were I got 246,575 22.4GHz radio resets!!

 

and saw it directly (no splunk) the key lines:

#reset: !!!BIG-hammer!!! 

#wl1: fatal error, reinitializing

#wl1: ap watchdog handler error

 

Best regards.

 

 

---

Please open a tac case.  You could have a different issue than the person who opened this thread.