Aruba Instant & Cloud Wi-Fi

Reply
New Contributor
Posts: 4
Registered: ‎05-08-2016

IAP-205 wireless interface restarts (wlc_reset)

Hi,
I am running 12 units IAP-205 with software 6.4.2.6-4.1.1.11.
In random period of time, some groups of users connected to one AP suffer from packet loss.
From syslogs I see that every day radios (5Ghz or 2.4Ghz) on different APs rebooting with some error.
All APs not overloaded and in the moment of failure can have 2 clients.
This is typical message I catch from AP after problem:
show log driver

[8151765.330000] failed to get phy_rxiqest
[8151876.506000] stats > 100: ccastats_us: 74016, acc->statss_ms: 74
[8152001.944000] stats > 100: ccastats_us: 74268, acc->statss_ms: 74
[8152064.714000] stats > 100: ccastats_us: 74187, acc->statss_ms: 74
[8152142.786000] wl0 wlc_dotxstatus: consumes 0 frames!
[8152142.786000] wl0 raw txstatus 0063 3A09 | 077A 0000 | 00000000 00000000 || 00007195 00000000 | 00000000
[8152142.786000] pktpend: 0 1 0 0 0 ap: 1
[8152142.786000] frameid 0x3a09 queue 1
[8152142.786000] wl0: wlc_bmac_txstatus:: bad txstatus 3A090063 2000077A 00000000 00000000 || 00007195 00000000 00000000 C7B080C8
[8152142.786000] error 1 caught in wlc_bmac_txstatus
[8152142.786000] wl0: wlc_dpc: txstatus fatal error
[8152142.786000] wl0: wlc_reset: !!!BIG-hammer!!!
[8152142.786000] wl0: 1 tx fifo 1 underflows!
[8152142.806000] wl0: wlc_txbf_init_sta: scb aleady has user index 0
[8152147.590000] wl0: ampdu_resp_timeout: cleaning up resp tid 0 waiting for seq 0x6fb for 200 ms
[8152356.796000] wl0: ampdu_resp_timeout: cleaning up resp tid 0 waiting for seq 0x9a for 200 ms

show log kernel

54616.806000] wl0: PHYTX error
[8154616.806000] wl0: PHYTX error
[8154616.806000] wl0: PHYTX error
[8154616.806000] wl0: PHYTX error
....
[8154616.856000] wl0: PSM microcode watchdog fired at 8138192 (seconds). Resetting.
[8154616.856000] ucode revision 876.1 
[8154616.856000] wl0: PSM microcode watchdog fired at 8138192 (seconds)
[8154616.856000] wl0: psmdebug 0x00ff861c, phydebug 0x00000048, psm_brc 0x00b0 psm_brc_1 0x00a2
[8154616.856000] wepctl 0x0015 wepstat 0x0003 wep_hdrloc 0x0026 wep_psdulen 1454
[8154616.856000] RXE_RXCNT: 0x0014 DAGG: ctl: 0x0000, bleft: 0x095f
[8154616.856000] offs: 0x0034, stat: 0x13db, len: 0x0000
[8154616.856000] phydebug
[8154616.856000] 0x0048
...
[8154616.858000] PC :
[8154616.858000] 0xff861c
[8154616.858000] 0xff861c
[8154616.858000] wlc_dump_aggfifo:
[8154616.858000] framerdy 0x2 bmccmd 1 framecnt 0x2 
[8154616.858000] AQM agg params 0xfff maxlen hi/lo 0x4 0x2040 minlen 0xc8 adjlen 0x8
[8154616.858000] AQM agg results 0x8001 len hi/lo: 0x0 0x5b2 BAbitmap(0-3) 0 0 0 0
[8154616.858000] wlc_dpc:605 wl0 ASSERT 'PSM Watchdog' macintstatus:0x2800
[8154616.858000] wl0: wlc_dpc: macintstatus & MI_GP0 non-zero, HAMMERING
[8154616.858000] wl0: wlc_reset: !!!BIG-hammer!!!
[8154616.878000] wl0: wlc_txbf_init_sta: scb aleady has user index 0
[8154773.658000] wl0: ampdu_resp_timeout: cleaning up resp tid 1 waiting for seq 0x65c for 200 ms
[8154996.036000] stats > 100: ccastats_us: 74133, acc->statss_ms: 74
[8155028.930000] wl0: ampdu_resp_timeout: cleaning up resp tid 0 waiting for seq 0xd8a for 200 ms

Peak CPU Util in the last one hour

----------------------------------
Timestamp CPU Util(%) Memory Util(%)
--------- ----------- --------------
2016-05-08 14:36:21 37 67

I guess it is some driver problem but I`m not sure. If you have some thoughts, please share.

Aruba Employee
Posts: 95
Registered: ‎09-10-2015

Re: IAP-205 wireless interface restarts (wlc_reset)

open a tac case, it has already been reported and under investigation.

Occasional Contributor I
Posts: 6
Registered: ‎01-09-2015

Re: IAP-205 wireless interface restarts (wlc_reset)

Is this a known issue? Please provide additional information such as bug IDs or existing case numbers for this issue.
New Contributor
Posts: 4
Registered: ‎05-08-2016

Re: IAP-205 wireless interface restarts (wlc_reset)

It is already resolved issue. Try latest stable firmware

Occasional Contributor I
Posts: 6
Registered: ‎01-09-2015

Re: IAP-205 wireless interface restarts (wlc_reset)

Thanks for the suggestion, but we are running 6.5.0.3, which I believe is the latest firmware. 

We see this interface reset error on ~70 of our AP-225 units around campus (out of ~1,700).

The "wl1: fatal error, reinitializing > wl1: wlc_reset: !!!BIG-hammer!!! " error is generated by these APs every 2 seconds. It only affects the 2.4Ghz radio and prevents any clients from associating to that radio. The 5Ghz radio continues to work fine and serve clients. Rebooting the AP eliminates the issue and the 2.4Ghz radio begins serving clients agian. 

Occasional Contributor I
Posts: 6
Registered: ‎01-09-2015

Re: IAP-205 wireless interface restarts (wlc_reset)

My Co-worker created a detailed post about this particular issue we are having:

https://community.arubanetworks.com/t5/Wireless-Access/Problem-with-radio-resets-quot-BIG-Hammer-quot/m-p/285430#M66902

 

Guru Elite
Posts: 21,018
Registered: ‎03-29-2007

Re: IAP-205 wireless interface restarts (wlc_reset)

Radio resets typically are due to errors due to interference. What is the arm min and max transmit power for your WLAN?


Colin Joseph
Aruba Customer Engineering

Looking for an Answer? Search the Community Knowledge Base Here: Community Knowledge Base

Occasional Contributor I
Posts: 6
Registered: ‎01-09-2015

Re: IAP-205 wireless interface restarts (wlc_reset)

Our 2.4Ghz ARM profile is set to range from 15 to 9.

Our 5Ghz ARM profile is set to range from 18 to 12.

We see this issue only on our 2.4Ghz radios. The radio resets every 2 seconds, preventing any client associations. This behavior continues throughout the night when I'd imagine the RF environment should be less noisy. 

Daytime noise floor is typically in the mid to upper -90 range for the APs experiencing this issue.

If we reboot the AP, the problem goes away for at least a few days, but may show up on other APs in the building. 

 

Thanks for your thoughts, we also have an open case with TAC. They claim to have seen this before and development wants to collect some log data from us. 

Guru Elite
Posts: 21,018
Registered: ‎03-29-2007

Re: IAP-205 wireless interface restarts (wlc_reset)

I would stick with TAC on this one. You are pulling together threads that may or may not be relevant to your situation, and that might delay relief.

Noise is mainly an indicator for non-wifi interference. High utilization can be caused by wifi, non wifi interference or a combination. That could cause radio resets. If you have a dense deployment, I would lower the max TX for 2.4ghz to 12 and see if it continues. Since TX power is logarithmic, every 3 that you decrement is an effective halving of transmit power which can make a great difference.

If you type "show ap arm state ap-name <name of AP> how many APs can you see on the 2.4ghz band?


Colin Joseph
Aruba Customer Engineering

Looking for an Answer? Search the Community Knowledge Base Here: Community Knowledge Base

Occasional Contributor I
Posts: 6
Registered: ‎01-09-2015

Re: IAP-205 wireless interface restarts (wlc_reset)

Thanks, we may try moving the 2.4Ghz max down to 12 and see if that changes the frequency of this problem. 

 

Trying your suggested command, some of the APs return no output besides the date. and time.

For the APs that do return ARM state info, the number of valid neighbors ranges from 10-20 with relevant neighbors in the 2-10 range. 

 

Search Airheads
Showing results for 
Search instead for 
Did you mean: