Problem with radio resets "BIG-Hammer"

[Dchris]

My partner recently found an error with a number of our access points around campus. He noticed the issue when troubleshooting an AP205H. The student connected to the device was having issues with disconnecting from the AP's 2.4 Ghz radio. After running a debug log he noticed that the radio had a large number of radio resets on the 2.4Ghz radio(460,000). After checking the Splunk logs he noticed that the error was named "wl1: wlc_reset: !!!BIG-Hammer!!!". After looking at the logs many of the AP's that are showing the BIG-hammer reset are resetting their 2.4Ghz radios every second or every other second. The number of resets are making the 2.4Ghz radios on the AP's with this issue unusable. 

After running some queries we have noticed that we are having this issue with about 100 of the almost 4,000 aps on campus.

Additional Information;

• The AP's with the issue are not all in the same building, VLAN, and Controller
• Resetting the AP seems to solve the issue but we are not yet sure how long it will take for the issue to reappear. 
• We are running OS Version 6.5.0.3
• We use Splunk to search for the trends. If you would like a command to spit out a nice query please let me know. 

We are going to work with TAC to see if they have any recommendations. We are currently looking for more information on this issue. Does anyone else have this issue? Can you search your logs for this issue on your setup? 

Thank you. 

[Dchris]

We have been doing some additional research on this issue and have not found much about it. My partner did find a post that posted previously explaining the same issue but the only comment on there was from an tech stating that it is a known issue. 

If anyone has more information about the issue please let us know. 

Link to previous post.

https://community.arubanetworks.com/t5/Aruba-Instant-Cloud-Wi-Fi/IAP-205-wireless-interface-restarts-wlc-reset/td-p/266304

[OttawaUser]

We have been having "Total Radio Resets" on AP 325 running 6.4.4.3 as well. The resets numbers are huge. 2.4 Ghz radio thousands, 5 Ghz hundreds. It is happening on 300+ APs.

In comparison, our AP 105 has very minimum number "Total Radio Resets" running the same 6.4.4.3. 

[cjoseph]

The AP325 is much, much more sensitive than the AP-105 ever was and so it is more susceptible to inteference.  It is not a good comparison.   There are quite a few reasons  for interference and radio resets and you need to work with TAC to understand what yours could be or if it is a bug.

[Ryan]

Sooooo, back to the quesiton at hand, WTH is the "BIG-hammer" issue? I have an open TAC case and they referred to the same thing. Is this a known bug? What causes the AP/radio to get into this state? Is there a fix in a later AOS? (I'm on 6.5.0.2.)

[Dchris]

Hello Ryan, 

We were told by our Aruba TAC that this is a known issue and they are currently working to fix it. In the past two weeks since my initial post my partner has been working with the aruba TAC to collect logs and data on the AP's that are having the issue. 

We do not have any idea what causes the Radios to go int othis state, we have tried to do a lot of investigating to see what could be causing the issue but nothing we have found has been conclusive. We have found that rebooting the AP in some cases will make the radio restes go away and in other cases they come back. 

The radio resets have become more of an issue on campus for us expecailly in areas with custom setups that rely on 2.4 Ghz coverage. We have a sports area that relies on one AP to cover a Media table and that AP's 2.4Ghz radio is basically inoperable for them. 

We will keep you posted on any updates on any updates we have and we will let you know when we get a solution from TAC. Please feel free to add any information that you get from your TAC. 

[Ryan]

FWIW, my workaround the fix the radio is to use spectrum local override and enable it then disable it on the 2.4ghz radio. Always fixes it.

[Dchris]

Good Morning Ryan, 

Thank you for the update on that. We also found yesterday that enabeling and disabeling ARM will fix the problem but in some locations the problem comes back. 

Update our case is that our TAC would like to upload a coustom image to a few of our AP's to see what happens. We are workign to figure out a way that we can upload this image now. We will keep you posted. 

Christopher

[Dchris]

We have a promising update, our TAC got back to us with the following information. 

"Our development team was able to identify the issue. The issue was due to the channel change and the APs radio was pushed to suspended state during the channel change and it gets recovered only after the radio reset. That’s the reason the AP radio did not reset when the ARM was disabled. The issue is verified and fixed in 6.5.0.4. The code is expected to release today."

We will let you know once we upgrade the code. We plan to wait a week or more before making the change. 

Christopher

[Ryan]

Christopher,

I look on Aruba support site under download software and only see 6.5.0.3 under "General Availability". 6.5.1.2 is under "Early Deployment".
On the HPE site, I see only 6.5.0.2 and 6.5.1.1.

Do you have 6.5.0.4 at this point?

And if anyone at HP/Aruba is listening, what's up with the inconsistency?

- Ryan -