Wireless Access

<WARN> |AP xxxxx-AP09@x.x.x.x sapd| AM d8:c7:c8:5e:80:e8: ARM HT Empty Channel Trigger new 36+/0/0 old 153-/50/0 new_rra 36+/8
Mar 16 13:26:13 stm[2400]: <400180> <WARN> |stm| handle_ap_message_response: BSS d8:c7:c8:5e:80:e8 nothing outstanding
Mar 16 13:26:13 stm[2400]: <400180> <WARN> |stm| handle_ap_message_response: BSS d8:c7:c8:5e:80:e9 nothing outstanding
Mar 16 13:26:13 stm[2400]: <400180> <WARN> |stm| handle_ap_message_response: BSS d8:c7:c8:5e:80:ea nothing outstanding
Mar 16 13:26:13 stm[2400]: <400180> <WARN> |stm| handle_ap_message_response: BSS d8:c7:c8:5e:80:eb nothing outstanding
Mar 16 13:26:13 stm[2400]: <400180> <WARN> |stm| handle_ap_message_response: BSS d8:c7:c8:5e:80:ec nothing outstanding
Mar 16 13:26:13 stm[2400]: <400180> <WARN> |stm| handle_ap_message_response: BSS d8:c7:c8:5e:80:ed nothing outstanding
Mar 16 13:26:13 stm[2400]: <400180> <WARN> |stm| handle_ap_message_response: BSS d8:c7:c8:5e:80:ee nothing outstanding
Mar 16 13:26:13 stm[2400]: <400180> <WARN> |stm| handle_ap_message_response: BSS d8:c7:c8:5e:80:ef nothing outstanding
Mar 16 13:26:22 sapd[1007]: <404085> <WARN> |AP xxxx-AP06@x.x.x.x sapd| AM d8:c7:c8:11:87:98: ARM HT Channel Interference Trigger new 36+/90 old 149+/139 new_rra 36+/8 TCI 10
Mar 16 13:26:22 stm[2400]: <400180> <WARN> |stm| handle_ap_message_response: BSS d8:c7:c8:11:87:98 nothing outstanding
Mar 16 13:26:22 stm[2400]: <400180> <WARN> |stm| handle_ap_message_response: BSS d8:c7:c8:11:87:99 nothing outstanding
Mar 16 13:26:22 stm[2400]: <400180> <WARN> |stm| handle_ap_message_response: BSS d8:c7:c8:11:87:9a nothing outstanding
Mar 16 13:26:22 stm[2400]: <400180> <WARN> |stm| handle_ap_message_response: BSS d8:c7:c8:11:87:9b nothing outstanding
Mar 16 13:26:22 stm[2400]: <400180> <WARN> |stm| handle_ap_message_response: BSS d8:c7:c8:11:87:9c nothing outstanding
Mar 16 13:26:22 stm[2400]: <400180> <WARN> |stm| handle_ap_message_response: BSS d8:c7:c8:11:87:9d nothing outstanding
Mar 16 13:26:22 stm[2400]: <400180> <WARN> |stm| handle_ap_message_response: BSS d8:c7:c8:11:87:9e nothing outstanding
Mar 16 13:26:22 stm[2400]: <400180> <WARN> |stm| handle_ap_message_response: BSS d8:c7:c8:11:87:9f nothing outstanding
Mar 16 13:26:25 sapd[837]: <404085> <WARN> |AP xxxxx-AP07@x.x.x.xsapd| AM d8:c7:c8:5e:77:28: ARM HT Channel Interference Trigger new 157+/65 old 36+/130 new_rra 157+/8 TCI 10
Mar 16 13:26:25 stm[2400]: <400180> <WARN> |stm| handle_ap_message_response: BSS d8:c7:c8:5e:77:28 nothing outstanding
Mar 16 13:26:25 stm[2400]: <400180> <WARN> |stm| handle_ap_message_response: BSS d8:c7:c8:5e:77:29 nothing outstanding
Mar 16 13:26:25 stm[2400]: <400180> <WARN> |stm| handle_ap_message_response: BSS d8:c7:c8:5e:77:2a nothing outstanding
Mar 16 13:26:25 stm[2400]: <400180> <WARN> |stm| handle_ap_message_response: BSS d8:c7:c8:5e:77:2b nothing outstanding
Mar 16 13:26:25 stm[2400]: <400180> <WARN> |stm| handle_ap_message_response: BSS d8:c7:c8:5e:77:2c nothing outstanding
Mar 16 13:26:25 stm[2400]: <400180> <WARN> |stm| handle_ap_message_response: BSS d8:c7:c8:5e:77:2d nothing outstanding
Mar 16 13:26:25 stm[2400]: <400180> <WARN> |stm| handle_ap_message_response: BSS d8:c7:c8:5e:77:2e nothing outstanding
Mar 16 13:26:25 stm[2400]: <400180> <WARN> |stm| handle_ap_message_response: BSS d8:c7:c8:5e:77:2f nothing outstanding

It is a cosmetic message.  It is fixed in 6.4.3.4  It was observed along with another controller crash, but the controller does not have to crash to see that message.

Thanks a lot for the detailed information on the log messages Mr.Colin. We are facing intermittent  connectivity (Randomly) issues with AP 105 with controller 3600 using 6.4.3.3.

The Symptoms were similar to below article like two minutes random discconection. So trying understand the log.

http://community.arubanetworks.com/t5/Wireless-Access/Limited-Access/m-p/123305/highlight/true#M26454

Thanks a lot once again.

There are many reasons for disconnects outside of that message.  How long have the disconnects been happening?

let me give a brief Description of the environment, Its university

They 6 controllers out of that 4 local controllers(3600) , 2 controllers manage AP-225  where classes are conducted and 2 controllers are managing AP-105 at hostel environment.

Master and standby master are 7220.

All the controlers are currently on 6.4.3.3 .They send survey quaterly basis and act accordingly

The disconnects are reported intially to us in the start of the year and informed complaints only from controllers having Ap-105. We found  Radar interference quite often on the logs on controllers of AP-105 so requested to disable the DFS channels only for AP-105. 

So they did changes as requested in March and have a feedback that the still issue persists. after Changes they did tests placing AIR Monitor in the environment  and having test clients which helped them to provide observation

Once we had detail of the symptom after the changes it was described precisely as mentioned in the article.

The ping drops suddenly and recovers after 1 or 2 minutes automatically.

We are planning to have test clients for 2 days  in the same environment and enable user-debug (having syslog server) for the clients  and to see any information will help us. 

We are planning to use the AIR-monitor as spectrum analyzers ?

Any commands to extract the logs spectrum analyzers which can help?

Any other suggestions as the issue is random and to monitor the clients 24/7 may be tough?

How many SSIDs do you have?

Do you have Drop Broadcast and Multicast Enabled on all your SSIDs?

What is the ARM Minimum and Maximum Transmit power in the ARM profile for those APs?

They had total 10 ( including two hiddent) and we asked to reduce. Currently they have 8 SSID's and they will reduce one more in futrue..

We did enable Drop Broadcast and Multicast on all SSID's .

BCMC optimization on all the Vlans.

ARM they left in default values.

Customer want to know the reason why it happens randomly (may be 3 or 4 times ) in a day. and act accordingly if need persists to add more AP's also. But detailed RCA is must. So any Suggestions in isolating the cause of the issue.

The default ARM values are minium 18 and maximum 127.  That is too much and could cause roaming and intereference issues.  The ARM values should be reduced to 12 minimum and 18 maximum.

You seem to be doing everything else right.  8 SSIDs is way too much.  You can see how many APs can see another AP by typing "show ap arm state ap-name <name of ap>".  Using that, you can determine possible cochannel intererence.

Thank you for the detailed information. Will try to check as suggested as customer want to know the cause before doing the changes.

Hope user-debug of the test clients which are going to be stationary

and the command you suggested "show ap arm state ap-name <name of ap>" will help to move forward on the issue. Will keep you posted as things progress especially afetr collecting the information from Site tomorrow.

Truthfullly, I rarely look at the user debugs, because they frequently do not say anything UNLESS that is the only client that is having problems.  If all of the clients are having problems, you need to monitor and improve the whole environment.  If improving the whole environment does not help (enable drop broadcasts, limiting SSIDs to 4 or less, constricting the power between 12 and 18), turning on the user debug might give you a clue.  The key to troubleshooting wireless is that since the environment is shared among all users, you need to first ensure that the environment has the best chance to succeed, and then any client that does not function properly, you can then focus on that specific client.

Good Luck.

Thanks Mr.Collin for the information. If my understanding is correct,  you want  to correct the environment with precise settings then troubleshoot.

Is there any other commands other than below which can help and support it is environmental issue

Show AP arm state ap-name  <ap-name> 

As said customer wants cause or proof before making any changes . 

I would look at the Dashboard Tab on the controller to see the RF utilization of access points.  It is probably good to see the RF utilization of the access points at midnight.  If it is at or about 10%, it is generated by too much coverage, too much power, or too many SSIDs, since the traffic observed would not be "data" traffic.  "show ap radio-summary" will also show the RF utilization of every AP on the commandline when the command is run.  This can be done as an alternative to the Dashboard tab on the controller.  

AM d8:c7:c8:5e:7b:a8: ARM Channel 36 Physical_Error_Rate 0 MAC_Error_Rate 70 Frame_Retry_Rate 0 arm_error_rate_threshold 50 arm_error_rate_wait_time 30
AM 18:64:72:55:3b:10: ARM Channel 149 Physical_Error_Rate 0 MAC_Error_Rate 0 Frame_Retry_Rate 82 arm_error_rate_threshold 50 arm_error_rate_wait_time 30
AM 18:64:72:56:32:a0: ARM Channel 1 Physical_Error_Rate 0 MAC_Error_Rate 0 Frame_Retry_Rate 81 arm_error_rate_threshold 50 arm_error_rate_wait_time 30

We are seeing MAC_error_rate most of the time for A radio and Frame_error_rate for G radio triggering the channel change.

May you help me what will lead to this and i hope frequent channel chnage can also cause the intermittent client issues rite?

In the ARM profile, I would change the Error Rate Threshold to 70 and the Error Rate Wait time to 90 to prevent that from happening.  Below is a chart from the 802.11ac Roaming and Optimization Guide showing the same:

http://community.arubanetworks.com/t5/Validated-Reference-Design/RF-and-Roaming-Optimization-for-Aruba-802-11ac-Networks/ta-p/227716

The default changes have been made in the new ArubaOS 6.4.4.x code.  For all others, the change would need to be made manually.

Thanks Mr.Colin for the information.

If the controller is upgarded to 6.4.4.x the changes will take effect automatically correct?

May you give few scenario why we may see this errors just for knowledge sake? Is it because of co-channel interference due to arm settings etc ?

Upgrading to 6.4.4.x will not change settings that you already have.  If you create a new profile on 6.4.4.x, it will have the new parameters.   It is easier to just change the two parameters outlined in the ARM profile, than upgrade.

The messages show when there is interference above a certain threshold for a certian amount of time.  For many clients, it is less disruptive to allow the AP to stay on the same channel, rather than change so soon, and hence the recommendation for the new parameters.

Thanks for detail information Mr.Colin. Customer is planning to upgrade to 6.4.4.x due to other issues.