Wireless Access

We recently upgraded our 7210 controller from 6.4.4.9 to 6.5.0.3. We have mostly AP-205s in place, with some 93-Hs and a few other one offs.  We are starting to receive complaints from students in dorm areas with the 205s about wireless issues.  In some cases, we've seen that they are attached to an AP on another floor or farther away than one very close to their room.  In other cases, we are starting to troubleshoot what look like garden variety signal / power / other issues that we've gone through in the past when tweaking the placement of APs.

 

Has anyone moved from 6.4 to 6.5 and experienced any issues?  No APs were added or moved during this upgrade, so we're starting to walk through normal troubleshooting procedures to see what we can figure out.  Thanks.

 

You could start looking at a couple of things:

- controller dashboard to have an idea if you guys have Aps reporting RF High utilization

- RF Management / ARM settings and what are the power levels set for 2.4 and 5

- 80 Mhz should be disabled under ARM

Please take a look at the RF and Roaming optimization VRD for 802.11ac
https://community.arubanetworks.com/t5/Validated-Reference-Design/RF-and-Roaming-Optimization-for-Aruba-802-11ac-Networks/ta-p/227716

Do you guys have in the same RF domain a mixture of 93H/205H ?

The 205s are not H models, just plain 205s.  The 93H models are used in some one off areas, such as a room off on its own away from the normal dorm halls where we have the 205s.

My apologies , the reason I asked is because either 205/205H are 802.11ac and if you mixed them with 93H (802.11n) in the same RF domain then you need to disabled VHT under HT-SSID-PROFILE and under ARM PROFILE

---

@jwhitaker@transy.edu wrote:

We recently upgraded our 7210 controller from 6.4.4.9 to 6.5.0.3. We have mostly AP-205s in place, with some 93-Hs and a few other one offs.  We are starting to receive complaints from students in dorm areas with the 205s about wireless issues.  In some cases, we've seen that they are attached to an AP on another floor or farther away than one very close to their room.  In other cases, we are starting to troubleshoot what look like garden variety signal / power / other issues that we've gone through in the past when tweaking the placement of APs.

 

Has anyone moved from 6.4 to 6.5 and experienced any issues?  No APs were added or moved during this upgrade, so we're starting to walk through normal troubleshooting procedures to see what we can figure out.  Thanks.

 

---

First look at the Channel Utilization for APs on the dashboard to understand how your network is doing.  Channel Utilization above 5% or more at midnight is not a good start.  That would indicate that the power on your access points are too high, causing clients to hold onto APs that they should not be on.  If it is above 40-50% during the day in sparse locations, you need to lower the power, cut rates, drop broadcasts or remove lower rates on all of your SSIDs to get the utilization down.

 

To take it a step further type "show ap arm state ap-name <name of ap>" to see how many APs each AP with a problem can see.  If an AP can see another AP on the same channel with an SNR of over 20, you will have cochannel interference that will affect your customer's performance.

 

I feel more people complain when you tell them that you have upgraded.  Look into the issues above first to cover your bases..

Hi Jason,

I don't want to lead you down the wrong path, but we upgraded to 6.5.0.3 back in November and recently came across an issue where a small population of AP-225s are resetting their 2.4GHz radios every 2 seconds - resulting in the SSID not being broadcasted on the 2.4GHz band - although the 5GHz SSID continues to function. I noticed this behavior back in December on one of our ticket-scanner AP where the manager said he could see it on his iPhone but not on the scanners -> when I saw his iPhone I immediately recognized the "5GHz delay" on the SSID. I rebooted the AP-225 and the problem went away. Only recently did I realize we may have a larger scale issue - my co-worker submitted a post after I discovered a 205H AP with about 13,000 radio resets in an area where a large population of APs only have maybe 10-15 at most - http://community.arubanetworks.com/t5/Wireless-Access/Problem-with-radio-resets-quot-BIG-Hammer-quot/m-p/285325

We haven’t had any reports beyond 1 or 2 about there being an issue as it appears a large amount of students are connecting to the 5Ghz radio – I was actually impressed that our 2.4 and 5 counts were 20% to 80%.

You could do a few quick samples on your AP-205 by running the following command and looking at "Total Radio Resets". However, some resets are due to interference as cjoseph as mentioned in our other posts - should compare similar models - 225 with a 225, 205 with a 205, and vice-versa.

 

show ap debug radio-stats ap-name <ap-name> radio 1

The big indicator is checking the driver-log and seeing BIG-hammer!!! reset over and over. A few 225s are generating these logs continuously.

show ap debug driver-log ap-name <ap-name>

[6333581.707517] chan dur      ibss           obss           interf       time
[6333581.707527] wl1: fatal error, reinitializing
[6333581.707538] wl1: wlc_reset: !!!BIG-hammer!!!
[6333581.762945] wl1 bmac set short slot 1, wl status 1
[6333581.793808] wl1: ap watchdog handler error
[6333583.707497] wl1: bcn inactivity detected HAMMERING
[6333583.707509] wl1: txbcnfrm 0 prev txbcnfrm 0 txbcn inactivity 2 timeout 2
[6333583.707521]
[6333583.707523] chan dur      ibss           obss           interf       time
[6333583.707533] wl1: fatal error, reinitializing
[6333583.707544] wl1: wlc_reset: !!!BIG-hammer!!!
[6333583.732442] stats > 100: ccastats_us: 8532, acc->statss_ms: 2
[6333583.763051] wl1 bmac set short slot 1, wl status 1
[6333583.793901] wl1: ap watchdog handler error
[6333585.707497] wl1: bcn inactivity detected HAMMERING
[6333585.707508] wl1: txbcnfrm 0 prev txbcnfrm 0 txbcn inactivity 2 timeout 2
[6333585.707520]
[6333585.707523] chan dur      ibss           obss           interf       time
[6333585.707533] wl1: fatal error, reinitializing
[6333585.707544] wl1: wlc_reset: !!!BIG-hammer!!!
[6333585.762967] wl1 bmac set short slot 1, wl status 1
[6333585.793819] wl1: ap watchdog handler error
[6333587.707005] wl1: bcn inactivity detected HAMMERING
[6333587.707017] wl1: txbcnfrm 0 prev txbcnfrm 0 txbcn inactivity 2 timeout 2
[6333587.707029]
[6333587.707031] chan dur      ibss           obss           interf       time
[6333587.707042] wl1: fatal error, reinitializing
[6333587.707052] wl1: wlc_reset: !!!BIG-hammer!!!
[6333587.762456] wl1 bmac set short slot 1, wl status 1
[6333587.793287] wl1: ap watchdog handler error
[6333589.707497] wl1: bcn inactivity detected HAMMERING
[6333589.707508] wl1: txbcnfrm 0 prev txbcnfrm 0 txbcn inactivity 2 timeout 2

Something of interest is that we don't have these problems on our 205Hs besides just the single outlier. This problem is largely noticeable in our 4th residence hall which is composed almost entirely of 205s - you actually led me to run the radio-stats command on our 233 APs and I only found maybe 4 or 5 that had the expected 10-15 counts. Now these stats could be normally considered all the home-devices that get brought into the rooms - but it's definitely an oddidity.

Finding out if your APs are exhibiting this issue is easier with a syslog server and setting the AP System Profile Driver Log to debugging (I tried the other levels one by one but didn't get the results till I reached debugging) -> note this is going to send a large amount of logs to your syslog server than normal. I consulted with our syslog admin prior to enabling this level and left it on for about 30 minutes to an hour. You should probably open a TAC case to be safe.

If you have splunk I could share the query our syslog admin created for us that has helped see where we have potential problems. We have a TAC case opened from last week and just yesterday they gathered some more logs to send to the development team.

Wow good tip.  I've found some of the target APs with upwards of 50K radio resets!  I have rebooted a few of them and continue to watch - did a simple reboot solve it for you, or do you still have APs exhibiting this behavior?  Thanks.

 

For the two AP-225s that have experienced the radio reset every 2 seconds - a reboot has resolved the issue so far. We haven't attempted anything on our apartment complex (AP-205s deployment) - as we're awaiting/gathering information for TAC. Also reviewing power levels for that particular residence hall and reading up on the VRD for Residence Halls.

There are so many variables, you should start by opening a TAC case...

:-) I'm laughing right now because "There are so many variables" are the exact words I tell folks every day and is very true. It's definitely been a great learning experience.


#AirheadsMobile

On a wired network, most devices perform uniformly.  That is not the case for a wireless network, so care must be taken to design it.  A wired network is not necessarily shared, so the performance of some devices do not determine the overall performance of the network.  In wireless, the bandwidth is shared among devices as well as among any type of energy that manages to appear on that band, so that creates many variables.  It is however possible to design a wireless network that takes all of those variables into account and make it perform like it should.  Nobody should just take wireless out the box, put it out there and expect it to perform well.  It should be designed, and periodically reviewed to ensure it is performing as expected.  In small networks, it is not especially hard to do.  In larger networks, where the feedback loop between consumers of the network and designers of the network, it is particularly difficult.  With all that being said, it is possible to tune the network synthetically and statistically to provide the best performance, even without alot of feedback.  I detail much of this at this link here:

 

http://community.arubanetworks.com/t5/Technology-Blog/Removing-the-Bottleneck-in-Wireless/ba-p/77978

We do understand the overall complexities of a wireless network and appreciate that not every client is equal.  That said, we had not advertised this firmware update, so when complaints started coming in after we did it, that made us suspicious that perhaps something was diffferent in this release - hence this post.  We are reviewing all of the recommended settings for signal strength, etc but we had already tuned most of these based on those recommendations a couple of years ago.  

Honestly,

 

I am just contributing ideas, because I have no idea what the state of your environment was before you had problems.  You can trust the ideas of what other people randomly say on this forum, or you can open a TAC case in parallel so that they can identify specific issues and come up with a solution based on what they see in your environment. 

 

It is just very painful to see people take random unsolicited advice on this forum without contacting TAC and going through weeks of pain...

cjoseph - totally hear you, and very much appreciate your contributions.  We've picked up a number of good thoughts from you on this forum in the past.  And we will engage TAC, but it is helpful (for us at least) to hear from other institutions like us, probably in similar dorm situations, and especially if they started seeing things on this particular version.  It helps us to do some troubleshooting on our own before we engage TAC so that we go in armed with some information.  Thanks!

Are you deploying your wireless in a dorm?  Have you seen the VRD for residence halls here?  http://community.arubanetworks.com/t5/Validated-Reference-Design/Next-Generation-Wireless-Architecture-for-Multimedia-Grade/ta-p/155598

You should PM me your email address and I will send you a link to send me your logs.  I can then recommend where you should go from there.

Yes - have seen that, but unfortunately went with 205s when we went with Aruba in 2014.  We do use 93H APs in some apartment-like dorms, but for normal "hall" type dorms, we still have these 205s.  And for the most part we do have good coverage, etc.  But there are always outliers, and we try to tweak and tune as best we can.

 

We upgraded from 6.4 to 6.5.0.3 as well.  We started getting complaints from our users in dorms as well stating that their signal dropped dramatically.  I noticed that the complaints were coming from the 2.4ghz users mainly.  Some users would connect and then disconnect.  Some users could see the SSID one minute and then the SSID would disappear.  Some things that I did that seemed to have fixed the problem was 1.  Increase the Arm power levels and 2.  I added a value in the "Local Probe Request Threshold" of the SSID profile.  The default is "0" and I added a value that ranged from "6-10".  This seemed to be the change that helped the most. 

cjoseph made an offline suggestion to remove 80MHz support from the ARM profiles that we had in place, as this is not desirable in high density deployments.  After making this change, we've had some reports that wireless has improved, and perhaps more concrete, we've seen the dashboard clear up a bit more than usual for APs with interference and channel busy.  We had already tuned a number of settings such as power (don't just turn it up, follow the min/max suggestions!) and other items, but this change did appear to make a difference for a number of APs that we sometimes see in the "orange" on that dashboard.  Thanks for that suggestion!  We are continuing to monitor for now.

 

---

@jwhitaker@transy.edu wrote:

cjoseph made an offline suggestion to remove 80MHz support from the ARM profiles that we had in place, as this is not desirable in high density deployments.  After making this change, we've had some reports that wireless has improved, and perhaps more concrete, we've seen the dashboard clear up a bit more than usual for APs with interference and channel busy.  We had already tuned a number of settings such as power (don't just turn it up, follow the min/max suggestions!) and other items, but this change did appear to make a difference for a number of APs that we sometimes see in the "orange" on that dashboard.  Thanks for that suggestion!  We are continuing to monitor for now.

 

---

To be clear, unchecking 80mhz channels will probably not help 2.4ghz performance, but it will make things more pleasant for 5ghz clients.  You might have fewer people complaining and in doing so, you might have more time to focus on fewer issues.

---

@JasonTango wrote:

We upgraded from 6.4 to 6.5.0.3 as well.  We started getting complaints from our users in dorms as well stating that their signal dropped dramatically.  I noticed that the complaints were coming from the 2.4ghz users mainly.  Some users would connect and then disconnect.  Some users could see the SSID one minute and then the SSID would disappear.  Some things that I did that seemed to have fixed the problem was 1.  Increase the Arm power levels and 2.  I added a value in the "Local Probe Request Threshold" of the SSID profile.  The default is "0" and I added a value that ranged from "6-10".  This seemed to be the change that helped the most. 

---

Settings a local probe request threshold *could* help some in a limited fashion.  You will run into trouble with clients that discover an AP through beacons as well as probe responses, because they will still be able to find an AP through beacons and still try to reach them.  The real way to solve the problem in a meaningful fashion of course is to turn the power down, and after that consider removing the lower rates on SSIDs.  Before doing either, you should take a snapshot of the Dashboard> Perform APs> Channel Utilization on the 2.4ghz band to see the before and after.

 

The key is reduced channel utilization in all situations.

OK... here goes the "me too" bandwagon.

 

We recently upgraded from Aruba OS 6.4.3.7 to Aruba OS 6.5.0.3 and are seeing some issues. And just to point out... prior to this upgrade we were receiving VERY few complaints at all. We were running very well for a long while on our previous version.

 

Since upgrading, the complaints have started come in. Mostly complaints of not being able to connect to an SSID or it dropping. This morning we (IT) were able to experience it first hand. The results were rather odd.

 

We got a complaint from a classroom that they could not connect to our 1X SSID. We went over and sat around a table outside of that classroom and had VERY strange results.

 

 - A Mac running OS X: would not connect at all. Was working in other areas just fine on that same 1X network..Turning off and on the wireless did not help.

Interesting.  Please do update us on your TAC case!  After making the 80Mhz adjustments mentioned earlier, we've heard "some better" so we are still doing some testing before engaging TAC.  

We currently have a TAC case open and Aruba is looking at several AP Tech-support log dumps as well as client debug logs.

 

So far we have tried several things on our end but have not seen any improvements. We disabled 80 MHz channels as well as making some changes to the default Client Match settings (based on our SE's suggestions). Unfortunately the issues continue.....

 

I just read through the release notes for 6.5.0.4 in hopes something would be a close match but don't see anything that really matches our issues.

 

Now we wait for TAC to go through the logs and see if they come up with anything.

 

Kevin,

Are you guys seeing these issues on a particular Access Point type / Client type?

Kevin,

Are you guys seeing these issues on a particular Access Point type / Client type?

No particular AP type. We are seeing them with AP-115's as well as AP-135's and AP-315's.

 

So far most noticeable on Mac's and Android devices but we do have complaints from some Windows 7 computers as well. Worth noting: My Windows 10 laptop is yet to show any of the symptoms at all.

Jut to close the loop on this... we opened a TAC case and how have things solved.

 

In our case we were feeding the VLAN for our 1X network to the controllers from a Clearpass server, so the controller had no VLAN assigned for that 1X network. Users got a vlan based on who they are. Some how between code bases this broke.

 

The fix (it is a system bug) was to assign a VLAN on the controller for that 1X network. They still get assigned a VLAN from our Clearpass servers, it just overrides the default vlan we assign (on the Controllers) up front. 

 

So an odd situation all around. But now fixed and working.

Kslate,

Please open your own thread. The subject is too general and it has no direction, so no one can focus on your specific issue.

We upgraded to 6.5.0.4, saw this item in the readme that seemed to match up what has been discussed in this thread:

 

Symptom: An AP failed to broadcast an SSID on the 802.11g radio. Improvements in the wireless driver of the AP resolved the issue. Scenario: Continuous 802.11g radio resets were observed on the AP. This issue was observed in 200 Series, AP-205H, 210 Series, and 220 Series access points running ArubaOS 6.5.0.3, ArubaOS 6.5.1.2, or earlier versions.

 

Now, we are not seeing those very high number of resets, and have had some reports of improvement from students that were complaining after the 6.5.0.3 upgrade.  Still monitoring, but we feel like this upgrade has resolved the issue.

 

Please keep us posted.  I am trying to get a maintennance window for this coming Monday so I can upgrade my wireless environment.  

Any complaints after upgrading to 6.5.0.4?  I upgraded Monday morning and so far so good but our students population is low due to spring break.

So far, so good for us on 6.5.0.4.  We no longer see the radio resets as mentioned earlier in this post, and a few particular students that had complained after 6.5.0.3 are now saying they are OK.

 

I don't normally respond in forums. I decided to because it might help. We just installed 12 Aruba-335's and had to use 6.5.0.3. This is just a business office. We don't bond. Even though I did set all SSID's HT to not enable usage of 80MHz. I found in the beginning of the deployment. The AP's where bonded. I did put in a TAC. We found that we had to turn to say none under 40MHz, none under 80MHz, and none under 120Mhz. Also in reg domain only check the channels to be 20MHz. uncheck the 40MHz, 80MHz and the 120MHz bonded channels.This cleared up the weird roaming issues, the high noise, and the high channel utilization. Maybe it helps.

Everybody who thinks they are having problems with the upgrade from 6.4.4.9 to 6.5.0.3 should consider upgrading to 6.5.0.4:

 

Please carefully read the release notes and load in your lab to test before you do.