Controllerless Networks

Hi everyone,

 

Fairly recently, our school district completed a replacement of our older controller-based AP-65s and 105s with all IAPs.  We have 40 sites, each of which has its own subnet and, therefore, its own IAP cluster.  Each site has an IAP-225 as the preferred master for the VC.  Some sites are all IAP-225s, but most are a mixture of a few IAP-225s and mostly IAP-205s.  The largest deployment has 142 IAPs, the second largest is 126, and the rest are all 100 or fewer.

 

Over the past month or so, we have started noticing 2 major issues, both with the IAP-205s:

 

1) After a few days of uptime, random IAP-205s throughout the district will stop handling DHCP requests properly.  We saw during a Wireshark packet capture that during the DHCP handshake process, the Discover packet went out, Offer came back, Request went out, but the ACK never came back from the client.  I don't know if that means the Offer made it to the AP and then never made it to the client, or if the Offer made it to the client but then the ACK never made it back to the AP.  But this particular problem has become rather epidemic, resulting in having to reboot entire IAP clusters on a weekly basis.  I've had a TAC case open on this for a few weeks, but we're not making much progress.  When this happens, I've been noticing that the IAPs in question have very little free memory as reported by the VC (like 4 MB or less), whereas they usually have about 30 or 40 MB free.

 

2) This happens less often, but sometimes, the CPU utilization on random IAP-205s gets pegged at 100%.  I just opened a TAC case on this as well.  When it happened today, I SSH'd into the IAP in question and did a "show cpu details", and a process called "dpimgr" was consuming 98% of the CPU.  I also did a "show tech-support" to send to TAC, but it took me a couple of tries because the IAP kept closing my SSH connection.

 

I'm starting to think the IAP-205s weren't designed well, i.e. perhaps they have too little memory and/or not enough CPU power, to be able to function well in a dense deployment.  Has anyone else had any experiences like this?  We've had no problems whatsoever with the IAP-225s.  I'm now wishing we had gone with ALL IAP-225s, but of course, that would have cost 3 times as much.

 

Thanks!

#AP225

how many users and what type of traffic are you seeing on the AP when this is happening?

The number of users can be low (less than 10) to high (50+) when the problem occurs.  As for what type of traffic... what do you mean?  There are usually devices connected that are still working (i.e. they already had IP addresses), so there's the typical HTTP, HTTPS, Bonjour, etc. stuff going by, if that's what you mean.

My thought was assesing user load on the AP. IE - is there heavy usage coming from multiple users when the issue occurs? The 205 is not a high user density AP

tsd - How do you define "high-density"?  There have been times where an IAP-205 has had 60+ clients on it.  A lot of the time, though, a large percentage of those clients are student devices (Smartphones, iPods, etc.) that are connected to our Guest network but aren't actually doing anything.  Aruba's info on their web site (http://www.arubanetworks.com/products/networking/access-points/200-series/) lists the 200-series as being ideal for "Medium density" WiFi environments, whatever that may mean.

 

cjoseph - PM coming your way.

If you have a large number of devices idling on your guest network, you could be hammering the captive portal with requests as applications try to get out to the Internet.

More and more devices are starting to have mechanisms that stop this but most still don't.


Thanks,
Tim

---

@cappalli wrote:
If you have a large number of devices idling on your guest network, you could be hammering the captive portal with requests as applications try to get out to the Internet.

---

There was a reference to captive portal hits in the release notes (from a previous version), but we're not using the Captive Portal.  We have a web filter that requires authentication before clients can get out, so many times, apps on a device won't work because of that, but that shouldn't affect the APs, since we're not using the actual Aruba Captive Portal feature.  In other words, clients can connect to our Guests network without having to authenticate or go through a Captive Portal, but they still have to then log in to the web filter before they can get out to the internet.  (They have to log in to the web filter regardless of whether they're using our Guest or Secure network.)

We ran into a "high resource utilization" issue on 6.4.0.3-4.1.0.1 and upgraded to 6.4.2.3-4.1.1.3

The issues have gone away.

 

We're running 205/275 at some locations and upgraded all locations to keep code the same everywhere. Our issues also took the form of near 100% memory use - but we generally saw it on 104/104 models.

I hope the upgrade helps your issues as well.

I upgraded our whole deployment to 6.4.2.3-4.1.1.3, but the issue has persisted.  Our local SEs thought it might have to do with broadcast traffic.  We are mostly on a single VLAN (the network just evolved that way) at each site, and we have some applications such as Lanschool that don't work on wireless machines if Broadcast Filtering is turned on.  The SEs suspect that may be the problem, though, so I turned boradcast filtering back on.  Last week was Spring Break, so we won't know until later this coming week if the problem goes away, since the problem only manifests itself after several days of actual use.

Well, unfortunately, turning broadcast filtering on did not help.  This week, we had a major epidemic.  I had to restart the entire wireless network at pretty much every school in the district yesterday.  At one site (a high school), about half of my APs stopped passing DHCP requests through again.  And I had a number of sites that showed high CPU utilization on IAP-205s and 204s.  Rather than trying to pinpoint and reboot the various affected APs, I just rebooted everything.  But this is becoming a major problem.  I haven't found a way in AirWave to, say, schedule a reboot every few days, so this has become a manual (and labor-intensive!) process.

 

Still working with TAC, although we haven't made any progress.

Upgrade you instant is to newer version: 4.1.1.4 , and see if your issues are gone.

bnewall,

 

Is your management VLAN separate from your user VLAN?

 

---

@cjoseph wrote:

bnewall,

 

Is your management VLAN separate from your user VLAN?

 

---

Sadly, no, at the moment.  We are going to address that this summer, as I'm sure it has at least something to do with it.  Interesting thing is, though, our 225s aren't having any issues, and we never ran into any problems with a single VLAN with our old controller-based network running AP-60s, 61s, 65s, 70s, 105s, and 124s.

 

bnewall,

 

The management VLAN being separate from the client vlan is crucial for scaling in an IAP network, and should be separate.  If you can make that a priority at a site where you have some flexibility, you could regain some stability in your network.

---

@kdisc98 wrote:
Upgrade you instant is to newer version: 4.1.1.4 , and see if your issues are gone.

---

Sorry, this response somehow got caught by our spam filter and I didn't see it right away.  I'm actually on 4.1.2.1, and the problems have not gone away.  Been on that version for a couple of weeks now.

Did you manage to find a solution to this, I'm having the same problems.

---

@Varmenni wrote:

Did you manage to find a solution to this, I'm having the same problems.

---

Yes, I did!  Sorry, I meant to update the thread and forgot to do so.  Actually, TAC came up with the solution:  Disable AppRF.  We aren't using it, and apparently, AppRF chews up a lot of CPU and memory resources.  Since disabling it, we haven't had anymore problems.

---

@bnewall wrote:

---

apparently, AppRF chews up a lot of CPU and memory resources.  Since disabling it, we haven't had anymore problems.

---

Well, let me rephrase that:  AppRF chews up a lot of CPU and memory resources when you're on a flat network with everything on VLAN 1 like we are.  Each of our schools is on its own subnet with a router, but within the school, everything is on a single VLAN.  So the broadcast domain ends up being huge, and the APs see a lot of traffic, so AppRF gets very busy.

This might be related .....

 

AppRF
Bug ID Description
119416
121017
Symptom: An increase in memory utilization was observed when AppRF was enabled on an IAP. A
change in the IAP code has resolved the memory leak issue.
Scenario: This issue occurred when the IAP could not reach BrightCloud® for AppRF application and
web filtering operations. This issue was found in IAPs running 6.4.2.6-4.1.1.7 or earlier release
versions.
Table 5: AppRF Fixed Issue

Yeah, I saw that mentioned in one of the more recent code updates.  We've updated, but I left AppRF turned off, since we're not using it anyway.

 

Thanks!

Hi guys, We are having similar issues with AP225s. In a classeroom of around 21 students the AP shows high memory utilization when everyone log off. (I know this sounded crazy).

I already did the test and after the laptops are log off the AP disconected the teacher laptop and any other in use. After around 10 minutes the AP is back again to normal.

Any ideas?? AP225 version 6.4.2.8

---

@viruscan wrote:

Hi guys, We are having similar issues with AP225s. In a classeroom of around 21 students the AP shows high memory utilization when everyone log off.

---

Do you have AppRF turned on?  That turned out to be the culprit for us.  It was more noticable on the 205s because they have less memory and CPU resources than the 225s.  But you might see it on the 225s as well.  Try turning AppRF off and see if that makes a difference.

AppRF is not active... Today we are going to recreated the scenario and do some tests to see what is going on with this. The wierd thing is that this only happend when students log off and only affected AP memory uses.

viruscan,

 

Can you please open a TAC case?  You mention that users get disconnected, but memory might not be the problem:  You might also have bad RF.  Please open a TAC case in parallel and open a new thread.  This thread was about the IAP-205, not the controller-based AP205.

---

@viruscan wrote:

Hi guys, We are having similar issues with AP225s. In a classeroom of around 21 students the AP shows high memory utilization when everyone log off. (I know this sounded crazy).

I already did the test and after the laptops are log off the AP disconected the teacher laptop and any other in use. After around 10 minutes the AP is back again to normal.

Any ideas?? AP225 version 6.4.2.8

---

viruscan,

 

What is "high memory utilization"?  High memory is relative.  The question is what is the performance, and if you are having client issues. Also, you are missing some digits in the version of Instant; please let us know the rest of that...

 

Thanks for the fast replay!

The AP downloand the image from the AP Controller every time they reboot. All the APs get the same image. We are not using Instant APs, which work independently from controller (thick APs).

What is "high memory utilization"?  High memory is relative. CPU uses is ok it show low uses but the memory go up to 85% uses and then other users get disconnected.

The version I founded is this:

WLAN Controller Information
Controller Name	ArubaMaster
Controller Model	Aruba7240-US
Country	US
Hardware	Restricted US
ArubaOS Version	6.4.2.8

bnewall, Please PM me your ticket number....

Are u using IDS? protection ? detection? are u using APPRF? try disable thoese services and see if it's effect the cpuload - also , be sure to use the lastest instantOS

---

@kdisc98 wrote:
Are u using IDS? protection ? detection? are u using APPRF? try disable thoese services and see if it's effect the cpuload - also , be sure to use the lastest instantOS

---

We are not using IDS at all.  We do have firewall rules defined for our Guest network.  At the risk of sounding stupid, I don't know if we're using AppRF.  (Where would I go to check?)  I don't recall specifically turning it on, so unless it's on by default, we're probably not using it.

 

We were running version 6.4.2.3-4.1.1.2 when we first started having the DHCP issue.  I backed down to 6.4.2.0-4.1.1.1, thinking that might fix it, but it didn't.  I then backed down again, to 6.4.2.0-4.1.1.0, which is the earliest code that is supported by the 205.  That didn't help, either.  Right around when I did that last downgrade, I saw that 6.4.2.3-4.1.1.2 was pulled from the download site.  I see that 6.4.2.3-4.1.1.3 is out now, and it does make a mention of high CPU utilization under certain circumstances:

 

Bug ID     Description
113152    Symptom: A higher CPU utilization was observed in IAPs. This issue is resolved by restricting the
Wi-Fi drivers from sending messages pertaining to client roaming when fast roaming is not enabled
on an SSID.
Scenario: This issue occurred because the Wi-Fi drivers sent incorrect message requests to the
STM process, which in turn sent incorrect PAPI messages. The issue was observed in IAPs running
Instant 6.4.0.2-4.1.x.x releases.

 

I haven't tried updating to 6.4.2.3-4.1.1.3 yet, but I'm willing to give it a shot if it will potentially fix one or both of these problems.