Higher Education

Reply
This is an open group. Sign in and click the "Join Group" button to become a group member and start posting.
Highlighted
Occasional Contributor I
Posts: 17
Registered: ‎04-29-2009
Anyone running 6.2.1.3 code and seeing APs randomly loosing connection to controllers?

Hello all, we implemented new centralized 7210 controllers in late July and since the return of our residents this past week we are having issues where our APs (125 and 135s) in number (20-30) are loosing contact with our controllers randomly. Investigation with Aruba TAC seems to show high datapath CPU utilization on the controllers when the issue crops up, and appears so far to look like it may be a code issue. We are running 6.2.1.3 on both 7210 and 3600 model controllers, the 3600 was in service all last year on 6.1.x code without this high datapath CPU occuring but now the datapath cpu is spiking on it as well as the 7210s. Port utilizations look great (around 10% on Gb links) and no errors to speak of on the ports. Just hopeful we are not alone with this issue.

Thanks in advance for any feedback

mike

MVP
Posts: 4,301
Registered: ‎07-20-2011
Re: Anyone running 6.2.1.3 code and seeing APs randomly loosing connection to controllers?

 

We are currently running 6.2.1.3 with three 7240's.

 

We been experiencing high CPU utilization on the AP125's (when running the show ap debug system-status ap-name <ap-name> even if those have no clients but I haven't seen the same behavior with AP105's or 135's

 

I currently have a case open.

 

Would like to know what you guys find out with TAC.

Thank you

Victor Fabian
Lead Mobility Engineer @ Integration Partners
AMFX | ACMX | ACDX | ACCX | CWAP | CWDP | CWNA
Regular Contributor I
Posts: 241
Registered: ‎04-03-2007
Re: Anyone running 6.2.1.3 code and seeing APs randomly loosing connection to controllers?

Yes please post any resolution of word from TAC. We are running 6.2.1.2 and are experienceing many AP reboots per day. The reason for reboot is given as out of memory. After reviewing our case the TAC recommended upgrading to 6.2.1.3 which resolves to known memory exhaustion issue.

 

Hoping not to trade one issue for another..

 

 Mike

Occasional Contributor I
Posts: 17
Registered: ‎04-29-2009
Re: Anyone running 6.2.1.3 code and seeing APs randomly loosing connection to controllers?

Hello All:

We have some feedback from TAC: the CPUs for datapath (show datapath utilization on controllers) was showing at least one CPU spiking to 100% when our APs were rebooting or bootstrapping and bouncing between master and backup LMS controllers. We had enabled Broadcast/Multicast (BCMC) Optimization on the SSID profiles, but apparently there is also the option to turn on BCMC Optimization on the VLAN interface as well (thank you Princeton for getting this feature I understand). This solved our issue, however, as we are a heavy iOS (iPad) environment the consequence of enabling BCMC on the VLAN interface is now the AirPrint and other AirGroup traffic will not work (we use PaperCut on a MAC server to allow our iPad users to print to campus printers). So we are between the proverbial Rock and a Hard Place until we can solve the BCMC issue on the VLAN the controllers and APs are located.

 

In three years with Aruba controllers/APs we have never had to use the BCMC Optimization on the VLAN interface, so we have either reached some threshold of BC traffic or we were concerned that we went from 6.1.3.x code to the 6.2 code may have caused the traffic we had to now be an issue for the controllers. So for you folks with APs dropping check periodically the datapath utilization, check the ap bss-table total times for APs to make sure they match closely the UP time of the AP (looking for APs bootstrapping), as well as the system log for any instances of heartbeat misses.

Hope this is helpful to someone, I'll be staring at packet traces for awhile to see if we can fully identify and resolve this errant BC/MC traffic

 

Mike

MVP
Posts: 501
Registered: ‎04-03-2007
Re: Anyone running 6.2.1.3 code and seeing APs randomly loosing connection to controllers?
How many APs are in your network that sees bouncing?
==========
Ryan Holland, ACDX #1 ACMX #1
The Ohio State University
Occasional Contributor I
Posts: 17
Registered: ‎04-29-2009
Re: Anyone running 6.2.1.3 code and seeing APs randomly loosing connection to controllers?

For us, we have 2 local controllers, one for dorms (7210) and one for remainder of campus (3600 curently, another 7210 will be deployed soon but we are leaving it in place for now just to see if we had a specific issue with the newer controller model). Dorm controller has about 84 APs (mostly 125s), the other has 104 APs (mix of 125 and 135s). We would have about 50% of the APs on either controller bounce to their backup (our 1 master controller), and at the time of the issue (datapath utilization at 100%), and the ping response would either get very long (1.5-2.5 seconds) or not repsond at all to the controllers (but they would ping our gateways just fine, under 2 ms). Typically we would see the datapath utililzation spike for about 3-5 minutes and then return to normal for an hour or more, then another episode would come again. Also, we had almost no case where both local controllers were experiencing the problem at the same time, which led us to believe it might be user traffic vs the LAN traffic on the VLAN interface, but we are told the BC/MC Optimization on the VLAN Interface on the controllers is operating on the inbound traffic from the LAN into the controllers.

 

Mike

MVP
Posts: 501
Registered: ‎04-03-2007
Re: Anyone running 6.2.1.3 code and seeing APs randomly loosing connection to controllers?
Wow. Ok. I was anticipating you saying 100s of APs. What type of broadcast traffic is coming into the controller during these bouncing periods? I've seen (MANY times) where a loop within datapath processing results in a CPU spike. I would not be surprised if there was a specific frame/packet that was mis-processed resulting in this issue. . . especially since it seems related to the code you are running.
==========
Ryan Holland, ACDX #1 ACMX #1
The Ohio State University
Guru Elite
Posts: 8,643
Registered: ‎09-08-2010
Re: Anyone running 6.2.1.3 code and seeing APs randomly loosing connection to controllers?
[ Edited ]

<removed> already asked


Tim Cappalli | Aruba Security TME
@timcappalli | timcappalli.me | ACMX #367 / ACCX #480
Contributor II
Posts: 59
Registered: ‎02-22-2011
Re: Anyone running 6.2.1.3 code and seeing APs randomly loosing connection to controllers?

We upgrade to code 6.2.1.2 and starting have 30 - 40 AP-125s reboot every day, so we upgrade to 6.2.1.3 and that resolved the problem of the APs rebooting. Now we have users complaining about slow wireless performance, which we have never really had before. I'm not sure if it's a code problem, but from 3.x to 6.1 we have had almost no complaints of wireless performance problems. 

 

Bob 

 

Occasional Contributor I
Posts: 17
Registered: ‎04-29-2009
Re: Anyone running 6.2.1.3 code and seeing APs randomly loosing connection to controllers?

Ryan:

I guess that is the $64k question (what BC/MC traffic is it that is causing the issue), and what we are trying to determine as quickly as we can. I am hoping we are not the only ones seeing this issue with these controllers/code version, our AP and user population is reltaively small with under 200 APs and about 3k total clients, so with much larger organizations out there using Aruba gear I was hoping not to be the only one experiencing this. Off to do more sniffing (looks like I picked the wrong week to stop sniffing glue).

Mike

Search Airheads
Showing results for 
Search instead for 
Did you mean: