Wireless Access

We are seeing High CPU on the controller randomly ( 2or 3 minutes) and effecting the connectivty of the users and performance

Do we have any inputs what are the CPU Queues 25 ,13, 17 and 23 from the show datapath utilization?

Do we have any document speaking the about different queues (like STM - a brief descirption about it) of CPU 8 to 31 of the command output show datapath utilization?

We did try changing snmp interval and moving to amon. BC/MC optimization on VAP and disabling ipv6 etc on the controller.

I diid attach output of the log.

Attachment(s)

ouptut.txt   3 KB 1 version

You could have spikes up to 80% with no material affect on user performance.  If the 80% is not sustained, it might not be an issue.  That message might or might not be important.  If you PM me, I can send you a link so that I can take a look at your logs.tar and see if anything seems wrong.

When you say "connectivity and performance issue", what do you mean?

Sure Colin. at the that specific moment when see the High CPU users loose connectivity and having trouble in connecting. if they are connected they have packet loss.

Have you already opened a TAC case for this behavior?

I personally would open a TAC case, as 99% actual is very unusual. TAC can look at the individual system logs from the debug tar logs and see what within fpapps is causing the issue. 

i did open a case with TAC for the same seeing the CPU values going above 95% but trying to understand anything i can do by knowing the CPU queues to narrow done as its also effecting production.

You should direct those questions to TAC, because they can see the full picture.

ok Thanks Colin. Will work with TAC on the same . I was just trying see if anyone had seen any similar issues due to any specific traffic or anyother things etc.

I wish we did know some of those more detailed investigative tips and tricks, but unfortunately most of us field facing guys are grunts. It's the guys in TAC that have the much needed deeper understanding of the CPU and control plane and usually they've seen previous cases to know what to look for right off the bat. Definatley though, as TAC how to narrow down/detail out CPU resource issues, most of the time they are happy to explain what to look for. Good luck!

It is a good idea to post here to see if anyone has seen similar symptoms.  In this case, you have quite a few things that could cause what you are seeing.  Some of those things are known and in your case, it might be something that has never been seen before...

We are still working with TAC on the same as it is Random Spike in CPU. Will keep you all posted if any good news

Did you ever find resolution to those datapath spikes? Did TAC share a workflow for you to determine what traffic was causing the spikes?

Hi Ryan,

Hi Ryan,

We've been experiencing high CPU spikes and loss of pings to our 7240XM controllers running 8.10.0.14 in a cluster of 12 XMs. The Aruba ACE team has already reviewed our wireless network and provided recommendations, which we have implemented. Things seemed to improve before the Christmas break when we were still on 8.10.0.12. However, after updating to 8.10.0.14 over the break, we started losing pings again to random controllers during class changes on campus.

Our wired Cisco network, on the other hand, remains clean with no issues. Any insights you may have on this matter would be greatly appreciated. We are actively monitoring the situation, and I have opened a new TAC case where we are documenting our findings.

Stay tuned.

I posted a message about our high CPU spikes during roaming at https://community.arubanetworks.com/discussion/after-students-come-back-stm-process-now-starved-for-cpu-and-associations-failing-at-a-very-high-percentage-rate