Security

Reply
Contributor I
Posts: 23
Registered: ‎10-07-2014

clearpass server crashing - BUG: SOFT LOCKUP - CPU#2 STUCK FOR 24s! [policy_server:16529]

Hi Everybody,

 

Has anyone experienced a ClearPass Publisher server crashing after a few hours of operations with the following messages displayed at console:

clearpass-bug-soft-lockup.jpg

The server has been reinstalled to expand disk capacity (it´s a clean deployment of CP-VA-25k, version 6.5.6, followed by a restore, under ESXi 6.0 update 2). The server stops responding in such a way that the only possible recovery is to power off the ClearPass VM in vCenter and power on again!

This weekend we had 4 lockups, and after the 2nd, I updated vmware tools to the most recent version, but the server continnued to crash.

Any ideas?

Thanks,

 

Heraldo.

Guru Elite
Posts: 7,852
Registered: ‎09-08-2010

Re: clearpass server crashing - BUG: SOFT LOCKUP - CPU#2 STUCK FOR 24s! [policy_server:16529]

It's best to open a TAC case for this.

Tim Cappalli | Aruba ClearPass TME
@timcappalli | ACMX #367 / ACCX #480 / ACEAP / CWSP
Contributor I
Posts: 23
Registered: ‎10-07-2014

Re: clearpass server crashing - BUG: SOFT LOCKUP - CPU#2 STUCK FOR 24s! [policy_server:16529]

Hi Cappalli,

Thanks for the response!

 

MVP
Posts: 750
Registered: ‎04-13-2009

Re: clearpass server crashing - BUG: SOFT LOCKUP - CPU#2 STUCK FOR 24s! [policy_server:16529]

What resources have been made available to the Clearpass VM? I had a similar issue in my lab when my estimate server was way oversubscribed.
Cheers
James

-------------------------------------------------------
-------------------@whereisjrw-------------------
------------------------blog-------------------------
ACCX #540 | ACMX #353 | ACDX #216
-----------Mobility First Expert #11----------
-------------------------------------------------------

If a reply adequately addresses your issue, please click on the "Accept as Solution" and "Give Kudos" button so this information can benefit other users via search.
Contributor I
Posts: 23
Registered: ‎10-07-2014

Re: clearpass server crashing - BUG: SOFT LOCKUP - CPU#2 STUCK FOR 24s! [policy_server:16529]

Hi Jrwhitehead,

Thanks for the response.

Clearpass VM was deployed with recommended settings (64GB ram, 2.18TB disk space, 12 virtual processors - our dell server has two 6 cores processors).

This weekend we had 4 crashes. What I did was after the 2nd crash was update vmware tools to the most recent version, but we still had 2 others crashes yesterday. This morning, before I restart the server, I also updated vmware compatibility to ESXi 6.0 or later (VM version 11). After the OVF deployment, compatibility was ESXi 5.0 or later (VM versoin 8). Maybe this update made some difference because so far the server is up and running as expected, no crashes or messages on the console since the reboot this morning. Fingers crossed to be only this!

You said your lab server was oversubscribed... What exactly was oversubscribed?

Thanks,

 

Aruba Employee
Posts: 370
Registered: ‎11-04-2011

Re: clearpass server crashing - BUG: SOFT LOCKUP - CPU#2 STUCK FOR 24s! [policy_server:16529]

You pretty sure have issues with your underlying hardware. These soft lockups indicate that the hardware cannot handle the load. It may be that you are running other VMs on the same hardware.

 

Check this: http://ubuntuforums.org/showthread.php?t=2205211 (solution was replacing the power supply of the computer)

Or this: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1009996 (I don't agree with the workaround as if you see these soft lockups performance of your ClearPass will be very poor)

Or this: http://unix.stackexchange.com/questions/70377/bug-soft-lockup-cpu-stuck-for-x-seconds

 

From the information that I have now, I would thoroughly check the hardware and the VMWare ESXi information. May be ESXi gives you warnings/errors... might any of your harddisks be bad (resulting in hughe disk io delays)? After you validated that the hardware does meet the ClearPass system requirements, can you replace hardware components? Like harddisk, power supply, whole server?

 

And yes, open a TAC case as well in parallel... however the messages come from the ClearPass kernel (which is the component closest to the hardware) and indicate issues/delays with the hardware it is installed on.

--
If you have urgent issues, please contact your Aruba partner or Aruba TAC.
MVP
Posts: 750
Registered: ‎04-13-2009

Re: clearpass server crashing - BUG: SOFT LOCKUP - CPU#2 STUCK FOR 24s! [policy_server:16529]

My CPUs & RAM was oversubscribed... HP Microservers are great for size and noise levels but sadly only support 16GB of RAM.. Anyway.. I'm interested to hear what TAC say.

 

Is Airwave supported on VM version 11?

Cheers
James

-------------------------------------------------------
-------------------@whereisjrw-------------------
------------------------blog-------------------------
ACCX #540 | ACMX #353 | ACDX #216
-----------Mobility First Expert #11----------
-------------------------------------------------------

If a reply adequately addresses your issue, please click on the "Accept as Solution" and "Give Kudos" button so this information can benefit other users via search.
Aruba Employee
Posts: 370
Registered: ‎11-04-2011

Re: clearpass server crashing - BUG: SOFT LOCKUP - CPU#2 STUCK FOR 24s! [policy_server:16529]

I have not seen issues with Airwave on VMware ESXi 6.0 and virtual hardware version 11.

 

In case you need a definitive answer, please ask Aruba TAC.

--
If you have urgent issues, please contact your Aruba partner or Aruba TAC.
Contributor I
Posts: 23
Registered: ‎10-07-2014

Re: clearpass server crashing - BUG: SOFT LOCKUP - CPU#2 STUCK FOR 24s! [policy_server:16529]

Hi Herman,

Thanks for the response and information!

Our ClearPass VM is running on  a dedicated server. No other VM´s on the same server. We have beem using this server for a long time to run ClearPass VM. Last week we expanded the disk capacity of this server and to do this we redeployed ClearPass OVF file, updated to 6.5.6 and restored the databases. The disks on this server are all new now. Before this disk expansion, we´ve never seen this issue. It started after ClearPass reinstallation. We also upgraded ESXi from 5.5 to 6.0 update 2, and maybe I think the issue has something to do with VM hardware compatibility. Just after redeployment, ClearPass VM compatibility was ESXi 5.0 or later (VM version 8). Yesterday morning, we updated VM compatibility  to ESXi 6.0 or later (VM version 11) and since the last reboot after this VM compatibility update, ClearPass is up and running, no crashes or soft lockup messages on the console.

If the messages were coming from ClearPass kernel, do you think that upgrading VM compatibility could have solved the issue?

Thanks again!

 

Aruba Employee
Posts: 370
Registered: ‎11-04-2011

Re: clearpass server crashing - BUG: SOFT LOCKUP - CPU#2 STUCK FOR 24s! [policy_server:16529]

Not sure if the upgrades solved the issue, but it can be well possible. ClearPass includes the VM tools that communicate between ESXi and the virtual machine. Both the virtual hardware as the VM tools interface change during an ESXi upgrade. I have ran ClearPass on ESXi 5.5 for long time, never seen it; I consider it more likely that the hardware you run ESXi on is better supported int ESXi 6.0; and that may be a good reason that the issue was resolved.

 

Good to hear that you were able to fix this issue.

--
If you have urgent issues, please contact your Aruba partner or Aruba TAC.
Search Airheads
Showing results for 
Search instead for 
Did you mean: