Security

Hi anyone,

 

right now I'm facing a problem that the policy server sertvice is crashing right after an AV update.

Is there a way to download old AV versions and import them?

it looks like a worldwide issue. i am currently online with TAC, they are telling me that it is a issue for more customers. seems like the av update indeed

Same here => I will try to check other customers...

I just give a try to roll back an old av update

The problem seems to start in:

 

AntiVirus & AntiSpyware Updates version: 1.48744

 

The error in event viewer is:
Source SYSTEM
Level ERROR
Category cpass-policy-server restarted
Action None
Timestamp Oct 11, 2017 10:14:41 CEST
Description
System monitor restarted cpass-policy-server, as it seemed to have stopped abruptly.

I wasn't lucky to downgrade the av/as updates.

I can confirm, it starts with 1.48744

Same here...as of 10:14 GMT+1 my Policy Server keeps crashing

a new update with revision ...45 is available, but it seems that the problem is still there

Theres a .46 already but no luck

This doesn't seem to effect Clearpass installations at 6.6.5 or below..

I confirm, our customer running 6.6.5 or below doesn't have the problem.  The ones running higher (not yet under 6.6.8) have it.

We are on  Version  6.6.7 and I can confirm, that policy server crashes.

:-(

We can confirm as well only 6.6.7 and 6.6.8 are affected; we have numerous customers affected currently.

i just started an implemantation,

did a restore db and then blocked internet access to avoid that update.

me too..  no assistance because all the engineers are too busy.  My servers have no policy running, as you can imagine it is a bit of a disaster for our network.

 

I saw the same issue on the logs.

 

hope it gets fixed soon.

Please hit save on your software update page and get the latest AV update. Should be now at 1.48751. start service if not already started. 

I confirm it.

 

Thank you

Everythings working again

1.48751 installed, and it is working again.

Just got the Service back. Not sure if there willl be an RCA that will get shared. 

 

This is what I found: 

Starting policy server at Wed Oct 11 04:51:00 EDT 2017:
log4cxx: Could not instantiate class [DOMConfigurator].
log4cxx: Class not found: DOMConfigurator
log4cxx: Could not instantiate configurator [DOMConfigurator].
log4cxx: No appender could be found for logger (Common.GlobalConfig).
log4cxx: Please initialize the log4cxx system properly.
Not in FIPS mode
terminate called after throwing an instance of 'std::out_of_range'
what(): map::at

Hi,

 

There is a new update (1.48751) for AV

and restart Policy Server and working :)

 

right now nearly 10k wireless and wired clients from all customers...

 

UPDATE: it looks like that update 1.48751 is running again

Confirmed, 1.48751 on 6.6.8.100017 followed by starting policy server on pub/sub seems like it's stabilised again.

Sigh of relief, the new AV update did the trick, things are working again.

 

Confirmed.

Hi, how would I invoke the AV update from the CLI in order to force it?

Just click 2 times.. Then go start your Policy service

Hi all,

 

my conclusion of this is to disable this auto update feature. Most of my customers do not use posture / onguard at the moment.

I know it was the first time this was happening but the business impact was/is to high.

 

The update of the policy server can be done by hand by downloading the new software / hotfixes and install them manualy.

 

And once again: thanks for the great community

How to disable the auto-update feature?

the quick and dirty option is to block the internet access (firewall / fake proxy)

I don't know if there is an option that disable the auto upte service?!

Cluster-wide parameters > General > "Auto check for available software updates"

But if I then click 'check status now' in the software update category, will the AV/AS updates then be installed automatically nevertheless?

Or is this limited to the firmware & patch updates?

 

update 1.48751 fixes the issue for me as well!

As for now, until it can be fully investigated you can remove the subscription ID.

Same issue.. Policy server service is getting crashed and we are using 6.6.7 version

Not sure it is due to the version 1.48744 as I have customer running it without any problems but have a customer having having problem and using it...

 

Customer not having the problem with this version of AV is under CPPM 6.6.5.

Customer having the problem with this version of AV is under CPPM 6.6.7.

 

Same problem here. Nothing like an A/V update to knock out wired and wireless authentication for thousands of endpoints across the country at once. Only saving grace is it's out-of-hours here in Australia and we're not running our global sites on this yet. Going to be interesting to see how Aruba communicate to all their CPPM customers how to fix this one.

 

Brett

I can confirm we are not seeing the problem in 6.6.5

We are seeing it on 6.6.8 pub and subscriber. Funny thing is we only just upgraded from 6.6.2 this morning, finally scheduled the upgrade on the day an update killed it.

I can confirm it as well. I don't see this issue with an 6.6.5

this is baaaad... Not possible to install older AV signatures it seems.. So now just hoping for the Engineering team to realise their mistake and update ASAP!

Same problem with 2 customer installations, on 6.6.7 and 6.6.8

I have 7 fully loaded 25k appliances down - does anyone know if restoring a backup, or manually loading a previous version of software would correct this?

I am having the same problem as well 400  + users screaming at me with no Wi-Fi

 

version 6.6.8.100017

See note above

Working for me now , can keep my job

wow, that was something new, and bad. I thought these things usually are tested PRIOR release.

 

The benefit of clearpass cluster just went partially down to toilet.
Now we are considering to isolate Clearpasses behind firewall to prevent AV/AS autoupdates. +20k pissed 802.1x users demands explanations and so far the best I can come up with is "we are currently browsing options".

It would have been nice for Aruba to let us know about the issue. Some people have already taken steps to restore backups in an attempt to get their systems working. I still also haven't seen any appology ? 

Tens of thousands of pounds for a system that can completely crash your entire infrastructure. This has been the biggest network outage we have had on site ever.

This forum has worked well in this issue and this fix was relatively swift.

 

We've had multiple hospital customers declare a major incident due to the impact this had to their clinical operations.

 

This has highlighted the requirement to change the recommended deployment options for Clearpass. 

 

 

We're working again too, but it does seem the CLI has a limit feature set :-\

Answer from HPE Aruba support:

 

ClearPass Team releases Posture and Profile Data Updates on an hourly basis.
 
The Posture and Profile Data update version 1.48743 which was released today had caused the Policy Service to crash causing authentication issues.
 
A defect RM42553 has been created for this issue.
 
The Dev Team has released an update 1.48751 which has resolved the issue.
 
Please ensure that the update 1.48751 is installed and the Policy Service is running on all the servers in the cluster, by following the below stated steps.
 
·         To install AV/AS Update version 1.48751, Please navigate to ClearPass Policy Manager GUI -> Administration-> Agents and Software Updates -> Software Updates page -> Click on 'Check Status Now".
 
·         Please navigate to ClearPass Policy Manager GUI -> Administration -> Server Manager -> Server Configuration -> Click on the name of the server -> Services Control -> Check for the status of the Policy server.
 
·         If the status is Stopped, please click on the Start button next to it, to start the service.

"A defect RM42553 has been created for this issue."

 

What does this mean?  Is this an internal reference number?

 

This was a massive outage for us with our Clearpass cluster seeing 40000+ endpoints per day :-(

Yes, it's an internal bug ID.

 

Its surprising how there has been no official statement from Aruba since the issues. It has been hours. Poor communication made things really bad, we couldnt get hold of support as they probably got inundated.

 

Thank god for Airheads as this was our only source of information for the entire period.

We have been constantly posting updates here. This is where we post updates.

 

When a RCA is complete, a public statement will be released.

Hey Tim

I think if you look back through this thread, you'll find the first official communication from Aruba (that I can tell) was when the problem was solved. That's two hours of silence and an inability to get ahold of anyone at the support centre with outages that affected hundreds of thousands of users. Don't get me wrong, I'm a huge fan and advocate for Aruba and specifically Clearpass but I have to agree with others, the lack of communication was disappointing.

Brett