Security

For most my customers we've deactivated the automatic update of "Posture Signature and Windows Hotfixes" due to the problem caused when one of those upddtes killed Clearpass a few years back. This has also been default FALSE since a certain release after THE INCIDENT.

Today I had one customer contact me which had this activated. At 13pm CET their Clearpass received a Posture update, and then Clearpass stopped responding to requests. Reboot had no effect. At roughly 2:30pm CET a new update was received which caused Clearpass to magically awaken and accept requests again.

Anyone else experienced this?

------------------------------
John-Egil Solberg |
ACMX | ACCX
------------------------------

Did the customer hit broadcast notifications under Monitoring > Live Monitoring > OnGuard Activity by any chance? We've some updates around this feature that's release noted in 6.8.9 that got released yesterday.


Original Message:
Sent: Mar 25, 2021 09:46 AM
From: John Solberg
Subject: Ooops, they did it again? Posture update killed Clearpass..

For most my customers we've deactivated the automatic update of "Posture Signature and Windows Hotfixes" due to the problem caused when one of those upddtes killed Clearpass a few years back. This has also been default FALSE since a certain release after THE INCIDENT.

Today I had one customer contact me which had this activated. At 13pm CET their Clearpass received a Posture update, and then Clearpass stopped responding to requests. Reboot had no effect. At roughly 2:30pm CET a new update was received which caused Clearpass to magically awaken and accept requests again.

Anyone else experienced this?

------------------------------
John-Egil Solberg |
ACMX | ACCX
------------------------------

We have not had other reports of this behavior and the signature failures would be something that impact a large customer base (not as bad as the October 2017 "INCIDENT", but a lot none the less).  If the customer is still using a version below 6.7.0 it is possible that there was a signature that was partly problematic for the system, but the system behavior was changed significantly for the 6.7 release to prevent this from occurring ever again.  

In addition to installing additional levels of internal checks on the signature releases we also modified the 6.7 code to pre-validate the signature before it loaded.  If the system failed the validation it will not actually load the signature.  In the event that the node does get an invalid signature installed, it is re-checked before the signature is replicated to other cluster nodes and will then fail-back to a safe signature to restore operation of the node.

I recommend collecting the logs and opening a TAC case quickly to try to determine the cause of the issue on this node.

------------------------------
Bryan Lechner
------------------------------

Original Message:
Sent: Mar 25, 2021 09:46 AM
From: John Solberg
Subject: Ooops, they did it again? Posture update killed Clearpass..

For most my customers we've deactivated the automatic update of "Posture Signature and Windows Hotfixes" due to the problem caused when one of those upddtes killed Clearpass a few years back. This has also been default FALSE since a certain release after THE INCIDENT.

Today I had one customer contact me which had this activated. At 13pm CET their Clearpass received a Posture update, and then Clearpass stopped responding to requests. Reboot had no effect. At roughly 2:30pm CET a new update was received which caused Clearpass to magically awaken and accept requests again.

Anyone else experienced this?

------------------------------
John-Egil Solberg |
ACMX | ACCX
------------------------------

I don't know the rootcause, but it was decided to upgrade from 6.8.8 to 6.8.9 - and that solved the issue..

During the problem we saw very few Radius entries in Access Tracker. Webauth worked fine.
Added packet dumps on the incoming and outgoing Radius traffic. We saw loads Radius coming in, but few Radius coming back from the Clearpass servers.
* A restart of the Radius service kickstarted so that it worked for a minute or two, then stopped again.
* Restarting Policy Service - same as above
* Reboot of the subscribers - same as above

AND - Oddly enough - changing the Auto posture and anti-virus profile update from False to True and visa-versa kickstarted it again so that it worked for a minute or two, then stopped again. Do you guys know which processes are restarted when this is done?

------------------------------
John-Egil Solberg |
ACMX | ACCX
------------------------------

Original Message:
Sent: Mar 25, 2021 06:13 PM
From: Bryan Lechner
Subject: Ooops, they did it again? Posture update killed Clearpass..

We have not had other reports of this behavior and the signature failures would be something that impact a large customer base (not as bad as the October 2017 "INCIDENT", but a lot none the less).  If the customer is still using a version below 6.7.0 it is possible that there was a signature that was partly problematic for the system, but the system behavior was changed significantly for the 6.7 release to prevent this from occurring ever again.  

In addition to installing additional levels of internal checks on the signature releases we also modified the 6.7 code to pre-validate the signature before it loaded.  If the system failed the validation it will not actually load the signature.  In the event that the node does get an invalid signature installed, it is re-checked before the signature is replicated to other cluster nodes and will then fail-back to a safe signature to restore operation of the node.

I recommend collecting the logs and opening a TAC case quickly to try to determine the cause of the issue on this node.

------------------------------
Bryan Lechner

Original Message:
Sent: Mar 25, 2021 09:46 AM
From: John Solberg
Subject: Ooops, they did it again? Posture update killed Clearpass..

For most my customers we've deactivated the automatic update of "Posture Signature and Windows Hotfixes" due to the problem caused when one of those upddtes killed Clearpass a few years back. This has also been default FALSE since a certain release after THE INCIDENT.

Today I had one customer contact me which had this activated. At 13pm CET their Clearpass received a Posture update, and then Clearpass stopped responding to requests. Reboot had no effect. At roughly 2:30pm CET a new update was received which caused Clearpass to magically awaken and accept requests again.

Anyone else experienced this?

------------------------------
John-Egil Solberg |
ACMX | ACCX
------------------------------

There are a number of things that are restarted in the processes above.  This is definitely something that is not related to the signatures then.  When the signatures were bad, the RADIUS service and the webauth would not have worked either, but both were actually running (the problem was that the policy engine itself was then stopped). 

You having RADIUS failures is something entirely different that should probably still be run through TAC to try to find the problem.  I don't want to have have this come back even after the upgrade because there is something else incorrect in the system.

------------------------------
Bryan Lechner
------------------------------

Original Message:
Sent: Mar 26, 2021 04:04 AM
From: John Solberg
Subject: Ooops, they did it again? Posture update killed Clearpass..

I don't know the rootcause, but it was decided to upgrade from 6.8.8 to 6.8.9 - and that solved the issue..

During the problem we saw very few Radius entries in Access Tracker. Webauth worked fine.
Added packet dumps on the incoming and outgoing Radius traffic. We saw loads Radius coming in, but few Radius coming back from the Clearpass servers.
* A restart of the Radius service kickstarted so that it worked for a minute or two, then stopped again.
* Restarting Policy Service - same as above
* Reboot of the subscribers - same as above

AND - Oddly enough - changing the Auto posture and anti-virus profile update from False to True and visa-versa kickstarted it again so that it worked for a minute or two, then stopped again. Do you guys know which processes are restarted when this is done?

------------------------------
John-Egil Solberg |
ACMX | ACCX

Original Message:
Sent: Mar 25, 2021 06:13 PM
From: Bryan Lechner
Subject: Ooops, they did it again? Posture update killed Clearpass..

We have not had other reports of this behavior and the signature failures would be something that impact a large customer base (not as bad as the October 2017 "INCIDENT", but a lot none the less).  If the customer is still using a version below 6.7.0 it is possible that there was a signature that was partly problematic for the system, but the system behavior was changed significantly for the 6.7 release to prevent this from occurring ever again.  

In addition to installing additional levels of internal checks on the signature releases we also modified the 6.7 code to pre-validate the signature before it loaded.  If the system failed the validation it will not actually load the signature.  In the event that the node does get an invalid signature installed, it is re-checked before the signature is replicated to other cluster nodes and will then fail-back to a safe signature to restore operation of the node.

I recommend collecting the logs and opening a TAC case quickly to try to determine the cause of the issue on this node.

------------------------------
Bryan Lechner

Original Message:
Sent: Mar 25, 2021 09:46 AM
From: John Solberg
Subject: Ooops, they did it again? Posture update killed Clearpass..

For most my customers we've deactivated the automatic update of "Posture Signature and Windows Hotfixes" due to the problem caused when one of those upddtes killed Clearpass a few years back. This has also been default FALSE since a certain release after THE INCIDENT.

Today I had one customer contact me which had this activated. At 13pm CET their Clearpass received a Posture update, and then Clearpass stopped responding to requests. Reboot had no effect. At roughly 2:30pm CET a new update was received which caused Clearpass to magically awaken and accept requests again.

Anyone else experienced this?

------------------------------
John-Egil Solberg |
ACMX | ACCX
------------------------------