Security

Hello all,

While I wait for TAC to get going on this, I thought I would poll and see if anyone has seen this previously.

I have a 4 node cluster authenticating MSCHAPv2 to and AD controller. (Yes, yes, deprecated, TLS, I know. Once I figure out how to pay for 10k certs for student devices I'll take a run at it.)

I am seeing approximately 20% of all wireless authentications timing out with a 9002 error. The timeouts are more or less equally distributed by service, location, controller (including eduroam users at other sites), device type, etc. Over a 24 hour period ALL clients experienced at least one timeout but successfully connected later on a retry.

The 20% failure rate is a constant. Overnight at 4 auths/second or daytime at 100, it's the same ratio. And they come in a batch, generally close to 8 at a time as seen in access tracker..

I've tried changing the AD controller being used, confirmed everything is on SSD storage. Nothing in the event logs on clearpass or in AD.

Has anyone seen anything like this before?

What is your WLAN hardware and what software version is running?  Version of CPPM?

Hello all,

While I wait for TAC to get going on this, I thought I would poll and see if anyone has seen this previously.

I have a 4 node cluster authenticating MSCHAPv2 to and AD controller. (Yes, yes, deprecated, TLS, I know. Once I figure out how to pay for 10k certs for student devices I'll take a run at it.)

I am seeing approximately 20% of all wireless authentications timing out with a 9002 error. The timeouts are more or less equally distributed by service, location, controller (including eduroam users at other sites), device type, etc. Over a 24 hour period ALL clients experienced at least one timeout but successfully connected later on a retry.

The 20% failure rate is a constant. Overnight at 4 auths/second or daytime at 100, it's the same ratio. And they come in a batch, generally close to 8 at a time as seen in access tracker..

I've tried changing the AD controller being used, confirmed everything is on SSD storage. Nothing in the event logs on clearpass or in AD.

Has anyone seen anything like this before?

My controllers are all 72xx running 8.10.13. But I have no idea what the controllers are that are being used at remote eduroam locations. 

CPPM is 6.10.8 + current patches.

TAC is having fun with this. So far we've decided that the AD environment and clearpass are OK, and I've been escalated to the controller team. 

What is your WLAN hardware and what software version is running?  Version of CPPM?

Hello all,

While I wait for TAC to get going on this, I thought I would poll and see if anyone has seen this previously.

I have a 4 node cluster authenticating MSCHAPv2 to and AD controller. (Yes, yes, deprecated, TLS, I know. Once I figure out how to pay for 10k certs for student devices I'll take a run at it.)

I am seeing approximately 20% of all wireless authentications timing out with a 9002 error. The timeouts are more or less equally distributed by service, location, controller (including eduroam users at other sites), device type, etc. Over a 24 hour period ALL clients experienced at least one timeout but successfully connected later on a retry.

The 20% failure rate is a constant. Overnight at 4 auths/second or daytime at 100, it's the same ratio. And they come in a batch, generally close to 8 at a time as seen in access tracker..

I've tried changing the AD controller being used, confirmed everything is on SSD storage. Nothing in the event logs on clearpass or in AD.

Has anyone seen anything like this before?

If you deployed PEAP without provisioning tool for your clients, this is expected. The 9002 means there is a timeout on the authentication, and in most cases this is a client configuration problem. Clients that are not (pre) provisioned to the SSID, accept the server certificate, are likely to prompt the end-user to accept the connection or not. If the end-user doesn't see the authentication popup, or is slow in responding, you will see a timeout.

If this is eduroam, the use of geteduroam or CAT tool can probably resolve the issue by correctly configuring your clients. Geteduroam may even solve your TLS client certificate challenge, depending on where you are located.

Please let us know if you were able to resolve the issue, and how.

My controllers are all 72xx running 8.10.13. But I have no idea what the controllers are that are being used at remote eduroam locations. 

CPPM is 6.10.8 + current patches.

TAC is having fun with this. So far we've decided that the AD environment and clearpass are OK, and I've been escalated to the controller team. 

What is your WLAN hardware and what software version is running?  Version of CPPM?

Hello all,

While I wait for TAC to get going on this, I thought I would poll and see if anyone has seen this previously.

I have a 4 node cluster authenticating MSCHAPv2 to and AD controller. (Yes, yes, deprecated, TLS, I know. Once I figure out how to pay for 10k certs for student devices I'll take a run at it.)

I am seeing approximately 20% of all wireless authentications timing out with a 9002 error. The timeouts are more or less equally distributed by service, location, controller (including eduroam users at other sites), device type, etc. Over a 24 hour period ALL clients experienced at least one timeout but successfully connected later on a retry.

The 20% failure rate is a constant. Overnight at 4 auths/second or daytime at 100, it's the same ratio. And they come in a batch, generally close to 8 at a time as seen in access tracker..

I've tried changing the AD controller being used, confirmed everything is on SSD storage. Nothing in the event logs on clearpass or in AD.

Has anyone seen anything like this before?

I am told this is usually a client issue and it not accepting the certificate.  I have also been told it is users in areas with weak signals walking out of the coverage area.  This I can understand more, but not at the levels we see (also about 20%).  Authentication takes less that a second and users do not usually move that fast.

We also see it on managed machines that are provisioned for TLS that still timeout.  The latest is our Cape Sensors started having the same problem recently. We know that they can connect (as they do not fail all of the time), they are not moving out of coverage.  So something else is going on.  We have the problem with 8 different Cape Sensors on Three different Controller clusters and fail  sometimes on all 7 of the clearpass servers we are currently running.

Yes, I will be opening a TAC case but it is hard when his is random and you are trying to get the perfect alignment of the planets so you can get data from the controller, radius server and packet dumps. 

If you deployed PEAP without provisioning tool for your clients, this is expected. The 9002 means there is a timeout on the authentication, and in most cases this is a client configuration problem. Clients that are not (pre) provisioned to the SSID, accept the server certificate, are likely to prompt the end-user to accept the connection or not. If the end-user doesn't see the authentication popup, or is slow in responding, you will see a timeout.

If this is eduroam, the use of geteduroam or CAT tool can probably resolve the issue by correctly configuring your clients. Geteduroam may even solve your TLS client certificate challenge, depending on where you are located.

Please let us know if you were able to resolve the issue, and how.

My controllers are all 72xx running 8.10.13. But I have no idea what the controllers are that are being used at remote eduroam locations. 

CPPM is 6.10.8 + current patches.

TAC is having fun with this. So far we've decided that the AD environment and clearpass are OK, and I've been escalated to the controller team. 

What is your WLAN hardware and what software version is running?  Version of CPPM?

Hello all,

While I wait for TAC to get going on this, I thought I would poll and see if anyone has seen this previously.

I have a 4 node cluster authenticating MSCHAPv2 to and AD controller. (Yes, yes, deprecated, TLS, I know. Once I figure out how to pay for 10k certs for student devices I'll take a run at it.)

I am seeing approximately 20% of all wireless authentications timing out with a 9002 error. The timeouts are more or less equally distributed by service, location, controller (including eduroam users at other sites), device type, etc. Over a 24 hour period ALL clients experienced at least one timeout but successfully connected later on a retry.

The 20% failure rate is a constant. Overnight at 4 auths/second or daytime at 100, it's the same ratio. And they come in a batch, generally close to 8 at a time as seen in access tracker..

I've tried changing the AD controller being used, confirmed everything is on SSD storage. Nothing in the event logs on clearpass or in AD.

Has anyone seen anything like this before?

Versions of software involved?

I am told this is usually a client issue and it not accepting the certificate.  I have also been told it is users in areas with weak signals walking out of the coverage area.  This I can understand more, but not at the levels we see (also about 20%).  Authentication takes less that a second and users do not usually move that fast.

We also see it on managed machines that are provisioned for TLS that still timeout.  The latest is our Cape Sensors started having the same problem recently. We know that they can connect (as they do not fail all of the time), they are not moving out of coverage.  So something else is going on.  We have the problem with 8 different Cape Sensors on Three different Controller clusters and fail  sometimes on all 7 of the clearpass servers we are currently running.

Yes, I will be opening a TAC case but it is hard when his is random and you are trying to get the perfect alignment of the planets so you can get data from the controller, radius server and packet dumps. 

If you deployed PEAP without provisioning tool for your clients, this is expected. The 9002 means there is a timeout on the authentication, and in most cases this is a client configuration problem. Clients that are not (pre) provisioned to the SSID, accept the server certificate, are likely to prompt the end-user to accept the connection or not. If the end-user doesn't see the authentication popup, or is slow in responding, you will see a timeout.

If this is eduroam, the use of geteduroam or CAT tool can probably resolve the issue by correctly configuring your clients. Geteduroam may even solve your TLS client certificate challenge, depending on where you are located.

Please let us know if you were able to resolve the issue, and how.

My controllers are all 72xx running 8.10.13. But I have no idea what the controllers are that are being used at remote eduroam locations. 

CPPM is 6.10.8 + current patches.

TAC is having fun with this. So far we've decided that the AD environment and clearpass are OK, and I've been escalated to the controller team. 

What is your WLAN hardware and what software version is running?  Version of CPPM?

Hello all,

While I wait for TAC to get going on this, I thought I would poll and see if anyone has seen this previously.

I have a 4 node cluster authenticating MSCHAPv2 to and AD controller. (Yes, yes, deprecated, TLS, I know. Once I figure out how to pay for 10k certs for student devices I'll take a run at it.)

I am seeing approximately 20% of all wireless authentications timing out with a 9002 error. The timeouts are more or less equally distributed by service, location, controller (including eduroam users at other sites), device type, etc. Over a 24 hour period ALL clients experienced at least one timeout but successfully connected later on a retry.

The 20% failure rate is a constant. Overnight at 4 auths/second or daytime at 100, it's the same ratio. And they come in a batch, generally close to 8 at a time as seen in access tracker..

I've tried changing the AD controller being used, confirmed everything is on SSD storage. Nothing in the event logs on clearpass or in AD.

Has anyone seen anything like this before?

Controllers 8.10.0.13
Clearpass 6.11.9

Versions of software involved?

I am told this is usually a client issue and it not accepting the certificate.  I have also been told it is users in areas with weak signals walking out of the coverage area.  This I can understand more, but not at the levels we see (also about 20%).  Authentication takes less that a second and users do not usually move that fast.

We also see it on managed machines that are provisioned for TLS that still timeout.  The latest is our Cape Sensors started having the same problem recently. We know that they can connect (as they do not fail all of the time), they are not moving out of coverage.  So something else is going on.  We have the problem with 8 different Cape Sensors on Three different Controller clusters and fail  sometimes on all 7 of the clearpass servers we are currently running.

Yes, I will be opening a TAC case but it is hard when his is random and you are trying to get the perfect alignment of the planets so you can get data from the controller, radius server and packet dumps. 

If you deployed PEAP without provisioning tool for your clients, this is expected. The 9002 means there is a timeout on the authentication, and in most cases this is a client configuration problem. Clients that are not (pre) provisioned to the SSID, accept the server certificate, are likely to prompt the end-user to accept the connection or not. If the end-user doesn't see the authentication popup, or is slow in responding, you will see a timeout.

If this is eduroam, the use of geteduroam or CAT tool can probably resolve the issue by correctly configuring your clients. Geteduroam may even solve your TLS client certificate challenge, depending on where you are located.

Please let us know if you were able to resolve the issue, and how.

My controllers are all 72xx running 8.10.13. But I have no idea what the controllers are that are being used at remote eduroam locations. 

CPPM is 6.10.8 + current patches.

TAC is having fun with this. So far we've decided that the AD environment and clearpass are OK, and I've been escalated to the controller team. 

What is your WLAN hardware and what software version is running?  Version of CPPM?

Hello all,

While I wait for TAC to get going on this, I thought I would poll and see if anyone has seen this previously.

I have a 4 node cluster authenticating MSCHAPv2 to and AD controller. (Yes, yes, deprecated, TLS, I know. Once I figure out how to pay for 10k certs for student devices I'll take a run at it.)

I am seeing approximately 20% of all wireless authentications timing out with a 9002 error. The timeouts are more or less equally distributed by service, location, controller (including eduroam users at other sites), device type, etc. Over a 24 hour period ALL clients experienced at least one timeout but successfully connected later on a retry.

The 20% failure rate is a constant. Overnight at 4 auths/second or daytime at 100, it's the same ratio. And they come in a batch, generally close to 8 at a time as seen in access tracker..

I've tried changing the AD controller being used, confirmed everything is on SSD storage. Nothing in the event logs on clearpass or in AD.

Has anyone seen anything like this before?

Versions of software involved?

I am told this is usually a client issue and it not accepting the certificate.  I have also been told it is users in areas with weak signals walking out of the coverage area.  This I can understand more, but not at the levels we see (also about 20%).  Authentication takes less that a second and users do not usually move that fast.

We also see it on managed machines that are provisioned for TLS that still timeout.  The latest is our Cape Sensors started having the same problem recently. We know that they can connect (as they do not fail all of the time), they are not moving out of coverage.  So something else is going on.  We have the problem with 8 different Cape Sensors on Three different Controller clusters and fail  sometimes on all 7 of the clearpass servers we are currently running.

Yes, I will be opening a TAC case but it is hard when his is random and you are trying to get the perfect alignment of the planets so you can get data from the controller, radius server and packet dumps. 

If you deployed PEAP without provisioning tool for your clients, this is expected. The 9002 means there is a timeout on the authentication, and in most cases this is a client configuration problem. Clients that are not (pre) provisioned to the SSID, accept the server certificate, are likely to prompt the end-user to accept the connection or not. If the end-user doesn't see the authentication popup, or is slow in responding, you will see a timeout.

If this is eduroam, the use of geteduroam or CAT tool can probably resolve the issue by correctly configuring your clients. Geteduroam may even solve your TLS client certificate challenge, depending on where you are located.

Please let us know if you were able to resolve the issue, and how.

My controllers are all 72xx running 8.10.13. But I have no idea what the controllers are that are being used at remote eduroam locations. 

CPPM is 6.10.8 + current patches.

TAC is having fun with this. So far we've decided that the AD environment and clearpass are OK, and I've been escalated to the controller team. 

What is your WLAN hardware and what software version is running?  Version of CPPM?

Hello all,

While I wait for TAC to get going on this, I thought I would poll and see if anyone has seen this previously.

I have a 4 node cluster authenticating MSCHAPv2 to and AD controller. (Yes, yes, deprecated, TLS, I know. Once I figure out how to pay for 10k certs for student devices I'll take a run at it.)

I am seeing approximately 20% of all wireless authentications timing out with a 9002 error. The timeouts are more or less equally distributed by service, location, controller (including eduroam users at other sites), device type, etc. Over a 24 hour period ALL clients experienced at least one timeout but successfully connected later on a retry.

The 20% failure rate is a constant. Overnight at 4 auths/second or daytime at 100, it's the same ratio. And they come in a batch, generally close to 8 at a time as seen in access tracker..

I've tried changing the AD controller being used, confirmed everything is on SSD storage. Nothing in the event logs on clearpass or in AD.

Has anyone seen anything like this before?

If you deployed PEAP without provisioning tool for your clients, this is expected. The 9002 means there is a timeout on the authentication, and in most cases this is a client configuration problem. Clients that are not (pre) provisioned to the SSID, accept the server certificate, are likely to prompt the end-user to accept the connection or not. If the end-user doesn't see the authentication popup, or is slow in responding, you will see a timeout.

If this is eduroam, the use of geteduroam or CAT tool can probably resolve the issue by correctly configuring your clients. Geteduroam may even solve your TLS client certificate challenge, depending on where you are located.

Please let us know if you were able to resolve the issue, and how.

My controllers are all 72xx running 8.10.13. But I have no idea what the controllers are that are being used at remote eduroam locations. 

CPPM is 6.10.8 + current patches.

TAC is having fun with this. So far we've decided that the AD environment and clearpass are OK, and I've been escalated to the controller team. 

What is your WLAN hardware and what software version is running?  Version of CPPM?

Hello all,

While I wait for TAC to get going on this, I thought I would poll and see if anyone has seen this previously.

I have a 4 node cluster authenticating MSCHAPv2 to and AD controller. (Yes, yes, deprecated, TLS, I know. Once I figure out how to pay for 10k certs for student devices I'll take a run at it.)

I am seeing approximately 20% of all wireless authentications timing out with a 9002 error. The timeouts are more or less equally distributed by service, location, controller (including eduroam users at other sites), device type, etc. Over a 24 hour period ALL clients experienced at least one timeout but successfully connected later on a retry.

The 20% failure rate is a constant. Overnight at 4 auths/second or daytime at 100, it's the same ratio. And they come in a batch, generally close to 8 at a time as seen in access tracker..

I've tried changing the AD controller being used, confirmed everything is on SSD storage. Nothing in the event logs on clearpass or in AD.

Has anyone seen anything like this before?

I have drilled this into the helpdesk for years. Step one troubleshooting is to run the cat tool. 

There seems to be two issues concurrently.

Clearpass is using domain controllers it shouldn't. For example, a node was joined to DC1, with DC2 and DC3 set as password servers. But I'm seeing failed authentications against DC4.

Failed authentications are being reported as timeouts instead of rejects.

I'm still seeing ~15% of all auth requests getting timeouts. (Not users - the retried requests inflate the percentage considerably)

TAC gave up. Said it was all of my users with bad passwords.

At this point I'm treating it as cosmetic . Almost all of these are phones and most just switch to data automatically and the users don't even care, and I have many other fish to fry.

If you deployed PEAP without provisioning tool for your clients, this is expected. The 9002 means there is a timeout on the authentication, and in most cases this is a client configuration problem. Clients that are not (pre) provisioned to the SSID, accept the server certificate, are likely to prompt the end-user to accept the connection or not. If the end-user doesn't see the authentication popup, or is slow in responding, you will see a timeout.

If this is eduroam, the use of geteduroam or CAT tool can probably resolve the issue by correctly configuring your clients. Geteduroam may even solve your TLS client certificate challenge, depending on where you are located.

Please let us know if you were able to resolve the issue, and how.

My controllers are all 72xx running 8.10.13. But I have no idea what the controllers are that are being used at remote eduroam locations. 

CPPM is 6.10.8 + current patches.

TAC is having fun with this. So far we've decided that the AD environment and clearpass are OK, and I've been escalated to the controller team. 

What is your WLAN hardware and what software version is running?  Version of CPPM?

Hello all,

While I wait for TAC to get going on this, I thought I would poll and see if anyone has seen this previously.

I have a 4 node cluster authenticating MSCHAPv2 to and AD controller. (Yes, yes, deprecated, TLS, I know. Once I figure out how to pay for 10k certs for student devices I'll take a run at it.)

I am seeing approximately 20% of all wireless authentications timing out with a 9002 error. The timeouts are more or less equally distributed by service, location, controller (including eduroam users at other sites), device type, etc. Over a 24 hour period ALL clients experienced at least one timeout but successfully connected later on a retry.

The 20% failure rate is a constant. Overnight at 4 auths/second or daytime at 100, it's the same ratio. And they come in a batch, generally close to 8 at a time as seen in access tracker..

I've tried changing the AD controller being used, confirmed everything is on SSD storage. Nothing in the event logs on clearpass or in AD.

Has anyone seen anything like this before?

As far as the failed authentications being seen as timeouts.  If you look under Clearpass - Configurations - Authentications - Methods - The method you use (MSChap) there is a field "Number of retries".  If the client fails auth, clearpass will just try again and many times that times out.  Why if it gets a failure and retries I am not sure.  This might at least clear up some of those.

I do have the hotfix patch 9 installed on clearpass.

I have drilled this into the helpdesk for years. Step one troubleshooting is to run the cat tool. 

There seems to be two issues concurrently.

Clearpass is using domain controllers it shouldn't. For example, a node was joined to DC1, with DC2 and DC3 set as password servers. But I'm seeing failed authentications against DC4.

Failed authentications are being reported as timeouts instead of rejects.

I'm still seeing ~15% of all auth requests getting timeouts. (Not users - the retried requests inflate the percentage considerably)

TAC gave up. Said it was all of my users with bad passwords.

At this point I'm treating it as cosmetic . Almost all of these are phones and most just switch to data automatically and the users don't even care, and I have many other fish to fry.

If you deployed PEAP without provisioning tool for your clients, this is expected. The 9002 means there is a timeout on the authentication, and in most cases this is a client configuration problem. Clients that are not (pre) provisioned to the SSID, accept the server certificate, are likely to prompt the end-user to accept the connection or not. If the end-user doesn't see the authentication popup, or is slow in responding, you will see a timeout.

If this is eduroam, the use of geteduroam or CAT tool can probably resolve the issue by correctly configuring your clients. Geteduroam may even solve your TLS client certificate challenge, depending on where you are located.

Please let us know if you were able to resolve the issue, and how.

My controllers are all 72xx running 8.10.13. But I have no idea what the controllers are that are being used at remote eduroam locations. 

CPPM is 6.10.8 + current patches.

TAC is having fun with this. So far we've decided that the AD environment and clearpass are OK, and I've been escalated to the controller team. 

What is your WLAN hardware and what software version is running?  Version of CPPM?

Hello all,

While I wait for TAC to get going on this, I thought I would poll and see if anyone has seen this previously.

I have a 4 node cluster authenticating MSCHAPv2 to and AD controller. (Yes, yes, deprecated, TLS, I know. Once I figure out how to pay for 10k certs for student devices I'll take a run at it.)

I am seeing approximately 20% of all wireless authentications timing out with a 9002 error. The timeouts are more or less equally distributed by service, location, controller (including eduroam users at other sites), device type, etc. Over a 24 hour period ALL clients experienced at least one timeout but successfully connected later on a retry.

The 20% failure rate is a constant. Overnight at 4 auths/second or daytime at 100, it's the same ratio. And they come in a batch, generally close to 8 at a time as seen in access tracker..

I've tried changing the AD controller being used, confirmed everything is on SSD storage. Nothing in the event logs on clearpass or in AD.

Has anyone seen anything like this before?