Security

 View Only
Expand all | Collapse all

CPPM 9002 errors with no obvious pattern

This thread has been viewed 38 times
  • 1.  CPPM 9002 errors with no obvious pattern

    Posted Sep 10, 2024 11:15 AM

    Hello all,

    While I wait for TAC to get going on this, I thought I would poll and see if anyone has seen this previously.

    I have a 4 node cluster authenticating MSCHAPv2 to and AD controller. (Yes, yes, deprecated, TLS, I know. Once I figure out how to pay for 10k certs for student devices I'll take a run at it.)

    I am seeing approximately 20% of all wireless authentications timing out with a 9002 error. The timeouts are more or less equally distributed by service, location, controller (including eduroam users at other sites), device type, etc. Over a 24 hour period ALL clients experienced at least one timeout but successfully connected later on a retry.

    The 20% failure rate is a constant. Overnight at 4 auths/second or daytime at 100, it's the same ratio. And they come in a batch, generally close to 8 at a time as seen in access tracker..

    I've tried changing the AD controller being used, confirmed everything is on SSD storage. Nothing in the event logs on clearpass or in AD.

    Has anyone seen anything like this before?



  • 2.  RE: CPPM 9002 errors with no obvious pattern

    Posted Sep 10, 2024 01:10 PM

    What is your WLAN hardware and what software version is running?  Version of CPPM?



    ------------------------------
    Carson Hulcher, ACEX#110
    ------------------------------



  • 3.  RE: CPPM 9002 errors with no obvious pattern

    Posted Sep 10, 2024 04:07 PM

    My controllers are all 72xx running 8.10.13. But I have no idea what the controllers are that are being used at remote eduroam locations. 

    CPPM is 6.10.8 + current patches.

    TAC is having fun with this. So far we've decided that the AD environment and clearpass are OK, and I've been escalated to the controller team. 




  • 4.  RE: CPPM 9002 errors with no obvious pattern

    Posted Sep 23, 2024 05:57 AM

    If you deployed PEAP without provisioning tool for your clients, this is expected. The 9002 means there is a timeout on the authentication, and in most cases this is a client configuration problem. Clients that are not (pre) provisioned to the SSID, accept the server certificate, are likely to prompt the end-user to accept the connection or not. If the end-user doesn't see the authentication popup, or is slow in responding, you will see a timeout.

    If this is eduroam, the use of geteduroam or CAT tool can probably resolve the issue by correctly configuring your clients. Geteduroam may even solve your TLS client certificate challenge, depending on where you are located.

    Please let us know if you were able to resolve the issue, and how.



    ------------------------------
    Herman Robers
    ------------------------
    If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

    In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.
    ------------------------------



  • 5.  RE: CPPM 9002 errors with no obvious pattern

    Posted Sep 24, 2024 04:21 PM

    I am told this is usually a client issue and it not accepting the certificate.  I have also been told it is users in areas with weak signals walking out of the coverage area.  This I can understand more, but not at the levels we see (also about 20%).  Authentication takes less that a second and users do not usually move that fast.

    We also see it on managed machines that are provisioned for TLS that still timeout.  The latest is our Cape Sensors started having the same problem recently. We know that they can connect (as they do not fail all of the time), they are not moving out of coverage.  So something else is going on.  We have the problem with 8 different Cape Sensors on Three different Controller clusters and fail  sometimes on all 7 of the clearpass servers we are currently running.

    Yes, I will be opening a TAC case but it is hard when his is random and you are trying to get the perfect alignment of the planets so you can get data from the controller, radius server and packet dumps. 




  • 6.  RE: CPPM 9002 errors with no obvious pattern

    Posted Sep 24, 2024 04:34 PM

    Versions of software involved?



    ------------------------------
    Carson Hulcher, ACEX#110
    ------------------------------



  • 7.  RE: CPPM 9002 errors with no obvious pattern

    Posted Sep 25, 2024 09:35 AM

    Controllers 8.10.0.13
    Clearpass 6.11.9




  • 8.  RE: CPPM 9002 errors with no obvious pattern

    Posted Sep 25, 2024 09:50 AM

    Have you applied the hotfix for ClearPass 6.11.9?



    ------------------------------
    Carson Hulcher, ACEX#110
    ------------------------------



  • 9.  RE: CPPM 9002 errors with no obvious pattern

    Posted Sep 25, 2024 11:34 AM

    Clearpass 6.10..8

    My controllers are 8.10.0.13, but I get the same results from anyone on eduroam on other campuses as well, including Aruba, Cisco, even Rukus gear.




  • 10.  RE: CPPM 9002 errors with no obvious pattern

    Posted Sep 25, 2024 12:04 PM

    I have drilled this into the helpdesk for years. Step one troubleshooting is to run the cat tool. 

    There seems to be two issues concurrently.

    Clearpass is using domain controllers it shouldn't. For example, a node was joined to DC1, with DC2 and DC3 set as password servers. But I'm seeing failed authentications against DC4.

    Failed authentications are being reported as timeouts instead of rejects.

    I'm still seeing ~15% of all auth requests getting timeouts. (Not users - the retried requests inflate the percentage considerably)

    TAC gave up. Said it was all of my users with bad passwords.

    At this point I'm treating it as cosmetic . Almost all of these are phones and most just switch to data automatically and the users don't even care, and I have many other fish to fry.




  • 11.  RE: CPPM 9002 errors with no obvious pattern

    Posted Sep 25, 2024 12:37 PM

    As far as the failed authentications being seen as timeouts.  If you look under Clearpass - Configurations - Authentications - Methods - The method you use (MSChap) there is a field "Number of retries".  If the client fails auth, clearpass will just try again and many times that times out.  Why if it gets a failure and retries I am not sure.  This might at least clear up some of those.

    I do have the hotfix patch 9 installed on clearpass.




  • 12.  RE: CPPM 9002 errors with no obvious pattern

    Posted Sep 25, 2024 01:57 PM

    That was helpful.

    My service is set up for [EAP-MSCHAPV2], [EAP-PEAP].

    Just for giggles I removed [EAP-PEAP]

    Most of my 9002 errors immediately changed to REJECT/9015 - Client does not support configured EAP methods

    Finally something to work with. Looks like these are primarily Apple "smart" devices.