Security

 View Only
Expand all | Collapse all

suddenly getting timeouts on authentications

This thread has been viewed 95 times
  • 1.  suddenly getting timeouts on authentications

    Posted Jul 13, 2022 01:46 PM
    Hi All,

    I have been migrating a site to be secured through clearpass today and after a minute or 15, I had everything working just fine. We use EAP-TLS on the clients. At some point, I suddenly started receiving TIMEOUTs and with that a Deny Access Profile. There has not been any change since we migrated over and started seeing the timeouts. These timeouts were the same on the Wireless service and on the wired portion of the network. I am not able to find why this would happen so suddenly, it's not even one location but all locations that have been working just fine.

    The only difference I can find is in the Authentication Method. A working authentication would have EAP-TLS as the method, and authentication with a timeout would have only EAP as the Authentication Method. I have never seen this before. I cant think of any reason why the method would not be EAP-TLS but only EAP without any addition like -TLS.

    Does anyone know what could be the cause of this? Would this be a supplicant issue, or could this be a clearpass-related issue? No other changes have been made by me or other engineers when I was doing the migration and testing. the Alert for the failing requests is:
    Error Code:
    9002
    Error Category:
    RADIUS protocol
    Error Message:
    Request timed out
    RADIUS Client did not complete EAP transaction

    When i check the logs of one of the authentications that timedout i only see the following that looks out of the ordinary:
    2022-07-13 11:29:11,764 [main SessId R00000509-01-62ce902d] ERROR RadiusServer.Radius - reqst_clean_list:

    Anyone that might be able to point me in the right direction? For now i just disabled dot1x on the network (it's still kind of pilot so not a very big issue there)

    Kind regards!


  • 2.  RE: suddenly getting timeouts on authentications

    Posted Jul 13, 2022 03:06 PM

    If you have an ACCEPT event for EAP-TLS and some timeouts or reject for EAP-PEAP, this could be caused by an incorrect supplicant profile, pls be sure supplicant is only trying certificate authentication, nor User or Machine

     

    Hope this help you

     

    Jorge

     






  • 3.  RE: suddenly getting timeouts on authentications

    Posted Jul 14, 2022 02:03 AM
    Hi Jorge,

    The client is not configured for EAP-PEAP. The supplicants are only configured to use EAP-TLS. When authentication is successful I also see the method being EAP-TLS. So it's not a supplicant that tries to do EAP-PEAP as a fallback method. With a timeout, I only see method EAP, and that is something I cant explain, why would a supplicant try to authenticate using a method that it does not support? There should always be something behind the EAP part, like EAP-PEAP, or in my case, EAP-TLS, why just EAP?


  • 4.  RE: suddenly getting timeouts on authentications

    Posted Jul 13, 2022 03:22 PM
    This sounds like a client issue.  The main things that come to mind are an improperly designed or degraded wireless deployment, a mis-configured supplicant, or a driver issue.


  • 5.  RE: suddenly getting timeouts on authentications

    Posted Jul 14, 2022 02:11 AM
    Hi,

    It sounded like that to me as well, but I need to be able to understand what is going wrong. We are not the sysadmins for the supplicants, we just do the wifi/networking part. Strange is that it has been working on a few sites for weeks just fine. I can't find a reason as to why the supplicant would not specify a proper EAP method for authentication when it has been doing this for weeks and in this case, this site was successful for an hour or two.

    It's also weird that we see this issue on wired and wireless clients. Both worked fine and suddenly everything is timing out on the clearpass server.

    I guess I should be making a trace as soon as this issue is there again to see if the Clearpass is sending back traffic to the NAD successfully. I'm pretty sure the NAD's are sending the authentications to the Clearpass server since I'm getting timeouts. It could be multiple issues but it still baffles me that the supplicant is sending a request based on EAP and not EAP-TLS, maybe someone can explain why I am seeing that?


  • 6.  RE: suddenly getting timeouts on authentications

    Posted Jul 14, 2022 12:18 PM
    Did you change the radius server certificate recently?

    ------------------------------
    Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

    HPE Design and Deploy Guides: https://community.arubanetworks.com/support/migrated-knowledge-base?attachments=&communitykey=dcc83c62-1a3a-4dd8-94dc-92968ea6fff1&pageindex=0&pagesize=12&search=&sort=most_recent&viewtype=card
    ------------------------------



  • 7.  RE: suddenly getting timeouts on authentications

    Posted Jul 14, 2022 02:39 PM
    This^ or did a certificate expire anywhere?  On ClearPass?  CA trust?  Client certificates?


  • 8.  RE: suddenly getting timeouts on authentications

    Posted Jul 15, 2022 09:29 AM
    Hi,

    No changes have been made to the certificates recently. And I have been testing the pilot location again and everything is working just fine now.
    It's pretty baffling that there is nothing that i can find that might explain the behaviour i am seeing.

    Could you let me know the best way forward if i see this again? Would a wireshark be the best option, or should i also enable other debugs? I could make a TAC case but i guess i have not enough information for the tac team to work with and i would like to be able to send them as much information as possible as soon as this happens again!

    Thanks in advance!


  • 9.  RE: suddenly getting timeouts on authentications

    Posted Jul 15, 2022 10:09 AM
    This sounds 100% like a client issue.  I would recommend ensuring all client NIC card and wireless drivers are up to date.


  • 10.  RE: suddenly getting timeouts on authentications

    Posted Jul 15, 2022 01:36 PM
    I am not so sure this is a client issue.  We have the same problems though we do use EAP-PEAP

    On the good connections, we (usually) see an auth method os EAP-PEAP,EAP-MSChapv2.  The bad ones show just EAP-PEAP mostly.  It appears that if completes the outer method but never finished the MSCHapv2 auth.  We also see the same error in the logs

    2022-07-15 13:10:58,047 [main SessId R0009aad9-18-62d19f71] ERROR RadiusServer.Radius - reqst_clean_list


    I am told that this is most likely an issue with transient users being in marginal signal errors.  That seems unlikely to me as the authentication time is so quick that a person walking would most likely have time to connect.  Possible they are in a weak signal or maybe they are on a bus that is going by fast enough.  But I still think we have way too many timeouts for that to be the case but on a large campus that is in the middle of the city, it very well may be.




  • 11.  RE: suddenly getting timeouts on authentications

    Posted Jul 18, 2022 02:52 AM
    I guess a timeout could mean a lot of things. Clearpass not getting a response from the client for example, or the client (authenticator) never getting the response from clearpass could also be an issue. But the thing that I am worried about is the method of authentication I am getting. It should not be only EAP, it should contain a method as well, like TLS or PEAP even.

    Since all my problems seem to have vanished like thin air, the most important question I still have is: How can I research this issue the best way when it comes up again. Now it is still a pilot phase, with 3 very small sites, but we want to start implementing 20 other sites which are much bigger in terms of users.

    Is there some best practice i should be using when researching a problem with clearpass (or supplicant/authenticator)?


  • 12.  RE: suddenly getting timeouts on authentications

    Posted Jul 18, 2022 08:51 AM
    Reviewing ClearPass and wireless controller logs is always a good place to start.  For instance, do clients disconnect from the wireless around the same time as a timeout?  Could this be a possible wireless interference issue?  What is your wireless environment at these sites?  What does the RF look like?


  • 13.  RE: suddenly getting timeouts on authentications

    Posted Jul 18, 2022 05:17 PM
    I would work with my Aruba/HPE Sales engineer to guide you through this pilot.  They could engage you with technical support or they might know the answer..

    ------------------------------
    Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

    HPE Design and Deploy Guides: https://community.arubanetworks.com/support/migrated-knowledge-base?attachments=&communitykey=dcc83c62-1a3a-4dd8-94dc-92968ea6fff1&pageindex=0&pagesize=12&search=&sort=most_recent&viewtype=card
    ------------------------------



  • 14.  RE: suddenly getting timeouts on authentications

    Posted Jul 19, 2022 09:12 AM
    Hi

    We had similar issue in our environment that user authentication request were getting timeout on clearpass & users were not access any services.
    During investigation, we found radius server were sending access challenges request to user but there were not repose from user side.
    However, we then found packets are getting fragmented in network. Then, on controller side, we reduced the mtu size for corporate ssid, after which we started seeing user authentication on clearpass side.

    Regards
    Pankaj



  • 15.  RE: suddenly getting timeouts on authentications

    Posted Feb 10, 2025 08:11 PM

    We have similar issues with network fragmentation, to what size dd you reduce the mtu?




  • 16.  RE: suddenly getting timeouts on authentications

    Posted Feb 14, 2025 06:30 AM

    Did you test/know already the end-to-end MTU?

    Try to find the maximum value for the icmp payload, which typically will be somewhere between 1000-1450 over WAN/VPN.

    If you know the maximum MTU, then go 50 or 100 bytes below that for your eap-fragmentation setting on switches/APs. ClearPass has a default of 1024 bytes EAP fragmentation size for sending, but is configurable.

    If you can move to RadSec, that would completely evade this issue.



    ------------------------------
    Herman Robers
    ------------------------
    If you have urgent issues, always contact your HPE Aruba Networking partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact HPE Aruba Networking TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or HPE Aruba Networking.

    In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.
    ------------------------------



  • 17.  RE: suddenly getting timeouts on authentications

    Posted Feb 18, 2025 07:05 AM

    Thanks for the response, I don't suppose you have one of  your fabulous videos on how to configure RadSec for Aruba switches, IAP or Mobility controller/conductor ?

     

    I am testing on a switch but it just stalls at

     

      Connection Status       : Waiting for socket creation

      Connection Error        : NA

     

    And ClearPass does show any entries in Event viewer for radsec, any ideas where to start?

     

    Regards,

    Paul.

     

    Paul Wheeler | IT Systems Administrator - Europe.

    AMETEK | PO Box 36 | 2 New Star Road | Leicester LE4 9JQ, UK

     

    Phone: +44 (0) 116 246 3006 |  Fax: +44 (0) 116 246 3060 | Mobile: +44 (0) 7824 692 684

    paul.wheeler@ametek.com | www.ametek.com

     

    DISCLAIMER: This message may contain privileged and confidential information. If you think for any reason this message has been addressed in error you must not copy or disseminate it and we would ask you to notify us immediately by return email to info.uk-lei@ametek.com. Internet emails are not necessarily secure. Taylor Hobson is part of AMETEK Ultra Precision Technologies, a division of AMETEK Inc (NYSE: AME) a leading global manufacturer of electronic instruments and electric motors. Taylor Hobson Limited is registered in England No. 3230332, with its address at PO Box 36, 2 New Star Road, Leicester, LE4 9JQ, England.

     






  • 18.  RE: suddenly getting timeouts on authentications

    Posted Feb 18, 2025 08:04 AM

    There is a video on EST and RadSec here. You may skip the EST part and just use the RadSec and use the factory device certificates.

    In written, there is a document/solution paper here.



    ------------------------------
    Herman Robers
    ------------------------
    If you have urgent issues, always contact your HPE Aruba Networking partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact HPE Aruba Networking TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or HPE Aruba Networking.

    In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.
    ------------------------------



  • 19.  RE: suddenly getting timeouts on authentications

    Posted Feb 18, 2025 09:35 AM

    Thanks again, some excellent information, unfortunately we are not yet able to run Central and OS10 and our Switches are AOS-S

    I followed these guides

    https://arubanetworking.hpe.com/techdocs/AOS-Switch/16.09/Aruba%202930F%26M%20Access%20Security%20Guide%20for%20AOS-S%20Switch%2016.09.pdf

    https://arubanetworking.hpe.com/techdocs/ArubaOS_8.11.2_Web_Help/Content/arubaos-solutions/auth-servers/enab-rads-radi-serv.htm

     

    The switch still does nothing but I am getting an error in ClearPass now for the controller

     

    TLS connection couldn't connect for IP.ADDR: Errors: error:14089086:SSL routines:ssl3_get_client_certificate:certificate verify failed

     

    So we made a step further but hit a new challenge!

     

     

    Regards,

    Paul.

     

    Paul Wheeler | IT Systems Administrator - Europe.

    AMETEK | PO Box 36 | 2 New Star Road | Leicester LE4 9JQ, UK

     

    Phone: +44 (0) 116 246 3006 |  Fax: +44 (0) 116 246 3060 | Mobile: +44 (0) 7824 692 684

    paul.wheeler@ametek.com | www.ametek.com

     

    DISCLAIMER: This message may contain privileged and confidential information. If you think for any reason this message has been addressed in error you must not copy or disseminate it and we would ask you to notify us immediately by return email to info.uk-lei@ametek.com. Internet emails are not necessarily secure. Taylor Hobson is part of AMETEK Ultra Precision Technologies, a division of AMETEK Inc (NYSE: AME) a leading global manufacturer of electronic instruments and electric motors. Taylor Hobson Limited is registered in England No. 3230332, with its address at PO Box 36, 2 New Star Road, Leicester, LE4 9JQ, England.

     






  • 20.  RE: suddenly getting timeouts on authentications

    Posted Feb 19, 2025 03:59 AM

    For AOS-S switches you can use following steps. If your switches are Aruba branded it is very likely that factory cert is already there. On older switches you need to provide the cert. Steps here assume that you will provide the cert.

    1. Create new TA-Profile for Radsec
      crypto pki ta-profile RADsec-profile

    2. Install Root CA cert to ta-profile
      copy tftp ta-certificate RADsec-profile <ip of tftp server> <CA cert filename>

    3. Create CSR request for ta-profile with usage all or radsec-client
      crypto pki create-csr certificate-name <cert name> ta-profile RADsec-profile key-type <key type> .... usage radsec-client

    4. Sign CSR and add SAN IP:<switch ip address> to the cert during signing

    5. Install signed cert
      crypto pki install-signed-certificate

    6. Radius server need to trust this CA

    7. Setup radius on switch
      radius-server host <radius ip> tls ...
      ...other radius configuration parameters with tls...
      aaa server-group radius <group name> host <radius ip> tls

    8. For troubleshooting you can use
          show radius host <host ip>
          debug security radsec
          debug security radius

    Best, Gorazd



    ------------------------------
    Gorazd Kikelj
    MVP Guru 2025
    ------------------------------