Network Management

Hi , We have been facing issues with one of our subscriber nodes of clearpass( in a cluster with publisher which is not experiencing this problem) , dot1x authentications are failing intermittently droping networks , errors shows EAP transactions timed out( both on wired and wireless) . I have ERT engineers from Aruba Clearpass and Switching team and no one is able to pinpoint the issue . This is happening on all Aruba OS , Aruba OS CX and Cisco Switches which are configured for this specific Clearpass Subscriber node , regardless of switch type .Sometimes it works and sometimes it does not .

No certificates have been updated on clearpass ends( this setup has been functional for a long time until Jan 2024) , these clearpass certs have been there for some time and are not expired at all , checked default MTU size on Switch which is 1500 for all end switches .

Functional Clearpass is in a different DC and non-functional CPPM is in a different DC . Any clues or cues..

This is going on for months and affecting HQ as well 

Are you using EAP-TLS? In that case, a common issue is that when the client sends it's client certificate, the resulting RADIUS packet from switch/AP/controller to ClearPass becomes too big. One solution is Jumbo frames end-to-end (so 1500 MTU is not enough), another is configuring your switches/APs for EAP fragmentation (with like 1000 bytes or so), or you can change to RadSec which by using TCP instead of UDP avoids the MTU issue all together. Changing to EC certificates on your client (and intermediates) may help as well as it reduces the size of the client certificate + intermediates.

ClearPass has an EAP-fragmentation by default (think with 1024 bytes), so from that side there should not be an issue. It's in general the path from switch to the RADIUS server causing these timeouts, and with EAP-TLS, TEAP or other authentication methods that use client certificates.

Did you already find out if/where/what RADIUS packets are dropped?

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.

Original Message:
Sent: Apr 12, 2024 05:45 PM
From: agn
Subject: Clearpass not able to authenticate do1x clients with EAP timeouts

Hi , We have been facing issues with one of our subscriber nodes of clearpass( in a cluster with publisher which is not experiencing this problem) , dot1x authentications are failing intermittently droping networks , errors shows EAP transactions timed out( both on wired and wireless) . I have ERT engineers from Aruba Clearpass and Switching team and no one is able to pinpoint the issue . This is happening on all Aruba OS , Aruba OS CX and Cisco Switches which are configured for this specific Clearpass Subscriber node , regardless of switch type .Sometimes it works and sometimes it does not .

No certificates have been updated on clearpass ends( this setup has been functional for a long time until Jan 2024) , these clearpass certs have been there for some time and are not expired at all , checked default MTU size on Switch which is 1500 for all end switches .

Functional Clearpass is in a different DC and non-functional CPPM is in a different DC . Any clues or cues..

This is going on for months and affecting HQ as well 

Thanks for responding .

So far we did packet captures on Clearpass where packets are for EAP are arriving fragmented ( from working and non working scenario) , on switch end also ERT engineers said they ahve not yet seen jumbo packets leaving the switch . How and where can i enable Rad Sec , looking for documentation already 

No we have not yet figured out where are packets are being dropped . only checked at L2 switcha and clearpass

Are you using EAP-TLS? In that case, a common issue is that when the client sends it's client certificate, the resulting RADIUS packet from switch/AP/controller to ClearPass becomes too big. One solution is Jumbo frames end-to-end (so 1500 MTU is not enough), another is configuring your switches/APs for EAP fragmentation (with like 1000 bytes or so), or you can change to RadSec which by using TCP instead of UDP avoids the MTU issue all together. Changing to EC certificates on your client (and intermediates) may help as well as it reduces the size of the client certificate + intermediates.

ClearPass has an EAP-fragmentation by default (think with 1024 bytes), so from that side there should not be an issue. It's in general the path from switch to the RADIUS server causing these timeouts, and with EAP-TLS, TEAP or other authentication methods that use client certificates.

Did you already find out if/where/what RADIUS packets are dropped?

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.

Original Message:
Sent: Apr 12, 2024 05:45 PM
From: agn
Subject: Clearpass not able to authenticate do1x clients with EAP timeouts

Hi , We have been facing issues with one of our subscriber nodes of clearpass( in a cluster with publisher which is not experiencing this problem) , dot1x authentications are failing intermittently droping networks , errors shows EAP transactions timed out( both on wired and wireless) . I have ERT engineers from Aruba Clearpass and Switching team and no one is able to pinpoint the issue . This is happening on all Aruba OS , Aruba OS CX and Cisco Switches which are configured for this specific Clearpass Subscriber node , regardless of switch type .Sometimes it works and sometimes it does not .

No certificates have been updated on clearpass ends( this setup has been functional for a long time until Jan 2024) , these clearpass certs have been there for some time and are not expired at all , checked default MTU size on Switch which is 1500 for all end switches .

Functional Clearpass is in a different DC and non-functional CPPM is in a different DC . Any clues or cues..

This is going on for months and affecting HQ as well 

Also there are 5-6 hops in middle so packet packture is a little bit tough thing to do right now , any other suggestion , but if packet captures is needed on each hop i can do that as well 

Thanks for responding .

So far we did packet captures on Clearpass where packets are for EAP are arriving fragmented ( from working and non working scenario) , on switch end also ERT engineers said they ahve not yet seen jumbo packets leaving the switch . How and where can i enable Rad Sec , looking for documentation already 

No we have not yet figured out where are packets are being dropped . only checked at L2 switcha and clearpass

Are you using EAP-TLS? In that case, a common issue is that when the client sends it's client certificate, the resulting RADIUS packet from switch/AP/controller to ClearPass becomes too big. One solution is Jumbo frames end-to-end (so 1500 MTU is not enough), another is configuring your switches/APs for EAP fragmentation (with like 1000 bytes or so), or you can change to RadSec which by using TCP instead of UDP avoids the MTU issue all together. Changing to EC certificates on your client (and intermediates) may help as well as it reduces the size of the client certificate + intermediates.

ClearPass has an EAP-fragmentation by default (think with 1024 bytes), so from that side there should not be an issue. It's in general the path from switch to the RADIUS server causing these timeouts, and with EAP-TLS, TEAP or other authentication methods that use client certificates.

Did you already find out if/where/what RADIUS packets are dropped?

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.

Original Message:
Sent: Apr 12, 2024 05:45 PM
From: agn
Subject: Clearpass not able to authenticate do1x clients with EAP timeouts

Hi , We have been facing issues with one of our subscriber nodes of clearpass( in a cluster with publisher which is not experiencing this problem) , dot1x authentications are failing intermittently droping networks , errors shows EAP transactions timed out( both on wired and wireless) . I have ERT engineers from Aruba Clearpass and Switching team and no one is able to pinpoint the issue . This is happening on all Aruba OS , Aruba OS CX and Cisco Switches which are configured for this specific Clearpass Subscriber node , regardless of switch type .Sometimes it works and sometimes it does not .

No certificates have been updated on clearpass ends( this setup has been functional for a long time until Jan 2024) , these clearpass certs have been there for some time and are not expired at all , checked default MTU size on Switch which is 1500 for all end switches .

Functional Clearpass is in a different DC and non-functional CPPM is in a different DC . Any clues or cues..

This is going on for months and affecting HQ as well 

What typically happens if MTU is the issue, is that you see the packet leaving the switch, but not arriving on the ClearPass. If that is the case, you can check somewhere in the middle if you can see the packet there, if not go further towards the switch, if so go further to the ClearPass. Note that when ClearPass is running as a VM, like ESXi, you have MTU settings on the hypervisor/vswitch/vnic as well. Once you know where the packets are lost, the solution is to increase the MTU there, but best is to make the full path jumbo. For RadSec, you may have a look at these tutorials.

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.

Original Message:
Sent: Apr 18, 2024 05:39 PM
From: agn
Subject: Clearpass not able to authenticate do1x clients with EAP timeouts

Also there are 5-6 hops in middle so packet packture is a little bit tough thing to do right now , any other suggestion , but if packet captures is needed on each hop i can do that as well 

Original Message:
Sent: Apr 18, 2024 05:38 PM
From: agn
Subject: Clearpass not able to authenticate do1x clients with EAP timeouts

Thanks for responding .

So far we did packet captures on Clearpass where packets are for EAP are arriving fragmented ( from working and non working scenario) , on switch end also ERT engineers said they ahve not yet seen jumbo packets leaving the switch . How and where can i enable Rad Sec , looking for documentation already 

No we have not yet figured out where are packets are being dropped . only checked at L2 switcha and clearpass

Original Message:
Sent: Apr 15, 2024 02:44 AM
From: Herman Robers
Subject: Clearpass not able to authenticate do1x clients with EAP timeouts

Are you using EAP-TLS? In that case, a common issue is that when the client sends it's client certificate, the resulting RADIUS packet from switch/AP/controller to ClearPass becomes too big. One solution is Jumbo frames end-to-end (so 1500 MTU is not enough), another is configuring your switches/APs for EAP fragmentation (with like 1000 bytes or so), or you can change to RadSec which by using TCP instead of UDP avoids the MTU issue all together. Changing to EC certificates on your client (and intermediates) may help as well as it reduces the size of the client certificate + intermediates.

ClearPass has an EAP-fragmentation by default (think with 1024 bytes), so from that side there should not be an issue. It's in general the path from switch to the RADIUS server causing these timeouts, and with EAP-TLS, TEAP or other authentication methods that use client certificates.

Did you already find out if/where/what RADIUS packets are dropped?

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.

Original Message:
Sent: Apr 12, 2024 05:45 PM
From: agn
Subject: Clearpass not able to authenticate do1x clients with EAP timeouts

Hi , We have been facing issues with one of our subscriber nodes of clearpass( in a cluster with publisher which is not experiencing this problem) , dot1x authentications are failing intermittently droping networks , errors shows EAP transactions timed out( both on wired and wireless) . I have ERT engineers from Aruba Clearpass and Switching team and no one is able to pinpoint the issue . This is happening on all Aruba OS , Aruba OS CX and Cisco Switches which are configured for this specific Clearpass Subscriber node , regardless of switch type .Sometimes it works and sometimes it does not .

No certificates have been updated on clearpass ends( this setup has been functional for a long time until Jan 2024) , these clearpass certs have been there for some time and are not expired at all , checked default MTU size on Switch which is 1500 for all end switches .

Functional Clearpass is in a different DC and non-functional CPPM is in a different DC . Any clues or cues..

This is going on for months and affecting HQ as well