Security

Hi there.  We have a large industrial site with Clearpass handling authentication and automatic VLAN selection for each area.  We have the system check for 802.1x details and it checks those against our domain controller machine list.  A GPO pushes out the certificates and makes sure the 802.1x service is on by default on our machines.  If the check is passed the machine is assigned the local data VLAN for that area.  The fail-through rule for this is a MAC address check.  The system checks against a host list which has our printers, access control devices, etc and if on the list they are also assigned the local data VLAN.  Ultimate fail-through put anything that fails these tests into a separate visitors VLAN which can't access any corporate stuff and is basically straight to internet.

I was called out on a Friday evening because one one of our areas could not access systems and servers.  A quick check showed that everyone regardless of machine type was failing through to the visitor VLAN.  I was forced to remove aaa authentication from the switches in that area to restore service.  I'm now looking at troubleshooting and I can't find anything obviously wrong.   We made no changes that day and checking through all the rules, roles & policies I can find nothing obviously wrong.  The PC's in that area will work anywhere else as they are running the service and have the certs.  Any machine I take down there fails the same way.  I've gone through all the switch configs and they appear the same as the 5 other areas on site.  The only clue I have is that when I check the logs during testing is that I can see the PC fail through to the visitor network, then it passes and gets the corporate VLAN, then fails through again.  This cycle seems to repeat continuously (see pic attached).

The server was updated a few days ago to 6.12.5.306119, well after the issue surfaced.  None of the other areas set up in the system are affected so I assume it's something in that specific rule set since I can't see anything different in the switch configs or the PC's.  Anyone run into something similar with a constant flipping between authorised and failing?  I'm stumped.

Hi there.  We have a large industrial site with Clearpass handling authentication and automatic VLAN selection for each area.  We have the system check for 802.1x details and it checks those against our domain controller machine list.  A GPO pushes out the certificates and makes sure the 802.1x service is on by default on our machines.  If the check is passed the machine is assigned the local data VLAN for that area.  The fail-through rule for this is a MAC address check.  The system checks against a host list which has our printers, access control devices, etc and if on the list they are also assigned the local data VLAN.  Ultimate fail-through put anything that fails these tests into a separate visitors VLAN which can't access any corporate stuff and is basically straight to internet.

I was called out on a Friday evening because one one of our areas could not access systems and servers.  A quick check showed that everyone regardless of machine type was failing through to the visitor VLAN.  I was forced to remove aaa authentication from the switches in that area to restore service.  I'm now looking at troubleshooting and I can't find anything obviously wrong.   We made no changes that day and checking through all the rules, roles & policies I can find nothing obviously wrong.  The PC's in that area will work anywhere else as they are running the service and have the certs.  Any machine I take down there fails the same way.  I've gone through all the switch configs and they appear the same as the 5 other areas on site.  The only clue I have is that when I check the logs during testing is that I can see the PC fail through to the visitor network, then it passes and gets the corporate VLAN, then fails through again.  This cycle seems to repeat continuously (see pic attached).

The server was updated a few days ago to 6.12.5.306119, well after the issue surfaced.  None of the other areas set up in the system are affected so I assume it's something in that specific rule set since I can't see anything different in the switch configs or the PC's.  Anyone run into something similar with a constant flipping between authorised and failing?  I'm stumped.

Hi. 

Check the logs on the switch and maybe enable 802.1x debugging. Lioks like something in send by enforcement profile is not what switch is expecting.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 20, 2025 10:52 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi there.  We have a large industrial site with Clearpass handling authentication and automatic VLAN selection for each area.  We have the system check for 802.1x details and it checks those against our domain controller machine list.  A GPO pushes out the certificates and makes sure the 802.1x service is on by default on our machines.  If the check is passed the machine is assigned the local data VLAN for that area.  The fail-through rule for this is a MAC address check.  The system checks against a host list which has our printers, access control devices, etc and if on the list they are also assigned the local data VLAN.  Ultimate fail-through put anything that fails these tests into a separate visitors VLAN which can't access any corporate stuff and is basically straight to internet.

I was called out on a Friday evening because one one of our areas could not access systems and servers.  A quick check showed that everyone regardless of machine type was failing through to the visitor VLAN.  I was forced to remove aaa authentication from the switches in that area to restore service.  I'm now looking at troubleshooting and I can't find anything obviously wrong.   We made no changes that day and checking through all the rules, roles & policies I can find nothing obviously wrong.  The PC's in that area will work anywhere else as they are running the service and have the certs.  Any machine I take down there fails the same way.  I've gone through all the switch configs and they appear the same as the 5 other areas on site.  The only clue I have is that when I check the logs during testing is that I can see the PC fail through to the visitor network, then it passes and gets the corporate VLAN, then fails through again.  This cycle seems to repeat continuously (see pic attached).

The server was updated a few days ago to 6.12.5.306119, well after the issue surfaced.  None of the other areas set up in the system are affected so I assume it's something in that specific rule set since I can't see anything different in the switch configs or the PC's.  Anyone run into something similar with a constant flipping between authorised and failing?  I'm stumped.

Thanks for the replies, gents.  The VLANs exist and are correct.  I've tried to get debugging working but there's no guide to actually debugging for the AOS used on the 2930M's that our network uses.  Every search for 802.1x or NAC debugging takes me straight to the HPE page for debugging 802.1x on OS-CX switches instead.

What I have discovered today is that it appears to be the MAC side of things that's causing the issue.  The actual 802.1x is working fine.  The port-access authenticator assigns the corporate VLAN correctly and the PC works.  If I turn on the MAC-based check that comes after it in the config the port fails through to the visitor VLAN immediately.  The check can't match the corporate PC's to a phone or printer MAC so it fails.  Why it's doing the check even through the first test has passed is what I don't know.  This config works elsewhere on the network but something must be different for the rule set here.  It is not stopping the checks and moves on to the MAC check which will always fail the corporate PC's.

For reference, here's an example of the aaa config on our switch ports.  

aaa accounting commands stop-only tacacs
aaa accounting exec start-stop tacacs
aaa accounting system stop-only tacacs
aaa authentication login privilege-mode
aaa authentication ssh login tacacs
aaa authentication ssh enable tacacs
aaa authentication port-access eap-radius
aaa port-access authenticator 1
aaa port-access authenticator 1 client-limit 2
aaa port-access authenticator active aaa port-access mac-based 1

aaa port-access mac-based 2 mac-pin

aaa port-access mac-based addr-format multi-colon

Original Message:
Sent: Jun 22, 2025 02:14 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi. 

Check the logs on the switch and maybe enable 802.1x debugging. Lioks like something in send by enforcement profile is not what switch is expecting.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 20, 2025 10:52 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi there.  We have a large industrial site with Clearpass handling authentication and automatic VLAN selection for each area.  We have the system check for 802.1x details and it checks those against our domain controller machine list.  A GPO pushes out the certificates and makes sure the 802.1x service is on by default on our machines.  If the check is passed the machine is assigned the local data VLAN for that area.  The fail-through rule for this is a MAC address check.  The system checks against a host list which has our printers, access control devices, etc and if on the list they are also assigned the local data VLAN.  Ultimate fail-through put anything that fails these tests into a separate visitors VLAN which can't access any corporate stuff and is basically straight to internet.

I was called out on a Friday evening because one one of our areas could not access systems and servers.  A quick check showed that everyone regardless of machine type was failing through to the visitor VLAN.  I was forced to remove aaa authentication from the switches in that area to restore service.  I'm now looking at troubleshooting and I can't find anything obviously wrong.   We made no changes that day and checking through all the rules, roles & policies I can find nothing obviously wrong.  The PC's in that area will work anywhere else as they are running the service and have the certs.  Any machine I take down there fails the same way.  I've gone through all the switch configs and they appear the same as the 5 other areas on site.  The only clue I have is that when I check the logs during testing is that I can see the PC fail through to the visitor network, then it passes and gets the corporate VLAN, then fails through again.  This cycle seems to repeat continuously (see pic attached).

The server was updated a few days ago to 6.12.5.306119, well after the issue surfaced.  None of the other areas set up in the system are affected so I assume it's something in that specific rule set since I can't see anything different in the switch configs or the PC's.  Anyone run into something similar with a constant flipping between authorised and failing?  I'm stumped.

Thanks for the replies, gents.  The VLANs exist and are correct.  I've tried to get debugging working but there's no guide to actually debugging for the AOS used on the 2930M's that our network uses.  Every search for 802.1x or NAC debugging takes me straight to the HPE page for debugging 802.1x on OS-CX switches instead.

What I have discovered today is that it appears to be the MAC side of things that's causing the issue.  The actual 802.1x is working fine.  The port-access authenticator assigns the corporate VLAN correctly and the PC works.  If I turn on the MAC-based check that comes after it in the config the port fails through to the visitor VLAN immediately.  The check can't match the corporate PC's to a phone or printer MAC so it fails.  Why it's doing the check even through the first test has passed is what I don't know.  This config works elsewhere on the network but something must be different for the rule set here.  It is not stopping the checks and moves on to the MAC check which will always fail the corporate PC's.

For reference, here's an example of the aaa config on our switch ports.  

aaa accounting commands stop-only tacacs
aaa accounting exec start-stop tacacs
aaa accounting system stop-only tacacs
aaa authentication login privilege-mode
aaa authentication ssh login tacacs
aaa authentication ssh enable tacacs
aaa authentication port-access eap-radius
aaa port-access authenticator 1
aaa port-access authenticator 1 client-limit 2
aaa port-access authenticator active aaa port-access mac-based 1

aaa port-access mac-based 2 mac-pin

aaa port-access mac-based addr-format multi-colon

Original Message:
Sent: Jun 22, 2025 02:14 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi. 

Check the logs on the switch and maybe enable 802.1x debugging. Lioks like something in send by enforcement profile is not what switch is expecting.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 20, 2025 10:52 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi there.  We have a large industrial site with Clearpass handling authentication and automatic VLAN selection for each area.  We have the system check for 802.1x details and it checks those against our domain controller machine list.  A GPO pushes out the certificates and makes sure the 802.1x service is on by default on our machines.  If the check is passed the machine is assigned the local data VLAN for that area.  The fail-through rule for this is a MAC address check.  The system checks against a host list which has our printers, access control devices, etc and if on the list they are also assigned the local data VLAN.  Ultimate fail-through put anything that fails these tests into a separate visitors VLAN which can't access any corporate stuff and is basically straight to internet.

I was called out on a Friday evening because one one of our areas could not access systems and servers.  A quick check showed that everyone regardless of machine type was failing through to the visitor VLAN.  I was forced to remove aaa authentication from the switches in that area to restore service.  I'm now looking at troubleshooting and I can't find anything obviously wrong.   We made no changes that day and checking through all the rules, roles & policies I can find nothing obviously wrong.  The PC's in that area will work anywhere else as they are running the service and have the certs.  Any machine I take down there fails the same way.  I've gone through all the switch configs and they appear the same as the 5 other areas on site.  The only clue I have is that when I check the logs during testing is that I can see the PC fail through to the visitor network, then it passes and gets the corporate VLAN, then fails through again.  This cycle seems to repeat continuously (see pic attached).

The server was updated a few days ago to 6.12.5.306119, well after the issue surfaced.  None of the other areas set up in the system are affected so I assume it's something in that specific rule set since I can't see anything different in the switch configs or the PC's.  Anyone run into something similar with a constant flipping between authorised and failing?  I'm stumped.

Hi Kenny,
heir is an example to enable debugging on an AOS-S switch. You can enable it globally for all ports or for a specific port.

deb security port-access authenticator include port 2
deb security port-access mac-based include port 2

You can also use debug security radius-server. However, this command is used for all sessions, depending on what is currently going on on the switch, it can become unmanageable.

It is more easier if you send debug output to a separate CLI session. Just open an additional session and enter debug destination session there.

The port configuration looks OK. I would also set auth-order and auth-priority explicitly and specify the unauth-vid.
I always use the following config:

aaa port-access authenticator 2
aaa port-access mac-based 2
aaa port-access mac-based 2 mac-pin
aaa port-access mac-based 2 unauth-vid 500
aaa port-access 2 auth-order authenticator mac-based
aaa port-access 2 auth-priority authenticator mac-based
aaa port-access 2 controlled-direction in

Thanks for the replies, gents.  The VLANs exist and are correct.  I've tried to get debugging working but there's no guide to actually debugging for the AOS used on the 2930M's that our network uses.  Every search for 802.1x or NAC debugging takes me straight to the HPE page for debugging 802.1x on OS-CX switches instead.

What I have discovered today is that it appears to be the MAC side of things that's causing the issue.  The actual 802.1x is working fine.  The port-access authenticator assigns the corporate VLAN correctly and the PC works.  If I turn on the MAC-based check that comes after it in the config the port fails through to the visitor VLAN immediately.  The check can't match the corporate PC's to a phone or printer MAC so it fails.  Why it's doing the check even through the first test has passed is what I don't know.  This config works elsewhere on the network but something must be different for the rule set here.  It is not stopping the checks and moves on to the MAC check which will always fail the corporate PC's.

For reference, here's an example of the aaa config on our switch ports.  

aaa accounting commands stop-only tacacs
aaa accounting exec start-stop tacacs
aaa accounting system stop-only tacacs
aaa authentication login privilege-mode
aaa authentication ssh login tacacs
aaa authentication ssh enable tacacs
aaa authentication port-access eap-radius
aaa port-access authenticator 1
aaa port-access authenticator 1 client-limit 2
aaa port-access authenticator active aaa port-access mac-based 1

aaa port-access mac-based 2 mac-pin

aaa port-access mac-based addr-format multi-colon

Original Message:
Sent: Jun 22, 2025 02:14 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi. 

Check the logs on the switch and maybe enable 802.1x debugging. Lioks like something in send by enforcement profile is not what switch is expecting.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 20, 2025 10:52 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi there.  We have a large industrial site with Clearpass handling authentication and automatic VLAN selection for each area.  We have the system check for 802.1x details and it checks those against our domain controller machine list.  A GPO pushes out the certificates and makes sure the 802.1x service is on by default on our machines.  If the check is passed the machine is assigned the local data VLAN for that area.  The fail-through rule for this is a MAC address check.  The system checks against a host list which has our printers, access control devices, etc and if on the list they are also assigned the local data VLAN.  Ultimate fail-through put anything that fails these tests into a separate visitors VLAN which can't access any corporate stuff and is basically straight to internet.

I was called out on a Friday evening because one one of our areas could not access systems and servers.  A quick check showed that everyone regardless of machine type was failing through to the visitor VLAN.  I was forced to remove aaa authentication from the switches in that area to restore service.  I'm now looking at troubleshooting and I can't find anything obviously wrong.   We made no changes that day and checking through all the rules, roles & policies I can find nothing obviously wrong.  The PC's in that area will work anywhere else as they are running the service and have the certs.  Any machine I take down there fails the same way.  I've gone through all the switch configs and they appear the same as the 5 other areas on site.  The only clue I have is that when I check the logs during testing is that I can see the PC fail through to the visitor network, then it passes and gets the corporate VLAN, then fails through again.  This cycle seems to repeat continuously (see pic attached).

The server was updated a few days ago to 6.12.5.306119, well after the issue surfaced.  None of the other areas set up in the system are affected so I assume it's something in that specific rule set since I can't see anything different in the switch configs or the PC's.  Anyone run into something similar with a constant flipping between authorised and failing?  I'm stumped.

Hi again.  Thanks for the correct debug commands gents, they worked fine.  I've attached a sample text showing the loop I'm seeing.

Waldemar - thanks for the config, I'll look into adding the additional commands you recommend.

Willem - we are not hitting our client limits.  I've got around 800 spare access licenses and 1100 spare OnGuard licenses so it's not that.  It looks like the switch is doing exactly what it is being told to do.  It seems to be authenticating using 802.1x and then being de-authenticated by the MAC check immediately after it.  I can see the VLAN flicking constantly between the corporate (501) and the visitor (95).

I have attached a copy of the debug log when I activated MAC authentication again as well as 802.1x on the port.  It seems to confirm that the user is authenticating properly and then the MAC address fail through check runs immediately after and kicks it into the visitor VLAN.  I don't know why the MAC check is running after a successful authentication but that seems to be the problem.

My network covers a large industrial site of 6 square miles so is split into 6 network areas.  Each area has its own distributions switches (8320x's) feeding the access switches (2930M's) and each area has its own group of data, voice and visitor VLANs.  Clearpass has profiles and rules for each VLAN in each area and the problem only affects this one area.  The other areas have near identical configuration on the switches and rules with only the VLAN numbers changing.   

Hi Kenny,
heir is an example to enable debugging on an AOS-S switch. You can enable it globally for all ports or for a specific port.

deb security port-access authenticator include port 2
deb security port-access mac-based include port 2

You can also use debug security radius-server. However, this command is used for all sessions, depending on what is currently going on on the switch, it can become unmanageable.

It is more easier if you send debug output to a separate CLI session. Just open an additional session and enter debug destination session there.

The port configuration looks OK. I would also set auth-order and auth-priority explicitly and specify the unauth-vid.
I always use the following config:

aaa port-access authenticator 2
aaa port-access mac-based 2
aaa port-access mac-based 2 mac-pin
aaa port-access mac-based 2 unauth-vid 500
aaa port-access 2 auth-order authenticator mac-based
aaa port-access 2 auth-priority authenticator mac-based
aaa port-access 2 controlled-direction in

Thanks for the replies, gents.  The VLANs exist and are correct.  I've tried to get debugging working but there's no guide to actually debugging for the AOS used on the 2930M's that our network uses.  Every search for 802.1x or NAC debugging takes me straight to the HPE page for debugging 802.1x on OS-CX switches instead.

What I have discovered today is that it appears to be the MAC side of things that's causing the issue.  The actual 802.1x is working fine.  The port-access authenticator assigns the corporate VLAN correctly and the PC works.  If I turn on the MAC-based check that comes after it in the config the port fails through to the visitor VLAN immediately.  The check can't match the corporate PC's to a phone or printer MAC so it fails.  Why it's doing the check even through the first test has passed is what I don't know.  This config works elsewhere on the network but something must be different for the rule set here.  It is not stopping the checks and moves on to the MAC check which will always fail the corporate PC's.

For reference, here's an example of the aaa config on our switch ports.  

aaa accounting commands stop-only tacacs
aaa accounting exec start-stop tacacs
aaa accounting system stop-only tacacs
aaa authentication login privilege-mode
aaa authentication ssh login tacacs
aaa authentication ssh enable tacacs
aaa authentication port-access eap-radius
aaa port-access authenticator 1
aaa port-access authenticator 1 client-limit 2
aaa port-access authenticator active aaa port-access mac-based 1

aaa port-access mac-based 2 mac-pin

aaa port-access mac-based addr-format multi-colon

Original Message:
Sent: Jun 22, 2025 02:14 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi. 

Check the logs on the switch and maybe enable 802.1x debugging. Lioks like something in send by enforcement profile is not what switch is expecting.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 20, 2025 10:52 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi there.  We have a large industrial site with Clearpass handling authentication and automatic VLAN selection for each area.  We have the system check for 802.1x details and it checks those against our domain controller machine list.  A GPO pushes out the certificates and makes sure the 802.1x service is on by default on our machines.  If the check is passed the machine is assigned the local data VLAN for that area.  The fail-through rule for this is a MAC address check.  The system checks against a host list which has our printers, access control devices, etc and if on the list they are also assigned the local data VLAN.  Ultimate fail-through put anything that fails these tests into a separate visitors VLAN which can't access any corporate stuff and is basically straight to internet.

I was called out on a Friday evening because one one of our areas could not access systems and servers.  A quick check showed that everyone regardless of machine type was failing through to the visitor VLAN.  I was forced to remove aaa authentication from the switches in that area to restore service.  I'm now looking at troubleshooting and I can't find anything obviously wrong.   We made no changes that day and checking through all the rules, roles & policies I can find nothing obviously wrong.  The PC's in that area will work anywhere else as they are running the service and have the certs.  Any machine I take down there fails the same way.  I've gone through all the switch configs and they appear the same as the 5 other areas on site.  The only clue I have is that when I check the logs during testing is that I can see the PC fail through to the visitor network, then it passes and gets the corporate VLAN, then fails through again.  This cycle seems to repeat continuously (see pic attached).

The server was updated a few days ago to 6.12.5.306119, well after the issue surfaced.  None of the other areas set up in the system are affected so I assume it's something in that specific rule set since I can't see anything different in the switch configs or the PC's.  Anyone run into something similar with a constant flipping between authorised and failing?  I'm stumped.

In the logs I see the message "Deauthentication request received".

Can you check in the access tracker if ClearPass is sending a CoA? In the access tracker details there should be an additional tab when a CoA is send

Hi again.  Thanks for the correct debug commands gents, they worked fine.  I've attached a sample text showing the loop I'm seeing.

Waldemar - thanks for the config, I'll look into adding the additional commands you recommend.

Willem - we are not hitting our client limits.  I've got around 800 spare access licenses and 1100 spare OnGuard licenses so it's not that.  It looks like the switch is doing exactly what it is being told to do.  It seems to be authenticating using 802.1x and then being de-authenticated by the MAC check immediately after it.  I can see the VLAN flicking constantly between the corporate (501) and the visitor (95).

I have attached a copy of the debug log when I activated MAC authentication again as well as 802.1x on the port.  It seems to confirm that the user is authenticating properly and then the MAC address fail through check runs immediately after and kicks it into the visitor VLAN.  I don't know why the MAC check is running after a successful authentication but that seems to be the problem.

My network covers a large industrial site of 6 square miles so is split into 6 network areas.  Each area has its own distributions switches (8320x's) feeding the access switches (2930M's) and each area has its own group of data, voice and visitor VLANs.  Clearpass has profiles and rules for each VLAN in each area and the problem only affects this one area.  The other areas have near identical configuration on the switches and rules with only the VLAN numbers changing.   

Hi Kenny,
heir is an example to enable debugging on an AOS-S switch. You can enable it globally for all ports or for a specific port.

deb security port-access authenticator include port 2
deb security port-access mac-based include port 2

You can also use debug security radius-server. However, this command is used for all sessions, depending on what is currently going on on the switch, it can become unmanageable.

It is more easier if you send debug output to a separate CLI session. Just open an additional session and enter debug destination session there.

The port configuration looks OK. I would also set auth-order and auth-priority explicitly and specify the unauth-vid.
I always use the following config:

aaa port-access authenticator 2
aaa port-access mac-based 2
aaa port-access mac-based 2 mac-pin
aaa port-access mac-based 2 unauth-vid 500
aaa port-access 2 auth-order authenticator mac-based
aaa port-access 2 auth-priority authenticator mac-based
aaa port-access 2 controlled-direction in

Thanks for the replies, gents.  The VLANs exist and are correct.  I've tried to get debugging working but there's no guide to actually debugging for the AOS used on the 2930M's that our network uses.  Every search for 802.1x or NAC debugging takes me straight to the HPE page for debugging 802.1x on OS-CX switches instead.

What I have discovered today is that it appears to be the MAC side of things that's causing the issue.  The actual 802.1x is working fine.  The port-access authenticator assigns the corporate VLAN correctly and the PC works.  If I turn on the MAC-based check that comes after it in the config the port fails through to the visitor VLAN immediately.  The check can't match the corporate PC's to a phone or printer MAC so it fails.  Why it's doing the check even through the first test has passed is what I don't know.  This config works elsewhere on the network but something must be different for the rule set here.  It is not stopping the checks and moves on to the MAC check which will always fail the corporate PC's.

For reference, here's an example of the aaa config on our switch ports.  

aaa accounting commands stop-only tacacs
aaa accounting exec start-stop tacacs
aaa accounting system stop-only tacacs
aaa authentication login privilege-mode
aaa authentication ssh login tacacs
aaa authentication ssh enable tacacs
aaa authentication port-access eap-radius
aaa port-access authenticator 1
aaa port-access authenticator 1 client-limit 2
aaa port-access authenticator active aaa port-access mac-based 1

aaa port-access mac-based 2 mac-pin

aaa port-access mac-based addr-format multi-colon

Original Message:
Sent: Jun 22, 2025 02:14 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi. 

Check the logs on the switch and maybe enable 802.1x debugging. Lioks like something in send by enforcement profile is not what switch is expecting.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 20, 2025 10:52 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi there.  We have a large industrial site with Clearpass handling authentication and automatic VLAN selection for each area.  We have the system check for 802.1x details and it checks those against our domain controller machine list.  A GPO pushes out the certificates and makes sure the 802.1x service is on by default on our machines.  If the check is passed the machine is assigned the local data VLAN for that area.  The fail-through rule for this is a MAC address check.  The system checks against a host list which has our printers, access control devices, etc and if on the list they are also assigned the local data VLAN.  Ultimate fail-through put anything that fails these tests into a separate visitors VLAN which can't access any corporate stuff and is basically straight to internet.

I was called out on a Friday evening because one one of our areas could not access systems and servers.  A quick check showed that everyone regardless of machine type was failing through to the visitor VLAN.  I was forced to remove aaa authentication from the switches in that area to restore service.  I'm now looking at troubleshooting and I can't find anything obviously wrong.   We made no changes that day and checking through all the rules, roles & policies I can find nothing obviously wrong.  The PC's in that area will work anywhere else as they are running the service and have the certs.  Any machine I take down there fails the same way.  I've gone through all the switch configs and they appear the same as the 5 other areas on site.  The only clue I have is that when I check the logs during testing is that I can see the PC fail through to the visitor network, then it passes and gets the corporate VLAN, then fails through again.  This cycle seems to repeat continuously (see pic attached).

The server was updated a few days ago to 6.12.5.306119, well after the issue surfaced.  None of the other areas set up in the system are affected so I assume it's something in that specific rule set since I can't see anything different in the switch configs or the PC's.  Anyone run into something similar with a constant flipping between authorised and failing?  I'm stumped.

Hi Willem.  I do not see any change of authentication notification or additional tab in the access tracker.  The 802.1 and MAC notifications look like every other notification with no additions.  It authenticates correctly using 802.1x, then it authenticates twice using the MAC rule which fails it through to the visitor VLAN.  For some reason it is authenticating against the MAC twice every time but I'm not sure why.  They look identical.

In the logs I see the message "Deauthentication request received".

Can you check in the access tracker if ClearPass is sending a CoA? In the access tracker details there should be an additional tab when a CoA is send

Hi again.  Thanks for the correct debug commands gents, they worked fine.  I've attached a sample text showing the loop I'm seeing.

Waldemar - thanks for the config, I'll look into adding the additional commands you recommend.

Willem - we are not hitting our client limits.  I've got around 800 spare access licenses and 1100 spare OnGuard licenses so it's not that.  It looks like the switch is doing exactly what it is being told to do.  It seems to be authenticating using 802.1x and then being de-authenticated by the MAC check immediately after it.  I can see the VLAN flicking constantly between the corporate (501) and the visitor (95).

I have attached a copy of the debug log when I activated MAC authentication again as well as 802.1x on the port.  It seems to confirm that the user is authenticating properly and then the MAC address fail through check runs immediately after and kicks it into the visitor VLAN.  I don't know why the MAC check is running after a successful authentication but that seems to be the problem.

My network covers a large industrial site of 6 square miles so is split into 6 network areas.  Each area has its own distributions switches (8320x's) feeding the access switches (2930M's) and each area has its own group of data, voice and visitor VLANs.  Clearpass has profiles and rules for each VLAN in each area and the problem only affects this one area.  The other areas have near identical configuration on the switches and rules with only the VLAN numbers changing.   

Hi Kenny,
heir is an example to enable debugging on an AOS-S switch. You can enable it globally for all ports or for a specific port.

deb security port-access authenticator include port 2
deb security port-access mac-based include port 2

You can also use debug security radius-server. However, this command is used for all sessions, depending on what is currently going on on the switch, it can become unmanageable.

It is more easier if you send debug output to a separate CLI session. Just open an additional session and enter debug destination session there.

The port configuration looks OK. I would also set auth-order and auth-priority explicitly and specify the unauth-vid.
I always use the following config:

aaa port-access authenticator 2
aaa port-access mac-based 2
aaa port-access mac-based 2 mac-pin
aaa port-access mac-based 2 unauth-vid 500
aaa port-access 2 auth-order authenticator mac-based
aaa port-access 2 auth-priority authenticator mac-based
aaa port-access 2 controlled-direction in

Thanks for the replies, gents.  The VLANs exist and are correct.  I've tried to get debugging working but there's no guide to actually debugging for the AOS used on the 2930M's that our network uses.  Every search for 802.1x or NAC debugging takes me straight to the HPE page for debugging 802.1x on OS-CX switches instead.

What I have discovered today is that it appears to be the MAC side of things that's causing the issue.  The actual 802.1x is working fine.  The port-access authenticator assigns the corporate VLAN correctly and the PC works.  If I turn on the MAC-based check that comes after it in the config the port fails through to the visitor VLAN immediately.  The check can't match the corporate PC's to a phone or printer MAC so it fails.  Why it's doing the check even through the first test has passed is what I don't know.  This config works elsewhere on the network but something must be different for the rule set here.  It is not stopping the checks and moves on to the MAC check which will always fail the corporate PC's.

For reference, here's an example of the aaa config on our switch ports.  

aaa accounting commands stop-only tacacs
aaa accounting exec start-stop tacacs
aaa accounting system stop-only tacacs
aaa authentication login privilege-mode
aaa authentication ssh login tacacs
aaa authentication ssh enable tacacs
aaa authentication port-access eap-radius
aaa port-access authenticator 1
aaa port-access authenticator 1 client-limit 2
aaa port-access authenticator active aaa port-access mac-based 1

aaa port-access mac-based 2 mac-pin

aaa port-access mac-based addr-format multi-colon

Original Message:
Sent: Jun 22, 2025 02:14 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi. 

Check the logs on the switch and maybe enable 802.1x debugging. Lioks like something in send by enforcement profile is not what switch is expecting.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 20, 2025 10:52 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi there.  We have a large industrial site with Clearpass handling authentication and automatic VLAN selection for each area.  We have the system check for 802.1x details and it checks those against our domain controller machine list.  A GPO pushes out the certificates and makes sure the 802.1x service is on by default on our machines.  If the check is passed the machine is assigned the local data VLAN for that area.  The fail-through rule for this is a MAC address check.  The system checks against a host list which has our printers, access control devices, etc and if on the list they are also assigned the local data VLAN.  Ultimate fail-through put anything that fails these tests into a separate visitors VLAN which can't access any corporate stuff and is basically straight to internet.

I was called out on a Friday evening because one one of our areas could not access systems and servers.  A quick check showed that everyone regardless of machine type was failing through to the visitor VLAN.  I was forced to remove aaa authentication from the switches in that area to restore service.  I'm now looking at troubleshooting and I can't find anything obviously wrong.   We made no changes that day and checking through all the rules, roles & policies I can find nothing obviously wrong.  The PC's in that area will work anywhere else as they are running the service and have the certs.  Any machine I take down there fails the same way.  I've gone through all the switch configs and they appear the same as the 5 other areas on site.  The only clue I have is that when I check the logs during testing is that I can see the PC fail through to the visitor network, then it passes and gets the corporate VLAN, then fails through again.  This cycle seems to repeat continuously (see pic attached).

The server was updated a few days ago to 6.12.5.306119, well after the issue surfaced.  None of the other areas set up in the system are affected so I assume it's something in that specific rule set since I can't see anything different in the switch configs or the PC's.  Anyone run into something similar with a constant flipping between authorised and failing?  I'm stumped.

The log doesn't show CoA.

Can you enable more debug logs on the switch. Just enable all the security logs for now

debug securitydebug destination session

Hi Willem.  I do not see any change of authentication notification or additional tab in the access tracker.  The 802.1 and MAC notifications look like every other notification with no additions.  It authenticates correctly using 802.1x, then it authenticates twice using the MAC rule which fails it through to the visitor VLAN.  For some reason it is authenticating against the MAC twice every time but I'm not sure why.  They look identical.

In the logs I see the message "Deauthentication request received".

Can you check in the access tracker if ClearPass is sending a CoA? In the access tracker details there should be an additional tab when a CoA is send

Hi again.  Thanks for the correct debug commands gents, they worked fine.  I've attached a sample text showing the loop I'm seeing.

Waldemar - thanks for the config, I'll look into adding the additional commands you recommend.

Willem - we are not hitting our client limits.  I've got around 800 spare access licenses and 1100 spare OnGuard licenses so it's not that.  It looks like the switch is doing exactly what it is being told to do.  It seems to be authenticating using 802.1x and then being de-authenticated by the MAC check immediately after it.  I can see the VLAN flicking constantly between the corporate (501) and the visitor (95).

I have attached a copy of the debug log when I activated MAC authentication again as well as 802.1x on the port.  It seems to confirm that the user is authenticating properly and then the MAC address fail through check runs immediately after and kicks it into the visitor VLAN.  I don't know why the MAC check is running after a successful authentication but that seems to be the problem.

My network covers a large industrial site of 6 square miles so is split into 6 network areas.  Each area has its own distributions switches (8320x's) feeding the access switches (2930M's) and each area has its own group of data, voice and visitor VLANs.  Clearpass has profiles and rules for each VLAN in each area and the problem only affects this one area.  The other areas have near identical configuration on the switches and rules with only the VLAN numbers changing.   

Hi Kenny,
heir is an example to enable debugging on an AOS-S switch. You can enable it globally for all ports or for a specific port.

deb security port-access authenticator include port 2
deb security port-access mac-based include port 2

You can also use debug security radius-server. However, this command is used for all sessions, depending on what is currently going on on the switch, it can become unmanageable.

It is more easier if you send debug output to a separate CLI session. Just open an additional session and enter debug destination session there.

The port configuration looks OK. I would also set auth-order and auth-priority explicitly and specify the unauth-vid.
I always use the following config:

aaa port-access authenticator 2
aaa port-access mac-based 2
aaa port-access mac-based 2 mac-pin
aaa port-access mac-based 2 unauth-vid 500
aaa port-access 2 auth-order authenticator mac-based
aaa port-access 2 auth-priority authenticator mac-based
aaa port-access 2 controlled-direction in

Thanks for the replies, gents.  The VLANs exist and are correct.  I've tried to get debugging working but there's no guide to actually debugging for the AOS used on the 2930M's that our network uses.  Every search for 802.1x or NAC debugging takes me straight to the HPE page for debugging 802.1x on OS-CX switches instead.

What I have discovered today is that it appears to be the MAC side of things that's causing the issue.  The actual 802.1x is working fine.  The port-access authenticator assigns the corporate VLAN correctly and the PC works.  If I turn on the MAC-based check that comes after it in the config the port fails through to the visitor VLAN immediately.  The check can't match the corporate PC's to a phone or printer MAC so it fails.  Why it's doing the check even through the first test has passed is what I don't know.  This config works elsewhere on the network but something must be different for the rule set here.  It is not stopping the checks and moves on to the MAC check which will always fail the corporate PC's.

For reference, here's an example of the aaa config on our switch ports.  

aaa accounting commands stop-only tacacs
aaa accounting exec start-stop tacacs
aaa accounting system stop-only tacacs
aaa authentication login privilege-mode
aaa authentication ssh login tacacs
aaa authentication ssh enable tacacs
aaa authentication port-access eap-radius
aaa port-access authenticator 1
aaa port-access authenticator 1 client-limit 2
aaa port-access authenticator active aaa port-access mac-based 1

aaa port-access mac-based 2 mac-pin

aaa port-access mac-based addr-format multi-colon

Original Message:
Sent: Jun 22, 2025 02:14 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi. 

Check the logs on the switch and maybe enable 802.1x debugging. Lioks like something in send by enforcement profile is not what switch is expecting.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 20, 2025 10:52 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi there.  We have a large industrial site with Clearpass handling authentication and automatic VLAN selection for each area.  We have the system check for 802.1x details and it checks those against our domain controller machine list.  A GPO pushes out the certificates and makes sure the 802.1x service is on by default on our machines.  If the check is passed the machine is assigned the local data VLAN for that area.  The fail-through rule for this is a MAC address check.  The system checks against a host list which has our printers, access control devices, etc and if on the list they are also assigned the local data VLAN.  Ultimate fail-through put anything that fails these tests into a separate visitors VLAN which can't access any corporate stuff and is basically straight to internet.

I was called out on a Friday evening because one one of our areas could not access systems and servers.  A quick check showed that everyone regardless of machine type was failing through to the visitor VLAN.  I was forced to remove aaa authentication from the switches in that area to restore service.  I'm now looking at troubleshooting and I can't find anything obviously wrong.   We made no changes that day and checking through all the rules, roles & policies I can find nothing obviously wrong.  The PC's in that area will work anywhere else as they are running the service and have the certs.  Any machine I take down there fails the same way.  I've gone through all the switch configs and they appear the same as the 5 other areas on site.  The only clue I have is that when I check the logs during testing is that I can see the PC fail through to the visitor network, then it passes and gets the corporate VLAN, then fails through again.  This cycle seems to repeat continuously (see pic attached).

The server was updated a few days ago to 6.12.5.306119, well after the issue surfaced.  None of the other areas set up in the system are affected so I assume it's something in that specific rule set since I can't see anything different in the switch configs or the PC's.  Anyone run into something similar with a constant flipping between authorised and failing?  I'm stumped.

Hi again.  I've attached the debug output from the moment I applied the config to the port again to turn on authentication.

One thing that might be of interest is that I just applied auth-order and auth-priority as suggested above by Lord.  I set it to authenticator and then MAC and turned on the port.  The result was that the port stuck on the default 'fake' VLAN 666 which all our ports are set to when nothing is plugged in.  The access tracker shows that 802.1x authentication worked but the VLAN does not get applied.

The log doesn't show CoA.

Can you enable more debug logs on the switch. Just enable all the security logs for now

debug securitydebug destination session

Hi Willem.  I do not see any change of authentication notification or additional tab in the access tracker.  The 802.1 and MAC notifications look like every other notification with no additions.  It authenticates correctly using 802.1x, then it authenticates twice using the MAC rule which fails it through to the visitor VLAN.  For some reason it is authenticating against the MAC twice every time but I'm not sure why.  They look identical.

In the logs I see the message "Deauthentication request received".

Can you check in the access tracker if ClearPass is sending a CoA? In the access tracker details there should be an additional tab when a CoA is send

Hi again.  Thanks for the correct debug commands gents, they worked fine.  I've attached a sample text showing the loop I'm seeing.

Waldemar - thanks for the config, I'll look into adding the additional commands you recommend.

Willem - we are not hitting our client limits.  I've got around 800 spare access licenses and 1100 spare OnGuard licenses so it's not that.  It looks like the switch is doing exactly what it is being told to do.  It seems to be authenticating using 802.1x and then being de-authenticated by the MAC check immediately after it.  I can see the VLAN flicking constantly between the corporate (501) and the visitor (95).

I have attached a copy of the debug log when I activated MAC authentication again as well as 802.1x on the port.  It seems to confirm that the user is authenticating properly and then the MAC address fail through check runs immediately after and kicks it into the visitor VLAN.  I don't know why the MAC check is running after a successful authentication but that seems to be the problem.

My network covers a large industrial site of 6 square miles so is split into 6 network areas.  Each area has its own distributions switches (8320x's) feeding the access switches (2930M's) and each area has its own group of data, voice and visitor VLANs.  Clearpass has profiles and rules for each VLAN in each area and the problem only affects this one area.  The other areas have near identical configuration on the switches and rules with only the VLAN numbers changing.   

Hi Kenny,
heir is an example to enable debugging on an AOS-S switch. You can enable it globally for all ports or for a specific port.

deb security port-access authenticator include port 2
deb security port-access mac-based include port 2

You can also use debug security radius-server. However, this command is used for all sessions, depending on what is currently going on on the switch, it can become unmanageable.

It is more easier if you send debug output to a separate CLI session. Just open an additional session and enter debug destination session there.

The port configuration looks OK. I would also set auth-order and auth-priority explicitly and specify the unauth-vid.
I always use the following config:

aaa port-access authenticator 2
aaa port-access mac-based 2
aaa port-access mac-based 2 mac-pin
aaa port-access mac-based 2 unauth-vid 500
aaa port-access 2 auth-order authenticator mac-based
aaa port-access 2 auth-priority authenticator mac-based
aaa port-access 2 controlled-direction in

Thanks for the replies, gents.  The VLANs exist and are correct.  I've tried to get debugging working but there's no guide to actually debugging for the AOS used on the 2930M's that our network uses.  Every search for 802.1x or NAC debugging takes me straight to the HPE page for debugging 802.1x on OS-CX switches instead.

What I have discovered today is that it appears to be the MAC side of things that's causing the issue.  The actual 802.1x is working fine.  The port-access authenticator assigns the corporate VLAN correctly and the PC works.  If I turn on the MAC-based check that comes after it in the config the port fails through to the visitor VLAN immediately.  The check can't match the corporate PC's to a phone or printer MAC so it fails.  Why it's doing the check even through the first test has passed is what I don't know.  This config works elsewhere on the network but something must be different for the rule set here.  It is not stopping the checks and moves on to the MAC check which will always fail the corporate PC's.

For reference, here's an example of the aaa config on our switch ports.  

aaa accounting commands stop-only tacacs
aaa accounting exec start-stop tacacs
aaa accounting system stop-only tacacs
aaa authentication login privilege-mode
aaa authentication ssh login tacacs
aaa authentication ssh enable tacacs
aaa authentication port-access eap-radius
aaa port-access authenticator 1
aaa port-access authenticator 1 client-limit 2
aaa port-access authenticator active aaa port-access mac-based 1

aaa port-access mac-based 2 mac-pin

aaa port-access mac-based addr-format multi-colon

Original Message:
Sent: Jun 22, 2025 02:14 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi. 

Check the logs on the switch and maybe enable 802.1x debugging. Lioks like something in send by enforcement profile is not what switch is expecting.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 20, 2025 10:52 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi there.  We have a large industrial site with Clearpass handling authentication and automatic VLAN selection for each area.  We have the system check for 802.1x details and it checks those against our domain controller machine list.  A GPO pushes out the certificates and makes sure the 802.1x service is on by default on our machines.  If the check is passed the machine is assigned the local data VLAN for that area.  The fail-through rule for this is a MAC address check.  The system checks against a host list which has our printers, access control devices, etc and if on the list they are also assigned the local data VLAN.  Ultimate fail-through put anything that fails these tests into a separate visitors VLAN which can't access any corporate stuff and is basically straight to internet.

I was called out on a Friday evening because one one of our areas could not access systems and servers.  A quick check showed that everyone regardless of machine type was failing through to the visitor VLAN.  I was forced to remove aaa authentication from the switches in that area to restore service.  I'm now looking at troubleshooting and I can't find anything obviously wrong.   We made no changes that day and checking through all the rules, roles & policies I can find nothing obviously wrong.  The PC's in that area will work anywhere else as they are running the service and have the certs.  Any machine I take down there fails the same way.  I've gone through all the switch configs and they appear the same as the 5 other areas on site.  The only clue I have is that when I check the logs during testing is that I can see the PC fail through to the visitor network, then it passes and gets the corporate VLAN, then fails through again.  This cycle seems to repeat continuously (see pic attached).

The server was updated a few days ago to 6.12.5.306119, well after the issue surfaced.  None of the other areas set up in the system are affected so I assume it's something in that specific rule set since I can't see anything different in the switch configs or the PC's.  Anyone run into something similar with a constant flipping between authorised and failing?  I'm stumped.

In majority of cases this mean that vlan in radius response is not found on switch.

Best, Gorazd

Hi again.  I've attached the debug output from the moment I applied the config to the port again to turn on authentication.

One thing that might be of interest is that I just applied auth-order and auth-priority as suggested above by Lord.  I set it to authenticator and then MAC and turned on the port.  The result was that the port stuck on the default 'fake' VLAN 666 which all our ports are set to when nothing is plugged in.  The access tracker shows that 802.1x authentication worked but the VLAN does not get applied.

The log doesn't show CoA.

Can you enable more debug logs on the switch. Just enable all the security logs for now

debug securitydebug destination session

Hi Willem.  I do not see any change of authentication notification or additional tab in the access tracker.  The 802.1 and MAC notifications look like every other notification with no additions.  It authenticates correctly using 802.1x, then it authenticates twice using the MAC rule which fails it through to the visitor VLAN.  For some reason it is authenticating against the MAC twice every time but I'm not sure why.  They look identical.

In the logs I see the message "Deauthentication request received".

Can you check in the access tracker if ClearPass is sending a CoA? In the access tracker details there should be an additional tab when a CoA is send

Hi again.  Thanks for the correct debug commands gents, they worked fine.  I've attached a sample text showing the loop I'm seeing.

Waldemar - thanks for the config, I'll look into adding the additional commands you recommend.

Willem - we are not hitting our client limits.  I've got around 800 spare access licenses and 1100 spare OnGuard licenses so it's not that.  It looks like the switch is doing exactly what it is being told to do.  It seems to be authenticating using 802.1x and then being de-authenticated by the MAC check immediately after it.  I can see the VLAN flicking constantly between the corporate (501) and the visitor (95).

I have attached a copy of the debug log when I activated MAC authentication again as well as 802.1x on the port.  It seems to confirm that the user is authenticating properly and then the MAC address fail through check runs immediately after and kicks it into the visitor VLAN.  I don't know why the MAC check is running after a successful authentication but that seems to be the problem.

My network covers a large industrial site of 6 square miles so is split into 6 network areas.  Each area has its own distributions switches (8320x's) feeding the access switches (2930M's) and each area has its own group of data, voice and visitor VLANs.  Clearpass has profiles and rules for each VLAN in each area and the problem only affects this one area.  The other areas have near identical configuration on the switches and rules with only the VLAN numbers changing.   

Hi Kenny,
heir is an example to enable debugging on an AOS-S switch. You can enable it globally for all ports or for a specific port.

deb security port-access authenticator include port 2
deb security port-access mac-based include port 2

You can also use debug security radius-server. However, this command is used for all sessions, depending on what is currently going on on the switch, it can become unmanageable.

It is more easier if you send debug output to a separate CLI session. Just open an additional session and enter debug destination session there.

The port configuration looks OK. I would also set auth-order and auth-priority explicitly and specify the unauth-vid.
I always use the following config:

aaa port-access authenticator 2
aaa port-access mac-based 2
aaa port-access mac-based 2 mac-pin
aaa port-access mac-based 2 unauth-vid 500
aaa port-access 2 auth-order authenticator mac-based
aaa port-access 2 auth-priority authenticator mac-based
aaa port-access 2 controlled-direction in

Thanks for the replies, gents.  The VLANs exist and are correct.  I've tried to get debugging working but there's no guide to actually debugging for the AOS used on the 2930M's that our network uses.  Every search for 802.1x or NAC debugging takes me straight to the HPE page for debugging 802.1x on OS-CX switches instead.

What I have discovered today is that it appears to be the MAC side of things that's causing the issue.  The actual 802.1x is working fine.  The port-access authenticator assigns the corporate VLAN correctly and the PC works.  If I turn on the MAC-based check that comes after it in the config the port fails through to the visitor VLAN immediately.  The check can't match the corporate PC's to a phone or printer MAC so it fails.  Why it's doing the check even through the first test has passed is what I don't know.  This config works elsewhere on the network but something must be different for the rule set here.  It is not stopping the checks and moves on to the MAC check which will always fail the corporate PC's.

For reference, here's an example of the aaa config on our switch ports.  

aaa accounting commands stop-only tacacs
aaa accounting exec start-stop tacacs
aaa accounting system stop-only tacacs
aaa authentication login privilege-mode
aaa authentication ssh login tacacs
aaa authentication ssh enable tacacs
aaa authentication port-access eap-radius
aaa port-access authenticator 1
aaa port-access authenticator 1 client-limit 2
aaa port-access authenticator active aaa port-access mac-based 1

aaa port-access mac-based 2 mac-pin

aaa port-access mac-based addr-format multi-colon

Original Message:
Sent: Jun 22, 2025 02:14 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi. 

Check the logs on the switch and maybe enable 802.1x debugging. Lioks like something in send by enforcement profile is not what switch is expecting.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 20, 2025 10:52 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi there.  We have a large industrial site with Clearpass handling authentication and automatic VLAN selection for each area.  We have the system check for 802.1x details and it checks those against our domain controller machine list.  A GPO pushes out the certificates and makes sure the 802.1x service is on by default on our machines.  If the check is passed the machine is assigned the local data VLAN for that area.  The fail-through rule for this is a MAC address check.  The system checks against a host list which has our printers, access control devices, etc and if on the list they are also assigned the local data VLAN.  Ultimate fail-through put anything that fails these tests into a separate visitors VLAN which can't access any corporate stuff and is basically straight to internet.

I was called out on a Friday evening because one one of our areas could not access systems and servers.  A quick check showed that everyone regardless of machine type was failing through to the visitor VLAN.  I was forced to remove aaa authentication from the switches in that area to restore service.  I'm now looking at troubleshooting and I can't find anything obviously wrong.   We made no changes that day and checking through all the rules, roles & policies I can find nothing obviously wrong.  The PC's in that area will work anywhere else as they are running the service and have the certs.  Any machine I take down there fails the same way.  I've gone through all the switch configs and they appear the same as the 5 other areas on site.  The only clue I have is that when I check the logs during testing is that I can see the PC fail through to the visitor network, then it passes and gets the corporate VLAN, then fails through again.  This cycle seems to repeat continuously (see pic attached).

The server was updated a few days ago to 6.12.5.306119, well after the issue surfaced.  None of the other areas set up in the system are affected so I assume it's something in that specific rule set since I can't see anything different in the switch configs or the PC's.  Anyone run into something similar with a constant flipping between authorised and failing?  I'm stumped.

The VLAN's are definitely on the switches.  There's data, voice, visitor and quarantine VLANs on all switches.  There are 10 switches in this area and all are configured the same and were working previously.  All switches in this area stopped working late on a Friday afternoon at the same time and I had to deactivate the authentication to get everyone back online.  The problem only affects this one area.  I am the only one who normally works on the Clearpass system and I made no changes to the switches or servers that day.  However, others do have access to the system who know even less than I do and one of them may have made a change. :-(  I think it more likely something on the server has been messed with rather than config on 10 switches. 

In majority of cases this mean that vlan in radius response is not found on switch.

Best, Gorazd

Hi again.  I've attached the debug output from the moment I applied the config to the port again to turn on authentication.

One thing that might be of interest is that I just applied auth-order and auth-priority as suggested above by Lord.  I set it to authenticator and then MAC and turned on the port.  The result was that the port stuck on the default 'fake' VLAN 666 which all our ports are set to when nothing is plugged in.  The access tracker shows that 802.1x authentication worked but the VLAN does not get applied.

The log doesn't show CoA.

Can you enable more debug logs on the switch. Just enable all the security logs for now

debug securitydebug destination session

Hi Willem.  I do not see any change of authentication notification or additional tab in the access tracker.  The 802.1 and MAC notifications look like every other notification with no additions.  It authenticates correctly using 802.1x, then it authenticates twice using the MAC rule which fails it through to the visitor VLAN.  For some reason it is authenticating against the MAC twice every time but I'm not sure why.  They look identical.

In the logs I see the message "Deauthentication request received".

Can you check in the access tracker if ClearPass is sending a CoA? In the access tracker details there should be an additional tab when a CoA is send

Hi again.  Thanks for the correct debug commands gents, they worked fine.  I've attached a sample text showing the loop I'm seeing.

Waldemar - thanks for the config, I'll look into adding the additional commands you recommend.

Willem - we are not hitting our client limits.  I've got around 800 spare access licenses and 1100 spare OnGuard licenses so it's not that.  It looks like the switch is doing exactly what it is being told to do.  It seems to be authenticating using 802.1x and then being de-authenticated by the MAC check immediately after it.  I can see the VLAN flicking constantly between the corporate (501) and the visitor (95).

I have attached a copy of the debug log when I activated MAC authentication again as well as 802.1x on the port.  It seems to confirm that the user is authenticating properly and then the MAC address fail through check runs immediately after and kicks it into the visitor VLAN.  I don't know why the MAC check is running after a successful authentication but that seems to be the problem.

My network covers a large industrial site of 6 square miles so is split into 6 network areas.  Each area has its own distributions switches (8320x's) feeding the access switches (2930M's) and each area has its own group of data, voice and visitor VLANs.  Clearpass has profiles and rules for each VLAN in each area and the problem only affects this one area.  The other areas have near identical configuration on the switches and rules with only the VLAN numbers changing.   

Hi Kenny,
heir is an example to enable debugging on an AOS-S switch. You can enable it globally for all ports or for a specific port.

deb security port-access authenticator include port 2
deb security port-access mac-based include port 2

You can also use debug security radius-server. However, this command is used for all sessions, depending on what is currently going on on the switch, it can become unmanageable.

It is more easier if you send debug output to a separate CLI session. Just open an additional session and enter debug destination session there.

The port configuration looks OK. I would also set auth-order and auth-priority explicitly and specify the unauth-vid.
I always use the following config:

aaa port-access authenticator 2
aaa port-access mac-based 2
aaa port-access mac-based 2 mac-pin
aaa port-access mac-based 2 unauth-vid 500
aaa port-access 2 auth-order authenticator mac-based
aaa port-access 2 auth-priority authenticator mac-based
aaa port-access 2 controlled-direction in

Thanks for the replies, gents.  The VLANs exist and are correct.  I've tried to get debugging working but there's no guide to actually debugging for the AOS used on the 2930M's that our network uses.  Every search for 802.1x or NAC debugging takes me straight to the HPE page for debugging 802.1x on OS-CX switches instead.

What I have discovered today is that it appears to be the MAC side of things that's causing the issue.  The actual 802.1x is working fine.  The port-access authenticator assigns the corporate VLAN correctly and the PC works.  If I turn on the MAC-based check that comes after it in the config the port fails through to the visitor VLAN immediately.  The check can't match the corporate PC's to a phone or printer MAC so it fails.  Why it's doing the check even through the first test has passed is what I don't know.  This config works elsewhere on the network but something must be different for the rule set here.  It is not stopping the checks and moves on to the MAC check which will always fail the corporate PC's.

For reference, here's an example of the aaa config on our switch ports.  

aaa accounting commands stop-only tacacs
aaa accounting exec start-stop tacacs
aaa accounting system stop-only tacacs
aaa authentication login privilege-mode
aaa authentication ssh login tacacs
aaa authentication ssh enable tacacs
aaa authentication port-access eap-radius
aaa port-access authenticator 1
aaa port-access authenticator 1 client-limit 2
aaa port-access authenticator active aaa port-access mac-based 1

aaa port-access mac-based 2 mac-pin

aaa port-access mac-based addr-format multi-colon

Original Message:
Sent: Jun 22, 2025 02:14 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi. 

Check the logs on the switch and maybe enable 802.1x debugging. Lioks like something in send by enforcement profile is not what switch is expecting.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 20, 2025 10:52 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi there.  We have a large industrial site with Clearpass handling authentication and automatic VLAN selection for each area.  We have the system check for 802.1x details and it checks those against our domain controller machine list.  A GPO pushes out the certificates and makes sure the 802.1x service is on by default on our machines.  If the check is passed the machine is assigned the local data VLAN for that area.  The fail-through rule for this is a MAC address check.  The system checks against a host list which has our printers, access control devices, etc and if on the list they are also assigned the local data VLAN.  Ultimate fail-through put anything that fails these tests into a separate visitors VLAN which can't access any corporate stuff and is basically straight to internet.

I was called out on a Friday evening because one one of our areas could not access systems and servers.  A quick check showed that everyone regardless of machine type was failing through to the visitor VLAN.  I was forced to remove aaa authentication from the switches in that area to restore service.  I'm now looking at troubleshooting and I can't find anything obviously wrong.   We made no changes that day and checking through all the rules, roles & policies I can find nothing obviously wrong.  The PC's in that area will work anywhere else as they are running the service and have the certs.  Any machine I take down there fails the same way.  I've gone through all the switch configs and they appear the same as the 5 other areas on site.  The only clue I have is that when I check the logs during testing is that I can see the PC fail through to the visitor network, then it passes and gets the corporate VLAN, then fails through again.  This cycle seems to repeat continuously (see pic attached).

The server was updated a few days ago to 6.12.5.306119, well after the issue surfaced.  None of the other areas set up in the system are affected so I assume it's something in that specific rule set since I can't see anything different in the switch configs or the PC's.  Anyone run into something similar with a constant flipping between authorised and failing?  I'm stumped.

Can you share a show tech all of the switch?  (maybe better via a DM)

The VLAN's are definitely on the switches.  There's data, voice, visitor and quarantine VLANs on all switches.  There are 10 switches in this area and all are configured the same and were working previously.  All switches in this area stopped working late on a Friday afternoon at the same time and I had to deactivate the authentication to get everyone back online.  The problem only affects this one area.  I am the only one who normally works on the Clearpass system and I made no changes to the switches or servers that day.  However, others do have access to the system who know even less than I do and one of them may have made a change. :-(  I think it more likely something on the server has been messed with rather than config on 10 switches. 

In majority of cases this mean that vlan in radius response is not found on switch.

Best, Gorazd

Hi again.  I've attached the debug output from the moment I applied the config to the port again to turn on authentication.

One thing that might be of interest is that I just applied auth-order and auth-priority as suggested above by Lord.  I set it to authenticator and then MAC and turned on the port.  The result was that the port stuck on the default 'fake' VLAN 666 which all our ports are set to when nothing is plugged in.  The access tracker shows that 802.1x authentication worked but the VLAN does not get applied.

The log doesn't show CoA.

Can you enable more debug logs on the switch. Just enable all the security logs for now

debug securitydebug destination session

Hi Willem.  I do not see any change of authentication notification or additional tab in the access tracker.  The 802.1 and MAC notifications look like every other notification with no additions.  It authenticates correctly using 802.1x, then it authenticates twice using the MAC rule which fails it through to the visitor VLAN.  For some reason it is authenticating against the MAC twice every time but I'm not sure why.  They look identical.

In the logs I see the message "Deauthentication request received".

Can you check in the access tracker if ClearPass is sending a CoA? In the access tracker details there should be an additional tab when a CoA is send

Hi again.  Thanks for the correct debug commands gents, they worked fine.  I've attached a sample text showing the loop I'm seeing.

Waldemar - thanks for the config, I'll look into adding the additional commands you recommend.

Willem - we are not hitting our client limits.  I've got around 800 spare access licenses and 1100 spare OnGuard licenses so it's not that.  It looks like the switch is doing exactly what it is being told to do.  It seems to be authenticating using 802.1x and then being de-authenticated by the MAC check immediately after it.  I can see the VLAN flicking constantly between the corporate (501) and the visitor (95).

I have attached a copy of the debug log when I activated MAC authentication again as well as 802.1x on the port.  It seems to confirm that the user is authenticating properly and then the MAC address fail through check runs immediately after and kicks it into the visitor VLAN.  I don't know why the MAC check is running after a successful authentication but that seems to be the problem.

My network covers a large industrial site of 6 square miles so is split into 6 network areas.  Each area has its own distributions switches (8320x's) feeding the access switches (2930M's) and each area has its own group of data, voice and visitor VLANs.  Clearpass has profiles and rules for each VLAN in each area and the problem only affects this one area.  The other areas have near identical configuration on the switches and rules with only the VLAN numbers changing.   

Hi Kenny,
heir is an example to enable debugging on an AOS-S switch. You can enable it globally for all ports or for a specific port.

deb security port-access authenticator include port 2
deb security port-access mac-based include port 2

You can also use debug security radius-server. However, this command is used for all sessions, depending on what is currently going on on the switch, it can become unmanageable.

It is more easier if you send debug output to a separate CLI session. Just open an additional session and enter debug destination session there.

The port configuration looks OK. I would also set auth-order and auth-priority explicitly and specify the unauth-vid.
I always use the following config:

aaa port-access authenticator 2
aaa port-access mac-based 2
aaa port-access mac-based 2 mac-pin
aaa port-access mac-based 2 unauth-vid 500
aaa port-access 2 auth-order authenticator mac-based
aaa port-access 2 auth-priority authenticator mac-based
aaa port-access 2 controlled-direction in

Thanks for the replies, gents.  The VLANs exist and are correct.  I've tried to get debugging working but there's no guide to actually debugging for the AOS used on the 2930M's that our network uses.  Every search for 802.1x or NAC debugging takes me straight to the HPE page for debugging 802.1x on OS-CX switches instead.

What I have discovered today is that it appears to be the MAC side of things that's causing the issue.  The actual 802.1x is working fine.  The port-access authenticator assigns the corporate VLAN correctly and the PC works.  If I turn on the MAC-based check that comes after it in the config the port fails through to the visitor VLAN immediately.  The check can't match the corporate PC's to a phone or printer MAC so it fails.  Why it's doing the check even through the first test has passed is what I don't know.  This config works elsewhere on the network but something must be different for the rule set here.  It is not stopping the checks and moves on to the MAC check which will always fail the corporate PC's.

For reference, here's an example of the aaa config on our switch ports.  

aaa accounting commands stop-only tacacs
aaa accounting exec start-stop tacacs
aaa accounting system stop-only tacacs
aaa authentication login privilege-mode
aaa authentication ssh login tacacs
aaa authentication ssh enable tacacs
aaa authentication port-access eap-radius
aaa port-access authenticator 1
aaa port-access authenticator 1 client-limit 2
aaa port-access authenticator active aaa port-access mac-based 1

aaa port-access mac-based 2 mac-pin

aaa port-access mac-based addr-format multi-colon

Original Message:
Sent: Jun 22, 2025 02:14 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi. 

Check the logs on the switch and maybe enable 802.1x debugging. Lioks like something in send by enforcement profile is not what switch is expecting.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 20, 2025 10:52 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi there.  We have a large industrial site with Clearpass handling authentication and automatic VLAN selection for each area.  We have the system check for 802.1x details and it checks those against our domain controller machine list.  A GPO pushes out the certificates and makes sure the 802.1x service is on by default on our machines.  If the check is passed the machine is assigned the local data VLAN for that area.  The fail-through rule for this is a MAC address check.  The system checks against a host list which has our printers, access control devices, etc and if on the list they are also assigned the local data VLAN.  Ultimate fail-through put anything that fails these tests into a separate visitors VLAN which can't access any corporate stuff and is basically straight to internet.

I was called out on a Friday evening because one one of our areas could not access systems and servers.  A quick check showed that everyone regardless of machine type was failing through to the visitor VLAN.  I was forced to remove aaa authentication from the switches in that area to restore service.  I'm now looking at troubleshooting and I can't find anything obviously wrong.   We made no changes that day and checking through all the rules, roles & policies I can find nothing obviously wrong.  The PC's in that area will work anywhere else as they are running the service and have the certs.  Any machine I take down there fails the same way.  I've gone through all the switch configs and they appear the same as the 5 other areas on site.  The only clue I have is that when I check the logs during testing is that I can see the PC fail through to the visitor network, then it passes and gets the corporate VLAN, then fails through again.  This cycle seems to repeat continuously (see pic attached).

The server was updated a few days ago to 6.12.5.306119, well after the issue surfaced.  None of the other areas set up in the system are affected so I assume it's something in that specific rule set since I can't see anything different in the switch configs or the PC's.  Anyone run into something similar with a constant flipping between authorised and failing?  I'm stumped.

The VLAN's are definitely on the switches.  There's data, voice, visitor and quarantine VLANs on all switches.  There are 10 switches in this area and all are configured the same and were working previously.  All switches in this area stopped working late on a Friday afternoon at the same time and I had to deactivate the authentication to get everyone back online.  The problem only affects this one area.  I am the only one who normally works on the Clearpass system and I made no changes to the switches or servers that day.  However, others do have access to the system who know even less than I do and one of them may have made a change. :-(  I think it more likely something on the server has been messed with rather than config on 10 switches. 

In majority of cases this mean that vlan in radius response is not found on switch.

Best, Gorazd

Hi again.  I've attached the debug output from the moment I applied the config to the port again to turn on authentication.

One thing that might be of interest is that I just applied auth-order and auth-priority as suggested above by Lord.  I set it to authenticator and then MAC and turned on the port.  The result was that the port stuck on the default 'fake' VLAN 666 which all our ports are set to when nothing is plugged in.  The access tracker shows that 802.1x authentication worked but the VLAN does not get applied.

The log doesn't show CoA.

Can you enable more debug logs on the switch. Just enable all the security logs for now

debug securitydebug destination session

Hi Willem.  I do not see any change of authentication notification or additional tab in the access tracker.  The 802.1 and MAC notifications look like every other notification with no additions.  It authenticates correctly using 802.1x, then it authenticates twice using the MAC rule which fails it through to the visitor VLAN.  For some reason it is authenticating against the MAC twice every time but I'm not sure why.  They look identical.

In the logs I see the message "Deauthentication request received".

Can you check in the access tracker if ClearPass is sending a CoA? In the access tracker details there should be an additional tab when a CoA is send

Hi again.  Thanks for the correct debug commands gents, they worked fine.  I've attached a sample text showing the loop I'm seeing.

Waldemar - thanks for the config, I'll look into adding the additional commands you recommend.

Willem - we are not hitting our client limits.  I've got around 800 spare access licenses and 1100 spare OnGuard licenses so it's not that.  It looks like the switch is doing exactly what it is being told to do.  It seems to be authenticating using 802.1x and then being de-authenticated by the MAC check immediately after it.  I can see the VLAN flicking constantly between the corporate (501) and the visitor (95).

I have attached a copy of the debug log when I activated MAC authentication again as well as 802.1x on the port.  It seems to confirm that the user is authenticating properly and then the MAC address fail through check runs immediately after and kicks it into the visitor VLAN.  I don't know why the MAC check is running after a successful authentication but that seems to be the problem.

My network covers a large industrial site of 6 square miles so is split into 6 network areas.  Each area has its own distributions switches (8320x's) feeding the access switches (2930M's) and each area has its own group of data, voice and visitor VLANs.  Clearpass has profiles and rules for each VLAN in each area and the problem only affects this one area.  The other areas have near identical configuration on the switches and rules with only the VLAN numbers changing.   

Hi Kenny,
heir is an example to enable debugging on an AOS-S switch. You can enable it globally for all ports or for a specific port.

deb security port-access authenticator include port 2
deb security port-access mac-based include port 2

You can also use debug security radius-server. However, this command is used for all sessions, depending on what is currently going on on the switch, it can become unmanageable.

It is more easier if you send debug output to a separate CLI session. Just open an additional session and enter debug destination session there.

The port configuration looks OK. I would also set auth-order and auth-priority explicitly and specify the unauth-vid.
I always use the following config:

aaa port-access authenticator 2
aaa port-access mac-based 2
aaa port-access mac-based 2 mac-pin
aaa port-access mac-based 2 unauth-vid 500
aaa port-access 2 auth-order authenticator mac-based
aaa port-access 2 auth-priority authenticator mac-based
aaa port-access 2 controlled-direction in

Thanks for the replies, gents.  The VLANs exist and are correct.  I've tried to get debugging working but there's no guide to actually debugging for the AOS used on the 2930M's that our network uses.  Every search for 802.1x or NAC debugging takes me straight to the HPE page for debugging 802.1x on OS-CX switches instead.

What I have discovered today is that it appears to be the MAC side of things that's causing the issue.  The actual 802.1x is working fine.  The port-access authenticator assigns the corporate VLAN correctly and the PC works.  If I turn on the MAC-based check that comes after it in the config the port fails through to the visitor VLAN immediately.  The check can't match the corporate PC's to a phone or printer MAC so it fails.  Why it's doing the check even through the first test has passed is what I don't know.  This config works elsewhere on the network but something must be different for the rule set here.  It is not stopping the checks and moves on to the MAC check which will always fail the corporate PC's.

For reference, here's an example of the aaa config on our switch ports.  

aaa accounting commands stop-only tacacs
aaa accounting exec start-stop tacacs
aaa accounting system stop-only tacacs
aaa authentication login privilege-mode
aaa authentication ssh login tacacs
aaa authentication ssh enable tacacs
aaa authentication port-access eap-radius
aaa port-access authenticator 1
aaa port-access authenticator 1 client-limit 2
aaa port-access authenticator active aaa port-access mac-based 1

aaa port-access mac-based 2 mac-pin

aaa port-access mac-based addr-format multi-colon

Original Message:
Sent: Jun 22, 2025 02:14 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi. 

Check the logs on the switch and maybe enable 802.1x debugging. Lioks like something in send by enforcement profile is not what switch is expecting.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 20, 2025 10:52 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi there.  We have a large industrial site with Clearpass handling authentication and automatic VLAN selection for each area.  We have the system check for 802.1x details and it checks those against our domain controller machine list.  A GPO pushes out the certificates and makes sure the 802.1x service is on by default on our machines.  If the check is passed the machine is assigned the local data VLAN for that area.  The fail-through rule for this is a MAC address check.  The system checks against a host list which has our printers, access control devices, etc and if on the list they are also assigned the local data VLAN.  Ultimate fail-through put anything that fails these tests into a separate visitors VLAN which can't access any corporate stuff and is basically straight to internet.

I was called out on a Friday evening because one one of our areas could not access systems and servers.  A quick check showed that everyone regardless of machine type was failing through to the visitor VLAN.  I was forced to remove aaa authentication from the switches in that area to restore service.  I'm now looking at troubleshooting and I can't find anything obviously wrong.   We made no changes that day and checking through all the rules, roles & policies I can find nothing obviously wrong.  The PC's in that area will work anywhere else as they are running the service and have the certs.  Any machine I take down there fails the same way.  I've gone through all the switch configs and they appear the same as the 5 other areas on site.  The only clue I have is that when I check the logs during testing is that I can see the PC fail through to the visitor network, then it passes and gets the corporate VLAN, then fails through again.  This cycle seems to repeat continuously (see pic attached).

The server was updated a few days ago to 6.12.5.306119, well after the issue surfaced.  None of the other areas set up in the system are affected so I assume it's something in that specific rule set since I can't see anything different in the switch configs or the PC's.  Anyone run into something similar with a constant flipping between authorised and failing?  I'm stumped.

Hi Kenny.

Can you please post Output tab result from access tracker and show vlan <802.1x vlan> on switch? Behaviour is consistent with wrong vlan value on radius response.

Best, Gorazd

The VLAN's are definitely on the switches.  There's data, voice, visitor and quarantine VLANs on all switches.  There are 10 switches in this area and all are configured the same and were working previously.  All switches in this area stopped working late on a Friday afternoon at the same time and I had to deactivate the authentication to get everyone back online.  The problem only affects this one area.  I am the only one who normally works on the Clearpass system and I made no changes to the switches or servers that day.  However, others do have access to the system who know even less than I do and one of them may have made a change. :-(  I think it more likely something on the server has been messed with rather than config on 10 switches. 

In majority of cases this mean that vlan in radius response is not found on switch.

Best, Gorazd

Hi again.  I've attached the debug output from the moment I applied the config to the port again to turn on authentication.

One thing that might be of interest is that I just applied auth-order and auth-priority as suggested above by Lord.  I set it to authenticator and then MAC and turned on the port.  The result was that the port stuck on the default 'fake' VLAN 666 which all our ports are set to when nothing is plugged in.  The access tracker shows that 802.1x authentication worked but the VLAN does not get applied.

The log doesn't show CoA.

Can you enable more debug logs on the switch. Just enable all the security logs for now

debug securitydebug destination session

Hi Willem.  I do not see any change of authentication notification or additional tab in the access tracker.  The 802.1 and MAC notifications look like every other notification with no additions.  It authenticates correctly using 802.1x, then it authenticates twice using the MAC rule which fails it through to the visitor VLAN.  For some reason it is authenticating against the MAC twice every time but I'm not sure why.  They look identical.

In the logs I see the message "Deauthentication request received".

Can you check in the access tracker if ClearPass is sending a CoA? In the access tracker details there should be an additional tab when a CoA is send

Hi again.  Thanks for the correct debug commands gents, they worked fine.  I've attached a sample text showing the loop I'm seeing.

Waldemar - thanks for the config, I'll look into adding the additional commands you recommend.

Willem - we are not hitting our client limits.  I've got around 800 spare access licenses and 1100 spare OnGuard licenses so it's not that.  It looks like the switch is doing exactly what it is being told to do.  It seems to be authenticating using 802.1x and then being de-authenticated by the MAC check immediately after it.  I can see the VLAN flicking constantly between the corporate (501) and the visitor (95).

I have attached a copy of the debug log when I activated MAC authentication again as well as 802.1x on the port.  It seems to confirm that the user is authenticating properly and then the MAC address fail through check runs immediately after and kicks it into the visitor VLAN.  I don't know why the MAC check is running after a successful authentication but that seems to be the problem.

My network covers a large industrial site of 6 square miles so is split into 6 network areas.  Each area has its own distributions switches (8320x's) feeding the access switches (2930M's) and each area has its own group of data, voice and visitor VLANs.  Clearpass has profiles and rules for each VLAN in each area and the problem only affects this one area.  The other areas have near identical configuration on the switches and rules with only the VLAN numbers changing.   

Hi Kenny,
heir is an example to enable debugging on an AOS-S switch. You can enable it globally for all ports or for a specific port.

deb security port-access authenticator include port 2
deb security port-access mac-based include port 2

You can also use debug security radius-server. However, this command is used for all sessions, depending on what is currently going on on the switch, it can become unmanageable.

It is more easier if you send debug output to a separate CLI session. Just open an additional session and enter debug destination session there.

The port configuration looks OK. I would also set auth-order and auth-priority explicitly and specify the unauth-vid.
I always use the following config:

aaa port-access authenticator 2
aaa port-access mac-based 2
aaa port-access mac-based 2 mac-pin
aaa port-access mac-based 2 unauth-vid 500
aaa port-access 2 auth-order authenticator mac-based
aaa port-access 2 auth-priority authenticator mac-based
aaa port-access 2 controlled-direction in

Thanks for the replies, gents.  The VLANs exist and are correct.  I've tried to get debugging working but there's no guide to actually debugging for the AOS used on the 2930M's that our network uses.  Every search for 802.1x or NAC debugging takes me straight to the HPE page for debugging 802.1x on OS-CX switches instead.

What I have discovered today is that it appears to be the MAC side of things that's causing the issue.  The actual 802.1x is working fine.  The port-access authenticator assigns the corporate VLAN correctly and the PC works.  If I turn on the MAC-based check that comes after it in the config the port fails through to the visitor VLAN immediately.  The check can't match the corporate PC's to a phone or printer MAC so it fails.  Why it's doing the check even through the first test has passed is what I don't know.  This config works elsewhere on the network but something must be different for the rule set here.  It is not stopping the checks and moves on to the MAC check which will always fail the corporate PC's.

For reference, here's an example of the aaa config on our switch ports.  

aaa accounting commands stop-only tacacs
aaa accounting exec start-stop tacacs
aaa accounting system stop-only tacacs
aaa authentication login privilege-mode
aaa authentication ssh login tacacs
aaa authentication ssh enable tacacs
aaa authentication port-access eap-radius
aaa port-access authenticator 1
aaa port-access authenticator 1 client-limit 2
aaa port-access authenticator active aaa port-access mac-based 1

aaa port-access mac-based 2 mac-pin

aaa port-access mac-based addr-format multi-colon

Original Message:
Sent: Jun 22, 2025 02:14 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi. 

Check the logs on the switch and maybe enable 802.1x debugging. Lioks like something in send by enforcement profile is not what switch is expecting.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 20, 2025 10:52 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi there.  We have a large industrial site with Clearpass handling authentication and automatic VLAN selection for each area.  We have the system check for 802.1x details and it checks those against our domain controller machine list.  A GPO pushes out the certificates and makes sure the 802.1x service is on by default on our machines.  If the check is passed the machine is assigned the local data VLAN for that area.  The fail-through rule for this is a MAC address check.  The system checks against a host list which has our printers, access control devices, etc and if on the list they are also assigned the local data VLAN.  Ultimate fail-through put anything that fails these tests into a separate visitors VLAN which can't access any corporate stuff and is basically straight to internet.

I was called out on a Friday evening because one one of our areas could not access systems and servers.  A quick check showed that everyone regardless of machine type was failing through to the visitor VLAN.  I was forced to remove aaa authentication from the switches in that area to restore service.  I'm now looking at troubleshooting and I can't find anything obviously wrong.   We made no changes that day and checking through all the rules, roles & policies I can find nothing obviously wrong.  The PC's in that area will work anywhere else as they are running the service and have the certs.  Any machine I take down there fails the same way.  I've gone through all the switch configs and they appear the same as the 5 other areas on site.  The only clue I have is that when I check the logs during testing is that I can see the PC fail through to the visitor network, then it passes and gets the corporate VLAN, then fails through again.  This cycle seems to repeat continuously (see pic attached).

The server was updated a few days ago to 6.12.5.306119, well after the issue surfaced.  None of the other areas set up in the system are affected so I assume it's something in that specific rule set since I can't see anything different in the switch configs or the PC's.  Anyone run into something similar with a constant flipping between authorised and failing?  I'm stumped.

Hi Gorazd.  Here's the output and I've attached 2 screenshots.  One is when the 802.1x works and the PC is authorised as normal, and the second is when the MAC check then immediately fails it through to the visitor VLAN (95).  Let me know if you need anything else.

Original Message:
Sent: Jun 24, 2025 02:37 PM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi Kenny.

Can you please post Output tab result from access tracker and show vlan <802.1x vlan> on switch? Behaviour is consistent with wrong vlan value on radius response.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 24, 2025 09:23 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

The VLAN's are definitely on the switches.  There's data, voice, visitor and quarantine VLANs on all switches.  There are 10 switches in this area and all are configured the same and were working previously.  All switches in this area stopped working late on a Friday afternoon at the same time and I had to deactivate the authentication to get everyone back online.  The problem only affects this one area.  I am the only one who normally works on the Clearpass system and I made no changes to the switches or servers that day.  However, others do have access to the system who know even less than I do and one of them may have made a change. :-(  I think it more likely something on the server has been messed with rather than config on 10 switches. 

Original Message:
Sent: Jun 24, 2025 08:51 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

In majority of cases this mean that vlan in radius response is not found on switch.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 24, 2025 08:40 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi again.  I've attached the debug output from the moment I applied the config to the port again to turn on authentication.

One thing that might be of interest is that I just applied auth-order and auth-priority as suggested above by Lord.  I set it to authenticator and then MAC and turned on the port.  The result was that the port stuck on the default 'fake' VLAN 666 which all our ports are set to when nothing is plugged in.  The access tracker shows that 802.1x authentication worked but the VLAN does not get applied.

Original Message:
Sent: Jun 24, 2025 07:35 AM
From: willembargeman
Subject: Clearpass authorisation flipping off and on

The log doesn't show CoA.

Can you enable more debug logs on the switch. Just enable all the security logs for now

debug securitydebug destination session

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 24, 2025 06:43 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi Willem.  I do not see any change of authentication notification or additional tab in the access tracker.  The 802.1 and MAC notifications look like every other notification with no additions.  It authenticates correctly using 802.1x, then it authenticates twice using the MAC rule which fails it through to the visitor VLAN.  For some reason it is authenticating against the MAC twice every time but I'm not sure why.  They look identical.

Original Message:
Sent: Jun 24, 2025 06:15 AM
From: willembargeman
Subject: Clearpass authorisation flipping off and on

In the logs I see the message "Deauthentication request received".

Can you check in the access tracker if ClearPass is sending a CoA? In the access tracker details there should be an additional tab when a CoA is send

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 24, 2025 06:07 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi again.  Thanks for the correct debug commands gents, they worked fine.  I've attached a sample text showing the loop I'm seeing.

Waldemar - thanks for the config, I'll look into adding the additional commands you recommend.

Willem - we are not hitting our client limits.  I've got around 800 spare access licenses and 1100 spare OnGuard licenses so it's not that.  It looks like the switch is doing exactly what it is being told to do.  It seems to be authenticating using 802.1x and then being de-authenticated by the MAC check immediately after it.  I can see the VLAN flicking constantly between the corporate (501) and the visitor (95).

I have attached a copy of the debug log when I activated MAC authentication again as well as 802.1x on the port.  It seems to confirm that the user is authenticating properly and then the MAC address fail through check runs immediately after and kicks it into the visitor VLAN.  I don't know why the MAC check is running after a successful authentication but that seems to be the problem.

My network covers a large industrial site of 6 square miles so is split into 6 network areas.  Each area has its own distributions switches (8320x's) feeding the access switches (2930M's) and each area has its own group of data, voice and visitor VLANs.  Clearpass has profiles and rules for each VLAN in each area and the problem only affects this one area.  The other areas have near identical configuration on the switches and rules with only the VLAN numbers changing.   

Original Message:
Sent: Jun 24, 2025 05:41 AM
From: Lord
Subject: Clearpass authorisation flipping off and on

Hi Kenny,
heir is an example to enable debugging on an AOS-S switch. You can enable it globally for all ports or for a specific port.

deb security port-access authenticator include port 2
deb security port-access mac-based include port 2

You can also use debug security radius-server. However, this command is used for all sessions, depending on what is currently going on on the switch, it can become unmanageable.

It is more easier if you send debug output to a separate CLI session. Just open an additional session and enter debug destination session there.

The port configuration looks OK. I would also set auth-order and auth-priority explicitly and specify the unauth-vid.
I always use the following config:

aaa port-access authenticator 2
aaa port-access mac-based 2
aaa port-access mac-based 2 mac-pin
aaa port-access mac-based 2 unauth-vid 500
aaa port-access 2 auth-order authenticator mac-based
aaa port-access 2 auth-priority authenticator mac-based
aaa port-access 2 controlled-direction in

------------------------------
Regards,

Waldemar
ACCX # 1377, ACEP, ACX - Network Security
If you find my answer useful, consider giving kudos and/or mark as solution

Original Message:
Sent: Jun 23, 2025 10:29 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Thanks for the replies, gents.  The VLANs exist and are correct.  I've tried to get debugging working but there's no guide to actually debugging for the AOS used on the 2930M's that our network uses.  Every search for 802.1x or NAC debugging takes me straight to the HPE page for debugging 802.1x on OS-CX switches instead.

What I have discovered today is that it appears to be the MAC side of things that's causing the issue.  The actual 802.1x is working fine.  The port-access authenticator assigns the corporate VLAN correctly and the PC works.  If I turn on the MAC-based check that comes after it in the config the port fails through to the visitor VLAN immediately.  The check can't match the corporate PC's to a phone or printer MAC so it fails.  Why it's doing the check even through the first test has passed is what I don't know.  This config works elsewhere on the network but something must be different for the rule set here.  It is not stopping the checks and moves on to the MAC check which will always fail the corporate PC's.

For reference, here's an example of the aaa config on our switch ports.  

aaa accounting commands stop-only tacacs
aaa accounting exec start-stop tacacs
aaa accounting system stop-only tacacs
aaa authentication login privilege-mode
aaa authentication ssh login tacacs
aaa authentication ssh enable tacacs
aaa authentication port-access eap-radius
aaa port-access authenticator 1
aaa port-access authenticator 1 client-limit 2
aaa port-access authenticator active aaa port-access mac-based 1

aaa port-access mac-based 2 mac-pin

aaa port-access mac-based addr-format multi-colon

Original Message:
Sent: Jun 22, 2025 02:14 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi. 

Check the logs on the switch and maybe enable 802.1x debugging. Lioks like something in send by enforcement profile is not what switch is expecting.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 20, 2025 10:52 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi there.  We have a large industrial site with Clearpass handling authentication and automatic VLAN selection for each area.  We have the system check for 802.1x details and it checks those against our domain controller machine list.  A GPO pushes out the certificates and makes sure the 802.1x service is on by default on our machines.  If the check is passed the machine is assigned the local data VLAN for that area.  The fail-through rule for this is a MAC address check.  The system checks against a host list which has our printers, access control devices, etc and if on the list they are also assigned the local data VLAN.  Ultimate fail-through put anything that fails these tests into a separate visitors VLAN which can't access any corporate stuff and is basically straight to internet.

I was called out on a Friday evening because one one of our areas could not access systems and servers.  A quick check showed that everyone regardless of machine type was failing through to the visitor VLAN.  I was forced to remove aaa authentication from the switches in that area to restore service.  I'm now looking at troubleshooting and I can't find anything obviously wrong.   We made no changes that day and checking through all the rules, roles & policies I can find nothing obviously wrong.  The PC's in that area will work anywhere else as they are running the service and have the certs.  Any machine I take down there fails the same way.  I've gone through all the switch configs and they appear the same as the 5 other areas on site.  The only clue I have is that when I check the logs during testing is that I can see the PC fail through to the visitor network, then it passes and gets the corporate VLAN, then fails through again.  This cycle seems to repeat continuously (see pic attached).

The server was updated a few days ago to 6.12.5.306119, well after the issue surfaced.  None of the other areas set up in the system are affected so I assume it's something in that specific rule set since I can't see anything different in the switch configs or the PC's.  Anyone run into something similar with a constant flipping between authorised and failing?  I'm stumped.

Hi Kenny,

Thank you for sharing the show tech. Please can you share a "show tech all" and not "show tech"?

The configuration it self does like correct.

In the logs I see mac-moves. Can there be a loop in the network? Please share the "show tech all" so I have more information available.

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125
------------------------------

Original Message:
Sent: Jun 25, 2025 03:36 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi Gorazd.  Here's the output and I've attached 2 screenshots.  One is when the 802.1x works and the PC is authorised as normal, and the second is when the MAC check then immediately fails it through to the visitor VLAN (95).  Let me know if you need anything else.

Original Message:
Sent: Jun 24, 2025 02:37 PM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi Kenny.

Can you please post Output tab result from access tracker and show vlan <802.1x vlan> on switch? Behaviour is consistent with wrong vlan value on radius response.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 24, 2025 09:23 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

The VLAN's are definitely on the switches.  There's data, voice, visitor and quarantine VLANs on all switches.  There are 10 switches in this area and all are configured the same and were working previously.  All switches in this area stopped working late on a Friday afternoon at the same time and I had to deactivate the authentication to get everyone back online.  The problem only affects this one area.  I am the only one who normally works on the Clearpass system and I made no changes to the switches or servers that day.  However, others do have access to the system who know even less than I do and one of them may have made a change. :-(  I think it more likely something on the server has been messed with rather than config on 10 switches. 

Original Message:
Sent: Jun 24, 2025 08:51 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

In majority of cases this mean that vlan in radius response is not found on switch.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 24, 2025 08:40 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi again.  I've attached the debug output from the moment I applied the config to the port again to turn on authentication.

One thing that might be of interest is that I just applied auth-order and auth-priority as suggested above by Lord.  I set it to authenticator and then MAC and turned on the port.  The result was that the port stuck on the default 'fake' VLAN 666 which all our ports are set to when nothing is plugged in.  The access tracker shows that 802.1x authentication worked but the VLAN does not get applied.

Original Message:
Sent: Jun 24, 2025 07:35 AM
From: willembargeman
Subject: Clearpass authorisation flipping off and on

The log doesn't show CoA.

Can you enable more debug logs on the switch. Just enable all the security logs for now

debug securitydebug destination session

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 24, 2025 06:43 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi Willem.  I do not see any change of authentication notification or additional tab in the access tracker.  The 802.1 and MAC notifications look like every other notification with no additions.  It authenticates correctly using 802.1x, then it authenticates twice using the MAC rule which fails it through to the visitor VLAN.  For some reason it is authenticating against the MAC twice every time but I'm not sure why.  They look identical.

Original Message:
Sent: Jun 24, 2025 06:15 AM
From: willembargeman
Subject: Clearpass authorisation flipping off and on

In the logs I see the message "Deauthentication request received".

Can you check in the access tracker if ClearPass is sending a CoA? In the access tracker details there should be an additional tab when a CoA is send

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 24, 2025 06:07 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi again.  Thanks for the correct debug commands gents, they worked fine.  I've attached a sample text showing the loop I'm seeing.

Waldemar - thanks for the config, I'll look into adding the additional commands you recommend.

Willem - we are not hitting our client limits.  I've got around 800 spare access licenses and 1100 spare OnGuard licenses so it's not that.  It looks like the switch is doing exactly what it is being told to do.  It seems to be authenticating using 802.1x and then being de-authenticated by the MAC check immediately after it.  I can see the VLAN flicking constantly between the corporate (501) and the visitor (95).

I have attached a copy of the debug log when I activated MAC authentication again as well as 802.1x on the port.  It seems to confirm that the user is authenticating properly and then the MAC address fail through check runs immediately after and kicks it into the visitor VLAN.  I don't know why the MAC check is running after a successful authentication but that seems to be the problem.

My network covers a large industrial site of 6 square miles so is split into 6 network areas.  Each area has its own distributions switches (8320x's) feeding the access switches (2930M's) and each area has its own group of data, voice and visitor VLANs.  Clearpass has profiles and rules for each VLAN in each area and the problem only affects this one area.  The other areas have near identical configuration on the switches and rules with only the VLAN numbers changing.   

Original Message:
Sent: Jun 24, 2025 05:41 AM
From: Lord
Subject: Clearpass authorisation flipping off and on

Hi Kenny,
heir is an example to enable debugging on an AOS-S switch. You can enable it globally for all ports or for a specific port.

deb security port-access authenticator include port 2
deb security port-access mac-based include port 2

You can also use debug security radius-server. However, this command is used for all sessions, depending on what is currently going on on the switch, it can become unmanageable.

It is more easier if you send debug output to a separate CLI session. Just open an additional session and enter debug destination session there.

The port configuration looks OK. I would also set auth-order and auth-priority explicitly and specify the unauth-vid.
I always use the following config:

aaa port-access authenticator 2
aaa port-access mac-based 2
aaa port-access mac-based 2 mac-pin
aaa port-access mac-based 2 unauth-vid 500
aaa port-access 2 auth-order authenticator mac-based
aaa port-access 2 auth-priority authenticator mac-based
aaa port-access 2 controlled-direction in

------------------------------
Regards,

Waldemar
ACCX # 1377, ACEP, ACX - Network Security
If you find my answer useful, consider giving kudos and/or mark as solution

Original Message:
Sent: Jun 23, 2025 10:29 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Thanks for the replies, gents.  The VLANs exist and are correct.  I've tried to get debugging working but there's no guide to actually debugging for the AOS used on the 2930M's that our network uses.  Every search for 802.1x or NAC debugging takes me straight to the HPE page for debugging 802.1x on OS-CX switches instead.

What I have discovered today is that it appears to be the MAC side of things that's causing the issue.  The actual 802.1x is working fine.  The port-access authenticator assigns the corporate VLAN correctly and the PC works.  If I turn on the MAC-based check that comes after it in the config the port fails through to the visitor VLAN immediately.  The check can't match the corporate PC's to a phone or printer MAC so it fails.  Why it's doing the check even through the first test has passed is what I don't know.  This config works elsewhere on the network but something must be different for the rule set here.  It is not stopping the checks and moves on to the MAC check which will always fail the corporate PC's.

For reference, here's an example of the aaa config on our switch ports.  

aaa accounting commands stop-only tacacs
aaa accounting exec start-stop tacacs
aaa accounting system stop-only tacacs
aaa authentication login privilege-mode
aaa authentication ssh login tacacs
aaa authentication ssh enable tacacs
aaa authentication port-access eap-radius
aaa port-access authenticator 1
aaa port-access authenticator 1 client-limit 2
aaa port-access authenticator active aaa port-access mac-based 1

aaa port-access mac-based 2 mac-pin

aaa port-access mac-based addr-format multi-colon

Original Message:
Sent: Jun 22, 2025 02:14 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi. 

Check the logs on the switch and maybe enable 802.1x debugging. Lioks like something in send by enforcement profile is not what switch is expecting.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 20, 2025 10:52 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi there.  We have a large industrial site with Clearpass handling authentication and automatic VLAN selection for each area.  We have the system check for 802.1x details and it checks those against our domain controller machine list.  A GPO pushes out the certificates and makes sure the 802.1x service is on by default on our machines.  If the check is passed the machine is assigned the local data VLAN for that area.  The fail-through rule for this is a MAC address check.  The system checks against a host list which has our printers, access control devices, etc and if on the list they are also assigned the local data VLAN.  Ultimate fail-through put anything that fails these tests into a separate visitors VLAN which can't access any corporate stuff and is basically straight to internet.

I was called out on a Friday evening because one one of our areas could not access systems and servers.  A quick check showed that everyone regardless of machine type was failing through to the visitor VLAN.  I was forced to remove aaa authentication from the switches in that area to restore service.  I'm now looking at troubleshooting and I can't find anything obviously wrong.   We made no changes that day and checking through all the rules, roles & policies I can find nothing obviously wrong.  The PC's in that area will work anywhere else as they are running the service and have the certs.  Any machine I take down there fails the same way.  I've gone through all the switch configs and they appear the same as the 5 other areas on site.  The only clue I have is that when I check the logs during testing is that I can see the PC fail through to the visitor network, then it passes and gets the corporate VLAN, then fails through again.  This cycle seems to repeat continuously (see pic attached).

The server was updated a few days ago to 6.12.5.306119, well after the issue surfaced.  None of the other areas set up in the system are affected so I assume it's something in that specific rule set since I can't see anything different in the switch configs or the PC's.  Anyone run into something similar with a constant flipping between authorised and failing?  I'm stumped.

Hi Kenny,

Thank you for sharing the show tech. Please can you share a "show tech all" and not "show tech"?

The configuration it self does like correct.

In the logs I see mac-moves. Can there be a loop in the network? Please share the "show tech all" so I have more information available.

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 25, 2025 03:36 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi Gorazd.  Here's the output and I've attached 2 screenshots.  One is when the 802.1x works and the PC is authorised as normal, and the second is when the MAC check then immediately fails it through to the visitor VLAN (95).  Let me know if you need anything else.

Original Message:
Sent: Jun 24, 2025 02:37 PM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi Kenny.

Can you please post Output tab result from access tracker and show vlan <802.1x vlan> on switch? Behaviour is consistent with wrong vlan value on radius response.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 24, 2025 09:23 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

The VLAN's are definitely on the switches.  There's data, voice, visitor and quarantine VLANs on all switches.  There are 10 switches in this area and all are configured the same and were working previously.  All switches in this area stopped working late on a Friday afternoon at the same time and I had to deactivate the authentication to get everyone back online.  The problem only affects this one area.  I am the only one who normally works on the Clearpass system and I made no changes to the switches or servers that day.  However, others do have access to the system who know even less than I do and one of them may have made a change. :-(  I think it more likely something on the server has been messed with rather than config on 10 switches. 

Original Message:
Sent: Jun 24, 2025 08:51 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

In majority of cases this mean that vlan in radius response is not found on switch.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 24, 2025 08:40 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi again.  I've attached the debug output from the moment I applied the config to the port again to turn on authentication.

One thing that might be of interest is that I just applied auth-order and auth-priority as suggested above by Lord.  I set it to authenticator and then MAC and turned on the port.  The result was that the port stuck on the default 'fake' VLAN 666 which all our ports are set to when nothing is plugged in.  The access tracker shows that 802.1x authentication worked but the VLAN does not get applied.

Original Message:
Sent: Jun 24, 2025 07:35 AM
From: willembargeman
Subject: Clearpass authorisation flipping off and on

The log doesn't show CoA.

Can you enable more debug logs on the switch. Just enable all the security logs for now

debug securitydebug destination session

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 24, 2025 06:43 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi Willem.  I do not see any change of authentication notification or additional tab in the access tracker.  The 802.1 and MAC notifications look like every other notification with no additions.  It authenticates correctly using 802.1x, then it authenticates twice using the MAC rule which fails it through to the visitor VLAN.  For some reason it is authenticating against the MAC twice every time but I'm not sure why.  They look identical.

Original Message:
Sent: Jun 24, 2025 06:15 AM
From: willembargeman
Subject: Clearpass authorisation flipping off and on

In the logs I see the message "Deauthentication request received".

Can you check in the access tracker if ClearPass is sending a CoA? In the access tracker details there should be an additional tab when a CoA is send

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 24, 2025 06:07 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi again.  Thanks for the correct debug commands gents, they worked fine.  I've attached a sample text showing the loop I'm seeing.

Waldemar - thanks for the config, I'll look into adding the additional commands you recommend.

Willem - we are not hitting our client limits.  I've got around 800 spare access licenses and 1100 spare OnGuard licenses so it's not that.  It looks like the switch is doing exactly what it is being told to do.  It seems to be authenticating using 802.1x and then being de-authenticated by the MAC check immediately after it.  I can see the VLAN flicking constantly between the corporate (501) and the visitor (95).

I have attached a copy of the debug log when I activated MAC authentication again as well as 802.1x on the port.  It seems to confirm that the user is authenticating properly and then the MAC address fail through check runs immediately after and kicks it into the visitor VLAN.  I don't know why the MAC check is running after a successful authentication but that seems to be the problem.

My network covers a large industrial site of 6 square miles so is split into 6 network areas.  Each area has its own distributions switches (8320x's) feeding the access switches (2930M's) and each area has its own group of data, voice and visitor VLANs.  Clearpass has profiles and rules for each VLAN in each area and the problem only affects this one area.  The other areas have near identical configuration on the switches and rules with only the VLAN numbers changing.   

Original Message:
Sent: Jun 24, 2025 05:41 AM
From: Lord
Subject: Clearpass authorisation flipping off and on

Hi Kenny,
heir is an example to enable debugging on an AOS-S switch. You can enable it globally for all ports or for a specific port.

deb security port-access authenticator include port 2
deb security port-access mac-based include port 2

You can also use debug security radius-server. However, this command is used for all sessions, depending on what is currently going on on the switch, it can become unmanageable.

It is more easier if you send debug output to a separate CLI session. Just open an additional session and enter debug destination session there.

The port configuration looks OK. I would also set auth-order and auth-priority explicitly and specify the unauth-vid.
I always use the following config:

aaa port-access authenticator 2
aaa port-access mac-based 2
aaa port-access mac-based 2 mac-pin
aaa port-access mac-based 2 unauth-vid 500
aaa port-access 2 auth-order authenticator mac-based
aaa port-access 2 auth-priority authenticator mac-based
aaa port-access 2 controlled-direction in

------------------------------
Regards,

Waldemar
ACCX # 1377, ACEP, ACX - Network Security
If you find my answer useful, consider giving kudos and/or mark as solution

Original Message:
Sent: Jun 23, 2025 10:29 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Thanks for the replies, gents.  The VLANs exist and are correct.  I've tried to get debugging working but there's no guide to actually debugging for the AOS used on the 2930M's that our network uses.  Every search for 802.1x or NAC debugging takes me straight to the HPE page for debugging 802.1x on OS-CX switches instead.

What I have discovered today is that it appears to be the MAC side of things that's causing the issue.  The actual 802.1x is working fine.  The port-access authenticator assigns the corporate VLAN correctly and the PC works.  If I turn on the MAC-based check that comes after it in the config the port fails through to the visitor VLAN immediately.  The check can't match the corporate PC's to a phone or printer MAC so it fails.  Why it's doing the check even through the first test has passed is what I don't know.  This config works elsewhere on the network but something must be different for the rule set here.  It is not stopping the checks and moves on to the MAC check which will always fail the corporate PC's.

For reference, here's an example of the aaa config on our switch ports.  

aaa accounting commands stop-only tacacs
aaa accounting exec start-stop tacacs
aaa accounting system stop-only tacacs
aaa authentication login privilege-mode
aaa authentication ssh login tacacs
aaa authentication ssh enable tacacs
aaa authentication port-access eap-radius
aaa port-access authenticator 1
aaa port-access authenticator 1 client-limit 2
aaa port-access authenticator active aaa port-access mac-based 1

aaa port-access mac-based 2 mac-pin

aaa port-access mac-based addr-format multi-colon

Original Message:
Sent: Jun 22, 2025 02:14 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi. 

Check the logs on the switch and maybe enable 802.1x debugging. Lioks like something in send by enforcement profile is not what switch is expecting.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 20, 2025 10:52 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi there.  We have a large industrial site with Clearpass handling authentication and automatic VLAN selection for each area.  We have the system check for 802.1x details and it checks those against our domain controller machine list.  A GPO pushes out the certificates and makes sure the 802.1x service is on by default on our machines.  If the check is passed the machine is assigned the local data VLAN for that area.  The fail-through rule for this is a MAC address check.  The system checks against a host list which has our printers, access control devices, etc and if on the list they are also assigned the local data VLAN.  Ultimate fail-through put anything that fails these tests into a separate visitors VLAN which can't access any corporate stuff and is basically straight to internet.

I was called out on a Friday evening because one one of our areas could not access systems and servers.  A quick check showed that everyone regardless of machine type was failing through to the visitor VLAN.  I was forced to remove aaa authentication from the switches in that area to restore service.  I'm now looking at troubleshooting and I can't find anything obviously wrong.   We made no changes that day and checking through all the rules, roles & policies I can find nothing obviously wrong.  The PC's in that area will work anywhere else as they are running the service and have the certs.  Any machine I take down there fails the same way.  I've gone through all the switch configs and they appear the same as the 5 other areas on site.  The only clue I have is that when I check the logs during testing is that I can see the PC fail through to the visitor network, then it passes and gets the corporate VLAN, then fails through again.  This cycle seems to repeat continuously (see pic attached).

The server was updated a few days ago to 6.12.5.306119, well after the issue surfaced.  None of the other areas set up in the system are affected so I assume it's something in that specific rule set since I can't see anything different in the switch configs or the PC's.  Anyone run into something similar with a constant flipping between authorised and failing?  I'm stumped.

Hi Ken,

I see a lot of MAC-moves for MAC d89ef3-04837d. The MAC address flaps between port 18 and Trk1.

addrmgrmovelist
mac address     vid    old port   new port   timestamp
-------------   ----   --------   --------   -----------------
d89ef3-04837d    501      18         Trk1    06/25/25 09:00:42
d89ef3-04837d    501      Trk1       18      06/25/25 09:02:43
d89ef3-04837d    501      18         Trk1    06/25/25 09:02:43
d89ef3-04837d    501      Trk1       18      06/25/25 09:02:43
d89ef3-04837d    501      18         Trk1    06/25/25 09:02:44
d89ef3-04837d    501      Trk1       18      06/25/25 09:04:45
d89ef3-04837d    501      18         Trk1    06/25/25 09:04:46
d89ef3-04837d    501      Trk1       18      06/25/25 09:04:46
d89ef3-04837d    501      18         Trk1    06/25/25 09:04:46
d89ef3-04837d    501      Trk1       18      06/25/25 09:06:48
d89ef3-04837d    501      18         Trk1    06/25/25 09:06:48
d89ef3-04837d    501      Trk1       18      06/25/25 09:06:48
d89ef3-04837d    501      18         Trk1    06/25/25 09:06:48
d89ef3-04837d    501      Trk1       18      06/25/25 09:08:50
d89ef3-04837d    501      18         Trk1    06/25/25 09:08:50
d89ef3-04837d    501      Trk1       18      06/25/25 09:08:50
d89ef3-04837d    501      18         Trk1    06/25/25 09:08:53
d89ef3-04837d    501      Trk1       18      06/25/25 09:10:52
d89ef3-04837d    501      18         Trk1    06/25/25 09:10:53
d89ef3-04837d    501      Trk1       18      06/25/25 09:10:53
d89ef3-04837d    501      18         Trk1    06/25/25 09:10:53
d89ef3-04837d    501      Trk1       18      06/25/25 09:12:55
d89ef3-04837d    501      18         Trk1    06/25/25 09:12:55
d89ef3-04837d    501      Trk1       18      06/25/25 09:12:55
d89ef3-04837d    501      18         Trk1    06/25/25 09:12:55

Can you trace down this MAC? I'm curious why this MAC is also seen on the uplink. 

Hi Kenny,

Thank you for sharing the show tech. Please can you share a "show tech all" and not "show tech"?

The configuration it self does like correct.

In the logs I see mac-moves. Can there be a loop in the network? Please share the "show tech all" so I have more information available.

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 25, 2025 03:36 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi Gorazd.  Here's the output and I've attached 2 screenshots.  One is when the 802.1x works and the PC is authorised as normal, and the second is when the MAC check then immediately fails it through to the visitor VLAN (95).  Let me know if you need anything else.

Original Message:
Sent: Jun 24, 2025 02:37 PM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi Kenny.

Can you please post Output tab result from access tracker and show vlan <802.1x vlan> on switch? Behaviour is consistent with wrong vlan value on radius response.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 24, 2025 09:23 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

The VLAN's are definitely on the switches.  There's data, voice, visitor and quarantine VLANs on all switches.  There are 10 switches in this area and all are configured the same and were working previously.  All switches in this area stopped working late on a Friday afternoon at the same time and I had to deactivate the authentication to get everyone back online.  The problem only affects this one area.  I am the only one who normally works on the Clearpass system and I made no changes to the switches or servers that day.  However, others do have access to the system who know even less than I do and one of them may have made a change. :-(  I think it more likely something on the server has been messed with rather than config on 10 switches. 

Original Message:
Sent: Jun 24, 2025 08:51 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

In majority of cases this mean that vlan in radius response is not found on switch.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 24, 2025 08:40 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi again.  I've attached the debug output from the moment I applied the config to the port again to turn on authentication.

One thing that might be of interest is that I just applied auth-order and auth-priority as suggested above by Lord.  I set it to authenticator and then MAC and turned on the port.  The result was that the port stuck on the default 'fake' VLAN 666 which all our ports are set to when nothing is plugged in.  The access tracker shows that 802.1x authentication worked but the VLAN does not get applied.

Original Message:
Sent: Jun 24, 2025 07:35 AM
From: willembargeman
Subject: Clearpass authorisation flipping off and on

The log doesn't show CoA.

Can you enable more debug logs on the switch. Just enable all the security logs for now

debug securitydebug destination session

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 24, 2025 06:43 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi Willem.  I do not see any change of authentication notification or additional tab in the access tracker.  The 802.1 and MAC notifications look like every other notification with no additions.  It authenticates correctly using 802.1x, then it authenticates twice using the MAC rule which fails it through to the visitor VLAN.  For some reason it is authenticating against the MAC twice every time but I'm not sure why.  They look identical.

Original Message:
Sent: Jun 24, 2025 06:15 AM
From: willembargeman
Subject: Clearpass authorisation flipping off and on

In the logs I see the message "Deauthentication request received".

Can you check in the access tracker if ClearPass is sending a CoA? In the access tracker details there should be an additional tab when a CoA is send

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 24, 2025 06:07 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi again.  Thanks for the correct debug commands gents, they worked fine.  I've attached a sample text showing the loop I'm seeing.

Waldemar - thanks for the config, I'll look into adding the additional commands you recommend.

Willem - we are not hitting our client limits.  I've got around 800 spare access licenses and 1100 spare OnGuard licenses so it's not that.  It looks like the switch is doing exactly what it is being told to do.  It seems to be authenticating using 802.1x and then being de-authenticated by the MAC check immediately after it.  I can see the VLAN flicking constantly between the corporate (501) and the visitor (95).

I have attached a copy of the debug log when I activated MAC authentication again as well as 802.1x on the port.  It seems to confirm that the user is authenticating properly and then the MAC address fail through check runs immediately after and kicks it into the visitor VLAN.  I don't know why the MAC check is running after a successful authentication but that seems to be the problem.

My network covers a large industrial site of 6 square miles so is split into 6 network areas.  Each area has its own distributions switches (8320x's) feeding the access switches (2930M's) and each area has its own group of data, voice and visitor VLANs.  Clearpass has profiles and rules for each VLAN in each area and the problem only affects this one area.  The other areas have near identical configuration on the switches and rules with only the VLAN numbers changing.   

Original Message:
Sent: Jun 24, 2025 05:41 AM
From: Lord
Subject: Clearpass authorisation flipping off and on

Hi Kenny,
heir is an example to enable debugging on an AOS-S switch. You can enable it globally for all ports or for a specific port.

deb security port-access authenticator include port 2
deb security port-access mac-based include port 2

You can also use debug security radius-server. However, this command is used for all sessions, depending on what is currently going on on the switch, it can become unmanageable.

It is more easier if you send debug output to a separate CLI session. Just open an additional session and enter debug destination session there.

The port configuration looks OK. I would also set auth-order and auth-priority explicitly and specify the unauth-vid.
I always use the following config:

aaa port-access authenticator 2
aaa port-access mac-based 2
aaa port-access mac-based 2 mac-pin
aaa port-access mac-based 2 unauth-vid 500
aaa port-access 2 auth-order authenticator mac-based
aaa port-access 2 auth-priority authenticator mac-based
aaa port-access 2 controlled-direction in

------------------------------
Regards,

Waldemar
ACCX # 1377, ACEP, ACX - Network Security
If you find my answer useful, consider giving kudos and/or mark as solution

Original Message:
Sent: Jun 23, 2025 10:29 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Thanks for the replies, gents.  The VLANs exist and are correct.  I've tried to get debugging working but there's no guide to actually debugging for the AOS used on the 2930M's that our network uses.  Every search for 802.1x or NAC debugging takes me straight to the HPE page for debugging 802.1x on OS-CX switches instead.

What I have discovered today is that it appears to be the MAC side of things that's causing the issue.  The actual 802.1x is working fine.  The port-access authenticator assigns the corporate VLAN correctly and the PC works.  If I turn on the MAC-based check that comes after it in the config the port fails through to the visitor VLAN immediately.  The check can't match the corporate PC's to a phone or printer MAC so it fails.  Why it's doing the check even through the first test has passed is what I don't know.  This config works elsewhere on the network but something must be different for the rule set here.  It is not stopping the checks and moves on to the MAC check which will always fail the corporate PC's.

For reference, here's an example of the aaa config on our switch ports.  

aaa accounting commands stop-only tacacs
aaa accounting exec start-stop tacacs
aaa accounting system stop-only tacacs
aaa authentication login privilege-mode
aaa authentication ssh login tacacs
aaa authentication ssh enable tacacs
aaa authentication port-access eap-radius
aaa port-access authenticator 1
aaa port-access authenticator 1 client-limit 2
aaa port-access authenticator active aaa port-access mac-based 1

aaa port-access mac-based 2 mac-pin

aaa port-access mac-based addr-format multi-colon

Original Message:
Sent: Jun 22, 2025 02:14 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi. 

Check the logs on the switch and maybe enable 802.1x debugging. Lioks like something in send by enforcement profile is not what switch is expecting.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 20, 2025 10:52 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi there.  We have a large industrial site with Clearpass handling authentication and automatic VLAN selection for each area.  We have the system check for 802.1x details and it checks those against our domain controller machine list.  A GPO pushes out the certificates and makes sure the 802.1x service is on by default on our machines.  If the check is passed the machine is assigned the local data VLAN for that area.  The fail-through rule for this is a MAC address check.  The system checks against a host list which has our printers, access control devices, etc and if on the list they are also assigned the local data VLAN.  Ultimate fail-through put anything that fails these tests into a separate visitors VLAN which can't access any corporate stuff and is basically straight to internet.

I was called out on a Friday evening because one one of our areas could not access systems and servers.  A quick check showed that everyone regardless of machine type was failing through to the visitor VLAN.  I was forced to remove aaa authentication from the switches in that area to restore service.  I'm now looking at troubleshooting and I can't find anything obviously wrong.   We made no changes that day and checking through all the rules, roles & policies I can find nothing obviously wrong.  The PC's in that area will work anywhere else as they are running the service and have the certs.  Any machine I take down there fails the same way.  I've gone through all the switch configs and they appear the same as the 5 other areas on site.  The only clue I have is that when I check the logs during testing is that I can see the PC fail through to the visitor network, then it passes and gets the corporate VLAN, then fails through again.  This cycle seems to repeat continuously (see pic attached).

The server was updated a few days ago to 6.12.5.306119, well after the issue surfaced.  None of the other areas set up in the system are affected so I assume it's something in that specific rule set since I can't see anything different in the switch configs or the PC's.  Anyone run into something similar with a constant flipping between authorised and failing?  I'm stumped.

Hi Ken,

I see a lot of MAC-moves for MAC d89ef3-04837d. The MAC address flaps between port 18 and Trk1.

addrmgrmovelistmac address     vid    old port   new port   timestamp-------------   ----   --------   --------   -----------------d89ef3-04837d    501      18         Trk1    06/25/25 09:00:42d89ef3-04837d    501      Trk1       18      06/25/25 09:02:43d89ef3-04837d    501      18         Trk1    06/25/25 09:02:43d89ef3-04837d    501      Trk1       18      06/25/25 09:02:43d89ef3-04837d    501      18         Trk1    06/25/25 09:02:44d89ef3-04837d    501      Trk1       18      06/25/25 09:04:45d89ef3-04837d    501      18         Trk1    06/25/25 09:04:46d89ef3-04837d    501      Trk1       18      06/25/25 09:04:46d89ef3-04837d    501      18         Trk1    06/25/25 09:04:46d89ef3-04837d    501      Trk1       18      06/25/25 09:06:48d89ef3-04837d    501      18         Trk1    06/25/25 09:06:48d89ef3-04837d    501      Trk1       18      06/25/25 09:06:48d89ef3-04837d    501      18         Trk1    06/25/25 09:06:48d89ef3-04837d    501      Trk1       18      06/25/25 09:08:50d89ef3-04837d    501      18         Trk1    06/25/25 09:08:50d89ef3-04837d    501      Trk1       18      06/25/25 09:08:50d89ef3-04837d    501      18         Trk1    06/25/25 09:08:53d89ef3-04837d    501      Trk1       18      06/25/25 09:10:52d89ef3-04837d    501      18         Trk1    06/25/25 09:10:53d89ef3-04837d    501      Trk1       18      06/25/25 09:10:53d89ef3-04837d    501      18         Trk1    06/25/25 09:10:53d89ef3-04837d    501      Trk1       18      06/25/25 09:12:55d89ef3-04837d    501      18         Trk1    06/25/25 09:12:55d89ef3-04837d    501      Trk1       18      06/25/25 09:12:55d89ef3-04837d    501      18         Trk1    06/25/25 09:12:55

Can you trace down this MAC? I'm curious why this MAC is also seen on the uplink. 

Hi Kenny,

Thank you for sharing the show tech. Please can you share a "show tech all" and not "show tech"?

The configuration it self does like correct.

In the logs I see mac-moves. Can there be a loop in the network? Please share the "show tech all" so I have more information available.

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 25, 2025 03:36 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi Gorazd.  Here's the output and I've attached 2 screenshots.  One is when the 802.1x works and the PC is authorised as normal, and the second is when the MAC check then immediately fails it through to the visitor VLAN (95).  Let me know if you need anything else.

Original Message:
Sent: Jun 24, 2025 02:37 PM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi Kenny.

Can you please post Output tab result from access tracker and show vlan <802.1x vlan> on switch? Behaviour is consistent with wrong vlan value on radius response.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 24, 2025 09:23 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

The VLAN's are definitely on the switches.  There's data, voice, visitor and quarantine VLANs on all switches.  There are 10 switches in this area and all are configured the same and were working previously.  All switches in this area stopped working late on a Friday afternoon at the same time and I had to deactivate the authentication to get everyone back online.  The problem only affects this one area.  I am the only one who normally works on the Clearpass system and I made no changes to the switches or servers that day.  However, others do have access to the system who know even less than I do and one of them may have made a change. :-(  I think it more likely something on the server has been messed with rather than config on 10 switches. 

Original Message:
Sent: Jun 24, 2025 08:51 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

In majority of cases this mean that vlan in radius response is not found on switch.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 24, 2025 08:40 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi again.  I've attached the debug output from the moment I applied the config to the port again to turn on authentication.

One thing that might be of interest is that I just applied auth-order and auth-priority as suggested above by Lord.  I set it to authenticator and then MAC and turned on the port.  The result was that the port stuck on the default 'fake' VLAN 666 which all our ports are set to when nothing is plugged in.  The access tracker shows that 802.1x authentication worked but the VLAN does not get applied.

Original Message:
Sent: Jun 24, 2025 07:35 AM
From: willembargeman
Subject: Clearpass authorisation flipping off and on

The log doesn't show CoA.

Can you enable more debug logs on the switch. Just enable all the security logs for now

debug securitydebug destination session

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 24, 2025 06:43 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi Willem.  I do not see any change of authentication notification or additional tab in the access tracker.  The 802.1 and MAC notifications look like every other notification with no additions.  It authenticates correctly using 802.1x, then it authenticates twice using the MAC rule which fails it through to the visitor VLAN.  For some reason it is authenticating against the MAC twice every time but I'm not sure why.  They look identical.

Original Message:
Sent: Jun 24, 2025 06:15 AM
From: willembargeman
Subject: Clearpass authorisation flipping off and on

In the logs I see the message "Deauthentication request received".

Can you check in the access tracker if ClearPass is sending a CoA? In the access tracker details there should be an additional tab when a CoA is send

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 24, 2025 06:07 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi again.  Thanks for the correct debug commands gents, they worked fine.  I've attached a sample text showing the loop I'm seeing.

Waldemar - thanks for the config, I'll look into adding the additional commands you recommend.

Willem - we are not hitting our client limits.  I've got around 800 spare access licenses and 1100 spare OnGuard licenses so it's not that.  It looks like the switch is doing exactly what it is being told to do.  It seems to be authenticating using 802.1x and then being de-authenticated by the MAC check immediately after it.  I can see the VLAN flicking constantly between the corporate (501) and the visitor (95).

I have attached a copy of the debug log when I activated MAC authentication again as well as 802.1x on the port.  It seems to confirm that the user is authenticating properly and then the MAC address fail through check runs immediately after and kicks it into the visitor VLAN.  I don't know why the MAC check is running after a successful authentication but that seems to be the problem.

My network covers a large industrial site of 6 square miles so is split into 6 network areas.  Each area has its own distributions switches (8320x's) feeding the access switches (2930M's) and each area has its own group of data, voice and visitor VLANs.  Clearpass has profiles and rules for each VLAN in each area and the problem only affects this one area.  The other areas have near identical configuration on the switches and rules with only the VLAN numbers changing.   

Original Message:
Sent: Jun 24, 2025 05:41 AM
From: Lord
Subject: Clearpass authorisation flipping off and on

Hi Kenny,
heir is an example to enable debugging on an AOS-S switch. You can enable it globally for all ports or for a specific port.

deb security port-access authenticator include port 2
deb security port-access mac-based include port 2

You can also use debug security radius-server. However, this command is used for all sessions, depending on what is currently going on on the switch, it can become unmanageable.

It is more easier if you send debug output to a separate CLI session. Just open an additional session and enter debug destination session there.

The port configuration looks OK. I would also set auth-order and auth-priority explicitly and specify the unauth-vid.
I always use the following config:

aaa port-access authenticator 2
aaa port-access mac-based 2
aaa port-access mac-based 2 mac-pin
aaa port-access mac-based 2 unauth-vid 500
aaa port-access 2 auth-order authenticator mac-based
aaa port-access 2 auth-priority authenticator mac-based
aaa port-access 2 controlled-direction in

------------------------------
Regards,

Waldemar
ACCX # 1377, ACEP, ACX - Network Security
If you find my answer useful, consider giving kudos and/or mark as solution

Original Message:
Sent: Jun 23, 2025 10:29 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Thanks for the replies, gents.  The VLANs exist and are correct.  I've tried to get debugging working but there's no guide to actually debugging for the AOS used on the 2930M's that our network uses.  Every search for 802.1x or NAC debugging takes me straight to the HPE page for debugging 802.1x on OS-CX switches instead.

What I have discovered today is that it appears to be the MAC side of things that's causing the issue.  The actual 802.1x is working fine.  The port-access authenticator assigns the corporate VLAN correctly and the PC works.  If I turn on the MAC-based check that comes after it in the config the port fails through to the visitor VLAN immediately.  The check can't match the corporate PC's to a phone or printer MAC so it fails.  Why it's doing the check even through the first test has passed is what I don't know.  This config works elsewhere on the network but something must be different for the rule set here.  It is not stopping the checks and moves on to the MAC check which will always fail the corporate PC's.

For reference, here's an example of the aaa config on our switch ports.  

aaa accounting commands stop-only tacacs
aaa accounting exec start-stop tacacs
aaa accounting system stop-only tacacs
aaa authentication login privilege-mode
aaa authentication ssh login tacacs
aaa authentication ssh enable tacacs
aaa authentication port-access eap-radius
aaa port-access authenticator 1
aaa port-access authenticator 1 client-limit 2
aaa port-access authenticator active aaa port-access mac-based 1

aaa port-access mac-based 2 mac-pin

aaa port-access mac-based addr-format multi-colon

Original Message:
Sent: Jun 22, 2025 02:14 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi. 

Check the logs on the switch and maybe enable 802.1x debugging. Lioks like something in send by enforcement profile is not what switch is expecting.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 20, 2025 10:52 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi there.  We have a large industrial site with Clearpass handling authentication and automatic VLAN selection for each area.  We have the system check for 802.1x details and it checks those against our domain controller machine list.  A GPO pushes out the certificates and makes sure the 802.1x service is on by default on our machines.  If the check is passed the machine is assigned the local data VLAN for that area.  The fail-through rule for this is a MAC address check.  The system checks against a host list which has our printers, access control devices, etc and if on the list they are also assigned the local data VLAN.  Ultimate fail-through put anything that fails these tests into a separate visitors VLAN which can't access any corporate stuff and is basically straight to internet.

I was called out on a Friday evening because one one of our areas could not access systems and servers.  A quick check showed that everyone regardless of machine type was failing through to the visitor VLAN.  I was forced to remove aaa authentication from the switches in that area to restore service.  I'm now looking at troubleshooting and I can't find anything obviously wrong.   We made no changes that day and checking through all the rules, roles & policies I can find nothing obviously wrong.  The PC's in that area will work anywhere else as they are running the service and have the certs.  Any machine I take down there fails the same way.  I've gone through all the switch configs and they appear the same as the 5 other areas on site.  The only clue I have is that when I check the logs during testing is that I can see the PC fail through to the visitor network, then it passes and gets the corporate VLAN, then fails through again.  This cycle seems to repeat continuously (see pic attached).

The server was updated a few days ago to 6.12.5.306119, well after the issue surfaced.  None of the other areas set up in the system are affected so I assume it's something in that specific rule set since I can't see anything different in the switch configs or the PC's.  Anyone run into something similar with a constant flipping between authorised and failing?  I'm stumped.

you didn't physically move the network connection of the device right? I don't understand why the MAC is flapping between the uplink and downlink (for the same VLAN). Normally that indicates a loop in the network. Can you check the core/distribution layer if you see MAC flaps for the MAC? And also try to remove the mac-pin configuration 

Hi Ken,

I see a lot of MAC-moves for MAC d89ef3-04837d. The MAC address flaps between port 18 and Trk1.

addrmgrmovelistmac address     vid    old port   new port   timestamp-------------   ----   --------   --------   -----------------d89ef3-04837d    501      18         Trk1    06/25/25 09:00:42d89ef3-04837d    501      Trk1       18      06/25/25 09:02:43d89ef3-04837d    501      18         Trk1    06/25/25 09:02:43d89ef3-04837d    501      Trk1       18      06/25/25 09:02:43d89ef3-04837d    501      18         Trk1    06/25/25 09:02:44d89ef3-04837d    501      Trk1       18      06/25/25 09:04:45d89ef3-04837d    501      18         Trk1    06/25/25 09:04:46d89ef3-04837d    501      Trk1       18      06/25/25 09:04:46d89ef3-04837d    501      18         Trk1    06/25/25 09:04:46d89ef3-04837d    501      Trk1       18      06/25/25 09:06:48d89ef3-04837d    501      18         Trk1    06/25/25 09:06:48d89ef3-04837d    501      Trk1       18      06/25/25 09:06:48d89ef3-04837d    501      18         Trk1    06/25/25 09:06:48d89ef3-04837d    501      Trk1       18      06/25/25 09:08:50d89ef3-04837d    501      18         Trk1    06/25/25 09:08:50d89ef3-04837d    501      Trk1       18      06/25/25 09:08:50d89ef3-04837d    501      18         Trk1    06/25/25 09:08:53d89ef3-04837d    501      Trk1       18      06/25/25 09:10:52d89ef3-04837d    501      18         Trk1    06/25/25 09:10:53d89ef3-04837d    501      Trk1       18      06/25/25 09:10:53d89ef3-04837d    501      18         Trk1    06/25/25 09:10:53d89ef3-04837d    501      Trk1       18      06/25/25 09:12:55d89ef3-04837d    501      18         Trk1    06/25/25 09:12:55d89ef3-04837d    501      Trk1       18      06/25/25 09:12:55d89ef3-04837d    501      18         Trk1    06/25/25 09:12:55

Can you trace down this MAC? I'm curious why this MAC is also seen on the uplink. 

Hi Kenny,

Thank you for sharing the show tech. Please can you share a "show tech all" and not "show tech"?

The configuration it self does like correct.

In the logs I see mac-moves. Can there be a loop in the network? Please share the "show tech all" so I have more information available.

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 25, 2025 03:36 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi Gorazd.  Here's the output and I've attached 2 screenshots.  One is when the 802.1x works and the PC is authorised as normal, and the second is when the MAC check then immediately fails it through to the visitor VLAN (95).  Let me know if you need anything else.

Original Message:
Sent: Jun 24, 2025 02:37 PM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi Kenny.

Can you please post Output tab result from access tracker and show vlan <802.1x vlan> on switch? Behaviour is consistent with wrong vlan value on radius response.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 24, 2025 09:23 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

The VLAN's are definitely on the switches.  There's data, voice, visitor and quarantine VLANs on all switches.  There are 10 switches in this area and all are configured the same and were working previously.  All switches in this area stopped working late on a Friday afternoon at the same time and I had to deactivate the authentication to get everyone back online.  The problem only affects this one area.  I am the only one who normally works on the Clearpass system and I made no changes to the switches or servers that day.  However, others do have access to the system who know even less than I do and one of them may have made a change. :-(  I think it more likely something on the server has been messed with rather than config on 10 switches. 

Original Message:
Sent: Jun 24, 2025 08:51 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

In majority of cases this mean that vlan in radius response is not found on switch.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 24, 2025 08:40 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi again.  I've attached the debug output from the moment I applied the config to the port again to turn on authentication.

One thing that might be of interest is that I just applied auth-order and auth-priority as suggested above by Lord.  I set it to authenticator and then MAC and turned on the port.  The result was that the port stuck on the default 'fake' VLAN 666 which all our ports are set to when nothing is plugged in.  The access tracker shows that 802.1x authentication worked but the VLAN does not get applied.

Original Message:
Sent: Jun 24, 2025 07:35 AM
From: willembargeman
Subject: Clearpass authorisation flipping off and on

The log doesn't show CoA.

Can you enable more debug logs on the switch. Just enable all the security logs for now

debug securitydebug destination session

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 24, 2025 06:43 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi Willem.  I do not see any change of authentication notification or additional tab in the access tracker.  The 802.1 and MAC notifications look like every other notification with no additions.  It authenticates correctly using 802.1x, then it authenticates twice using the MAC rule which fails it through to the visitor VLAN.  For some reason it is authenticating against the MAC twice every time but I'm not sure why.  They look identical.

Original Message:
Sent: Jun 24, 2025 06:15 AM
From: willembargeman
Subject: Clearpass authorisation flipping off and on

In the logs I see the message "Deauthentication request received".

Can you check in the access tracker if ClearPass is sending a CoA? In the access tracker details there should be an additional tab when a CoA is send

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 24, 2025 06:07 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi again.  Thanks for the correct debug commands gents, they worked fine.  I've attached a sample text showing the loop I'm seeing.

Waldemar - thanks for the config, I'll look into adding the additional commands you recommend.

Willem - we are not hitting our client limits.  I've got around 800 spare access licenses and 1100 spare OnGuard licenses so it's not that.  It looks like the switch is doing exactly what it is being told to do.  It seems to be authenticating using 802.1x and then being de-authenticated by the MAC check immediately after it.  I can see the VLAN flicking constantly between the corporate (501) and the visitor (95).

I have attached a copy of the debug log when I activated MAC authentication again as well as 802.1x on the port.  It seems to confirm that the user is authenticating properly and then the MAC address fail through check runs immediately after and kicks it into the visitor VLAN.  I don't know why the MAC check is running after a successful authentication but that seems to be the problem.

My network covers a large industrial site of 6 square miles so is split into 6 network areas.  Each area has its own distributions switches (8320x's) feeding the access switches (2930M's) and each area has its own group of data, voice and visitor VLANs.  Clearpass has profiles and rules for each VLAN in each area and the problem only affects this one area.  The other areas have near identical configuration on the switches and rules with only the VLAN numbers changing.   

Original Message:
Sent: Jun 24, 2025 05:41 AM
From: Lord
Subject: Clearpass authorisation flipping off and on

Hi Kenny,
heir is an example to enable debugging on an AOS-S switch. You can enable it globally for all ports or for a specific port.

deb security port-access authenticator include port 2
deb security port-access mac-based include port 2

You can also use debug security radius-server. However, this command is used for all sessions, depending on what is currently going on on the switch, it can become unmanageable.

It is more easier if you send debug output to a separate CLI session. Just open an additional session and enter debug destination session there.

The port configuration looks OK. I would also set auth-order and auth-priority explicitly and specify the unauth-vid.
I always use the following config:

aaa port-access authenticator 2
aaa port-access mac-based 2
aaa port-access mac-based 2 mac-pin
aaa port-access mac-based 2 unauth-vid 500
aaa port-access 2 auth-order authenticator mac-based
aaa port-access 2 auth-priority authenticator mac-based
aaa port-access 2 controlled-direction in

------------------------------
Regards,

Waldemar
ACCX # 1377, ACEP, ACX - Network Security
If you find my answer useful, consider giving kudos and/or mark as solution

Original Message:
Sent: Jun 23, 2025 10:29 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Thanks for the replies, gents.  The VLANs exist and are correct.  I've tried to get debugging working but there's no guide to actually debugging for the AOS used on the 2930M's that our network uses.  Every search for 802.1x or NAC debugging takes me straight to the HPE page for debugging 802.1x on OS-CX switches instead.

What I have discovered today is that it appears to be the MAC side of things that's causing the issue.  The actual 802.1x is working fine.  The port-access authenticator assigns the corporate VLAN correctly and the PC works.  If I turn on the MAC-based check that comes after it in the config the port fails through to the visitor VLAN immediately.  The check can't match the corporate PC's to a phone or printer MAC so it fails.  Why it's doing the check even through the first test has passed is what I don't know.  This config works elsewhere on the network but something must be different for the rule set here.  It is not stopping the checks and moves on to the MAC check which will always fail the corporate PC's.

For reference, here's an example of the aaa config on our switch ports.  

aaa accounting commands stop-only tacacs
aaa accounting exec start-stop tacacs
aaa accounting system stop-only tacacs
aaa authentication login privilege-mode
aaa authentication ssh login tacacs
aaa authentication ssh enable tacacs
aaa authentication port-access eap-radius
aaa port-access authenticator 1
aaa port-access authenticator 1 client-limit 2
aaa port-access authenticator active aaa port-access mac-based 1

aaa port-access mac-based 2 mac-pin

aaa port-access mac-based addr-format multi-colon

Original Message:
Sent: Jun 22, 2025 02:14 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi. 

Check the logs on the switch and maybe enable 802.1x debugging. Lioks like something in send by enforcement profile is not what switch is expecting.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 20, 2025 10:52 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi there.  We have a large industrial site with Clearpass handling authentication and automatic VLAN selection for each area.  We have the system check for 802.1x details and it checks those against our domain controller machine list.  A GPO pushes out the certificates and makes sure the 802.1x service is on by default on our machines.  If the check is passed the machine is assigned the local data VLAN for that area.  The fail-through rule for this is a MAC address check.  The system checks against a host list which has our printers, access control devices, etc and if on the list they are also assigned the local data VLAN.  Ultimate fail-through put anything that fails these tests into a separate visitors VLAN which can't access any corporate stuff and is basically straight to internet.

I was called out on a Friday evening because one one of our areas could not access systems and servers.  A quick check showed that everyone regardless of machine type was failing through to the visitor VLAN.  I was forced to remove aaa authentication from the switches in that area to restore service.  I'm now looking at troubleshooting and I can't find anything obviously wrong.   We made no changes that day and checking through all the rules, roles & policies I can find nothing obviously wrong.  The PC's in that area will work anywhere else as they are running the service and have the certs.  Any machine I take down there fails the same way.  I've gone through all the switch configs and they appear the same as the 5 other areas on site.  The only clue I have is that when I check the logs during testing is that I can see the PC fail through to the visitor network, then it passes and gets the corporate VLAN, then fails through again.  This cycle seems to repeat continuously (see pic attached).

The server was updated a few days ago to 6.12.5.306119, well after the issue surfaced.  None of the other areas set up in the system are affected so I assume it's something in that specific rule set since I can't see anything different in the switch configs or the PC's.  Anyone run into something similar with a constant flipping between authorised and failing?  I'm stumped.

you didn't physically move the network connection of the device right? I don't understand why the MAC is flapping between the uplink and downlink (for the same VLAN). Normally that indicates a loop in the network. Can you check the core/distribution layer if you see MAC flaps for the MAC? And also try to remove the mac-pin configuration 

Hi Ken,

I see a lot of MAC-moves for MAC d89ef3-04837d. The MAC address flaps between port 18 and Trk1.

addrmgrmovelistmac address     vid    old port   new port   timestamp-------------   ----   --------   --------   -----------------d89ef3-04837d    501      18         Trk1    06/25/25 09:00:42d89ef3-04837d    501      Trk1       18      06/25/25 09:02:43d89ef3-04837d    501      18         Trk1    06/25/25 09:02:43d89ef3-04837d    501      Trk1       18      06/25/25 09:02:43d89ef3-04837d    501      18         Trk1    06/25/25 09:02:44d89ef3-04837d    501      Trk1       18      06/25/25 09:04:45d89ef3-04837d    501      18         Trk1    06/25/25 09:04:46d89ef3-04837d    501      Trk1       18      06/25/25 09:04:46d89ef3-04837d    501      18         Trk1    06/25/25 09:04:46d89ef3-04837d    501      Trk1       18      06/25/25 09:06:48d89ef3-04837d    501      18         Trk1    06/25/25 09:06:48d89ef3-04837d    501      Trk1       18      06/25/25 09:06:48d89ef3-04837d    501      18         Trk1    06/25/25 09:06:48d89ef3-04837d    501      Trk1       18      06/25/25 09:08:50d89ef3-04837d    501      18         Trk1    06/25/25 09:08:50d89ef3-04837d    501      Trk1       18      06/25/25 09:08:50d89ef3-04837d    501      18         Trk1    06/25/25 09:08:53d89ef3-04837d    501      Trk1       18      06/25/25 09:10:52d89ef3-04837d    501      18         Trk1    06/25/25 09:10:53d89ef3-04837d    501      Trk1       18      06/25/25 09:10:53d89ef3-04837d    501      18         Trk1    06/25/25 09:10:53d89ef3-04837d    501      Trk1       18      06/25/25 09:12:55d89ef3-04837d    501      18         Trk1    06/25/25 09:12:55d89ef3-04837d    501      Trk1       18      06/25/25 09:12:55d89ef3-04837d    501      18         Trk1    06/25/25 09:12:55

Can you trace down this MAC? I'm curious why this MAC is also seen on the uplink. 

Hi Kenny,

Thank you for sharing the show tech. Please can you share a "show tech all" and not "show tech"?

The configuration it self does like correct.

In the logs I see mac-moves. Can there be a loop in the network? Please share the "show tech all" so I have more information available.

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 25, 2025 03:36 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi Gorazd.  Here's the output and I've attached 2 screenshots.  One is when the 802.1x works and the PC is authorised as normal, and the second is when the MAC check then immediately fails it through to the visitor VLAN (95).  Let me know if you need anything else.

Original Message:
Sent: Jun 24, 2025 02:37 PM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi Kenny.

Can you please post Output tab result from access tracker and show vlan <802.1x vlan> on switch? Behaviour is consistent with wrong vlan value on radius response.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 24, 2025 09:23 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

The VLAN's are definitely on the switches.  There's data, voice, visitor and quarantine VLANs on all switches.  There are 10 switches in this area and all are configured the same and were working previously.  All switches in this area stopped working late on a Friday afternoon at the same time and I had to deactivate the authentication to get everyone back online.  The problem only affects this one area.  I am the only one who normally works on the Clearpass system and I made no changes to the switches or servers that day.  However, others do have access to the system who know even less than I do and one of them may have made a change. :-(  I think it more likely something on the server has been messed with rather than config on 10 switches. 

Original Message:
Sent: Jun 24, 2025 08:51 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

In majority of cases this mean that vlan in radius response is not found on switch.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 24, 2025 08:40 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi again.  I've attached the debug output from the moment I applied the config to the port again to turn on authentication.

One thing that might be of interest is that I just applied auth-order and auth-priority as suggested above by Lord.  I set it to authenticator and then MAC and turned on the port.  The result was that the port stuck on the default 'fake' VLAN 666 which all our ports are set to when nothing is plugged in.  The access tracker shows that 802.1x authentication worked but the VLAN does not get applied.

Original Message:
Sent: Jun 24, 2025 07:35 AM
From: willembargeman
Subject: Clearpass authorisation flipping off and on

The log doesn't show CoA.

Can you enable more debug logs on the switch. Just enable all the security logs for now

debug securitydebug destination session

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 24, 2025 06:43 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi Willem.  I do not see any change of authentication notification or additional tab in the access tracker.  The 802.1 and MAC notifications look like every other notification with no additions.  It authenticates correctly using 802.1x, then it authenticates twice using the MAC rule which fails it through to the visitor VLAN.  For some reason it is authenticating against the MAC twice every time but I'm not sure why.  They look identical.

Original Message:
Sent: Jun 24, 2025 06:15 AM
From: willembargeman
Subject: Clearpass authorisation flipping off and on

In the logs I see the message "Deauthentication request received".

Can you check in the access tracker if ClearPass is sending a CoA? In the access tracker details there should be an additional tab when a CoA is send

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 24, 2025 06:07 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi again.  Thanks for the correct debug commands gents, they worked fine.  I've attached a sample text showing the loop I'm seeing.

Waldemar - thanks for the config, I'll look into adding the additional commands you recommend.

Willem - we are not hitting our client limits.  I've got around 800 spare access licenses and 1100 spare OnGuard licenses so it's not that.  It looks like the switch is doing exactly what it is being told to do.  It seems to be authenticating using 802.1x and then being de-authenticated by the MAC check immediately after it.  I can see the VLAN flicking constantly between the corporate (501) and the visitor (95).

I have attached a copy of the debug log when I activated MAC authentication again as well as 802.1x on the port.  It seems to confirm that the user is authenticating properly and then the MAC address fail through check runs immediately after and kicks it into the visitor VLAN.  I don't know why the MAC check is running after a successful authentication but that seems to be the problem.

My network covers a large industrial site of 6 square miles so is split into 6 network areas.  Each area has its own distributions switches (8320x's) feeding the access switches (2930M's) and each area has its own group of data, voice and visitor VLANs.  Clearpass has profiles and rules for each VLAN in each area and the problem only affects this one area.  The other areas have near identical configuration on the switches and rules with only the VLAN numbers changing.   

Original Message:
Sent: Jun 24, 2025 05:41 AM
From: Lord
Subject: Clearpass authorisation flipping off and on

Hi Kenny,
heir is an example to enable debugging on an AOS-S switch. You can enable it globally for all ports or for a specific port.

deb security port-access authenticator include port 2
deb security port-access mac-based include port 2

You can also use debug security radius-server. However, this command is used for all sessions, depending on what is currently going on on the switch, it can become unmanageable.

It is more easier if you send debug output to a separate CLI session. Just open an additional session and enter debug destination session there.

The port configuration looks OK. I would also set auth-order and auth-priority explicitly and specify the unauth-vid.
I always use the following config:

aaa port-access authenticator 2
aaa port-access mac-based 2
aaa port-access mac-based 2 mac-pin
aaa port-access mac-based 2 unauth-vid 500
aaa port-access 2 auth-order authenticator mac-based
aaa port-access 2 auth-priority authenticator mac-based
aaa port-access 2 controlled-direction in

------------------------------
Regards,

Waldemar
ACCX # 1377, ACEP, ACX - Network Security
If you find my answer useful, consider giving kudos and/or mark as solution

Original Message:
Sent: Jun 23, 2025 10:29 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Thanks for the replies, gents.  The VLANs exist and are correct.  I've tried to get debugging working but there's no guide to actually debugging for the AOS used on the 2930M's that our network uses.  Every search for 802.1x or NAC debugging takes me straight to the HPE page for debugging 802.1x on OS-CX switches instead.

What I have discovered today is that it appears to be the MAC side of things that's causing the issue.  The actual 802.1x is working fine.  The port-access authenticator assigns the corporate VLAN correctly and the PC works.  If I turn on the MAC-based check that comes after it in the config the port fails through to the visitor VLAN immediately.  The check can't match the corporate PC's to a phone or printer MAC so it fails.  Why it's doing the check even through the first test has passed is what I don't know.  This config works elsewhere on the network but something must be different for the rule set here.  It is not stopping the checks and moves on to the MAC check which will always fail the corporate PC's.

For reference, here's an example of the aaa config on our switch ports.  

aaa accounting commands stop-only tacacs
aaa accounting exec start-stop tacacs
aaa accounting system stop-only tacacs
aaa authentication login privilege-mode
aaa authentication ssh login tacacs
aaa authentication ssh enable tacacs
aaa authentication port-access eap-radius
aaa port-access authenticator 1
aaa port-access authenticator 1 client-limit 2
aaa port-access authenticator active aaa port-access mac-based 1

aaa port-access mac-based 2 mac-pin

aaa port-access mac-based addr-format multi-colon

Original Message:
Sent: Jun 22, 2025 02:14 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi. 

Check the logs on the switch and maybe enable 802.1x debugging. Lioks like something in send by enforcement profile is not what switch is expecting.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 20, 2025 10:52 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi there.  We have a large industrial site with Clearpass handling authentication and automatic VLAN selection for each area.  We have the system check for 802.1x details and it checks those against our domain controller machine list.  A GPO pushes out the certificates and makes sure the 802.1x service is on by default on our machines.  If the check is passed the machine is assigned the local data VLAN for that area.  The fail-through rule for this is a MAC address check.  The system checks against a host list which has our printers, access control devices, etc and if on the list they are also assigned the local data VLAN.  Ultimate fail-through put anything that fails these tests into a separate visitors VLAN which can't access any corporate stuff and is basically straight to internet.

I was called out on a Friday evening because one one of our areas could not access systems and servers.  A quick check showed that everyone regardless of machine type was failing through to the visitor VLAN.  I was forced to remove aaa authentication from the switches in that area to restore service.  I'm now looking at troubleshooting and I can't find anything obviously wrong.   We made no changes that day and checking through all the rules, roles & policies I can find nothing obviously wrong.  The PC's in that area will work anywhere else as they are running the service and have the certs.  Any machine I take down there fails the same way.  I've gone through all the switch configs and they appear the same as the 5 other areas on site.  The only clue I have is that when I check the logs during testing is that I can see the PC fail through to the visitor network, then it passes and gets the corporate VLAN, then fails through again.  This cycle seems to repeat continuously (see pic attached).

The server was updated a few days ago to 6.12.5.306119, well after the issue surfaced.  None of the other areas set up in the system are affected so I assume it's something in that specific rule set since I can't see anything different in the switch configs or the PC's.  Anyone run into something similar with a constant flipping between authorised and failing?  I'm stumped.

I took another look at the debugoutput (Authentication debug.txt) that Kenny shared.
In line 87 it says  "Switch Mac move indication received", the client is then deauthenticated.

This matches the MAC is flapping between the uplink and accessport.
Is there really a loop in this LAN segment?

you didn't physically move the network connection of the device right? I don't understand why the MAC is flapping between the uplink and downlink (for the same VLAN). Normally that indicates a loop in the network. Can you check the core/distribution layer if you see MAC flaps for the MAC? And also try to remove the mac-pin configuration 

Hi Ken,

I see a lot of MAC-moves for MAC d89ef3-04837d. The MAC address flaps between port 18 and Trk1.

addrmgrmovelistmac address     vid    old port   new port   timestamp-------------   ----   --------   --------   -----------------d89ef3-04837d    501      18         Trk1    06/25/25 09:00:42d89ef3-04837d    501      Trk1       18      06/25/25 09:02:43d89ef3-04837d    501      18         Trk1    06/25/25 09:02:43d89ef3-04837d    501      Trk1       18      06/25/25 09:02:43d89ef3-04837d    501      18         Trk1    06/25/25 09:02:44d89ef3-04837d    501      Trk1       18      06/25/25 09:04:45d89ef3-04837d    501      18         Trk1    06/25/25 09:04:46d89ef3-04837d    501      Trk1       18      06/25/25 09:04:46d89ef3-04837d    501      18         Trk1    06/25/25 09:04:46d89ef3-04837d    501      Trk1       18      06/25/25 09:06:48d89ef3-04837d    501      18         Trk1    06/25/25 09:06:48d89ef3-04837d    501      Trk1       18      06/25/25 09:06:48d89ef3-04837d    501      18         Trk1    06/25/25 09:06:48d89ef3-04837d    501      Trk1       18      06/25/25 09:08:50d89ef3-04837d    501      18         Trk1    06/25/25 09:08:50d89ef3-04837d    501      Trk1       18      06/25/25 09:08:50d89ef3-04837d    501      18         Trk1    06/25/25 09:08:53d89ef3-04837d    501      Trk1       18      06/25/25 09:10:52d89ef3-04837d    501      18         Trk1    06/25/25 09:10:53d89ef3-04837d    501      Trk1       18      06/25/25 09:10:53d89ef3-04837d    501      18         Trk1    06/25/25 09:10:53d89ef3-04837d    501      Trk1       18      06/25/25 09:12:55d89ef3-04837d    501      18         Trk1    06/25/25 09:12:55d89ef3-04837d    501      Trk1       18      06/25/25 09:12:55d89ef3-04837d    501      18         Trk1    06/25/25 09:12:55

Can you trace down this MAC? I'm curious why this MAC is also seen on the uplink. 

Hi Kenny,

Thank you for sharing the show tech. Please can you share a "show tech all" and not "show tech"?

The configuration it self does like correct.

In the logs I see mac-moves. Can there be a loop in the network? Please share the "show tech all" so I have more information available.

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 25, 2025 03:36 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi Gorazd.  Here's the output and I've attached 2 screenshots.  One is when the 802.1x works and the PC is authorised as normal, and the second is when the MAC check then immediately fails it through to the visitor VLAN (95).  Let me know if you need anything else.

Original Message:
Sent: Jun 24, 2025 02:37 PM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi Kenny.

Can you please post Output tab result from access tracker and show vlan <802.1x vlan> on switch? Behaviour is consistent with wrong vlan value on radius response.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 24, 2025 09:23 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

The VLAN's are definitely on the switches.  There's data, voice, visitor and quarantine VLANs on all switches.  There are 10 switches in this area and all are configured the same and were working previously.  All switches in this area stopped working late on a Friday afternoon at the same time and I had to deactivate the authentication to get everyone back online.  The problem only affects this one area.  I am the only one who normally works on the Clearpass system and I made no changes to the switches or servers that day.  However, others do have access to the system who know even less than I do and one of them may have made a change. :-(  I think it more likely something on the server has been messed with rather than config on 10 switches. 

Original Message:
Sent: Jun 24, 2025 08:51 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

In majority of cases this mean that vlan in radius response is not found on switch.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 24, 2025 08:40 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi again.  I've attached the debug output from the moment I applied the config to the port again to turn on authentication.

One thing that might be of interest is that I just applied auth-order and auth-priority as suggested above by Lord.  I set it to authenticator and then MAC and turned on the port.  The result was that the port stuck on the default 'fake' VLAN 666 which all our ports are set to when nothing is plugged in.  The access tracker shows that 802.1x authentication worked but the VLAN does not get applied.

Original Message:
Sent: Jun 24, 2025 07:35 AM
From: willembargeman
Subject: Clearpass authorisation flipping off and on

The log doesn't show CoA.

Can you enable more debug logs on the switch. Just enable all the security logs for now

debug securitydebug destination session

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 24, 2025 06:43 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi Willem.  I do not see any change of authentication notification or additional tab in the access tracker.  The 802.1 and MAC notifications look like every other notification with no additions.  It authenticates correctly using 802.1x, then it authenticates twice using the MAC rule which fails it through to the visitor VLAN.  For some reason it is authenticating against the MAC twice every time but I'm not sure why.  They look identical.

Original Message:
Sent: Jun 24, 2025 06:15 AM
From: willembargeman
Subject: Clearpass authorisation flipping off and on

In the logs I see the message "Deauthentication request received".

Can you check in the access tracker if ClearPass is sending a CoA? In the access tracker details there should be an additional tab when a CoA is send

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 24, 2025 06:07 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi again.  Thanks for the correct debug commands gents, they worked fine.  I've attached a sample text showing the loop I'm seeing.

Waldemar - thanks for the config, I'll look into adding the additional commands you recommend.

Willem - we are not hitting our client limits.  I've got around 800 spare access licenses and 1100 spare OnGuard licenses so it's not that.  It looks like the switch is doing exactly what it is being told to do.  It seems to be authenticating using 802.1x and then being de-authenticated by the MAC check immediately after it.  I can see the VLAN flicking constantly between the corporate (501) and the visitor (95).

I have attached a copy of the debug log when I activated MAC authentication again as well as 802.1x on the port.  It seems to confirm that the user is authenticating properly and then the MAC address fail through check runs immediately after and kicks it into the visitor VLAN.  I don't know why the MAC check is running after a successful authentication but that seems to be the problem.

My network covers a large industrial site of 6 square miles so is split into 6 network areas.  Each area has its own distributions switches (8320x's) feeding the access switches (2930M's) and each area has its own group of data, voice and visitor VLANs.  Clearpass has profiles and rules for each VLAN in each area and the problem only affects this one area.  The other areas have near identical configuration on the switches and rules with only the VLAN numbers changing.   

Original Message:
Sent: Jun 24, 2025 05:41 AM
From: Lord
Subject: Clearpass authorisation flipping off and on

Hi Kenny,
heir is an example to enable debugging on an AOS-S switch. You can enable it globally for all ports or for a specific port.

deb security port-access authenticator include port 2
deb security port-access mac-based include port 2

You can also use debug security radius-server. However, this command is used for all sessions, depending on what is currently going on on the switch, it can become unmanageable.

It is more easier if you send debug output to a separate CLI session. Just open an additional session and enter debug destination session there.

The port configuration looks OK. I would also set auth-order and auth-priority explicitly and specify the unauth-vid.
I always use the following config:

aaa port-access authenticator 2
aaa port-access mac-based 2
aaa port-access mac-based 2 mac-pin
aaa port-access mac-based 2 unauth-vid 500
aaa port-access 2 auth-order authenticator mac-based
aaa port-access 2 auth-priority authenticator mac-based
aaa port-access 2 controlled-direction in

------------------------------
Regards,

Waldemar
ACCX # 1377, ACEP, ACX - Network Security
If you find my answer useful, consider giving kudos and/or mark as solution

Original Message:
Sent: Jun 23, 2025 10:29 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Thanks for the replies, gents.  The VLANs exist and are correct.  I've tried to get debugging working but there's no guide to actually debugging for the AOS used on the 2930M's that our network uses.  Every search for 802.1x or NAC debugging takes me straight to the HPE page for debugging 802.1x on OS-CX switches instead.

What I have discovered today is that it appears to be the MAC side of things that's causing the issue.  The actual 802.1x is working fine.  The port-access authenticator assigns the corporate VLAN correctly and the PC works.  If I turn on the MAC-based check that comes after it in the config the port fails through to the visitor VLAN immediately.  The check can't match the corporate PC's to a phone or printer MAC so it fails.  Why it's doing the check even through the first test has passed is what I don't know.  This config works elsewhere on the network but something must be different for the rule set here.  It is not stopping the checks and moves on to the MAC check which will always fail the corporate PC's.

For reference, here's an example of the aaa config on our switch ports.  

aaa accounting commands stop-only tacacs
aaa accounting exec start-stop tacacs
aaa accounting system stop-only tacacs
aaa authentication login privilege-mode
aaa authentication ssh login tacacs
aaa authentication ssh enable tacacs
aaa authentication port-access eap-radius
aaa port-access authenticator 1
aaa port-access authenticator 1 client-limit 2
aaa port-access authenticator active aaa port-access mac-based 1

aaa port-access mac-based 2 mac-pin

aaa port-access mac-based addr-format multi-colon

Original Message:
Sent: Jun 22, 2025 02:14 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi. 

Check the logs on the switch and maybe enable 802.1x debugging. Lioks like something in send by enforcement profile is not what switch is expecting.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 20, 2025 10:52 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi there.  We have a large industrial site with Clearpass handling authentication and automatic VLAN selection for each area.  We have the system check for 802.1x details and it checks those against our domain controller machine list.  A GPO pushes out the certificates and makes sure the 802.1x service is on by default on our machines.  If the check is passed the machine is assigned the local data VLAN for that area.  The fail-through rule for this is a MAC address check.  The system checks against a host list which has our printers, access control devices, etc and if on the list they are also assigned the local data VLAN.  Ultimate fail-through put anything that fails these tests into a separate visitors VLAN which can't access any corporate stuff and is basically straight to internet.

I was called out on a Friday evening because one one of our areas could not access systems and servers.  A quick check showed that everyone regardless of machine type was failing through to the visitor VLAN.  I was forced to remove aaa authentication from the switches in that area to restore service.  I'm now looking at troubleshooting and I can't find anything obviously wrong.   We made no changes that day and checking through all the rules, roles & policies I can find nothing obviously wrong.  The PC's in that area will work anywhere else as they are running the service and have the certs.  Any machine I take down there fails the same way.  I've gone through all the switch configs and they appear the same as the 5 other areas on site.  The only clue I have is that when I check the logs during testing is that I can see the PC fail through to the visitor network, then it passes and gets the corporate VLAN, then fails through again.  This cycle seems to repeat continuously (see pic attached).

The server was updated a few days ago to 6.12.5.306119, well after the issue surfaced.  None of the other areas set up in the system are affected so I assume it's something in that specific rule set since I can't see anything different in the switch configs or the PC's.  Anyone run into something similar with a constant flipping between authorised and failing?  I'm stumped.

Kenny,

What type of switch is the distribution layer (EUGRAC501)? If it is AOS-S you can you the following commands to check the MAC moves.

show mac-address d89ef3-04837d detail
edomtset
edomtset
addrmgrmovelist

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125
------------------------------

Original Message:
Sent: Jun 25, 2025 06:01 AM
From: Lord
Subject: Clearpass authorisation flipping off and on

I took another look at the debugoutput (Authentication debug.txt) that Kenny shared.
In line 87 it says  "Switch Mac move indication received", the client is then deauthenticated.

This matches the MAC is flapping between the uplink and accessport.
Is there really a loop in this LAN segment?

------------------------------
Regards,

Waldemar
ACCX # 1377, ACEP, ACX - Network Security
If you find my answer useful, consider giving kudos and/or mark as solution

Original Message:
Sent: Jun 25, 2025 05:26 AM
From: willembargeman
Subject: Clearpass authorisation flipping off and on

you didn't physically move the network connection of the device right? I don't understand why the MAC is flapping between the uplink and downlink (for the same VLAN). Normally that indicates a loop in the network. Can you check the core/distribution layer if you see MAC flaps for the MAC? And also try to remove the mac-pin configuration 

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 25, 2025 05:10 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Original Message:
Sent: 6/25/2025 4:50:00 AM
From: willembargeman
Subject: RE: Clearpass authorisation flipping off and on

Hi Ken,

I see a lot of MAC-moves for MAC d89ef3-04837d. The MAC address flaps between port 18 and Trk1.

addrmgrmovelistmac address     vid    old port   new port   timestamp-------------   ----   --------   --------   -----------------d89ef3-04837d    501      18         Trk1    06/25/25 09:00:42d89ef3-04837d    501      Trk1       18      06/25/25 09:02:43d89ef3-04837d    501      18         Trk1    06/25/25 09:02:43d89ef3-04837d    501      Trk1       18      06/25/25 09:02:43d89ef3-04837d    501      18         Trk1    06/25/25 09:02:44d89ef3-04837d    501      Trk1       18      06/25/25 09:04:45d89ef3-04837d    501      18         Trk1    06/25/25 09:04:46d89ef3-04837d    501      Trk1       18      06/25/25 09:04:46d89ef3-04837d    501      18         Trk1    06/25/25 09:04:46d89ef3-04837d    501      Trk1       18      06/25/25 09:06:48d89ef3-04837d    501      18         Trk1    06/25/25 09:06:48d89ef3-04837d    501      Trk1       18      06/25/25 09:06:48d89ef3-04837d    501      18         Trk1    06/25/25 09:06:48d89ef3-04837d    501      Trk1       18      06/25/25 09:08:50d89ef3-04837d    501      18         Trk1    06/25/25 09:08:50d89ef3-04837d    501      Trk1       18      06/25/25 09:08:50d89ef3-04837d    501      18         Trk1    06/25/25 09:08:53d89ef3-04837d    501      Trk1       18      06/25/25 09:10:52d89ef3-04837d    501      18         Trk1    06/25/25 09:10:53d89ef3-04837d    501      Trk1       18      06/25/25 09:10:53d89ef3-04837d    501      18         Trk1    06/25/25 09:10:53d89ef3-04837d    501      Trk1       18      06/25/25 09:12:55d89ef3-04837d    501      18         Trk1    06/25/25 09:12:55d89ef3-04837d    501      Trk1       18      06/25/25 09:12:55d89ef3-04837d    501      18         Trk1    06/25/25 09:12:55

Can you trace down this MAC? I'm curious why this MAC is also seen on the uplink. 

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 25, 2025 04:25 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Original Message:
Sent: 6/25/2025 3:54:00 AM
From: willembargeman
Subject: RE: Clearpass authorisation flipping off and on

Hi Kenny,

Thank you for sharing the show tech. Please can you share a "show tech all" and not "show tech"?

The configuration it self does like correct.

In the logs I see mac-moves. Can there be a loop in the network? Please share the "show tech all" so I have more information available.

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 25, 2025 03:36 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi Gorazd.  Here's the output and I've attached 2 screenshots.  One is when the 802.1x works and the PC is authorised as normal, and the second is when the MAC check then immediately fails it through to the visitor VLAN (95).  Let me know if you need anything else.

Original Message:
Sent: Jun 24, 2025 02:37 PM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi Kenny.

Can you please post Output tab result from access tracker and show vlan <802.1x vlan> on switch? Behaviour is consistent with wrong vlan value on radius response.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 24, 2025 09:23 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

The VLAN's are definitely on the switches.  There's data, voice, visitor and quarantine VLANs on all switches.  There are 10 switches in this area and all are configured the same and were working previously.  All switches in this area stopped working late on a Friday afternoon at the same time and I had to deactivate the authentication to get everyone back online.  The problem only affects this one area.  I am the only one who normally works on the Clearpass system and I made no changes to the switches or servers that day.  However, others do have access to the system who know even less than I do and one of them may have made a change. :-(  I think it more likely something on the server has been messed with rather than config on 10 switches. 

Original Message:
Sent: Jun 24, 2025 08:51 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

In majority of cases this mean that vlan in radius response is not found on switch.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 24, 2025 08:40 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi again.  I've attached the debug output from the moment I applied the config to the port again to turn on authentication.

One thing that might be of interest is that I just applied auth-order and auth-priority as suggested above by Lord.  I set it to authenticator and then MAC and turned on the port.  The result was that the port stuck on the default 'fake' VLAN 666 which all our ports are set to when nothing is plugged in.  The access tracker shows that 802.1x authentication worked but the VLAN does not get applied.

Original Message:
Sent: Jun 24, 2025 07:35 AM
From: willembargeman
Subject: Clearpass authorisation flipping off and on

The log doesn't show CoA.

Can you enable more debug logs on the switch. Just enable all the security logs for now

debug securitydebug destination session

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 24, 2025 06:43 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi Willem.  I do not see any change of authentication notification or additional tab in the access tracker.  The 802.1 and MAC notifications look like every other notification with no additions.  It authenticates correctly using 802.1x, then it authenticates twice using the MAC rule which fails it through to the visitor VLAN.  For some reason it is authenticating against the MAC twice every time but I'm not sure why.  They look identical.

Original Message:
Sent: Jun 24, 2025 06:15 AM
From: willembargeman
Subject: Clearpass authorisation flipping off and on

In the logs I see the message "Deauthentication request received".

Can you check in the access tracker if ClearPass is sending a CoA? In the access tracker details there should be an additional tab when a CoA is send

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 24, 2025 06:07 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi again.  Thanks for the correct debug commands gents, they worked fine.  I've attached a sample text showing the loop I'm seeing.

Waldemar - thanks for the config, I'll look into adding the additional commands you recommend.

Willem - we are not hitting our client limits.  I've got around 800 spare access licenses and 1100 spare OnGuard licenses so it's not that.  It looks like the switch is doing exactly what it is being told to do.  It seems to be authenticating using 802.1x and then being de-authenticated by the MAC check immediately after it.  I can see the VLAN flicking constantly between the corporate (501) and the visitor (95).

I have attached a copy of the debug log when I activated MAC authentication again as well as 802.1x on the port.  It seems to confirm that the user is authenticating properly and then the MAC address fail through check runs immediately after and kicks it into the visitor VLAN.  I don't know why the MAC check is running after a successful authentication but that seems to be the problem.

My network covers a large industrial site of 6 square miles so is split into 6 network areas.  Each area has its own distributions switches (8320x's) feeding the access switches (2930M's) and each area has its own group of data, voice and visitor VLANs.  Clearpass has profiles and rules for each VLAN in each area and the problem only affects this one area.  The other areas have near identical configuration on the switches and rules with only the VLAN numbers changing.   

Original Message:
Sent: Jun 24, 2025 05:41 AM
From: Lord
Subject: Clearpass authorisation flipping off and on

Hi Kenny,
heir is an example to enable debugging on an AOS-S switch. You can enable it globally for all ports or for a specific port.

deb security port-access authenticator include port 2
deb security port-access mac-based include port 2

You can also use debug security radius-server. However, this command is used for all sessions, depending on what is currently going on on the switch, it can become unmanageable.

It is more easier if you send debug output to a separate CLI session. Just open an additional session and enter debug destination session there.

The port configuration looks OK. I would also set auth-order and auth-priority explicitly and specify the unauth-vid.
I always use the following config:

aaa port-access authenticator 2
aaa port-access mac-based 2
aaa port-access mac-based 2 mac-pin
aaa port-access mac-based 2 unauth-vid 500
aaa port-access 2 auth-order authenticator mac-based
aaa port-access 2 auth-priority authenticator mac-based
aaa port-access 2 controlled-direction in

------------------------------
Regards,

Waldemar
ACCX # 1377, ACEP, ACX - Network Security
If you find my answer useful, consider giving kudos and/or mark as solution

Original Message:
Sent: Jun 23, 2025 10:29 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Thanks for the replies, gents.  The VLANs exist and are correct.  I've tried to get debugging working but there's no guide to actually debugging for the AOS used on the 2930M's that our network uses.  Every search for 802.1x or NAC debugging takes me straight to the HPE page for debugging 802.1x on OS-CX switches instead.

What I have discovered today is that it appears to be the MAC side of things that's causing the issue.  The actual 802.1x is working fine.  The port-access authenticator assigns the corporate VLAN correctly and the PC works.  If I turn on the MAC-based check that comes after it in the config the port fails through to the visitor VLAN immediately.  The check can't match the corporate PC's to a phone or printer MAC so it fails.  Why it's doing the check even through the first test has passed is what I don't know.  This config works elsewhere on the network but something must be different for the rule set here.  It is not stopping the checks and moves on to the MAC check which will always fail the corporate PC's.

For reference, here's an example of the aaa config on our switch ports.  

aaa accounting commands stop-only tacacs
aaa accounting exec start-stop tacacs
aaa accounting system stop-only tacacs
aaa authentication login privilege-mode
aaa authentication ssh login tacacs
aaa authentication ssh enable tacacs
aaa authentication port-access eap-radius
aaa port-access authenticator 1
aaa port-access authenticator 1 client-limit 2
aaa port-access authenticator active aaa port-access mac-based 1

aaa port-access mac-based 2 mac-pin

aaa port-access mac-based addr-format multi-colon

Original Message:
Sent: Jun 22, 2025 02:14 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi. 

Check the logs on the switch and maybe enable 802.1x debugging. Lioks like something in send by enforcement profile is not what switch is expecting.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 20, 2025 10:52 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi there.  We have a large industrial site with Clearpass handling authentication and automatic VLAN selection for each area.  We have the system check for 802.1x details and it checks those against our domain controller machine list.  A GPO pushes out the certificates and makes sure the 802.1x service is on by default on our machines.  If the check is passed the machine is assigned the local data VLAN for that area.  The fail-through rule for this is a MAC address check.  The system checks against a host list which has our printers, access control devices, etc and if on the list they are also assigned the local data VLAN.  Ultimate fail-through put anything that fails these tests into a separate visitors VLAN which can't access any corporate stuff and is basically straight to internet.

I was called out on a Friday evening because one one of our areas could not access systems and servers.  A quick check showed that everyone regardless of machine type was failing through to the visitor VLAN.  I was forced to remove aaa authentication from the switches in that area to restore service.  I'm now looking at troubleshooting and I can't find anything obviously wrong.   We made no changes that day and checking through all the rules, roles & policies I can find nothing obviously wrong.  The PC's in that area will work anywhere else as they are running the service and have the certs.  Any machine I take down there fails the same way.  I've gone through all the switch configs and they appear the same as the 5 other areas on site.  The only clue I have is that when I check the logs during testing is that I can see the PC fail through to the visitor network, then it passes and gets the corporate VLAN, then fails through again.  This cycle seems to repeat continuously (see pic attached).

The server was updated a few days ago to 6.12.5.306119, well after the issue surfaced.  None of the other areas set up in the system are affected so I assume it's something in that specific rule set since I can't see anything different in the switch configs or the PC's.  Anyone run into something similar with a constant flipping between authorised and failing?  I'm stumped.

Kenny,

What type of switch is the distribution layer (EUGRAC501)? If it is AOS-S you can you the following commands to check the MAC moves.

show mac-address d89ef3-04837d detailedomtsetedomtsetaddrmgrmovelist

I took another look at the debugoutput (Authentication debug.txt) that Kenny shared.
In line 87 it says  "Switch Mac move indication received", the client is then deauthenticated.

This matches the MAC is flapping between the uplink and accessport.
Is there really a loop in this LAN segment?

you didn't physically move the network connection of the device right? I don't understand why the MAC is flapping between the uplink and downlink (for the same VLAN). Normally that indicates a loop in the network. Can you check the core/distribution layer if you see MAC flaps for the MAC? And also try to remove the mac-pin configuration 

Hi Ken,

I see a lot of MAC-moves for MAC d89ef3-04837d. The MAC address flaps between port 18 and Trk1.

addrmgrmovelistmac address     vid    old port   new port   timestamp-------------   ----   --------   --------   -----------------d89ef3-04837d    501      18         Trk1    06/25/25 09:00:42d89ef3-04837d    501      Trk1       18      06/25/25 09:02:43d89ef3-04837d    501      18         Trk1    06/25/25 09:02:43d89ef3-04837d    501      Trk1       18      06/25/25 09:02:43d89ef3-04837d    501      18         Trk1    06/25/25 09:02:44d89ef3-04837d    501      Trk1       18      06/25/25 09:04:45d89ef3-04837d    501      18         Trk1    06/25/25 09:04:46d89ef3-04837d    501      Trk1       18      06/25/25 09:04:46d89ef3-04837d    501      18         Trk1    06/25/25 09:04:46d89ef3-04837d    501      Trk1       18      06/25/25 09:06:48d89ef3-04837d    501      18         Trk1    06/25/25 09:06:48d89ef3-04837d    501      Trk1       18      06/25/25 09:06:48d89ef3-04837d    501      18         Trk1    06/25/25 09:06:48d89ef3-04837d    501      Trk1       18      06/25/25 09:08:50d89ef3-04837d    501      18         Trk1    06/25/25 09:08:50d89ef3-04837d    501      Trk1       18      06/25/25 09:08:50d89ef3-04837d    501      18         Trk1    06/25/25 09:08:53d89ef3-04837d    501      Trk1       18      06/25/25 09:10:52d89ef3-04837d    501      18         Trk1    06/25/25 09:10:53d89ef3-04837d    501      Trk1       18      06/25/25 09:10:53d89ef3-04837d    501      18         Trk1    06/25/25 09:10:53d89ef3-04837d    501      Trk1       18      06/25/25 09:12:55d89ef3-04837d    501      18         Trk1    06/25/25 09:12:55d89ef3-04837d    501      Trk1       18      06/25/25 09:12:55d89ef3-04837d    501      18         Trk1    06/25/25 09:12:55

Can you trace down this MAC? I'm curious why this MAC is also seen on the uplink. 

Hi Kenny,

Thank you for sharing the show tech. Please can you share a "show tech all" and not "show tech"?

The configuration it self does like correct.

In the logs I see mac-moves. Can there be a loop in the network? Please share the "show tech all" so I have more information available.

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 25, 2025 03:36 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi Gorazd.  Here's the output and I've attached 2 screenshots.  One is when the 802.1x works and the PC is authorised as normal, and the second is when the MAC check then immediately fails it through to the visitor VLAN (95).  Let me know if you need anything else.

Original Message:
Sent: Jun 24, 2025 02:37 PM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi Kenny.

Can you please post Output tab result from access tracker and show vlan <802.1x vlan> on switch? Behaviour is consistent with wrong vlan value on radius response.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 24, 2025 09:23 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

The VLAN's are definitely on the switches.  There's data, voice, visitor and quarantine VLANs on all switches.  There are 10 switches in this area and all are configured the same and were working previously.  All switches in this area stopped working late on a Friday afternoon at the same time and I had to deactivate the authentication to get everyone back online.  The problem only affects this one area.  I am the only one who normally works on the Clearpass system and I made no changes to the switches or servers that day.  However, others do have access to the system who know even less than I do and one of them may have made a change. :-(  I think it more likely something on the server has been messed with rather than config on 10 switches. 

Original Message:
Sent: Jun 24, 2025 08:51 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

In majority of cases this mean that vlan in radius response is not found on switch.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 24, 2025 08:40 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi again.  I've attached the debug output from the moment I applied the config to the port again to turn on authentication.

One thing that might be of interest is that I just applied auth-order and auth-priority as suggested above by Lord.  I set it to authenticator and then MAC and turned on the port.  The result was that the port stuck on the default 'fake' VLAN 666 which all our ports are set to when nothing is plugged in.  The access tracker shows that 802.1x authentication worked but the VLAN does not get applied.

Original Message:
Sent: Jun 24, 2025 07:35 AM
From: willembargeman
Subject: Clearpass authorisation flipping off and on

The log doesn't show CoA.

Can you enable more debug logs on the switch. Just enable all the security logs for now

debug securitydebug destination session

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 24, 2025 06:43 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi Willem.  I do not see any change of authentication notification or additional tab in the access tracker.  The 802.1 and MAC notifications look like every other notification with no additions.  It authenticates correctly using 802.1x, then it authenticates twice using the MAC rule which fails it through to the visitor VLAN.  For some reason it is authenticating against the MAC twice every time but I'm not sure why.  They look identical.

Original Message:
Sent: Jun 24, 2025 06:15 AM
From: willembargeman
Subject: Clearpass authorisation flipping off and on

In the logs I see the message "Deauthentication request received".

Can you check in the access tracker if ClearPass is sending a CoA? In the access tracker details there should be an additional tab when a CoA is send

------------------------------
Willem Bargeman
Systems Engineer Aruba
ACEX #125

Original Message:
Sent: Jun 24, 2025 06:07 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi again.  Thanks for the correct debug commands gents, they worked fine.  I've attached a sample text showing the loop I'm seeing.

Waldemar - thanks for the config, I'll look into adding the additional commands you recommend.

Willem - we are not hitting our client limits.  I've got around 800 spare access licenses and 1100 spare OnGuard licenses so it's not that.  It looks like the switch is doing exactly what it is being told to do.  It seems to be authenticating using 802.1x and then being de-authenticated by the MAC check immediately after it.  I can see the VLAN flicking constantly between the corporate (501) and the visitor (95).

I have attached a copy of the debug log when I activated MAC authentication again as well as 802.1x on the port.  It seems to confirm that the user is authenticating properly and then the MAC address fail through check runs immediately after and kicks it into the visitor VLAN.  I don't know why the MAC check is running after a successful authentication but that seems to be the problem.

My network covers a large industrial site of 6 square miles so is split into 6 network areas.  Each area has its own distributions switches (8320x's) feeding the access switches (2930M's) and each area has its own group of data, voice and visitor VLANs.  Clearpass has profiles and rules for each VLAN in each area and the problem only affects this one area.  The other areas have near identical configuration on the switches and rules with only the VLAN numbers changing.   

Original Message:
Sent: Jun 24, 2025 05:41 AM
From: Lord
Subject: Clearpass authorisation flipping off and on

Hi Kenny,
heir is an example to enable debugging on an AOS-S switch. You can enable it globally for all ports or for a specific port.

deb security port-access authenticator include port 2
deb security port-access mac-based include port 2

You can also use debug security radius-server. However, this command is used for all sessions, depending on what is currently going on on the switch, it can become unmanageable.

It is more easier if you send debug output to a separate CLI session. Just open an additional session and enter debug destination session there.

The port configuration looks OK. I would also set auth-order and auth-priority explicitly and specify the unauth-vid.
I always use the following config:

aaa port-access authenticator 2
aaa port-access mac-based 2
aaa port-access mac-based 2 mac-pin
aaa port-access mac-based 2 unauth-vid 500
aaa port-access 2 auth-order authenticator mac-based
aaa port-access 2 auth-priority authenticator mac-based
aaa port-access 2 controlled-direction in

------------------------------
Regards,

Waldemar
ACCX # 1377, ACEP, ACX - Network Security
If you find my answer useful, consider giving kudos and/or mark as solution

Original Message:
Sent: Jun 23, 2025 10:29 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Thanks for the replies, gents.  The VLANs exist and are correct.  I've tried to get debugging working but there's no guide to actually debugging for the AOS used on the 2930M's that our network uses.  Every search for 802.1x or NAC debugging takes me straight to the HPE page for debugging 802.1x on OS-CX switches instead.

What I have discovered today is that it appears to be the MAC side of things that's causing the issue.  The actual 802.1x is working fine.  The port-access authenticator assigns the corporate VLAN correctly and the PC works.  If I turn on the MAC-based check that comes after it in the config the port fails through to the visitor VLAN immediately.  The check can't match the corporate PC's to a phone or printer MAC so it fails.  Why it's doing the check even through the first test has passed is what I don't know.  This config works elsewhere on the network but something must be different for the rule set here.  It is not stopping the checks and moves on to the MAC check which will always fail the corporate PC's.

For reference, here's an example of the aaa config on our switch ports.  

aaa accounting commands stop-only tacacs
aaa accounting exec start-stop tacacs
aaa accounting system stop-only tacacs
aaa authentication login privilege-mode
aaa authentication ssh login tacacs
aaa authentication ssh enable tacacs
aaa authentication port-access eap-radius
aaa port-access authenticator 1
aaa port-access authenticator 1 client-limit 2
aaa port-access authenticator active aaa port-access mac-based 1

aaa port-access mac-based 2 mac-pin

aaa port-access mac-based addr-format multi-colon

Original Message:
Sent: Jun 22, 2025 02:14 AM
From: GorazdKikelj
Subject: Clearpass authorisation flipping off and on

Hi. 

Check the logs on the switch and maybe enable 802.1x debugging. Lioks like something in send by enforcement profile is not what switch is expecting.

Best, Gorazd

------------------------------
Gorazd Kikelj
MVP Guru 2025

Original Message:
Sent: Jun 20, 2025 10:52 AM
From: Kenny_10_Bellys
Subject: Clearpass authorisation flipping off and on

Hi there.  We have a large industrial site with Clearpass handling authentication and automatic VLAN selection for each area.  We have the system check for 802.1x details and it checks those against our domain controller machine list.  A GPO pushes out the certificates and makes sure the 802.1x service is on by default on our machines.  If the check is passed the machine is assigned the local data VLAN for that area.  The fail-through rule for this is a MAC address check.  The system checks against a host list which has our printers, access control devices, etc and if on the list they are also assigned the local data VLAN.  Ultimate fail-through put anything that fails these tests into a separate visitors VLAN which can't access any corporate stuff and is basically straight to internet.

I was called out on a Friday evening because one one of our areas could not access systems and servers.  A quick check showed that everyone regardless of machine type was failing through to the visitor VLAN.  I was forced to remove aaa authentication from the switches in that area to restore service.  I'm now looking at troubleshooting and I can't find anything obviously wrong.   We made no changes that day and checking through all the rules, roles & policies I can find nothing obviously wrong.  The PC's in that area will work anywhere else as they are running the service and have the certs.  Any machine I take down there fails the same way.  I've gone through all the switch configs and they appear the same as the 5 other areas on site.  The only clue I have is that when I check the logs during testing is that I can see the PC fail through to the visitor network, then it passes and gets the corporate VLAN, then fails through again.  This cycle seems to repeat continuously (see pic attached).

The server was updated a few days ago to 6.12.5.306119, well after the issue surfaced.  None of the other areas set up in the system are affected so I assume it's something in that specific rule set since I can't see anything different in the switch configs or the PC's.  Anyone run into something similar with a constant flipping between authorised and failing?  I'm stumped.

Can you try to use the following commands on the CX switch to see the mac moves?

show mac-address-table mac-move
show mac-address-table mac-move address <mac-address>