Security

I am part way through rolling out Clearpass across my Aruba campus network and a strange issue is starting to show up.  I am getting reports from users around the site that they are being randomly dropped from the network for a few seconds maybe 3 or 4 times a day.  More and more people are reporting this and when I check the network switches I'm seeing something odd.

When I look at the ports of the affected users there's additional MAC addresses showing up.  I have the client and address limit settings on the port to limit it to 2 devices, basically a VoIP phone and a PC.  When I look at the switch I see 3 MAC addresses, the expected addresses getting the correct VLANs and then a third PC's MAC address getting put onto the visitor/fail through VLAN.  If I increase the clearpass client limit on the port I will get more spurious MAC addresses showing up on the port, all being assigned to the fail through VLAN.  The MAC's are not related to the client directly connected to the port, and there are no other devices connected to that port.

The MAC addresses appear to be real, in that the MAC vendor codes are real, but I cannot see them anywhere on our network.  The users start picking up 2 gateways, one for the correct VLAN and one from the fail through VLAN or a 192.168.0.1.  This effectively knocks them off the network.  I have also seen them get a DNS server in the 192 range as well.  I have no clue where it's coming from.  I have DHCP snooping on my switches so I don't believe it's a local port with a home router plugged in or something.

Anyone any idea why ports with a single PC or a PC & phone plugged in will start randomly looking like I've plugged additional devices in?  It seems to flip through them and the actual user gets disconnected for a few seconds.  I've no clue why this is happening.  I've rebooted the switches to get them up to the latest firmware levels just in case.  Switches are Aruba 2930M's and 5400's and Clearpass was updated a couple of months ago.  Very odd.

Any possibility a network loop might have been introduced unintentionally by an end-user?

We had reports of connectivity issues right about Christmas Break in a residence hall. We noticed that several wired MAC addresses appeared to be moving back and forth to one port that hosted a 205H with three bridged ports on the bottom. A student had plugged two ends of same cord into AP itself (for cable management :-p). The "broadcast/multicast traffic" was going out the AP - then back in - so the source mac address of other devices were showing up on that port without the device being actually connected. We enabled loop detection on the 205Hs after that.

Might give you a place to start. We also had some people accidentally plug a VoIP Phone into two wall jacks (PC Port into one wall jack and Switch Port into another wall jack). However, in those cases the Cisco Switch places port into err-disabled mode I think because of BPDU Guard.

Hope this helps.
Original Message:
Sent: Aug 25, 2022 11:02 AM
From: Kenny Mitchell
Subject: Unknown additional MAC addresses showing up on ports

I am part way through rolling out Clearpass across my Aruba campus network and a strange issue is starting to show up.  I am getting reports from users around the site that they are being randomly dropped from the network for a few seconds maybe 3 or 4 times a day.  More and more people are reporting this and when I check the network switches I'm seeing something odd.

When I look at the ports of the affected users there's additional MAC addresses showing up.  I have the client and address limit settings on the port to limit it to 2 devices, basically a VoIP phone and a PC.  When I look at the switch I see 3 MAC addresses, the expected addresses getting the correct VLANs and then a third PC's MAC address getting put onto the visitor/fail through VLAN.  If I increase the clearpass client limit on the port I will get more spurious MAC addresses showing up on the port, all being assigned to the fail through VLAN.  The MAC's are not related to the client directly connected to the port, and there are no other devices connected to that port.

The MAC addresses appear to be real, in that the MAC vendor codes are real, but I cannot see them anywhere on our network.  The users start picking up 2 gateways, one for the correct VLAN and one from the fail through VLAN or a 192.168.0.1.  This effectively knocks them off the network.  I have also seen them get a DNS server in the 192 range as well.  I have no clue where it's coming from.  I have DHCP snooping on my switches so I don't believe it's a local port with a home router plugged in or something.

Anyone any idea why ports with a single PC or a PC & phone plugged in will start randomly looking like I've plugged additional devices in?  It seems to flip through them and the actual user gets disconnected for a few seconds.  I've no clue why this is happening.  I've rebooted the switches to get them up to the latest firmware levels just in case.  Switches are Aruba 2930M's and 5400's and Clearpass was updated a couple of months ago.  Very odd.

I don't think that's the issue here.  I've had chaos ensue when someone plugged in both ports on a phone as well and it only took down that subnet area.  I also have contractors who rock up with a home router, plug it in and bring down a subnet too.  I now run DHCP Snooping and loop protect on all my switches to prevent this from happening again.

What I'm seeing is all over my entire campus, but only on Clearpass enabled switches.  It's not confined to one subnet, switch or area.  Maybe one in 10 or so ports just has unknown MAC addresses appear on the port, causing the real user to get knocked off every now and again.  It feels to me like something is plugged in as well but there's no way it can be.
Original Message:
Sent: Aug 26, 2022 11:02 AM
From: Christopher Johnson
Subject: Unknown additional MAC addresses showing up on ports

Any possibility a network loop might have been introduced unintentionally by an end-user?

We had reports of connectivity issues right about Christmas Break in a residence hall. We noticed that several wired MAC addresses appeared to be moving back and forth to one port that hosted a 205H with three bridged ports on the bottom. A student had plugged two ends of same cord into AP itself (for cable management :-p). The "broadcast/multicast traffic" was going out the AP - then back in - so the source mac address of other devices were showing up on that port without the device being actually connected. We enabled loop detection on the 205Hs after that.

Might give you a place to start. We also had some people accidentally plug a VoIP Phone into two wall jacks (PC Port into one wall jack and Switch Port into another wall jack). However, in those cases the Cisco Switch places port into err-disabled mode I think because of BPDU Guard.

Hope this helps.
Original Message:
Sent: Aug 25, 2022 11:02 AM
From: Kenny Mitchell
Subject: Unknown additional MAC addresses showing up on ports

I am part way through rolling out Clearpass across my Aruba campus network and a strange issue is starting to show up.  I am getting reports from users around the site that they are being randomly dropped from the network for a few seconds maybe 3 or 4 times a day.  More and more people are reporting this and when I check the network switches I'm seeing something odd.

When I look at the ports of the affected users there's additional MAC addresses showing up.  I have the client and address limit settings on the port to limit it to 2 devices, basically a VoIP phone and a PC.  When I look at the switch I see 3 MAC addresses, the expected addresses getting the correct VLANs and then a third PC's MAC address getting put onto the visitor/fail through VLAN.  If I increase the clearpass client limit on the port I will get more spurious MAC addresses showing up on the port, all being assigned to the fail through VLAN.  The MAC's are not related to the client directly connected to the port, and there are no other devices connected to that port.

The MAC addresses appear to be real, in that the MAC vendor codes are real, but I cannot see them anywhere on our network.  The users start picking up 2 gateways, one for the correct VLAN and one from the fail through VLAN or a 192.168.0.1.  This effectively knocks them off the network.  I have also seen them get a DNS server in the 192 range as well.  I have no clue where it's coming from.  I have DHCP snooping on my switches so I don't believe it's a local port with a home router plugged in or something.

Anyone any idea why ports with a single PC or a PC & phone plugged in will start randomly looking like I've plugged additional devices in?  It seems to flip through them and the actual user gets disconnected for a few seconds.  I've no clue why this is happening.  I've rebooted the switches to get them up to the latest firmware levels just in case.  Switches are Aruba 2930M's and 5400's and Clearpass was updated a couple of months ago.  Very odd.

So a couple of options I have seen previously that are similar here:

• Software on the computer is presenting multiple NIC cards.  We've seen this before on Airheads with specialized video conference software.  
• VM/VirtualBox software on the PC not set to NAT mode.
• Switch code bug that leaks the MAC address table across multiple switch ports.  What is your NAD?  Code version?

Original Message:
Sent: Aug 31, 2022 04:17 AM
From: Kenny Mitchell
Subject: Unknown additional MAC addresses showing up on ports

I don't think that's the issue here.  I've had chaos ensue when someone plugged in both ports on a phone as well and it only took down that subnet area.  I also have contractors who rock up with a home router, plug it in and bring down a subnet too.  I now run DHCP Snooping and loop protect on all my switches to prevent this from happening again.

What I'm seeing is all over my entire campus, but only on Clearpass enabled switches.  It's not confined to one subnet, switch or area.  Maybe one in 10 or so ports just has unknown MAC addresses appear on the port, causing the real user to get knocked off every now and again.  It feels to me like something is plugged in as well but there's no way it can be.
Original Message:
Sent: Aug 26, 2022 11:02 AM
From: Christopher Johnson
Subject: Unknown additional MAC addresses showing up on ports

Any possibility a network loop might have been introduced unintentionally by an end-user?

We had reports of connectivity issues right about Christmas Break in a residence hall. We noticed that several wired MAC addresses appeared to be moving back and forth to one port that hosted a 205H with three bridged ports on the bottom. A student had plugged two ends of same cord into AP itself (for cable management :-p). The "broadcast/multicast traffic" was going out the AP - then back in - so the source mac address of other devices were showing up on that port without the device being actually connected. We enabled loop detection on the 205Hs after that.

Might give you a place to start. We also had some people accidentally plug a VoIP Phone into two wall jacks (PC Port into one wall jack and Switch Port into another wall jack). However, in those cases the Cisco Switch places port into err-disabled mode I think because of BPDU Guard.

Hope this helps.
Original Message:
Sent: Aug 25, 2022 11:02 AM
From: Kenny Mitchell
Subject: Unknown additional MAC addresses showing up on ports

I am part way through rolling out Clearpass across my Aruba campus network and a strange issue is starting to show up.  I am getting reports from users around the site that they are being randomly dropped from the network for a few seconds maybe 3 or 4 times a day.  More and more people are reporting this and when I check the network switches I'm seeing something odd.

When I look at the ports of the affected users there's additional MAC addresses showing up.  I have the client and address limit settings on the port to limit it to 2 devices, basically a VoIP phone and a PC.  When I look at the switch I see 3 MAC addresses, the expected addresses getting the correct VLANs and then a third PC's MAC address getting put onto the visitor/fail through VLAN.  If I increase the clearpass client limit on the port I will get more spurious MAC addresses showing up on the port, all being assigned to the fail through VLAN.  The MAC's are not related to the client directly connected to the port, and there are no other devices connected to that port.

The MAC addresses appear to be real, in that the MAC vendor codes are real, but I cannot see them anywhere on our network.  The users start picking up 2 gateways, one for the correct VLAN and one from the fail through VLAN or a 192.168.0.1.  This effectively knocks them off the network.  I have also seen them get a DNS server in the 192 range as well.  I have no clue where it's coming from.  I have DHCP snooping on my switches so I don't believe it's a local port with a home router plugged in or something.

Anyone any idea why ports with a single PC or a PC & phone plugged in will start randomly looking like I've plugged additional devices in?  It seems to flip through them and the actual user gets disconnected for a few seconds.  I've no clue why this is happening.  I've rebooted the switches to get them up to the latest firmware levels just in case.  Switches are Aruba 2930M's and 5400's and Clearpass was updated a couple of months ago.  Very odd.

Hi again.  After much experimentation and checking it looks like your option of leaking MAC addresses is the most probable.  What we have found is that the additional MAC addresses showing up on the ports are actual MAC addresses of PC's on other ports on the switch.  For some reason they are being duplicated to a second port, Clearpass gets a request to check the MAC address and then assigns the duplicate to our guest VLAN.  This causes the user number on the port to exceed our settings and every few minutes the addresses fight for dominance and the real user gets kicked off.  The process then starts again.  If I increase the user count then more MAC addresses eventually show up until the user is kicked again.

I am going to raise a support case with Aruba for this one.  The switches on site are on 16.11.0004 and Clearpass is on 6.9.6 but I've got a 6.10.5 upgrade queued up.
Original Message:
Sent: Aug 31, 2022 08:12 AM
From: Unknown User
Subject: Unknown additional MAC addresses showing up on ports

So a couple of options I have seen previously that are similar here:

• Software on the computer is presenting multiple NIC cards.  We've seen this before on Airheads with specialized video conference software.  
• VM/VirtualBox software on the PC not set to NAT mode.
• Switch code bug that leaks the MAC address table across multiple switch ports.  What is your NAD?  Code version?

Original Message:
Sent: Aug 31, 2022 04:17 AM
From: Kenny Mitchell
Subject: Unknown additional MAC addresses showing up on ports

I don't think that's the issue here.  I've had chaos ensue when someone plugged in both ports on a phone as well and it only took down that subnet area.  I also have contractors who rock up with a home router, plug it in and bring down a subnet too.  I now run DHCP Snooping and loop protect on all my switches to prevent this from happening again.

What I'm seeing is all over my entire campus, but only on Clearpass enabled switches.  It's not confined to one subnet, switch or area.  Maybe one in 10 or so ports just has unknown MAC addresses appear on the port, causing the real user to get knocked off every now and again.  It feels to me like something is plugged in as well but there's no way it can be.
Original Message:
Sent: Aug 26, 2022 11:02 AM
From: Christopher Johnson
Subject: Unknown additional MAC addresses showing up on ports

Any possibility a network loop might have been introduced unintentionally by an end-user?

We had reports of connectivity issues right about Christmas Break in a residence hall. We noticed that several wired MAC addresses appeared to be moving back and forth to one port that hosted a 205H with three bridged ports on the bottom. A student had plugged two ends of same cord into AP itself (for cable management :-p). The "broadcast/multicast traffic" was going out the AP - then back in - so the source mac address of other devices were showing up on that port without the device being actually connected. We enabled loop detection on the 205Hs after that.

Might give you a place to start. We also had some people accidentally plug a VoIP Phone into two wall jacks (PC Port into one wall jack and Switch Port into another wall jack). However, in those cases the Cisco Switch places port into err-disabled mode I think because of BPDU Guard.

Hope this helps.
Original Message:
Sent: Aug 25, 2022 11:02 AM
From: Kenny Mitchell
Subject: Unknown additional MAC addresses showing up on ports

I am part way through rolling out Clearpass across my Aruba campus network and a strange issue is starting to show up.  I am getting reports from users around the site that they are being randomly dropped from the network for a few seconds maybe 3 or 4 times a day.  More and more people are reporting this and when I check the network switches I'm seeing something odd.

When I look at the ports of the affected users there's additional MAC addresses showing up.  I have the client and address limit settings on the port to limit it to 2 devices, basically a VoIP phone and a PC.  When I look at the switch I see 3 MAC addresses, the expected addresses getting the correct VLANs and then a third PC's MAC address getting put onto the visitor/fail through VLAN.  If I increase the clearpass client limit on the port I will get more spurious MAC addresses showing up on the port, all being assigned to the fail through VLAN.  The MAC's are not related to the client directly connected to the port, and there are no other devices connected to that port.

The MAC addresses appear to be real, in that the MAC vendor codes are real, but I cannot see them anywhere on our network.  The users start picking up 2 gateways, one for the correct VLAN and one from the fail through VLAN or a 192.168.0.1.  This effectively knocks them off the network.  I have also seen them get a DNS server in the 192 range as well.  I have no clue where it's coming from.  I have DHCP snooping on my switches so I don't believe it's a local port with a home router plugged in or something.

Anyone any idea why ports with a single PC or a PC & phone plugged in will start randomly looking like I've plugged additional devices in?  It seems to flip through them and the actual user gets disconnected for a few seconds.  I've no clue why this is happening.  I've rebooted the switches to get them up to the latest firmware levels just in case.  Switches are Aruba 2930M's and 5400's and Clearpass was updated a couple of months ago.  Very odd.

Hello Kenny_10_Bellys,

I have the same problem with Comware switch 5130.

and did you fix the Problem?

Thank you
Original Message:
Sent: Sep 14, 2022 05:04 AM
From: Kenny Mitchell
Subject: Unknown additional MAC addresses showing up on ports

Hi again.  After much experimentation and checking it looks like your option of leaking MAC addresses is the most probable.  What we have found is that the additional MAC addresses showing up on the ports are actual MAC addresses of PC's on other ports on the switch.  For some reason they are being duplicated to a second port, Clearpass gets a request to check the MAC address and then assigns the duplicate to our guest VLAN.  This causes the user number on the port to exceed our settings and every few minutes the addresses fight for dominance and the real user gets kicked off.  The process then starts again.  If I increase the user count then more MAC addresses eventually show up until the user is kicked again.

I am going to raise a support case with Aruba for this one.  The switches on site are on 16.11.0004 and Clearpass is on 6.9.6 but I've got a 6.10.5 upgrade queued up.
Original Message:
Sent: Aug 31, 2022 08:12 AM
From: Unknown User
Subject: Unknown additional MAC addresses showing up on ports

So a couple of options I have seen previously that are similar here:

• Software on the computer is presenting multiple NIC cards.  We've seen this before on Airheads with specialized video conference software.  
• VM/VirtualBox software on the PC not set to NAT mode.
• Switch code bug that leaks the MAC address table across multiple switch ports.  What is your NAD?  Code version?

Original Message:
Sent: Aug 31, 2022 04:17 AM
From: Kenny Mitchell
Subject: Unknown additional MAC addresses showing up on ports

I don't think that's the issue here.  I've had chaos ensue when someone plugged in both ports on a phone as well and it only took down that subnet area.  I also have contractors who rock up with a home router, plug it in and bring down a subnet too.  I now run DHCP Snooping and loop protect on all my switches to prevent this from happening again.

What I'm seeing is all over my entire campus, but only on Clearpass enabled switches.  It's not confined to one subnet, switch or area.  Maybe one in 10 or so ports just has unknown MAC addresses appear on the port, causing the real user to get knocked off every now and again.  It feels to me like something is plugged in as well but there's no way it can be.
Original Message:
Sent: Aug 26, 2022 11:02 AM
From: Christopher Johnson
Subject: Unknown additional MAC addresses showing up on ports

Any possibility a network loop might have been introduced unintentionally by an end-user?

We had reports of connectivity issues right about Christmas Break in a residence hall. We noticed that several wired MAC addresses appeared to be moving back and forth to one port that hosted a 205H with three bridged ports on the bottom. A student had plugged two ends of same cord into AP itself (for cable management :-p). The "broadcast/multicast traffic" was going out the AP - then back in - so the source mac address of other devices were showing up on that port without the device being actually connected. We enabled loop detection on the 205Hs after that.

Might give you a place to start. We also had some people accidentally plug a VoIP Phone into two wall jacks (PC Port into one wall jack and Switch Port into another wall jack). However, in those cases the Cisco Switch places port into err-disabled mode I think because of BPDU Guard.

Hope this helps.
Original Message:
Sent: Aug 25, 2022 11:02 AM
From: Kenny Mitchell
Subject: Unknown additional MAC addresses showing up on ports

I am part way through rolling out Clearpass across my Aruba campus network and a strange issue is starting to show up.  I am getting reports from users around the site that they are being randomly dropped from the network for a few seconds maybe 3 or 4 times a day.  More and more people are reporting this and when I check the network switches I'm seeing something odd.

When I look at the ports of the affected users there's additional MAC addresses showing up.  I have the client and address limit settings on the port to limit it to 2 devices, basically a VoIP phone and a PC.  When I look at the switch I see 3 MAC addresses, the expected addresses getting the correct VLANs and then a third PC's MAC address getting put onto the visitor/fail through VLAN.  If I increase the clearpass client limit on the port I will get more spurious MAC addresses showing up on the port, all being assigned to the fail through VLAN.  The MAC's are not related to the client directly connected to the port, and there are no other devices connected to that port.

The MAC addresses appear to be real, in that the MAC vendor codes are real, but I cannot see them anywhere on our network.  The users start picking up 2 gateways, one for the correct VLAN and one from the fail through VLAN or a 192.168.0.1.  This effectively knocks them off the network.  I have also seen them get a DNS server in the 192 range as well.  I have no clue where it's coming from.  I have DHCP snooping on my switches so I don't believe it's a local port with a home router plugged in or something.

Anyone any idea why ports with a single PC or a PC & phone plugged in will start randomly looking like I've plugged additional devices in?  It seems to flip through them and the actual user gets disconnected for a few seconds.  I've no clue why this is happening.  I've rebooted the switches to get them up to the latest firmware levels just in case.  Switches are Aruba 2930M's and 5400's and Clearpass was updated a couple of months ago.  Very odd.