Controllerless Networks

Hello all,

we have 15 Instant APs (IAP-105) that we are using for WI-FI services in a small office. recently I have observed excessive ARP traffic on the Management network. so excessive that  up to 93% of traffic on the IAP management VLAN is ARP traffic. (I have a separate VLAN for AP Management traffic. this VLAN is untagged on the switch port, and other data from SSIDs is tagged) the ARP traffic takes several forms:

* ARP floods of an AP requesting the MAC of another.
* Gratuitous ARP traffic from the Controller AP to the Virtual Controller IP (it should hold that IP itself)
* ARP floods from the Gateway of the Subnet (a Cisco Firewall) looking for an IP that doesn't exist on the subnet. (I have a case with Cisco about that traffic).

ARP traffic is fine and I have it contained on the AP management VLAN, except it's overwhelming the management plane on a switch that's the DHCP server for the VLAN.

any ideas what's the issue and how to stop it?

I'd appreciate any input.

Do you have packet captures? How many packets per second do you see on each of the categories?

That a gateway is doing ARPs for IPs that are not in use is probably a scan (or clients) trying to reach those IPs. If the firewall allows that traffic, it will do an ARP request for the IP. So for that one, check on the other interfaces/logs what client is trying to reach those IPs, or bring up a client with that IP and see the source IP as soon as the ARP is successful, or create a static ARP on your gateway.

The first step is to determine if the traffic is normal, if you see the same ARP multiple times a second, that is not normal, and you may reach out to your Aruba partner or Aruba support to get it analyzed. If you see repeating the same AP requesting ARP for the same other AP, it may be good to check if the AP is responding, and there may be an ACL in your network or so preventing the response to reach the original AP, or you may have a damaged cable that even may work in only one direction. The generic information above is far from sufficient to tell anything useful.

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.
------------------------------

Original Message:
Sent: Sep 15, 2021 04:08 AM
From: Aws Al-Dabbagh
Subject: Excessive ARPs between APs

Hello all,

we have 15 Instant APs (IAP-105) that we are using for WI-FI services in a small office. recently I have observed excessive ARP traffic on the Management network. so excessive that  up to 93% of traffic on the IAP management VLAN is ARP traffic. (I have a separate VLAN for AP Management traffic. this VLAN is untagged on the switch port, and other data from SSIDs is tagged) the ARP traffic takes several forms:

* ARP floods of an AP requesting the MAC of another.
* Gratuitous ARP traffic from the Controller AP to the Virtual Controller IP (it should hold that IP itself)
* ARP floods from the Gateway of the Subnet (a Cisco Firewall) looking for an IP that doesn't exist on the subnet. (I have a case with Cisco about that traffic).

ARP traffic is fine and I have it contained on the AP management VLAN, except it's overwhelming the management plane on a switch that's the DHCP server for the VLAN.

any ideas what's the issue and how to stop it?

I'd appreciate any input.

yes, I have packet captures.

I see tens of packets. but they are intermittent. every few minutes I'll see an ARP flood (more than 50 packets) of one AP IP address checking for another. then a different IP for a different AP doing the same, then every 15-30 minutes the Gateway is checking for an IP that doesn't exist. I'm checking the firewall logs and there's no traffic from anywhere else on the network to that IP.

I do see the same ARP multiple times a second. Aruba support replied that the IAP-105 is end of support as of August last year.

I can provide more info if needed to troubleshoot the issue. and I'd appreciate any support.

Thanks.

------------------------------
Aws Al-Dabbagh
------------------------------

Original Message:
Sent: Sep 16, 2021 08:09 AM
From: Herman Robers
Subject: Excessive ARPs between APs

Do you have packet captures? How many packets per second do you see on each of the categories?

That a gateway is doing ARPs for IPs that are not in use is probably a scan (or clients) trying to reach those IPs. If the firewall allows that traffic, it will do an ARP request for the IP. So for that one, check on the other interfaces/logs what client is trying to reach those IPs, or bring up a client with that IP and see the source IP as soon as the ARP is successful, or create a static ARP on your gateway.

The first step is to determine if the traffic is normal, if you see the same ARP multiple times a second, that is not normal, and you may reach out to your Aruba partner or Aruba support to get it analyzed. If you see repeating the same AP requesting ARP for the same other AP, it may be good to check if the AP is responding, and there may be an ACL in your network or so preventing the response to reach the original AP, or you may have a damaged cable that even may work in only one direction. The generic information above is far from sufficient to tell anything useful.

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.

Original Message:
Sent: Sep 15, 2021 04:08 AM
From: Aws Al-Dabbagh
Subject: Excessive ARPs between APs

Hello all,

we have 15 Instant APs (IAP-105) that we are using for WI-FI services in a small office. recently I have observed excessive ARP traffic on the Management network. so excessive that  up to 93% of traffic on the IAP management VLAN is ARP traffic. (I have a separate VLAN for AP Management traffic. this VLAN is untagged on the switch port, and other data from SSIDs is tagged) the ARP traffic takes several forms:

* ARP floods of an AP requesting the MAC of another.
* Gratuitous ARP traffic from the Controller AP to the Virtual Controller IP (it should hold that IP itself)
* ARP floods from the Gateway of the Subnet (a Cisco Firewall) looking for an IP that doesn't exist on the subnet. (I have a case with Cisco about that traffic).

ARP traffic is fine and I have it contained on the AP management VLAN, except it's overwhelming the management plane on a switch that's the DHCP server for the VLAN.

any ideas what's the issue and how to stop it?

I'd appreciate any input.

Could it be that you have an L2 loop in your network? That ARP flood is really strange. IF it is always the same AP, then check the cable of that AP, replace the cable, move to another port and see if that fixes it.

ARP request from a gateway/firewall to non-existing IPs are very likely initiated by the gw/firewall, DHCP server that is checking if an IP is in use, or NMS/Security scanners, or other systems on your network scanning for those IPs. The firewall should get the MAC address before it can send IP traffic, and ARP is the way of doing that. Unfortunately, until there is a successful ARP, you will not see the source IP. If the IP (or one of the IPs) is always the same, you can bring up a client with that IP and capture the IP traffic for the source IP.

If you share the packet capture, others may have a look at it.

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.
------------------------------

Original Message:
Sent: Sep 19, 2021 05:42 AM
From: Aws Al-Dabbagh
Subject: Excessive ARPs between APs

yes, I have packet captures.

I see tens of packets. but they are intermittent. every few minutes I'll see an ARP flood (more than 50 packets) of one AP IP address checking for another. then a different IP for a different AP doing the same, then every 15-30 minutes the Gateway is checking for an IP that doesn't exist. I'm checking the firewall logs and there's no traffic from anywhere else on the network to that IP.

I do see the same ARP multiple times a second. Aruba support replied that the IAP-105 is end of support as of August last year.

I can provide more info if needed to troubleshoot the issue. and I'd appreciate any support.

Thanks.

------------------------------
Aws Al-Dabbagh

Original Message:
Sent: Sep 16, 2021 08:09 AM
From: Herman Robers
Subject: Excessive ARPs between APs

Do you have packet captures? How many packets per second do you see on each of the categories?

That a gateway is doing ARPs for IPs that are not in use is probably a scan (or clients) trying to reach those IPs. If the firewall allows that traffic, it will do an ARP request for the IP. So for that one, check on the other interfaces/logs what client is trying to reach those IPs, or bring up a client with that IP and see the source IP as soon as the ARP is successful, or create a static ARP on your gateway.

The first step is to determine if the traffic is normal, if you see the same ARP multiple times a second, that is not normal, and you may reach out to your Aruba partner or Aruba support to get it analyzed. If you see repeating the same AP requesting ARP for the same other AP, it may be good to check if the AP is responding, and there may be an ACL in your network or so preventing the response to reach the original AP, or you may have a damaged cable that even may work in only one direction. The generic information above is far from sufficient to tell anything useful.

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.

Original Message:
Sent: Sep 15, 2021 04:08 AM
From: Aws Al-Dabbagh
Subject: Excessive ARPs between APs

Hello all,

we have 15 Instant APs (IAP-105) that we are using for WI-FI services in a small office. recently I have observed excessive ARP traffic on the Management network. so excessive that  up to 93% of traffic on the IAP management VLAN is ARP traffic. (I have a separate VLAN for AP Management traffic. this VLAN is untagged on the switch port, and other data from SSIDs is tagged) the ARP traffic takes several forms:

* ARP floods of an AP requesting the MAC of another.
* Gratuitous ARP traffic from the Controller AP to the Virtual Controller IP (it should hold that IP itself)
* ARP floods from the Gateway of the Subnet (a Cisco Firewall) looking for an IP that doesn't exist on the subnet. (I have a case with Cisco about that traffic).

ARP traffic is fine and I have it contained on the AP management VLAN, except it's overwhelming the management plane on a switch that's the DHCP server for the VLAN.

any ideas what's the issue and how to stop it?

I'd appreciate any input.

The issue has been resolved. Turned out 1 AP was causing the issue. We found it by shutting down each AP for 15 minutes and monitoring ARP traffic during the same time.

I still don't understand how 1 AP (even if broken) can cause other APs, even the gateway, to send random traffic. Further troubleshooting found that while that AP was on, other APs were flooding ARP requests to each other. And the Cisco Firewall was sending ARP requests to random IPs.

 

Anyway, Thanks for your support, Herman.

------------------------------
Aws Al-Dabbagh
------------------------------

Original Message:
Sent: Sep 20, 2021 07:40 AM
From: Herman Robers
Subject: Excessive ARPs between APs

Could it be that you have an L2 loop in your network? That ARP flood is really strange. IF it is always the same AP, then check the cable of that AP, replace the cable, move to another port and see if that fixes it.

ARP request from a gateway/firewall to non-existing IPs are very likely initiated by the gw/firewall, DHCP server that is checking if an IP is in use, or NMS/Security scanners, or other systems on your network scanning for those IPs. The firewall should get the MAC address before it can send IP traffic, and ARP is the way of doing that. Unfortunately, until there is a successful ARP, you will not see the source IP. If the IP (or one of the IPs) is always the same, you can bring up a client with that IP and capture the IP traffic for the source IP.

If you share the packet capture, others may have a look at it.

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.

Original Message:
Sent: Sep 19, 2021 05:42 AM
From: Aws Al-Dabbagh
Subject: Excessive ARPs between APs

yes, I have packet captures.

I see tens of packets. but they are intermittent. every few minutes I'll see an ARP flood (more than 50 packets) of one AP IP address checking for another. then a different IP for a different AP doing the same, then every 15-30 minutes the Gateway is checking for an IP that doesn't exist. I'm checking the firewall logs and there's no traffic from anywhere else on the network to that IP.

I do see the same ARP multiple times a second. Aruba support replied that the IAP-105 is end of support as of August last year.

I can provide more info if needed to troubleshoot the issue. and I'd appreciate any support.

Thanks.

------------------------------
Aws Al-Dabbagh

Original Message:
Sent: Sep 16, 2021 08:09 AM
From: Herman Robers
Subject: Excessive ARPs between APs

Do you have packet captures? How many packets per second do you see on each of the categories?

That a gateway is doing ARPs for IPs that are not in use is probably a scan (or clients) trying to reach those IPs. If the firewall allows that traffic, it will do an ARP request for the IP. So for that one, check on the other interfaces/logs what client is trying to reach those IPs, or bring up a client with that IP and see the source IP as soon as the ARP is successful, or create a static ARP on your gateway.

The first step is to determine if the traffic is normal, if you see the same ARP multiple times a second, that is not normal, and you may reach out to your Aruba partner or Aruba support to get it analyzed. If you see repeating the same AP requesting ARP for the same other AP, it may be good to check if the AP is responding, and there may be an ACL in your network or so preventing the response to reach the original AP, or you may have a damaged cable that even may work in only one direction. The generic information above is far from sufficient to tell anything useful.

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.

Original Message:
Sent: Sep 15, 2021 04:08 AM
From: Aws Al-Dabbagh
Subject: Excessive ARPs between APs

Hello all,

we have 15 Instant APs (IAP-105) that we are using for WI-FI services in a small office. recently I have observed excessive ARP traffic on the Management network. so excessive that  up to 93% of traffic on the IAP management VLAN is ARP traffic. (I have a separate VLAN for AP Management traffic. this VLAN is untagged on the switch port, and other data from SSIDs is tagged) the ARP traffic takes several forms:

* ARP floods of an AP requesting the MAC of another.
* Gratuitous ARP traffic from the Controller AP to the Virtual Controller IP (it should hold that IP itself)
* ARP floods from the Gateway of the Subnet (a Cisco Firewall) looking for an IP that doesn't exist on the subnet. (I have a case with Cisco about that traffic).

ARP traffic is fine and I have it contained on the AP management VLAN, except it's overwhelming the management plane on a switch that's the DHCP server for the VLAN.

any ideas what's the issue and how to stop it?

I'd appreciate any input.

As mentioned it could be a cable issue. Don't think it is an AP issue, unless the ethernet port got damaged which is unlikely. You can try with another cable/short patch cable direct to the switch if the issue is resolved.

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.
------------------------------

Original Message:
Sent: Sep 22, 2021 07:38 AM
From: Aws Al-Dabbagh
Subject: Excessive ARPs between APs

The issue has been resolved. Turned out 1 AP was causing the issue. We found it by shutting down each AP for 15 minutes and monitoring ARP traffic during the same time.

I still don't understand how 1 AP (even if broken) can cause other APs, even the gateway, to send random traffic. Further troubleshooting found that while that AP was on, other APs were flooding ARP requests to each other. And the Cisco Firewall was sending ARP requests to random IPs.

 

Anyway, Thanks for your support, Herman.

------------------------------
Aws Al-Dabbagh

Original Message:
Sent: Sep 20, 2021 07:40 AM
From: Herman Robers
Subject: Excessive ARPs between APs

Could it be that you have an L2 loop in your network? That ARP flood is really strange. IF it is always the same AP, then check the cable of that AP, replace the cable, move to another port and see if that fixes it.

ARP request from a gateway/firewall to non-existing IPs are very likely initiated by the gw/firewall, DHCP server that is checking if an IP is in use, or NMS/Security scanners, or other systems on your network scanning for those IPs. The firewall should get the MAC address before it can send IP traffic, and ARP is the way of doing that. Unfortunately, until there is a successful ARP, you will not see the source IP. If the IP (or one of the IPs) is always the same, you can bring up a client with that IP and capture the IP traffic for the source IP.

If you share the packet capture, others may have a look at it.

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.

Original Message:
Sent: Sep 19, 2021 05:42 AM
From: Aws Al-Dabbagh
Subject: Excessive ARPs between APs

yes, I have packet captures.

I see tens of packets. but they are intermittent. every few minutes I'll see an ARP flood (more than 50 packets) of one AP IP address checking for another. then a different IP for a different AP doing the same, then every 15-30 minutes the Gateway is checking for an IP that doesn't exist. I'm checking the firewall logs and there's no traffic from anywhere else on the network to that IP.

I do see the same ARP multiple times a second. Aruba support replied that the IAP-105 is end of support as of August last year.

I can provide more info if needed to troubleshoot the issue. and I'd appreciate any support.

Thanks.

------------------------------
Aws Al-Dabbagh

Original Message:
Sent: Sep 16, 2021 08:09 AM
From: Herman Robers
Subject: Excessive ARPs between APs

Do you have packet captures? How many packets per second do you see on each of the categories?

That a gateway is doing ARPs for IPs that are not in use is probably a scan (or clients) trying to reach those IPs. If the firewall allows that traffic, it will do an ARP request for the IP. So for that one, check on the other interfaces/logs what client is trying to reach those IPs, or bring up a client with that IP and see the source IP as soon as the ARP is successful, or create a static ARP on your gateway.

The first step is to determine if the traffic is normal, if you see the same ARP multiple times a second, that is not normal, and you may reach out to your Aruba partner or Aruba support to get it analyzed. If you see repeating the same AP requesting ARP for the same other AP, it may be good to check if the AP is responding, and there may be an ACL in your network or so preventing the response to reach the original AP, or you may have a damaged cable that even may work in only one direction. The generic information above is far from sufficient to tell anything useful.

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.

Original Message:
Sent: Sep 15, 2021 04:08 AM
From: Aws Al-Dabbagh
Subject: Excessive ARPs between APs

Hello all,

we have 15 Instant APs (IAP-105) that we are using for WI-FI services in a small office. recently I have observed excessive ARP traffic on the Management network. so excessive that  up to 93% of traffic on the IAP management VLAN is ARP traffic. (I have a separate VLAN for AP Management traffic. this VLAN is untagged on the switch port, and other data from SSIDs is tagged) the ARP traffic takes several forms:

* ARP floods of an AP requesting the MAC of another.
* Gratuitous ARP traffic from the Controller AP to the Virtual Controller IP (it should hold that IP itself)
* ARP floods from the Gateway of the Subnet (a Cisco Firewall) looking for an IP that doesn't exist on the subnet. (I have a case with Cisco about that traffic).

ARP traffic is fine and I have it contained on the AP management VLAN, except it's overwhelming the management plane on a switch that's the DHCP server for the VLAN.

any ideas what's the issue and how to stop it?

I'd appreciate any input.

I'm not sure it could be the cable. we installed another AP using the same cable and switch port. it has been working fine for the past 24 hours.

if the issue recurs, I'll consider your solution.

thanks again for your insights.

------------------------------
Aws Al-Dabbagh
------------------------------

Original Message:
Sent: Sep 22, 2021 08:08 AM
From: Herman Robers
Subject: Excessive ARPs between APs

As mentioned it could be a cable issue. Don't think it is an AP issue, unless the ethernet port got damaged which is unlikely. You can try with another cable/short patch cable direct to the switch if the issue is resolved.

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.

Original Message:
Sent: Sep 22, 2021 07:38 AM
From: Aws Al-Dabbagh
Subject: Excessive ARPs between APs

The issue has been resolved. Turned out 1 AP was causing the issue. We found it by shutting down each AP for 15 minutes and monitoring ARP traffic during the same time.

I still don't understand how 1 AP (even if broken) can cause other APs, even the gateway, to send random traffic. Further troubleshooting found that while that AP was on, other APs were flooding ARP requests to each other. And the Cisco Firewall was sending ARP requests to random IPs.

 

Anyway, Thanks for your support, Herman.

------------------------------
Aws Al-Dabbagh

Original Message:
Sent: Sep 20, 2021 07:40 AM
From: Herman Robers
Subject: Excessive ARPs between APs

Could it be that you have an L2 loop in your network? That ARP flood is really strange. IF it is always the same AP, then check the cable of that AP, replace the cable, move to another port and see if that fixes it.

ARP request from a gateway/firewall to non-existing IPs are very likely initiated by the gw/firewall, DHCP server that is checking if an IP is in use, or NMS/Security scanners, or other systems on your network scanning for those IPs. The firewall should get the MAC address before it can send IP traffic, and ARP is the way of doing that. Unfortunately, until there is a successful ARP, you will not see the source IP. If the IP (or one of the IPs) is always the same, you can bring up a client with that IP and capture the IP traffic for the source IP.

If you share the packet capture, others may have a look at it.

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.

Original Message:
Sent: Sep 19, 2021 05:42 AM
From: Aws Al-Dabbagh
Subject: Excessive ARPs between APs

yes, I have packet captures.

I see tens of packets. but they are intermittent. every few minutes I'll see an ARP flood (more than 50 packets) of one AP IP address checking for another. then a different IP for a different AP doing the same, then every 15-30 minutes the Gateway is checking for an IP that doesn't exist. I'm checking the firewall logs and there's no traffic from anywhere else on the network to that IP.

I do see the same ARP multiple times a second. Aruba support replied that the IAP-105 is end of support as of August last year.

I can provide more info if needed to troubleshoot the issue. and I'd appreciate any support.

Thanks.

------------------------------
Aws Al-Dabbagh

Original Message:
Sent: Sep 16, 2021 08:09 AM
From: Herman Robers
Subject: Excessive ARPs between APs

Do you have packet captures? How many packets per second do you see on each of the categories?

That a gateway is doing ARPs for IPs that are not in use is probably a scan (or clients) trying to reach those IPs. If the firewall allows that traffic, it will do an ARP request for the IP. So for that one, check on the other interfaces/logs what client is trying to reach those IPs, or bring up a client with that IP and see the source IP as soon as the ARP is successful, or create a static ARP on your gateway.

The first step is to determine if the traffic is normal, if you see the same ARP multiple times a second, that is not normal, and you may reach out to your Aruba partner or Aruba support to get it analyzed. If you see repeating the same AP requesting ARP for the same other AP, it may be good to check if the AP is responding, and there may be an ACL in your network or so preventing the response to reach the original AP, or you may have a damaged cable that even may work in only one direction. The generic information above is far from sufficient to tell anything useful.

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.

Original Message:
Sent: Sep 15, 2021 04:08 AM
From: Aws Al-Dabbagh
Subject: Excessive ARPs between APs

Hello all,

we have 15 Instant APs (IAP-105) that we are using for WI-FI services in a small office. recently I have observed excessive ARP traffic on the Management network. so excessive that  up to 93% of traffic on the IAP management VLAN is ARP traffic. (I have a separate VLAN for AP Management traffic. this VLAN is untagged on the switch port, and other data from SSIDs is tagged) the ARP traffic takes several forms:

* ARP floods of an AP requesting the MAC of another.
* Gratuitous ARP traffic from the Controller AP to the Virtual Controller IP (it should hold that IP itself)
* ARP floods from the Gateway of the Subnet (a Cisco Firewall) looking for an IP that doesn't exist on the subnet. (I have a case with Cisco about that traffic).

ARP traffic is fine and I have it contained on the AP management VLAN, except it's overwhelming the management plane on a switch that's the DHCP server for the VLAN.

any ideas what's the issue and how to stop it?

I'd appreciate any input.

Hi, just reading your post. I was wondering how you detected the excessive ARP issue and if there was any noticeable impact to users?
Original Message:
Sent: Sep 22, 2021 08:15 AM
From: Aws Al-Dabbagh
Subject: Excessive ARPs between APs

I'm not sure it could be the cable. we installed another AP using the same cable and switch port. it has been working fine for the past 24 hours.

if the issue recurs, I'll consider your solution.

thanks again for your insights.

------------------------------
Aws Al-Dabbagh

Original Message:
Sent: Sep 22, 2021 08:08 AM
From: Herman Robers
Subject: Excessive ARPs between APs

As mentioned it could be a cable issue. Don't think it is an AP issue, unless the ethernet port got damaged which is unlikely. You can try with another cable/short patch cable direct to the switch if the issue is resolved.

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.

Original Message:
Sent: Sep 22, 2021 07:38 AM
From: Aws Al-Dabbagh
Subject: Excessive ARPs between APs

The issue has been resolved. Turned out 1 AP was causing the issue. We found it by shutting down each AP for 15 minutes and monitoring ARP traffic during the same time.

I still don't understand how 1 AP (even if broken) can cause other APs, even the gateway, to send random traffic. Further troubleshooting found that while that AP was on, other APs were flooding ARP requests to each other. And the Cisco Firewall was sending ARP requests to random IPs.

 

Anyway, Thanks for your support, Herman.

------------------------------
Aws Al-Dabbagh

Original Message:
Sent: Sep 20, 2021 07:40 AM
From: Herman Robers
Subject: Excessive ARPs between APs

Could it be that you have an L2 loop in your network? That ARP flood is really strange. IF it is always the same AP, then check the cable of that AP, replace the cable, move to another port and see if that fixes it.

ARP request from a gateway/firewall to non-existing IPs are very likely initiated by the gw/firewall, DHCP server that is checking if an IP is in use, or NMS/Security scanners, or other systems on your network scanning for those IPs. The firewall should get the MAC address before it can send IP traffic, and ARP is the way of doing that. Unfortunately, until there is a successful ARP, you will not see the source IP. If the IP (or one of the IPs) is always the same, you can bring up a client with that IP and capture the IP traffic for the source IP.

If you share the packet capture, others may have a look at it.

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.

Original Message:
Sent: Sep 19, 2021 05:42 AM
From: Aws Al-Dabbagh
Subject: Excessive ARPs between APs

yes, I have packet captures.

I see tens of packets. but they are intermittent. every few minutes I'll see an ARP flood (more than 50 packets) of one AP IP address checking for another. then a different IP for a different AP doing the same, then every 15-30 minutes the Gateway is checking for an IP that doesn't exist. I'm checking the firewall logs and there's no traffic from anywhere else on the network to that IP.

I do see the same ARP multiple times a second. Aruba support replied that the IAP-105 is end of support as of August last year.

I can provide more info if needed to troubleshoot the issue. and I'd appreciate any support.

Thanks.

------------------------------
Aws Al-Dabbagh

Original Message:
Sent: Sep 16, 2021 08:09 AM
From: Herman Robers
Subject: Excessive ARPs between APs

Do you have packet captures? How many packets per second do you see on each of the categories?

That a gateway is doing ARPs for IPs that are not in use is probably a scan (or clients) trying to reach those IPs. If the firewall allows that traffic, it will do an ARP request for the IP. So for that one, check on the other interfaces/logs what client is trying to reach those IPs, or bring up a client with that IP and see the source IP as soon as the ARP is successful, or create a static ARP on your gateway.

The first step is to determine if the traffic is normal, if you see the same ARP multiple times a second, that is not normal, and you may reach out to your Aruba partner or Aruba support to get it analyzed. If you see repeating the same AP requesting ARP for the same other AP, it may be good to check if the AP is responding, and there may be an ACL in your network or so preventing the response to reach the original AP, or you may have a damaged cable that even may work in only one direction. The generic information above is far from sufficient to tell anything useful.

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.

Original Message:
Sent: Sep 15, 2021 04:08 AM
From: Aws Al-Dabbagh
Subject: Excessive ARPs between APs

Hello all,

we have 15 Instant APs (IAP-105) that we are using for WI-FI services in a small office. recently I have observed excessive ARP traffic on the Management network. so excessive that  up to 93% of traffic on the IAP management VLAN is ARP traffic. (I have a separate VLAN for AP Management traffic. this VLAN is untagged on the switch port, and other data from SSIDs is tagged) the ARP traffic takes several forms:

* ARP floods of an AP requesting the MAC of another.
* Gratuitous ARP traffic from the Controller AP to the Virtual Controller IP (it should hold that IP itself)
* ARP floods from the Gateway of the Subnet (a Cisco Firewall) looking for an IP that doesn't exist on the subnet. (I have a case with Cisco about that traffic).

ARP traffic is fine and I have it contained on the AP management VLAN, except it's overwhelming the management plane on a switch that's the DHCP server for the VLAN.

any ideas what's the issue and how to stop it?

I'd appreciate any input.