Wired Intelligent Edge

Hi,

We have tens of HP 2920 switches, which generally work fine. But there is a group of them connected to a cisco trunk port where there is a lot of ARP traffic – up to 500 ARP requests per second. HP 2920 switches connected to this high ARP broadcast segment sometimes fail to reply to arp requests.

I have set up a port mirror on one of the switches and I can ARP requests are received, but sometimes 20-30 of them seem to be ignored before finally, 2920 sends back a reply.

Has someone seen this behaviour? Is there some kind of ARP rate limiting? Any advise how further debug this?

That's too much ARP - the switch can only resolve 25 simultaneously.

Re-design the link - find out what all the ARP is for.

It's just a network with a lot of IP addresses. Not too many devices, but each device has many IP addresses. Also, there is a fair share of unused IP addresses, and due to non-stop network scanning (vulnerability scanners), these generate a lot of ARP requests too.

So, are you saying HP 2920 is inspecting at most 25 broadcast ARP requests per second and if the ARP request trying to resolve switch's own IP address is not among the lucky 25 ones it's not going to respond? Is this configurable?

Thanks!

Not 25 per second, just 25 simultaneously. Per second would depend on how long it takes to resolve each one.

The 2920 is not a router, nor is it a real Layer3 switch, so it isn't really designed to be doing lots of ARP resolution.

I definitely don't have 25 ARP requests going at the same time to this switch. This is an L2 switch, and the only IP traffic it handles is monitoring of the switch itself – i.e. SNMP and ICMP from single monitoring host IP. This is confirmed also by the following two facts: 1. there's only one entry in the switch's ARP table (show arp) and 2. only the monitoring host has this switch's IP address in its ARP cache.

It turned out the monitoring host's ARP cache size was too small and overflowed often, so it had to redo the ARP often. Once I increased the ARP cache sufficiently, the time between the ARP requests to the switch increased enough and now the problem is gone. At the same time, the total amount of ARP requests is still in the vicinity of 500/second.

Anyway, as far as the problem is gone I'm not going to investigate this further.

Thank you for your willingness to help!

You say it's a Layer2 switch, but still mentioned lots of ARP requests to the switch. Which got me thinking:
If the Layer2 switch's management IP address is not in a management VLAN, but is instead in a VLAN where there are heaps of other hosts doing whatever they do, then the Layer2 switch would need to read every ARP request to see if it needs to answer it.
(I have no idea whether this would impede the switch's ability to forward ARP broadcasts, I kind of doubt it really).

Is the IP address on the switch on the same VLAN as a bunch of other hosts?

If you've not implemented any management VLAN for your network devices, maybe try that.