Wired Intelligent Edge

Hi.

One of our offices have been plagued by a hard to find issue for a while now, and i suspect the issue to be with the network switch at some level, even though we replaced the switch, and the same issue prevails.

The office is set up with pretty much the same equipment as all our other offices.

Equipment: 2620 PPOE+ (J9624A) (previously 15.17 firmware, now 16.02..14. same issue on both versions)

Issue facts:

Some ARP entries disappear from the arp table. This does not happen to the other offices we have the same switches in. Items that can disappear from the ARP table is a ESXi server and an Access Point.
I then have to ping those items from the switch before i can access them from a remote location
they are set up properly with gateway settings and such.

Equipment that are connected through HP unmanaged 8/16 port switches will loose network access (we tried replaing all table switches with new ones with no luck). If i replace one of those table switches with a default config 2620, and connect the equipment to it directly, the equipment stays connected with no issue.

When they loose network access, it seems they only loose access to external addresses, while internal devices are still reachable(ISP has checked their side, and found no issue. even replaced their modem just to be sure)

Even though some equipment looses access to stuff that is external to the 2620 main switch, other equipment will at the same time work. and its not the same equipment each time. it appears to be random.

I have tried activating spaning-tree and loop-protect, but no loops are detected.

The configuration is pretty basic. i have tried debug syslogging, but that does not really give me anything understandable. Here is the show config

; J9624A Configuration Editor; Created on release #RA.16.02.0014
; Ver #0e:01.10.82.34.41.1c.28.f3.84.9c.63.ff.37.2f:b3
hostname "Drammen-Hoved2620"
mirror-port 13
fault-finder broadcast-storm sensitivity high
fault-finder bad-driver sensitivity high
fault-finder bad-transceiver sensitivity high
fault-finder bad-cable sensitivity high
fault-finder too-long-cable sensitivity high
fault-finder over-bandwidth sensitivity high
fault-finder loss-of-link sensitivity high
fault-finder duplex-mismatch-hdx sensitivity high
fault-finder duplex-mismatch-fdx sensitivity high
fault-finder link-flap sensitivity high
logging 172.20.0.80
max-vlans 50
timesync sntp
sntp unicast
sntp server priority 1 172.20.0.37
time daylight-time-rule western-europe
time timezone 60
ip route 0.0.0.0 0.0.0.0 192.168.254.77
ip routing
snmp-server community "public" unrestricted
snmp-server enable traps mac-notify
snmp-server enable traps mac-count-notify
vlan 1
name "DEFAULT_VLAN"
no untagged 1-26
untagged 27-28
no ip address
exit
vlan 1040
name "Powertech"
untagged 1
ip address 192.168.254.78 255.255.255.252
exit
vlan 1041
name "Ansatt"
untagged 2-8,25
tagged 26
ip address 10.130.152.1 255.255.255.0
exit
vlan 1042
name "Gjest"
tagged 26
ip address 10.130.153.1 255.255.255.0
ip helper-address 10.130.152.10
exit
vlan 1043
name "Tynnklient"
untagged 9-24
tagged 26
ip address 10.130.154.1 255.255.254.0
ip helper-address 10.130.152.10
exit
no tftp server
loop-protect 2-24
loop-protect trap loop-detected
loop-protect disable-timer 600
no dhcp config-file-update
no dhcp image-file-update
device-profile name "default-ap-profile"
cos 0
exit

Any help on this would be much appreciated :)

Stop using the 2620 as a Layer-3 switch.

It isn't really good enough to be used as a network core. Do you have a better switch you can use as your core switch?

I'm afraid not. But we are using it with the same setup on larger locations without issue, so i doubt it's because the switch is not good enough.

I have 20 locations set up pretty much the same, with this switch, and on one of those locations it is making a lot of issues.

In any case, what would you recommend as a layer 3 core switch on a location with 100 users(15 employees, and the rest zero client student machines using 1M each)?
2920?

Another thing that might be relevant. I'm using the non-poe model on most of those 20 locations, and i'm starting to see similar behaviour on another location with the POE+ model. Could it be that the POE+ model is so different from the non-poe model that it doesnt work as well?

Depending on which Layer3 functionality you need, either 2920 or 3800.

The latest models are now 2930 and 3810 I think.

But, back to your problem - is there a difference in firmware versions?

I had something similar once (different vendor) and after some very close attention I discovered that the MAC address belonging to the Host was migrating from the Access port to the uplink port, which meant all traffic that was meant to be going to the Host was instead going to the uplink, resulting in total connectivity failure.
(In this case it was a problem with the MAC-address table, not the ARP table.)
The problem only occurred when the uplink was an aggregation of 2 or more physical ports.
The problem was fixed by a firmware upgrade. 

In your case, a problem with the ARP table entries would usually indicate
 - duplicate IP addresses
 - something sending proxy ARP messages
 - dodgy gratuitous ARPs of some description
 - ARP table filling up, older entries are flushed
but more likely some kind of firmware bug I suppose.

Thanks for the suggestions :)

I decided to purchase a 2920, and replaced it yesterday. So far everything seems to be working better, but i neet to wait a few days before being certain.

The configuration on the 2920 is pretty much the exact same, just adjusted for it being a Gb switch, and used PoE injectors for the 2 APs i had connected.

If it keeps being stable, i suspect a bug with the firmware for the PoE+ version of these switches, as i have not seen this issue with the non-PoE ones.

It is now a few days later, and the network has been stable after removing the PoE version of 2620 as a main switch. it is now running as a secondary switch, and seem to be doing fine.

So my recommendation is that no one use the 2620 PoE as a main switch, no matter how small the location.

No issues, even on larger locations with the 2620 non-poe, though, so that one is good.

I have a similar issue with 2 HP 2620 switches in seperate environments a rollback from RA.16.02.0014 to RA.15.17.0009 did solve the issue for me in both cases.