Wired Intelligent Edge

Errors Occur regularlyPort shows often at 100%Frames/s are off the wall!One of my helpdesk guys spotted one of our 2910AL's in ProCurveManager showing some nasty looking errors.

I have been investigating and resolved a few minor issues but the switch is still reporting massive utilisation to PCM.

This is totaly abnormal compared to the other switches in the network.

Trunk 10 is an LACP trunk to a Comware based switch, same as any other of our edge switches.

I have disabled one link on the trunk
I have disabled all edge devices on the switch. so the only active link is the one remaining on the LACP pair.
The config is very straight forward.

I can't see any abnormal levels of traffic at the other end of the link.

Seems to be predomenantly UDP and ARP.

Any ideas where to go next.

Rob

This is the port stats for 60 seconds after being cleared, it shows there is not huge traffic according to the port countters.

 MAC Address      : 002347-b63bff
  Link Status      : Up
  Totals (Since boot or last clear) :
   Bytes Rx        : 221,287            Bytes Tx        : 41,175
   Unicast Rx      : 90                 Unicast Tx      : 118
   Bcast/Mcast Rx  : 1835               Bcast/Mcast Tx  : 8
  Errors (Since boot or last clear) :
   FCS Rx          : 0                  Drops Tx        : 0
   Alignment Rx    : 0                  Collisions Tx   : 0
   Runts Rx        : 0                  Late Colln Tx   : 0
   Giants Rx       : 1                  Excessive Colln : 0
   Total Rx Errors : 1                  Deferred Tx     : 0

Hello,

From what I can see of the Wireshark packet analysis there is HSRP (Cisco's first hop routing redundancy protocol) message traffic in IPv6 form on UDP port 2029 going over your Trunk link into your switch.

Do you actually have Cisco devices running IPv6 that are offering up a floating IPv6 address to the network? To the best of my knowledge both the Comware and (now) Aruba-OS switches only offer flavours of VRRP which is the non-propriatory version and has its own unique signature. If you definitely don't have those devices in your network it could always be malware or something equally unpleasant.

It is possible someone has "commandeered" UDP port 2029 for their own application but that woud be unusual.

There have been some weird bugs on NICs where a pair of them supposedly sleeping would "discover each other to death" with IPv6 packets and cause a huge broadcast storm but I haven't yet seen any other example with HSRPv6.

Please keep digging and let us know what you find. If you ever get the opportunity to move on from PCM to the IMC management and monitoring software platform (free download with 60 day trial available) you'll find even more tools (sFlow, real time counters, SNMP, syslog aggregation) for doing this kind of thing. I am a bit of a fan. :-)

Many thanks

Ian

Ian

There are a couple of CISCO ASA's , but I don't think they are the cause.

This is the only switch in the network that is reporting this random traffic in PCM.

I think it may be a Firmware Bug ( no surprise realy )

http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=mmr_kc-0118894

Flashing the System and rebooting on Monday.

SW-HO-COM-2(config)# walkmib 1.3.6.1.2.1.2.2.1.2 | inc .63
ifDescr.63 = Trk10
ifDescr.12463 = HP Switch software loopback interface
SW-HO-COM-2(config)# getmib ifspeed.63
ifSpeed.63 = 2,000,000,000
SW-HO-COM-2(config)# getmib ifspeed.63
ifSpeed.63 = 2,000,000,000
SW-HO-COM-2(config)# repeat 1
ifSpeed.63 = 2,000,000,000
ifSpeed.63 = 0
ifSpeed.63 = 0
ifSpeed.63 = 2,000,000,000
ifSpeed.63 = 2,000,000,000
ifSpeed.63 = 0
ifSpeed.63 = 0
ifSpeed.63 = 0

Good spot!

I take it that you will be upgrading to firmware 16.x ASAP and Aruba-fying your switch??

I did wonder where the odd HSRP traffic was coming from I guess that was a Red Herring.

I've added some tags to the post so that hopefully it can be found easily by anyone else who experiences the same problem

Thanks

Ian