Wired Intelligent Edge

Seem to be getting connection spikes internally, any help is much appreciated!

Core is Aruba 5406rZL2 & Edge Switches are 2930F's.

Happens across site, latency can be seen when pinging the core switch from an Edge switch just around the corner from the core. Response time is 1-9ms but every 2-3 pings it jumps to 44-90ms+ - this also occurs when connected directly to the core switch (end users report the issue as sudden lag when and unresponsiveness accessing webpages.)

Initially thought this was a loop but have slowly been whittling that down by going across site and disconnecting switches, did find a potential "rogue AP" connected but still occurring after that device was disconnected (I still have 4 switches to disconnect to rule this out unless routing loop)

The core is connected to the router on port 1/E1 and I have seen collision warnings in the event viewer, when checking port statistics I could see RX errors increasing when re-running, only other ports on the core I see TX errors on are for the physical backup server in the same rack.

Loop protect is turned on all Edge switches but not the core, not a networking specialist but appreciate the help in getting to the bottom of this! (can include config is requested)

Will

Pinging a switch is not a good gauge for forwarding latency. ICMP echo requests are handled by the CPU/control plane at low priority. If there's anything better to do, ping replies need to wait. Also, many network devices throttle ICMP rates to limit CPU load.

You can use ping for such a diagnosis across the switch(es) only when you're sure that the destination replies without any delay - Windows and Linux hosts usually do. Most often, a tool like iperf3 with a carefully crafted test configuration can tell you much more and more reliably so than a simple ping.

Collision warnings are a common sign of a duplex mismatch - the classic scenario is when you force full duplex on one side and leave the other in default configuration. Forcing full duplex disable Auto Negotiation, causing the default side to fall back to half duplex.

Generally, you should never set the duplex manually unless there's a very good reason for that, you do it carefully, on both link ends, and document that setting well. Auto negotiation is mandatory for gigabit links and faster, so they may also degrade to Fast Ethernet which might not be intended.

End to End communication latency and speed is better evaluated if you test your infrastructure with specific client/server tools like iperf deployed directly on baremetal or virtualized hosts connected to your network infrastructure. It should be pretty easy to measure round trip time consistenly below 300/400 us (0.3/0.4 ms) - or better - for communications traversing just few hops over non saturated and non oversubscribed links.

Try to understand what's going on inside your Core Switch (especially if its routing functions are involved) from various point of view (ports status, STP, Broadcast, VLANs, routing, links flappings, etc.) and then extend the diagnostic up to the edge.

Original Message:
Sent: Feb 23, 2023 04:17 AM
From: Will59
Subject: Aruba 5406rZL2 Internal Latency

Seem to be getting connection spikes internally, any help is much appreciated!

Core is Aruba 5406rZL2 & Edge Switches are 2930F's.

Happens across site, latency can be seen when pinging the core switch from an Edge switch just around the corner from the core. Response time is 1-9ms but every 2-3 pings it jumps to 44-90ms+ - this also occurs when connected directly to the core switch (end users report the issue as sudden lag when and unresponsiveness accessing webpages.)

Initially thought this was a loop but have slowly been whittling that down by going across site and disconnecting switches, did find a potential "rogue AP" connected but still occurring after that device was disconnected (I still have 4 switches to disconnect to rule this out unless routing loop)

The core is connected to the router on port 1/E1 and I have seen collision warnings in the event viewer, when checking port statistics I could see RX errors increasing when re-running, only other ports on the core I see TX errors on are for the physical backup server in the same rack.

Loop protect is turned on all Edge switches but not the core, not a networking specialist but appreciate the help in getting to the bottom of this! (can include config is requested)

Will

The core is connected to the router on port 1/E1 and I have seen collision warnings in the event viewer, when checking port statistics I could see RX errors increasing when re-running, only other ports on the core I see TX errors on are for the physical backup server in the same rack.

You should check more in depth the reasons for these RX/TX errors and if they are harmful. It can be overload of the switch port, like if you have a 1Gbps port to your backup server, and multiple hosts backing up data at full speed that exceeds the 1Gbps. This even more is an issue if drops happen on the link between your switches.

If you see collission warnings, you may have a duplex mismatch on the port and setting it to full-duplex (assuming the connected device supports it) may help. You should not see collissions on fully switched networks, unless you connect really old devices.

I would in this case get a network specialist involved, for example your Aruba Partner, or Aruba Support if you have access to that. It's too risky to change settings if you don't know what the issue is and what the possible negative results are of your changes.

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.
------------------------------

Original Message:
Sent: Feb 23, 2023 04:17 AM
From: Will59
Subject: Aruba 5406rZL2 Internal Latency

Seem to be getting connection spikes internally, any help is much appreciated!

Core is Aruba 5406rZL2 & Edge Switches are 2930F's.

Happens across site, latency can be seen when pinging the core switch from an Edge switch just around the corner from the core. Response time is 1-9ms but every 2-3 pings it jumps to 44-90ms+ - this also occurs when connected directly to the core switch (end users report the issue as sudden lag when and unresponsiveness accessing webpages.)

Initially thought this was a loop but have slowly been whittling that down by going across site and disconnecting switches, did find a potential "rogue AP" connected but still occurring after that device was disconnected (I still have 4 switches to disconnect to rule this out unless routing loop)

The core is connected to the router on port 1/E1 and I have seen collision warnings in the event viewer, when checking port statistics I could see RX errors increasing when re-running, only other ports on the core I see TX errors on are for the physical backup server in the same rack.

Loop protect is turned on all Edge switches but not the core, not a networking specialist but appreciate the help in getting to the bottom of this! (can include config is requested)

Will