Wired Intelligent Edge

I have an Aruba 6000 that is connected upstream to a Cisco Nexus 7K.  I have been having intermittent performance issues for the past few weeks and started delving into configuration as well as possible path related issues.  I noticed on my controller that I'm taking a fairly large amount of inbound/crc/giant errors.  After swapping cables, moving connections from different ports, hard coding duplex, etc I finally opened a TAC case.  TAC identified that my MTU on my gigabitethernet interfaces on the controller is set to 1500 while it's set to 9216 on the upstream Nexus 7K.  I cannot change the MTU for an individual interface on the 7K as it's a global change that affects all interfaces.  So now I'm left trying to determine if:

 

1.  The issues with inbound errors are in fact a result of different MTU between switch gigabitethernet interfaces and controller

2.  If the inbound errors are having an actual impact

 

I've tried clearing counters to get an idea of whether or not these inbound errors are the result of some PMTU discovery firing off at a predetermined interval, but it doesn't appear to be the case as they increment nearly instantanously and consistently.

 

Thoughts?

 

I have it the other way around, because we have yet to schedule an outage window to turn on jumbo frame support on the core network and across the distribution network.  In my setup, the giant errors, as expected, occur on the core network side and are obviously just PMTU.  These do NOT include CRC errors, and it would surprise me if the controller logs CRC errors just for giant frames.

 

In any case, assuming the 6000 supports it,  why not turn on jumbo support on the controller and see if the performance problems go away?

 

I don't have N7Ks at my disposal but the following suggests that for L2 interfaces you should be able to set a per-interface MTU either to the systemwide jumbo size, or to 1500:

 

http://www.ccierants.com/2013/10/ccie-dc-definitive-jumbo-frames.html

 

Well, unfortunately the 6000 doesn't support jumbo frames, so there is no option to enable it on the controller.  My understanding is that this is supported in the newer 7200/7400 series controllers.  Interestingly I noticed that even for the 10Gb interfaces it's still an MTU of 1500.

 

As for the Nexus 7K I actually read the same blog post, but have different findings.  Perhaps it is a version issue, but from everything I've read (and see) the MTU is set by the network-qos policy map, which is set to 9216.  I can only change it system wide...which would impact our VM/SANS environment so that's a no-go.  But to your point I don't see why a giant frame would be logged as a CRC, but we shall see.  I think what I'm going to do is put another switch in between my 7K and the controller and set the MTU on that switch to 1500 and see if the CRC errors continue.  If they do then I know it's not an MTU issue...if they don't then I know that the MTU difference is causing an issue...then I just need to figure out if there is an impact.

 

 

To test the theory of MTU mismatch being the problem, ping the nexus from the controller.  Set the packet size higher than 1500 bytes and set the do not fragment flag.  FYI, you'll need AOS 6.3 to use the df-flag.

 

I expect that since the MTU on the switch is 1500 bytes the packet would be dropped before it hits the output queue, but will be interesting to see if the counters on the Nexus interface increment.

 

---- Edit -----

After thinking about it, I don't believe the df-flag is going to matter in this case since it's probably a L2 segment.  Should only matter on a L3 segment.

I wonder if there is something buggy going on here with ArubaOS.

 

one of my clients just showed me they have been having the same issues on HP 2520 switches on pretty much every port the AP's are connected to.

 

Running 6.3.1.7 and a mix of AP-225's, AP-125's, AP-135s.

 

Yet to really look into it but thought i'd throw that out there in case there is something else happening.

 

Default MTU on switches is 1522 and Aruba controller is set to 1500 so no configuration mismatch. Could the AP's be sending spurious bad frames?

 

Scott

Scott,

Yours is most likely a result of MTU path discovery that the APs perform:

http://community.arubanetworks.com/t5/ArubaOS-and-Controllers/Oversize-jumbo-frame-errors/m-p/5421/highlight/true#M1860

This will not result in performance issues for your clients.

It's definitely a result of large packets.  When I ran 1000 pings with 2000 byte packets from the upstream switch and set the DF-BIT i see an increase in CRC errors in 1000 (+ a handful more). 

 

What I'm confused about now is why I'm getting all of these large packets?  We have our APs set to 1500 and our RAPs set to even smaller.  So what is doing PMTU so often?  In one two week period I saw over a million of these errors.  I know that someone said that they aren't impacting...but I'm not entirely convinced just yet...though i can't see how they would be as long as the packets aren't coming from our APs...though then that raises the question of where/what they are coming from.

 

 

Thanks, i wasn't aware of this behaviour.

 

Scott

 

FWIW,

 

Just noticed in the ArubaOS MAS release notes that a known issue #85529 that causes MTU probes to be logged under InputErrorBytes.

 

Perhaps that issue applies to controller ArubaOS as well.

 

 

Issue number   85529 is specifically about the S3500 and an AP135 connected to it.

 

Hi,

thank for the suggestion.

because i'm having this issue with HP switch, i've read your post.

If the HP "prefer" packet with maximum size 1522 bytes (tagged), and AP's packet are 1578 bytes, can i change some parameters on Aruba and configure it to use MTU of 1500 bytes?

I would like to not use Jumbo packets on the switch..

Regards,

 

N

the link mentioned on the first page shows you where you can update it on the controller.

 

http://community.arubanetworks.com/t5/ArubaOS-and-Controllers/Oversize-jumbo-frame-errors/m-p/5421/highlight/true#M1860

 

it mentions, 3.x, but it is still there with 6.x