Wired Intelligent Edge

Hi

We are migrating a cisco nexus campus core to a new aruba 8325 vsx core.

 

for the migration process we connected the ciscos witchLACP mc-lag.

 

On those interfaces we are seeing a high number of tx dropped packets..

 

Interface 1/1/43 is up
Admin state is up
Description:
Hardware: Ethernet, MAC Address: 54:80:28:fd:7e:ba
MTU 1500
Type SFP+SR
qos trust dscp
Speed 10000 Mb/s
Auto-Negotiation is off
Flow-control: off
Rx
6892493124 input packets 3158464638088 bytes
417 input error 58560220 dropped
406 CRC/FCS
Tx
14267183972 output packets 4354343600902 bytes
0 input error 601800434 dropped
0 collision

 

we can't figure out which traffic is beeing dropped.. looks like the queues get overrun and the switch starts to drop packets... but we have like 500Mbps - 1000Mbps traffic on a 10Gbps link..

 

Applications don't seem to be affected...

 

has anybody observed this behaviour aswell?

 

Hi nla2,

 

I see 406 CRC/FCS errors on the interface that could be a faulty SFP or cable. Please reset the counters of this interface and look if the CRC/FCS error growing again.

 

Also look if the vlan configuration is the same on both sites, beter dont use "permit all vlans" because when vlans missing on the switch the receiving switch the packets wil dropped on the ingress interface of the switch where the vlans dont exist.

 

Hopeful this help you

MTU? 1500 Bytes on a 10Gbps physical link speed could be not an usual value (Cisco Nexus side)...did you check?

 

Another thing to help diagnose: post show lacp interfaces multi-chassis command's sanitized output to discover how VSX (Primary and Secondary) matches remote peer (Cisco Nexus)...is Cisco Nexus configured with a "normal" LACP EtherChannel (required) against the VSX?

 

Clearly learning about the whole (sanitized) VSX configuration and having more details about LACP EtherChannel (Cisco Nexus side) would be of help.

 

What SKU (equal for both Aruba 8325) was used to setup the VSX (JL624A/JL625A)? ...I ask because seeing 10Gbps link on port 1/1/43...means that 4th Group (ports 37-48) were set to operate all at 10Gbps and not at 25Gbps (default). Correct?

Hi guys

 

@mkk 

- CRC errors are not growing. SFPs and cables look fine. I guess we got them as we plugged the fiber cables.

- About the VLANs: yes we have different VLAN mapping between 8325 and Nexus. Basically the Nexus is allowing all VLANs on the trunk also vlans that are not really used anymore. On the 8325 we are allowing only the VLANs that matter.

Given this I was expecting drops on the ingress (rx) of the interface but i still can't explain the tx drops. Or am I understanding this wrong?

 

@parnassus
- MTU: Yes I checked. MTU is 1500 on every interface and VLAN of the Nexus. There are no jumbo configurations.

- The output of show lacp interfaces multi-chassis

 

sw003-dist# show lacp interfaces multi-chassis

State abbreviations :
A - Active P - Passive F - Aggregable I - Individual
S - Short-timeout L - Long-timeout N - InSync O - OutofSync
C - Collecting D - Distributing
X - State m/c expired E - Default neighbor state

Actor details of all interfaces:
------------------------------------------------------------------------------
Intf Aggregate Port Port State System-ID System Aggr
name id Priority Priority Key
------------------------------------------------------------------------------
1/1/43 lag10(mc) 43 1 ALFNCD 54:80:28:ff:ff:01 65534 10

Partner details of all interfaces:
------------------------------------------------------------------------------
Intf Aggregate Partner Port State System-ID System Aggr
name Port-id Priority Priority Key
------------------------------------------------------------------------------
1/1/43 lag10(mc) 1041 32768 ALFNCD 64:a0:e7:41:69:c1 32768 224

Remote Actor details of all interfaces:
------------------------------------------------------------------------------
Intf Aggregate Port Port State System-ID System Aggr
name id Priority Priority Key
------------------------------------------------------------------------------
1/1/43 lag10(mc) 1043 1 ALFNCD 54:80:28:ff:ff:01 65534 10

Remote Partner details of all interfaces:
------------------------------------------------------------------------------
Intf Aggregate Partner Port State System-ID System Aggr
name Port-id Priority Priority Key
------------------------------------------------------------------------------
1/1/43 lag10(mc) 1051 32768 ASFNC 64:a0:e7:41:69:c1 32768 224

 

On the Nexus side I see this..

 

sw002.gg0.hgn# sh port-channel summary
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
225 Po225(SU) Eth LACP Eth4/17(P) Eth4/27(P)

sw002.gg0.hgn# sh lacp interface eth 4/17
Interface Ethernet4/17 is up
Channel group is 225 port channel is Po225
PDUs sent: 8971
PDUs rcvd: 8974
Markers sent: 0
Markers rcvd: 0
Marker response sent: 0
Marker response rcvd: 0
Unknown packets rcvd: 0
Illegal packets rcvd: 0
Lag Id: [ [(8000, 64-a0-e7-41-69-c1, e0, 8000, 411), (fffe, 54-80-28-ff-ff-1, a,
1, 2b)] ]
Operational as aggregated link since Wed Jul 10 07:20:09 2019

Local Port: Eth4/17 MAC Address= 64-a0-e7-41-69-c1
System Identifier=0x8000, Port Identifier=0x8000,0x411
Operational key=224
LACP_Activity=active
LACP_Timeout=Long Timeout (30s)
Synchronization=IN_SYNC
Collecting=true
Distributing=true
Partner information refresh timeout=Long Timeout (90s)
Actor Admin State=61
Actor Oper State=61
Neighbor: 0x2b
MAC Address= 54-80-28-ff-ff-1
System Identifier=0xfffe, Port Identifier=0x1,0x2b
Operational key=10
LACP_Activity=active
LACP_Timeout=Long Timeout (30s)
Synchronization=IN_SYNC
Collecting=true
Distributing=true
Partner Admin State=61
Partner Oper State=61
Aggregate or Individual(True=1)= 1

sw002.gg0.hgn# sh lacp interface eth 4/27
Interface Ethernet4/27 is up
Channel group is 225 port channel is Po225
PDUs sent: 8980
PDUs rcvd: 8982
Markers sent: 0
Markers rcvd: 0
Marker response sent: 0
Marker response rcvd: 0
Unknown packets rcvd: 0
Illegal packets rcvd: 0
Lag Id: [ [(8000, 64-a0-e7-41-69-c1, e0, 8000, 41b), (fffe, 54-80-28-ff-ff-1, a,
1, 413)] ]
Operational as aggregated link since Wed Jul 10 07:16:15 2019

Local Port: Eth4/27 MAC Address= 64-a0-e7-41-69-c1
System Identifier=0x8000, Port Identifier=0x8000,0x41b
Operational key=224
LACP_Activity=active
LACP_Timeout=Long Timeout (30s)
Synchronization=IN_SYNC
Collecting=true
Distributing=true
Partner information refresh timeout=Long Timeout (90s)
Actor Admin State=61
Actor Oper State=61
Neighbor: 0x413
MAC Address= 54-80-28-ff-ff-1
System Identifier=0xfffe, Port Identifier=0x1,0x413
Operational key=10
LACP_Activity=active
LACP_Timeout=Long Timeout (30s)
Synchronization=IN_SYNC
Collecting=true
Distributing=true
Partner Admin State=61
Partner Oper State=61
Aggregate or Individual(True=1)= 1

 

- 10G/25G: Correct. The port-group is set to operate at 10Gbps.

 

 

Edit:

We have mcast traffic streaming from the nexus and igmp snooping enabled on the vlans streaming mcast traffic. Could the drops be mcast traffic dropped by igmp to avoid mcast traffic going back to the nexus?

 

 

Cheers, looking all outputs you provided the only one that has a questionable part is the show lacp interfaces multi-chassis where the Remote Partner (so what the VSX Secondary node sees about its peer) shows this:

Remote Partner details of all interfaces:
------------------------------------------------------------------------------
Intf Aggregate Partner Port State System-ID System Aggr
name Port-id Priority Priority Key
------------------------------------------------------------------------------
1/1/43 lag10(mc) 1051 32768 ASFNC 64:a0:e7:41:69:c1 32768 224

I don't understand why lag10 (1/1/43) on VSX Secondary is reported as ASFNC instead of being ALFNCD (and more than not Distributing versus Distributing...what make me alarm is the Short versus Long, considering Long is reported also on Cisco side...). Maybe it's just "cosmetic"...but a check on why it's ASFNC and not ALFNCD wouldn't be a bad idea.

Hey

 

Yeah, I've noticed that aswell.. but.. sh lacp interface 1/1/43 vsx-peer gives me this..

 

sw003-dist# sh lacp interfaces 1/1/43

State abbreviations :
A - Active P - Passive F - Aggregable I - Individual
S - Short-timeout L - Long-timeout N - InSync O - OutofSync
C - Collecting D - Distributing
X - State m/c expired E - Default neighbor state

Aggregate-name : lag10(multi-chassis)
-------------------------------------------------
Actor Partner
-------------------------------------------------
Port-id | 43 | 1041
Port-priority | 1 | 32768
Key | 10 | 224
State | ALFNCD | ALFNCD
System-ID | 54:80:28:ff:ff:01 | 64:a0:e7:41:69:c1
System-priority | 65534 | 32768

 

 

 

sw003-dist# sh lacp interfaces 1/1/43 vsx-peer

State abbreviations :
A - Active P - Passive F - Aggregable I - Individual
S - Short-timeout L - Long-timeout N - InSync O - OutofSync
C - Collecting D - Distributing
X - State m/c expired E - Default neighbor state

Aggregate-name : lag10(multi-chassis)
-------------------------------------------------
Actor Partner
-------------------------------------------------
Port-id | 1043 | 1051
Port-priority | 1 | 32768
Key | 10 | 224
State | ALFNCD | ALFNCD
System-ID | 54:80:28:ff:ff:01 | 64:a0:e7:41:69:c1
System-priority | 65534 | 32768

 

Double check on the vsx member..

 

sw004-dist# sh lacp interfaces 1/1/43

State abbreviations :
A - Active P - Passive F - Aggregable I - Individual
S - Short-timeout L - Long-timeout N - InSync O - OutofSync
C - Collecting D - Distributing
X - State m/c expired E - Default neighbor state

Aggregate-name : lag10(multi-chassis)
-------------------------------------------------
Actor Partner
-------------------------------------------------
Port-id | 1043 | 1051
Port-priority | 1 | 32768
Key | 10 | 224
State | ALFNCD | ALFNCD
System-ID | 54:80:28:ff:ff:01 | 64:a0:e7:41:69:c1
System-priority | 65534 | 32768

 

everything is reported as ALFNCD..

 

 

Oh, I see. One more time the reported status of remote partner looks randomly unreliable (or, at least, not trustable as other commands' output report)...what ArubaOS-CX version is your VSX currently running on?

Hey

Yeah, maybe.

It's running on 10.02.0031.

 

I honestly don't think that the drops are related to a LACP mismatch. 

The nexus output shows that everything is ok.

 

We have mcast traffic streaming from the nexus and igmp snooping enabled on the vlans with mcast traffic. Could the drops be mcast traffic dropped by igmp to avoid mcast traffic going back over the lacp link to the nexus?

 

 

Do you have fix the issue ?

Hi

 

The drops were mostly unknown multicast traffic beeing dropped by igmp snooping.

Cheers

Hi,

 

There is a command in cisco " spanning-tree etherchannel guard misconfig". IT is very usefull when we connect Cisco with 3rd party LACP devices. We can implment this and check the syslog of Cisco what error message is displayed and take action accordingly.

 

Regards

Shivas

ACLP C flag instead of CD is a cosmetic bug that has been reported a while ago and that should have been fixed in 10.3 (at least).

...or maybe 10.02.0060 (if we're referring to CR 54328).