Wired Intelligent Edge

Hello community,

I have a weird mac-address-table problem in a rather complex network topology.

Problem Description

On SW3-5940-IRF client MAC addresses are learned (correctly) on the LAG interface to downstream access layer switch SW5-2930F-VSF. In this case the client which reports connectivity problems is connected to SW5.

Every few seconds the client mac address is seen on the LAG interface to SW1-5406zl and SW2-5406zl. This leads to packet loss of about 50% to 60% for the clients connected to SW5.

CLI Output which shows mac address change

[SW3/4-5940-IRF]display mac-address 0C54-1111-4444
MAC Address VLAN ID State Port/Nickname Aging
0C54-1111-4444 200 Learned BAGG4 Y
[L01DS000]display mac-address 0C54-15C6-FBBF
MAC Address VLAN ID State Port/Nickname Aging
0C54-1111-4444 200 Learned BAGG150 Y

Firmware Versions

Switches SW1 and SW2 5406zl run firmware version K.16.02.0014

SW3 and SW4 (IRF): 5940 run firmware 7.1.070, Release 2612P02

Access Layer VSF switch SW5 and SW6: 2930F run firmware WC.16.08.0009

Topology Description

SW1-5406zl is connected to SW2-5406zl with a switch-interconnect on a trunk interface.

SW1-5406zl and SW2-5406zl have a distributed LACP trunk configured on the uplinks ports to SW3-5940-IRF. Both uplinks are conncted to one IRF switch. SW4-5940-IRF is the second IRF cluster member, but has no uplinks to SW1-5406zl and SW2-5406zl.

Clients are connected to access layer switch SW5-2930F-VSF and SW6-2930F-VSF. SW5-2930F-VSF has an uplink to SW3-5940-IRF and SW6-2930F-VSF to SW4-5940-IRF. This uplink is configured as a regular LACP LAG.

Temporary Solution

When I disable one uplink between SW3-5940-IRF and SW2-5406zl clients do not report any connectivity problems. The client mac addresses on SW3-5940-IRF are never shown on the uplink to SW1-5406zl.

To me it looks like a problem specific to distributed trunking with uplinks to IRF clusters. I also have another 5940 Comware Switch connected to SW1-5406zl and SW2-5406zl where I don't have this problem.
Has anybody found a similar problem?

Thanks in advance for your help

Tomorrow I will post the uplink configuration and a network topology map.

Interesting, would be nice to see sanitized configurations used on the HP 5400 zl pair for Distributed Trunking.

Hello Parnassus.

The distributed trunk configuration is very basic without keepalives.

ISC Configuration

SW1-5406zl
trunk E1-E2 trk1 trunk
switch-interconnect trk1

SW2-5406zl
trunk E1-E2 trk1 trunk
switch-interconnect trk1

Distributed trunk configuration to SW3-5940-IRF
SW1-5406zl and SW2-5406zl

trunk E5 trk110 dt-lacp

VLANs on trk110 are the same on SW1-5406zl and SW2-5406zl. I ran the command "show vlan ports trk110 detail" on both switches an made a diff on the CLI output.

Bridge Aggregation configuration on SW3-5940-IRF to SW1-5406zl and SW2-5406zl

$VLAN-List matches the tagged vlans on trk110 on SW1-5406zl and SW2-5406zl

interface Bridge-Aggregation150
description LACP SW1-5406zl
port link-type trunk
undo port trunk permit vlan 1
port trunk permit vlan $VLAN-List
link-aggregation mode dynamic
#
interface Ten-GigabitEthernet1/0/45
port link-mode bridge
description UPLINK SW1-5406zl
port link-type trunk
undo port trunk permit vlan 1
port trunk permit vlan $VLAN List
undo stp enable
port link-aggregation group 150
#
interface Ten-GigabitEthernet1/0/46
port link-mode bridge
description UPLINK SW2-5406zl
port link-type trunk
undo port trunk permit vlan 1
port trunk permit vlan $VLAN-List
shutdown
undo stp enable
port link-aggregation group 150

Please let me know if you need more information. Do you have an idea how to troubleshoot this problem?

Attached is the network diagram.

If interface XGE1/0/46 is enabled, on SW3/SW4 client mac address is moving between interfaces Bridge-Aggregation150 to SW1/SW2 and the Uplink to SW5/SW6.

If interface XGE1/0/46 is down, no problems occur.

Attachment(s)

network-topology.pdf   50 KB 1 version

Hello Parnassus.

The distributed trunk configuration is very basic without keepalives.

ISC Configuration

SW1-5406zl
trunk E1-E2 trk1 trunk
switch-interconnect trk1

SW2-5406zl
trunk E1-E2 trk1 trunk
switch-interconnect trk1

Distributed trunk configuration to SW3-5940-IRF
SW1-5406zl and SW2-5406zl

trunk E5 trk110 dt-lacp

VLANs on trk110 are the same on SW1-5406zl and SW2-5406zl. I ran the command "show vlan ports trk110 detail" on both switches an made a diff on the CLI output.

Bridge Aggregation configuration on SW3-5940-IRF to SW1-5406zl and SW2-5406zl

$VLAN-List matches the tagged vlans on trk110 on SW1-5406zl and SW2-5406zl

interface Bridge-Aggregation150
description LACP SW1-5406zl
port link-type trunk
undo port trunk permit vlan 1
port trunk permit vlan $VLAN-List
link-aggregation mode dynamic
#
interface Ten-GigabitEthernet1/0/45
port link-mode bridge
description UPLINK SW1-5406zl
port link-type trunk
undo port trunk permit vlan 1
port trunk permit vlan $VLAN List
undo stp enable
port link-aggregation group 150
#
interface Ten-GigabitEthernet1/0/46
port link-mode bridge
description UPLINK SW2-5406zl
port link-type trunk
undo port trunk permit vlan 1
port trunk permit vlan $VLAN-List
shutdown
undo stp enable
port link-aggregation group 150

Please let me know if you need more information. Do you have an idea how to troubleshoot this problem?

Provided that your network configuration is OK (DT, IRF, Access Switches for both Layer 2 and Layer 3)...does moving that particular host on another port on the VSF (or moving it on the IRF directly) - provided that that port will be untagged member of the same VLAN id it has actually - cause the issue to reappear again (same MAC Address learned on the wrong interface)?

If so the issue is on the host itself (Host seems to have a Intel NIC: is the device firmware OK? is the software driver OK? Is there a possible relationship with OS used or running OS's Services/Applications which are able to manipulate MAC Address?)...in other terms...if you try a - just an example - Live Linux distribution, which is very simple to try without touching the actual host's disk - on that very same host will the issue happen again systematically as it is happening now?