Comware

Hello,

I am trying to run 3 HPE 5510HI switches in IRF with BFD MAD configured. IRF is working fine, when I try to simulate split-brain scenario and shutdown all 10Gb ports on Master device, all other physical 1Gb ports are shut down. Should I MAD exclude 1Gb ports from shutdown, so that devices could establish BFD sessions? It is not clearly stated in any manual, but it seems logical to me.

Hi!

As is explained here it looks so (it's reasonable)...especially reading the statement on the Basic principle of MAD paragraph: "... the other one will shutdown all the local interfaces (except the MAD-excluded interfaces)."

Maybe the command mad exclude interface can help here (reference, as example: HP 5920 & 5900 Switch Series IRF Configuration Guide)

Thanks for the reply. Its strange that this is not mentioned in official configuration guides. Will try to test it and will report results here in forum.

Hi Tomas,

It seems mentioned here: HPE FlexNetwork 5510 HI Switch Series IRF Configuration Guide (on Page 30: "Excluding a port from the shutdown action upon detection of multi-active collision").

If I read the reported statement "By default, all ports (except the console and IRF physical interfaces) shut down automatically when the IRF fabric transits to the Recovery state." correctly a question rises: Are BFD-MAD dedicated ports considered not part of the IRF dedicated physical interfaces' group and so they're not automatically excluded from the imposed shutdown?

I think after enabling BFD MAD all 10Gb ports automatically were excluded, but not 1Gb ports used for BFD MAD.

I cannot stil lconfigure IRF with BFD MAD, I will paste here my setup, and what happens after I disconenct Master from the ring.

Here is my IRF setup:

irf mac-address persistent timer
 irf auto-update enable
 undo irf link-delay
 irf member 1 priority 30
 irf member 2 priority 20
 irf member 3 priority 10

irf-port 1/1
 port group interface Ten-GigabitEthernet1/0/49
 port group interface Ten-GigabitEthernet1/0/50
#
irf-port 1/2
 port group interface Ten-GigabitEthernet1/0/51
 port group interface Ten-GigabitEthernet1/0/52
#
irf-port 2/1
 port group interface Ten-GigabitEthernet2/0/51
 port group interface Ten-GigabitEthernet2/0/52
#
irf-port 2/2
 port group interface Ten-GigabitEthernet2/0/49
 port group interface Ten-GigabitEthernet2/0/50
#
irf-port 3/1
 port group interface Ten-GigabitEthernet3/0/51
 port group interface Ten-GigabitEthernet3/0/52
#
irf-port 3/2
 port group interface Ten-GigabitEthernet3/0/49
 port group interface Ten-GigabitEthernet3/0/50

mad exclude interface GigabitEthernet1/0/1
 mad exclude interface GigabitEthernet1/0/2
 mad exclude interface GigabitEthernet2/0/1
 mad exclude interface GigabitEthernet2/0/2
 mad exclude interface GigabitEthernet3/0/1
 mad exclude interface GigabitEthernet3/0/2

Here is all IRF information when IRF ring is working ok:

<IRF>dis irf
MemberID    Role    Priority  CPU-Mac         Description
 *+1        Master  30        00e0-fc0f-8c02  ---
   2        Standby 20        00e0-fc0f-8c03  ---
   3        Standby 10        00e0-fc0f-8c04  ---
--------------------------------------------------
 * indicates the device is the master.
 + indicates the device through which the user logs in.

 The bridge MAC of the IRF is: 2c23-3aee-60cf
 Auto upgrade                : yes
 Mac persistent              : 6 min
 Domain ID                   : 0

<IRF>dis mad verbose
Multi-active recovery state: No
Excluded ports (user-configured):
  GigabitEthernet1/0/1
  GigabitEthernet1/0/2
  GigabitEthernet2/0/1
  GigabitEthernet2/0/2
  GigabitEthernet3/0/1
  GigabitEthernet3/0/2
Excluded ports (system-configured):
  Ten-GigabitEthernet1/0/49
  Ten-GigabitEthernet1/0/50
  Ten-GigabitEthernet1/0/51
  Ten-GigabitEthernet1/0/52
  Ten-GigabitEthernet2/0/49
  Ten-GigabitEthernet2/0/50
  Ten-GigabitEthernet2/0/51
  Ten-GigabitEthernet2/0/52
  Ten-GigabitEthernet3/0/49
  Ten-GigabitEthernet3/0/50
  Ten-GigabitEthernet3/0/51
  Ten-GigabitEthernet3/0/52
MAD ARP disabled.
MAD ND disabled.
MAD LACP disabled.
MAD BFD enabled interface: Vlan-interface3
  MAD status                 : Normal
  Member ID   MAD IP address       Neighbor   MAD status
  1           192.168.2.1/24       2          Normal
  1           192.168.2.1/24       3          Normal
  2           192.168.2.2/24       1          Normal
  2           192.168.2.2/24       3          Normal
  3           192.168.2.3/24       1          Normal
  3           192.168.2.3/24       2          Normal

<IRF>dis irf topology
                              Topology Info
 -------------------------------------------------------------------------
               IRF-Port1                IRF-Port2
 MemberID    Link       neighbor      Link       neighbor    Belong To
 1           UP         2             UP         3           00e0-fc0f-8c02
 2           UP         3             UP         1           00e0-fc0f-8c02
 3           UP         1             UP         2           00e0-fc0f-8c02

<IRF>dis bfd session
 Total Session Num: 2     Up Session Num: 0     Init Mode: Active

 IPv4 Session Working Under Ctrl Mode:

 LD/RD          SourceAddr      DestAddr        State    Holdtime    Interface
 97/0           192.168.2.1     192.168.2.2     Down        /        Vlan3
 98/0           192.168.2.1     192.168.2.3     Down        /        Vlan3

<IRF>display vlan 3
 VLAN ID: 3
 VLAN type: Static
 Route interface: Configured
 IPv4 address: 192.168.2.1
 IPv4 subnet mask: 255.255.255.0
 Description: VLAN 0003
 Name: VLAN 0003
 Tagged ports:   None
 Untagged ports:
    GigabitEthernet1/0/1          GigabitEthernet1/0/2
    GigabitEthernet2/0/1          GigabitEthernet2/0/2
    GigabitEthernet3/0/1          GigabitEthernet3/0/2

Then I do Split-Brain (by shuting down all Ten gig ports on Master device (ID=1)):

<IRF>dis irf
MemberID    Role    Priority  CPU-Mac         Description
 *+1        Master  30        00e0-fc0f-8c02  ---
--------------------------------------------------
 * indicates the device is the master.
 + indicates the device through which the user logs in.

 The bridge MAC of the IRF is: 2c23-3aee-60cf
 Auto upgrade                : yes
 Mac persistent              : 6 min
 Domain ID                   : 0

<IRF>dis mad verbose
Multi-active recovery state: No
Excluded ports (user-configured):
  GigabitEthernet1/0/1
  GigabitEthernet1/0/2
Excluded ports (system-configured):
  Ten-GigabitEthernet1/0/49
  Ten-GigabitEthernet1/0/50
  Ten-GigabitEthernet1/0/51
  Ten-GigabitEthernet1/0/52
MAD ARP disabled.
MAD ND disabled.
MAD LACP disabled.
MAD BFD enabled interface: Vlan-interface3
  MAD status                 : Faulty
  Member ID   MAD IP address       Neighbor   MAD status
  1           192.168.2.1/24       2          Normal
  1           192.168.2.1/24       3          Faulty

<IRF>dis irf topology
                              Topology Info
 -------------------------------------------------------------------------
               IRF-Port1                IRF-Port2
 MemberID    Link       neighbor      Link       neighbor    Belong To
 1           DOWN       ---           DOWN       ---         00e0-fc0f-8c02

<IRF>dis bfd session
 Total Session Num: 2     Up Session Num: 0     Init Mode: Active

 IPv4 Session Working Under Ctrl Mode:

 LD/RD          SourceAddr      DestAddr        State    Holdtime    Interface
 97/97          192.168.2.1     192.168.2.2     Init     3800ms      Vlan3
 98/0           192.168.2.1     192.168.2.3     Down        /        Vlan3

Session with SRC 192.168.2.1 and DST 192.168.2.2 is constantly changing from Init to Down to Init.

Session with SRC 192.168.2.1 and DST 192.168.2.3 is constantly Down.

What is happening there?

In case someone is facing the same issue, I have found this document:

http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=mmr_sf-EN_US000005491

And in their configuration, they have:

UPDATE:

I have loged in to Devices 2 and 3 via console and both of them are constantly giving these log messages:

%Jan 1 05:39:25:306 2013 IRF BFD/5/BFD_CHANGE_FSM: Sess[192.168.2.2/192.168.2.1, LD/RD:97/97, Interface:Vlan3, SessType:Ctrl, LinkType:INET], Sta: UP->DOWN, Diag: 3
%Jan 1 05:39:25:413 2013 IRF BFD/5/BFD_CHANGE_FSM: Sess[192.168.2.2/192.168.2.1, LD/RD:97/97, Interface:Vlan3, SessType:Ctrl, LinkType:INET], Sta: DOWN->INIT, Diag: 0
%Jan 1 05:39:25:414 2013 IRF BFD/5/BFD_CHANGE_FSM: Sess[192.168.2.2/192.168.2.1, LD/RD:97/97, Interface:Vlan3, SessType:Ctrl, LinkType:INET], Sta: INIT->UP, Diag: 0
%Jan 1 05:39:25:414 2013 IRF DEV/1/MAD_DETECT: Multi-active devices detected, please fix it.

Wait.

Initially you wrote "I am trying to run 3 HPE 5510HI switches in IRF with BFD MAD configured": does your IRF scenario is a 3 chassis IRF fabrics (without any Intermediate Device) or a 2 chassis IRF fabrics (with the third Switch acting as the Intermediate Device)?

From the dis irf topology result it is a 3 chassis IRF fabrics (three members).

For a 3 chassis IRF fabrics I mean there is a Master and two Subordinates (connected with a Daisy-chain or a Ring topology).

Isn't that BFD MAD works only in a 2 chassis IRF fabric scenario with or withouth an Intermediate Device and, as soon as you open up to a 3 (or 4) chassis IRF fabric scenario you're forced to use an Intermediate Device?

Yes - for more than 2 switches in IRF fabric, using BFD MAD, you need to use an intermediate device, and disable STP on the BFD MAD interfaces.

The IRF configuration guide should have a configuiration examples section titled - "BFD MAD-enabled IRF configuration example".

Please try to emulate what is shown there.

I am a bit confused. Actually there is a configuration example showing more than 2 devices connected in a ring topology with full mesh BFD MAD connections:

http://www.h3c.com.hk/Technical_Support___Documents/Technical_Documents/Switches/H3C_S5800_Series_Switches/Configuration/Typical_Configuration_Example/H3C_S5820X[S5800]_Configuration_Examples-6W100/#_Ref270520212

I must to admit that BFD MAD requirements and restrictions seem to vary reading documentation (e.g. HPE FlexNetwork 10500 Switch Series versus HPE FlexNetwork 5510 HI Switch Series).

For the HPE FlexNetwork 5510 HI Switch Series, in contrast to what is reported for the bigger HPE FlexNetwork 10500 Switch Series in its IRF Configuration Guide, there is not a clear statement about the Intermediate Device requirement when IRF members are more than two (exactly as it happens reading the H3C documentation reported above which doesn't report that requirement).

See this:

taken from the MAD Mechanisms paragraph of the IRF Configuration Guide for the HPE FlexNetwork 5510 HI Switch Series.

When using the common Ethernet ports...notice the statement: "Each member device has a BFD MAD link to an intermediate device, or all member devices have a BFD MAD link to each other." which admits a Mesh configuration ("all to each other") if the Intermediate Device is not available...at least as I understand it.

Last minute edit:

Among BFD MAD Disadvantages are reported:

• If no intermediate device is used, any two IRF members must have a BFD MAD link to each other.
• If an intermediate device is used, every IRF member must have a BFD MAD link to the intermediate device.

So it seems that the magic of two IRF members' threshold associated with the presence/absence of an Intermediate Device is there.

Hi,

I think for MAD types we should concern what H3C site presenting (reference screen-shoot attached), but:

1. If you have daisy chain topology (min. 3 IRF nodes) , you cannot use LACP MAD without intermediate switch.

2. Also you cannot use BFD MAD with daisy chain , because not having full mesh (BFD sessions between IRF nodes).

So, for 3 IRF nodes you can use LACP MAD with intermediate switch OR you can setup/change to IRF ring topology and having BFD MAD possible.

Michal

My setup is IRF ring topology and I cannot understand why one bfd link is UP and another one is DOWN:

[IRF]dis bfd session
Total Session Num: 2 Up Session Num: 1 Init Mode: Active

IPv4 Session Working Under Ctrl Mode:

LD/RD SourceAddr DestAddr State Holdtime Interface
97/97 192.168.2.1 192.168.2.2 Up 1384ms Vlan3
98/0 192.168.2.1 192.168.2.3 Down / Vlan3