Wired Intelligent Edge

Hello,

What's the logic that is ruling the VSX ISL ports' sequence output provided by the show vsx lacp aggregates command (below an example on the VSX I've setup, command executed on primary Aruba 8320 node):

Aruba-8320-1# show vsx lacp aggregates lag128
                    Local-peer                        Remote-peer
-------------------------------------------------------------------------------

Aggregate-name    : lag128                            lag128
Interfaces        : 1/1/49 1/1/52 1/1/53 1/1/50       1/1/50 1/1/53 1/1/49 1/1/52 
Heartbeat rate    : slow                              slow
Hash              : l3-src-dst                        l3-src-dst
Aggregate mode    : active                            active

Note that show vsx lacp aggregates lag128 vsx-peer command shows the same sequence on each respective peer (clearly local/remote peer's references are swapped)...VSX is In-Sync, Keepalive is operational and DAC Cables used for VSL ISL were manually connected sequencially (first 1/1/49 to 1/1/49, few seconds after 1/1/50 to 1/1/50, then 1/1/52 to 1/1/52 and finally 1/1/53 to 1/1/53...1/1/51 and 1/1/54 were not used at all).

Also note the (stripped) output of show interface brief:

1/1/47    --      routed --          no      down    No XCVR installed      --    
1/1/48    --      routed 1000SX      yes     up                             1000  
1/1/49    1       trunk  QSFP+DA1    yes     up                             40000 
1/1/50    1       trunk  QSFP+DA1    yes     up                             40000 
1/1/51    --      routed --          no      down    No XCVR installed      --    
1/1/52    1       trunk  QSFP+DA1    yes     up                             40000 
1/1/53    1       trunk  QSFP+DA1    yes     up                             40000 
1/1/54    --      routed --          no      down    No XCVR installed      --    
lag128    1       trunk  --          yes     up      --                     160000  

And this show lldp neighbor-info vsx-peer that shows that four ports I used in VSX ISL, as seen from the remote VSX peer, are all conneted sequencially (49 with 49, 50 with 50 and so on...) and aren't crossed:

Aruba-8320-1# show lldp neighbor-info vsx-peer 

LLDP Neighbor Information 
=========================

Total Neighbor Entries          : 5
Total Neighbor Entries Deleted  : 0
Total Neighbor Entries Dropped  : 0
Total Neighbor Entries Aged-Out : 0

LOCAL-PORT  CHASSIS-ID         PORT-ID       PORT-DESC     TTL      SYS-NAME    
--------------------------------------------------------------------------------
1/1/48      d0:67:26:xx:xx:xx  1/1/48        1/1/48        120      Aruba-8320-1
1/1/49      d0:67:26:xx:xx:xx  1/1/49        1/1/49        120      Aruba-8320-1
1/1/50      d0:67:26:xx:xx:xx  1/1/50        1/1/50        120      Aruba-8320-1
1/1/52      d0:67:26:xx:xx:xx  1/1/52        1/1/52        120      Aruba-8320-1
1/1/53      d0:67:26:xx:xx:xx  1/1/53        1/1/53        120      Aruba-8320-1

(Port 1/1/48 on both ends is the Keepalive link).

I'm really curious...

Interesting question!

I took a quick look at a VSX pair of 8320s in the TME lab and I am seeing the same behavior you observed.  What is more interesting is that the LAG we are using for ISL (lag 10) has three interfaces configured as members.  I've removed all but one cable to see if any changes occur and the output of 'show vsx lacp interface lag10' consistently shows three members in the LAG with the local peer listing the ports "out of sequence" and the remote-peer list the ports as I would expect:

SWHQ-AGG1A# sh vsx lacp aggregates lag10
Local-peer Remote-peer
-------------------------------------------------------------------------------

Aggregate-name : lag10 lag10
Interfaces : 1/1/52 1/1/53 1/1/51 1/1/51 1/1/52 1/1/53

Heartbeat rate : slow slow
Hash : l3-src-dst l3-src-dst
Aggregate mode : active active

I'll ping some contacts in developmenet and share back what I find.

To confirm, you didn't notice any problems/issues with packet forwarding just in the output of the 'show' commands?

-t

Hi, after a full reboot of the VSX by performing this sequence:

1 - Power-down Aruba 8320-1 (Primary)

2 - Power-down Aruba 8320-2 (Secondary)

3 - Physically relocate both Aruba 8320 into definitive Rack position

4 - Power-up Aruba 8320-1 (Primary)

5 - Power-up Aruba 8320-2 (Secondary) about 20 seconds after powering up Aruba 8320-1 (so the first ready is the Primary).

Now the VSX ISL LAG 128 (lag128) member ports' sequence on Secondary peer looks like this:

Aruba-8320-2# show vsx lacp aggregates lag128
                    Local-peer                        Remote-peer
-------------------------------------------------------------------------------

Aggregate-name    : lag128                            lag128
Interfaces        : 1/1/50 1/1/49 1/1/53 1/1/52       1/1/52 1/1/49 1/1/50 1/1/53 
Heartbeat rate    : slow                              slow
Hash              : l3-src-dst                        l3-src-dst
Aggregate mode    : active                            active

and, conversely, the Primary peer - queried by the Secondary - shows that:

Aruba-8320-2# show vsx lacp aggregates lag128 vsx-peer
                    Local-peer                        Remote-peer
-------------------------------------------------------------------------------

Aggregate-name    : lag128                            lag128
Interfaces        : 1/1/50 1/1/49 1/1/53 1/1/52       1/1/52 1/1/49 1/1/50 1/1/53
Heartbeat rate    : slow                              slow
Hash              : l3-src-dst                        l3-src-dst
Aggregate mode    : active                            active

So it looks like the sequence ordering changes after a reboot (DAC Cables are still connected sequencially: 49 with 49, 50 with 50, 52 with 52 and 53 with 53).

Note how the running-config (JSON) sequence about lag128's member ports is represented on the Secondary:

 "lag128": {
            "admin": "up",
            "description": "VSX-ISL-LAG128-Link",
            "interfaces": [
                "1%2F1%2F50",
                "1%2F1%2F49",
                "1%2F1%2F53",
                "1%2F1%2F52"
            ],
            "lacp": "active",
            "name": "lag128",
            "other_config": {
                "mclag_enabled": "false"
            },
            "vlan_mode": "native-tagged",
            "vlan_tag": "1",
            "vsx_sync": [
                "^vlan.*"
            ]
        }

I didn't notice any issues with packet forwarding BUT that's because simply the VSX I'm working with is still in its staging phase so no Multi-Chassis LAGs were defined (so there is no usage at Layer 2 level...), I'm going to start with deployment in a few days and then I hope the setup will be bullet-proof.

Hi Davide,

I'll share your additional observations with development and report back when I have an update for you.

I don't forsee any problems with your configuration -- especially as the 'show lacp interface' shows that you have a LAG formed between the VSX peers.

I have a question for you:  We generally see a pair of 40G links used for the LAG supporting ISL.  Did a bandwidth requirement drive you to use 4x40G interfaces?  What is your  solution for the keepalive configuration?  If you have free interfaces, our leading practice is to create a keepalive VRF with a pair of interface in a LAG.  You can also use any L3 path between the VSX peers if you don't have free interfaces.  Here is a config sample:

vrf VSX_KEEPALIVE

interface 1/1/47
no shutdown
lag 11
exit

interface 1/1/48
no shutdown
lag 11
exit

interface lag 11
vrf attach VSX_KEEPALIVE
ip address 192.168.1.1/24
lacp mode active
exit

vsx
<config omitted>

keepalive peer 192.168.1.2 source 192.168.1.1 vrf VSX_KEEPALIVE

-t

Todd thanks for your support!

Our configuration was extensively discussed in other threads (basically our scenario is a Layer 2 isolated network with a lot of VSX LAGs AKA MC-LAGs to downstream servers and all-flash storages).

Our setup is 4x40Gbps lag128 for VSX ISL (this part will truly deserve another thread...since I've few questions about the potential usage - call it how traffic is going to be distributed - of the ISL LAGs during normal operations, not only during a node failure) and the Keepalive was implemented on a separate VRF (I called it "VSX-ISL-Keepalive") and uses the 1/1/48 ports on both peers equipped with SFP J4858C HPE X121 1G SFP LC SX Transceiver interconnected using 0,5 meter fiber optic cable&nbsp;(Aruba told us it is enough for the rate of traffic the Keepalive will see).

I asked if Keepalive (which is a routed interface) could/should be done using a LAG (Layer 3 LAG) instead of using just a single link...in my vision that idea was to enhance resiliency of the heartbeat between both peers...but it was suggested me that it was too much and not worth the the pair of additional SFP+ slots lost for that (my initial idea was to use a LAG of two SFP transceivers - say port 1/1/47 and 1/1/48 - to form the Keepalive's LAG...see this thread posted few days ago about that).

So...up to now, what I've done looks good enough...to me.

VSX is already formed...all is in the staging area (we postponed the go-live for first week of September) so I've some time to explore the system and report what I find.

Kind regards, Davide.

Hi Davide,

Vincent is one of my TME peers.  He is right -- there is not a bandwidth need.  If you have  free interfaces and cables, there is no harm in using a LAG.  This does provide minimal extra protection as the keepalive is "important" during an ISL failure.  I'd use a LAG and call it "cheap insurance" 

-t

I agree with you (it was my initial idea), if I'll find two additional SFP J4858C Transceiver spares I'll rearrange ISL Keepalive configuration by converting it from single link to a two links' LAG.

It looks like that also VSX LAGs (Multi-Chassis LAGs) are affected by an unusual output sequence for member ports:

Aruba-8320-1# show vsx lacp aggregates lag2
                    Local-peer                Remote-peer
-------------------------------------------------------------------------------

Aggregate-name    : lag2 (multi-chassis)      lag2 (multi-chassis)
Interfaces        : 1/1/3 1/1/4               1/1/4 1/1/3
Remote-interfaces : 1/1/4 1/1/3               1/1/3 1/1/4
Heartbeat rate    : slow                      slow
Hash              : l3-src-dst                l3-src-dst
Aggregate mode    : active                    active

Notice how the sequence is 1/1/3, 1/1/4 on Local peer (VSX Primary) and 1/1/4, 1/1/3 on Remote peer (VSX Secondary).