Wired Intelligent Edge

 View Only
  • 1.  Spanning tree uplinks on CX 8100 VSX pair

    Posted 28 days ago
      |   view attached

    I have 2 CX 8100 Aggregation switches configured with VSX.  Latest 10.13.1101 code

    Upstream is a pair of CX 8360s in VSX as well.  Each Agg switch has a LAG to the 2 cores LAG5/6.  It is not a MC-LAG on the 8100s

    Agg1 switch is working normally but Agg2 switch insists on traversing the inter-switch-link to get back to the cores.

    I am getting a STP cost of 802 on the (upstream) MC-LAG vs cost of 800 on Agg1

    The cost is only 801 when going over the ISL and other switch's LAG.

    I can't set the cost of the ISL.  It won't let it change.

    I just want this to work as expected.

    DC-8100-Agg2# sh span
    Spanning tree status      : Enabled Protocol: MSTP

    MST0
      Root ID    Priority   : 8192
                 MAC-Address: 02:02:00:00:01:00
                 Hello time(in seconds):2  Max Age(in seconds):20
                 Forward Delay(in seconds):15

      Bridge ID  Priority  : 16384
                 MAC-Address: 02:01:00:00:05:00
                 Hello time(in seconds):2  Max Age(in seconds):20
                 Forward Delay(in seconds):15

    Port         Role           State      Cost           Priority   Type             BPDU-Tx    BPDU-Rx    TCN-Tx     TCN-Rx
    ------------ -------------- ---------- -------------- ---------- ---------------- ---------- ---------- ---------- ----------
    ....
    lag6         Alternate      Blocking   802            64         P2P              2          465792     0          8
    lag256       Root           Forwarding 1              64         P2P              5          465792     2          4

    DC-8100-Agg1# sh span
    Spanning tree status      : Enabled Protocol: MSTP

    MST0
      Root ID    Priority   : 8192
                 MAC-Address: 02:02:00:00:01:00
                 Hello time(in seconds):2  Max Age(in seconds):20
                 Forward Delay(in seconds):15

      Bridge ID  Priority  : 16384
                 MAC-Address: 02:01:00:00:05:00
                 Hello time(in seconds):2  Max Age(in seconds):20
                 Forward Delay(in seconds):15

    Port         Role           State      Cost           Priority   Type             BPDU-Tx    BPDU-Rx    TCN-Tx     TCN-Rx
    ------------ -------------- ---------- -------------- ---------- ---------------- ---------- ---------- ---------- ----------
    ......
    lag5         Root           Forwarding 800            64         P2P              9          558994     8          8
    lag256       Designated     Forwarding 1              64         P2P              606440     47893      20         12

    Number of topology changes    : 23
    Last topology change occurred : 453774 seconds ago

    Attachment(s)



  • 2.  RE: Spanning tree uplinks on CX 8100 VSX pair

    Posted 28 days ago
    Edited by thomasbnc 28 days ago

    Hi

     

    Thanks for reaching out to the community.

     

    Your setup is interesting but also rather unusual. Could you please elaborate a bit more on your motivation to not build an MC-LAG uplink to the core from your Agg1/Agg2 switches (also possible with 4 links if needed/desired)?

    An MC-LAG to the upstream layer could provide redundancy and the benefit of a "local" uplink which could actively forward the traffic rather than sending it over the ISL.

    <edit>

    The triangle Core ---(lag5)----> Agg1 ---- (ISL)----> Agg2 ---(lag6)--->Core builds a loop. The link from Agg2 (VSX secondary) to the Core needs to get blocked to avoid a looping condition. To do that, I guess that VSX adjusts the cost of the uplink to be just above the other path to the root which is in total 800 + 1 = 801. So, I'm not surprised by that.

    </edit>

    An ISL can by definition never go into a blocked state from an STP point of view. Please refer to the VSX guide for details:
    https://arubanetworking.hpe.com/techdocs/AOS-CX/10.13/PDF/vsx.pdf

    page 75: "ISL is always part of STP, nonblocking and it sends and receives BPDUs."

     

    There is another excellent guide as a reference for best practices in the context of VSX:
    https://support.hpe.com/hpesc/public/docDisplay?docId=a00094242en_us&docLocale=en_US

     

    Regards,

    Thomas

     

     






  • 3.  RE: Spanning tree uplinks on CX 8100 VSX pair

    Posted 28 days ago

    This is surely unusual topology. We recommend to have square topology per se between Core and Agg, if both are in VSX of its own. And also, MC-LAG between them (the layers). This way we avoid such STP issues. Here in your setup, you have not used MC-LAG and it kind of creates a triangle setup between the CORE VSX and AGG-1 (which is in-real a VSX too). ISL is meant not to be blocked by design or else will affect the sync. 



    ------------------------------
    [Aniruddha][Jasu]
    ------------------------------



  • 4.  RE: Spanning tree uplinks on CX 8100 VSX pair

    Posted 28 days ago

    As other community members replied, this topology leads to L2 loop, so spanning-tree will block one path.

    Please see the VSX Configuration Best Practices for a topology where a single LAG is used between TOR and AGG, like in this picture (extracted from the new revision of the VSX Best Practices that will be published soon).




  • 5.  RE: Spanning tree uplinks on CX 8100 VSX pair

    Posted 28 days ago

    I was hoping to avoid MC-LAG on the Agg 8100s.   The previous switches in this place were independent but connected via MC-LAG to the cores.

    I will need to schedule an outage to change the uplinks to a MC-LAG with 4 members.




  • 6.  RE: Spanning tree uplinks on CX 8100 VSX pair

    Posted 28 days ago

    If you don't want MCLAG on the core, then don't use L2 trunking between Core nodes, or avoid carrying over Core ISL downstream VLANs.

    Typical topology where there is no MCLAG on Core:




  • 7.  RE: Spanning tree uplinks on CX 8100 VSX pair

    Posted 28 days ago

    Such a change can be done without interruption. Just remove / re-add lag5 on the core side as MC-LAG interface. While you are doing this, lag6 takes over the traffic from your Agg switches. When done, reconfigure (remove, re-add) lag5 on Agg1 as MC-LAG with one or two link(s) to the MC-LAG on the core. Then configure lag5 (as MC-LAG) on Agg2 and add one of the local links from lag6 (non-MC-LAG) to lag5 (MC-LAG). This should switchover traffic again to lag5. Then decommission lag6 and add the remaining links to the MC-LAG lag5. Spanning-Tree will manage your loops. Just allow enough time after each step to go through the various STP states and let the topology converge. 




  • 8.  RE: Spanning tree uplinks on CX 8100 VSX pair

    Posted 28 days ago
    Edited by Mflowers@beta.team 28 days ago

    "Such a change can be done without interruption"
    Please do not listen to this advice.  Spanning tree will need to converge during this change and it will 100% cause some interruption during convergence.

    I do not know what you are supporting with these switches and what your clients are.   The time it takes spanning-tree to converge might just be a small blip to your end users (if your clients are users/people) or it could cause I/O timeouts/iSCSI links to fail (if you are supporting VM infrastructure with shared storage).

    Your plan to do this off hours/maintenance window is a wise move.  Even if you expect the change to be a small blip in the network, it is still a good idea to be careful when possible.  We all make mistakes and better to make a mistake during off hours.

    This is a large ingress/egress point of your network and not something to take lightly. 




  • 9.  RE: Spanning tree uplinks on CX 8100 VSX pair

    Posted 28 days ago

    This sounds like it is working as expected and normally for how you have it setup.

    Instead of look at it like 4 different switches, lets look at it as 2 different switches (VSX-CORE, VSX-AGG).  The ISL link, lets stop thinking of it as another wired connection and as the internal switch port to port communication.

    So what does your architecture look like now?

    VSX-CORE
     | lag5   | lag6
    VSX-AGG

    As you can see in my very poor ASCII diagram above, lag5 and lag6 are two different links and if both were online at the same time you would have a loop.  In order for data to get from VSX-AGG it has to move the traffic to the data plane of the switch and forward it out only one of the lag ports (lag6).

    Just remember, look at VSX\VSF switches as if they were one physical switch when thinking about how packets are forwarded.  The data plane in VSX/VSF is virtualized into one single data plane and not two different data planes.

    If you would want it to work so that lag5 and lag6 are both forwarding and the ISL link would be down then you need to separate the data plane into two independent data planes (remove VSX).

    You are mashing two switches together into one single one and then asking "I have two links from this one switch going to this other one switch.  Why is one link getting blocked?"