Wired Intelligent Edge

 View Only

  • 1.  Stacking Member Loss/Reboot

    Posted Oct 11, 2022 11:15 AM
    Hey All,

    I manage a small network at a school district.  Recently, we've been getting warnings in our logs about a member of a stack having a loss of communication and then shutting down and rebooting. 

    Here is an output of the logs for when the event occurs:

    I 10/10/22 11:52:52 04992 vsf: ST1-CMDR: VSF port 1/50 is in error state
    I 10/10/22 11:52:52 04992 vsf: ST1-CMDR: VSF link 2 is down
    W 10/10/22 11:52:52 03270 stacking: ST1-CMDR: Topology is a Chain
    W 10/10/22 11:52:52 03258 stacking: ST1-CMDR: Member switch with Member ID 8
    removed due to loss of communication
    I 10/10/22 11:52:52 03272 stacking: ST1-CMDR: Stack fragment active
    I 10/10/22 11:52:52 04992 vsf: ST1-CMDR: VSF port 1/50 is down
    I 10/10/22 11:52:52 00077 ports: ST1-CMDR: port 8/1 is now off-line
    I 10/10/22 11:52:52 00077 ports: ST1-CMDR: port 8/3 is now off-line
    I 10/10/22 11:52:52 00077 ports: ST1-CMDR: port 8/4 is now off-line
    I 10/10/22 11:52:52 00077 ports: ST1-CMDR: port 8/5 is now off-line
    I 10/10/22 11:52:52 00077 ports: ST1-CMDR: port 8/9 is now off-line
    I 10/10/22 11:52:52 00077 ports: ST1-CMDR: port 8/20 is now off-line
    I 10/10/22 11:52:52 00077 ports: ST1-CMDR: port 8/24 is now off-line
    I 10/10/22 11:52:52 00077 ports: ST1-CMDR: port 8/30 in Trk3 is now off-line
    I 10/10/22 11:52:52 00077 ports: ST1-CMDR: port 8/31 in Trk4 is now off-line
    I 10/10/22 11:52:52 00077 ports: ST1-CMDR: port 8/32 in Trk5 is now off-line
    I 10/10/22 11:52:52 00077 ports: ST1-CMDR: port 8/33 in Trk6 is now off-line
    I 10/10/22 11:52:52 00077 ports: ST1-CMDR: port 8/34 in Trk7 is now off-line
    I 10/10/22 11:52:52 00077 ports: ST1-CMDR: port 8/35 in Trk8 is now off-line
    I 10/10/22 11:52:52 00077 ports: ST1-CMDR: port 8/36 in Trk9 is now off-line
    I 10/10/22 11:52:52 00077 ports: ST1-CMDR: port 8/38 in Trk11 is now off-line
    I 10/10/22 11:52:52 00077 ports: ST1-CMDR: port 8/39 in Trk12 is now off-line
    I 10/10/22 11:52:52 00077 ports: ST1-CMDR: port 8/40 in Trk13 is now off-line
    I 10/10/22 11:52:52 00077 ports: ST1-CMDR: port 8/41 in Trk14 is now off-line
    I 10/10/22 11:52:52 00077 ports: ST1-CMDR: port 8/42 in Trk15 is now off-line
    I 10/10/22 11:52:52 00077 ports: ST1-CMDR: port 8/43 in Trk16 is now off-line
    I 10/10/22 11:52:52 00077 ports: ST1-CMDR: port 8/45 in Trk18 is now off-line
    I 10/10/22 11:52:52 00077 ports: ST1-CMDR: port 8/46 in Trk19 is now off-line
    I 10/10/22 11:52:52 00077 ports: ST1-CMDR: port 8/48 in Trk1 is now off-line
    I 10/10/22 11:52:52 00079 ports: ST1-CMDR: trunk Trk6 is now inactive
    I 10/10/22 11:52:52 00077 ports: ST1-CMDR: port 7/33 in Trk6 is now off-line
    I 10/10/22 11:52:52 00079 ports: ST1-CMDR: trunk Trk10 is now inactive
    I 10/10/22 11:52:52 00077 ports: ST1-CMDR: port 7/37 in Trk10 is now off-line
    I 10/10/22 11:52:52 00079 ports: ST1-CMDR: trunk Trk11 is now inactive
    I 10/10/22 11:52:52 00077 ports: ST1-CMDR: port 7/38 in Trk11 is now off-line
    I 10/10/22 11:52:52 00079 ports: ST1-CMDR: trunk Trk18 is now inactive
    I 10/10/22 11:52:52 00077 ports: ST1-CMDR: port 7/45 in Trk18 is now off-line
    I 10/10/22 11:52:52 04992 vsf: ST7-STBY: VSF port 7/50 is in error state
    I 10/10/22 11:52:52 04992 vsf: ST7-STBY: VSF link 2 is down
    I 10/10/22 11:52:52 04992 vsf: ST7-STBY: VSF port 7/50 is down
    I 10/10/22 11:52:52 00079 ports: ST1-CMDR: trunk Trk3 is now inactive
    I 10/10/22 11:52:52 00077 ports: ST1-CMDR: port 7/30 in Trk3 is now off-line
    I 10/10/22 11:52:52 00079 ports: ST1-CMDR: trunk Trk4 is now inactive
    I 10/10/22 11:52:53 00077 ports: ST1-CMDR: port 7/31 in Trk4 is now off-line
    I 10/10/22 11:52:53 00079 ports: ST1-CMDR: trunk Trk15 is now inactive
    I 10/10/22 11:52:53 00077 ports: ST1-CMDR: port 7/42 in Trk15 is now off-line
    I 10/10/22 11:52:53 00079 ports: ST1-CMDR: trunk Trk16 is now inactive
    I 10/10/22 11:52:53 00077 ports: ST1-CMDR: port 7/43 in Trk16 is now off-line
    I 10/10/22 11:53:07 00077 ports: ST1-CMDR: port 4/40 is now off-line
    I 10/10/22 11:53:14 00435 ports: ST1-CMDR: port 4/40 is Blocked by STP
    I 10/10/22 11:53:16 00076 ports: ST1-CMDR: port 4/40 is now on-line
    I 10/10/22 11:54:14 04988 vsf: ST1-CMDR: VSF link 2 port 1/50 up: Peer validated
    I 10/10/22 11:54:14 04987 vsf: ST1-CMDR: VSF link 2 up: Peer has mac
    548028-b91900
    I 10/10/22 11:54:18 04992 vsf: ST1-CMDR: VSF link 2 is up
    I 10/10/22 11:54:18 03271 stacking: ST1-CMDR: Topology is a Ring
    I 10/10/22 11:54:18 03803 chassis: ST1-CMDR: System Self test completed on
    8/1-52
    I 10/10/22 11:54:18 04988 vsf: ST7-STBY: VSF link 2 port 7/50 up: Peer validated
    I 10/10/22 11:54:18 04987 vsf: ST7-STBY: VSF link 2 up: Peer has mac
    548028-b91900
    I 10/10/22 11:54:18 04992 vsf: ST7-STBY: VSF link 2 is up
    I 10/10/22 11:54:18 03125 mgr: ST1-CMDR: Startup configuration changed by SNMP.
    New seq. number 4407
    I 10/10/22 11:54:21 02555 chassis: ST1-CMDR: Co-processor Ready
    I 10/10/22 11:54:21 00256 ports: ST1-CMDR: Port 8/49 is reserved for VSF use
    I 10/10/22 11:54:21 00256 ports: ST1-CMDR: Port 8/50 is reserved for VSF use
    I 10/10/22 11:54:21 00539 stacking: ST1-CMDR: Initial sync to member 8 starting
    I 10/10/22 11:54:21 00405 ports: ST1-CMDR: port 8/49 xcvr hot-swap insert.
    I 10/10/22 11:54:21 00405 ports: ST1-CMDR: port 8/50 xcvr hot-swap insert.
    I 10/10/22 11:54:23 00435 ports: ST1-CMDR: port 7/48 is Blocked by STP
    I 10/10/22 11:54:23 00076 ports: ST1-CMDR: port 7/48 in Trk1 is now on-line
    I 10/10/22 11:54:23 00539 stacking: ST1-CMDR: Initial sync to member 8 complete
    I 10/10/22 11:54:23 03260 stacking: ST8-MMBR: Member booted
    I 10/10/22 11:54:23 03272 stacking: ST1-CMDR: Stack active

    I have tried replacing the fiber as well as the SFP with no luck.  This incident happens about 2-4 times a day.

    I have also included our switch config for that stack:

    VSF Domain ID : 1
    MAC Address : 548028-261b8b
    VSF Topology : Ring
    VSF Status : Active
    Uptime : 77d 1h 1m
    VSF MAD : None
    VSF Port Speed : 10G
    Software Version : WC.16.08.0001

    Name : HSE-M01-STACK1
    Contact :
    Location :


    Member ID : 1
    MAC Address : 548028-261b80
    Type : JL256A
    Model : Aruba JL256A 2930F-48G-PoE+-4SFP+ Switch
    Priority : 255
    Status : Commander
    ROM Version : WC.16.01.0005
    Serial Number : CN80HKX24L
    Uptime : 77d 1h 1m
    CPU Utilization : 2%
    Memory - Total : 340,857,344 bytes
    Free : 217,298,088 bytes
    VSF Links -
    #1 : Active, Peer member 2
    #2 : Active, Peer member 8



    Member ID : 2
    MAC Address : 548028-b94240
    Type : JL256A
    Model : Aruba JL256A 2930F-48G-PoE+-4SFP+ Switch
    Priority : 128
    Status : Member
    ROM Version : WC.16.01.0005
    Serial Number : CN80HKX65J
    Uptime : 77d 1h 1m
    CPU Utilization : 0%
    Memory - Total : 340,857,344 bytes
    Free : 250,588,000 bytes
    VSF Links -
    #1 : Active, Peer member 1
    #2 : Active, Peer member 3



    Member ID : 3
    MAC Address : 548028-b9f980
    Type : JL256A
    Model : Aruba JL256A 2930F-48G-PoE+-4SFP+ Switch
    Priority : 128
    Status : Member
    ROM Version : WC.16.01.0005
    Serial Number : CN80HKX5B0
    Uptime : 77d 1h 1m
    CPU Utilization : 0%
    Memory - Total : 340,857,344 bytes
    Free : 250,588,000 bytes
    VSF Links -
    #1 : Active, Peer member 2
    #2 : Active, Peer member 4



    Member ID : 4
    MAC Address : 548028-b90940
    Type : JL256A
    Model : Aruba JL256A 2930F-48G-PoE+-4SFP+ Switch
    Priority : 128
    Status : Member
    ROM Version : WC.16.01.0005
    Uptime : 77d 1h 1m
    CPU Utilization : 0%
    Memory - Total : 340,857,344 bytes
    Free : 250,727,540 bytes
    VSF Links -
    #1 : Active, Peer member 3
    #2 : Active, Peer member 5



    Member ID : 5
    MAC Address : 548028-268b00
    Type : JL256A
    Model : Aruba JL256A 2930F-48G-PoE+-4SFP+ Switch
    Priority : 128
    Status : Member
    ROM Version : WC.16.01.0005
    Uptime : 63d 5h 9m
    CPU Utilization : 0%
    Memory - Total : 340,857,344 bytes
    Free : 250,719,596 bytes
    VSF Links -
    #1 : Active, Peer member 4
    #2 : Active, Peer member 6



    Member ID : 6
    MAC Address : 548028-c25f40
    Type : JL256A
    Model : Aruba JL256A 2930F-48G-PoE+-4SFP+ Switch
    Priority : 128
    Status : Member
    ROM Version : WC.16.01.0005
    Uptime : 77d 1h 0m
    CPU Utilization : 0%
    Memory - Total : 340,857,344 bytes
    Free : 250,727,540 bytes
    VSF Links -
    #1 : Active, Peer member 5
    #2 : Active, Peer member 7



    Member ID : 7
    MAC Address : 548028-b977c0
    Type : JL256A
    Model : Aruba JL256A 2930F-48G-PoE+-4SFP+ Switch
    Priority : 128
    Status : Standby
    ROM Version : WC.16.01.0005
    Uptime : 77d 1h 0m
    CPU Utilization : 2%
    Memory - Total : 340,857,344 bytes
    Free : 234,860,252 bytes
    VSF Links -
    #1 : Active, Peer member 6
    #2 : Active, Peer member 8



    Member ID : 8
    MAC Address : 548028-b91900
    Type : JL256A
    Model : Aruba JL256A 2930F-48G-PoE+-4SFP+ Switch
    Priority : 128
    Status : Member
    ROM Version : WC.16.01.0005
    Uptime : 0d 3h 3m
    CPU Utilization : 0%
    Memory - Total : 340,857,344 bytes
    Free : 250,727,424 bytes
    VSF Links -
    #1 : Active, Peer member 7
    #2 : Active, Peer member 1

    I am fairly new to networks and to Aruba switching so just let me know if anyone needs any additional information.

    Any help is appreciated!


  • 2.  RE: Stacking Member Loss/Reboot

    Posted Oct 11, 2022 06:52 PM
    Hi! it could be an Hardware related issue (Switch? Cabling? Transceivers?) if, every time, the Switch which is rebooting daily is always the very same VSF Member (Member ID 8 connected to Commander ID 1 and Standby ID 7) and involved ports are always the same (8/49 and 8/50 on Member ID 8, corresponding ports on Commander and Standby neighbor switches go in Error state and then down too).

    In any case a check of VSF configuration would be of help (Transceivers/DACs included).

    Please note that the ArubaOS-Switch software version WC.16.08.0001 is the very first build (11/2018) of WC.16.08 release (see here), you have to consider that WC.16.08 reached the 0026 build in mid 2022 so an update would not a peregrine idea to have your VSF stack updated first. Also WC.16.08 was followed by 16.09, 16.10 and 16.11 and so an upgrade to one of those software lines would be positive too (I guess that 16.08 is not going to receive updates beyond 0026 but I could be wrong).

    Have a look at Chapter 20 "VSF Virtual Switching Framework" of this guide (2930F ArubaOS-Switch 16.11), also this nice (VSF/Backplane) best practice guide too (along with Chapter 20 that guide is also useful for maintenance purposes).

    To validate VSF status you have few commands (probably you already know them):

    show vsf
    show vsf detail
    show vsf link
    show vsf link details

    Could you eventually provide us the sanitized output of show vsf link details command?
    Original Message


Related Content