Network Management

 View Only
last person joined: yesterday 

Keep an informative eye on your network with IMC and AirWave network management solutions.

HP 6120XG switch LACP is blocked - error condition

This thread has been viewed 3 times
  • 1.  HP 6120XG switch LACP is blocked - error condition

    Posted Apr 07, 2022 08:26 AM

    Folks, 

    I am dealing with very strange issue related LACP and not sure if its bug of 6120XG switch or something else. so let me explain my issue here. 

    Network Diagram https://ibb.co/tPm7f13

    I have two Cisco Nexus 9396PX TOR switches in Cisco vPC design for milti-chassis connected to HP blade switch 6120XG. 

    This is in production last 1 years and had no issue. but recently noticed my all LACP links went down even all switches are up. During debug i found Cisco Switch was not receiving any LACP packets to bring this bundle up. (That conclude something is wrong with HP 6120XG switch) Seems like HP switch is freez or something even i can ssh on to switch and run all commands. 

    HP 6120XG firmware version: Z.14.58

    Cisco Nexus 9396PX logs during incident, Here is the logs which i found on Cisco and HP switches. 

    2022 Mar 31 05:28:39 swt-phx-k006-leaf-2-2 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel412: first operational port changed from Ethernet1/3 to Ethernet1/4
    2022 Mar 31 05:28:39 swt-phx-k006-leaf-2-2 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel412: Ethernet1/3 is down
    2022 Mar 31 05:28:39 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel412,bandwidth changed to 10000000 Kbit
    2022 Mar 31 05:28:39 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/3 is down (Initializing)
    2022 Mar 31 05:28:40 swt-phx-k006-leaf-2-2 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel412: first operational port changed from Ethernet1/4 to none
    2022 Mar 31 05:28:40 swt-phx-k006-leaf-2-2 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel412: Ethernet1/4 is down
    2022 Mar 31 05:28:40 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel412 is down (No operational members)
    2022 Mar 31 05:28:40 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel412,bandwidth changed to 100000 Kbit
    2022 Mar 31 05:28:40 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/4 is down (Initializing)
    2022 Mar 31 05:28:40 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel412 is down (No operational members)
    2022 Mar 31 05:28:49 swt-phx-k006-leaf-2-2 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/3: Ethernet1/3 is suspended
    2022 Mar 31 05:28:50 swt-phx-k006-leaf-2-2 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/4: Ethernet1/4 is suspended
    2022 Mar 31 06:05:19 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/3 is down (Link failure)
    2022 Mar 31 06:05:19 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/4 is down (Link failure)
    2022 Mar 31 06:05:19 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel412 is down (No operational members)
    2022 Mar 31 06:08:26 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel412 is down (No operational members) (message repeated 2
     times)
    2022 Mar 31 06:08:26 swt-phx-k006-leaf-2-2 %ETHPORT-5-SPEED: Interface Ethernet1/3, operational speed changed to 10 Gbps
    2022 Mar 31 06:08:26 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/3, operational duplex mode changed to Full
    2022 Mar 31 06:08:26 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/3, operational Receive Flow Control state changed to off
    2022 Mar 31 06:08:26 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/3, operational Transmit Flow Control state changed to off
    2022 Mar 31 06:08:26 swt-phx-k006-leaf-2-2 %ETHPORT-5-SPEED: Interface port-channel412, operational speed changed to 10 Gbps
    2022 Mar 31 06:08:26 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_DUPLEX: Interface port-channel412, operational duplex mode changed to Full
    2022 Mar 31 06:08:26 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel412, operational Receive Flow Control state changed to off
    2022 Mar 31 06:08:26 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel412, operational Transmit Flow Control state changed to off
    2022 Mar 31 06:08:27 swt-phx-k006-leaf-2-2 %ETHPORT-5-SPEED: Interface Ethernet1/4, operational speed changed to 10 Gbps
    2022 Mar 31 06:08:27 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/4, operational duplex mode changed to Full
    2022 Mar 31 06:08:27 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/4, operational Receive Flow Control state changed to off
    2022 Mar 31 06:08:27 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/4, operational Transmit Flow Control state changed to off
    2022 Mar 31 06:08:31 swt-phx-k006-leaf-2-2 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel412: Ethernet1/3 is up
    2022 Mar 31 06:08:31 swt-phx-k006-leaf-2-2 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel412: first operational port changed from none to Ethernet1/3
    2022 Mar 31 06:08:31 swt-phx-k006-leaf-2-2 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel412: Ethernet1/4 is up
    2022 Mar 31 06:08:31 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel412,bandwidth changed to 10000000 Kbit
    2022 Mar 31 06:08:31 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_UP: Interface Ethernet1/3 is up in mode trunk
    2022 Mar 31 06:08:31 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_UP: Interface port-channel412 is up in mode trunk
    2022 Mar 31 06:08:31 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel412,bandwidth changed to 20000000 Kbit
    2022 Mar 31 06:08:31 swt-phx-k006-leaf-2-2 %ETHPORT-5-IF_UP: Interface Ethernet1/4 is up in mode trunk

    LACP logs

    21) FSM:<Ethernet1/3> Transition at 472440 usecs after Thu Mar 31 05:28:39 2022
        Previous state: [LACP_ST_PORT_MEMBER_COLLECTING_AND_DISTRIBUTING_ENABLED]
        Triggered event: [LACP_EV_RECEIVE_PARTNER_PDU_TIMED_OUT]
        Next state: [LACP_ST_PORT_IS_DOWN_OR_LACP_IS_DISABLED]
    
    22) FSM:<Ethernet1/3> Transition at 485224 usecs after Thu Mar 31 05:28:39 2022
        Previous state: [LACP_ST_PORT_IS_DOWN_OR_LACP_IS_DISABLED]
        Triggered event: [LACP_EV_LACP_DOWN_OR_PORT_DOWN]
        Next state: [FSM_ST_NO_CHANGE]
    
    23) FSM:<Ethernet1/3> Transition at 513668 usecs after Thu Mar 31 05:28:39 2022
        Previous state: [LACP_ST_PORT_IS_DOWN_OR_LACP_IS_DISABLED]
        Triggered event: [LACP_EV_PORT_HW_PATH_DISABLED]
        Next state: [FSM_ST_NO_CHANGE]
    
    24) FSM:<Ethernet1/3> Transition at 553383 usecs after Thu Mar 31 05:28:39 2022
        Previous state: [LACP_ST_PORT_IS_DOWN_OR_LACP_IS_DISABLED]
        Triggered event: [LACP_EV_CLNUP_PHASE_II]
        Next state: [LACP_ST_PORT_IS_DOWN_OR_LACP_IS_DISABLED]
    
    25) FSM:<Ethernet1/3> Transition at 572723 usecs after Thu Mar 31 05:28:39 2022
        Previous state: [LACP_ST_PORT_IS_DOWN_OR_LACP_IS_DISABLED]
        Triggered event: [LACP_EV_LACP_ENABLED_AND_PORT_UP]
        Next state: [LACP_ST_DETACHED_LAG_NOT_DETERMINED]
    
    26) FSM:<Ethernet1/3> Transition at 573092 usecs after Thu Mar 31 05:28:40 2022
        Previous state: [LACP_ST_DETACHED_LAG_NOT_DETERMINED]
        Triggered event: [LACP_EV_PERIODIC_TRANSMIT_TIMER_EXPIRED]
        Next state: [FSM_ST_NO_CHANGE]
    
    27) FSM:<Ethernet1/3> Transition at 574523 usecs after Thu Mar 31 05:28:41 2022
        Previous state: [LACP_ST_DETACHED_LAG_NOT_DETERMINED]
        Triggered event: [LACP_EV_PERIODIC_TRANSMIT_TIMER_EXPIRED]
        Next state: [FSM_ST_NO_CHANGE]
    
    28) FSM:<Ethernet1/3> Transition at 575875 usecs after Thu Mar 31 05:28:42 2022
        Previous state: [LACP_ST_DETACHED_LAG_NOT_DETERMINED]
        Triggered event: [LACP_EV_PERIODIC_TRANSMIT_TIMER_EXPIRED]
        Next state: [FSM_ST_NO_CHANGE]
    
    29) FSM:<Ethernet1/3> Transition at 577793 usecs after Thu Mar 31 05:28:43 2022
        Previous state: [LACP_ST_DETACHED_LAG_NOT_DETERMINED]
        Triggered event: [LACP_EV_PERIODIC_TRANSMIT_TIMER_EXPIRED]
        Next state: [FSM_ST_NO_CHANGE]

    On HP 6120XG switch

    Mar 31 05:37:25 172.30.9.34 00393 lacp:  Port 18 is blocked - error condition
    Mar 31 05:37:25 172.30.9.34 00435 ports:  port 18 is Blocked by LACP
    Mar 31 05:37:51 172.30.9.34 00393 lacp:  Port 19 is blocked - error condition
    Mar 31 05:37:51 172.30.9.34 00435 ports:  port 19 is Blocked by LACP
    Mar 31 05:38:03 172.30.9.34 00393 lacp:  Port 20 is blocked - error condition
    Mar 31 05:38:03 172.30.9.34 00435 ports:  port 20 is Blocked by LACP
    Mar 31 05:38:36 172.30.9.34 00393 lacp:  Port 21 is blocked - error condition
    Mar 31 05:38:36 172.30.9.34 00435 ports:  port 21 is Blocked by LACP
    Mar 31 05:38:36 172.30.9.34 00079 ports:  trunk Trk1 is now inactive
    Mar 31 05:38:36 172.30.9.34 00002 vlan:  DEFAULT_VLAN virtual LAN disabled
    Mar 31 05:41:36 172.30.9.34 00075 system:  Out of pkt buffers; miss count: 2
    Mar 31 05:43:57 172.30.9.34 02631 SNTP:  Server not found at 132.163.96.1.
    Mar 31 05:44:06 172.30.9.34 02631 SNTP:  Server not found at 132.163.97.1.
    Mar 31 05:44:06 172.30.9.34 00414 SNTP:  Unable to reach configured SNTP servers

     

    Cisco Nexus switch port configuration

    interface port-channel412
      description ** Link to enc-k004-1-b **
      switchport mode trunk
      switchport trunk allowed vlan 60,62-70,100
      mtu 9216
      speed 10000
      vpc 412

     

    HP 6120XG config

    hostname "swt-enc-k004-1-b"
    interface 23
       disable
       lacp Active
    exit
    interface 24
       disable
       lacp Active
    exit
    trunk 18-21 Trk1 LACP
    vlan 1
       name "DEFAULT_VLAN"
       untagged 17,22-24,Trk1
       ip address dhcp-bootp
       no untagged 1-16
       exit
    vlan 100
       name "ostack_1_public1"
       tagged 1-16,Trk1
       no ip address
       jumbo
       exit
    vlan 63
       name "ostack_1_br-vxlan"
       tagged 1-16,Trk1
       no ip address
       jumbo
       exit
    vlan 64
       name "ops"
       tagged 1-16,Trk1
       no ip address
       jumbo
       exit
    vlan 65
       name "ostack_1_br-mgmt"
       tagged 1-16,Trk1
       no ip address
       jumbo
       exit
    vlan 66
       name "ostack_1_ops"
       tagged 1-16,Trk1
       no ip address
       jumbo
       exit
    vlan 67
       name "ostack_1_odmz"
       tagged 1-16,Trk1
       no ip address
       jumbo
       exit
    vlan 68
       name "ostack_1_idmz"
       tagged 1-16,Trk1
       no ip address
       jumbo
       exit
    vlan 69
       name "ostack_lab"
       tagged 1-16,Trk1
       no ip address
       jumbo
       exit
    vlan 70
       name "pxe"
       untagged 1-16
       tagged Trk1
       no ip address
       jumbo
       exit
    vlan 62
       name "ostack_1_br-lbaas"
       tagged 1-16,Trk1
       no ip address
       jumbo
       exit
    vlan 60
       name "ostack_1_br-storage"
       tagged 1-16,Trk1
       no ip address
       jumbo
       exit
    
    logging 172.30.8.7 oobm
    logging facility syslog
    timesync sntp
    sntp unicast
    sntp server priority 1 132.163.96.1 3 oobm
    sntp server priority 2 132.163.97.1 3 oobm
    snmp-server community "XXXXXX" operator
    spanning-tree
    spanning-tree 1 admin-edge-port
    spanning-tree 1 bpdu-protection
    spanning-tree 2 admin-edge-port
    spanning-tree 2 bpdu-protection
    spanning-tree 3 admin-edge-port
    spanning-tree 3 bpdu-protection
    spanning-tree 4 admin-edge-port
    spanning-tree 4 bpdu-protection
    spanning-tree 5 admin-edge-port
    spanning-tree 5 bpdu-protection
    spanning-tree 6 admin-edge-port
    spanning-tree 6 bpdu-protection
    spanning-tree 7 admin-edge-port
    spanning-tree 7 bpdu-protection
    spanning-tree 8 admin-edge-port
    spanning-tree 8 bpdu-protection
    spanning-tree 9 admin-edge-port
    spanning-tree 9 bpdu-protection
    spanning-tree 10 admin-edge-port
    spanning-tree 10 bpdu-protection
    spanning-tree 11 admin-edge-port
    spanning-tree 11 bpdu-protection
    spanning-tree 12 admin-edge-port
    spanning-tree 12 bpdu-protection
    spanning-tree 13 admin-edge-port
    spanning-tree 13 bpdu-protection
    spanning-tree 14 admin-edge-port
    spanning-tree 14 bpdu-protection
    spanning-tree 15 admin-edge-port
    spanning-tree 15 bpdu-protection
    spanning-tree 16 admin-edge-port
    spanning-tree 16 bpdu-protection
    spanning-tree Trk1 priority 4
    oobm
       ip address 172.30.9.34 255.255.248.0
       ip default-gateway 172.30.8.1
       exit
    password manager
    password operator

     

    After reboot HP blade 6120XG LACP came back and traffic started flowing,  This happened in past to other racks also but everytime we just reboot switch and it come back. (We don't have HPE support where we can ask to find out if its BUG of HP 6120XG switch or something else in my config) 



    ------------------------------
    Satish Patel
    ------------------------------