Wired Intelligent Edge

Recently we added a second 5406 and setup a VSF.

We have noticed that when we ssh (on the OOBM of the cluster) the shell often becomes unresponsive for short successive periods.

"show cpu" shows high usage in such periods, often listing processes which should not to the best of my knowledge be running (HTTP, we disabled web-management).

We're probably going to be contacting support soon, but any pointers in the mean time would be great.

Hi! any other relevant information would help...you described what you did but you didn't provided any "evidence", logging or specific details such as software version used or in form of the output of some usual diagnostic show commands (related to VSF).

Difficult to guess blindly...

Hi parnassus, I'd be more then happy to provide more evidence, just not really sure where to look.

The following is the output of show cpu, please note that tftp and web are set to off:

no tftp client
no tftp server
no web-management

show logging doesn't show anything out of the ordinary as far as I can tell.

# show cpu

84 percent busy, from 9 sec ago
1 sec ave: 99 percent busy
5 sec ave: 98 percent busy
1 min ave: 35 percent busy


Task usage for last 4 seconds
% CPU | Description
-------+--------------------------
3.1 | Idle
1.5 | Hardware Mgmt
1.1 | System Services
91.5 | HTTP
2.3 | IPsec
0.4 | TFTP

As far as HW/SW goes:

(5400) J9850A running KB.16.10.0007
module 1/A type j9993a
module 1/B type j9993a
module 1/C type j9996a
module 1/D type j9996a
module 1/F type j9990a
module 2/A type j9993a
module 2/B type j9993a
module 2/C type j9996a
module 2/D type j9996a
module 2/E type j9995a
module 2/F type j9990a

Well...that 91.5% of (busy) CPU dedicated to HTTP service is quite interesting (if the Web Management is truly disabled who else is using HTTP?).

Have you tried also the no web-management ssl command too?

What show cpu slot all command reports?

Is the VSF actually OK? does the show vsf command report anything strange?

Do you have process-tracking enabled (to be enabled before the show cpu command with process-tracking command; to disable use the no process-tracking command after exiting with Ctrl-C)? I don't believe.

Please paste the output of show flash and show uptime commands too (remove any Serial Number or MAC address before posting).

We have SNMP and Web Management both enabled but our VSF (older KB.16.05.0007 version) runs with these values:

VSF# show cpu

5 percent busy, from 300 sec ago
1 sec ave: 2 percent busy
5 sec ave: 4 percent busy
1 min ave: 3 percent busy

Thanks for the quick reply!

The modules CPUs are all almost idle so the high CPU usage is only in the management module (if I understand the difference between show cpu and show cpu slot all)

# show vsf

 VSF Domain ID    : 7
 MAC Address      : xxxxxx-xxxxxx
 VSF Topology     : Chain
 VSF Status       : Active
 Uptime           : 23d 20h 24m
 VSF MAD          : OOBM
 VSF Port Speed   : 40G
 Software Version : KB.16.10.0007

 Mbr
 ID  MAC Address       Model                                 Pri Status
 --- ----------------- ------------------------------------- --- ---------------
  1  xxxxxx-xxxxxx     HP J9850A Switch 5406Rzl2             128 Commander
  2  xxxxxx-xxxxxx     HP J9850A Switch 5406Rzl2             128 Standby

# show flash
Image             Size (bytes) Date     Version
----------------- ------------ -------- --------------
Primary Image    :    34237409 04/17/20 KB.16.10.0007
Secondary Image  :    33773635 11/19/18 KB.16.07.0003

Boot ROM Version
----------------
Primary Boot ROM Version   : KB.16.01.0009
Secondary Boot ROM Version : KB.16.01.0009

Default Boot Image   : Primary
Default Boot ROM     : Primary

# show uptime
 VSF Virtual Chassis
 23d 20h 27m

 VSF Member 1
 23d 20h 27m

 VSF Member 2
 6d 17h 49m

The difference in uptime is because member 1 was created on the side, everything was switched to member 1 after which member 2 (which was the device we already had) was updated, reset and added into the VSF.

What's about VSF OOBM MAD settings? the show oobm ip vsf member 1 (and show oobm ip vsf member 2) is the command to use.

# show oobm ip vsf member 1

  IPv4 Status          : Enabled
  IPv4 Default Gateway : x.x.x.1

            |                                     Address  Interface
 VSF-member | IP Config IP Address/Prefix Length  Status   Status
 ---------- + --------- ------------------------- -------- ---------
 1          | manual    x.x.x.75/24             Active   Up

# show oobm ip vsf member 2

  IPv4 Status          : Enabled
  IPv4 Default Gateway : x.x.x.1

            |                                     Address  Interface
 VSF-member | IP Config IP Address/Prefix Length  Status   Status
 ---------- + --------- ------------------------- -------- ---------
 2          | manual    x.x.x.77/24             Active   Up

# show running-config oobm

Running configuration:

oobm
   ip address x.x.x.75 255.255.255.0
   ip default-gateway x.x.x.1
   ipv6 enable
   ipv6 address dhcp full
   vsf member 1
      ip address x.x.x.76 255.255.255.0
      ip default-gateway x.x.x.1
      exit
   vsf member 2
      ip address x.x.x.77 255.255.255.0
      ip default-gateway x.x.x.1
      exit
   exit

There is also a vlan interface on the oobm subnet just in case the oobm switch goes down.

@parnassus (or someone else) any insights?

Hi! according to this KB Article settings the OoBM IP Address explicitly for VSF Member 1 is not necessary if you already assigned a global OoBM IP Address. I don't know if (re)configuring as per KB will have a beneficial effect or not in relation to your issue.

Another two things:

1. VSF Members priorities: VSF Commander and VSF Standby priorities are equals...best practices suggests to set VSF Commander with an higher priority.
2. Perform checks related to HTTP/HTTPS Web Management and related processes (as example mine VSF has HTTP access Enabled but disabling both HTTP and HTTPS should help, isn't it? I also disabled activate provision and software-update features):

VSF# show web-management 

 Web Management - Server Configuration

  HTTP Access    : Enabled
  HTTPS Access   : Disabled
  Idle Timeout   : 600 seconds
  Management URL : http://h17007.www1.hpe.com/device_help
  Support URL    : http://www.arubanetworks.com/products/networking/
  User Interface : Improved
  Listen Mode    : both

VSF# show activate provision 

 Configuration and Status - Activate Provision Service

  Activate Provision Service   : Disabled
  Activate Server Address      : device.arubanetworks.com
  Activation Key               : Not Available

VSF# show activate software-update 

 Configuration and Status - Activate Software Update

  Activate Server Address      : device.arubanetworks.com
  Activate Server Polling      : Disabled
  Installed Software Version   : KB.16.05.0007
  Server Software Version      : Not available - polling disabled.
  Server Software Image URL    : Not available - polling disabled.

Hey just pinging back the case resolution.

We ended up contacting support and a newer firmware (KB.16.01.0009) had been released, after upgrading to it the CPU issue was resolved.