Wired Intelligent Edge

Hey

 

I have had several TAC cases with Aruba about laggy SSH sessions to AOS swithes. There are never any solutions, and it seems like they are not able to understand why it happens. I use SSH for management of Cisco, aruba, linux and procurve and this only happens to units with ArubaOS. 

 

Anyone else seen this ?

What laggy behavior do you see exactly? Is it a constant latency between typing a letter and seeing it back on screen? Is it that the session hangs for seconds every now and then (like there is no IP connectivity for short time)? Is it that commands, when entered, take time to process output? Is it on specific commands or large outputs?

 

This kind of issues typically has to do with something in the network like packet loss, MTU path problems or flapping interfaces (though it has been a while that I last saw network interfaces flapping).

 

If Aruba Support doesn't provide you a satisfactory solution, feel free to escalate your issue or involve your local Aruba representatives.

 

 

To me it would seem like a drop in connectivity. But as far as I can see everything works ok, except the SSH to Aruba switches. It both hangs and has slow text input. 

 

Since this only occur on ArubaOS units and for example not on a older provision unit or Cisco unit in the same rack, Im kind of stuck. 

 

 

perhaps make a wireshark capture of both a constant ping to the switch and your ssh session setting up, typing a bit until the problem is known to be observed, then exit and stop the ping.

 

I think there are tools online you can anonymise/change the IPs inside the pcap if thats an issue.  The capture might shed some light on where the lag is coming from (or at least conclusively prove it's the switch switch, which gives you a bigger stick with support).

 

I can confirm that laggy behaviour. I tried different versions of ArubaOS but this issue seems to relate to any YA16.xx firmware.

Ocassionally the reaction on inputs over ssh is very slow and hangs sometimes for a second. So I would say it is more the "constant latency between typing a letter and seeing it back on screen" thing Herman mentioned. Sometimes it feels as you have to type the input twice to be recognized.

Are you using central authentication like TACACS?

In fact, I am using tacacs for ssh authentication:

 

aaa authentication ssh login tacacs
aaa authentication ssh enable tacacs

I already tried with local ssh authentication only, the issue is still the same.

This was a bug which is now fixed in the 16.08.0003 release. We experienced the same and created a case for this.

Authentication
CR_0000248908
Symptom/Scenario: When attempting to SSH to a switch where aaa authentication ssh login tacacs is configured and the TACACS server is reachable, the switch experiences a one second delay between the TACACS TCP handshake and the actual authentication request

I have seen that in the release notes and tried the 16.08.0003 release as well. But this laggy behaviour occurs after the tacacs login, even on this latest release. While configuring or troubleshooting the reaction on every typed character is sometimes incredible slow (the delay between typing and seeing the character typed in the ssh windows). My guess is that the ssh process is way under-priorized.

What we have seen is a delay in the TACACS authorization request. Every time you type a command it is validated by the TACACS server.
I haven't test the new release so I'm not sure if it is fixed.

I just got some time to investigate on this behaviour further. Even with a blank configuration there are delays in the ssh session from time to time. The full config:

; J9774A Configuration Editor; Created on release #YA.16.08.0003
; Ver #14:01.44.00.04.19.02.13.98.82.34.61.18.28.f3.84.9c.63.ff.37.27:45
hostname "HP-2530-8G-PoEP"
snmp-server community "public" unrestricted
vlan 1
   name "DEFAULT_VLAN"
   untagged 1-10
   ip address dhcp-bootp
   exit

Is nobody else facing this issue?

 

 

I touch rarely our Aruba 2530...so I didn't notice the laggy SSH...but it's laggy...both on YA.16.08.0002 and on YA.16.08.0003 (just updated to test).

 

I see the CPU flying high...up to 45% (with a 34% average on 1 minute) to then decrease to 13/15% Busy (with an 14% average on 1 minute).

 

I also did a process-tracking...

 

HP J9776A 2530-24G Switch
Software revision YA.16.08.0003

DL1-103# show cpu

26 percent busy, from 300 sec ago
1 sec ave: 47 percent busy
5 sec ave: 49 percent busy
1 min ave: 34 percent busy

DL1-103# show cpu

43 percent busy, from 4 sec ago
1 sec ave: 40 percent busy
5 sec ave: 44 percent busy
1 min ave: 34 percent busy

DL1-103# show cpu

40 percent busy, from 2 sec ago
1 sec ave: 41 percent busy
5 sec ave: 41 percent busy
1 min ave: 34 percent busy

DL1-103# show cpu

43 percent busy, from 4 sec ago
1 sec ave: 44 percent busy
5 sec ave: 42 percent busy
1 min ave: 34 percent busy

DL1-103# show cpu

44 percent busy, from 2 sec ago
1 sec ave: 45 percent busy
5 sec ave: 43 percent busy
1 min ave: 34 percent busy

DL1-103# show cpu

42 percent busy, from 5 sec ago
1 sec ave: 42 percent busy
5 sec ave: 42 percent busy
1 min ave: 34 percent busy

DL1-103# show cpu

42 percent busy, from 2 sec ago
1 sec ave: 42 percent busy
5 sec ave: 41 percent busy
1 min ave: 34 percent busy

DL1-103# show cpu

0 percent busy, from 0 sec ago
1 sec ave: 42 percent busy
5 sec ave: 41 percent busy
1 min ave: 34 percent busy

DL1-103# show cpu

44 percent busy, from 1 sec ago
1 sec ave: 44 percent busy
5 sec ave: 42 percent busy
1 min ave: 34 percent busy

DL1-103# process-tracking
ATTENTION : You are enabling the 'Process-Tracking' capability for the module.
            This can affect the performance of the module.

Do you want to enable Process-Tracking functionality [y/N]? y
Process-tracking started
Process-tracking time interval : 30 seconds

DL1-103# show cpu process
Process tracker state: ACTIVE
Process tracking time: 30 seconds
                                 Total     %    Time Since    Times      Max
  Process Name        Priority   Time     CPU   Last Ran       Ran      Time
--------------------+----------+--------+-----+-----------+----------+-------
   Sessions & I/O-7        146    78 ms     1      775 us        344     5 ms
    Hardware Mgmt-7        193   397 ms     6       35 ms       5703   190 us
             Idle-3          1     1 s     20       39 ms      60733    96 us
             Idle-1        226     2 s     26        3 ms      29467   146 us
 System Services-43          1     1 s     17       14 ms        654     5 ms
             Idle-0        226     1 s     23        3 ms      29513   191 us
Process tracker state: ACTIVE
Process tracking time: 30 seconds
                                 Total     %    Time Since    Times      Max
  Process Name        Priority   Time     CPU   Last Ran       Ran      Time
--------------------+----------+--------+-----+-----------+----------+-------
    Hardware Mgmt-7        193   789 ms     7      152 ms      11323   190 us
             Idle-3          1     2 s     21      227 us     113407    98 us
             Idle-1        226     3 s     27      324 us      54876   333 us
 System Services-43          1     2 s     16       95 ms       1070     5 ms
             Idle-0        226     3 s     24      269 us      54928   203 us
Process tracker state: ACTIVE
Process tracking time: 30 seconds
                                 Total     %    Time Since    Times      Max
  Process Name        Priority   Time     CPU   Last Ran       Ran      Time
--------------------+----------+--------+-----+-----------+----------+-------
    Hardware Mgmt-7        193     1 s      7      299 ms      16358   190 us
             Idle-3          1     3 s     20      314 us     165339   103 us
             Idle-1        226     4 s     27      233 us      80066   333 us
 System Services-43          1     3 s     17       13 ms       1654     5 ms
             Idle-0        226     4 s     24      339 us      80157   203 us
Process tracker state: ACTIVE
Process tracking time: 30 seconds
                                 Total     %    Time Since    Times      Max
  Process Name        Priority   Time     CPU   Last Ran       Ran      Time
--------------------+----------+--------+-----+-----------+----------+-------
    Hardware Mgmt-7        193   878 ms     7      425 ms      12569   201 us
             Idle-3          1     3 s     20       14 ms     127546   322 us
             Idle-1        226     3 s     26        4 ms      61854   333 us
 System Services-43          1     2 s     18      231 us       1399     5 ms
             Idle-0        226     3 s     23        4 ms      61876   201 us
Process tracker state: ACTIVE
Process tracking time: 30 seconds
                                 Total     %    Time Since    Times      Max
  Process Name        Priority   Time     CPU   Last Ran       Ran      Time
--------------------+----------+--------+-----+-----------+----------+-------
    Hardware Mgmt-7        193   658 ms     7        4 ms       9415   201 us
             Idle-3          1     2 s     20      232 us      91654   322 us
             Idle-1        226     2 s     26      321 us      44359   143 us
 System Services-43          1     2 s     17       32 ms        977     5 ms
             Idle-0        226     2 s     24      256 us      44368   180 us
Process tracker state: ACTIVE
Process tracking time: 30 seconds
                                 Total     %    Time Since    Times      Max
  Process Name        Priority   Time     CPU   Last Ran       Ran      Time
--------------------+----------+--------+-----+-----------+----------+-------
    Hardware Mgmt-7        193     1 s      7      111 ms      14443   201 us
             Idle-3          1     3 s     20      349 us     141578   322 us
             Idle-1        226     4 s     26      267 us      68552   165 us
 System Services-43          1     3 s     18       17 ms       1628     5 ms
             Idle-0        226     3 s     23      372 us      68586   200 us
Process tracker state: ACTIVE
Process tracking time: 30 seconds
                                 Total     %    Time Since    Times      Max
  Process Name        Priority   Time     CPU   Last Ran       Ran      Time
--------------------+----------+--------+-----+-----------+----------+-------
    Hardware Mgmt-7        193   747 ms     7      293 ms      10691   199 us
             Idle-3          1     2 s     20      305 us     105253   287 us
             Idle-1        226     3 s     26      317 us      50952   165 us
 System Services-43          1     2 s     19        1 ms       1217     5 ms
             Idle-0        226     2 s     23      232 us      51014   200 us
Process tracker state: ACTIVE
Process tracking time: 30 seconds
                                 Total     %    Time Since    Times      Max
  Process Name        Priority   Time     CPU   Last Ran       Ran      Time
--------------------+----------+--------+-----+-----------+----------+-------
    Hardware Mgmt-7        193     1 s      7      435 ms      15717   199 us
             Idle-3          1     3 s     20      311 us     157962   287 us
             Idle-1        226     4 s     26      230 us      76497   165 us
 System Services-43          1     3 s     18       19 ms       1822     5 ms
             Idle-0        226     4 s     23      335 us      76572   203 us
Process tracker state: ACTIVE
Process tracking time: 30 seconds
                                 Total     %    Time Since    Times      Max
  Process Name        Priority   Time     CPU   Last Ran       Ran      Time
--------------------+----------+--------+-----+-----------+----------+-------
    Hardware Mgmt-7        193   877 ms     7       48 ms      12575   192 us
             Idle-3          1     2 s     21        4 ms     122411   287 us
             Idle-1        226     3 s     26        4 ms      59198   145 us
 System Services-43          1     2 s     17      233 us       1365     5 ms
             Idle-0        226     3 s     24        4 ms      59232   333 us
Process tracker state: ACTIVE
Process tracking time: 30 seconds
                                 Total     %    Time Since    Times      Max
  Process Name        Priority   Time     CPU   Last Ran       Ran      Time
--------------------+----------+--------+-----+-----------+----------+-------
    Hardware Mgmt-7        193     1 s      7      248 ms      17590   192 us
             Idle-3          1     4 s     20        5 ms     172719   287 us
             Idle-1        226     4 s     26      262 us      83557   145 us
 System Services-43          1     3 s     18      363 us       1917     5 ms
             Idle-0        226     4 s     23      222 us      83625   333 us
Process tracker state: ACTIVE
Process tracking time: 30 seconds
                                 Total     %    Time Since    Times      Max
  Process Name        Priority   Time     CPU   Last Ran       Ran      Time
--------------------+----------+--------+-----+-----------+----------+-------
    Hardware Mgmt-7        193   965 ms     7      383 ms      13815   210 us
             Idle-3          1     3 s     20      361 us     134549   114 us
             Idle-1        226     4 s     26      395 us      65074   341 us
 System Services-43          1     3 s     19        7 ms       1502     5 ms
             Idle-0        226     3 s     23      309 us      65180   333 us
Process tracker state: ACTIVE
Process tracking time: 30 seconds
                                 Total     %    Time Since    Times      Max
  Process Name        Priority   Time     CPU   Last Ran       Ran      Time
--------------------+----------+--------+-----+-----------+----------+-------
    Hardware Mgmt-7        193   726 ms     7      117 us      10400   210 us
             Idle-3          1     2 s     20        4 ms      97937   304 us
             Idle-1        226     3 s     26        4 ms      47313   341 us
 System Services-43          1     2 s     19       11 ms       1204     5 ms
             Idle-0        226     2 s     23        4 ms      47379   193 us
Process tracker state: ACTIVE
Process tracking time: 30 seconds
                                 Total     %    Time Since    Times      Max
  Process Name        Priority   Time     CPU   Last Ran       Ran      Time
--------------------+----------+--------+-----+-----------+----------+-------
    Hardware Mgmt-7        193     1 s      7       40 ms      15703   210 us
             Idle-3          1     3 s     20      305 us     146213   304 us
             Idle-1        226     4 s     25      318 us      70638   341 us
 System Services-43          1     3 s     20       79 ms       1904     5 ms
             Idle-0        226     3 s     22      233 us      70758   194 us
Process tracker state: ACTIVE
Process tracking time: 30 seconds
                                 Total     %    Time Since    Times      Max
  Process Name        Priority   Time     CPU   Last Ran       Ran      Time
--------------------+----------+--------+-----+-----------+----------+-------
    Hardware Mgmt-7        193   825 ms     7      119 ms      11814   190 us
             Idle-3          1     2 s     19      305 us     108942   304 us
             Idle-1        226     3 s     25      317 us      52668   147 us
 System Services-43          1     2 s     22       64 ms       1487     5 ms
             Idle-0        226     3 s     22      232 us      52733   196 us
Process tracker state: ACTIVE
Process tracking time: 30 seconds
                                 Total     %    Time Since    Times      Max
  Process Name        Priority   Time     CPU   Last Ran       Ran      Time
--------------------+----------+--------+-----+-----------+----------+-------
    Hardware Mgmt-7        193     1 s      7      213 ms      16833   192 us
             Idle-3          1     3 s     19      313 us     159806   304 us
             Idle-1        226     4 s     25      231 us      77333   147 us
 System Services-43          1     3 s     21       32 ms       2181     5 ms
             Idle-0        226     4 s     22      336 us      77437   198 us
DL1-103#

DL1-103# no process-tracking
Process-tracking stopped

DL1-103# show cpu

29 percent busy, from 127 sec ago
1 sec ave: 24 percent busy
5 sec ave: 29 percent busy
1 min ave: 32 percent busy

DL1-103# show cpu

24 percent busy, from 1 sec ago
1 sec ave: 24 percent busy
5 sec ave: 26 percent busy
1 min ave: 32 percent busy

DL1-103# show cpu

24 percent busy, from 1 sec ago
1 sec ave: 24 percent busy
5 sec ave: 23 percent busy
1 min ave: 32 percent busy

DL1-103# show cpu

17 percent busy, from 1 sec ago
1 sec ave: 17 percent busy
5 sec ave: 22 percent busy
1 min ave: 31 percent busy

DL1-103# show cpu

17 percent busy, from 1 sec ago
1 sec ave: 17 percent busy
5 sec ave: 21 percent busy
1 min ave: 31 percent busy

DL1-103# show cpu

18 percent busy, from 2 sec ago
1 sec ave: 18 percent busy
5 sec ave: 18 percent busy
1 min ave: 31 percent busy

DL1-103# show cpu

15 percent busy, from 9 sec ago
1 sec ave: 11 percent busy
5 sec ave: 13 percent busy
1 min ave: 28 percent busy

DL1-103# show cpu

11 percent busy, from 2 sec ago
1 sec ave: 11 percent busy
5 sec ave: 11 percent busy
1 min ave: 28 percent busy

DL1-103# show cpu

16 percent busy, from 4 sec ago
1 sec ave: 16 percent busy
5 sec ave: 15 percent busy
1 min ave: 27 percent busy

DL1-103# show cpu

16 percent busy, from 1 sec ago
1 sec ave: 16 percent busy
5 sec ave: 16 percent busy
1 min ave: 26 percent busy

DL1-103# show cpu

13 percent busy, from 80 sec ago
1 sec ave: 14 percent busy
5 sec ave: 14 percent busy
1 min ave: 13 percent busy

DL1-103# show cpu

13 percent busy, from 27 sec ago
1 sec ave: 15 percent busy
5 sec ave: 15 percent busy
1 min ave: 14 percent busy

DL1-103# show cpu

7 percent busy, from 160 sec ago
1 sec ave: 48 percent busy
5 sec ave: 49 percent busy
1 min ave: 6 percent busy

What I noticed is that with CPU at 34% on average the prompt was very laggy, with CPU at 14% (or lower) on average the CLI lagginess is difficult to notice (at least typing slowly)...the test I did is typing: show vvvvvvvvvvvvvvvvvvvvvvvvv at some point you can't stop v to be printed at prompt even if I stopped to press the v key on the keyboard 7 or 8 seconds before, the very same doing a deletion of those letters...just to see how fast is the prompt...I would say it prints jerkily (it's not fluid) at prompt.

 

Maybe there is a background process which is slamming the Aruba 2530's CPU.

Great! Sounds like a first hint for this issue. Is there something else I could help with? Will this issue be tracked as a bug and investigated further? For us this is a big thing as we have to manage dozens of the 2530 series devices.

OK, who will open a case on Aruba Support Portal (ASP)? ;-)

I can do it in about two weeks. Aruba Support is always a little time-consuming so I have to take that into account ;-P

I tried troubleshooting this with Aruba support a year ago with no success. They got wireshark dumps and technical information but it seemed like no one understood this problem. Seems strange since it hits diffrent models with 16.04+ fw. 

 

What I have seen is that it hits most models, but not the 2930.