Wireless Access

My 7210 is only a few weeks old but I've had a couple issues lately with it going offline.  When it goes offline I can't ping it, ssh to it, or connect via the Web GUI.  However, it is accessible via console cable.  I have no errors on the uplink switchport.

 

I've got a ticket open and they are looking at a couple crash.tar files from me.  I'm just curious if anyone else is having the same issue.

 

I'm running 6.2.0.1

 

 

#7210

mmm...seems like a full fpapps process fail.

 

Did u looked on your cpu % ?

show cpuload

show memory

show processes

 

Are u using the Controller as a DHCP server ? if yes ..to how much leases?

#7210

I'm not using the controller as a DHCP server

#7210

ok :) so it's not a DHCP issue :)

so please run the 4 commands above via CLI - and write here your output.

 

#7210

(TheDude) # show storage
Filesystem                Size      Used Available Use% Mounted on
none                      2.0G     16.8M      2.0G   1% /tmp
/dev/usb/flash3           1.5G    112.2M      1.3G   8% /flash

(TheDude) #show memory

Memory (Kb): total: 5716096, used: 1012544, free: 4703552

(TheDude) #show cpuload

user 0.2%, system 0.4%, idle 99.4%

(TheDude) #
(TheDude) #show processes

%CPU S   PID  PPID    VSZ   RSS F  NI START     TIME      EIP CMD
 0.2 S     1     0   5440  2432 4   0 14:01 00:00:10 2ac6419c init      
 0.0 S     2     0      0     0 1   0 14:01 00:00:00 00000000 [kthreadd]
 0.0 S     3     2      0     0 1   - 14:01 00:00:00 00000000 [migration/0]
 0.0 S     4     2      0     0 1   0 14:01 00:00:00 00000000 [ksoftirqd/0]
 0.0 S     5     2      0     0 5   - 14:01 00:00:00 00000000 [watchdog/0]
 0.0 S     6     2      0     0 1   - 14:01 00:00:00 00000000 [migration/1]
 0.0 S     7     2      0     0 1   0 14:01 00:00:00 00000000 [ksoftirqd/1]
 0.0 S     8     2      0     0 5   - 14:01 00:00:00 00000000 [watchdog/1]
 0.0 S     9     2      0     0 1   - 14:01 00:00:00 00000000 [migration/2]
 0.0 S    10     2      0     0 1   0 14:01 00:00:00 00000000 [ksoftirqd/2]
 0.0 S    11     2      0     0 5   - 14:01 00:00:00 00000000 [watchdog/2]
 0.0 S    12     2      0     0 1   - 14:01 00:00:00 00000000 [migration/3]
 0.0 S    13     2      0     0 1   0 14:01 00:00:00 00000000 [ksoftirqd/3]
 0.0 S    14     2      0     0 5   - 14:01 00:00:00 00000000 [watchdog/3]
 0.0 S    15     2      0     0 1   - 14:01 00:00:00 00000000 [migration/4]
 0.0 S    16     2      0     0 1   0 14:01 00:00:00 00000000 [ksoftirqd/4]
 0.0 S    17     2      0     0 5   - 14:01 00:00:00 00000000 [watchdog/4]
 0.0 S    18     2      0     0 1   - 14:01 00:00:00 00000000 [migration/5]
 0.0 S    19     2      0     0 1   0 14:01 00:00:00 00000000 [ksoftirqd/5]
 0.0 S    20     2      0     0 5   - 14:01 00:00:00 00000000 [watchdog/5]
 0.0 S    21     2      0     0 1   - 14:01 00:00:00 00000000 [migration/6]
 0.0 S    22     2      0     0 1   0 14:01 00:00:00 00000000 [ksoftirqd/6]
 0.0 S    23     2      0     0 5   - 14:01 00:00:00 00000000 [watchdog/6]
 0.0 S    24     2      0     0 1   - 14:01 00:00:00 00000000 [migration/7]
 0.0 S    25     2      0     0 1   0 14:01 00:00:00 00000000 [ksoftirqd/7]
 0.0 S    26     2      0     0 5   - 14:01 00:00:00 00000000 [watchdog/7]
 0.0 S    27     2      0     0 1   0 14:01 00:00:00 00000000 [events/0]
 0.0 S    28     2      0     0 1   0 14:01 00:00:00 00000000 [events/1]
 0.0 S    29     2      0     0 1   0 14:01 00:00:00 00000000 [events/2]
 0.0 S    30     2      0     0 1   0 14:01 00:00:00 00000000 [events/3]
 0.0 S    31     2      0     0 1   0 14:01 00:00:00 00000000 [events/4]
 0.0 S    32     2      0     0 1   0 14:01 00:00:00 00000000 [events/5]
 0.0 S    33     2      0     0 1   0 14:01 00:00:00 00000000 [events/6]
 0.0 S    34     2      0     0 1   0 14:01 00:00:00 00000000 [events/7]
 0.0 S    35     2      0     0 1   0 14:01 00:00:00 00000000 [khelper]
 0.0 S    38     2      0     0 1   0 14:01 00:00:00 00000000 [async/mgr]
 0.0 S   116     2      0     0 1   0 14:01 00:00:00 00000000 [sync_supers]
 0.0 S   118     2      0     0 1   0 14:01 00:00:00 00000000 [bdi-default]
 0.0 S   120     2      0     0 1   0 14:01 00:00:00 00000000 [kblockd/0]
 0.0 S   121     2      0     0 1   0 14:01 00:00:00 00000000 [kblockd/1]
 0.0 S   122     2      0     0 1   0 14:01 00:00:00 00000000 [kblockd/2]
 0.0 S   123     2      0     0 1   0 14:01 00:00:00 00000000 [kblockd/3]
 0.0 S   124     2      0     0 1   0 14:01 00:00:00 00000000 [kblockd/4]
 0.0 S   125     2      0     0 1   0 14:01 00:00:00 00000000 [kblockd/5]
 0.0 S   126     2      0     0 1   0 14:01 00:00:00 00000000 [kblockd/6]
 0.0 S   127     2      0     0 1   0 14:01 00:00:00 00000000 [kblockd/7]
 0.0 S   137     2      0     0 1   0 14:01 00:00:00 00000000 [khubd]
 0.0 S   195     2      0     0 1   0 14:01 00:00:00 00000000 [rpciod/0]
 0.0 S   196     2      0     0 1   0 14:01 00:00:00 00000000 [rpciod/1]
 0.0 S   197     2      0     0 1   0 14:01 00:00:00 00000000 [rpciod/2]
 0.0 S   198     2      0     0 1   0 14:01 00:00:00 00000000 [rpciod/3]
 0.0 S   199     2      0     0 1   0 14:01 00:00:00 00000000 [rpciod/4]
 0.0 S   200     2      0     0 1   0 14:01 00:00:00 00000000 [rpciod/5]
 0.0 S   201     2      0     0 1   0 14:01 00:00:00 00000000 [rpciod/6]
 0.0 S   202     2      0     0 1   0 14:01 00:00:00 00000000 [rpciod/7]
 0.0 S   215     2      0     0 1   0 14:01 00:00:00 00000000 [khungtaskd]
 0.0 S   216     2      0     0 1   0 14:01 00:00:00 00000000 [kswapd0]
 0.0 S   217     2      0     0 1   0 14:01 00:00:00 00000000 [aio/0]
 0.0 S   218     2      0     0 1   0 14:01 00:00:00 00000000 [aio/1]
 0.0 S   219     2      0     0 1   0 14:01 00:00:00 00000000 [aio/2]
 0.0 S   220     2      0     0 1   0 14:01 00:00:00 00000000 [aio/3]
 0.0 S   221     2      0     0 1   0 14:01 00:00:00 00000000 [aio/4]
 0.0 S   222     2      0     0 1   0 14:01 00:00:00 00000000 [aio/5]
 0.0 S   223     2      0     0 1   0 14:01 00:00:00 00000000 [aio/6]
 0.0 S   224     2      0     0 1   0 14:01 00:00:00 00000000 [aio/7]
 0.0 S   225     2      0     0 1   0 14:01 00:00:00 00000000 [nfsiod]
 0.0 S   226     2      0     0 1   0 14:01 00:00:00 00000000 [crypto/0]
 0.0 S   227     2      0     0 1   0 14:01 00:00:00 00000000 [crypto/1]
 0.0 S   228     2      0     0 1   0 14:01 00:00:00 00000000 [crypto/2]
 0.0 S   229     2      0     0 1   0 14:01 00:00:00 00000000 [crypto/3]
 0.0 S   230     2      0     0 1   0 14:01 00:00:00 00000000 [crypto/4]
 0.0 S   231     2      0     0 1   0 14:01 00:00:00 00000000 [crypto/5]
 0.0 S   232     2      0     0 1   0 14:01 00:00:00 00000000 [crypto/6]
 0.0 S   233     2      0     0 1   0 14:01 00:00:00 00000000 [crypto/7]
 0.0 S   819     2      0     0 1   0 14:01 00:00:00 00000000 [mtdblockd]
 0.0 S  1020     2      0     0 1   0 14:01 00:00:00 00000000 [scsi_eh_0]
 0.0 S  1021     2      0     0 1   0 14:01 00:00:00 00000000 [usb-storage]
 0.0 S  1068     1   3584  1600 4   - 14:01 00:00:00 2abf4c38 /mswitch/bin/watchdog enable 0
 0.0 S  1071     1   3584  1536 4   - 14:01 00:00:00 2abf4c38 /mswitch/bin/watchdog enable 1
 0.0 S  1074     1   3584  1536 4   - 14:01 00:00:00 2abf4c38 /mswitch/bin/watchdog enable 2
 0.0 S  1077     1   3584  1536 4   - 14:01 00:00:00 2abf4c38 /mswitch/bin/watchdog enable 3
 0.0 S  1080     1   3584  1536 4   - 14:01 00:00:00 2abf4c38 /mswitch/bin/watchdog enable 4
 0.0 S  1083     1   3584  1536 4   - 14:01 00:00:00 2abf4c38 /mswitch/bin/watchdog enable 5
 0.0 S  1086     1   3584  1536 4   - 14:01 00:00:00 2abf4c38 /mswitch/bin/watchdog enable 6
 0.0 S  1089     1   3584  1536 4   - 14:01 00:00:00 2abf4c38 /mswitch/bin/watchdog enable 7
 0.0 S  1104     1   3456  1856 5  -4 14:01 00:00:01 2abca448 /sbin/udevd -d
 0.0 S  2664     2      0     0 1   0 14:01 00:00:00 00000000 [kjournald]
 0.0 S  2763     1   5440  2432 4   0 14:01 00:00:00 2ac64360 /bin/sh /mswitch/bin/syslogd_start
 0.0 S  2767  2763   6272  4736 4   0 14:01 00:00:00 2ac9a448 /mswitch/bin/syslogd -x -r -n -m 0 -f /mswitch/conf/syslog.conf
 0.0 S  3203     1   7488  4672 4   0 14:02 00:00:01 2af2a448 /mswitch/bin/nanny /mswitch/bin/nanny_list 0
 0.0 S  3212  3203   5440  2432 4   0 14:02 00:00:00 2ac64360 /bin/sh /mswitch/bin/dbstart
 0.3 S  3244  3203  43520 38528 4   0 14:02 00:00:12 2b27a448 /mswitch/bin/arci-cli-helper
 0.4 S  3245  3203  43712 38528 4   0 14:02 00:00:14 2b041e88 /mswitch/bin/fpcli
 0.0 S  3249  3203   7680  4736 0   0 14:02 00:00:00 2aefa448 /mswitch/bin/packet_filter
 0.0 S  3250  3203   9984  6720 0   0 14:02 00:00:01 2b06a448 /mswitch/bin/certmgr
 0.0 S  3251  3203   5056  3392 4   0 14:02 00:00:00 2acba448 /mswitch/bin/cryptoPOST
 0.0 S  3252  3203   4864  3136 4   0 14:02 00:00:00 2ac9a448 /mswitch/bin/sbConsoled 0
 0.0 S  3253  3203   5120  3456 0   0 14:02 00:00:00 2acda448 /mswitch/bin/pubsub
 0.3 S  3254  3203  45184 39872 0   0 14:02 00:00:13 2b29a448 /mswitch/bin/cfgm
 0.0 S  3255  3203   8064  5120 4   0 14:02 00:00:00 2af1a448 /mswitch/bin/syslogdwrap
 0.0 S  3256  3203  11136  6720 4   0 14:02 00:00:01 2b1ca448 /mswitch/bin/aaa
 0.0 S  3258  3212   5440  2432 4   0 14:02 00:00:00 2ac64360 /bin/sh /mswitch/mysql/bin/safe_mysqld --pid-file=/var/mysql/mysql.pid
 0.7 S  3287  3258 149760  8128 0   0 14:02 00:00:26 2ae8a4ac /mswitch/mysql/libexec/mysqld --basedir=/mswitch/mysql --datadir=/var/mysql --big-tables --pid-file=/var/mysql/mysql.pid
 0.9 S  3293  3203 322048 152000 0  0 14:02 00:00:32 2b02bd8c /mswitch/bin/fpapps
 0.0 S  3296  3203   8896  5824 4   0 14:02 00:00:00 2af1a448 /mswitch/bin/pim
 0.0 S  3297  3203  11072  6592 0   0 14:02 00:00:00 2b1aa448 /mswitch/bin/licensemgr
 0.0 S  3306  3203  25280 20608 4   0 14:02 00:00:02 2b26a448 /mswitch/bin/isakmpd
 0.6 S  3309  3203  45184 16064 4   0 14:02 00:00:21 2b1fa4ac /mswitch/bin/wms -l 5
 0.0 S  3327  3203  12480  7808 0   0 14:02 00:00:02 2b24a448 /mswitch/bin/profmgr
 0.0 S  3392  3203  10112  5376 0   0 14:02 00:00:00 2b15a448 /mswitch/bin/rfm
 0.4 S  3426  3203  26112 15616 4   0 14:02 00:00:13 2b34a448 /mswitch/bin/auth
 1.0 S  3427  3203  76928 23744 4   0 14:02 00:00:36 2b33a4ac /mswitch/bin/stm
 0.0 S  3429  3203   6144  4160 4   0 14:02 00:00:01 2adaa448 /mswitch/bin/rtpa
 0.0 S  3432  3203  11136  6656 0   0 14:02 00:00:02 2b1aa448 /mswitch/bin/udbserver
 0.0 S  3435  3203   7936  5440 4   0 14:02 00:00:02 2ae6a448 /mswitch/bin/dhcpdwrap
 0.0 S  3440  3203   6144  3968 0   0 14:02 00:00:00 2ad4a448 /mswitch/bin/radvdwrap
 0.0 S  3442  3203  10752  6336 0   0 14:02 00:00:00 2b11a448 /mswitch/bin/mobileip
 0.0 S  3443  3203  10368  5952 4   0 14:02 00:00:00 2b12a448 /mswitch/bin/phonehome
 0.4 S  3444  3203  10688  6528 4   0 14:02 00:00:13 2b17a448 /mswitch/bin/hwMon
 0.0 S  3445  3203   5120  3264 4   0 14:02 00:00:00 2ac64c38 /mswitch/bin/sbHeartbeat -t 1 -m 10
 0.2 S  3446  3203  12672  8064 4   0 14:02 00:00:08 2b15a448 /mswitch/bin/snmpd
 0.0 S  3447  3203  12736  8576 4   0 14:02 00:00:02 2b15a448 /mswitch/bin/trapd
 0.0 S  3448  3203   8448  5184 4   0 14:02 00:00:00 2af8a448 /mswitch/bin/ntpwrap
 0.0 S  3451  3203   9984  5760 0   0 14:02 00:00:00 2af2a448 /mswitch/bin/dbsync
 0.0 S  3459  3203  10496  6080 4   0 14:02 00:00:01 2b13a448 /mswitch/bin/slb
 0.0 S  3460  3203   5440  2496 4   0 14:02 00:00:00 2aca4220 /sbin/klogd -n
 0.0 S  3461  3203   5312  3648 4   0 14:02 00:00:01 2ad2a448 /mswitch/bin/resolvwrap
 0.0 S  3462  3203   8512  6336 4   0 14:02 00:00:00 2af0a448 /etc/ssh/sshd -D -f /etc/ssh/sshd_config
 0.0 S  3463  3203  10944  6208 4   0 14:02 00:00:00 2b1aa448 /mswitch/bin/cts
 0.0 S  3466  3203   5504  3328 0   0 14:02 00:00:00 2ace4c38 /mswitch/bin/dbbk periodic-backup 10800
 0.0 S  3467  3203   7808  4672 0   0 14:02 00:00:00 2aefa448 /mswitch/bin/qpdq
 0.0 S  3468  3203   8320  5184 4   0 14:02 00:00:00 2af6a448 /mswitch/bin/httpd_wrap
 0.0 S  3471  3203  19712  8512 4   0 14:02 00:00:00 2af5a448 /mswitch/bin/l2tpd -D
 0.0 S  3472  3203  17280  6144 4   0 14:02 00:00:00 2af7a448 /mswitch/bin/pptpd -f
 0.0 S  3476  3203  16320  7808 0   0 14:02 00:00:00 2b21a448 /mswitch/bin/fw_visibility
 0.0 S  3487  3203  10432  6016 4   0 14:02 00:00:00 2b0fa448 /mswitch/bin/ospf
 0.0 S  3488  3203   4096  2560 4   0 14:02 00:00:00 2abca448 /usr/sbin/dnsmasq -d -o
 0.0 S  3489  3203   7680  4544 0   0 14:02 00:00:01 2aefa448 /mswitch/bin/util_proc
 0.0 S  3490  3203  10240  5888 0   0 14:02 00:00:01 2af5a448 /mswitch/bin/cpsec
 0.0 S  3491  3203   5440  2496 4   0 14:02 00:00:00 2ac64360 /bin/sh /usr/sbin/mem_mon -v -t 5
 0.0 S  3501  3203  11200  7424 0   0 14:02 00:00:01 2b1ea448 /mswitch/bin/spectrum
 0.0 S  3502  3203   6528  4480 4   0 14:02 00:00:00 2adea448 /mswitch/bin/iapmgr
 0.0 S  3554  3448   8512  8512 4   0 14:02 00:00:00 2af8a448 /mswitch/bin/ntpd -g -L
 0.0 S  3585  3468  16640 11648 4   0 14:03 00:00:01 2b4ca448 /mswitch/apache/bin/httpd -DFOREGROUND -DSSL
 0.0 S  7160  3491   5248  1792 4   0 14:57 00:00:00 2ac64c38 sleep 300             
 0.3 S  7169  3462   8768  6720 4   0 14:57 00:00:00 2af0a448 sshd: admin@pts/0                          
 0.0 S  7174  7169   3392  1344 4   0 14:57 00:00:00 2abca448 -sshwrap
 0.0 S  7190  3212   5248  1792 4   0 14:58 00:00:00 2ac64c38 sleep 10             
 0.0 R  7191  3245   4736  2944 4   0 14:58 00:00:00 2abff2b8 ps --user nobody -N -wo %cpu,state,pid,ppid,vsz,rss,flags,nice,start_time,time,eip,cmd

#7210

nothing that jumping into my eyes...

• are u running Debug on clients maybe??
• are u using captive portal on the controller?
• when this problem happning - is after a few minutes the controller is back up..or u should restart it manully?
• this controller is connected to an Airwave server - if yes,what is the AirWave version.
  

 

 

please supply me answers...i'm trying to point trouble your issue. (even due it's a new HW and also new AOS)

 

 

#7210

I can't find it now, but I thought I saw an error message during one of the outage times that said something like "Layer2/3 suspended" or something like that.

 

 

Thanks for looking at it. 

#7210

I do have it connected to a AirWave, but the airwave looses connectivity as well.  I dont really know where to look for any other info on there.  Let me know if there is a place to find it.

 

I'm not running a captive portal.  I am pretty basic in terms of setup.  I only have 2 SSIDs - one is open and the other is 802.1x pointing to a NPS server.

 

I dont think I am running debug on any clients. 
And the box will not come back online.  I hadn't added this to my network monitor and it went down over the weekend and didnt come back up until I saw it monday morning and rebooted it.

 

 

#7210

Try to disconnect (just for test it)  the controller from AIRWAVE (disable the SNMP connectivity between the two) **it might be the Airwave making your controller process to carsh**

if the problem stopped so - the issue is between the AirWave version to your AOS version

 

(Check it! - it's important)

and update me.

 

#7210

I will give that a try if it happens again.  Thanks for looking at it.

#7210

 

 

ok - I'm waiting to hear! (hope it will stop your SSH/GUI/interface ports crash)

and from there we will continue, to understand if the Airwave was the source and what exactly.

 

 

#7210

Try upgrading it to the verion 6.2.0.2

it could be a bug...

Also you should upgrade it just in case of this:  If you read the release notes of 6.2.0.2

 

 

Symptom: An internal controller module stopped responding, corrupting data on the controller and

causing the controller to become unusable. The only resolution is to replace the controller. This

issue is resolved in ArubaOS 6.2.0.2.

Scenario: This issue was observed on M3 and 3000 Series controllers running ArubaOS 6.2.0.0

and 6.2.0.1."

 

6.2.0.2 fixes this... just in case you should upgrade it... as like it says it just have been observed on M3 and 3000  for now... but still...

 

When werd things happening without no  explanation i always try to upgrade it and it solve the issue like 90% of the times in my case.  

 

Cheers

Carlos

#7210

This is an A7210 and is not affected by this bug.

 

This bug (77230) is platform dependant, honestly it only affects M3 and A3x00 controllers and not A72x0 controllers.

 

Cheers

 

Giles

#7210

Giles,

 

I'm not sure what you are saying, but I have a 7210 and the bug with the AMSDU exists on my controller.

 

Quote from TAC "As an update the bug has been fixed, the fix will be available in 6.2.0.3 which is yet to be released."

 

 

#7210

I think my reply got crossed with the other post.

I've checked your ticket and eng report your issue is bug 74048 which is fixed in AOS 6.2.0.3 due for release shortly.

 

I was referring to the other post where the suggestion was to upgrade to AOS 6.2.0.2, as that was not related to your issue.

 

Sorry for the confusion.

 

#7210

No problem. 

#7210

If i assist u some how - Please K+ (the star+ button)  :)  me, thanks.

 

If u would like me to dig into it more... Answer the questions i asked in the  previous post.

 

Regards.

 

Me

#7210

Hello we have a ARUBA 6000 Chase with 2 cards and for some reason the card 0 reboots and this is happening every day? sodtware version of the card 5.20.2.1
#7210

Andy,

 

Please open a case with support.

#7210

I hope TAC are helping you out.

 

If possible if it goes off line again I would issue 'tar logs tech-support' from the CLI, this will store a file called logs.tar on the flash, after rebooting the controller to regain service, copy the file off and send to TAC.

The file will include the 'show cpu' etc.

 

I would not think this is caused by AirWave, if the controller's SNMP process has issues or the CLI - those processes would re-start and would have no affect on the IP stack of the controller.

 

Something else is happening here which we will get to the bottom of, I'm sure the crash files will be assisting TAC to ge to the root cause.

 

 

 

#7210

Turns out I found a bug in the OS. 

 

They had me disable the AMSDU setting in the Stateful Firewall configs and everything has been great.

 

I've been told that this bug will be fixed in the upcoming release.

#7210

hehe i knew it was a bug! ;P

#7210