Wireless Access

Hi all, have an old controller running 5x software.  It reloads itself when blacklist gets to 300 entries.

Any ideas ??

Thanks

You should open a support case.  That should not happen.

Colin, indeed, problem is controller out of support, just seeing if anyone had seen the issue.

How is the blacklisting being done?

What version of ArubaOS?

What controller platform?

Colin, its an old 6000 controler (well 2 x 6000), master local, running 5.0.4.13 firmware.  Blacklist clients in the local are showing as 'user defined' but blacklist clients is checked in the VAP profiles.

The local controller has no issues, same box, same firmware.  It has blacklist clients there 200 days etc ....

Type "show version" on the box to get the reboot reason.

up 42 days, last reboot: user reboot

I have seen the 'user reboot' before in other issues where the controller has self-booted

---

@stunshot wrote:

Colin, its an old 6000 controler (well 2 x 6000), master local, running 5.0.4.13 firmware.  Blacklist clients in the local are showing as 'user defined' but blacklist clients is checked in the VAP profiles.

 

The local controller has no issues, same box, same firmware.  It has blacklist clients there 200 days etc ....

---

Stunshot, is it running on SC hardware then in late 5.0.4.x on SC1/SC2 will be very low on memory during run time if the number of APs is high, such as around 100 and over. It's possible it's running out of memory, in which case there are not a lot of options.

Can you show output of "show memory" while this controller is in runtime, can also show "show memory debug" as an attachment too.

regards

-jeff

Jeff, thanks for the reply, yes the memory is very full, show memory gives

Total 256060, used 252069, free 3971

Output of memory debug,

APs and master controller) #show memory debug

memory snapshot: Tue Jul 29 06:30:25 2014

=========================================

Memory (Kb): total: 256060, free: 3571

4104 /mswitch/bin/l2tpd -D

3720 /mswitch/bin/isakmpd

3656 /mswitch/bin/profmgr

2880 /mswitch/bin/udbserver

26676 /mswitch/bin/stm

1928 /mswitch/bin/mobileip

1924 /mswitch/bin/snmpd

18076 /mswitch/bin/fpapps

15064 /mswitch/bin/cfgm

1476 /mswitch/bin/ospf

1452 /mswitch/bin/slb

1372 /mswitch/bin/dbsync

1328 /mswitch/bin/pptpd -f

12968 /mswitch/bin/wms -l 5

1220 /mswitch/bin/pim

10032 /mswitch/bin/auth

Querying excessively large apps

auth     :  9/20M (cur/typical)  0M over

stm      : 26/25M (cur/typical)  1M over

wms      : 12/18M (cur/typical)  0M over

cfgm     : 14/23M (cur/typical)  0M over

fpapps   : 17/20M (cur/typical)  0M over

profmgr  :  3/ 8M (cur/typical)  0M over

isakmpd  :  3/ 6M (cur/typical)  0M over

l2tpd    :  4/ 8M (cur/typical)  0M over

mobileip :  1/ 4M (cur/typical)  0M over

ospf     :  1/ 4M (cur/typical)  0M over

pim      :  1/ 2M (cur/typical)  0M over

pptpd    :  1/ 5M (cur/typical)  0M over

dbsync   :  1/ 3M (cur/typical)  0M over

slb      :  1/ 4M (cur/typical)  0M over

snmpd    :  1/ 5M (cur/typical)  0M over

udbserver:  2/ 4M (cur/typical)  0M over

Large files under /tmp

======================

/tmp/.var/mysql/wms:-rw-r-----    1 root     root      2167547 Jul 29 06:30 probe_sta_table.ISD

/tmp/.var/mysql/wms:-rw-r-----    1 root     root      1299456 Jul 29 06:30 probe_sta_table.ISM

%CPU S   PID  PPID  RSS CMD

 4.0 S 31897   536 26676 stm

 0.0 S   839   679 18076 fpapps

 0.0 S   840   679 18076 fpapps

 0.0 S   842   679 18076 fpapps

 0.0 S   843   679 18076 fpapps

 0.0 S   844   679 18076 fpapps

 0.0 S   841   679 18076 fpapps

 0.0 S   848   679 18076 fpapps

 0.0 S   849   679 18076 fpapps

 0.0 S   850   679 18076 fpapps

 0.0 S   816   679 18076 fpapps

 0.0 S   829   679 18076 fpapps

 0.3 S   954   679 18076 fpapps

 0.0 S   838   679 18076 fpapps

 0.0 S   679   678 18076 fpapps

 0.0 S   845   679 18076 fpapps

 0.0 S   756   679 18076 fpapps

 0.0 S   678   536 18076 fpapps

 0.0 S   815   679 18076 fpapps

 0.0 S   680   679 18076 fpapps

 0.0 S   814   679 18076 fpapps

 0.0 S   837   679 18076 fpapps

 0.1 S   809   679 18076 fpapps

 0.0 S   672   536 15064 cfgm

 0.0 S   552   536 14888 fpcli

 2.9 S   702   536 12968 wms -l 5

 0.1 S   728   585 10520 mysql/libexec/mysqld

 0.0 S   675   585 10520 mysql/libexec/mysqld

 0.0 S 16898   585 10520 mysql/libexec/mysqld

 0.0 S   888   585 10520 mysql/libexec/mysqld

 0.0 S  1014   585 10520 mysql/libexec/mysqld

 0.0 S   585   584 10520 mysql/libexec/mysqld

 0.2 S 32028   585 10520 mysql/libexec/mysqld

 2.3 S   712   585 10520 mysql/libexec/mysqld

 0.0 S   586   585 10520 mysql/libexec/mysqld

 0.0 S   584   559 10520 mysql/libexec/mysqld

 0.0 S   692   585 10520 mysql/libexec/mysqld

 0.0 S   725   585 10520 mysql/libexec/mysqld

 0.0 S   538   536 10212 arci-cli-helper

 0.5 S   715   536 10032 auth

 0.0 S   699   536 4104 l2tpd -D

 0.0 S  1036   771 3804 apache/bin/httpd -DFOR

 0.0 S   701   536 3720 isakmpd

 0.0 S   710   536 3656 profmgr

 0.0 S   724   536 2880 udbserver

 0.2 S   736   536 2604 trapd

 0.0 S   985   737 2588 ntpd -g -L

 0.0 S  1006   726 2152 dhcpd -f -lf /tmp/dhcpd.lease e

 0.0 S  1008  1007 2152 dhcpd -f -lf /tmp/dhcpd.lease e

 0.0 S  1007  1006 2152 dhcpd -f -lf /tmp/dhcpd.lease e

 0.0 S   727   536 1928 mobileip

 0.0 S   735   536 1924 snmpd

 0.0 S   691   536 1732 licensemgr

 0.4 S 20227   761 1568 /etc/ssh/sshd: admin@pts/0

 0.0 S   778   536 1476 ospf

 0.0 S   745   536 1452 slb

 0.0 S   764   536 1384 cts

 0.0 S   738   536 1372 dbsync

 0.0 S   700   536 1328 pptpd -f

 0.0 S   786   536 1292 cpsec

 0.0 S   674   536 1260 aaa

 0.0 S   555   536 1220 certmgr

 0.0 S   681   536 1220 pim

 0.0 S   761   536 1184 /etc/ssh/sshd -D -f /etc/ssh/ss

 0.0 S   713   536 1160 rfm

 0.0 S   673   536 1144 syslogdwrap

 0.0 S   726   536 1120 dhcpdwrap

 0.0 S   536     1 1084 nanny bin

 0.0 S   737   536 1028 ntpwrap

 0.0 S   554   536  972 packet_filter

 0.0 S   746   536  956 ads

 0.2 S   729   536  948 hwMon

 0.0 S   773   536  944 msghh

 0.0 S   686   536  928 p8mgr

Filesystem           1k-blocks      Used Available Use% Mounted on

/dev/root                71957     68212      3745  95% /

none                     71680      9108     62572  13% /tmp

/dev/hda3               153271     57531     87826  40% /flash

        total:    used:    free:  shared: buffers:  cached:

Mem:  262205440 259448832  2756608        0  2834432 101826560

Swap:        0        0        0

MemTotal:       256060 kB

MemFree:          2692 kB

MemShared:           0 kB

Buffers:          2768 kB

Cached:          99440 kB

SwapCached:          0 kB

Active:         165868 kB

Inactive:        75128 kB

HighTotal:           0 kB

HighFree:            0 kB

LowTotal:       256060 kB

LowFree:          2692 kB

SwapTotal:           0 kB

SwapFree:            0 kB

slabinfo - version: 1.1

kmem_cache            60     72    108    2    2    1

tcp_tw_bucket          0     80     96    0    2    1

tcp_bind_bucket       16    113     32    1    1    1

tcp_open_request       0     59     64    0    1    1

inet_peer_cache      207    236     64    4    4    1

ip_fib_hash           62    113     32    1    1    1

ip_dst_cache         514    888    160   28   37    1

arp_cache            308    680     96   13   17    1

blkdev_requests      128    160     96    4    4    1

nfs_write_data         0      0    416    0    0    1

nfs_read_data          0      0    384    0    0    1

nfs_page               0      0     96    0    0    1

journal_head           6    936     48    3   12    1

revoke_table           1    253     12    1    1    1

revoke_record          0    113     32    0    1    1

dnotify cache          0      0     20    0    0    1

file lock cache        0     40     96    0    1    1

fasync cache           0      0     16    0    0    1

uid_cache              1    113     32    1    1    1

skbuff_head_cache    498   1056    160   43   44    1

sock                 448    455    768   90   91    1

sigqueue               0     29    132    0    1    1

cdev_cache           607    678     32    6    6    1

bdev_cache             2     59     64    1    1    1

mnt_cache             12     59     64    1    1    1

inode_cache         1860   1920    480  235  240    1

dentry_cache        1778   1950    128   65   65    1

filp                2392   2400    128   80   80    1

names_cache            0      3   4096    0    3    1

buffer_head        19228  34360     96  859  859    1

mm_struct             71     90    128    3    3    1

vm_area_struct      3478   3560     96   87   89    1

fs_cache              70    113     32    1    1    1

files_cache           70     81    416    8    9    1

signal_act            73     78   1312   25   26    1

size-131072(DMA)       0      0 131072    0    0   32

size-131072            0      0 131072    0    0   32

size-65536(DMA)        0      0  65536    0    0   16

size-65536             0      0  65536    0    0   16

size-32768(DMA)        0      0  32768    0    0    8

size-32768             1      1  32768    1    1    8

size-16384(DMA)        0      0  16384    0    0    4

size-16384             0      4  16384    0    4    4

size-8192(DMA)         0      0   8192    0    0    2

size-8192              1      2   8192    1    2    2

size-4096(DMA)         0      0   4096    0    0    1

size-4096             22     23   4096   22   23    1

size-2048(DMA)         0      0   2048    0    0    1

size-2048            421    518   2048  219  259    1

size-1024(DMA)         0      0   1024    0    0    1

size-1024             35     64   1024   10   16    1

size-512(DMA)          0      0    512    0    0    1

size-512              41     80    512    6   10    1

size-256(DMA)          0      0    256    0    0    1

size-256              43    330    256   12   22    1

size-128(DMA)          0      0    128    0    0    1

size-128            1193   1440    128   40   48    1

size-64(DMA)           0      0     64    0    0    1

size-64              267    354     64    6    6    1

size-32(DMA)           0      0     32    0    0    1

size-32              617    904     32    8    8    1

Proto Recv-Q Send-Q Local Address           Foreign Address         State

(St Helier APs and master controller) #

I need to clear files, the larges being flasbackup.tar.gz at 8488794, not sure I can remove this??

Hi stunshot

Total 256060, used 252069, free 3971

Once you are down under 10M free, you are at risk of going out of memory for any number of reasons, including:

> sudden influx of users (special event, open day etc.)

> tar logs techsupport -- for your current free memory, i would outright ban the use of tar logs tech.

> more APs coming online

> sudden change in WMS data (which causes on board database to grow, thus size in memory) - in fact this applies to anything in the database, including the blacklist users.

This controller is only "just" stable - if there is anyway you can deload it a little that may be a good idea, in a master/local the local may be a little more free since it's not handling the WMS data, you could possibly load it slightly higher.

please note that deleting the flashbackup file won't have any bearing on this, so you can just leave it.

regards

-jeff

Jeff, cheers for your prompt reply.  So to clarify , leave the flashbackup ?  how about if i delete all the old config files ? I believe it wont let me delete the running config ?

They are running way more than 100 AP's so ultimately ths needs a controller upgrade... we are working on that.

cheers

hi stunshot

Nothing which lives under /flash/config  (i.e. what you see when you type 'dir' at the CLI in the controller) directly impacts the amount of free memory that you see when you type "show memory", so no point to worry about cleaning them up.

Those file are part of the flash file system, you can check it with 'show storage', it might be low, might not be. Either way, the low free memory is what is going to bring the controller down (which also comprises of things like the /tmp filesystem, which is used to pre-load software images and tar logs techsupport etc).

Based on the logs you attached before in tandem with 42 days plus uptime, it doesn't look like it's leaking, just 'redlined' as far as utilisation goes.

regards

-jeff

Cheers Jeff, so it might just be a case of the controller and its age just not being up to the job ?

Hi stunshot,

Based on what you have shown and said so far, yes I believe that is the case. Low mem on 5.0.x builds on older SC cards is something that I have seen quite frequently for last 1-2 years, especially when the load starts to reach up around 100 APs and higher.

regards

-jeff