Wireless Access

last person joined: 7 hours ago 

Access network design for branch, remote, outdoor and campus locations with Aruba access points, and mobility controllers.
Expand all | Collapse all

controller reboots when blacklist reaches 300

  • 1.  controller reboots when blacklist reaches 300

    Posted Jul 24, 2014 04:30 AM

    Hi all, have an old controller running 5x software.  It reloads itself when blacklist gets to 300 entries.

     

    Any ideas ??

     

    Thanks



  • 2.  RE: controller reboots when blacklist reaches 300

    Posted Jul 24, 2014 06:58 AM

    You should open a support case.  That should not happen.



  • 3.  RE: controller reboots when blacklist reaches 300

    Posted Jul 24, 2014 07:22 AM

    Colin, indeed, problem is controller out of support, just seeing if anyone had seen the issue.



  • 4.  RE: controller reboots when blacklist reaches 300

    Posted Jul 24, 2014 10:32 AM

    How is the blacklisting being done?

    What version of ArubaOS?

    What controller platform?

     



  • 5.  RE: controller reboots when blacklist reaches 300

    Posted Jul 24, 2014 11:03 AM

    Colin, its an old 6000 controler (well 2 x 6000), master local, running 5.0.4.13 firmware.  Blacklist clients in the local are showing as 'user defined' but blacklist clients is checked in the VAP profiles.

     

    The local controller has no issues, same box, same firmware.  It has blacklist clients there 200 days etc ....



  • 6.  RE: controller reboots when blacklist reaches 300

    Posted Jul 24, 2014 11:06 AM

    Type "show version" on the box to get the reboot reason.

     



  • 7.  RE: controller reboots when blacklist reaches 300

    Posted Jul 24, 2014 11:15 AM

    up 42 days, last reboot: user reboot

     

    I have seen the 'user reboot' before in other issues where the controller has self-booted



  • 8.  RE: controller reboots when blacklist reaches 300

    Posted Jul 29, 2014 12:20 AM

    @stunshot wrote:

    Colin, its an old 6000 controler (well 2 x 6000), master local, running 5.0.4.13 firmware.  Blacklist clients in the local are showing as 'user defined' but blacklist clients is checked in the VAP profiles.

     

    The local controller has no issues, same box, same firmware.  It has blacklist clients there 200 days etc ....


    Stunshot, is it running on SC hardware then in late 5.0.4.x on SC1/SC2 will be very low on memory during run time if the number of APs is high, such as around 100 and over. It's possible it's running out of memory, in which case there are not a lot of options.

     

    Can you show output of "show memory" while this controller is in runtime, can also show "show memory debug" as an attachment too.

     

    regards

    -jeff



  • 9.  RE: controller reboots when blacklist reaches 300

    Posted Jul 29, 2014 01:39 AM

    Jeff, thanks for the reply, yes the memory is very full, show memory gives

     

    Total 256060, used 252069, free 3971

     

    Output of memory debug,

     

    APs and master controller) #show memory debug

     

    memory snapshot: Tue Jul 29 06:30:25 2014

    =========================================

    Memory (Kb): total: 256060, free: 3571

     

    4104 /mswitch/bin/l2tpd -D

    3720 /mswitch/bin/isakmpd

    3656 /mswitch/bin/profmgr

    2880 /mswitch/bin/udbserver

    26676 /mswitch/bin/stm

    1928 /mswitch/bin/mobileip

    1924 /mswitch/bin/snmpd

    18076 /mswitch/bin/fpapps

    15064 /mswitch/bin/cfgm

    1476 /mswitch/bin/ospf

    1452 /mswitch/bin/slb

    1372 /mswitch/bin/dbsync

    1328 /mswitch/bin/pptpd -f

    12968 /mswitch/bin/wms -l 5

    1220 /mswitch/bin/pim

    10032 /mswitch/bin/auth

     

    Querying excessively large apps

     

    auth     :  9/20M (cur/typical)  0M over

    stm      : 26/25M (cur/typical)  1M over

    wms      : 12/18M (cur/typical)  0M over

    cfgm     : 14/23M (cur/typical)  0M over

    fpapps   : 17/20M (cur/typical)  0M over

    profmgr  :  3/ 8M (cur/typical)  0M over

    isakmpd  :  3/ 6M (cur/typical)  0M over

    l2tpd    :  4/ 8M (cur/typical)  0M over

    mobileip :  1/ 4M (cur/typical)  0M over

    ospf     :  1/ 4M (cur/typical)  0M over

    pim      :  1/ 2M (cur/typical)  0M over

    pptpd    :  1/ 5M (cur/typical)  0M over

    dbsync   :  1/ 3M (cur/typical)  0M over

    slb      :  1/ 4M (cur/typical)  0M over

    snmpd    :  1/ 5M (cur/typical)  0M over

    udbserver:  2/ 4M (cur/typical)  0M over

     

    Large files under /tmp

    ======================

    /tmp/.var/mysql/wms:-rw-r-----    1 root     root      2167547 Jul 29 06:30 probe_sta_table.ISD

    /tmp/.var/mysql/wms:-rw-r-----    1 root     root      1299456 Jul 29 06:30 probe_sta_table.ISM

    %CPU S   PID  PPID  RSS CMD

     4.0 S 31897   536 26676 stm

     0.0 S   839   679 18076 fpapps

     0.0 S   840   679 18076 fpapps

     0.0 S   842   679 18076 fpapps

     0.0 S   843   679 18076 fpapps

     0.0 S   844   679 18076 fpapps

     0.0 S   841   679 18076 fpapps

     0.0 S   848   679 18076 fpapps

     0.0 S   849   679 18076 fpapps

     0.0 S   850   679 18076 fpapps

     0.0 S   816   679 18076 fpapps

     0.0 S   829   679 18076 fpapps

     0.3 S   954   679 18076 fpapps

     0.0 S   838   679 18076 fpapps

     0.0 S   679   678 18076 fpapps

     0.0 S   845   679 18076 fpapps

     0.0 S   756   679 18076 fpapps

     0.0 S   678   536 18076 fpapps

     0.0 S   815   679 18076 fpapps

     0.0 S   680   679 18076 fpapps

     0.0 S   814   679 18076 fpapps

     0.0 S   837   679 18076 fpapps

     0.1 S   809   679 18076 fpapps

     0.0 S   672   536 15064 cfgm

     0.0 S   552   536 14888 fpcli

     2.9 S   702   536 12968 wms -l 5

     0.1 S   728   585 10520 mysql/libexec/mysqld

     0.0 S   675   585 10520 mysql/libexec/mysqld

     0.0 S 16898   585 10520 mysql/libexec/mysqld

     0.0 S   888   585 10520 mysql/libexec/mysqld

     0.0 S  1014   585 10520 mysql/libexec/mysqld

     0.0 S   585   584 10520 mysql/libexec/mysqld

     0.2 S 32028   585 10520 mysql/libexec/mysqld

     2.3 S   712   585 10520 mysql/libexec/mysqld

     0.0 S   586   585 10520 mysql/libexec/mysqld

     0.0 S   584   559 10520 mysql/libexec/mysqld

     0.0 S   692   585 10520 mysql/libexec/mysqld

     0.0 S   725   585 10520 mysql/libexec/mysqld

     0.0 S   538   536 10212 arci-cli-helper

     0.5 S   715   536 10032 auth

     0.0 S   699   536 4104 l2tpd -D

     0.0 S  1036   771 3804 apache/bin/httpd -DFOR

     0.0 S   701   536 3720 isakmpd

     0.0 S   710   536 3656 profmgr

     0.0 S   724   536 2880 udbserver

     0.2 S   736   536 2604 trapd

     0.0 S   985   737 2588 ntpd -g -L

     0.0 S  1006   726 2152 dhcpd -f -lf /tmp/dhcpd.lease e

     0.0 S  1008  1007 2152 dhcpd -f -lf /tmp/dhcpd.lease e

     0.0 S  1007  1006 2152 dhcpd -f -lf /tmp/dhcpd.lease e

     0.0 S   727   536 1928 mobileip

     0.0 S   735   536 1924 snmpd

     0.0 S   691   536 1732 licensemgr

     0.4 S 20227   761 1568 /etc/ssh/sshd: admin@pts/0

     0.0 S   778   536 1476 ospf

     0.0 S   745   536 1452 slb

     0.0 S   764   536 1384 cts

     0.0 S   738   536 1372 dbsync

     0.0 S   700   536 1328 pptpd -f

     0.0 S   786   536 1292 cpsec

     0.0 S   674   536 1260 aaa

     0.0 S   555   536 1220 certmgr

     0.0 S   681   536 1220 pim

     0.0 S   761   536 1184 /etc/ssh/sshd -D -f /etc/ssh/ss

     0.0 S   713   536 1160 rfm

     0.0 S   673   536 1144 syslogdwrap

     0.0 S   726   536 1120 dhcpdwrap

     0.0 S   536     1 1084 nanny bin

     0.0 S   737   536 1028 ntpwrap

     0.0 S   554   536  972 packet_filter

     0.0 S   746   536  956 ads

     0.2 S   729   536  948 hwMon

     0.0 S   773   536  944 msghh

     0.0 S   686   536  928 p8mgr

     

    Filesystem           1k-blocks      Used Available Use% Mounted on

    /dev/root                71957     68212      3745  95% /

    none                     71680      9108     62572  13% /tmp

    /dev/hda3               153271     57531     87826  40% /flash

     

            total:    used:    free:  shared: buffers:  cached:

    Mem:  262205440 259448832  2756608        0  2834432 101826560

    Swap:        0        0        0

    MemTotal:       256060 kB

    MemFree:          2692 kB

    MemShared:           0 kB

    Buffers:          2768 kB

    Cached:          99440 kB

    SwapCached:          0 kB

    Active:         165868 kB

    Inactive:        75128 kB

    HighTotal:           0 kB

    HighFree:            0 kB

    LowTotal:       256060 kB

    LowFree:          2692 kB

    SwapTotal:           0 kB

    SwapFree:            0 kB

     

    slabinfo - version: 1.1

    kmem_cache            60     72    108    2    2    1

    tcp_tw_bucket          0     80     96    0    2    1

    tcp_bind_bucket       16    113     32    1    1    1

    tcp_open_request       0     59     64    0    1    1

    inet_peer_cache      207    236     64    4    4    1

    ip_fib_hash           62    113     32    1    1    1

    ip_dst_cache         514    888    160   28   37    1

    arp_cache            308    680     96   13   17    1

    blkdev_requests      128    160     96    4    4    1

    nfs_write_data         0      0    416    0    0    1

    nfs_read_data          0      0    384    0    0    1

    nfs_page               0      0     96    0    0    1

    journal_head           6    936     48    3   12    1

    revoke_table           1    253     12    1    1    1

    revoke_record          0    113     32    0    1    1

    dnotify cache          0      0     20    0    0    1

    file lock cache        0     40     96    0    1    1

    fasync cache           0      0     16    0    0    1

    uid_cache              1    113     32    1    1    1

    skbuff_head_cache    498   1056    160   43   44    1

    sock                 448    455    768   90   91    1

    sigqueue               0     29    132    0    1    1

    cdev_cache           607    678     32    6    6    1

    bdev_cache             2     59     64    1    1    1

    mnt_cache             12     59     64    1    1    1

    inode_cache         1860   1920    480  235  240    1

    dentry_cache        1778   1950    128   65   65    1

    filp                2392   2400    128   80   80    1

    names_cache            0      3   4096    0    3    1

    buffer_head        19228  34360     96  859  859    1

    mm_struct             71     90    128    3    3    1

    vm_area_struct      3478   3560     96   87   89    1

    fs_cache              70    113     32    1    1    1

    files_cache           70     81    416    8    9    1

    signal_act            73     78   1312   25   26    1

    size-131072(DMA)       0      0 131072    0    0   32

    size-131072            0      0 131072    0    0   32

    size-65536(DMA)        0      0  65536    0    0   16

    size-65536             0      0  65536    0    0   16

    size-32768(DMA)        0      0  32768    0    0    8

    size-32768             1      1  32768    1    1    8

    size-16384(DMA)        0      0  16384    0    0    4

    size-16384             0      4  16384    0    4    4

    size-8192(DMA)         0      0   8192    0    0    2

    size-8192              1      2   8192    1    2    2

    size-4096(DMA)         0      0   4096    0    0    1

    size-4096             22     23   4096   22   23    1

    size-2048(DMA)         0      0   2048    0    0    1

    size-2048            421    518   2048  219  259    1

    size-1024(DMA)         0      0   1024    0    0    1

    size-1024             35     64   1024   10   16    1

    size-512(DMA)          0      0    512    0    0    1

    size-512              41     80    512    6   10    1

    size-256(DMA)          0      0    256    0    0    1

    size-256              43    330    256   12   22    1

    size-128(DMA)          0      0    128    0    0    1

    size-128            1193   1440    128   40   48    1

    size-64(DMA)           0      0     64    0    0    1

    size-64              267    354     64    6    6    1

    size-32(DMA)           0      0     32    0    0    1

    size-32              617    904     32    8    8    1

     

    Proto Recv-Q Send-Q Local Address           Foreign Address         State

     

     

    (St Helier APs and master controller) #

     

     

     

    I need to clear files, the larges being flasbackup.tar.gz at 8488794, not sure I can remove this??



  • 10.  RE: controller reboots when blacklist reaches 300

    Posted Jul 29, 2014 02:17 AM

    Hi stunshot

     

    Total 256060, used 252069, free 3971

     

    Once you are down under 10M free, you are at risk of going out of memory for any number of reasons, including:

     

    > sudden influx of users (special event, open day etc.)

    > tar logs techsupport -- for your current free memory, i would outright ban the use of tar logs tech.

    > more APs coming online

    > sudden change in WMS data (which causes on board database to grow, thus size in memory) - in fact this applies to anything in the database, including the blacklist users.

     

    This controller is only "just" stable - if there is anyway you can deload it a little that may be a good idea, in a master/local the local may be a little more free since it's not handling the WMS data, you could possibly load it slightly higher.

     

    please note that deleting the flashbackup file won't have any bearing on this, so you can just leave it.

     

    regards

    -jeff



  • 11.  RE: controller reboots when blacklist reaches 300

    Posted Jul 29, 2014 02:45 AM

    Jeff, cheers for your prompt reply.  So to clarify , leave the flashbackup ?  how about if i delete all the old config files ? I believe it wont let me delete the running config ?

     

    They are running way more than 100 AP's so ultimately ths needs a controller upgrade... we are working on that.

     

    cheers



  • 12.  RE: controller reboots when blacklist reaches 300

    Posted Jul 29, 2014 02:56 AM

    hi stunshot

    Nothing which lives under /flash/config  (i.e. what you see when you type 'dir' at the CLI in the controller) directly impacts the amount of free memory that you see when you type "show memory", so no point to worry about cleaning them up.

     

    Those file are part of the flash file system, you can check it with 'show storage', it might be low, might not be. Either way, the low free memory is what is going to bring the controller down (which also comprises of things like the /tmp filesystem, which is used to pre-load software images and tar logs techsupport etc).

     

    Based on the logs you attached before in tandem with 42 days plus uptime, it doesn't look like it's leaking, just 'redlined' as far as utilisation goes.

     

    regards

    -jeff



  • 13.  RE: controller reboots when blacklist reaches 300

    Posted Jul 29, 2014 03:31 AM

    Cheers Jeff, so it might just be a case of the controller and its age just not being up to the job ?



  • 14.  RE: controller reboots when blacklist reaches 300

    Posted Jul 29, 2014 09:25 AM

    Hi stunshot,

     

    Based on what you have shown and said so far, yes I believe that is the case. Low mem on 5.0.x builds on older SC cards is something that I have seen quite frequently for last 1-2 years, especially when the load starts to reach up around 100 APs and higher.

     

    regards

    -jeff