Controllerless Networks

Hi!

I have WLAN which include 1x IAP 135, 2x IAP 175P and 1x IAP 175AC (mesh). All APs works in Instant mode. One of IAP 175P is mesh portal and IAP 175AC is mesh AP. On mesh portal I see high cpu utilization. Average number of clients in rush hours is about 70 clients. 

CPU utilization on mesh portal:

CPU utilization on mesh AP:

Total clients and usage:

Do you have any ideas? Thanks in advance for helping me!

Anyone can help me with this issue? I still have problem with CPU which (probably) cause stucking mesh point in down state for about half time in every day.

Hi friend,

How many child APs uder this portal ?

can you share the output of " Show aps", show ap mesh neighbours" and "show log system"

Please share the required information.

There is only one child connected to the portal. 

Unfortunately i don't have access to CLI, only through web GUI.

Hi,

You can send the summary support through support menu( More--Support).

Please share the required information.

show aps

AP List
-------
Name               IP Address    Mode    Spectrum  Clients  Type   Mesh Role  2.4 Channel  2.4 Power (dB)  2.4 Utilization (%)  2.4 Noise Floor (dBm)  5.0 Channel  5.0 Power (dB)  5.0 Utilization (%)  5.0 Noise Floor (dBm)  Need Antenna Config  From Port  Config Id  Config Csum  Age
----               ----------    ----    --------  -------  ----   ---------  -----------  --------------  -------------------  ---------------------  -----------  --------------  -------------------  ---------------------  -------------------  ---------  ---------  -----------  ---
6c:f3:7f:ce:3f:a4  192.168.1.6*  access  disable   7        135    Portal     6            19              6(good)              -94(good)              60+          22              0(good)              -92(good)              No                   none       7          39754        8d:19h:59m:12s
24:de:c6:cf:dc:e5  192.168.1.7   access  disable   27       175P   Portal     11           20              9(good)              -96(good)              116+         24              0(good)              -93(good)              No                   none       7          39754        8d:19h:58m:55s
24:de:c6:c7:29:7a  192.168.1.4   access  disable   2        135    Portal     11           19              2(good)              -90(good)              44+          22              0(good)              -93(good)              No                   none       7          39754        8d:19h:57m:34s
24:de:c6:cf:dc:5f  192.168.1.5   access  disable   15       175P   Portal     1            20              13(good)             -96(good)              108+         24              20(good)             -93(good)              No                   none       7          39754        2h:42m:40s
d8:c7:c8:c8:04:b5  192.168.1.10  access  disable   11       175AC  Point      11           20              13(good)             -96(good)              108+         24              22(good)             -93(good)              No                   none       7          39754        42m:28s

show ap mesh neighbours

Neighbor list
-------------
MAC                Portal             Channel  Age  Hops  Cost  Relation   Flags  RSSI  Rate Tx/Rx  A-Req  A-Resp  A-Fail  HT-Details       Cluster ID
---                ------             -------  ---  ----  ----  --------   -----  ----  ----------  -----  ------  ------  ----------       ----------
d8:c7:c8:00:4b:59  24:de:c6:7d:c5:f8  108+     0    1     7.00  C 42m:56s  HLK    45    6/6         4      4       0       HT-40MHzsgi-2ss  42b09b0d6e45167b6e1f0483915d056
d8:c7:c8:00:4b:58  24:de:c6:7d:c5:f8  108+     0    1     7.00  N 42m:54s  HLK    45    -           0      0       0       HT-40MHzsgi-2ss  42b09b0d6e45167b6e1f0483915d056

Total count: 2, Children: 1
Relation: P = Parent; C = Child; N = Neighbor; B = Blacklisted-neighbor
Flags: R = Recovery-mode; S = Sub-threshold link; D = Reselection backoff; F = Auth-failure; H = High Throughput; L = Legacy allowed
	K = Connected; U = Upgrading; G = Descendant-upgrading; Z = Config pending; Y = Assoc-resp/Auth pending
	a = SAE Accepted; b = SAE Blacklisted-neighbour; e = SAE Enabled; u = portal-unreachable; o = opensystem

show log system 

Jan 21 12:21:07  stm[1682]: <399801>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  An internal system error has occurred at file rc_acct_instant.c function rc_interim_acct_req line 650.
Jan 21 12:21:07  stm[1682]: <399801>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  An internal system error has occurred at file rc_acct_instant.c function rc_acct_common_vplist line 107.
Jan 21 12:21:07  stm[1682]: <399801>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  An internal system error has occurred at file rc_acct_instant.c function rc_interim_acct_req line 650.
Jan 21 12:23:09  stm[1682]: <399801>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  An internal system error has occurred at file rc_acct_instant.c function rc_acct_common_vplist line 107.
Jan 21 12:23:09  stm[1682]: <399801>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  An internal system error has occurred at file rc_acct_instant.c function rc_interim_acct_req line 650.
Jan 21 12:23:09  stm[1682]: <399801>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  An internal system error has occurred at file rc_acct_instant.c function rc_acct_common_vplist line 107.
Jan 21 12:23:09  stm[1682]: <399801>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  An internal system error has occurred at file rc_acct_instant.c function rc_interim_acct_req line 650.
Jan 21 12:23:38  stm[1682]: <304001>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  Unexpected stm (Station management) runtime error at update_sta_flag, 2069, Doesn't send rc_stop_acct_req when captive portal + auth mode
Jan 21 12:25:07  stm[1682]: <304001>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  Unexpected stm (Station management) runtime error at update_sta_flag, 2069, Doesn't send rc_stop_acct_req when captive portal + auth mode
Jan 21 12:25:12  stm[1682]: <399801>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  An internal system error has occurred at file rc_acct_instant.c function rc_acct_common_vplist line 107.
Jan 21 12:25:12  stm[1682]: <399801>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  An internal system error has occurred at file rc_acct_instant.c function rc_interim_acct_req line 650.
Jan 21 12:25:12  stm[1682]: <399801>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  An internal system error has occurred at file rc_acct_instant.c function rc_acct_common_vplist line 107.
Jan 21 12:25:12  stm[1682]: <399801>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  An internal system error has occurred at file rc_acct_instant.c function rc_interim_acct_req line 650.
Jan 21 12:26:43  stm[1682]: <304001>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  Unexpected stm (Station management) runtime error at update_sta_flag, 2069, Doesn't send rc_stop_acct_req when captive portal + auth mode
Jan 21 12:27:14  stm[1682]: <399801>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  An internal system error has occurred at file rc_acct_instant.c function rc_acct_common_vplist line 107.
Jan 21 12:27:14  stm[1682]: <399801>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  An internal system error has occurred at file rc_acct_instant.c function rc_interim_acct_req line 650.
Jan 21 12:27:14  stm[1682]: <399801>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  An internal system error has occurred at file rc_acct_instant.c function rc_acct_common_vplist line 107.
Jan 21 12:27:14  stm[1682]: <399801>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  An internal system error has occurred at file rc_acct_instant.c function rc_interim_acct_req line 650.
Jan 21 12:27:56  stm[1682]: <304001>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  Unexpected stm (Station management) runtime error at update_sta_flag, 2069, Doesn't send rc_stop_acct_req when captive portal + auth mode
Jan 21 12:28:38  stm[1682]: <304001>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  Unexpected stm (Station management) runtime error at update_sta_flag, 2069, Doesn't send rc_stop_acct_req when captive portal + auth mode

Jan 21 12:29:16  stm[1682]: <399801>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  An internal system error has occurred at file rc_acct_instant.c function rc_acct_common_vplist line 107.
Jan 21 12:29:16  stm[1682]: <399801>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  An internal system error has occurred at file rc_acct_instant.c function rc_interim_acct_req line 650.
Jan 21 12:29:16  stm[1682]: <399801>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  An internal system error has occurred at file rc_acct_instant.c function rc_acct_common_vplist line 107.
Jan 21 12:29:16  stm[1682]: <399801>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  An internal system error has occurred at file rc_acct_instant.c function rc_interim_acct_req line 650.
Jan 21 12:29:44  stm[1682]: <304055>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm| |ap| Unexpected stm (Station management) runtime error at rap_bridge_user_handler, 11755, couldn't find sap fnct:rap_bridge_user_handler, bss:00:00:00:00:00:00
Jan 21 12:30:53  cli[1676]: <341004>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 cli|  AP 192.168.1.5: Client e4:2d:02:45:9f:45 connect fail because DHCP request timed out
Jan 21 12:31:19  stm[1682]: <399801>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  An internal system error has occurred at file rc_acct_instant.c function rc_acct_common_vplist line 107.
Jan 21 12:31:19  stm[1682]: <399801>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  An internal system error has occurred at file rc_acct_instant.c function rc_interim_acct_req line 650.
Jan 21 12:31:19  stm[1682]: <399801>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  An internal system error has occurred at file rc_acct_instant.c function rc_acct_common_vplist line 107.
Jan 21 12:31:19  stm[1682]: <399801>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  An internal system error has occurred at file rc_acct_instant.c function rc_interim_acct_req line 650.
Jan 21 12:32:49  stm[1682]: <304001>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  Unexpected stm (Station management) runtime error at update_sta_flag, 2069, Doesn't send rc_stop_acct_req when captive portal + auth mode
Jan 21 12:33:21  stm[1682]: <399801>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  An internal system error has occurred at file rc_acct_instant.c function rc_acct_common_vplist line 107.
Jan 21 12:33:21  stm[1682]: <399801>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  An internal system error has occurred at file rc_acct_instant.c function rc_interim_acct_req line 650.
Jan 21 12:34:06  stm[1682]: <304001>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  Unexpected stm (Station management) runtime error at update_sta_flag, 2069, Doesn't send rc_stop_acct_req when captive portal + auth mode
Jan 21 12:35:04  stm[1682]: <304001>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  Unexpected stm (Station management) runtime error at update_sta_flag, 2069, Doesn't send rc_stop_acct_req when captive portal + auth mode
Jan 21 12:35:22  cli[1676]: <341004>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 cli|  AP 192.168.1.10: Client cc:3a:61:37:a8:78 connect fail because DHCP request timed out
Jan 21 12:35:23  stm[1682]: <304001>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  Unexpected stm (Station management) runtime error at update_sta_flag, 2069, Doesn't send rc_stop_acct_req when captive portal + auth mode
Jan 21 12:37:10  stm[1682]: <304001>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  Unexpected stm (Station management) runtime error at update_sta_flag, 2069, Doesn't send rc_stop_acct_req when captive portal + auth mode
Jan 21 12:38:18  stm[1682]: <304001>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  Unexpected stm (Station management) runtime error at update_sta_flag, 2069, Doesn't send rc_stop_acct_req when captive portal + auth mode
Jan 21 12:39:19  stm[1682]: <304001>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  Unexpected stm (Station management) runtime error at update_sta_flag, 2069, Doesn't send rc_stop_acct_req when captive portal + auth mode

 when captive portal + auth mode
Jan 21 12:45:53  stm[1682]: <304001>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  Unexpected stm (Station management) runtime error at update_sta_flag, 2069, Doesn't send rc_stop_acct_req when captive portal + auth mode
Jan 21 12:48:31  cli[1676]: <341004>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 cli|  AP 192.168.1.5: Client 30:75:12:ab:37:b3 connect fail because DHCP request timed out
Jan 21 12:48:41  stm[1682]: <304001>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  Unexpected stm (Station management) runtime error at update_sta_flag, 2069, Doesn't send rc_stop_acct_req when captive portal + auth mode
Jan 21 12:50:04  cli[1676]: <341004>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 cli|  AP 192.168.1.7: Client 00:18:60:64:31:1b connect fail because DHCP request timed out
Jan 21 12:50:23  stm[1682]: <304001>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  Unexpected stm (Station management) runtime error at update_sta_flag, 2069, Doesn't send rc_stop_acct_req when captive portal + auth mode
Jan 21 12:57:08  stm[1682]: <304001>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  Unexpected stm (Station management) runtime error at update_sta_flag, 2069, Doesn't send rc_stop_acct_req when captive portal + auth mode
Jan 21 12:58:31  cli[1676]: <341004>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 cli|  AP 192.168.1.10: Client 20:54:76:e0:65:e3 connect fail because DHCP request timed out
Jan 21 12:58:38  stm[1682]: <304001>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  Unexpected stm (Station management) runtime error at update_sta_flag, 2069, Doesn't send rc_stop_acct_req when captive portal + auth mode
Jan 21 12:59:45  stm[1682]: <304055>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm| |ap| Unexpected stm (Station management) runtime error at rap_bridge_user_handler, 11755, couldn't find sap fnct:rap_bridge_user_handler, bss:00:00:00:00:00:00
Jan 21 13:00:08  stm[1682]: <304001>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  Unexpected stm (Station management) runtime error at update_sta_flag, 2069, Doesn't send rc_stop_acct_req when captive portal + auth mode
Jan 21 13:02:23  stm[1682]: <304001>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  Unexpected stm (Station management) runtime error at update_sta_flag, 2069, Doesn't send rc_stop_acct_req when captive portal + auth mode
Jan 21 13:03:38  stm[1682]: <304001>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  Unexpected stm (Station management) runtime error at update_sta_flag, 2069, Doesn't send rc_stop_acct_req when captive portal + auth mode
Jan 21 13:03:52  cli[1676]: <341004>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 cli|  AP 192.168.1.5: Client 88:c9:d0:d7:b7:86 connect fail because DHCP request timed out
Jan 21 13:04:08  stm[1682]: <304001>  |AP 6c:f3:7f:ce:3f:a4@192.168.1.6 stm|  Unexpected stm (Station management) runtime error at update_sta_flag, 2069, Doesn't send rc_stop_acct_req when captive portal + auth mode

All logs originate from virtual controller (except mesh neighbours which include log from mesh portal)

All logs are originating from VC except mesh neighbours which include logs from mesh portal.

Attachment(s)

logs.txt   14 KB 1 version

As far as the mesh portal dropping it appears it's on channel 108 which is a DFS channel. It's entirely possible it's detecting radar on the channel and having to go off channel. If you do a show log wireless all do you see any radar events happening?

Yes, "radar detected" appears on mesh portal.

In fact near the site (about 300m) there is transmitting station like this:

http://pliki.radiopolska.pl/galeria/getpict.php?country=1&filename=mainthumb&id_loc=40

Attachment(s)

radar_events.txt   11 KB 1 version

I would disable DFS channels for your APs in a situation like this, and in most other situations as well unless you're very strapped for channels. Some clients still don't support DFS channels and will fail to connect if an AP is on DFS creating a coverage hole. In this case as well it obviously causes your mesh to drop when the radar comes onto the channel.

It is possible to disable DFS in instant mode? 

You can change the allowed channels arm can use by do the following in the CLI:

(Instant AP)(config)# arm
(Instant AP)(ARM)# a-channels <channel-list>
(Instant AP)(ARM)# end
(Instant AP)# commit apply

http://community.arubanetworks.com/aruba/attachments/aruba/IAP/6226/1/Aruba%20Instant%206.4.0.2-4.1%20CLI%20Reference%20Guide.pdf

You can also do it in the GUI (had to get into my instant demo to take a look). If you go to RF -> Show Advanced Options and select Customize Valid Channels under Access Point control then you can edit the allowed channels. See attached image.

Thanks a lot! In logs there was problem with channel 100+. I changed allowed channel from 36-64. Unfortunately this is global settings (in fact I could set it only on particular AP) - I will monitor behaviour the whole WLAN from radio point and I hope it will be work.