Wireless Access

Reply
Frequent Contributor II

Wireless client network issues

We are having issues with our clients on our network randomly losing connection or having very high latency and slow page loads.  I called into TAC and they weren't very helpful.  An hour on the phone and I didn't feel like they were looking in the right direction.

 

Here is a snapshot from our system log.  we have about 2500 users online at any given point.  Does anything look alarming in this log that would be a cause for concern?  We are running 6.5.3.4 on a 7210 controller with 291 AP's.  We have dual 10 GB uplinks as well.  Where else could I look for strange slow performance across the entire network?

 

Jan 22 13:45:58 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 11' has dropped below  30%  threshold (actual:22%).

Jan 22 13:46:58 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 13' has dropped below  30%  threshold (actual:28%).

Jan 22 13:46:58 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 14' has dropped below  30%  threshold (actual:20%).

Jan 22 13:46:58 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 15' has dropped below  30%  threshold (actual:20%).

Jan 22 13:47:58 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 11' has exceeded  30%  threshold (actual:46%).

Jan 22 13:47:58 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 13' has exceeded  30%  threshold (actual:45%).

Jan 22 13:47:58 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 14' has exceeded  30%  threshold (actual:36%).

Jan 22 13:47:58 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 15' has exceeded  30%  threshold (actual:36%).

Jan 22 13:48:02  KERNEL(amsroombandap-14378@10.110.252.119): [152569.565418] net_ratelimit: 2 callbacks suppressed

Jan 22 13:49:10  KERNEL(amsroombandap-14378@10.110.252.119): [152636.792014] net_ratelimit: 3 callbacks suppressed

Jan 22 13:49:58 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 11' has dropped below  30%  threshold (actual:29%).

Jan 22 13:49:58 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 14' has dropped below  30%  threshold (actual:26%).

Jan 22 13:49:58 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 15' has dropped below  30%  threshold (actual:19%).

Jan 22 13:50:58 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 11' has exceeded  30%  threshold (actual:39%).

Jan 22 13:50:58 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 14' has exceeded  30%  threshold (actual:43%).

Jan 22 13:52:58 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 14' has dropped below  30%  threshold (actual:24%).

Jan 22 13:52:58 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 15' has exceeded  30%  threshold (actual:32%).

Jan 22 13:53:58  KERNEL(amsroom810ap-12765@10.110.252.153): [152936.971525] net_ratelimit: 19 callbacks suppressed

Jan 22 13:53:58 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 14' has exceeded  30%  threshold (actual:35%).

Jan 22 13:54:59 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 15' has dropped below  30%  threshold (actual:26%).

Jan 22 13:55:59 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 11' has dropped below  30%  threshold (actual:23%).

Jan 22 13:55:59 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 13' has dropped below  30%  threshold (actual:20%).

Jan 22 13:55:59 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 14' has dropped below  30%  threshold (actual:20%).

Jan 22 13:55:59 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 15' has exceeded  30%  threshold (actual:35%).

Jan 22 13:56:59 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 13' has exceeded  30%  threshold (actual:31%).

Jan 22 13:56:59 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 14' has exceeded  30%  threshold (actual:36%).

Jan 22 13:57:59 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 11' has exceeded  30%  threshold (actual:51%).

                                                  

Jan 22 13:57:59 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 13' has dropped below  30%  threshold (actual:28%).

Jan 22 13:57:59 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 15' has dropped below  30%  threshold (actual:24%).

Jan 22 13:58:59 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 12' has exceeded  30%  threshold (actual:39%).

Jan 22 13:58:59 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 13' has exceeded  30%  threshold (actual:59%).

Jan 22 13:58:59 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 15' has exceeded  30%  threshold (actual:58%).

Jan 22 13:59:43 :399803:  <4204> <ERRS> |authmgr|  An internal system error has occurred at file aal.c function server_cbh line 634 error server_cbh(634): outstanding_auths already 0, can't decrement further. server_name: campusLDAP, server_group: ACSWirelessSecureServerGroup.

Jan 22 13:59:50 :399803:  <5624> <ERRS> |authmgr|  An internal system error has occurred at file user.c function decrement_authserver_outstanding_auths line 10231 error decrement_authserver_outstanding_auths(10231): outstanding_auths already 0, can't decrement further. server_name: campusLDAP, server_group: ACSWirelessSecureServerGroup.

Jan 22 14:00:59 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 12' has dropped below  30%  threshold (actual:26%).

Jan 22 14:01:59 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 13' has dropped below  30%  threshold (actual:28%).

Jan 22 14:01:59 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 15' has dropped below  30%  threshold (actual:24%).

Jan 22 14:02:59 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 15' has exceeded  30%  threshold (actual:53%).

Jan 22 14:03:16 :355002:  <5126> <DBUG> |cert_dwnld|  cert_downld_mgr_papi_recv_cb: Recv from 127.0.0.1:8407

Jan 22 14:03:16 :355002:  <5126> <DBUG> |cert_dwnld|  cert_downld_mgr_papi_recv_cb: MessageCode: 1005 len 161 data_len 85 Type 1

Jan 22 14:03:16 :348306:  <4643> <DBUG> |ipstm|  papi_rcv_cb, Recvd PAPI Message, len=161

Jan 22 14:03:16 :306603:  <4643> <DBUG> |ipstm| Processing the Debug Information

Jan 22 14:03:16 :355002:  <5126> <DBUG> |cert_dwnld|  cert_downld_mgr_papi_recv_cb: Recv from 127.0.0.1:8407

Jan 22 14:03:16 :355002:  <5126> <DBUG> |cert_dwnld|  cert_downld_mgr_papi_recv_cb: MessageCode: 1005 len 131 data_len 55 Type 1

Jan 22 14:03:16 :348306:  <4643> <DBUG> |ipstm|  papi_rcv_cb, Recvd PAPI Message, len=131

Jan 22 14:03:16 :306603:  <4643> <DBUG> |ipstm| Processing the Debug Information

Jan 22 14:03:16 :355002:  <5126> <DBUG> |cert_dwnld|  cert_downld_mgr_papi_recv_cb: Recv from 127.0.0.1:8407

Jan 22 14:03:16 :355002:  <5126> <DBUG> |cert_dwnld|  cert_downld_mgr_papi_recv_cb: MessageCode: 1005 len 91 data_len 15 Type 1

Jan 22 14:03:16 :348306:  <4643> <DBUG> |ipstm|  papi_rcv_cb, Recvd PAPI Message, len=91

Jan 22 14:03:16 :306603:  <4643> <DBUG> |ipstm| Processing the Debug Information

Jan 22 14:03:16 :355002:  <5126> <DBUG> |cert_dwnld|  cert_downld_mgr_papi_recv_cb: Recv from 127.0.0.1:8407

Jan 22 14:03:16 :355002:  <5126> <DBUG> |cert_dwnld|  cert_downld_mgr_papi_recv_cb: MessageCode: 1005 len 91 data_len 15 Type 1

Jan 22 14:03:16 :348306:  <4643> <DBUG> |ipstm|  papi_rcv_cb, Recvd PAPI Message, len=91

Jan 22 14:03:16 :306603:  <4643> <DBUG> |ipstm| Processing the Debug Information

Jan 22 14:03:59 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 13' has exceeded  30%  threshold (actual:36%).

Jan 22 14:03:59 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 14' has dropped below  30%  threshold (actual:28%).

Jan 22 14:04:59 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 12' has exceeded  30%  threshold (actual:51%).

Jan 22 14:04:59 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 14' has exceeded  30%  threshold (actual:37%).

Jan 22 14:05:59 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 12' has dropped below  30%  threshold (actual:18%).

Jan 22 14:05:59 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 13' has dropped below  30%  threshold (actual:23%).

Jan 22 14:05:59 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 15' has dropped below  30%  threshold (actual:26%).

Jan 22 14:06:23  KERNEL(amsroom613ap-12769@10.110.252.129): [153681.877237] net_ratelimit: 5 callbacks suppressed

Jan 22 14:07:00 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 12' has exceeded  30%  threshold (actual:33%).

Jan 22 14:07:00 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 15' has exceeded  30%  threshold (actual:40%).

Jan 22 14:08:00 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 12' has dropped below  30%  threshold (actual:15%).

Jan 22 14:08:00 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 14' has dropped below  30%  threshold (actual:23%).

Jan 22 14:09:00 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 12' has exceeded  30%  threshold (actual:33%).

Jan 22 14:09:00 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 13' has exceeded  30%  threshold (actual:38%).

Jan 22 14:09:00 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 15' has dropped below  30%  threshold (actual:26%).

Jan 22 14:10:00 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 12' has dropped below  30%  threshold (actual:28%).

Jan 22 14:10:00 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 13' has dropped below  30%  threshold (actual:25%).

Jan 22 14:10:00 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 14' has exceeded  30%  threshold (actual:42%).

Jan 22 14:10:00 :399838:  <4053> <WARN> |fpapps|  Resource 'Datapath CPU 15' has exceeded  30%  threshold (actual:51%).

 

 

Frequent Contributor II

Re: Wireless client network issues

Here are some more errors on the controller system log.

 

Jan 22 12:56:50 :399803:  <5624> <ERRS> |authmgr|  An internal system error has occurred at file user.c function decrement_authserver_outstanding_auths line 10231 error decrement_authserver_outstanding_auths(10231): outstanding_auths already 0, can't decrement further. server_name: campusLDAP, server_group: ACSWirelessSecureServerGroup.

Jan 22 12:56:52 :399803:  <5624> <ERRS> |authmgr|  An internal system error has occurred at file user.c function decrement_authserver_outstanding_auths line 10231 error decrement_authserver_outstanding_auths(10231): outstanding_auths already 0, can't decrement further. server_name: campusLDAP, server_group: ACSWirelessSecureServerGroup.

Jan 22 12:56:54 :399803:  <5624> <ERRS> |authmgr|  An internal system error has occurred at file user.c function decrement_authserver_outstanding_auths line 10231 error decrement_authserver_outstanding_auths(10231): outstanding_auths already 0, can't decrement further. server_name: campusLDAP, server_group: ACSWirelessSecureServerGroup.

Jan 22 12:56:54 :399803:  <5624> <ERRS> |authmgr|  An internal system error has occurred at file user.c function decrement_authserver_outstanding_auths line 10231 error decrement_authserver_outstanding_auths(10231): outstanding_auths already 0, can't decrement further. server_name: campusLDAP, server_group: ACSWirelessSecureServerGroup.

Jan 22 12:56:57 :399803:  <5624> <ERRS> |authmgr|  An internal system error has occurred at file user.c function decrement_authserver_outstanding_auths line 10231 error decrement_authserver_outstanding_auths(10231): outstanding_auths already 0, can't decrement further. server_name: campusLDAP, server_group: ACSWirelessSecureServerGroup.

Jan 22 12:57:03 :399803:  <5624> <ERRS> |authmgr|  An internal system error has occurred at file user.c function decrement_authserver_outstanding_auths line 10231 error decrement_authserver_outstanding_auths(10231): outstanding_auths already 0, can't decrement further. server_name: campusLDAP, server_group: ACSWirelessSecureServerGroup.

Jan 22 12:57:35 :399803:  <5624> <ERRS> |authmgr|  An internal system error has occurred at file user.c function decrement_authserver_outstanding_auths line 10231 error decrement_authserver_outstanding_auths(10231): outstanding_auths already 0, can't decrement further. server_name: campusLDAP, server_group: ACSWirelessSecureServerGroup.

Jan 22 12:57:47 :399803:  <5624> <ERRS> |authmgr|  An internal system error has occurred at file user.c function decrement_authserver_outstanding_auths line 10231 error decrement_authserver_outstanding_auths(10231): outstanding_auths already 0, can't decrement further. server_name: campusLDAP, server_group: ACSWirelessSecureServerGroup.

Jan 22 12:57:56 :399803:  <4204> <ERRS> |authmgr|  An internal system error has occurred at file aal.c function server_cbh line 634 error server_cbh(634): outstanding_auths already 0, can't decrement further. server_name: campusLDAP, server_group: ACSWirelessSecureServerGroup.

Jan 22 12:58:22 :399803:  <5624> <ERRS> |authmgr|  An internal system error has occurred at file user.c function decrement_authserver_outstanding_auths line 10231 error decrement_authserver_outstanding_auths(10231): outstanding_auths already 0, can't decrement further. server_name: campusLDAP, server_group: ACSWirelessSecureServerGroup.

Jan 22 12:58:35 :399803:  <5624> <ERRS> |authmgr|  An internal system error has occurred at file user.c function decrement_authserver_outstanding_auths line 10231 error decrement_authserver_outstanding_auths(10231): outstanding_auths already 0, can't decrement further. server_name: campusLDAP, server_group: ACSWirelessSecureServerGroup.

Guru Elite

Re: Wireless client network issues

There is not enough information to understand what could be going on.  The first set of logs are cosmetic.  It just means the system was over 30% for a little in terms of data.  That would not explain your issue.  The second set of logs are inconclusive.

 

If at any time TAC is not helpful, you can ask for an escalation.

******************
Answers and views expressed by me on this forum are my own and not necessarily the position of Aruba Networks or Hewlett Packard Enterprise.
******************
Search Airheads
cancel
Showing results for 
Search instead for 
Did you mean: