Wireless Access

I have been seeing a bunch of these on my controller. I have a tac case opened for this customer (a bunch of issues, APs not being able to talk back to the controller, Users not being able to load external captive portal, httpd crashing, etc.)

Sep 22 17:14:34 <stm 304067> <ERRS> |AP 00111aaa051E:1647:4919@10.22.3.10 stm| PAPI_Send failed: papi_send_status_callback, 2731: Connection timed out; from 8451 to 216.123.55.169:8214 len 109 type 3092
Sep 22 17:14:34 <stm 304067> <ERRS> |AP 0111aaa01234:850:4618@10.22.2.28 stm| PAPI_Send failed: papi_send_status_callback, 2731: Connection timed out; from 8451 to 216.123.55.169:8214 len 108 type 3092

According to the syslog guide it means:

PAPI_Send failed: [func:%s], [line:%d]: [error:%s]; from [sport:%d] to
[toaddr:%p]:[toport:%d] len [len:%d] type [type:%d]
Description: Inter-process communication message failed to reach the target.

Any ideas?

AOS 6.3.1.8

4x7220s 

all masters

Mainly, check the distro ethernet for packet loss or issues of that ilk.  Are these messages happening with other services on the controller, or just STM?

---

@pmonardo wrote:

I have been seeing a bunch of these on my controller. I have a tac case opened for this customer (a bunch of issues, APs not being able to talk back to the controller, Users not being able to load external captive portal, httpd crashing, etc.)

 

Sep 22 17:14:34 <stm 304067> <ERRS> |AP 00111aaa051E:1647:4919@10.22.3.10 stm| PAPI_Send failed: papi_send_status_callback, 2731: Connection timed out; from 8451 to 216.123.55.169:8214 len 109 type 3092
Sep 22 17:14:34 <stm 304067> <ERRS> |AP 0111aaa01234:850:4618@10.22.2.28 stm| PAPI_Send failed: papi_send_status_callback, 2731: Connection timed out; from 8451 to 216.123.55.169:8214 len 108 type 3092

 

According to the syslog guide it means:

PAPI_Send failed: [func:%s], [line:%d]: [error:%s]; from [sport:%d] to
[toaddr:%p]:[toport:%d] len [len:%d] type [type:%d]
Description: Inter-process communication message failed to reach the target.

 

Any ideas?

 

AOS 6.3.1.8

4x7220s 

all masters

---

Pmonardo,

Those messages by themselves are cosmetic unless they are sustained.  More debugging would have to be done to determine if they are service-impacting.

I can tell you guys for sure that this particular network is suffering from httpd crashes on all the controllers, APs seem to randomly lose their tunnel or reboot, authenticated users seem to drop off the network.
EDIT: I guess attaching a pic to email replies does not work.

This is a graph of authenticated users and from noon to 6pm, a huge drop and for a major coffee chain across the country is not correct.

Anyway, I was more curious to see if that error message would point to something.

Thanks

pmonardo, I cannot see your picture, but TAC is the best route.  Please let us know what they find.

Hello, Pasquale.

What did you find to be the issue?  Your symptoms sound very much like something I am experiencing as well.

Pasquale

Sep 22 17:14:34 <stm 304067> <ERRS> |AP 00111aaa051E:1647:4919@10.22.3.10 stm| PAPI_Send failed: papi_send_status_callback, 2731: Connection timed out; from 8451 to 216.123.55.169:8214 len 109 type 3092

this message is saying is that the AP 10.22.3.10 tried to send a message from AP STM (8451) to the controller AUTH (8214).. That should not time out, so yes, I would say that you have something going on, either between this AP and the controller, or if this message is more widespread across multiple APs, then more likely on the controller and specificlly involving AUTH.

Start by checking CPU, throttles, radius (or whatever back end auth is being used), security and user debug, 'show ipc statistics app-id 8214' etc.

By the way, for the captive portal in a hotspot like this, what is the model of controller here and is the httpd cpu running high ?

regards

-jeff