Monitoring, Management & Location Tracking

Problem:

AMP version : 8.0.11

Issue reported : Client count discrepancy between controller and Airwave when preferred AMON is enabled.

Client count discrepancy between controller and Airwave when preferred amon is enabled.

Troubleshooting done:

1. We may not see any crash or error message in Amp logs related to the client count discrepancy.

2. But we will see Amon data loss from the system performance page,

4. From the rabbitmq list queues we could notice that the amon messages are getting filled up. Example from the below output the AMP_papi_facade is filling up with huge amon messages.

[root@airwave log]# rabbitmqctl list_queues
Listing queues ...
AMP_amon        0
AMP_amon_decoder        0
AMP_amon_euq_request    0
AMP_amon_euq_response   0
AMP_papi_facade 106058321
AMP_pef_euq_request     0
AMP_wms_offload 0
scriptorium_queue       0
...done.

This issue happens with only controllers upgraded to the version 6.4.3.1 and later. Since the amon buffer size is reduced in those codes to 1400  so AMP continuously receive amon messages which packs the rabbitmq list queue.

Note: Upgrading the controller to 6.4.3.9 will still hold amon buffer size as 1400, only new install of 6.4.3.9 will have the amon buffer size as 32000.

Solution:

To resolve this issue instantly without upgrading the controller we can increase the amon msg buffer size by executing the following command on the controller,

#configure terminal

#amon msg-buffer-size 32000

#exit

#save configuration

After increasing the amon msg buffer size to 32000 on controller we can monitor AMP for the accurate client count.