Monitoring, Management & Location Tracking

Basic AMON troubleshooting on Airwave

Aruba Employee

SummaryThis article tells us the basic steps to check if Airwave is processing the data that is received by AMON feed

 

Question: What are the basic steps to verify if Airwave is processing AMON feed or not?

 

Below are few troubleshooting steps to check if Airwave is receiving AMON feed from the controllers:

STEP 1 :

On Airwave, check if the  AMONAggregator and PAPI daemonss are running. Type 'wd' and we will get the below output and check for the highlighted options to be running:

 

 

ONE.jpg

 

STEP 2:

We can also monitor the AMON graphs under SYSTEM --> Performance page of the Airwave server.

Example:

 

TWO.jpg

 

 

This gives more details on the AMON Packets and Messages processing rate.

Also, if the AMON message loss graph is blank, it means there is no data loss and the requests are being processed properly on the Airwave server.



STEP 3:

Then try checking the logging:

# qlog enable amon_dump_raw

The above command will create a file names 'amon_dump_raw' under /var/log/amp_diag directory.

# tail -f /var/log/amp_diag/amon_dump_raw

è Need to wait for the dump to populate.
è This log captures data after it has come into AirWave, but before it hits the processing queues.

 
From there, you can translate the data to be human readable using the below decode command:

# qlog_decoder --decoder papi --nofollow /var/log/amp_diag/amon_dump_raw


Once the file is decoded, we can the file to see the data in human readable format which will give us more information on the data that is being sent from the controller to Airwave.

Version history
Revision #:
1 of 1
Last update:
‎10-29-2014 09:44 AM
Updated by:
 
Labels (1)
Contributors
Comments
justink84

@

 

Thanks for the article, it gave me a place to start looking. Was running a wireshark on airwave via tcpdump, to find out some insight as to what was going on with AMON and PAPI traffic. There wasn't much use as the packets were being delievered although there was still an issue with it getting processed into airwave. It was suspected there was something with the payload that was not being processed. 

 

Ended up having to use the following command "qlog enable amon" as the amon_dump_raw was not showing up in /var/log/amp_diag/ folder. 

 

I was able to see some insight in regards to what was going on at this point. I was receiving the following error in the log file. 

 

[root@jk-amp-01 mercury]# cat /var/log/amp_diag/amon
1460724791.732955 24641 papi|dropping payload from 172.16.22.13 because id is not found
1460724792.982381 24641 papi|dropping payload from 172.16.22.13 because id is not found
1460724792.982667 24641 papi|dropping payload from 172.16.22.13 because id is not found

 

Since my contorller was configured to use the loopback interface as the controller-ip, this is what was causing me issues with getting the payload of papi traffic to be processed by airwave correctely. First I did try changing airwave to have the mgmt IP for the controller of 172.16.22.13, vs the loopback. This still did not help at all. 

 

(jk-aruba7005) #show controller-ip

Switch IP Address: 172.16.20.100

Switch IP is configured to be loopback interface

 

After configuring my controller to use the L3 interface vlan199 (172.16.22.13), the payload was then interperitated correctly by airwave, and it was processing amon traffic correctly. 

 

[root@jk-amp-01 mercury]# cat /var/log/amp_diag/amon

1460725292.565372 24641 papi|dropping payload from 172.16.22.13 because id is not found
1460725299.505907 24641 papi|dropping payload from 172.16.22.13 because id is not found
1460725322.634027 24641 papi|dropping payload from 172.16.22.13 because id is not found
1460725467.677425 24523 publish_controller_vap_stats|published stats for 0 aps
1460725725.080516 24641 papi|dropping payload from 172.16.22.13 because id is not found
1460725731.480864 24647 slice_by_node_id|publish to AMP_inline_stats MQ for AMON_DHCP_STATION_INFO_MESSAGE took 0.000330 seconds
1460725741.130303 24647 slice_by_node_id|publish to AMP_amon_1 MQ for AMON_USER_INFO_MESSAGE took 0.000201 seconds
1460725746.151027 24647 slice_by_node_id|publish to AMP_amon_1 MQ for AMON_USER_INFO_MESSAGE took 0.000135 seconds
1460725746.152604 24647 slice_by_node_id|publish to AMP_inline_stats MQ for AMON_DOT1X_MESSAGE took 0.000123 seconds
1460725750.025544 24647 slice_by_node_id|publish to AMP_inline_stats MQ for AMON_PASSIVE_AP_STATION_STATS_MESSAGE took 0.000141 seconds
1460725751.173871 24647 slice_by_node_id|publish to AMP_amon_1 MQ for AMON_USER_INFO_MESSAGE took 0.000133 seconds
1460725755.170318 24647 slice_by_node_id|publish to AMP_amon_1 MQ for AMON_VAP_STATS_MESSAGE took 0.000136 seconds
1460725755.171294 24647 slice_by_node_id|publish to AMP_amon_1 MQ for AMON_VAP_STATS_MESSAGE took 0.000131 seconds

 

 

Not sure exactly what the problem was wiht using the loopback as the controller-ip in this case. I have inspected the wiresharks from both before / after changing the controller-ip. There isn't really anything that sticks out and the field for the ID: 0x4972 is populated under the PAPI values for both before and after wiresharks. 

 

I suspect that since the papi traffic was being sent to airwave with the L3 interface IP from vlan199 vs sourceing with the IP of the loopback interface, this was the main culprit and I was forced to change teh controller back to the L3 interface vs loopback. I also had tried multiple versions of code, without any success either (6.4.3.7 and 6.4.4.5)

 

Justin

 

Search Airheads
cancel
Showing results for 
Search instead for 
Did you mean: 
Is this a frequent problem?

Request an official Aruba knowledge base article to be written by our experts.