Network Management

Hi, I did a clean installation of the AMP 8.2.6.1 from ISO to VM and configured the connection with the controller. But It looks like AMP doesn't receive or controller doesn't send any AMON data. Controller and AMP are in the one VLAN and can ping each other. AMP pools controller via SNMP and collects APs and users statistics. But AppRF, Clarity, etc. is blank.

Following is my setup:

(aruba1) #show mgmt-servers                   
List of Management Servers
--------------------------
Type     Primary Server  Profile
----     --------------  -------
AirWave  192.168.112.8   default-amp
Num Rows:1

(aruba1) #show mgmt-server profile default-amp
Mgmt Config profile "default-amp" (Predefined (editable))
---------------------------------------------------------
Parameter          Value
---------          -----
Stats              Enabled
Tag                Enabled
Sessions           Enabled
Monitored Info     Disabled
Misc               Enabled
Location           Enabled
UCC Monitoring     Disabled
AirGroup Info      Disabled
Inline DHCP stats  Enabled
Inline AP stats    Enabled
Inline Auth stats  Enabled
Inline DNS stats   Enabled

but wms doesn't send any messages

(aruba1) #show mgmt-server message-counters process wms

Message Counter History
-----------------------
Message Number  Time  Packets
--------------  ----  -------

In AMP - Setup - Performance graphs named AMON Message Processing Rate and AMON Packet Arrival / Processing Rate show zero receive/process rate for AMON packets.

Can anybody hint me what is wrong with my setup and where are AMON messages lost?

 

Thanks. Martin

Might be asking the obvious but have you enabled AMON?

 

AMP Setup> General > Additional AMP Options 

enable amon data collection
prefer amon vs snmp

Are the WLC's credentials correct, do you see them as showing as OK?

On occasion I have had to restart the AMP Services on 8.2.6 as the AMON Packet Process was also showing 0. 

 

From the System > Status page > Restart AMP Services

Hi,

"Enable AMON data collection" is enabled. If I enable "Prefer AMON vs. SNMP" option, I also lost data about connected clients, because they are collected via SNMP at this moment, and AMON doesn't work.

 

I reentered WLC credentials and reload services. Will see.

 

Sure, just check your SNMP queue isn't full either. I had an issue once
where there was 2x redundant SNMP servers configured which caused the queue
to fill up and traps be dropped.

(Aruba620) #show snmp inform stats

Inform queue size is 250

SNMP INFORM STATS
-----------------
HOST PORT INFORMS-INQUEUE OVERFLOW TOTAL INFORMS
---- ---- --------------- -------- -------------

Setting "Prefer AMON" in Airwave should not be part of a troubleshooting step, because it does not affect if AMON is received.

AMON is not sent over SNMP, so the SNMP queues are not involved.

Please see the article here: http://community.arubanetworks.com/t5/Community-Tribal-Knowledge-Base/Why-you-should-always-enable-AMON-on-your-Aruba-Controller-when/ta-p/72104 about what you should have configured.  It seems that you do not have monitored info enabled in your AMON profile and that could be critical.

My SNMP queue is empty as well. I followed linked document. Default AMP profile should be enough? There is no firewall between AMP and WLC.

Now I noticed that System - Status pages show that PAPI Message Handler is in downstate. I reloaded AMP processes and restarted VM, but still:

PAPI Message Handler     Down     /var/log/papi_handler

What's the error in the papi_handler log?  Anything in the amon_aggregator log as well?  The service being down is of concern since the papi_handler is where the AMON msgs first get processed.  A support case should be opened to dig into this further.

papi_handler log just repeats:

2018-05-21 23:28:38.808 INFO [main] - CONF_PATH /usr/local/airwave/lib/scala/papi_handler.conf
2018-05-21 23:28:53.501 INFO [main] - CONF_PATH /usr/local/airwave/lib/scala/papi_handler.conf

and also nothing is interesting in the amon_aggregator log

Mon May 21 15:52:42 2018: Started (PID: 29140)
Mon May 21 15:52:42 2018: Postgres PID: 29141
Mon May 21 16:26:09 2018: Started (PID: 2042)
Mon May 21 16:26:09 2018: Postgres PID: 2043

Is there any way how to debug processes in CLI? I'm skilled Linux admin, but access to the generic console is limited in AMP 8.2.

 

System - Event log is full of:

 

Mon May 21 23:41:22 2018 	System 	System 	Restarting service PAPI Message Handler 				
Mon May 21 23:41:07 2018 	System 	System 	Restarting service PAPI Message Handler

It repeats every 15 seconds. Something must be wrong with PAPI collector daemon.

HI,

 

papi_hanlder is the daemon which hanldes AMONAggregator and PEFProcessor Daemons which hanldes client montiorng /PEF aggregate session data.

 

We generally see papi restart, if we have system perfromance issue. can you check the system>performance page and see whether server have sufficent CPU/RAM memory.

 

Check service_watcher/message log in /var/log directory for any memory limit errors and also papi_hanlder log for erros..

 

 

I suggest a support case since additional logging needs to be enabled, and only support has backdoor access beyond the limited AMPCLI menu.

Hi, I thank all of you for your hints. Luckily I discovered the cause of this issue. I found an older thread here in forums mentioning problems with AMP processes related to AMP hostname resolving in DNS and hostname binding to the TLS certificate.

My deployment of the AMP is in private management VLAN using, and there is no DNS resolver service. I installed AMP with its private address in CN field of TLS certificate. I changed hostname which means subsequent regeneration of cert three times (-> amp -> airwave -> airwave.local). I saw PAPI collector running after the third change of hostname.

So my conclusion is that something related to the hostname and its resolvability makes PAPI collector for failing during startup.

Martin

Thanks for that update.  That's a helpful scenario to know.

Hopefully, it will help others.