Network Management

got a number of 650 controllers running 6.3.1.5 that are polled by AirWave 7.7.13. in the logging messages show up that point to too heavy SMP polling according to TAC (snmp[2382]: PAPI_Send: To: 7f000001:8366 Type:0x4 Timed out.).

my polling settings for this group are:

Up/Down Status Polling Period: 60 seconds
AP Interface Polling Period: 5 min
Client Data Polling Period: 5 min
Thin AP Discovery Polling Period: 2 min
Device-to-Device Link Polling Period: 30 min
802.11 Counters Polling Period: 5 min
Rogue AP and Device Location Data Polling Period: 30 min
CDP Neighbor Data Polling Period: 5min

what are others using for AirWave polling of 650s? do you see these messages or does it work fine?

What type of deployment are we looking at (office/school/hotspot)? How many controllers/APs in the network?  What's the average client load?

Other things worth noting are the SNMP settings from Device Setup -> Communication

And the number of monitoring/config/audit processes allowed on AMP Setup -> General

full setup is a master (3xxx) with a dozen locals (650). this specific local is an office deployment, 3 SSIDs, 4 APs connected and a maximum of 40 users, usually less, average 20.

SNMP Timeout (3-60 sec): 3
SNMP Retries (1-40): 3

Monitoring Processes (1-4): 2    
Maximum number of configuration processes (1-20): 5
Maximum number of audit processes (1-20): 3
SNMP Fetcher Count (2-6): 2

What's the TAC case number?

Based on the info, I might assume that the AMP is making a request that the controller is not responding to in under 3 seconds which might result in a re-request causing additional SNMP load.  I would suggest increasing the SNMP timeout max at 60s (not retries).  That way AirWave is not re-requesting the same info while the controller may already be compiling the data response (hopefully reducing SNMP load).

Prior to making the above change: How is network latency in this environment?  If you take a tcpdump of a single controller to AMP, do you see multiple SNMP requests for the same data topic in a short period of time?

TAC case 1653177

latency seems pretty fine, it is an MPLS cloud, peaks to 50ms. never the less ill try to higher timeout to see what  happens.

i also moved this controller to a group with the default polling settings which does help with getting the messages occur less often (went from like every minute to every hour), but still it happens.

higher timeout didn't seem to do anything. something else to try? or just accept that these 650s will give some timeouts?

turned out to be bug 87717

this specific error (port 8366) represents the hardware monitor, in this case the error is saying that some snmp query tried to talk to the hardware monitor and it timed out.

which was also described on airheads in the past and support confirmed it most likely won't be fixed:

http://community.arubanetworks.com/t5/Unified-Wired-Wireless-Access/What-is-the-log-of-snmp-1606-PAPI-Send-To-7f000001-8366-Type0x4/m-p/154678/highlight/true#M32851