What is the best way to get detailed information on why a small number of APs at a remote branch are going down? Assume the network itself has been fully investigated, and I'm just looking to understand from the AP's and the controller's point of view what is going on.
APs are CAPs in tunnel mode on 6.4.3 7200.
APs connect over a private WAN to controllers some distance away (20ms+).
HA has been disabled, so it's just basic LMS traffic
We have Airwave reporting APs down only some of the time, but by checking the Network Summary in the master controller one can see they are bouncing far more often.
I would like to know what particular packets have to be missed for the controller to consider the AP down. Also how which commands/debugs to clearly capture this information.
thanks
edit: if it's not clear this is an intermittent, so APs are constantly 'bouncing' (but not rebooting) through the day, status changing minute to minute
edit: no physical access to AP or controller
edit: some system logs. note I can't look at the AP debugs, since the APs are constantly reconnecting.
(C000001WC002) #show ap debug log ap-name AP004034WP006
Module SAPM Client is busy. Please try later
(C000001WC002) #show log system all | include AP004034WP004
Aug 28 07:57:11 :305007: <INFO> |stm| AP AP004034WP004 bootstrapped
Aug 28 07:57:25 :305003: <INFO> |stm| AP AP004034WP004 is down
Aug 28 07:57:25 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP004(10.x.4.33), AP mac 40:e3:d6:c4:5b:74, AP status 2
Aug 28 07:57:25 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP004 already deleted
Aug 28 07:57:43 :305007: <INFO> |stm| AP AP004034WP004 bootstrapped
Aug 28 07:57:57 :305003: <INFO> |stm| AP AP004034WP004 is down
Aug 28 07:57:57 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP004(10.x.4.33), AP mac 40:e3:d6:c4:5b:74, AP status 2
Aug 28 07:57:57 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP004 already deleted
Aug 28 07:58:14 :305007: <INFO> |stm| AP AP004034WP004 bootstrapped
Aug 28 07:58:27 :305003: <INFO> |stm| AP AP004034WP004 is down
Aug 28 07:58:27 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP004(10.x.4.33), AP mac 40:e3:d6:c4:5b:74, AP status 2
Aug 28 07:58:27 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP004 already deleted
Aug 28 07:58:50 :305007: <INFO> |stm| AP AP004034WP004 bootstrapped
Aug 28 07:59:03 :305003: <INFO> |stm| AP AP004034WP004 is down
Aug 28 07:59:03 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP004(10.x.4.33), AP mac 40:e3:d6:c4:5b:74, AP status 2
Aug 28 07:59:03 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP004 already deleted
Aug 28 07:59:20 :305007: <INFO> |stm| AP AP004034WP004 bootstrapped
Aug 28 07:59:33 :305003: <INFO> |stm| AP AP004034WP004 is down
Aug 28 07:59:33 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP004(10.x.4.33), AP mac 40:e3:d6:c4:5b:74, AP status 2
Aug 28 07:59:33 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP004 already deleted
Aug 28 07:59:52 :305007: <INFO> |stm| AP AP004034WP004 bootstrapped
Aug 28 08:00:05 :305003: <INFO> |stm| AP AP004034WP004 is down
Aug 28 08:00:05 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP004(10.x.4.33), AP mac 40:e3:d6:c4:5b:74, AP status 2
Aug 28 08:00:05 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP004 already deleted
Aug 28 08:00:22 :305007: <INFO> |stm| AP AP004034WP004 bootstrapped
Aug 28 08:00:35 :305003: <INFO> |stm| AP AP004034WP004 is down
Aug 28 08:00:35 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP004(10.x.4.33), AP mac 40:e3:d6:c4:5b:74, AP status 2
Aug 28 08:00:35 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP004 already deleted
Aug 28 08:01:00 :305007: <INFO> |stm| AP AP004034WP004 bootstrapped
Aug 28 08:01:13 :305003: <INFO> |stm| AP AP004034WP004 is down
Aug 28 08:01:13 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP004(10.x.4.33), AP mac 40:e3:d6:c4:5b:74, AP status 2
Aug 28 08:01:13 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP004 already deleted
(C000001WC002) #show log system all | include AP004034WP006
Aug 28 07:56:56 :305007: <INFO> |stm| AP AP004034WP006 bootstrapped
Aug 28 07:57:09 :305003: <INFO> |stm| AP AP004034WP006 is down
Aug 28 07:57:09 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP006(10.x.4.35), AP mac 40:e3:d6:c4:5c:16, AP status 2
Aug 28 07:57:09 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP006 already deleted
Aug 28 07:57:32 :305007: <INFO> |stm| AP AP004034WP006 bootstrapped
Aug 28 07:57:45 :305003: <INFO> |stm| AP AP004034WP006 is down
Aug 28 07:57:45 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP006(10.x.4.35), AP mac 40:e3:d6:c4:5c:16, AP status 2
Aug 28 07:57:45 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP006 already deleted
Aug 28 07:58:04 :305007: <INFO> |stm| AP AP004034WP006 bootstrapped
Aug 28 07:58:17 :305003: <INFO> |stm| AP AP004034WP006 is down
Aug 28 07:58:17 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP006(10.x.4.35), AP mac 40:e3:d6:c4:5c:16, AP status 2
Aug 28 07:58:17 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP006 already deleted
Aug 28 07:58:35 :305007: <INFO> |stm| AP AP004034WP006 bootstrapped
Aug 28 07:58:47 :305003: <INFO> |stm| AP AP004034WP006 is down
Aug 28 07:58:47 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP006(10.x.4.35), AP mac 40:e3:d6:c4:5c:16, AP status 2
Aug 28 07:58:47 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP006 already deleted
Aug 28 07:59:09 :305007: <INFO> |stm| AP AP004034WP006 bootstrapped
Aug 28 07:59:23 :305003: <INFO> |stm| AP AP004034WP006 is down
Aug 28 07:59:23 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP006(10.x.4.35), AP mac 40:e3:d6:c4:5c:16, AP status 2
Aug 28 07:59:23 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP006 already deleted
Aug 28 07:59:41 :305007: <INFO> |stm| AP AP004034WP006 bootstrapped
Aug 28 07:59:53 :305003: <INFO> |stm| AP AP004034WP006 is down
Aug 28 07:59:53 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP006(10.x.4.35), AP mac 40:e3:d6:c4:5c:16, AP status 2
Aug 28 07:59:53 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP006 already deleted
Aug 28 08:00:13 :305007: <INFO> |stm| AP AP004034WP006 bootstrapped
Aug 28 08:00:27 :305003: <INFO> |stm| AP AP004034WP006 is down
Aug 28 08:00:27 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP006(10.x.4.35), AP mac 40:e3:d6:c4:5c:16, AP status 2
Aug 28 08:00:27 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP006 already deleted
Aug 28 08:00:44 :305007: <INFO> |stm| AP AP004034WP006 bootstrapped
Aug 28 08:00:57 :305003: <INFO> |stm| AP AP004034WP006 is down
Aug 28 08:00:57 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP006(10.x.4.35), AP mac 40:e3:d6:c4:5c:16, AP status 2
Aug 28 08:00:57 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP006 already deleted
Aug 28 08:01:15 :305007: <INFO> |stm| AP AP004034WP006 bootstrapped
Aug 28 08:01:29 :305003: <INFO> |stm| AP AP004034WP006 is down
Aug 28 08:01:29 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP006(10.x.4.35), AP mac 40:e3:d6:c4:5c:16, AP status 2
Aug 28 08:01:29 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP006 already deleted
Aug 28 08:01:56 :305007: <INFO> |stm| AP AP004034WP006 bootstrapped
(C000001WC002) #show log system all | include AP004007WP001
Aug 28 07:58:31 KERNEL(AP004007WP001@10.x.4.19): [24007.766047] ASAP_FIREWALL_REM_MCAST: 1 1 0 0, no vap
Aug 28 07:58:50 :305007: <INFO> |stm| AP AP004007WP001 bootstrapped
Aug 28 07:59:03 :305003: <INFO> |stm| AP AP004007WP001 is down
Aug 28 07:59:03 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004007WP001(10.x.4.19), AP mac 40:e3:d6:c4:5e:14, AP status 2
Aug 28 07:59:03 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004007WP001 already deleted
Aug 28 07:59:15 :311004: <WARN> |AP AP004007WP001@10.x.4.19 sapd| Missed 8 heartbeats; rebootstrapping
Aug 28 07:59:18 KERNEL(AP004007WP001@10.x.4.19): [24054.357025] ASAP_FIREWALL_REM_MCAST: 1 1 0 0, no vap
Aug 28 07:59:31 :305007: <INFO> |stm| AP AP004007WP001 bootstrapped
Aug 28 07:59:43 :305003: <INFO> |stm| AP AP004007WP001 is down
Aug 28 07:59:43 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004007WP001(10.x.4.19), AP mac 40:e3:d6:c4:5e:14, AP status 2
Aug 28 07:59:43 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004007WP001 already deleted
Aug 28 07:59:57 :311004: <WARN> |AP AP004007WP001@10.x.4.19 sapd| Missed 8 heartbeats; rebootstrapping
Aug 28 07:59:59 KERNEL(AP004007WP001@10.x.4.19): [24095.996832] ASAP_FIREWALL_REM_MCAST: 1 1 0 0, no vap
Aug 28 08:00:15 :305007: <INFO> |stm| AP AP004007WP001 bootstrapped
Aug 28 08:00:27 :305003: <INFO> |stm| AP AP004007WP001 is down
Aug 28 08:00:27 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004007WP001(10.x.4.19), AP mac 40:e3:d6:c4:5e:14, AP status 2
Aug 28 08:00:27 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004007WP001 already deleted
Aug 28 08:00:41 :311004: <WARN> |AP AP004007WP001@10.x.4.19 sapd| Missed 8 heartbeats; rebootstrapping
Aug 28 08:00:43 KERNEL(AP004007WP001@10.x.4.19): [24139.777733] ASAP_FIREWALL_REM_MCAST: 1 1 0 0, no vap
Aug 28 08:00:56 :305007: <INFO> |stm| AP AP004007WP001 bootstrapped
Aug 28 08:01:09 :305003: <INFO> |stm| AP AP004007WP001 is down
Aug 28 08:01:09 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004007WP001(10.x.4.19), AP mac 40:e3:d6:c4:5e:14, AP status 2
Aug 28 08:01:09 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004007WP001 already deleted
Aug 28 08:01:22 :311004: <WARN> |AP AP004007WP001@10.x.4.19 sapd| Missed 8 heartbeats; rebootstrapping
Aug 28 08:01:24 KERNEL(AP004007WP001@10.x.4.19): [24180.645200] ASAP_FIREWALL_REM_MCAST: 1 1 0 0, no vap
Aug 28 08:01:48 :305007: <INFO> |stm| AP AP004007WP001 bootstrapped
Aug 28 08:02:01 :305003: <INFO> |stm| AP AP004007WP001 is down
Aug 28 08:02:01 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004007WP001(10.x.4.19), AP mac 40:e3:d6:c4:5e:14, AP status 2
Aug 28 08:02:01 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004007WP001 already deleted
Aug 28 08:02:14 :311004: <WARN> |AP AP004007WP001@10.x.4.19 sapd| Missed 8 heartbeats; rebootstrapping
Aug 28 08:02:16 KERNEL(AP004007WP001@10.x.4.19): [24233.064751] ASAP_FIREWALL_REM_MCAST: 1 1 0 0, no vap
Aug 28 08:02:26 :305007: <INFO> |stm| AP AP004007WP001 bootstrapped