Wireless Access

Reply
Regular Contributor I

Troubleshooting AP down errors

What is the best way to get detailed information on why a small number of APs at a remote branch are going down? Assume the network itself has been fully investigated, and I'm just looking to understand from the AP's and the controller's point of view what is going on. 

 

APs are CAPs in tunnel mode on 6.4.3 7200. 

APs connect over a private WAN to controllers some distance away (20ms+).

HA has been disabled, so it's just basic LMS traffic

We have Airwave reporting APs down only some of the time, but by checking the Network Summary in the master controller one can see they are bouncing far more often.

 

I would like to know what particular packets have to be missed for the controller to consider the AP down. Also how which commands/debugs to clearly capture this information.

 

thanks

 

edit: if it's not clear this is an intermittent, so APs are constantly 'bouncing' (but not rebooting) through the day, status changing minute to minute

 

edit: no physical access to AP or controller

 

edit: some system logs. note I can't look at the AP debugs, since the APs are constantly reconnecting.

 

(C000001WC002) #show ap debug log ap-name AP004034WP006
Module SAPM Client is busy. Please try later

(C000001WC002) #show log system all | include AP004034WP004 Aug 28 07:57:11 :305007: <INFO> |stm| AP AP004034WP004 bootstrapped Aug 28 07:57:25 :305003: <INFO> |stm| AP AP004034WP004 is down Aug 28 07:57:25 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP004(10.x.4.33), AP mac 40:e3:d6:c4:5b:74, AP status 2 Aug 28 07:57:25 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP004 already deleted Aug 28 07:57:43 :305007: <INFO> |stm| AP AP004034WP004 bootstrapped Aug 28 07:57:57 :305003: <INFO> |stm| AP AP004034WP004 is down Aug 28 07:57:57 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP004(10.x.4.33), AP mac 40:e3:d6:c4:5b:74, AP status 2 Aug 28 07:57:57 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP004 already deleted Aug 28 07:58:14 :305007: <INFO> |stm| AP AP004034WP004 bootstrapped Aug 28 07:58:27 :305003: <INFO> |stm| AP AP004034WP004 is down Aug 28 07:58:27 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP004(10.x.4.33), AP mac 40:e3:d6:c4:5b:74, AP status 2 Aug 28 07:58:27 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP004 already deleted Aug 28 07:58:50 :305007: <INFO> |stm| AP AP004034WP004 bootstrapped Aug 28 07:59:03 :305003: <INFO> |stm| AP AP004034WP004 is down Aug 28 07:59:03 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP004(10.x.4.33), AP mac 40:e3:d6:c4:5b:74, AP status 2 Aug 28 07:59:03 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP004 already deleted Aug 28 07:59:20 :305007: <INFO> |stm| AP AP004034WP004 bootstrapped Aug 28 07:59:33 :305003: <INFO> |stm| AP AP004034WP004 is down Aug 28 07:59:33 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP004(10.x.4.33), AP mac 40:e3:d6:c4:5b:74, AP status 2 Aug 28 07:59:33 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP004 already deleted Aug 28 07:59:52 :305007: <INFO> |stm| AP AP004034WP004 bootstrapped Aug 28 08:00:05 :305003: <INFO> |stm| AP AP004034WP004 is down Aug 28 08:00:05 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP004(10.x.4.33), AP mac 40:e3:d6:c4:5b:74, AP status 2 Aug 28 08:00:05 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP004 already deleted Aug 28 08:00:22 :305007: <INFO> |stm| AP AP004034WP004 bootstrapped Aug 28 08:00:35 :305003: <INFO> |stm| AP AP004034WP004 is down Aug 28 08:00:35 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP004(10.x.4.33), AP mac 40:e3:d6:c4:5b:74, AP status 2 Aug 28 08:00:35 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP004 already deleted Aug 28 08:01:00 :305007: <INFO> |stm| AP AP004034WP004 bootstrapped Aug 28 08:01:13 :305003: <INFO> |stm| AP AP004034WP004 is down Aug 28 08:01:13 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP004(10.x.4.33), AP mac 40:e3:d6:c4:5b:74, AP status 2 Aug 28 08:01:13 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP004 already deleted (C000001WC002) #show log system all | include AP004034WP006 Aug 28 07:56:56 :305007: <INFO> |stm| AP AP004034WP006 bootstrapped Aug 28 07:57:09 :305003: <INFO> |stm| AP AP004034WP006 is down Aug 28 07:57:09 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP006(10.x.4.35), AP mac 40:e3:d6:c4:5c:16, AP status 2 Aug 28 07:57:09 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP006 already deleted Aug 28 07:57:32 :305007: <INFO> |stm| AP AP004034WP006 bootstrapped Aug 28 07:57:45 :305003: <INFO> |stm| AP AP004034WP006 is down Aug 28 07:57:45 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP006(10.x.4.35), AP mac 40:e3:d6:c4:5c:16, AP status 2 Aug 28 07:57:45 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP006 already deleted Aug 28 07:58:04 :305007: <INFO> |stm| AP AP004034WP006 bootstrapped Aug 28 07:58:17 :305003: <INFO> |stm| AP AP004034WP006 is down Aug 28 07:58:17 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP006(10.x.4.35), AP mac 40:e3:d6:c4:5c:16, AP status 2 Aug 28 07:58:17 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP006 already deleted Aug 28 07:58:35 :305007: <INFO> |stm| AP AP004034WP006 bootstrapped Aug 28 07:58:47 :305003: <INFO> |stm| AP AP004034WP006 is down Aug 28 07:58:47 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP006(10.x.4.35), AP mac 40:e3:d6:c4:5c:16, AP status 2 Aug 28 07:58:47 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP006 already deleted Aug 28 07:59:09 :305007: <INFO> |stm| AP AP004034WP006 bootstrapped Aug 28 07:59:23 :305003: <INFO> |stm| AP AP004034WP006 is down Aug 28 07:59:23 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP006(10.x.4.35), AP mac 40:e3:d6:c4:5c:16, AP status 2 Aug 28 07:59:23 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP006 already deleted Aug 28 07:59:41 :305007: <INFO> |stm| AP AP004034WP006 bootstrapped Aug 28 07:59:53 :305003: <INFO> |stm| AP AP004034WP006 is down Aug 28 07:59:53 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP006(10.x.4.35), AP mac 40:e3:d6:c4:5c:16, AP status 2 Aug 28 07:59:53 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP006 already deleted Aug 28 08:00:13 :305007: <INFO> |stm| AP AP004034WP006 bootstrapped Aug 28 08:00:27 :305003: <INFO> |stm| AP AP004034WP006 is down Aug 28 08:00:27 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP006(10.x.4.35), AP mac 40:e3:d6:c4:5c:16, AP status 2 Aug 28 08:00:27 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP006 already deleted Aug 28 08:00:44 :305007: <INFO> |stm| AP AP004034WP006 bootstrapped Aug 28 08:00:57 :305003: <INFO> |stm| AP AP004034WP006 is down Aug 28 08:00:57 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP006(10.x.4.35), AP mac 40:e3:d6:c4:5c:16, AP status 2 Aug 28 08:00:57 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP006 already deleted Aug 28 08:01:15 :305007: <INFO> |stm| AP AP004034WP006 bootstrapped Aug 28 08:01:29 :305003: <INFO> |stm| AP AP004034WP006 is down Aug 28 08:01:29 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004034WP006(10.x.4.35), AP mac 40:e3:d6:c4:5c:16, AP status 2 Aug 28 08:01:29 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004034WP006 already deleted Aug 28 08:01:56 :305007: <INFO> |stm| AP AP004034WP006 bootstrapped (C000001WC002) #show log system all | include AP004007WP001 Aug 28 07:58:31 KERNEL(AP004007WP001@10.x.4.19): [24007.766047] ASAP_FIREWALL_REM_MCAST: 1 1 0 0, no vap Aug 28 07:58:50 :305007: <INFO> |stm| AP AP004007WP001 bootstrapped Aug 28 07:59:03 :305003: <INFO> |stm| AP AP004007WP001 is down Aug 28 07:59:03 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004007WP001(10.x.4.19), AP mac 40:e3:d6:c4:5e:14, AP status 2 Aug 28 07:59:03 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004007WP001 already deleted Aug 28 07:59:15 :311004: <WARN> |AP AP004007WP001@10.x.4.19 sapd| Missed 8 heartbeats; rebootstrapping Aug 28 07:59:18 KERNEL(AP004007WP001@10.x.4.19): [24054.357025] ASAP_FIREWALL_REM_MCAST: 1 1 0 0, no vap Aug 28 07:59:31 :305007: <INFO> |stm| AP AP004007WP001 bootstrapped Aug 28 07:59:43 :305003: <INFO> |stm| AP AP004007WP001 is down Aug 28 07:59:43 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004007WP001(10.x.4.19), AP mac 40:e3:d6:c4:5e:14, AP status 2 Aug 28 07:59:43 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004007WP001 already deleted Aug 28 07:59:57 :311004: <WARN> |AP AP004007WP001@10.x.4.19 sapd| Missed 8 heartbeats; rebootstrapping Aug 28 07:59:59 KERNEL(AP004007WP001@10.x.4.19): [24095.996832] ASAP_FIREWALL_REM_MCAST: 1 1 0 0, no vap Aug 28 08:00:15 :305007: <INFO> |stm| AP AP004007WP001 bootstrapped Aug 28 08:00:27 :305003: <INFO> |stm| AP AP004007WP001 is down Aug 28 08:00:27 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004007WP001(10.x.4.19), AP mac 40:e3:d6:c4:5e:14, AP status 2 Aug 28 08:00:27 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004007WP001 already deleted Aug 28 08:00:41 :311004: <WARN> |AP AP004007WP001@10.x.4.19 sapd| Missed 8 heartbeats; rebootstrapping Aug 28 08:00:43 KERNEL(AP004007WP001@10.x.4.19): [24139.777733] ASAP_FIREWALL_REM_MCAST: 1 1 0 0, no vap Aug 28 08:00:56 :305007: <INFO> |stm| AP AP004007WP001 bootstrapped Aug 28 08:01:09 :305003: <INFO> |stm| AP AP004007WP001 is down Aug 28 08:01:09 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004007WP001(10.x.4.19), AP mac 40:e3:d6:c4:5e:14, AP status 2 Aug 28 08:01:09 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004007WP001 already deleted Aug 28 08:01:22 :311004: <WARN> |AP AP004007WP001@10.x.4.19 sapd| Missed 8 heartbeats; rebootstrapping Aug 28 08:01:24 KERNEL(AP004007WP001@10.x.4.19): [24180.645200] ASAP_FIREWALL_REM_MCAST: 1 1 0 0, no vap Aug 28 08:01:48 :305007: <INFO> |stm| AP AP004007WP001 bootstrapped Aug 28 08:02:01 :305003: <INFO> |stm| AP AP004007WP001 is down Aug 28 08:02:01 :343005: <INFO> |mdns| mdns_parse_sapm_packet 939 PAPI Recv from sapm : AP name(ip) AP004007WP001(10.x.4.19), AP mac 40:e3:d6:c4:5e:14, AP status 2 Aug 28 08:02:01 :343005: <INFO> |mdns| mdns_del_ap_info_tbl 304 ap_table:ap entry for AP004007WP001 already deleted Aug 28 08:02:14 :311004: <WARN> |AP AP004007WP001@10.x.4.19 sapd| Missed 8 heartbeats; rebootstrapping Aug 28 08:02:16 KERNEL(AP004007WP001@10.x.4.19): [24233.064751] ASAP_FIREWALL_REM_MCAST: 1 1 0 0, no vap Aug 28 08:02:26 :305007: <INFO> |stm| AP AP004007WP001 bootstrapped

 


--
ACMA ACMP
Search Airheads
cancel
Showing results for 
Search instead for 
Did you mean: