Wireless Access

Reply
Frequent Contributor I
Posts: 73
Registered: ‎08-16-2011

Is it possible to log AP missed heartbeats?

Hello, 

 

Running ArubaOS 6.1.3.7.  Trying to troubleshoot APs behind a mesh link that randomly bootstrap & occasionally reboot.  Wondering if it is possible to configure logging to collect missed heartbeats from APs?

 

My guess is that, based on what help says, the process is "approc" ; especially given that "sapd" & "sapm" don't appear to be options presented by the help system.  

 

What would be the proper level?

 

     # logging level <NOTICE> | <INFO> | <DEBUG> system process approc subcat ap

 

What is the differences between that & specifying :

 

     # logging level <NOTICE> | <INFO> | <DEBUG> ap-debug <name> subcat all process approc

 

TIA, 

 

--Raf

 

--Raf
Guru Elite
Posts: 21,010
Registered: ‎03-29-2007

Re: Is it possible to log AP missed heartbeats?

[ Edited ]

Well, if you miss a heartbeat, that means you don't have connectivity.  Normally an access point sends the syslog that it has missed "x" amount of heartbeats and that it is rebootstrapping, so it needs connectivity to send that message.  No individual messages are sent for an individual miss of an heartbeat.  By default, that it is in the system log, without configuring anything:

 

 

Apr  9 08:12:49  sapd[560]: <311005> <WARN> |AP AP34@10.50.1.89 sapd|  Missed 8 heartbeats on wired AP interface 3; rebootstrapping

 



Colin Joseph
Aruba Customer Engineering

Looking for an Answer? Search the Community Knowledge Base Here: Community Knowledge Base

MVP
Posts: 4,269
Registered: ‎07-20-2011

Re: Is it possible to log AP missed heartbeats?

You can check if other APs are bootstrapping or rebooting by using this command :

 

show ap debug counters

 

If you are seeing this behavior a lot you might have to take a look at your network layout / traffic / Layer 1 .

 

Another thing you could do is increase the heartbeat value from 8 to 25 under the ap system-profile 

Thank you

Victor Fabian
Lead Mobility Engineer @ Integration Partners
AMFX | ACMX | ACDX | ACCX | CWAP | CWDP | CWNA
Frequent Contributor I
Posts: 73
Registered: ‎08-16-2011

Re: Is it possible to log AP missed heartbeats?

Thanks,

These messages (sapd) don't seem to make it to my remote log host. I see a few sapd warnings on power as far back as the logs on the controller go, however I do not see the same messages in my remote log.

 

#show logging facility      

Remote Logging Facility is local1


#show logging server   

Remote Server: 192.168.36.64

FACILITY MAPPING TABLE
----------------------
local-facility  severity  remote-facility
--------------  --------  ---------------
All             All       local1



#show logging level verbose 

LOGGING LEVELS
--------------
Facility  Level          Sub Category  Process
--------  -----          ------------  -------
network   warnings       N/A           N/A
network   debugging      dhcp          N/A
security  informational  N/A           N/A
security  warnings       aaa           N/A
security  informational  dot1x         N/A
security  warnings       ids           N/A
security  warnings       ids-ap        N/A
system    informational  N/A           N/A
user      informational  N/A           N/A
user      informational  dot1x         N/A
wireless  informational  N/A           N/A

 

I'll have to think a bit more about why the controller does not log missed heartbeats via sapm. More importantly, why these messages aren't making it to my log host.

 

More on the specifics of my issue in a bit.

 

 

--Raf
MVP
Posts: 4,269
Registered: ‎07-20-2011

Re: Is it possible to log AP missed heartbeats?

In the log message cjoseph posted you can see that these come as warning , so you probably should set the logging system level warning instead of informational.

 

Another thing is that you might need to do this from the master controller, I usually see these messages on the master controller.

 

Hope this helps

Thank you

Victor Fabian
Lead Mobility Engineer @ Integration Partners
AMFX | ACMX | ACDX | ACCX | CWAP | CWDP | CWNA
Frequent Contributor I
Posts: 73
Registered: ‎08-16-2011

Re: Is it possible to log AP missed heartbeats?

So here is what we're experiencing...

 

We have 4 APs & a Switch behind a mesh link in building that has no fiber service (which I'm told requires abatement). 

 

AFAICT, the mesh links don't report any problems, but the APs behind the link seem to bootstrap, anywhere between 3 - 6 times a day; we have Airwave sending us AP Device Up/Down messages.

 

#show ap debug counters group armory   

AP Counters
-----------
Name         Group   IP Address      Configs Sent  Configs Acked  AP Boots Sent  AP Boots Acked  Bootstraps (Total)  Reboots
----         -----   ----------      ------------  -------------  -------------  --------------  ------------------  -------
army-1-2d52  armory  192.168.55.114  5             5              0              0               1          (1    )  0
army-1-2df8  armory  192.168.54.116  5             5              0              0               1          (1    )  0
army-1-2e08  armory  192.168.54.115  20            18             0              0               1          (1    )  0
army-1-2e20  armory  192.168.54.114  8             7              0              0               1          (1    )  0
Total APs :4

 

Here are the counters from the Mesh Point & Portal.  The Point & Portal are on a different controller than the APs.

 

#show ap mesh topology aruba-mesh 

Mesh Cluster Name: aruba-mesh
-----------------------------
Name                Mesh Role   Parent              Path Cost  Node Cost  Link Cost  Hop Count  RSSI  Rate Tx/Rx  Last Update  Uplink Age    #Children
----                ---------   ------              ---------  ---------  ---------  ---------  ----  ----------  -----------  ----------    ---------
armory-mesh-point   Point (N)   nesbit-mesh-portal  11         0          10         1          26    54/48       1m:27s       1m:31s        0
nesbit-mesh-portal  Portal (N)  -                   0          1          0          0          0     -           1m:31s       3d:4h:1m:34s  1

Total APs :2
(R): Recovery AP. (N): 11N Enabled. For Portals 'Uplink Age' equals uptime.


#show ap mesh active aruba-mesh     

Mesh Cluster Name: aruba-mesh
-----------------------------
Name                Group            IP Address      BSSID              Band/Ch/EIRP/MaxEIRP  MTU   Enet Ports     Mesh Role  Parent              #Children  AP Type  Uptime
----                -----            ----------      -----              --------------------  ---   ----------     ---------  ------              ---------  -------  ------
armory-mesh-point   mesh-aps-armory  192.168.55.148  d8:c7:c8:b3:9b:50  802.11a/149/20/20           Tunnel/Tunnel  Point      nesbit-mesh-portal  0          124      3d:3h:59m:43s
nesbit-mesh-portal  mesh-aps-armory  192.168.55.115  d8:c7:c8:b3:9f:d0  802.11a/149/21/21     1500  -/Off          Portal     -                   1          124      3d:3h:54m:25s

Total APs :2

#show ap debug counters group mesh-aps-armory AP Counters ----------- Name Group IP Address Configs Sent Configs Acked AP Boots Sent AP Boots Acked Bootstraps (Total) Reboots ---- ----- ---------- ------------ ------------- ------------- -------------- ------------------ ------- armory-mesh-point mesh-aps-armory 192.168.55.148 7 7 0 0 2 (2 ) 1 nesbit-mesh-portal mesh-aps-armory 192.168.55.115 2 2 0 0 1 (2 ) 2 Total APs :2

 I'm now gonna bombard you w/ some info...

 

#show ap mesh debug counters ap-name armory-mesh-point

Mesh Packet Counters
--------------------
Interface  Echo Sent  Echo Recv  Probe Req  Probe Resp   Assoc Req   Assoc Resp   Assoc Fail  Link up/down  Resel.  Switch  Other Mgmt
---------  ---------  ---------  ---------  ----------   ---------   ----------   ----------  ------------  ------  ------  ----------
Parent     0          0          0          0            0           0            0           0             -       -       0
Child      269136     267435     634        119(119 HT)  182(56 HT)  177 (56 HT)  5           118           0       0       2583082

Received Packet Statistics: Total 3119036, Mgmt 2586652 (dropped non-mesh 0), Data 532384 (dropped unassociated 0)HT: pns=0 ans=0 pnr=119 ars=56 arr=0 anr=56

Recovery Profile Usage Counters
-------------------------------
Item                         Value
----                         -----
Enter recovery mode          1
Exit recovery mode           1
Total connections to switch  0

Mesh loop-prevention Sequence No.:272792

Mesh timer ticks:273118

Change-record: HT-link renegotiation, linkdown:2s, linkup:2s, previous portal:d8:c7:c8:b3:9f:d0, previous parent: d8:c7:c8:b3:9f:d0

#show ap mesh debug counters ap-name nesbit-mesh-portal

Mesh Packet Counters
--------------------
Interface  Echo Sent  Echo Recv  Probe Req  Probe Resp   Assoc Req   Assoc Resp  Assoc Fail  Link up/down  Resel.  Switch  Other Mgmt
---------  ---------  ---------  ---------  ----------   ---------   ----------  ----------  ------------  ------  ------  ----------
Parent     270679     266753     602        602(602 HT)  177(56 HT)  177(56 HT)  0           176           -       -       0

Received Packet Statistics: Total 3126998, Mgmt 2592799 (dropped non-mesh 0), Data 534199 (dropped unassociated 0)HT: pns=602 ans=56 pnr=0 ars=0 arr=56 anr=0

Recovery Profile Usage Counters
-------------------------------
Item                         Value
----                         -----
Enter recovery mode          0
Exit recovery mode           0
Total connections to switch  0

Mesh loop-prevention Sequence No.:272806

Mesh timer ticks:272805

 

I'll need to sort through the profiles to make sure we're looking at the right threshold setting for these APs. 

 

If you see anything glaring, or can make any suggestions, I'm all ears.  Additionally, we're working on moving the mesh point (or portal) so that we have a clear line of sight.  I believe there is a tree in the way at the moment, and being that its spring, it's about to come alive w/ folliage.

 

More as things develop.  Thanks,

 

--Raf
Frequent Contributor I
Posts: 73
Registered: ‎08-16-2011

Re: Is it possible to log AP missed heartbeats?

Thanks for the suggestion...

 

I am under the impression that the information messages would include all levels below it.  The 6.1 CLI Resource Guide (CRG) seems to confirm that.

 

"There are eight logging severity levels, each with its associated types of messages. Each level also includes the levels below it. For example, if you set the logging level to informational (6), all messages from level 0 through level 5 (from emergencies through notifications) are also logged. The warnings severity level is set by default for all message categories and subcategories."

 

I'm going to search through the Master's logs to see if there is anything of use.  If I can't find anything, I may have to explicitly set it for system approc messages.  However, if it is as cjoseph has indicated and the APs aren't able to send notification due to a loss of connectivity, I'm not sure whatelse can be done about it.  At the very least, I should be able to see messages after they restore connectivity. 

 

Thanks,

 

--Raf
MVP
Posts: 4,269
Registered: ‎07-20-2011

Re: Is it possible to log AP missed heartbeats?

 

Can you run the following commands on the mesh portal and point to determine if there's any RF issues with the 802.11a band :

 

show  ap arm rf-summary ap-name 

 

show ap arm history ap-name

 

Please share 

 

 

 

 

Thank you

Victor Fabian
Lead Mobility Engineer @ Integration Partners
AMFX | ACMX | ACDX | ACCX | CWAP | CWDP | CWNA
Frequent Contributor I
Posts: 73
Registered: ‎08-16-2011

Re: Is it possible to log AP missed heartbeats?

Here is the output for the rf-summary :

 

#show  ap arm rf-summary ap-name armory-mesh-point

Channel Summary
---------------
channel  retry  phy-err  mac-err  noise  cov-idx  intf_idx
-------  -----  -------  -------  -----  -------  --------
161      0      0        0        92     0/0      0/0//0/0
1        0      0        0        0      0/0      0/0//0/0
48       0      0        0        0      0/0      0/0//0/0
165      0      0        0        91     0/0      0/0//0/0
5        0      0        0        0      0/0      0/0//0/0
6        0      0        0        0      0/0      0/0//0/0
7        0      0        0        0      0/0      0/0//0/0
11       0      0        0        0      0/0      0/0//0/0
149      17     0        8        94     16/0     71/0//0/0
36       0      0        0        0      0/0      0/0//0/0
153      0      0        0        96     0/0      0/0//0/0
40       0      0        0        0      0/0      0/0//0/0
157      0      0        0        92     0/0      0/0//0/0
44       0      0        0        0      0/0      0/0//0/0
HT Channel Summary
------------------
channel_pair  Pairwise_intf_index
------------  -------------------
1-5           0
7-11          0
149-153       71
36-40         0
157-161       0
44-48         0

Interface Name           :wifi0
Current ARM Assignment   :149/127
Covered channels a/g     :0/0
Free channels a/g        :9/0
ARM Edge State           :enable
Last check channel/pwr   :3d:4h:33m:25s/3d:4h:33m:25s
Last change channel/pwr  :3d:4h:33m:25s/3d:4h:33m:25s
Next Check channel/pwr   :0s/0s

Interface Name           :wifi1
Current ARM Assignment   :11/127
Covered channels a/g     :0/1
Free channels a/g        :0/2
ARM Edge State           :disable
Last check channel/pwr   :3d:4h:33m:25s/3d:4h:33m:25s
Last change channel/pwr  :3d:4h:33m:25s/3d:4h:33m:25s
Next Check channel/pwr   :0s/0s


#show  ap arm rf-summary ap-name nesbit-mesh-portal

Channel Summary
---------------
channel  retry  phy-err  mac-err  noise  cov-idx  intf_idx
-------  -----  -------  -------  -----  -------  --------
161      0      0        0        89     0/0      25/17//10/0
1        0      0        0        0      0/0      0/0//0/0
48       0      0        0        86     4/0      22/0//0/0
165      0      0        0        90     0/0      0/10//0/4
5        0      0        0        0      0/0      0/0//0/0
6        0      0        0        0      0/0      0/0//0/0
7        0      0        0        0      0/0      0/0//0/0
11       0      0        0        0      0/0      0/0//0/0
149      6      0        4        95     22/0     77/0//5/0
36       0      0        0        86     7/0      64/0//0/0
153      0      0        0        90     0/0      0/43//0/0
40       0      0        0        86     0/0      0/21//0/0
157      0      0        0        88     8/0      53/8//1/4
44       0      0        0        86     0/0      0/7//0/0
HT Channel Summary
------------------
channel_pair  Pairwise_intf_index
------------  -------------------
1-5           0
7-11          0
149-153       125
36-40         85
157-161       118
44-48         29

Interface Name           :wifi0
Current ARM Assignment   :149/127
Covered channels a/g     :4/0
Free channels a/g        :5/0
ARM Edge State           :enable
Last check channel/pwr   :3d:4h:33m:14s/3d:4h:33m:14s
Last change channel/pwr  :3d:4h:33m:14s/3d:4h:33m:14s
Next Check channel/pwr   :0s/0s

Interface Name           :wifi1
Current ARM Assignment   :11/127
Covered channels a/g     :0/1
Free channels a/g        :0/2
ARM Edge State           :disable
Last check channel/pwr   :3d:4h:33m:14s/3d:4h:33m:14s
Last change channel/pwr  :3d:4h:33m:14s/3d:4h:33m:14s
Next Check channel/pwr   :0s/0s

 

There isn't any ARM history.  I believe we have disabled ARM for the two.  Below are the profiles used:

 

#show ap-group mesh-aps-armory

AP group "mesh-aps-armory"
--------------------------
Parameter                                Value
---------                                -----
Virtual AP                               N/A
802.11a radio profile                    Mesh_outdoor
802.11g radio profile                    no_arm_mesh
Ethernet interface 0 port configuration  default
Ethernet interface 1 port configuration  default
Ethernet interface 2 port configuration  shutdown
Ethernet interface 3 port configuration  shutdown
Ethernet interface 4 port configuration  shutdown
AP system profile                        spare4
VoIP Call Admission Control profile      default
802.11a Traffic Management profile       N/A
802.11g Traffic Management profile       N/A
Regulatory Domain profile                default
RF Optimization profile                  default
RF Event Thresholds profile              default
IDS profile                              default
Mesh Radio profile                       armory-mesh-radio
Mesh Cluster profile                     armory-mesh-cluster  priority 1
Provisioning profile                     N/A
AP authorization profile                 N/A



#show rf dot11g-radio-profile no_arm_mesh

802.11g radio profile "no_arm_mesh"
-----------------------------------
Parameter                                      Value
---------                                      -----
Radio enable                                   Enabled
Mode                                           ap-mode
High throughput enable (radio)                 Disabled
Channel                                        N/A
Beacon Period                                  100 msec
Non-Wi-Fi Interference Immunity                2
Beacon Regulate                                Disabled
Transmit EIRP                                  127 dBm
Advertise 802.11d and 802.11h Capabilities     Disabled
TPC Power                                      15 dBm
Spectrum load balancing                        Disabled
Spectrum Load balancing mode                   channel
Spectrum load balancing update interval (sec)  30 seconds
Spectrum load balancing threshold (%)          20 percent
Advertized regulatory max EIRP                 0
Spectrum Load Balancing domain                 N/A
RX Sensitivity Tuning Based Channel Reuse      disable
RX Sensitivity Threshold                       0 -dBm
Enable CSA                                     Disabled
CSA Count                                      4
Reduce Cell Size (Rx Sensitivity)              0 dB
Management Frame Throttle interval             1 sec
Management Frame Throttle Limit                20
ARM/WIDS Override                              Disabled
Protection for 802.11b Clients                 Enabled
Adaptive Radio Management (ARM) Profile        no_arm_enable_MeSh
High-throughput Radio Profile                  default-g
Maximum Distance                               0 meters
Spectrum Monitoring                            Disabled
Spectrum Monitoring Profile                    default-g
AM Scanning Profile                            default


#show rf dot11a-radio-profile Mesh_outdoor

802.11a radio profile "Mesh_outdoor"
------------------------------------
Parameter                                      Value
---------                                      -----
Radio enable                                   Enabled
Mode                                           ap-mode
High throughput enable (radio)                 Disabled
Channel                                        149
Beacon Period                                  100 msec
Non-Wi-Fi Interference Immunity                2
Beacon Regulate                                Disabled
Transmit EIRP                                  127 dBm
Advertise 802.11d and 802.11h Capabilities     Disabled
TPC Power                                      15 dBm
Spectrum load balancing                        Disabled
Spectrum Load balancing mode                   channel
Spectrum load balancing update interval (sec)  30 seconds
Spectrum load balancing threshold (%)          20 percent
Advertized regulatory max EIRP                 0
Spectrum Load Balancing domain                 N/A
RX Sensitivity Tuning Based Channel Reuse      disable
RX Sensitivity Threshold                       0 -dBm
Enable CSA                                     Disabled
CSA Count                                      4
Reduce Cell Size (Rx Sensitivity)              0 dB
Management Frame Throttle interval             1 sec
Management Frame Throttle Limit                20
ARM/WIDS Override                              Disabled
Adaptive Radio Management (ARM) Profile        no_arm_enable_MeSh
High-throughput Radio Profile                  default-a
Maximum Distance                               0 meters
Spectrum Monitoring                            Disabled
Spectrum Monitoring Profile                    default-a
AM Scanning Profile                            default


#show rf arm-profile no_arm_enable_MeSh

Adaptive Radio Management (ARM) profile "no_arm_enable_MeSh"
------------------------------------------------------------
Parameter                         Value
---------                         -----
Assignment                        disable
Allowed bands for 40MHz channels  a-only
Client Aware                      Enabled
Max Tx EIRP                       127 dBm
Min Tx EIRP                       33 dBm
Multi Band Scan                   Enabled
Rogue AP Aware                    Disabled
Scan Interval                     10 sec
Active Scan                       Disabled
Scanning                          Enabled
Scan Time                         110 msec
VoIP Aware Scan                   Disabled
Power Save Aware Scan             Disabled
Video Aware Scan                  Enabled
Ideal Coverage Index              10
Acceptable Coverage Index         4
Free Channel Index                25
Backoff Time                      240 sec
Error Rate Threshold              50 %
Error Rate Wait Time              30 sec
Noise Threshold                   75 -dBm
Noise Wait Time                   120 sec
Minimum Scan Time                 8
Load aware Scan Threshold         1250000 Bps
Mode Aware Arm                    Disabled
Scan Mode                         all-reg-domain

 

Thanks,

 

--Raf
MVP
Posts: 4,269
Registered: ‎07-20-2011

Re: Is it possible to log AP missed heartbeats?

 

I noticed you disabled HT too ?

 

Based on the rf output you provided it doesn't look like there any RF issues on the 802.11a at least at the moment you took those.

 

You may want to increase the Bootstrap threshold to 25 under AP system profile .

Thank you

Victor Fabian
Lead Mobility Engineer @ Integration Partners
AMFX | ACMX | ACDX | ACCX | CWAP | CWDP | CWNA
Search Airheads
Showing results for 
Search instead for 
Did you mean: