Wireless Access

last person joined: 14 hours ago 

Access network design for branch, remote, outdoor, and campus locations with HPE Aruba Networking access points and mobility controllers.
Expand all | Collapse all

Is it possible to log AP missed heartbeats?

This thread has been viewed 23 times
  • 1.  Is it possible to log AP missed heartbeats?

    Posted Apr 15, 2013 04:33 PM

    Hello, 

     

    Running ArubaOS 6.1.3.7.  Trying to troubleshoot APs behind a mesh link that randomly bootstrap & occasionally reboot.  Wondering if it is possible to configure logging to collect missed heartbeats from APs?

     

    My guess is that, based on what help says, the process is "approc" ; especially given that "sapd" & "sapm" don't appear to be options presented by the help system.  

     

    What would be the proper level?

     

         # logging level <NOTICE> | <INFO> | <DEBUG> system process approc subcat ap

     

    What is the differences between that & specifying :

     

         # logging level <NOTICE> | <INFO> | <DEBUG> ap-debug <name> subcat all process approc

     

    TIA, 

     

    --Raf

     



  • 2.  RE: Is it possible to log AP missed heartbeats?

    EMPLOYEE
    Posted Apr 15, 2013 07:46 PM

    Well, if you miss a heartbeat, that means you don't have connectivity.  Normally an access point sends the syslog that it has missed "x" amount of heartbeats and that it is rebootstrapping, so it needs connectivity to send that message.  No individual messages are sent for an individual miss of an heartbeat.  By default, that it is in the system log, without configuring anything:

     

     

    Apr  9 08:12:49  sapd[560]: <311005> <WARN> |AP AP34@10.50.1.89 sapd|  Missed 8 heartbeats on wired AP interface 3; rebootstrapping
    
    

     



  • 3.  RE: Is it possible to log AP missed heartbeats?

    Posted Apr 16, 2013 09:18 AM

    You can check if other APs are bootstrapping or rebooting by using this command :

     

    show ap debug counters

     

    If you are seeing this behavior a lot you might have to take a look at your network layout / traffic / Layer 1 .

     

    Another thing you could do is increase the heartbeat value from 8 to 25 under the ap system-profile 



  • 4.  RE: Is it possible to log AP missed heartbeats?

    Posted Apr 17, 2013 09:55 AM

    So here is what we're experiencing...

     

    We have 4 APs & a Switch behind a mesh link in building that has no fiber service (which I'm told requires abatement). 

     

    AFAICT, the mesh links don't report any problems, but the APs behind the link seem to bootstrap, anywhere between 3 - 6 times a day; we have Airwave sending us AP Device Up/Down messages.

     

    #show ap debug counters group armory   
    
    AP Counters
    -----------
    Name         Group   IP Address      Configs Sent  Configs Acked  AP Boots Sent  AP Boots Acked  Bootstraps (Total)  Reboots
    ----         -----   ----------      ------------  -------------  -------------  --------------  ------------------  -------
    army-1-2d52  armory  192.168.55.114  5             5              0              0               1          (1    )  0
    army-1-2df8  armory  192.168.54.116  5             5              0              0               1          (1    )  0
    army-1-2e08  armory  192.168.54.115  20            18             0              0               1          (1    )  0
    army-1-2e20  armory  192.168.54.114  8             7              0              0               1          (1    )  0
    Total APs :4

     

    Here are the counters from the Mesh Point & Portal.  The Point & Portal are on a different controller than the APs.

     

    #show ap mesh topology aruba-mesh

    Mesh Cluster Name: aruba-mesh
    -----------------------------
    Name                Mesh Role   Parent              Path Cost  Node Cost  Link Cost  Hop Count  RSSI  Rate Tx/Rx  Last Update  Uplink Age    #Children
    ----                ---------   ------              ---------  ---------  ---------  ---------  ----  ----------  -----------  ----------    ---------
    armory-mesh-point   Point (N)   nesbit-mesh-portal  11         0          10         1          26    54/48       1m:27s       1m:31s        0
    nesbit-mesh-portal  Portal (N)  -                   0          1          0          0          0     -           1m:31s       3d:4h:1m:34s  1

    Total APs :2
    (R): Recovery AP. (N): 11N Enabled. For Portals 'Uplink Age' equals uptime.


    #show ap mesh active aruba-mesh     

    Mesh Cluster Name: aruba-mesh
    -----------------------------
    Name                Group            IP Address      BSSID              Band/Ch/EIRP/MaxEIRP  MTU   Enet Ports     Mesh Role  Parent              #Children  AP Type  Uptime
    ----                -----            ----------      -----              --------------------  ---   ----------     ---------  ------              ---------  -------  ------
    armory-mesh-point   mesh-aps-armory  192.168.55.148  d8:c7:c8:b3:9b:50  802.11a/149/20/20           Tunnel/Tunnel  Point      nesbit-mesh-portal  0          124      3d:3h:59m:43s
    nesbit-mesh-portal  mesh-aps-armory  192.168.55.115  d8:c7:c8:b3:9f:d0  802.11a/149/21/21     1500  -/Off          Portal     -                   1          124      3d:3h:54m:25s

    Total APs :2

    #show ap debug counters group mesh-aps-armory AP Counters ----------- Name Group IP Address Configs Sent Configs Acked AP Boots Sent AP Boots Acked Bootstraps (Total) Reboots ---- ----- ---------- ------------ ------------- ------------- -------------- ------------------ ------- armory-mesh-point mesh-aps-armory 192.168.55.148 7 7 0 0 2 (2 ) 1 nesbit-mesh-portal mesh-aps-armory 192.168.55.115 2 2 0 0 1 (2 ) 2 Total APs :2

     I'm now gonna bombard you w/ some info...

     

    #show ap mesh debug counters ap-name armory-mesh-point
    
    Mesh Packet Counters
    --------------------
    Interface  Echo Sent  Echo Recv  Probe Req  Probe Resp   Assoc Req   Assoc Resp   Assoc Fail  Link up/down  Resel.  Switch  Other Mgmt
    ---------  ---------  ---------  ---------  ----------   ---------   ----------   ----------  ------------  ------  ------  ----------
    Parent     0          0          0          0            0           0            0           0             -       -       0
    Child      269136     267435     634        119(119 HT)  182(56 HT)  177 (56 HT)  5           118           0       0       2583082
    
    Received Packet Statistics: Total 3119036, Mgmt 2586652 (dropped non-mesh 0), Data 532384 (dropped unassociated 0)HT: pns=0 ans=0 pnr=119 ars=56 arr=0 anr=56
    
    Recovery Profile Usage Counters
    -------------------------------
    Item                         Value
    ----                         -----
    Enter recovery mode          1
    Exit recovery mode           1
    Total connections to switch  0
    
    Mesh loop-prevention Sequence No.:272792
    
    Mesh timer ticks:273118
    
    Change-record: HT-link renegotiation, linkdown:2s, linkup:2s, previous portal:d8:c7:c8:b3:9f:d0, previous parent: d8:c7:c8:b3:9f:d0
    
    #show ap mesh debug counters ap-name nesbit-mesh-portal
    
    Mesh Packet Counters
    --------------------
    Interface  Echo Sent  Echo Recv  Probe Req  Probe Resp   Assoc Req   Assoc Resp  Assoc Fail  Link up/down  Resel.  Switch  Other Mgmt
    ---------  ---------  ---------  ---------  ----------   ---------   ----------  ----------  ------------  ------  ------  ----------
    Parent     270679     266753     602        602(602 HT)  177(56 HT)  177(56 HT)  0           176           -       -       0
    
    Received Packet Statistics: Total 3126998, Mgmt 2592799 (dropped non-mesh 0), Data 534199 (dropped unassociated 0)HT: pns=602 ans=56 pnr=0 ars=0 arr=56 anr=0
    
    Recovery Profile Usage Counters
    -------------------------------
    Item                         Value
    ----                         -----
    Enter recovery mode          0
    Exit recovery mode           0
    Total connections to switch  0
    
    Mesh loop-prevention Sequence No.:272806
    
    Mesh timer ticks:272805
    
    

     

    I'll need to sort through the profiles to make sure we're looking at the right threshold setting for these APs. 

     

    If you see anything glaring, or can make any suggestions, I'm all ears.  Additionally, we're working on moving the mesh point (or portal) so that we have a clear line of sight.  I believe there is a tree in the way at the moment, and being that its spring, it's about to come alive w/ folliage.

     

    More as things develop.  Thanks,

     



  • 5.  RE: Is it possible to log AP missed heartbeats?

    Posted Apr 17, 2013 10:27 AM

     

    Can you run the following commands on the mesh portal and point to determine if there's any RF issues with the 802.11a band :

     

    show  ap arm rf-summary ap-name 

     

    show ap arm history ap-name

     

    Please share 

     

     

     

     



  • 6.  RE: Is it possible to log AP missed heartbeats?

    Posted Apr 17, 2013 10:41 AM

    Here is the output for the rf-summary :

     

    #show  ap arm rf-summary ap-name armory-mesh-point
    
    Channel Summary
    ---------------
    channel  retry  phy-err  mac-err  noise  cov-idx  intf_idx
    -------  -----  -------  -------  -----  -------  --------
    161      0      0        0        92     0/0      0/0//0/0
    1        0      0        0        0      0/0      0/0//0/0
    48       0      0        0        0      0/0      0/0//0/0
    165      0      0        0        91     0/0      0/0//0/0
    5        0      0        0        0      0/0      0/0//0/0
    6        0      0        0        0      0/0      0/0//0/0
    7        0      0        0        0      0/0      0/0//0/0
    11       0      0        0        0      0/0      0/0//0/0
    149      17     0        8        94     16/0     71/0//0/0
    36       0      0        0        0      0/0      0/0//0/0
    153      0      0        0        96     0/0      0/0//0/0
    40       0      0        0        0      0/0      0/0//0/0
    157      0      0        0        92     0/0      0/0//0/0
    44       0      0        0        0      0/0      0/0//0/0
    HT Channel Summary
    ------------------
    channel_pair  Pairwise_intf_index
    ------------  -------------------
    1-5           0
    7-11          0
    149-153       71
    36-40         0
    157-161       0
    44-48         0
    
    Interface Name           :wifi0
    Current ARM Assignment   :149/127
    Covered channels a/g     :0/0
    Free channels a/g        :9/0
    ARM Edge State           :enable
    Last check channel/pwr   :3d:4h:33m:25s/3d:4h:33m:25s
    Last change channel/pwr  :3d:4h:33m:25s/3d:4h:33m:25s
    Next Check channel/pwr   :0s/0s
    
    Interface Name           :wifi1
    Current ARM Assignment   :11/127
    Covered channels a/g     :0/1
    Free channels a/g        :0/2
    ARM Edge State           :disable
    Last check channel/pwr   :3d:4h:33m:25s/3d:4h:33m:25s
    Last change channel/pwr  :3d:4h:33m:25s/3d:4h:33m:25s
    Next Check channel/pwr   :0s/0s
    
    
    #show  ap arm rf-summary ap-name nesbit-mesh-portal
    
    Channel Summary
    ---------------
    channel  retry  phy-err  mac-err  noise  cov-idx  intf_idx
    -------  -----  -------  -------  -----  -------  --------
    161      0      0        0        89     0/0      25/17//10/0
    1        0      0        0        0      0/0      0/0//0/0
    48       0      0        0        86     4/0      22/0//0/0
    165      0      0        0        90     0/0      0/10//0/4
    5        0      0        0        0      0/0      0/0//0/0
    6        0      0        0        0      0/0      0/0//0/0
    7        0      0        0        0      0/0      0/0//0/0
    11       0      0        0        0      0/0      0/0//0/0
    149      6      0        4        95     22/0     77/0//5/0
    36       0      0        0        86     7/0      64/0//0/0
    153      0      0        0        90     0/0      0/43//0/0
    40       0      0        0        86     0/0      0/21//0/0
    157      0      0        0        88     8/0      53/8//1/4
    44       0      0        0        86     0/0      0/7//0/0
    HT Channel Summary
    ------------------
    channel_pair  Pairwise_intf_index
    ------------  -------------------
    1-5           0
    7-11          0
    149-153       125
    36-40         85
    157-161       118
    44-48         29
    
    Interface Name           :wifi0
    Current ARM Assignment   :149/127
    Covered channels a/g     :4/0
    Free channels a/g        :5/0
    ARM Edge State           :enable
    Last check channel/pwr   :3d:4h:33m:14s/3d:4h:33m:14s
    Last change channel/pwr  :3d:4h:33m:14s/3d:4h:33m:14s
    Next Check channel/pwr   :0s/0s
    
    Interface Name           :wifi1
    Current ARM Assignment   :11/127
    Covered channels a/g     :0/1
    Free channels a/g        :0/2
    ARM Edge State           :disable
    Last check channel/pwr   :3d:4h:33m:14s/3d:4h:33m:14s
    Last change channel/pwr  :3d:4h:33m:14s/3d:4h:33m:14s
    Next Check channel/pwr   :0s/0s
    
    

     

    There isn't any ARM history.  I believe we have disabled ARM for the two.  Below are the profiles used:

     

    #show ap-group mesh-aps-armory
    
    AP group "mesh-aps-armory"
    --------------------------
    Parameter                                Value
    ---------                                -----
    Virtual AP                               N/A
    802.11a radio profile                    Mesh_outdoor
    802.11g radio profile                    no_arm_mesh
    Ethernet interface 0 port configuration  default
    Ethernet interface 1 port configuration  default
    Ethernet interface 2 port configuration  shutdown
    Ethernet interface 3 port configuration  shutdown
    Ethernet interface 4 port configuration  shutdown
    AP system profile                        spare4
    VoIP Call Admission Control profile      default
    802.11a Traffic Management profile       N/A
    802.11g Traffic Management profile       N/A
    Regulatory Domain profile                default
    RF Optimization profile                  default
    RF Event Thresholds profile              default
    IDS profile                              default
    Mesh Radio profile                       armory-mesh-radio
    Mesh Cluster profile                     armory-mesh-cluster  priority 1
    Provisioning profile                     N/A
    AP authorization profile                 N/A
    
    
    
    #show rf dot11g-radio-profile no_arm_mesh
    
    802.11g radio profile "no_arm_mesh"
    -----------------------------------
    Parameter                                      Value
    ---------                                      -----
    Radio enable                                   Enabled
    Mode                                           ap-mode
    High throughput enable (radio)                 Disabled
    Channel                                        N/A
    Beacon Period                                  100 msec
    Non-Wi-Fi Interference Immunity                2
    Beacon Regulate                                Disabled
    Transmit EIRP                                  127 dBm
    Advertise 802.11d and 802.11h Capabilities     Disabled
    TPC Power                                      15 dBm
    Spectrum load balancing                        Disabled
    Spectrum Load balancing mode                   channel
    Spectrum load balancing update interval (sec)  30 seconds
    Spectrum load balancing threshold (%)          20 percent
    Advertized regulatory max EIRP                 0
    Spectrum Load Balancing domain                 N/A
    RX Sensitivity Tuning Based Channel Reuse      disable
    RX Sensitivity Threshold                       0 -dBm
    Enable CSA                                     Disabled
    CSA Count                                      4
    Reduce Cell Size (Rx Sensitivity)              0 dB
    Management Frame Throttle interval             1 sec
    Management Frame Throttle Limit                20
    ARM/WIDS Override                              Disabled
    Protection for 802.11b Clients                 Enabled
    Adaptive Radio Management (ARM) Profile        no_arm_enable_MeSh
    High-throughput Radio Profile                  default-g
    Maximum Distance                               0 meters
    Spectrum Monitoring                            Disabled
    Spectrum Monitoring Profile                    default-g
    AM Scanning Profile                            default
    
    
    #show rf dot11a-radio-profile Mesh_outdoor
    
    802.11a radio profile "Mesh_outdoor"
    ------------------------------------
    Parameter                                      Value
    ---------                                      -----
    Radio enable                                   Enabled
    Mode                                           ap-mode
    High throughput enable (radio)                 Disabled
    Channel                                        149
    Beacon Period                                  100 msec
    Non-Wi-Fi Interference Immunity                2
    Beacon Regulate                                Disabled
    Transmit EIRP                                  127 dBm
    Advertise 802.11d and 802.11h Capabilities     Disabled
    TPC Power                                      15 dBm
    Spectrum load balancing                        Disabled
    Spectrum Load balancing mode                   channel
    Spectrum load balancing update interval (sec)  30 seconds
    Spectrum load balancing threshold (%)          20 percent
    Advertized regulatory max EIRP                 0
    Spectrum Load Balancing domain                 N/A
    RX Sensitivity Tuning Based Channel Reuse      disable
    RX Sensitivity Threshold                       0 -dBm
    Enable CSA                                     Disabled
    CSA Count                                      4
    Reduce Cell Size (Rx Sensitivity)              0 dB
    Management Frame Throttle interval             1 sec
    Management Frame Throttle Limit                20
    ARM/WIDS Override                              Disabled
    Adaptive Radio Management (ARM) Profile        no_arm_enable_MeSh
    High-throughput Radio Profile                  default-a
    Maximum Distance                               0 meters
    Spectrum Monitoring                            Disabled
    Spectrum Monitoring Profile                    default-a
    AM Scanning Profile                            default
    
    
    #show rf arm-profile no_arm_enable_MeSh
    
    Adaptive Radio Management (ARM) profile "no_arm_enable_MeSh"
    ------------------------------------------------------------
    Parameter                         Value
    ---------                         -----
    Assignment                        disable
    Allowed bands for 40MHz channels  a-only
    Client Aware                      Enabled
    Max Tx EIRP                       127 dBm
    Min Tx EIRP                       33 dBm
    Multi Band Scan                   Enabled
    Rogue AP Aware                    Disabled
    Scan Interval                     10 sec
    Active Scan                       Disabled
    Scanning                          Enabled
    Scan Time                         110 msec
    VoIP Aware Scan                   Disabled
    Power Save Aware Scan             Disabled
    Video Aware Scan                  Enabled
    Ideal Coverage Index              10
    Acceptable Coverage Index         4
    Free Channel Index                25
    Backoff Time                      240 sec
    Error Rate Threshold              50 %
    Error Rate Wait Time              30 sec
    Noise Threshold                   75 -dBm
    Noise Wait Time                   120 sec
    Minimum Scan Time                 8
    Load aware Scan Threshold         1250000 Bps
    Mode Aware Arm                    Disabled
    Scan Mode                         all-reg-domain

     

    Thanks,

     



  • 7.  RE: Is it possible to log AP missed heartbeats?

    Posted Apr 17, 2013 12:05 PM

     

    I noticed you disabled HT too ?

     

    Based on the rf output you provided it doesn't look like there any RF issues on the 802.11a at least at the moment you took those.

     

    You may want to increase the Bootstrap threshold to 25 under AP system profile .



  • 8.  RE: Is it possible to log AP missed heartbeats?

    Posted Apr 17, 2013 09:31 AM

    Thanks,

    These messages (sapd) don't seem to make it to my remote log host. I see a few sapd warnings on power as far back as the logs on the controller go, however I do not see the same messages in my remote log.

     

    #show logging facility      
    
    Remote Logging Facility is local1
    
    
    #show logging server   
    
    Remote Server: 192.168.36.64
    
    FACILITY MAPPING TABLE
    ----------------------
    local-facility  severity  remote-facility
    --------------  --------  ---------------
    All             All       local1
    
    
    
    #show logging level verbose 
    
    LOGGING LEVELS
    --------------
    Facility  Level          Sub Category  Process
    --------  -----          ------------  -------
    network   warnings       N/A           N/A
    network   debugging      dhcp          N/A
    security  informational  N/A           N/A
    security  warnings       aaa           N/A
    security  informational  dot1x         N/A
    security  warnings       ids           N/A
    security  warnings       ids-ap        N/A
    system    informational  N/A           N/A
    user      informational  N/A           N/A
    user      informational  dot1x         N/A
    wireless  informational  N/A           N/A

     

    I'll have to think a bit more about why the controller does not log missed heartbeats via sapm. More importantly, why these messages aren't making it to my log host.

     

    More on the specifics of my issue in a bit.

     

     



  • 9.  RE: Is it possible to log AP missed heartbeats?

    Posted Apr 17, 2013 09:51 AM

    In the log message cjoseph posted you can see that these come as warning , so you probably should set the logging system level warning instead of informational.

     

    Another thing is that you might need to do this from the master controller, I usually see these messages on the master controller.

     

    Hope this helps



  • 10.  RE: Is it possible to log AP missed heartbeats?

    Posted Apr 17, 2013 10:27 AM

    Thanks for the suggestion...

     

    I am under the impression that the information messages would include all levels below it.  The 6.1 CLI Resource Guide (CRG) seems to confirm that.

     

    "There are eight logging severity levels, each with its associated types of messages. Each level also includes the levels below it. For example, if you set the logging level to informational (6), all messages from level 0 through level 5 (from emergencies through notifications) are also logged. The warnings severity level is set by default for all message categories and subcategories."

     

    I'm going to search through the Master's logs to see if there is anything of use.  If I can't find anything, I may have to explicitly set it for system approc messages.  However, if it is as cjoseph has indicated and the APs aren't able to send notification due to a loss of connectivity, I'm not sure whatelse can be done about it.  At the very least, I should be able to see messages after they restore connectivity. 

     

    Thanks,

     



  • 11.  RE: Is it possible to log AP missed heartbeats?

    EMPLOYEE
    Posted Apr 17, 2013 05:43 PM

    @Raf wrote:

    Thanks for the suggestion...

     

    I am under the impression that the information messages would include all levels below it.  The 6.1 CLI Resource Guide (CRG) seems to confirm that.

     

    "There are eight logging severity levels, each with its associated types of messages. Each level also includes the levels below it. For example, if you set the logging level to informational (6), all messages from level 0 through level 5 (from emergencies through notifications) are also logged. The warnings severity level is set by default for all message categories and subcategories."

     

    I'm going to search through the Master's logs to see if there is anything of use.  If I can't find anything, I may have to explicitly set it for system approc messages.  However, if it is as cjoseph has indicated and the APs aren't able to send notification due to a loss of connectivity, I'm not sure whatelse can be done about it.  At the very least, I should be able to see messages after they restore connectivity. 

     

    Thanks,

     


    What is the distance between the mesh portal and mesh point?  What Access points and antennas?  How high off of the ground?  RSSI of 26 is decent, but you might want it to be better for your purposes...



  • 12.  RE: Is it possible to log AP missed heartbeats?

    Posted Apr 17, 2013 10:46 PM

    @cjoseph wrote:

    What is the distance between the mesh portal and mesh point?  What Access points and antennas?  How high off of the ground?  RSSI of 26 is decent, but you might want it to be better for your purposes...


    Pardon my ignorance, by better do you mean a higher or lower RSSI value? 

     

    Not more than 100 ft.   (http://goo.gl/maps/m2XIr  --  you can almost see the point antenna; essentially across that walk way)

     

    Aruba AP 124.  I don't know the specific antenna being used.  It looks like an AP-ANT-92. 

     

    Portal is on roof about 2nd story level; I'd say about 18 - 20 ft high.  

    Point is also on roof about 3rd story level; 30 - 40 ft high. 

     

    My colleague and I were out on the roof near the Portal trying to get a better SNR.  We were getting mid to upper 30's w/ a few mid-upper teens on every 5th or 6th sample.  I forgot the exact command we were using, but it was outputing 3-4 different SNR values & ack'ed SNR as well.  We were looking at it very aggressively, every second or 2, while we tried to better align the antennas.  We plan on visiting the Point on Monday. 

     

    We have tentative plans to run fiber into this building, however that will take a month or longer (depending on how much abatement needs to be done). 

     

    Thanks, 

     

     



  • 13.  RE: Is it possible to log AP missed heartbeats?

    EMPLOYEE
    Posted Apr 18, 2013 04:09 AM

    Higher is of course, better.

     

    If you haven't please look at our outdoor expert Chuck L's videos on antenna types and aligning antennas here:  http://www.arubanetworks.com/technology/engineer-videos/.  Please also take a look at our outdoor VRD on Outdoor Point to Point Deployments here:  http://www.arubanetworks.com/technology/reference-design-guides/ for much more detailed information about how to  fine tune your connection.