Wireless Access

last person joined: 8 hours ago 

Access network design for branch, remote, outdoor and campus locations with Aruba access points, and mobility controllers.
Expand all | Collapse all

Bootstrapping of AP215

  • 1.  Bootstrapping of AP215

    Posted May 21, 2018 12:36 PM
      |   view attached

    I have a Campus environment with (45) 215 AP's.  In this environment a few of my AP's continuously bootstrap.  This is caused by missing heartbeats.   On th e7205 Controller I can ping all AP's until they bootstrap.  I have checked the ports and protocols on my firewall and even had the dreaded "permit any any" to test and the devices still bootstrap.  I have changed the default heartbeat count to 10 at 60 seconds intervals.  Along with changing the MTU to 1500 still booting.  I have upgraded to the 6.5.4.3 OS. Other than opening a case for this matter I thought I would ask a question.  What could cause this.

    Attachment(s)

    txt
    Discussion with Aruba.txt   198K 1 version


  • 2.  RE: Bootstrapping of AP215

    Posted May 22, 2018 03:14 AM

    That sounds strange. I wonder if you have checked the following document, it basically shows some recommendations about how to adjust the bootrstrap threshold and prioritze AP heartbeat: http://www.arubanetworks.com/techdocs/ArubaOS_64x_WebHelp/Content/ArubaFrameStyles/AP_Config/Optimize_Over_Low_Link.htm

     

    HTH



  • 3.  RE: Bootstrapping of AP215

    Posted May 22, 2018 07:54 AM

    Hi Kevin,

     

    Yes I have read the document in question and have the bootstrap threshold to 60 instead of the default.  Still having missed heartbeats.  I think it is the oversaturation of the link that could be the problem.  

     



  • 4.  RE: Bootstrapping of AP215

    Posted May 23, 2018 02:26 PM

    Hello,

     

    i also found that one of my AP has an error within the log.  May 22 05:45:26 authmgr[4114]: <522038> <4114> <NOTI> |authmgr| username=ac:a3:1e:c5:e0:aa MAC=ac:a3:1e:c5:e0:aa IP=x.x.x.126 Authentication result=Authentication Successful method=TRANSPORT-VPN server=Internal 



  • 5.  RE: Bootstrapping of AP215

    Posted May 23, 2018 03:23 PM

    Hi,

     

    Do you have a VPN concentrator where your controller resides?

    It seems that the AP is trying to establish a VPN connection (assuming you are using CPSec) to another device that may not be the controller 

     

    I wonder if you can look at the controller logs and the traffic sent by the AP

     

    Kevin



  • 6.  RE: Bootstrapping of AP215

    Posted May 23, 2018 04:05 PM
      |   view attached

    Hi Kevin,

     

    I do not have a VPN concentrator at this location.  We have VPN on our Firewall only and this traffic is within our LAN... the IP address of the device is 10.141.197.126

    Attachment(s)

    txt
    Boot.txt   3.19MB 1 version


  • 7.  RE: Bootstrapping of AP215

    Posted May 24, 2018 10:25 AM

    I found this:

    May  9 13:09:28  stm[4117]: <305049> <4117> <WARN> |stm|  Unsecure AP "LIX-AP4" (MAC ac:a3:1e:c5:e0:aa, IP 205.165.197.126) has been denied access because Control Plane Security is enabled and the AP is not approved.

     

    It seems that the AP has not been added to the whitelist in the controller. in particular, these two APs may not be added to the whitelist:

    ac:a3:1e:c5:e0:aa

    ac:a3:1e:c5:e2:0e

    IP addresses: x.x.x.126 and x.x.x.125

     

    EDIT: link about whitelists: https://www.arubanetworks.com/techdocs/ArubaOS_64x_WebHelp/Content/ArubaFrameStyles/Control_Plane/Whitelists_on_Campus_and_Remote_APs.htm

     

    Please note that CPsec is not intended for use with RAPs

     

    I wonder if you could check that

    Kevin



  • 8.  RE: Bootstrapping of AP215

    Posted May 24, 2018 10:30 AM

    Hi Kevin,

    On those days that is correct they were not added to the whitelist.  Since then that matter has been taken care of.  I had replaced the older LIX AP's with 2 factory reset devices.  The problem only maginified then, not only did those two start having with bootstrap but now all four of my AP's at that location started.  Which is the issue now.....

     



  • 9.  RE: Bootstrapping of AP215

    Posted May 24, 2018 10:36 AM

    That sounds bad. I wonder if you could send some logs again to check this in more detail

    Kevin



  • 10.  RE: Bootstrapping of AP215

    Posted May 24, 2018 10:43 AM

    Which logs please?  That was the from the command of "sh log all"  from the controller.  I have a ".pcap" file using Air Monitor to check out issue.  I have a show tech file about the problem



  • 11.  RE: Bootstrapping of AP215

    Posted May 24, 2018 10:46 AM

    sh log all and sh tech should give us good leads

    Kevin



  • 12.  RE: Bootstrapping of AP215

    Posted May 24, 2018 11:05 AM
      |   view attached

    Here is one of many....this one will have data from three of the AP's on location

    Attachment(s)

    txt
    Tech support LIX-AP3.txt   10.64MB 1 version


  • 13.  RE: Bootstrapping of AP215

    Posted May 24, 2018 11:49 AM

    This seems to be harder than expected. I would recommend to open a TAC case with Aruba support to find the root cause.

    Kevin



  • 14.  RE: Bootstrapping of AP215

    Posted May 24, 2018 12:28 PM

    Hi Kevin,

     

    I have found something else strange, on the Cisco interface the AP are accumulating giants and total drops.  On one of my interfaces with 5 minutes of traffic I had a lot of giants.

     

    GigabitEthernet1/0/27 is up, line protocol is up (connected)
      Hardware is Gigabit Ethernet, address is d42c.443e.da1b (bia d42c.443e.da1b)
      MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
         reliability 255/255, txload 1/255, rxload 1/255
      Encapsulation ARPA, loopback not set
      Keepalive set (10 sec)
      Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
      input flow-control is off, output flow-control is unsupported
      ARP type: ARPA, ARP Timeout 04:00:00
      Last input never, output 00:00:00, output hang never
      Last clearing of "show interface" counters 1d21h
      Input queue: 0/4096/0/0 (size/max/drops/flushes); Total output drops: 0
      Queueing strategy: fifo
      Output queue: 0/4096 (size/max)
      5 minute input rate 4000 bits/sec, 2 packets/sec
      5 minute output rate 11000 bits/sec, 7 packets/sec
         888261 packets input, 209335625 bytes, 0 no buffer
         Received 5825 broadcasts (5576 multicasts)
         0 runts, 1358 giants, 0 throttles
         0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
         0 watchdog, 5576 multicast, 0 pause input
         0 input packets with dribble condition detected
         1549199 packets output, 447064097 bytes, 0 underruns
         0 output errors, 0 collisions, 0 interface resets
         0 unknown protocol drops
         0 babbles, 0 late collision, 0 deferred
         0 lost carrier, 0 no carrier, 0 pause output
         0 output buffer failures, 0 output buffers swapped out



  • 15.  RE: Bootstrapping of AP215

    Posted May 24, 2018 01:07 PM

    Alright, then I think there are two options:

     

    - Enable jumbo frames on the network

    - Reduce the SAP MTU frame size to 1500 (the answer is in page 1): http://community.arubanetworks.com/t5/Access-Points/AP-sending-quot-too-large-quot-frames/td-p/3507/page/2

     

    HTH

    Kevin



  • 16.  RE: Bootstrapping of AP215

    Posted May 24, 2018 01:51 PM

    kevin,

     

    At one time I had the MTU at 1500 but read a document to leave the setting blank that way the AP will chose it's MTUU settings automatically.  So far that has worked.  We already have jumbo frames enabled on the LAN.  Yet I am still getting giants, at certain times of the day we have a large data push due to our business model.  Yet even without that data push across the LAN these AP are getting giants.  Prior to this setting we had all AP's on a VPN connection to the firewall.  With budgeting we had to get rid othe VPN and add the connection to our main corporate connection.  Which we have opened all Wireless ports to allow traffic. 



  • 17.  RE: Bootstrapping of AP215

    Posted May 24, 2018 02:11 PM

    That is interesting. At this point, I would suggest to get in touch with aruba TAC to look further into this issue

     EDIT: are the SSIDs in tunnel or bridge mode?

    Kevin



  • 18.  RE: Bootstrapping of AP215

    Posted May 24, 2018 02:41 PM

    @browneyed_wifiwrote:

    We already have jumbo frames enabled on the LAN.  Yet I am still getting giants, at certain times of the day we have a large data push due to our business model.


    Based on the show int stats provided earlier, it does not appear that jumbo frames are enabled for this interface. If so, the show interface should appear as below:

     

    3560-Desk#show int gig 0/4

    GigabitEthernet0/4 is up, line protocol is up (connected)

      Hardware is Gigabit Ethernet, address is 2401.c740.bc84 (bia 2401.c740.bc84)

      Description: "NAS"

      MTU 9000 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

         reliability 255/255, txload 1/255, rxload 1/255

      Encapsulation ARPA, loopback not set

      Keepalive set (10 sec)

      Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX

      input flow-control is off, output flow-control is unsupported

      ARP type: ARPA, ARP Timeout 04:00:00

      Last input never, output 00:00:00, output hang never

      Last clearing of "show interface" counters never

      Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

      Queueing strategy: fifo

      Output queue: 0/40 (size/max)

      5 minute input rate 0 bits/sec, 0 packets/sec

      5 minute output rate 7000 bits/sec, 12 packets/sec

         268 packets input, 43039 bytes, 0 no buffer

         Received 120 broadcasts (22 multicasts)

         0 runts, 0 giants, 0 throttles

         0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

         0 watchdog, 22 multicast, 0 pause input

         0 input packets with dribble condition detected

         7418 packets output, 703268 bytes, 0 underruns

         0 output errors, 0 collisions, 1 interface resets

         0 unknown protocol drops

         0 babbles, 0 late collision, 0 deferred

         0 lost carrier, 0 no carrier, 0 pause output

         0 output buffer failures, 0 output buffers swapped out

    3560-Desk#

     

    Since the MTU for your interface is still showing 1500, it appears as though the switch isn't supporting jumbo frames. What does "show system mtu" return with run on your switch?

     

    3560-Desk#show system mtu

     

    System MTU size is 1500 bytes

    System Jumbo MTU size is 9000 bytes

    System Alternate MTU size is 1500 bytes

    Routing MTU size is 1500 bytes

    3560-Desk#



  • 19.  RE: Bootstrapping of AP215

    Posted May 24, 2018 03:35 PM

    Hi Charles,

    CiscoSW# sh system mtu

    System MTU size is 1500 bytes
    System Jumbo MTU size is 1500 bytes
    System Alternate MTU size is 1500 bytes
    Routing MTU size is 1500 bytes

     



  • 20.  RE: Bootstrapping of AP215

    Posted May 24, 2018 03:58 PM

    Hey, since we are using the POE injectors.  Would that cause a issue with the bootstrapping problem.  Can the injectors start to go out and cause this issue?



  • 21.  RE: Bootstrapping of AP215

    Posted May 24, 2018 04:19 PM

    Ap-215 needs 802.3af power (15.4 W). So, if the midspan (injector) provides that there should not be any issue. Furthermore, if the APs do not get sufficient power, they usually shut down one radio (this can be found with console access to the AP).

     

    In addition, if you use an injector; make sure that PoE is disabled in that switchport

    Kevin 



  • 22.  RE: Bootstrapping of AP215

    Posted May 25, 2018 09:11 AM

    Good Morning Kevin,

     

    The switches that we are using are not POE compatible.  These are old 2960G/X series that is why we had to get those 3501G PowerDsine injectors.



  • 23.  RE: Bootstrapping of AP215

    Posted May 25, 2018 01:00 PM

    Hi James,

     

    Then I guess that the issue is related to the jumbo frames. I know that the APs send large frames periodically to perform Path MTU Discovery but this should not affect the performance that much.

     

    I wonder if you can do some packet capture/port mirroring to the traffic between the AP and the switch to see what is going on

     

    Kevin



  • 24.  RE: Bootstrapping of AP215

    Posted May 30, 2018 10:52 AM

    Hi Kevin,

     

    I did take your advice and opened a TAC ticket.  They are wondering the same as you.  Yes I have some pcap files you can look at.  It seemed that every so often I would lose connectivity.  We checked the circuit and found nothing as of late.



  • 25.  RE: Bootstrapping of AP215

    Posted Jun 05, 2018 08:21 AM

    Good morning 

    Update of the ticket.  AP are still bootstrapping in certain locations.  We had a update from Cisco that may have played a part in this issue.  I also took the controller back to last working AOS.



  • 26.  RE: Bootstrapping of AP215

    Posted Jun 05, 2018 08:29 AM

    Glad to read that this is moving forward. Hope you manage to fix it. You said you rolled back to the previous AOS version, is there any bug related to it in AOS?

    Kevin



  • 27.  RE: Bootstrapping of AP215

    Posted Jun 05, 2018 09:00 AM

    Kevin,

     

    Not sure, but 6.5.4.3 did not work well with Cisco IOS 15.2(6) E1 for the 2960X, nor with the 3850 IOS 03.06.  Every port that the AP's were connected to had drops and giants.  Once I rolled back,  all of them went away.  Most of the campus AP are working without bootstrapping yet a few are still having issues.  Most are due to some interference and Rogues.