Wireless Access

I have a Campus environment with (45) 215 AP's.  In this environment a few of my AP's continuously bootstrap.  This is caused by missing heartbeats.   On th e7205 Controller I can ping all AP's until they bootstrap.  I have checked the ports and protocols on my firewall and even had the dreaded "permit any any" to test and the devices still bootstrap.  I have changed the default heartbeat count to 10 at 60 seconds intervals.  Along with changing the MTU to 1500 still booting.  I have upgraded to the 6.5.4.3 OS. Other than opening a case for this matter I thought I would ask a question.  What could cause this.

Attachment(s)

Discussion with Aruba.txt   198 KB 1 version

That sounds strange. I wonder if you have checked the following document, it basically shows some recommendations about how to adjust the bootrstrap threshold and prioritze AP heartbeat: http://www.arubanetworks.com/techdocs/ArubaOS_64x_WebHelp/Content/ArubaFrameStyles/AP_Config/Optimize_Over_Low_Link.htm

HTH

Hi Kevin,

Yes I have read the document in question and have the bootstrap threshold to 60 instead of the default.  Still having missed heartbeats.  I think it is the oversaturation of the link that could be the problem.  

Hello,

i also found that one of my AP has an error within the log.  May 22 05:45:26 authmgr[4114]: <522038> <4114> <NOTI> |authmgr| username=ac:a3:1e:c5:e0:aa MAC=ac:a3:1e:c5:e0:aa IP=x.x.x.126 Authentication result=Authentication Successful method=TRANSPORT-VPN server=Internal 

Hi,

Do you have a VPN concentrator where your controller resides?

It seems that the AP is trying to establish a VPN connection (assuming you are using CPSec) to another device that may not be the controller 

I wonder if you can look at the controller logs and the traffic sent by the AP

Kevin

Hi Kevin,

I do not have a VPN concentrator at this location.  We have VPN on our Firewall only and this traffic is within our LAN... the IP address of the device is 10.141.197.126

Attachment(s)

Boot.txt   3.19 MB 1 version

I found this:

May  9 13:09:28  stm[4117]: <305049> <4117> <WARN> |stm|  Unsecure AP "LIX-AP4" (MAC ac:a3:1e:c5:e0:aa, IP 205.165.197.126) has been denied access because Control Plane Security is enabled and the AP is not approved.

It seems that the AP has not been added to the whitelist in the controller. in particular, these two APs may not be added to the whitelist:

ac:a3:1e:c5:e0:aa

ac:a3:1e:c5:e2:0e

IP addresses: x.x.x.126 and x.x.x.125

EDIT: link about whitelists: https://www.arubanetworks.com/techdocs/ArubaOS_64x_WebHelp/Content/ArubaFrameStyles/Control_Plane/Whitelists_on_Campus_and_Remote_APs.htm

Please note that CPsec is not intended for use with RAPs

I wonder if you could check that

Kevin

Hi Kevin,

On those days that is correct they were not added to the whitelist.  Since then that matter has been taken care of.  I had replaced the older LIX AP's with 2 factory reset devices.  The problem only maginified then, not only did those two start having with bootstrap but now all four of my AP's at that location started.  Which is the issue now.....

That sounds bad. I wonder if you could send some logs again to check this in more detail

Kevin

Which logs please?  That was the from the command of "sh log all"  from the controller.  I have a ".pcap" file using Air Monitor to check out issue.  I have a show tech file about the problem

sh log all and sh tech should give us good leads

Kevin

Here is one of many....this one will have data from three of the AP's on location

Attachment(s)

Tech support LIX-AP3.txt   10.64 MB 1 version

This seems to be harder than expected. I would recommend to open a TAC case with Aruba support to find the root cause.

Kevin

Hi Kevin,

I have found something else strange, on the Cisco interface the AP are accumulating giants and total drops.  On one of my interfaces with 5 minutes of traffic I had a lot of giants.

GigabitEthernet1/0/27 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is d42c.443e.da1b (bia d42c.443e.da1b)
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:00, output hang never
  Last clearing of "show interface" counters 1d21h
  Input queue: 0/4096/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/4096 (size/max)
  5 minute input rate 4000 bits/sec, 2 packets/sec
  5 minute output rate 11000 bits/sec, 7 packets/sec
     888261 packets input, 209335625 bytes, 0 no buffer
     Received 5825 broadcasts (5576 multicasts)
     0 runts, 1358 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 5576 multicast, 0 pause input
     0 input packets with dribble condition detected
     1549199 packets output, 447064097 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

Alright, then I think there are two options:

- Enable jumbo frames on the network

- Reduce the SAP MTU frame size to 1500 (the answer is in page 1): http://community.arubanetworks.com/t5/Access-Points/AP-sending-quot-too-large-quot-frames/td-p/3507/page/2

HTH

Kevin

kevin,

At one time I had the MTU at 1500 but read a document to leave the setting blank that way the AP will chose it's MTUU settings automatically.  So far that has worked.  We already have jumbo frames enabled on the LAN.  Yet I am still getting giants, at certain times of the day we have a large data push due to our business model.  Yet even without that data push across the LAN these AP are getting giants.  Prior to this setting we had all AP's on a VPN connection to the firewall.  With budgeting we had to get rid othe VPN and add the connection to our main corporate connection.  Which we have opened all Wireless ports to allow traffic. 

That is interesting. At this point, I would suggest to get in touch with aruba TAC to look further into this issue

 EDIT: are the SSIDs in tunnel or bridge mode?

Kevin

---

@browneyed_wifiwrote:

We already have jumbo frames enabled on the LAN.  Yet I am still getting giants, at certain times of the day we have a large data push due to our business model.

---

Based on the show int stats provided earlier, it does not appear that jumbo frames are enabled for this interface. If so, the show interface should appear as below:

3560-Desk#show int gig 0/4

GigabitEthernet0/4 is up, line protocol is up (connected)

  Hardware is Gigabit Ethernet, address is 2401.c740.bc84 (bia 2401.c740.bc84)

  Description: "NAS"

  MTU 9000 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX

  input flow-control is off, output flow-control is unsupported

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input never, output 00:00:00, output hang never

  Last clearing of "show interface" counters never

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 0 bits/sec, 0 packets/sec

  5 minute output rate 7000 bits/sec, 12 packets/sec

     268 packets input, 43039 bytes, 0 no buffer

     Received 120 broadcasts (22 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 22 multicast, 0 pause input

     0 input packets with dribble condition detected

     7418 packets output, 703268 bytes, 0 underruns

     0 output errors, 0 collisions, 1 interface resets

     0 unknown protocol drops

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 pause output

     0 output buffer failures, 0 output buffers swapped out

3560-Desk#

Since the MTU for your interface is still showing 1500, it appears as though the switch isn't supporting jumbo frames. What does "show system mtu" return with run on your switch?

3560-Desk#show system mtu

System MTU size is 1500 bytes

System Jumbo MTU size is 9000 bytes

System Alternate MTU size is 1500 bytes

Routing MTU size is 1500 bytes

3560-Desk#

Hi Charles,

CiscoSW# sh system mtu

System MTU size is 1500 bytes
System Jumbo MTU size is 1500 bytes
System Alternate MTU size is 1500 bytes
Routing MTU size is 1500 bytes

Hey, since we are using the POE injectors.  Would that cause a issue with the bootstrapping problem.  Can the injectors start to go out and cause this issue?

Ap-215 needs 802.3af power (15.4 W). So, if the midspan (injector) provides that there should not be any issue. Furthermore, if the APs do not get sufficient power, they usually shut down one radio (this can be found with console access to the AP).

In addition, if you use an injector; make sure that PoE is disabled in that switchport

Kevin 

Good Morning Kevin,

The switches that we are using are not POE compatible.  These are old 2960G/X series that is why we had to get those 3501G PowerDsine injectors.

Hi James,

Then I guess that the issue is related to the jumbo frames. I know that the APs send large frames periodically to perform Path MTU Discovery but this should not affect the performance that much.

I wonder if you can do some packet capture/port mirroring to the traffic between the AP and the switch to see what is going on

Kevin

Hi Kevin,

I did take your advice and opened a TAC ticket.  They are wondering the same as you.  Yes I have some pcap files you can look at.  It seemed that every so often I would lose connectivity.  We checked the circuit and found nothing as of late.

Good morning 

Update of the ticket.  AP are still bootstrapping in certain locations.  We had a update from Cisco that may have played a part in this issue.  I also took the controller back to last working AOS.

Glad to read that this is moving forward. Hope you manage to fix it. You said you rolled back to the previous AOS version, is there any bug related to it in AOS?

Kevin

Kevin,

Not sure, but 6.5.4.3 did not work well with Cisco IOS 15.2(6) E1 for the 2960X, nor with the 3850 IOS 03.06.  Every port that the AP's were connected to had drops and giants.  Once I rolled back,  all of them went away.  Most of the campus AP are working without bootstrapping yet a few are still having issues.  Most are due to some interference and Rogues.