I have a Campus environment with (45) 215 AP's. In this environment a few of my AP's continuously bootstrap. This is caused by missing heartbeats. On th e7205 Controller I can ping all AP's until they bootstrap. I have checked the ports and protocols on my firewall and even had the dreaded "permit any any" to test and the devices still bootstrap. I have changed the default heartbeat count to 10 at 60 seconds intervals. Along with changing the MTU to 1500 still booting. I have upgraded to the 18.104.22.168 OS. Other than opening a case for this matter I thought I would ask a question. What could cause this.
That sounds strange. I wonder if you have checked the following document, it basically shows some recommendations about how to adjust the bootrstrap threshold and prioritze AP heartbeat: http://www.arubanetworks.com/techdocs/ArubaOS_64x_WebHelp/Content/ArubaFrameStyles/AP_Config/Optimize_Over_Low_Link.htm
Yes I have read the document in question and have the bootstrap threshold to 60 instead of the default. Still having missed heartbeats. I think it is the oversaturation of the link that could be the problem.
i also found that one of my AP has an error within the log. May 22 05:45:26 authmgr: <522038> <4114> <NOTI> |authmgr| username=ac:a3:1e:c5:e0:aa MAC=ac:a3:1e:c5:e0:aa IP=x.x.x.126 Authentication result=Authentication Successful method=TRANSPORT-VPN server=Internal
Do you have a VPN concentrator where your controller resides?
It seems that the AP is trying to establish a VPN connection (assuming you are using CPSec) to another device that may not be the controller
I wonder if you can look at the controller logs and the traffic sent by the AP
I do not have a VPN concentrator at this location. We have VPN on our Firewall only and this traffic is within our LAN... the IP address of the device is 10.141.197.126
I found this:
May 9 13:09:28 stm: <305049> <4117> <WARN> |stm| Unsecure AP "LIX-AP4" (MAC ac:a3:1e:c5:e0:aa, IP 22.214.171.124) has been denied access because Control Plane Security is enabled and the AP is not approved.
It seems that the AP has not been added to the whitelist in the controller. in particular, these two APs may not be added to the whitelist:
IP addresses: x.x.x.126 and x.x.x.125
EDIT: link about whitelists: https://www.arubanetworks.com/techdocs/ArubaOS_64x_WebHelp/Content/ArubaFrameStyles/Control_Plane/Whitelists_on_Campus_and_Remote_APs.htm
Please note that CPsec is not intended for use with RAPs
I wonder if you could check that
On those days that is correct they were not added to the whitelist. Since then that matter has been taken care of. I had replaced the older LIX AP's with 2 factory reset devices. The problem only maginified then, not only did those two start having with bootstrap but now all four of my AP's at that location started. Which is the issue now.....
That sounds bad. I wonder if you could send some logs again to check this in more detail
Which logs please? That was the from the command of "sh log all" from the controller. I have a ".pcap" file using Air Monitor to check out issue. I have a show tech file about the problem
sh log all and sh tech should give us good leads
Here is one of many....this one will have data from three of the AP's on location
This seems to be harder than expected. I would recommend to open a TAC case with Aruba support to find the root cause.
I have found something else strange, on the Cisco interface the AP are accumulating giants and total drops. On one of my interfaces with 5 minutes of traffic I had a lot of giants.
GigabitEthernet1/0/27 is up, line protocol is up (connected) Hardware is Gigabit Ethernet, address is d42c.443e.da1b (bia d42c.443e.da1b) MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX input flow-control is off, output flow-control is unsupported ARP type: ARPA, ARP Timeout 04:00:00 Last input never, output 00:00:00, output hang never Last clearing of "show interface" counters 1d21h Input queue: 0/4096/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/4096 (size/max) 5 minute input rate 4000 bits/sec, 2 packets/sec 5 minute output rate 11000 bits/sec, 7 packets/sec 888261 packets input, 209335625 bytes, 0 no buffer Received 5825 broadcasts (5576 multicasts) 0 runts, 1358 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog, 5576 multicast, 0 pause input 0 input packets with dribble condition detected 1549199 packets output, 447064097 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets 0 unknown protocol drops 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier, 0 pause output 0 output buffer failures, 0 output buffers swapped out
Alright, then I think there are two options:
- Enable jumbo frames on the network
- Reduce the SAP MTU frame size to 1500 (the answer is in page 1): http://community.arubanetworks.com/t5/Access-Points/AP-sending-quot-too-large-quot-frames/td-p/3507/page/2
At one time I had the MTU at 1500 but read a document to leave the setting blank that way the AP will chose it's MTUU settings automatically. So far that has worked. We already have jumbo frames enabled on the LAN. Yet I am still getting giants, at certain times of the day we have a large data push due to our business model. Yet even without that data push across the LAN these AP are getting giants. Prior to this setting we had all AP's on a VPN connection to the firewall. With budgeting we had to get rid othe VPN and add the connection to our main corporate connection. Which we have opened all Wireless ports to allow traffic.
That is interesting. At this point, I would suggest to get in touch with aruba TAC to look further into this issue
EDIT: are the SSIDs in tunnel or bridge mode?
@browneyed_wifiwrote:We already have jumbo frames enabled on the LAN. Yet I am still getting giants, at certain times of the day we have a large data push due to our business model.
We already have jumbo frames enabled on the LAN. Yet I am still getting giants, at certain times of the day we have a large data push due to our business model.
Based on the show int stats provided earlier, it does not appear that jumbo frames are enabled for this interface. If so, the show interface should appear as below:
3560-Desk#show int gig 0/4
GigabitEthernet0/4 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 2401.c740.bc84 (bia 2401.c740.bc84)
MTU 9000 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 7000 bits/sec, 12 packets/sec
268 packets input, 43039 bytes, 0 no buffer
Received 120 broadcasts (22 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 22 multicast, 0 pause input
0 input packets with dribble condition detected
7418 packets output, 703268 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
Since the MTU for your interface is still showing 1500, it appears as though the switch isn't supporting jumbo frames. What does "show system mtu" return with run on your switch?
3560-Desk#show system mtu
System MTU size is 1500 bytes
System Jumbo MTU size is 9000 bytes
System Alternate MTU size is 1500 bytes
Routing MTU size is 1500 bytes
CiscoSW# sh system mtu
System MTU size is 1500 bytesSystem Jumbo MTU size is 1500 bytesSystem Alternate MTU size is 1500 bytesRouting MTU size is 1500 bytes
Hey, since we are using the POE injectors. Would that cause a issue with the bootstrapping problem. Can the injectors start to go out and cause this issue?
Ap-215 needs 802.3af power (15.4 W). So, if the midspan (injector) provides that there should not be any issue. Furthermore, if the APs do not get sufficient power, they usually shut down one radio (this can be found with console access to the AP).
In addition, if you use an injector; make sure that PoE is disabled in that switchport
Good Morning Kevin,
The switches that we are using are not POE compatible. These are old 2960G/X series that is why we had to get those 3501G PowerDsine injectors.
Then I guess that the issue is related to the jumbo frames. I know that the APs send large frames periodically to perform Path MTU Discovery but this should not affect the performance that much.
I wonder if you can do some packet capture/port mirroring to the traffic between the AP and the switch to see what is going on
I did take your advice and opened a TAC ticket. They are wondering the same as you. Yes I have some pcap files you can look at. It seemed that every so often I would lose connectivity. We checked the circuit and found nothing as of late.
Update of the ticket. AP are still bootstrapping in certain locations. We had a update from Cisco that may have played a part in this issue. I also took the controller back to last working AOS.
Glad to read that this is moving forward. Hope you manage to fix it. You said you rolled back to the previous AOS version, is there any bug related to it in AOS?
Not sure, but 126.96.36.199 did not work well with Cisco IOS 15.2(6) E1 for the 2960X, nor with the 3850 IOS 03.06. Every port that the AP's were connected to had drops and giants. Once I rolled back, all of them went away. Most of the campus AP are working without bootstrapping yet a few are still having issues. Most are due to some interference and Rogues.
At Aruba, we believe that the most dynamic customer experiences happen at the Edge. Our mission is to deliver innovative solutions that harness data at the Edge to drive powerful business outcomes.
© Copyright 2020 Hewlett Packard Enterprise Development LPAll Rights Reserved.