Access Points

Reply
Aruba
Posts: 1,285
Registered: ‎08-29-2007

APs bouncing between controllers, broken tunnel.

Hi,

We are currently using 3.3.1.31 and have noticed some situations occasionally where whole groups of APs will fail over to its other controller. The APs all remain up, but this does bounce the bssid.

The following messages show on the system log.


Nov 11 08:44:23 :311006: |AP LHR5085T5A3-A4-AP00247-00:0b:86:c6:64:ea@10.36.47.56 sapd| Broken tunnel to switch detected on radio 0 VAP 3; rebootstrapping
Nov 11 08:44:23 :311006: |AP LHR4028T4-A3-AP00058-00:0b:86:c7:25:da@10.15.47.40 sapd| Broken tunnel to switch detected on radio 0 VAP 0; rebootstrapping
Nov 11 08:44:23 :311006: |AP LHR5079T5A2-A1-AP00206-00:0b:86:c5:bf:fa@10.35.47.40 sapd| Broken tunnel to switch detected on radio 0 VAP 3; rebootstrapping




The problem is, apart from potentially a very brief outage, is that when the APs moved back across to their designated controller, a trap is sent to say that the AP(s) are down, but because the trap to say that the AP is up is being sent from another controller, these events are not cleared from our NMS systems.

Any ideas on what is happening here in the first place?

Thanks

Michael

If my post is helpful please give kudos, or mark as solved if it answers your post.

ACCP, ACMP, ACMX #294
mclarke@arubanetworks.com
Guru Elite
Posts: 20,582
Registered: ‎03-29-2007

How Many


Hi,

We are currently using 3.3.1.31 and have noticed some situations occasionally where whole groups of APs will fail over to its other controller. The APs all remain up, but this does bounce the bssid.

The following messages show on the system log.



The problem is, apart from potentially a very brief outage, is that when the APs moved back across to their designated controller, a trap is sent to say that the AP(s) are down, but because the trap to say that the AP is up is being sent from another controller, these events are not cleared from our NMS systems.

Any ideas on what is happening here in the first place?

Thanks

Michael




Michael,

How many APs do you have active on that primary controller and how is it connected to the network physically?


Colin Joseph
Aruba Customer Engineering

Looking for an Answer? Search the Community Knowledge Base Here: Community Knowledge Base

Aruba Employee
Posts: 509
Registered: ‎07-03-2008

Re: APs bouncing between controllers, broken tunnel.

Michael - I recently saw the same phenomenon, check out post:

http://airheads.arubanetworks.com/vBulletin/showthread.php?t=1540

I recommend you open a support case. I'm still in monitoring mode for a solution, but what it is very strongly pointing to is non-802.11 interference. I usually see one or two APs drop from the local to the master at a time, not whole groups like you are, so it could be a different issue.

If you do a "show ap debug radio-stats ap-name radio 1 advanced" (and radio 0), check to see if there are any "Resets BeacQ Stucks" or CRC errors. Also, check for any errors on the switchports that the APs are connected to. If they are clean, the issues could be in RF. Tuning interference immunity on my 'g' radio profile for that site has done the trick so far. It was determined it was most likely non-802.11 interference causing the issue because there were no significant interference from 802.11 devices in the air. No AP has dropped for a little over two weeks now (knock wood).

Again, TAC is going to be able to help you a lot more, but you may end up needed to get past first level support to get to your root cause.
Guru Elite
Posts: 20,582
Registered: ‎03-29-2007

Bootstrapping

Just to be clear; a bootstrap occurs when an AP has missed heartbeats equal to the bootstrap threshold in the AP system-profile of that AP (7 by default). Heartbeats are typically sent one per second from an AP. "show ap debug counters" will tell you if an AP is bootstrapping. A broken tunnel is registered when an AP gets an "ICMP unreachable" from the controller it is trying to contact or if any other device in the path sends an ICMP uncreachable. At that point the AP will bootstrap. Common troubleshooting procedure is like mentioned before, ensuring that the AP has negotiated properly with the switch and there are no errors on the port. Improper negotiation and packet loss will cause or contribute to the possibility of a bootstrap. If an AP has negotiated poorly, an AP will not rebootstrap until it has missed (bootstrap threshold) heartbeats. At times, the combination of poor negotiation + congestion will cause it to bootstrap; this is why it is important to find out if one, or both are causing this situation. Monitor your controller physical port that the AP traffic comes in on and the switch/router port it is connected to ensure that you are not experiencing congestion/errors that will cause/contribute to the situation above. If many APs bootstrap, the controller port could be congested at the time, and you might have to do a port channel to widen that pipe. When APs bootstrap, they turn off their radios until they find another controller, which is why you will always see the radio go down. This normally allows clients to then re-ip in the correct range in the event the AP has to find another controller.

Everything that was said above relates strictly to AP to controller connectivity. Interference/ Beacon Queue resets, interference immunity or CRC errors have to do with client connectivity and not AP bootstraps, specifically. These issues may result in a client seeing an access point, but not being able to connect or connect reliably. All in all, you may have to open a support case, but ensuring that the AP has a healthy path to the controller will deal with a whole range of connectivity issues before it gets to that.


Colin Joseph
Aruba Customer Engineering

Looking for an Answer? Search the Community Knowledge Base Here: Community Knowledge Base

Aruba Employee
Posts: 509
Registered: ‎07-03-2008

Re: APs bouncing between controllers, broken tunnel.

In my case though Colin, there was so much interference that the APs couldn't process even their own beacons. This caused heartbeats to be missed, and the AP to disassociate from the controller.
Aruba
Posts: 1,285
Registered: ‎08-29-2007

Re: APs bouncing between controllers, broken tunnel.

Hi,

Thanks for the suggestions guys. Just to clarify, this event happened yesterday to only one pair of controllers and affected a whole bunch of APs, about ~80. I only included a couple of lines from the log. It also happened to a few other, but not all, pairs of controllers about a week before, and also involved dozens of APs.

The controllers are all locals, paired up in a vrrp, connected with two gig connections, in a port channel configuration. There are negligible errors on the ports and the utilisation is low. The Cisco logs also do not point to anything happening at the time.

I think a TAc case is the best option here.

Thanks

If my post is helpful please give kudos, or mark as solved if it answers your post.

ACCP, ACMP, ACMX #294
mclarke@arubanetworks.com
Aruba Employee
Posts: 664
Registered: ‎04-15-2009

Re: APs bouncing between controllers, broken tunnel.

Michael,

You can open a TAC case, but I highly suspect you are hitting bug ID 34661. It is fixed in 3.3.1.30, 3.3.2.18, and 3.4.0.3. The bug is caused by a VRRP timer and the symptom is that APs will bootstrap at random times. You can do a "show vrrp x staticstics" (where x is your vrrp group number). See if the last line in the output on the backup ("Number times became VRRP master") is increasing. If it is, and the VRRP is NOT actually transitioning to the backup, you are hitting that bug.
Search Airheads
Showing results for 
Search instead for 
Did you mean: