Access Points

Reply
Aruba Employee
Posts: 509
Registered: ‎07-03-2008

Troubleshooting Tips - APs failing to master

I have a situation at one of my remote sites where APs will randomly lose their connection to their local controller and drop back to the master. This doesn't happen every day and the site is an hour away, so I wanted to see what everyone thought would be good remote troubleshooting/data gathering steps. Interestingly enough, this is the most basic and straightforward install I have, yet it gives me endless trouble.

Attached is a high-level diagram of the site's LAN and WAN connection back to its primary data center. I have an Aruba 200 controller as the site local controller and five APs...three 65s and two 70s. Controller is running AOS 3.3.2.19. The site only has two cisco 2950 switches and a single VLAN. The 2950s are connected to each other via their gigE ports in an L2 etherchannel. Three APs are on one switch along with the controller, and the other two are on the second switch.

Randomly, one or maybe two of the APs will lose their connection to the local controller and fail back to the master, where it runs happily until LMS preemption kicks in and it goes back to the local. The only logs I really see in the local controller are below:

Sep 4 14:05:29 :311006: |AP REA-01-AP02@10.42.0.90 sapd| Broken tunnel to switch detected on radio 0 VAP 0; rebootstrapping
Sep 4 14:52:47 :311006: |AP REA-01-AP02@10.42.0.90 sapd| Broken tunnel to switch detected on radio 0 VAP 1; rebootstrapping
Sep 8 10:45:38 :311006: |AP REA-01-AP03@10.42.0.98 sapd| Broken tunnel to switch detected on radio 0 VAP 0; rebootstrapping
Sep 8 10:51:25 :311006: |AP REA-01-AP04@10.42.0.49 sapd| Broken tunnel to switch detected on radio 1 VAP 1; rebootstrapping
Sep 8 13:02:09 :311006: |AP REA-01-AP03@10.42.0.98 sapd| Broken tunnel to switch detected on radio 0 VAP 0; rebootstrapping
Sep 8 13:05:34 :311006: |AP REA-01-AP04@10.42.0.49 sapd| Broken tunnel to switch detected on radio 0 VAP 0; rebootstrapping
Sep 14 14:06:24 :311006: |AP REA-01-AP03@10.42.0.98 sapd| Broken tunnel to switch detected on radio 0 VAP 0; rebootstrapping
Sep 14 14:07:44 :311006: |AP REA-01-AP05@10.42.0.78 sapd| Broken tunnel to switch detected on radio 1 VAP 1; rebootstrapping
Sep 14 14:20:13 :316023: |wms| PAPI_Send failed to probe IP 10.42.0.98 from WMS
Sep 14 14:35:30 :311006: |AP REA-01-AP03@10.42.0.98 sapd| Broken tunnel to switch detected on radio 0 VAP 0; rebootstrapping

AP04/05 are the 70s, AP01/02/03 are the 65s. AP03 seems to be the biggest offender. Everything I check on the LAN is spotless. No switchport errors, no utilization issues, nothing. There are no reported general LAN issues reported at the site either. The 2950s are at the latest code as well. What else from an Aruba perspective can I even look at to figure out what may be the issue with the broken connections?

TIA.
Occasional Contributor II
Posts: 35
Registered: ‎04-29-2009

Re: Troubleshooting Tips - APs failing to master

Hi Mike,

Are you seeing any events that corrolate to the timestamps when do a 'show fault' or 'show fault history' ?
Aruba Employee
Posts: 509
Registered: ‎07-03-2008

Re: Troubleshooting Tips - APs failing to master

Yes, the faults show the AP and its BSSIDs as down at those times.
Occasional Contributor I
Posts: 9
Registered: ‎04-03-2007

Re: Troubleshooting Tips - APs failing to master

I'm sure you've checked this, but I think we saw something like this in a scenario where the local was running different code than the master....then again it was awhile ago so memory may not be serving me correctly...
Aruba Employee
Posts: 509
Registered: ‎07-03-2008

Re: Troubleshooting Tips - APs failing to master

Hi Garrett - Yeah, everything's on the same code, 3.3.2.19.
New Contributor
Posts: 2
Registered: ‎11-15-2007

Troubleshooting Tips - APs failing to master

Mike,

Try the below steps

1. Run ap tech-support and look for rebootstrapping reason. command "show ap tech-support "
2. Is AP03 connected to the same switch where the controller is connected.
3. Are you sure if AP failover to master you don't see this the same problem?
4. reprovision AP03 to master and see if you hit the problem.
5. when did you upgrade the controller? did you see the same problem in the previous code also?
6. have some on at the site to swap the AP's and see if it fixes ( to isolate the hardware problem)
7. try to disable L2 ethch and see.
8. if above steps didnt help enable ap debug get the logs.tar with tech-support & ap tech-support contact Aruba support.

Cheers
Murali
Aruba Employee
Posts: 509
Registered: ‎07-03-2008

Re: Troubleshooting Tips - APs failing to master

Thanks Murali - One interesting thing from one of the AP tech-supports was that the controller is sending ICMP port unreachables to the AP.

asap_gre_err: Received ICMP (DEST_UNREACH, PROT_UNREACH) from <6>for vap 0:0

Kind of shows me that there's something up with the controller. I'm going to open up a TAC case at this point.
Search Airheads
Showing results for 
Search instead for 
Did you mean: