Wireless Access

Running into a random issue with some RAP users across the US. The RAPs are RAP-3WNP connecting to a 7220 controller. We are experiencing network drops for the users, although it appears their RAPs are staying connected, as the uptime does not change. 

 

I enabled debugging for a specific RAP and I am getting some traffic that indicates multiple instances of "Adding new Gateway MAC", sometimes two the same second to two different MAC addresses.

 

All of the RAPs hit the same public address and are NAT to the same controller. Not all RAPs or users are having the issue. This started about 4 days ago and is intermittent. 

 

We've also done some packet captures and it seems like the devices connected to the RAPs are frequently sending ARP requests for their gateway, almost like it is changing. Don't know if these two are related, just trying to collect the data right now. 

 

Anybody notice anything similar to this or have any ideas what else to check? 

The "uptime" parameter will only reset if the RAP cold or warm reboots.  I would check the "show ap bss-table" which would show how long the radio is up.  That would reset when the AP rebootstraps for shorter connectivity issues.

I did just find this in the logs:

 

Aug 22 11:07:19 :305003: <4110> <INFO> |stm| AP 20:4c:03:09:40:54 is down
Aug 22 11:07:19 :400110: <4110> <DBUG> |stm| AP 20:4c:03:09:40:54: Delete AP state for bssid 70:3a:0e:69:d8:40; Deauth 0 Clear 1
Aug 22 11:07:24 :305007: <4110> <INFO> |stm| AP 20:4c:03:09:40:54 bootstrapped

 

The AP losing connectivity to the controller would cause this?

Yes. Type "show AP debug counters" to see how many times that AP has rebootstrapped.

I would increase the bootstrap threshold in the AP system profile to 45 to see if that solves the issue. The rap bootstrap threshold minimum is 30 even though it says 8 in the AP system profile.

Does the problem occur with the same groups of a ccess points all the time?

it doesn't appear to be happening to all RAPs in the same AP group.  The one I looked at has 67 boostraps (320 total) at this time. If I make that change, does it affect the currently connected RAPs?

Changing that parameter affects all RAPs in that AP group. Changing the bootstrap threshold merely makes the rap more resilient to outages.

If it is happening to all RAPs, you would look at the infrastructure closer to the controller. If it is only happening to a subset of RAPs, I would look at the local site. I would also take the tech-support of the worst access point and have TAC analyze it.

We've done some more digging and found that the IPSec tunnel stays up, but we're only noticing one-way communication. The pings are coming from the controller to the client on the other end of the RAP, but the return traffic is being dropped and never reaching the controller. We only have about 25 users, out of 180 experiencing the issues. Different ISPs from their homes, but same internet back to our controller. Any suggestions or ideas? 

 

We do have a TAC case open and escalated to address this issue, but there are a lot of logs and captures being requested, which will take me some time.  Figured I would work this from a few avenues.

Turned out some RAP users weren't using Eth0 as uplink to their home network, possibly causing their home router to answer ARP requests for default gateway of other users.

 

All users that dropped at the same time had the same internal home network IP scheme of 192.168.0.x. 

 

We had another event, which turned out to be unrelated, but I started another discussion which resolved this one also. See here: 

 

http://community.arubanetworks.com/t5/Wireless-Access/Same-AP-Name-for-Multiple-Users/m-p/458761#M84059