Wireless Access

Hey Community, 

 

We are experiencing some weird intermittent outages with our RAP deployment, we have been consistently having about 25 (same RAPs every time) drop connectivity for about 30 seconds, then come back up. The RAP tunnel seems to stay up, but the client machine cannot be pinged. We ran packet capture and see the pings reaching the client, but nothing being returned. 

 

This seemingly went away on it's own over the course of a week.

 

Today, out of the blue, all of our RAPs dropped and lost connectivity (200 or so) at the same time. We rebooted our RAP controller, and when everything came back up, we had multiple users in the user-table showing the same AP Name. These same users were stuck in the logon role and we were not seeing their authetications and we could not ping users who appeared to be authenticated. After we disabled the Eth1 and Eth2 ports on the RAP that showed up for multiple users, we were able to ping and most users were able to authenticate successfully.

 

I am leaning toward some type of loop introduced, but not sure if this issue is the same or different than the original one. We do not have spanning-tree enabled for RAP ports at this time. We are planning to disable Eth2 for all RAPs and just leave Eth0 (uplink) and Eth1 for PCs. 

 

Any other experience with same AP name for multiple users or AP wired port loops that sounds like this? Trying to figure out what's going on, but struggling. 

 

We do have a TAC Case opened and escalated, but I need more eyes on this.

 

Thanks!

Make sure that users do not accidentally have an eth1 or eth2 connected to their router.  If that happens, the router on a far end will propagate its default gateway arp entry to other users/devices.  You can sometimes figure this out if hyou have untrusted ports and users with strange ip addresses (local soho ip addresses e.g. 192.168.1.x) end up in the user table.  You can prevent this by editing the validuser ACL to only allow ip address ranges that users should only get if connected to one of your wired ports.  You can also do this by blocking SOHO ip address ranges in your validuser ACL...

interesting, we do have untrusted wired ports, but I don't recall seeing unusual IPs in the user-table (doesn't mean it didn't happen, just don't recall). I will certainly look at that tomorrow. Any other possible reason for a group of RAPs (different model, different AP group) to drop at the same time? We did setup VRRP for the NAT, but are moving away from it as it's not supported. No failovers, but that is another factor we've considered as a problem. Weird thing is that RAPs have been running fine for months until last week.


#AirheadsMobile

If you have someone who dual-connected their router to a RAP, the default gateway ARP entry can be propagated without showing up in the table.  You can shut down the ethernet1 and 2 ports while it happens to see if access points that had problems recover.  

 

Someone plugging in enet1 or 2 into the router is the #1 reason for what you are seeing happening.  The only permanent proactive solution I have seen is by putting the local router subnet into the validuser ACL with a deny, so that none of the default gateways or any other ip addresses from a local router will propagate to other devices.  The below entry in the validuser ACL will block all traffic from the 192.168.1.x subnet:

 

  192.168.1.0 255.255.255.0  any          any                   deny                             Low                                                           4

 

I just confirmed we have a few users with a 192.168.1.X IP address in the user table. Our IP range is 10.X.X.X so it is likley from a home user. This sounds like someone is plugging in the wrong port from the RAP to their home network?

Yes.  Find out what AP that user is coming from and alert that user, OR use the validuser ACL to block those devices to be proactive.

Just to clarify, would the users with the 192.168 address be the ones possibly causing the issue or just were victims of the issue?

Users with the 192.168.x addresses were bridged into the controller datapath because an enet1 or enet2 was plugged into the controller.  Those are devices that are local to the SOHO router like printers and home devices that should not be on the corporate network.  The user table should say the AP name and the physical port they are plugged into.  That is your RAP that has the SOHO router incorrectly double plugged into enet1 and 2.

We've found a handful of RAP users plugged into other than Eth0 for their WAN uplink and have been working on moving them. I'm still showing (2) user entries with 192.168 addresses, but we've confirmed they have the proper setup. 

 

The only thing I'm still wondering, we are set to Split Tunnel on the AP wired profile, but split tunneling is not actually enforced by our ACLs or anything. Do you think that could introduce unusual IPs into the user-table as well? 

 

We dedicated this controller to just RAPs, so we weren't using PEF licensing, but we are looking into it so we can put 192.168.x.x into the validuser-acl as a block. 

 

Thanks for all the help, we tracked this for a week and were struggling to identify a root cause.