Higher Education

We are experiencing an uncommon situation on our campus wireless network.  An unknown aspect of our network is causing our Android clients to exercise several bugs in the Android operating system with regards to DHCP.  As a result these clients continue to use IP addresses well after their leases expire causing IP conflicts and failures to create new user table entries on the controller.

 

At this point, I am trying to determine what is different in our network configuration which is causing this issue.  If you have a few moments I would greatly appreciate any feedback you can give on the following questions.  If you are actually experiencing this issue I would appreciate a chance to compare experiences with you.

 

DHCP Configuration

• What type of DHCP server are you using?
• Are you using real world or private address space?
• What size DHCP pools do you use?
• Are your pools available across all APs or assigned to specific buildings?
• What are your lease timers?
• Where is your DHCP relay configured?

Controller configuration

• What is your controller architecture?
• What version of ArubaOS are you running?
• Do you have “Enforce DHCP” configured?
• What is your setting for “User Idle Timeout”?
• What is your setting for “Station Ageout Time”?

Thank you for your time,

 

John Pearson

Wright State University

 

wright-johnp,

 

It could help if you answer all of your questions for your environment and we might be able to help you from there.  Which android clients and which bug are you referring to?

 

Colin,

 

Just FYI:  I've been looking into this Android issue for several years.  I no longer think that the devices can be fixed; however, most institutions do not experience it (Princeton is an example of another large institution with this issue).  At this point, I am simply trying to identify anything that we have configured here outside the norm in an attempt to stop exercising these bugs.

 

Here's information on our existing configuration:

 

DHCP:

 

1) Server is a generic Bind 9 implementation on Redhat.

2) The vast majority of our wireless IP space is private; however, we do have around 8000 real world addresses assigned.  Our problem occurs on both.

3) We currently use multiple /22s with VLAN grouping.

4) Our campus is divided into 7 regions with 5 x /22s for address space.

5) Our lease timers are 15 minutes.

 

Note the large address pools and small lease timers are in place in an attempt to mitigate the DHCP issue.

 

Controller:

 

1) We have 2 x 7240 controllers configured as master/standby.

2) Running 6.3.1.11 currently.  Our SE has recommended an upgrade to 6.4.2.4.

3) We do not have "Enforce DHCP" configured.  I have tested this, and it is not effective for the issue at hand.

4) "User Idle Timeout" is at 300 seconds.

5) "Station Ageout Time" is at 1000 seconds.

Note the large address pools and small lease timers are in place in an attempt to mitigate the DHCP issue.

 

A quick description of the issues we see with Android:  1) The devices respond to gratuitous ARPs while asleep.  This prevents the controller from aging them out of the user table 2) Upon waking from sleep the devices continue to use their previous leased address without communicating with DHCP.  In many cases the address has been freed and assigned to a new device.  3) The Android device can begin using a second IP address without a DHCP conversation.  This address is one that has been leased by the device previously.

 

For more information  I will point to the detailed analysis at Princeton:  https://www.net.princeton.edu/android/  

 

I will add to this that Google has closed all of these issues as resolved in Android 4.2.  However, I have seen these issues on every version of Android from its inception through the current release.

 

Thank you in advance for any insight.

 

John

 

wright-johnp,

 

Let us see who else can add their experiences to your post here.  I just wanted your experience to be out there so that others could respond.

 

This sounds awful. The client you describe is not abiding by the RFC for DHCP. We see >100K unique clients daily, but we have not come across this issue. (Not to say it’s not there; we just haven’t heard/found it.) I would attempt to reproduce it in a lab environment with no encryption so you can more easily perform packet captures. Most curious would be to see if you can see it never sending DHCPDISCOVER/DHCPREQUEST messages. If you do find this to be prevalent, I’d have them derive addressing in RFC1918 space inside some supernet as to not affect your other clients.

We have many controllers.
AOS 6.3.2.11
Not enforcing DHCP in aaa profile.
User idle = 300s
DHCP max/default/min lease time = 600s
BOOTP disallowed

- Ryan -

I still do not understand exactly why people use such short lease times, especially when they have authenticated networks.  Do some vlan pools if you have a lot of users and use longer lease times.  This to me would solve the problem.  Though we do not have as many users (not even close) we have never seen any issues with androide devices, but that is most likely due to a much longer lease time for our DHCP server.

 

Having only a 15 minute lease timer seems excessive.  Bump it out a couple of days and I would imagine you would not see this issue crop up.  I think the small lease times are what are causing the issue...

 

For guest networks it would make sense to make it shorter, but even then 15 minutes is just way too short... Bump it out a couple of hours.

 

Dan, I assume our environments are just simply different. We give clients public IP space and have a limited amount of addressing available. We support ~50,000 concurrent clients and use VLAN pooling predominately with /23s. We have 10 minute lease times to ensure that a student pulling their iPhone out of their pocket, looking at the time, and then putting it back in their pocket does not consume an IP address longer than necessary. We experimented going from 1hr -> 30min -> 15min and eventually settled on 10min.

Hopefully this provides an example for why short lease times are essential for some environments (like ours).

- Ryan -

Like Ryan, we also have short lease times for the exact same reasons. Ours are currently set at 15 minutes in most circumstances.

David
--------------------------
David Morton
Director, Mobile Communications
University of Washington

(config)# ipv6 enable

 

 

 

 

 

 

Sorry, I had to. :-)

Aruba’s IPv6 knob is hardly magical…. unless it builds a NAT64 supporting core along with it . . . :)

"Upon waking from sleep the devices continue to use their previous leased address without communicating with DHCP." Is this only noticed because the device is using the same IP Address, or has traffic been captured to verify that there is no DHCP communication at that time? I read the Princeton issues and it stated specifically the device is using an address that was handed out to another device. If traffic was captured and verified that no DHCP traffic from the problem device exists, then ignore. If not, verify that DHCP traffic does/doesn't exist through a capture. If it exists, then it could be an issue with stale arp cache. DHCP tries to ping the address but is unable to and then allows the address to be assigned. 

@Dan, Ryan, David,

Thanks for your replies. Much like Ryan and David we are attempting to balance pool availability with the size of our collision domains and lease timers. The Android issue that we are experiencing makes maintaining available addresses even more difficult; larger pool sizes and smaller lease timers are an attempt to mitigate this.

 

@Tim

Yes, the IPv6 knob... I'm hoping to save that for the day before I retire... ;)

 

@Dwillynt

Here's a step-by-step version of what we're seeing. Given the randomness of the clients involved along with the sheer amount of traffic involved getting a capture that contains all pieces of this is problematic.

 

1) Android device (we'll call this "the offender") obtains an address via DHCP (standard 4 way handshake).
2) The offender maintains its address for a period of time, then ceases communication with the DHCP server (seen via DHCP logs).
3) At this point, the offender still has a MAC/IP pairing in the Aruba controller's user table. This entry will not time out because many mobile devices respond to gratuitous ARPs without waking the host operating system.
4) At some point the DHCP server marks the offender's IP as available for use.
5) A second device ("the victim") is assigned the IP (seen via DHCP logs). The victim receives a "limited connectivity" error because:
6) The offender's user table entry still exists. The controller cannot create a second user table entry for the same IP.
7) If the offender's user table entry is cleared a user table entry is created for the victim. This immediately resolves the victim's connectivity issue.

 

John

 

In the fall of 2013 we had similar issues with Androids running (KitKat?) coming onto campus obtaining and holding leases after a sleep cycle that *should have* expired them from the user tabel and the DHCP lease bind. The effect was issues with new devices grabbed what should be a valid lease. Just like at Princeton. If I recall, enabling both Enforce DHCP and prohibit-ip-spoofing caused the ruckuss to slow down. IP spoofing is enabled by default and makes the controller deny multiple MAC addresses using same IP address. Note that this command must be run on each controller. It is not pushed out by the master.

 

John, it looks like you have lease times longer than idle times, which is what you want. Ideally, the device should be removed from the user table before it's lease expires and not the other way around. Experimenting with time diffs we ended up with 16 minute DHCP lease times and a 14 minute Global User idle timeout. Lease times are short strictly for IPV4 conservation. Otherwise, we would happily have them longer. Uber short lease times can do a number on the DHCP server's CPU.

 

Also, I'm not sure if the controller "learns" anything from a DHCP transaction when Enforce DHCP is enabled (e.g. does it know when the lease half life is up?), other than that a DHCP conversation actually took place. How did you verify that command wasn't applicable?

 

Mike

Mike,

 

Thanks for the reply.  I will definitely have to consider Enforce DHCP.  It's relevant and your issues were resolved.

 

In the testing that on Enforce DHCP that we've done, the Android device:

 

1) Associates with the network.

2) Leases an address.

3) Has an entry created in the user table.  DHCP is marked valid for this entry.

4) Goes to sleep for a period of time which should see its user table entry deleted.  However, the device responds to gratuitous ARPs from the controller while asleep; the controller does not delete the entry.

5) At this point the lease is expired and subsequently marked free.  There is no DHCP conversation between the Android and the DHCP server.  However, DHCP in the user table entry is still set to valid.  This state does not appear to be maintained beyond initial creation.

6) A second device is assigned the "available" address via DHCP.  A user table entry cannot be created because the Android's entry is being maintained via gratuitous ARP.

 

John

 

Step 5 mentioned shouldn't be happening. If the mobile device is answering gratuitous arps, then it should answer when the new device sends out the gratuitous arp to see if the address is already used. The victim device should then refuse the already used address. I would look to see where that process is failing.