Higher Education

Hi,

 

Always surprises with new ARUBA codes now a days, I am in a bad shape with the new issue i am facing with 6000, now the box is with 6.3.0.2, we have DHCP with scopes enabled for 4000 ips, however during morning DHCP lease out all ips even though the number of users are not more than 1600, vlan-assignment count and dhcp lease count are not matching at all, also noted that DHCP report shows that users getting multiple ip for same device, contacted ARUBA support, they suggested to change the hash algorithm to even, no luck, then fast age enabled, removed vlan pool name and added vlans manually, now waiting for the next day, realy a nightmare situation evry morning, could someone come up with an advise.

 

Thank you

 

 

I don't have experience with Aruba's internal DHCP server, but can you decrease the lease time to something aggressive like 10 minutes?

Edit: Unrelated

We use microsoft DHCP, we tried all different settings with leasetime, the issue doesnt point to the lease time. vlan assignement table show a figure which is compeltely lower figure that the total dhcp addresses leased out, also what triggered to lease multiple ips for same device in DHCP.

What model of controller do your have? Controllers older that the 7200 series have a maximun of 512 leases and are not designed for Production use.

I notice the limit on the 7210 is 5000 leases.

#7210

---

@nointerference wrote:

We use microsoft DHCP, we tried all different settings with leasetime, the issue doesnt point to the lease time. vlan assignement table show a figure which is compeltely lower figure that the total dhcp addresses leased out, also what triggered to lease multiple ips for same device in DHCP.

---

Wireshark at the DHCP server.  With an external (to the controller) DHCP server that would be my first step in figuring out "also what triggered to lease multiple ips for same device in DHCP."    Are the multiple leases per device offered at the same time? 

Oops. I missed the detail that you are using a Microsoft DHCP server

 

:(

We use IPAM appliances for our DHCP. Due to class change times (overlap of users in the building - those leaving class and those awaiting to walk into a class), we have our lease times set for 15 min minimum lease time, 20 min default lease time, and 21 min maximum lease time. We had to keep the maximum close to the default because Apple devices (at one time) tended to ignore the default time and went for the maximum time.

it looks like not a DHCP server or a lease issue, how same device can have different ip, also we do have enogh ips in dhcp scope to handle the number of devices. It's all happening in first 40 mins, but controller outputs shows not that many devices is assigned in those scopes. Show vlan-assign.

 

 

---

@nointerference wrote:

it looks like not a DHCP server or a lease issue, how same device can have different ip, also we do have enogh ips in dhcp scope to handle the number of devices. It's all happening in first 40 mins, but controller outputs shows not that many devices is assigned in those scopes. Show vlan-assign.

 

 

---

Well, it sounds like your DHCP server disagrees.  Sounds like controller shows n number of clients; but your dhcp server believes it has dished out more (n+something) IPs than that.  My advice stands - I'd start with a packet capture (and/or a detailed analysis of the DHCP log) at the DHCP server.

 

Are you seeing lots of DHCP NAKs from clients?  Are you seeing multiple Discover/Requests per device?

we had wireshark not seen anything unusual, dhcp server is common for wired and wireless, but i am sure no sharing b/w them.

We had similar issue when we tried the even VLAN assignment. Users will connect and get a lease on VLAN1 (for example), then they will disconnect and connect again before their lease for VLAN1 expire and after the user entry in the controller times out, and get put by the controller in another VLAN2 but the even assignment where they get another lease. This can happen multiple times for the same user and he ends up with number of active leases in different VLANS. We have never had problems with hash based VLAN assignment , so this may not be exactly your case.

Hell dpi,

 

we are very close in syptom with even algorithm, after change to even from hash i had same indications as u mentioned, how u sorted out the issue, were u had DHCP scope full issue. Hash the issue we faced is that twelve vlans in the pool, 3 r fully subscribed but controller still assign clients the vlan id of fully subscribed scopes.

We reverted back to hash and haven't used the even VLANs since. If you have hash for a VLAN assignment algorithm and still have full DHCP scopes, you may need to add more VLANs to the pool.

Our understanding for the even VLAN failure/feature is that because the controllers don't communicate with the DHCP servers they don't know if the users already have DHCP lease on one of the subnets/VLANs. The controller will keep information for the last VLAN assignment until the user entry the controller user table expires, and after that if the same user reconnects he will get put into the next in line VLAN (which most likely won't be the same as before).

There may be a way to match the lease time with the user table entry timeout, but that will lead to too short leases or too long timeout  (at least in our case).

fwiw, we are using 10 minute leases. If you have a lot of churn in your wireless network, consider shorter leases.

- Ryan -

Ah! Welcome aboard the ?vlan pooling hash algorithm is uncomfortably inefficient? train!

I track this very closely in our environment and observe up to a 20% variance between how subscribed the most and least utilized network are. 20% is just awful. The problem is that while the controller does see the DHCP traffic from the client/server, it does nothing with this from an intelligence standpoint. This is something wherein I envy Cisco customers. Cisco?s ?VLAN select? has the controller aware of DHCP traffic. If it observes that clients are not getting a lease from a particular vlan, that vlan is marked dirty is essentially taken out of the pool for a period of time. In your case then, those 3 fully subscribed would not be in the pool, thus leaving 9 available for use by the hash algorithm.

I have Aruba performing a deep analysis/comparison between the efficiencies of hash vs. even, and I?m hopeful they can support their claim that even is more efficient by illustrating this with metrics. I?ll certainly share with anyone that wants it (if I ever receive it).

For now, our solution is to throw a ton of address space at the problem, but moving forward, we?re also going to place our students? mobile devices (phones/tablets) onto RFC1918 space.

- Ryan -

A couple items:

1. show ap vlan-usage indicates what Mac addresses in the controller's station table have been assigned to each vlan. This is separate from L3/dhcp information. I just wanted to clear this up in case of ambiguity.

2. If I'm a client and receive IP-1, then immediately send out a DHCPDISCOVER packet, the dhcp server (per the RFC) will send a DHCPOFFER ...UNLESS your vendor (MS) has a "one lease per client" option. Can you verify there are discovers from clients within the max lease time window after they've already received an address?

3. What are your lease times configured?

Hello Ryan,

 

1) show ap vlan-usage, and show vlan-assignment, are they give same info?, how to find L3/dhcp info.

 

 

 

I will check for ur second question.

thanks

 

 

Hmm . . . oddly enough, they?re different it seems. When I run these two commands, I get the same format (vlan and client #s) but with different values. I don?t know from where each is populated, so maybe Aruba will see this thread and chime in.

Where to get L3/dhcp info? Your DHCP server. My experience is that using Windows event viewer for MS DHCP logs is hideous. Take a look at ?Splunk? as a logging repository, which could do a lot of MS DHCP analysis for you.

- Ryan -

---

@nointerference wrote:

Hi,

 

Always surprises with new ARUBA codes now a days, I am in a bad shape with the new issue i am facing with 6000, now the box is with 6.3.0.2, we have DHCP with scopes enabled for 4000 ips, however during morning DHCP lease out all ips even though the number of users are not more than 1600, vlan-assignment count and dhcp lease count are not matching at all, also noted that DHCP report shows that users getting multiple ip for same device, contacted ARUBA support, they suggested to change the hash algorithm to even, no luck, then fast age enabled, removed vlan pool name and added vlans manually, now waiting for the next day, realy a nightmare situation evry morning, could someone come up with an advise.

 

Thank you

 

 

---

Nointerference,

 

There is no direct connection between the controller and how many leases are provided via the DHCP server.  The controller can attempt to hash clients via the mac address, or load balance them via "even" pools, but there are a number of factors that can affect that balance.  The controller, after user derivation or pooling determines that a user belongs in a VLAN and just bridges that traffic to that VLAN.  After that, your underlying network and clients do  more to determine what goes on with address depletion.

 

Here are factors that can influence DHCP depletion:

 

- DHCP server having reserved leases - That will lead to unbalanced DHCP addresses.

- DHCP server scopes having uneven lease times - That would mean some VLANs would get utilized before others, because leases would not be released evenly.

- DHCP scopes having uneven sizes - This would mean some VLANs get utilized before others, because the controllers assumes that all VLANs are the same size.

- DHCP scope lease time that is shorter than the user-idle-timeout on the controller - This would mean that the controller would keep users in the user table longer than the scope will.  Which means that if a DHCP address is released, but the ip address of that user is still in the user table, the controller will flag it as a duplicate ip address and possibly not let it connect.

- User or server derivation rules that place users in specific VLANS, depleting DHCP scopes, but falling outside of load balancing mechanisms - The controller does not count users that are placed in VLANs in this manner in the "even" load balancing scheme.

- DHCP scopes shared among more than one Virtual AP - Each Virtual AP has its own balancing mechanism, and it does not account for other Virtual APs using the same VLANS.  Load balancing is only kept track of within the specific Virtual AP that it is assigned from.

 

To keep your VLANs even you should:

 

- Not share VLANs between wired and wireless devices

- Do not share VLANs between virtual APs

- Do not share VLANs between controllers

- Use EVEN balancing with the Named VLAN feature

- Do not use user derivation or server derivation to put users in VLANs that are already in Virtual APs that you want to remain balanced

- Make sure VLAN sizes are all identical within the single Virtual AP

- Make sure all VLANs have the same lease lengths

- Use "Enforce DHCP" on the AAA profile so that "Ghost Addresses" like phone WAN addresses or VMWARE addresses do not end up in the user table

- Manually clear out ALL leases on all scopes to ensure that you start from scratch.  Make sure no clients are on the WLAN at that time, or take the WLAN out of service before you clear.

- Make sure that you have no DHCP reservations

 

Outside of DHCP depletion, in general:

- TEST all versions of code and configurations in a lab, before you upgrade.  Aruba makes changes and outlines the changes in the release notes, but not every possibility is tested before code is released.  That means there will be design decisions made by users that Aruba is not aware of, or did not test and does not understand the impact of changes.  It falls on the Network Administrator to test new code to ensure that his/her custom design continues to work as intended.

 

Thank you for the post which is very much helpful for anyone who visits this community.

 

I need to get clarified with few things.

 

How can I resolve the issue with hash algorithm which oversubscribe some scopes in DHCP where most of them are not even 50% utilized.

 

Then we tried even with preserve vlan, and also without preserve vlan setting. We had issue of oversubscribing all vlans by each device having multiple ips.

 

Thank you.

---

@nointerference wrote:

Thank you for the post which is very much helpful for anyone who visits this community.

 

I need to get clarified with few things.

 

How can I resolve the issue with hash algorithm which oversubscribe some scopes in DHCP where most of them are not even 50% utilized.

 

Then we tried even with preserve vlan, and also without preserve vlan setting. We had issue of oversubscribing all vlans by each device having multiple ips.

 

Thank you.

---

I just posted a whole list of things that you can check, to see if it is affecting your situation.  Some of the things are in the Aruba configuration and other things are in your infrastructure.  Please check through all carefully to see if any contribute to your issue.

 

"Preserve VLAN" will only complicate your situation and should not be used, so that you can get to the bottom of your issue.

 

If you are using HASH, statistically there will be wasted ip addresses and you need to provide enough addresses and VLANs to account for that.  The distribution will NOT be even.  

 

Ultimately, you might have to get a consultant to come into your environment and work with TAC to determine what is the issue.  TAC cannot know everything about your network over the phone...

 

COuld you expalin little more about Use EVEN balancing with the Named VLAN feature.

 

I always appreciate TAC for their support, they are very much helpful.After all i do believe it is the network administrator who is accountable for the local setup.

There is nothing additional to explain.  Even will distribute clients in a Virtual access point evenly between the VLANs defined.

 

I listed things before, that even if you have EVEN vlans configured, will make it uneven.  You need to check to see if you have any of those situations...

nointerference,

 

Regarding multiple IPs per client, please re-read what I posted earlier:

 

2. If I'm a client and receive IP-1, then immediately send out a DHCPDISCOVER packet, the dhcp server (per the RFC) will send a DHCPOFFER ...UNLESS your vendor (MS) has a "one lease per client" option. Can you verify there are discovers from clients within the max lease time window after they've already received an address?

 So, can you look into your DHCP server to restrict this?

---

@Ryan wrote:

nointerference,

 

Regarding multiple IPs per client, please re-read what I posted earlier:

 

2. If I'm a client and receive IP-1, then immediately send out a DHCPDISCOVER packet, the dhcp server (per the RFC) will send a DHCPOFFER ...UNLESS your vendor (MS) has a "one lease per client" option. Can you verify there are discovers from clients within the max lease time window after they've already received an address?

 So, can you look into your DHCP server to restrict this?

---

Ryan,

 

Do you have any screenshots of your DHCP server where this restriction is configured?

 

I?m not using MS DHCP service like you are. We?re using Bluecat?s IPAM solution for DNS/DHCP. There?s a deployment option for ?one lease per client?. I would look through Microsoft?s documentation to see if there?s an equivalent.