Wireless Access

Experiencing a very frustrating issue with our current RAP architecture.  The DHCP scope that serves users behind our RAPs gets flooded with BAD_ADDRESS entries. This causes all addresses to become available and users are unable to get on the network.  We have 25-30 scopes in our environment and this is the only scope that it's happing on.  Architecture looks like this:

 

7210 HeadEnd ===INTERNET===RAP-155/RAP-3 === Small Switch

 

The SSID and the wired connections in the switch are all on the same VLAN/network.  The default gateway handed out to all clients on that network is our core routers, not the controller.  Not sure if that matters. 

 

Anyone come across a similar situation?  We've sniffed the DHCP server VLAN and there is no smoking gun, yet the scope continues to fill up.  

I have a TAC case open, however I thought I'd throw it out here as well.  Thanks!

 

 

What is the DHCP server and where is it located? Is it on the local VLAN, or behind the core router?

 

 

We use Windows Server 2012 R2, which sits on a server VLAN on our core (Cisco Nexus 7K's). 

We have 2 DHCP servers actually....scope is /22 and split in half between the servers. 

And the BAD_ADDRESS messages show up on the DHCP servers, correct? What is relaying DHCP to the servers? Is the controller acting as a relay, or the core router? 

Correct.  There is a DHCP helper on the L3 interface for the network for RAP users that points users/devices to the DHCP servers to acquire an address.  Controller is acting as a relay.

So you can see what I'm talking about, I cleared all the BAD ADDRESS entries out of DHCP about an hour ago and here is what's already repopulating:

Most of what I've found for that error seems to be related to either rogue DHCP servers in the same scope, or devices that misbehave and don't renew properly. If you have a wireshark capture of the DHCP sequence (filter on bootp to just see the DHCP conversations), do you see any patterns such as a common mac address in the requests that lead to DHCP NAKs? 

Hi

Did you find any solution? In the last few days, the wireless scopes have been filled with bad_address, there are some clients who fill the entire scoop. I have not seen the same error on the cable scoop. We run DHCP on Windows 2016.

 

M

No solution yet.  We've separated the RAP wireless from the RAP wired by assigning the wireless clients a new VLAN.  The wireless scopes are now completely clean with no BAD_ADDRESS entries accumulating.  The wired side continues to accumulate.  This has something to do with a Cisco switch behind a RAP.  TAC is researching, but I am not hopeful.

By manually putting an IP helper on the vlan on the controller we've slowed down the issue, although it still exists.  Instead of 20+ bad address errors per day, now we're getting only 2 or 3.  I also decreased the lease times to 8 hours from 8 days.  This seems to have had no effect. 

I'm going to research the "enforce dhcp" option on the controller and see if that may be a potential solution.  

You're using the controller as an DHCP relay? In my environment I used the SVI on a Cisco switch. Not saying you're incorrect just throwing that out there.

 

You're using two different DHCP servers for a single scope? Sounds like a race to respond. How would one server be aware of an addr which was handed out by the other server?

 

I have all my helpers on the SVI's as well (Cisco Nexus 7K as core).  I've always left off helpers on the vlans and their SVIs on the controllers.  To try and troubleshoot this issue I added them to the vlan that was having the issue.

yes, I'm using 2 different DHCP servers.  However, we have failover configured so that both servers are always in sync.  Reservations only need to be made on one of the servers, etc. 

Two thoughts. 1. In your topology diagram when you say Small Switch are you describing an unmamanged switch? 2. Check the windows event viewer for DHCP. Lots of juicy info in there.

No, the switch is managed, but it is all vlan 1/out-of-the-box with 802.1x turned on.  Looked at the event viewer already and yes, there's a ton of info in there, but can't find anything pertinent to what I'm experiencing.  

This morning, however, I may have found a smoking gun.  Happened to be monitoring DHCP when a bad address error came.  Was able to look up the IP in the ARP table on the Nexus 7K's and get the MAC. Then looked up the MAC on the controller and was able to trace it down to the RAP and Cisco switch hanging off of it.  Noticed several dot1x errors and port up/port down entries inthe switch logs.  So i looked at our ClearPass access tracker and found the device was bouncing between dot1x and mac auth. This may be ClearPass related, but I'm on with TAC right now and will dig deeper. 

I think you're on the right track looking into the enforce DHCP option. I would still suggest investigating your DHCP traffic to see if there isn't a rogue DHCP server/device that's answering clients that you're not aware of. It could also be an issue with the overlapping scopes between the two Windows 2012 DHCP servers. I haven't personally implemented that kind of failover with Windows DHCP servers. With 2012, my experience was always that split scopes were more reliable. To do DHCP redundancy (at that time), it usually meant going to Infoblox or BlueCat for an appropriate synchronized solution.

Were you able to find a fix for this? I am experiencing the same issue. We have multiple sites all connected Via MPLS with Centralized DHCP. I added the DHCP server IP in as a IP helper address on the Router at the branch site. Was curiouse if that needs to just be the IP of the Clearpass server. 

Is it causing problems? You might want to check the DHCP server logs. If it's Windows, it's all in Event Viewer.  You were correct to add the DHCP server as an IP helper address (otherwise the layer 2 DHCP requests would never reach the server). I believe you CAN add clearpass as an IP helper, for example when you're using Clearpass as a NAC but I don't think that's relevant to your situation.

The issue subsided when I split the wired and wireless into 2 separate scopes.  They were previously all on one.  I also shortened the lease time to 8 hours (from 8 days).

In answer to your question, you should have the ClearPass IP as a helper AND your DHCP server.  The DHCP server will provide the IP and the ClearPass IP will allow ClearPass to see the DHCP request and add it to the endpoints database.  

I removed the Clearpass configuration on my switchport so it would no longer authenticate with the NAC server. This resolved the issue. It looks to be a Clearpass NAC config issue.

Any update from Aruba TAC on this issue. I am experiencing it as well

I added the following command to the switch and it resolved my issues.

 

    ip device tracking probe auto-source override

We ran into a BAD_ADDRESS issue as well while testing Aruba gear in our environment. We escalated through engineering, and it was determined the culprit was the Openflow controller (used by AirGroup) causing an incorrect packet to be returned to the client during a DHCP lease/renewal that cause the client to think the address was already in use on the network, so it would reject the lease thus the BAD_ADDRESS on the server. Disabling Openflow resolved the issue. We then turned Openflow back on (from /mm) with a new 'host timeout' timer of 1800 from the default 300, and this helped as well. I think we still see the occasional issue, but not enough to cause impact. I haven't heard if there is any permanent fix yet. 

How were you able to turn off this feature on the controller side?? What commands etc??

---

@cm119 wrote:

We ran into a BAD_ADDRESS issue as well while testing Aruba gear in our environment. We escalated through engineering, and it was determined the culprit was the Openflow controller (used by AirGroup) causing an incorrect packet to be returned to the client during a DHCP lease/renewal that cause the client to think the address was already in use on the network, so it would reject the lease thus the BAD_ADDRESS on the server. Disabling Openflow resolved the issue. We then turned Openflow back on (from /mm) with a new 'host timeout' timer of 1800 from the default 300, and this helped as well. I think we still see the occasional issue, but not enough to cause impact. I haven't heard if there is any permanent fix yet. 

---

 

I was having the same issue. Turning off Spanning Tree in the Web UI seemed to solve the issue for me.

Maybe not.. .still getting bad addresses.

---

@cm119 wrote:

We ran into a BAD_ADDRESS issue as well while testing Aruba gear in our environment. We escalated through engineering, and it was determined the culprit was the Openflow controller (used by AirGroup) causing an incorrect packet to be returned to the client during a DHCP lease/renewal that cause the client to think the address was already in use on the network, so it would reject the lease thus the BAD_ADDRESS on the server. Disabling Openflow resolved the issue. We then turned Openflow back on (from /mm) with a new 'host timeout' timer of 1800 from the default 300, and this helped as well. I think we still see the occasional issue, but not enough to cause impact. I haven't heard if there is any permanent fix yet. 

---

FYI, if you check recent release notes, they have now made the openflow controller 'host timeout' default value 3600 from 300 seconds. So I think this tells you they had the defaults wrong to begin with. If you are having this issue, and are on any previous code where the timout was set to 300, I would recommend updating to 3600 to see if it resolves the issue. Check the recent (8.5.0.x I think) release notes for the info and commands. 

Having a very similar problem. Did you find the root cause or a solution?

I added the following command to the switch and it resolved my issues.

ip device tracking probe auto-source override

Sorry, I know this is an old post but this tripped me up this week and this is how I got the othe bottom of it

 

DHCP Scope: Full of BAD_ADDRESS Entries

 

Pete