Wireless Access

I first noticed the issue because some of our servers became unavailabe to our wireless users.  Our server subnet is 10.0.0.x and IOS devices were starting to show up under GUI and user-table of having 10.0.0.x IPs.  It's like some ARP poising is happening.

 

Our setup:

We have 2 aaa profiles.  Secure and guest.

Both have the enforce DHCP option check.

We use an external DHCP server, however it has NO 10.0.0.x scope in it.

 

 

Could this be a software bug?  We are running Version: 6.3.1.8 on a Aruba 6000.  Perhaps someone else has seen something similar to this..

 

Here is from the CLI:

show user-table | include 10.0.0
10.0.0.2 44:d8:84:80:bd:4c guest 00:05:47
10.0.0.5 d8:d1:cb:a3:25:60 guest 00:06:32
10.0.0.6 78:31:c1:3a:9c:a0 guest 00:05:54
10.0.0.7 0c:3e:9f:3f:7e:cb guest 00:06:45
10.0.0.8 c0:63:94:ae:0c:1f guest 00:04:35
10.0.0.9 88:1f:a1:d8:96:da guest 00:04:43
10.0.0.10 0c:3e:9f:dd:c1:94 guest 00:05:36
10.0.0.11 dc:86:d8:90:4e:13 guest 00:05:41
10.0.0.13 8c:29:37:7c:23:d5 guest 00:04:46
10.0.0.21 3c:ab:8e:d4:7f:19 guest 00:06:30
10.0.0.28 e4:25:e7:96:68:cc guest 00:06:19
10.0.0.66 64:76:ba:e4:5e:fc guest 00:06:21
10.0.0.99 2c:be:08:c0:33:26 guest 00:00:02

Seeing similar results.  Only two of 20,000+.  Any reason why?

 This could require some investigation.  Type "show user mac <mac address of device>" and see if it has more than one ip address.  Does it say what access point those users are on?

 

I ran the command on a single MAC addy and it does have 2 addresses and it's connected to a single AP.

 

 

However, I am seeing this on multiple APs throught our site.  It's not confined to any single AP.

Type show user-table ip <10.x.x.x> | include DHCP to see how it got the address in the first place:

 

If it got it from DHCP, double-check that your AAA profile has enforce dhcp..

 

Doesen't seem to like that command

 

Here is what I typed:

 

(192.168.1.3) # show user-table ip 192.168.1.121 | include DHCP
DHCP device-id info - Index: 89, Option: 010F03062C2E2F1F2179F92B, Device:  Group: Windows
Address is from DHCP: yes

 

 

Interresting..

 

#show user-table ip 10.0.0.180 | include DHCP
Address is from DHCP: yes

 

I also verified that the enforce DHCP option is checked.

do this:

 

config t
logging level debugging network subcat dhcp
logging level debugging network process dhcpd

 Then, kick your client with two ip addresses off the network like this:

 

aaa user delete mac <mac address>

 

Then type "show log network 50""

 

Oct 2 14:26:11 :202541:  <DBUG> |dhcpdwrap| |dhcp| Received DHCP packet from Datpath, sos msg hdr flags 0x440 opcode 0x5a ingress 0x10020 vlan 1 egress 0x1 src mac 90:68:c3:ed:d0:31
Oct 2 14:26:11 :202536:  <DBUG> |dhcpdwrap| |dhcp| Datapath vlan1: REQUEST 90:68:c3:ed:d0:31 Transaction ID:0xda19b573 reqIP=192.168.1.98 Options 3d:019068c3edd031 39:05dc 3c:6468637063642d352e352e36 0c:616e64726f69642d65633466376639303138653534353335 37:012103060f1a1c333a3b
Oct 2 14:26:11 :202523:  <DBUG> |dhcpdwrap| |dhcp| dhcprelay: dev=eth1, length=315, from_port=68, op=1, giaddr=0.0.0.0
Oct 2 14:26:11 :202532:  <DBUG> |dhcpdwrap| |dhcp| got 2 relay servers
Oct 2 14:26:11 :202533:  <DBUG> |dhcpdwrap| |dhcp| Relayed: DISCOVER server=192.168.1.32 giaddr=192.168.1.3 MAC=90:68:c3:ed:d0:31
Oct 2 14:26:11 :202533:  <DBUG> |dhcpdwrap| |dhcp| Relayed: DISCOVER server=192.168.1.31 giaddr=192.168.1.3 MAC=90:68:c3:ed:d0:31
Oct 2 14:26:11 :202541:  <DBUG> |dhcpdwrap| |dhcp| Received DHCP packet from Datpath, sos msg hdr flags 0x42 opcode 0x5a ingress 0x0 vlan 1 egress 0x2140 src mac 00:0b:86:61:24:b0
Oct 2 14:26:11 :202536:  <DBUG> |dhcpdwrap| |dhcp| Datapath vlan1: REQUEST 90:68:c3:ed:d0:31 Transaction ID:0xda19b573 reqIP=192.168.1.98 Options 3d:019068c3edd031 39:05dc 3c:6468637063642d352e352e36 0c:616e64726f69642d65633466376639303138653534353335 37:012103060f1a1c333a3b
Oct 2 14:26:11 :202523:  <DBUG> |dhcpdwrap| |dhcp| dhcprelay: dev=eth1, length=315, from_port=67, op=1, giaddr=192.168.1.3
Oct 2 14:26:11 :202541:  <DBUG> |dhcpdwrap| |dhcp| Received DHCP packet from Datpath, sos msg hdr flags 0x42 opcode 0x5a ingress 0x2140 vlan 1 egress 0x10020 src mac 74:9d:dc:4b:08:41
Oct 2 14:26:11 :202544:  <DBUG> |dhcpdwrap| |dhcp| Datapath vlan1: ACK 90:68:c3:ed:d0:31 Transaction ID:0xda19b573 clientIP=192.168.1.98

 

 

 

Ok, the client never re-connected, so I am going back through the logs to see if I can find the origional connecton.

 

Err.. logging was not enabled prior, nevermind.

Device Type: iPhone6,1/7.0.6 (11B651)

?

Sorry all the staff and students went home for the day so I am not seeing any rogue devices.

 

Are they all iphone 6?  They are all at least apple.  If they are all iphones, I might have an idea what's going on, but it's a long windy story.

 

They are all iOS devices thus far. I was thinking it was air drop or something, but I can't replicate it myself.

Jes,

 

Let us just look for anything that has a 10.x.x.x address tomorrow.  We want to know if the controller sees that it got its ip address from DHCP or not.  If not, then we do not have enforce DHCP on the correct AAA profile.  If it DOES get it from DHCP, according to the controller, then the show log network 50 statement should tell you what DHCP server gave it what ip address and  you might have a rogue DHCP server.

 

Can I search the logs someway?  I show the last 6000 lines and that is the span of 3 minutes.  Lot's of info.

Show log network all | include 10.

Whole lot of nothing in the logs so far.

The logs are just saying all users who get IP addresses. We want to find users who got 10.x addresses and find out if we see where they got them from. TAC should be able to help you with this.

 

Eh, go lookig for trouble and you find it...

 

Yes I can verify I have these too, mostly from Androids in my case.

 

My working hypothesis is the controllers mistakenly snooping an address out of a initial DHCPRequest, ARP, or IP packet from a stale home wifi or mobile carrier connection.  All the strange addresses are either 10.x, assigned to carriers, or the firewalled-but-public DoD ranges that big carriers squat on.  There's too much chunking for these to be just randmly distributed.

 

The clients actually get normal on-network addresses and those are the ones they are using, it is just a stray user-table entry.  Having servers in 10.0.0.x in general sets you up for more collisions with home networks.  It may be worth developing a plan to gradually move servers to a less obvious subnet of 10.

 

I took debug logs, compared them to user table entries, and found that there was DHCP activity by these clients at the same time these user entries are created.  However the logs for the traffic only ever has the correct address that the client got from our external DHCP server, no sign of the bad adresses there.

 

(Also had a stray client manage to accidentally linger on the control plane long enough to pull an AP IP address, despite denyall initial/preauth role.  That's a different issue.   I'll have to assign a default vlan to that profile someday to avoid the race condition there.)

 

 

Here is from the log this morning.

 

 

Show log network all | include 10.0.0
Oct 3 08:09:19 :202536: <DBUG> |dhcpdwrap| |dhcp| Datapath vlan354: REQUEST 78:31:c1:3a:9c:a0 Transaction ID:0xea88cbe7 reqIP=10.0.0.6 Options 37:0103060f77fc 39:05dc 3d:017831c13a9ca0 33:0076a700 0c:546564732d6950686f6e65
Oct 3 08:09:21 :202536: <DBUG> |dhcpdwrap| |dhcp| Datapath vlan354: REQUEST 78:31:c1:3a:9c:a0 Transaction ID:0xea88cbe7 reqIP=10.0.0.6 Options 37:0103060f77fc 39:05dc 3d:017831c13a9ca0 33:0076a700 0c:546564732d6950686f6e65

You should do "show log network all | include 9c:a0

So that you can see everything for that Mac address.

That search yealds no results.

 

I think I will give this a try for now.

 

https://arubanetworkskb.secure.force.com/pkb/articles/HowTo/R-40

 

 

Honestly enforce dhcp should be all you need.

I have that option checked on the AAA profiles that we use.

Jes,

 

I wanted to see if you had a rogue DHCP server on your network, because the ip addresses are coming from somewhere, but it is too hard to determine that through a forum.

 

Please continue to work closely with TAC to find out what is going on, since you are already engaged with them.

 

I could accept the rogue DHCP idea, but this is happening across multiple subnets.  

Okay.  It could be a range of things.  Please work with TAC to determine the issue.  On this forum, we would just be guessing.

 

Removed..

I added the following and the issue of strange IPs in the logs has seemingly vanished.

 

config t
ip access-list session validuser
network 10.0.0.0 255.255.0.0 any any deny position 1
network 192.168.0.0 255.255.0.0 any any deny position 2
show ip access-list validuser

Using the validuser ACL is an Aruba best practice.

---

@cappalli wrote:

Using the validuser ACL is an Aruba best practice.

---

One drawback of using the Validuser ACL is that if your DHCP server for some reason is not giving out ip addresses, you will receive no hint of that in the controller.  You would have to allow .169 addresses in the ValidUser ACL to see .169 users in the controller...

 

Also it's not airtight for this general issue because there could be some strange topologies where getting these ghost user-table entries for an address that actually is in the valid user range would be problematic.  Mostly I expect they would be cleared if another user hopped on that address though and previously held addresses would be safe via IP spoofing protection.

 

But a good workaround for the majority of cases.

 

if i see this with mobile phone devices (that actually have 3g / 4g) enabled i always it asume it is their mobile provider assigned IP.

Hopefully you have a rule in your ACL to allow traffic for your valid users...

Yes


Priority  Source                   Destination  Service  Action  TimeRange  Log  Expired  Queue  TOS  8021P  Blacklist  Mirror  DisScan  ClassifyMedia  IPv4/6
--------  ------                   -----------  -------  ------  ---------  ---  -------  -----  ---  -----  ---------  ------  -------  -------------  ------
1         10.0.0.0 255.255.255.0   any          any      deny                             Low                                                           4
2         192.168.0.0 255.255.0.0  any          any      deny                             Low                                                           4
3         169.254.0.0 255.255.0.0  any          any      deny                             Low                                                           4
4         127.0.0.0 255.0.0.0      any          any      deny                             Low                                                           4
5         224.0.0.0 240.0.0.0      any          any      deny                             Low                                                           4
6         255.255.255.255          any          any      deny                             Low                                                           4
7         240.0.0.0 240.0.0.0      any          any      deny                             Low                                                           4
8         any                      any          any      permit                           Low                                                           4
9         fe80::                   any          any      deny                             Low                                                           6
10        fc00::/7                 any          any      permit                           Low                                                           6
11        fe80::/64                any          any      permit                           Low                                                           6
12        ipv6-reserved-range      any          any      deny                             Low                                                           6
13        any                      any          any      permit                           Low                                                           6

 

 

We run with Enforce DHCP, Prohibit IP spoofing, Prohibit ARP spoofing, Prevent DHCP exhaustion, and a filter on the users preventing them from sending DHCP server response packets.

 

We still get these entries.

 

Something isn't getting initialized early enough in the filters area, and I don't think the DHCP request is needed to cause the user-table entry to build, because I am not seeing them; it is probably being built from any old packet.

 

I'd be interested to know if the valid-users policy actually works.