Wireless Access

So I'm having sort of a weird problem with my controller (Aruba 6.1.3.4).  Here's a breakdown of my network topology

VLAN 102 Wireless 192.168.102.0/24

VLAN 100 Internal 192.168.100.0/23

VLAN 25 guest 10.10.25.0/24

All of the VLANs are connected via Layer 3 through a Cisco ASA

The controller IP is configured on VLAN 100, which is also where all of the APs are located

The Problem:

When someone on a VLAN other than 100 tries to access the controller (using its VLAN 100 IP address) the controller tries to respond using it's VLAN 100 IP address, but it tags the traffic for the VLAN of the original request

Example:

User on the guest VLAN 25 pings the controller IP 192.168.100.xxx

Ping gets sent through the ASA and arrives at the controller on VLAN 100

Controller receives the ping, then responds on VLAN 25, but using IP 192.168.100.xxx

The ASA drops the response, because the connection doesn't exist (wrong interface)

What could possibly be causing this?

Do you have "IP Nat INSIDE" on any of your VLANs?

Doesn't look like it.  Here's the relevant config:

ip cp-redirect-address 192.168.100.17
ip NAT pool dynamic-srcnat 0.0.0.0 0.0.0.0
ip access-list eth validuserethacl
permit any
!
ip access-list session validuser
network 169.254.0.0 255.255.0.0 any any deny
any any any permit
ipv6 any any any permit
!
ip access-list session vmware-acl
!
ip access-list session citrix-acl
!
ip access-list session ra-guard
!
ip access-list session captiveportal6
!

vlan 25 "Guest"
vlan 100 "Internal"
vlan 102

no spanning-tree

interface gigabitethernet 1/0
description "GE1/0"
trusted
trusted vlan 1-4094
!

interface gigabitethernet 1/1
description "GE1/1"
trusted
trusted vlan 1-4094
!

interface gigabitethernet 1/2
description "GE1/2"
trusted
trusted vlan 1-4094
!

interface gigabitethernet 1/3
description "GE1/3"
trusted
trusted vlan 1-4094
!

interface gigabitethernet 1/4
description "GE1/4"
trusted
trusted vlan 1-4094
!

interface gigabitethernet 1/5
description "GE1/5"
trusted
trusted vlan 1-4094
!

interface gigabitethernet 1/6
description "GE1/6"
trusted
trusted vlan 1-4094
!

interface gigabitethernet 1/7
description "GE1/7"
trusted
trusted vlan 1-4094
!

interface port-channel 0
add gigabitethernet 1/4
add gigabitethernet 1/5
trusted
trusted vlan 1-4094
switchport mode trunk
switchport access vlan 102
!

interface vlan 100
ip address 192.168.100.17 255.255.254.0
no ip routing
!

interface vlan 1
no ip routing
shutdown
!

interface vlan 25
ip address 10.10.25.17 255.255.255.0
no ip routing
ip helper-address 192.168.100.50
!

interface vlan 102
ip address 192.168.102.17 255.255.255.0
no ip routing
ip helper-address 192.168.100.50
!

ip default-gateway 192.168.100.1
no uplink wired vlan 1
uplink disable

Well thats a weird config

You should not need the ip helper on your wireless controller, if you got your ASA  Cisco routing thats for sure...

And it looks like it as you got no ip routing on your interface vlans.. you should just need the ip helper on the one that  its routing on your Cisco ASA

Besides  there is something else that got my attention

interface port-channel 0
add gigabitethernet 1/4
add gigabitethernet 1/5
trusted
trusted vlan 1-4094
switchport mode trunk
switchport access vlan 102

Where are you passing your guest vlan and your internal vlan  to your device that is routing? i mean tagging it i dont see them that you are tagging it like switchport trunk allowed vlan 102,100,25?

I know im not helping you in your quetion but im really curious on how is that working?

Look like for example i just logged in, in one of  the controllers i have configured

interface port-channel 0
        add gigabitethernet 1/0
        add gigabitethernet 1/1                   
        add gigabitethernet 1/2
        add gigabitethernet 1/3
        trusted
        trusted vlan 1,10,12-13,20-24
        switchport mode trunk
        switchport trunk allowed vlan 1,10-13,20-24

interface vlan 1
        ip address 192.168.1.60 255.255.255.0
!

interface vlan 12
        no ip routing
!

interface vlan 13
        no ip routing
!

interface vlan 20
        ip address 192.168.20.1 255.255.255.0
        no ip routing
!

interface vlan 21
        no ip routing
!

interface vlan 22                                 
        no ip routing
!

interface vlan 23
        no ip routing
!

interface vlan 24
        no ip routing
!

Can you explain me ? how can your guest vlan on your controller communicate with for example ASA defaultt gateway of that network without going through layer 2 on tags or without natting through the controller? 

Maybe here is your issue?

well im looking forward to see the resolution! its a weird case you got here

Basically it's set up like this:

Cisco ASA <-- Trunk with all VLANs --> Switch <-- Trunk with all VLANs --> Wireless Controller

Edit: I should clarify that the Port channel interface is the only interface that is actually active.  None of the other ports are used.

The ASA is the default gateway for every VLAN, and routing/firewall is all it does.  DHCP is handled by a server on the internal VLAN (which is why I needed to configure ip-helper).  I guess I could have the ASA or the switch do the DHCP forwarding as well, and actually I'm going to check to make sure it's not set up to do that already (though it shouldn't cause any issues in theory, since they'd both be forwarding to the same server).

I put the "no ip routing" because the controller should not be allowing any traffic to pass between VLANs without going through the ASA.  That could potentially be a security issue, and very likely be a network management issue.  I couldn't really find any good explanation of this feature, so it's quite possible I've misunderstood what it is actually for.

I've just allowed all VLANs on the controller's trunk port because that was the default configuration.  The switch is set to ignore any untagged traffic, and traffic on unused VLANs wouldn't actually go anywhere so I'm not too concerned about that, unless you think it could be causing issues?

This configuration has actually been working very well for the past few days, with the exception of the problem I opened this thread for.  I wouldn't even really care about this, but it's breaking captive portal because users in the guest VLAN can't access the wireless controller to see the login page.

Hope all this is clear, thanks for the input!

The best solution is to use the "ip cp-redirect-address" command to redirect users to the ip address of the controller on the guest VLAN for the captive portal

config t

ip cp-redirect-address 10.10.25.x

Interesting, I guess if you specify "switchport mode trunk" without specifically allowing any vlans it just adds them all.  It's definitely sending regular traffic over all of my VLANs no problem.  If I disable captive portal the guest users traffic goes through the guest VLAN and the internal users go over the internal VLAN.

As for the "ip cp-redirect-address" suggestion, I did in fact try that first.  Unfortunately the controller wouldn't respond to requests from the captive users on that interface.  I think this is because I have a basic license on the controller, and the default firewall rule for captive users only allows them to access the alias named "controller" over http/https which I assume is just the controller's primary IP (In this case, 192.168.100.17 on VLAN 100).

I'll reconfirm this when I get back in the office on Monday, but I'm quite positive that changing the ip cp-redirect address to anything but the controller's primary IP address didn't work.  When I test it on Monday I'll run wireshark on the captive user computer and see if I can glean any more information about what is happening.

---

@scottraymond wrote:

Interesting, I guess if you specify "switchport mode trunk" without specifically allowing any vlans it just adds them all.  It's definitely sending regular traffic over all of my VLANs no problem.  If I disable captive portal the guest users traffic goes through the guest VLAN and the internal users go over the internal VLAN.

 

As for the "ip cp-redirect-address" suggestion, I did in fact try that first.  Unfortunately the controller wouldn't respond to requests from the captive users on that interface.  I think this is because I have a basic license on the controller, and the default firewall rule for captive users only allows them to access the alias named "controller" over http/https which I assume is just the controller's primary IP (In this case, 192.168.100.17 on VLAN 100).

 

I'll reconfirm this when I get back in the office on Monday, but I'm quite positive that changing the ip cp-redirect address to anything but the controller's primary IP address didn't work.  When I test it on Monday I'll run wireshark on the captive user computer and see if I can glean any more information about what is happening.

---

If you type "show trunk" it will probably show you that allowed VLANs is all.  Whenever you create a new VLAN, it will add it automatically.  Please change the ip cp-redirect address to the ip address of the VLAN.  Whenever a captive portal user requests the controller's page, it is designed to return the page on that specific ip address and it was made for your particular situation.  Having it as the management ip address is what makes it answer from there.

Interesting Collin

Thanks for the information

So here are my results from testing using Wireshark

Scenario 1:

Guest network is placed in VLAN 100

ip cp-redirect-address 192.168.100.17

Client attempts to visit google.com

Client gets temporary redirect from spoofed IP address of google.com via the Aruba

Client does DNS lookup for redirect, and successfully connects to the captive portal page

Scenario 2:

Guest network is placed in VLAN 100

ip cp-redirect-address 10.10.25.17

Client attempts to visit google.com

Client gets temporary redirect from spoofed IP address of google.com via the Aruba

Client does DNS lookup for redirect, and received the spoofed DNS reply pointing to 10.10.25.17

Client attempts to connect to 10.10.25.17

In wireshark I see tons of "TCP ACKed unseen segment" errors

Client is unable to connect to captive portal

Scenario 3:

Guest network is placed in VLAN 25

ip cp-redirect-address 192.168.100.17

Client attempts to visit google.com

Client receives no response at all and times out

Scneario 4:

Guest network is placed in VLAN 25

ip cp-redirect-address 10.10.25.17

Client attempts to visit google.com

Client receives no response at all and times out

Thanks for that testing.  Scenario 1 is how it *should* be configured and work.

Unfortunately, putting my guest users in the secure network is not an option for me.  Those users need to be firewalled off from the rest of the network by the ASA, and therefore must be located in VLAN 25.  I guess my only other option is to actually place the controller and all of the APs in the guest network, but that is just as bad for security.

Is my network setup really that strange here?  I cannot fathom why this one feature (captive portal) would not be working, when all other traffic is being routed between the VLANs/subnets without fail.  At this point I'm almost ready to give up the captive portal functionality and just use a static WPA2 key for my guest network.

Are you allowing DNS traffic on the 10.20.25 subnet?  DNS is necessary for that first connection.  What is the DNS server that clients get in that subnet?

Sorry, I should've mentioned that in my testing results.  DNS for google.com worked in all of the scenarios.  Both subnets (internal and guest) use our internal DNS server that is located on the internal network (VLAN 100).

It was only once it actually tried to connect to the IP for google.com that the connection failed for users behind captive portal.  There was simply no response.

Try just typing in the controller's ip address in the browser to initiate the captive portal.

From the guest network, no response from either the local interface on VLAN 25 or the main interface on VLAN 100.  I checked, and the ASA is configured to allow the guest network to access the VLAN 100 IP address.  I'm even getting hits on that rule, but the user never receives any response.

The big question(s)  are:

Does the controller's ip interface on the same VLAN as the client even show up in the client's ARP table when the client tries to access it?  

What is the client's "logon" role rights?

I hadn't thought of checking the ARP tables, but yes, the entry for the controllers IP on the same VLAN does appear and is correct.

Here's the rules for my logon group created by the WLAN wizard

I've been trying to think through this logically (apologies, I'm fairly new to aruba).

Using the ip cp-redirect command, does that logically (within the controller) forward the host to it's captive portal so there is no further traffic, or does it actually have to create a packet itself and then send it.

Eg)

Guest (Vlan25 tunnel) -> AP (100) -> FW (100) -> Controller (100)(25)

10.10.25.X                                                          redirect to 10.10.25.17

At this point, when the guest hits the controller looking to go to the ip cp-redirect address within Vlan25's subnet (10.10.25.17) does the controller just logically know this is the captive portal or does it have to create a packet with the source of 10.10.25.17 and destination of 192.168.100.17 even though this is itself and forward it to the firewall again?

Normally you'd expect a device to just happily look at a routing table and then sort it internally but if it's forced into Layer2 only mode, what does the controller do?

The reason I ask is I'm wondering if the ASA is receiving and sending data on the same interface which is denied by default (same-security-traffic intra-interface) and this is the reason traffic is being dropped? (Although if the firewall is the L3 device, and has subinterfaces for each VLAN then this shouldnt be the case even if the controller forwarded packets back)

Just food for thought perhaps.

I think a big part of the issue here is that the controller is trying to be a Layer 3 device, but it's not doing it consistently or it's doing it wrong.  I still can't wrap my head around how the packets are even hitting the ASA on the wrong interface.  It works out something like this (which makes no sense):

Client on 10.10.25.x sends packet for 192.168.100.x to the ASA on VLAN 25

Controller receives packet from the ASA on VLAN 100

Controller sends response for 10.10.25.x to the ASA on VLAN 25

ASA drops the packet because it is received on the wrong interface

All other issues aside, why would the controller even send this packet to the ASA?  If it's responding on VLAN 25 (which it shouldn't), then it doesn't even need to use the gateway since it's already on that network.

A'ha!  I finally figured it out!

The ASA was configured with a unidirectional NAT rule between the internal and guest networks.  I changed the rules to be bidirectional and everything is working properly now.

I'll admit my understanding of NAT is not the greatest, so I'm not entirely sure why this was causing all these issues.  All other traffic between the guest network and internal network was functioning fine previously.  Maybe someone with some more in-depth networking knowledge can chime in and better explain why this happened?

Edit:

Ah nice to hear it is fixable!

ASA wise, what NAT rules did you have in place? (Is there a reason to use NAT as well as ACLs? :P)

Here's my before and after config, for anyone who's curious!

Before:

nat (inside,guest) source static any any unidirectional

After:

nat (inside,guest) source static 192.168.100.0/23 192.168.100.0/23 destination static 10.10.25.0/24 10.10.25.0/24

With the ASA Version 8.3+ the NAT command structure has been completely changed and new firewall/NAT checks are in place to prevent NAT overlapping and spoofing.

Having an "any any unidirectional" I have no idea what would happen:P  Is nat-control enabled so if NAT is not performed the packet is dropped?

Also are your AP's in tunnel or bridge mode?

I think those rules are the result of the upgrade from a previous version of ASA.  I inherited that config from the previous team who is no longer working here.  I'm not very well-versed in the more complicated NAT scenarios, so I just left it as-is, haha.

nat-control isn't enabled, and the APs are in tunnel mode.

Yeah it seems quite a few people have suffered from issues after upgrading from sub 8.3 to 8.3+ whereby additional commands or keywords have been automatically changed and added into new statements into previously configured commands, the unidirectional keyword being one of them reported!

Without NAT control, routing between interfaces can occur without a require NAT statement so I'm going to guess that it was routing in one direction, and then trying to NAT in the other?

nat (internal,guest) any any unidirectional  -  to me that reads traffic from Vlan100 -> Vlan25, NAT.

So the original packet came from Vlan25, hits the ASA default gateway on 10.10.25.1

ASA routes the packet but does not NAT, so the source is still 10.10.25.17 and dest is 192.168.100.17.

Return traffic, NAT occurs so that 192.168.100.17 is NAT'd into.. well, any, so I have no idea what that might end up as

Client potentially receives a packet from a different source IP and drops it not knowning whats going on?

By entering the IP ranges for each interface specifically (even though you're not actually NATing, as they're NAT'ed to themselves) it's performing two-way NAT, so regardless which end initiates it'll NAT 10range into 192range or vice versa and then the return traffic obviously is picked up by the active xlate entry, NAT'd back and arrives back at the client successfully.

Also NAT can override the routing table if an active NAT statement exists in the xlate table for your packet IP destination so might have caused this but without being able to test or see I cannot say for sure what statement was within the table if any.

I've probably not been hugely helpful but it's interesting though :)

Okay i do understadn everything you say and i already know what the no ip routing does but anywaywas thanks for telling, the thing is that in what line you are permitting all the vlans on the controller to be trunked?

That was the part i didnt see and thats why i find itconfusing...

As far i know trusting all the vlans does not mean im trunking them all... it just that im trusting them  but not doing 802.1q

Unless that on aruba controller doing just switch mode trunk will allow all the vlans?

okay thats the confusing part... hehe

in what line does it says you are allowing all the vlans? i lll like to test tho... :)