Wireless Access

We are seeing a strange behavior on a pair of Aruba controllers. We found it after we introduced an EAST/WEST firewall between some VLANs in our environment. We have AirWave in VLAN129 it and sends its request to the controllers' addresses in VLAN 254. We noticed UDP traffic between 254 and 129 was fine but TCP and ICMP traffic was not.

The controllers have their ip-default-gateway in VLAN254 and no other routes. The controllers also have IP addresses on some other VLAN Interfaces. VLAN129 (where the AirWave server lives) is one of the interfaces. In our troubleshooting efforts we noticed that both controllers try to respond to TCP traffic from the AirWave server using their IP addresses in VLAN 129 instead of responding with their VLAN254 Interfaces.

The FortiNet firewall is blocking those conversations because it is set up to ‘block’ asymmetrical routes. When we allow asymmetrical routing the TCP/ICMP traffic works but the TRACEROUTES revealed that the packets were using VLAN129 to get to AirWave even when I forced the  TRACEROUTE to use the controller’s VLAN254 interface as the source.

To ‘fix’ this we added a route to the controller that would force traffic from both controllers to use VLAN254’s gateway for the AirWave host IP in VLAN129. We removed asymmetric routing support and we tested. It worked!

Is this bug or expected behavior? (Questions about the 'design' notwitstanding of course. I also could have fixed it by having AirWave poll the controller's IPs in VLAN129) Why wouldn't the controller use the same interface in a TCP conversation.

Thanks!

What firmware version is running on your controller?

What are the IP addresses for the controllers' VLAN 129 and VLAN 254 interfaces?

From the CLI, what is the output you get from running: 

show controller-up

Version 6.5.3.4

controller IP is the loopback

loopback IP is same subnet as VLAN 254

controllers also have IP in VLAN 129

Airwave is in VLAN 129

TCP packets between AirWave to loopback and/or interface IP in 254 exhibits same behavior - asymmetric route

---

@bhubert wrote:

Version 6.5.3.4

controller IP is the loopback

loopback IP is same subnet as VLAN 254

controllers also have IP in VLAN 129

Airwave is in VLAN 129

TCP packets between AirWave to loopback and/or interface IP in 254 exhibits same behavior - asymmetric route

 

To ‘fix’ this we added a route to the controller that would force traffic from both controllers to use VLAN254’s gateway for the AirWave host IP in VLAN129. We removed asymmetric routing support and we tested. It worked!

---

This behavior is correct, and here's why...

The controller has more than just the default route in the routing table. If you run "show ip route" from the CLI, you'll see the default route of last resort, in addition to any connected interfaces. The only interface that will not show a connected route is the loopback interface, since it is assumed to have a /32.

When traffic arrives to the controller's mgmt interface (the loopback interface), the controller will consult the routing table for determining which interface the response should go out of. As Mark mentioned, connected interfaces have a higher preference than static routes, in addition to being more specific. In this case, the connected interface on VLAN 129 is the shortest path, so it gets used.

Your fix works because it provides a more specific route. Rather than a subnet route pointing out VLAN 129, you provided a host specific route (/32) out the other VLAN. 

I might have to think through this more, but since the Controller has an IP interface in VLAN129, that means it creates a Connected entry in the routing table. The default gateway is a static route, meaning the Connected route takes precedence. Maybe someone can validate my thinking here, but did you try shutting down the VLAN interface for 129 and just leave the VLAN 129.

Also TCP is connection-based protocol, so it requires a response when sending traffic, same with ICMP. UDP is connection-less, so it just sends the data without expecting return traffic.

Don't know if I'm barking up the right tree or not, but just a thought.

---

@mharing wrote:

I might have to think through this more, but since the Controller has an IP interface in VLAN129, that means it creates a Connected entry in the routing table. The default gateway is a static route, meaning the Connected route takes precedence. Maybe someone can validate my thinking here, but did you try shutting down the VLAN interface for 129 and just leave the VLAN 129.

---

This is my thinking too. In addition, since most people do not configure the controller-ip manually, the controller determines the IP to use for management on it's own. I suspect the controller has selected VLAN 129 for it's controller-ip, rather than the interface on VLAN 254 as the OP is expecting.

I'm certain that would work but (long story short) I cannot remove that interface on the production controllers.

I'm just wondering why a TCP/ICMP connection would fail to the all the hosts in VLAN129. I couldn't ping the gateway or AirWave from the controller using VLAN254s as source. Only when we enabled asymm on the E/W firewall did it allow traffic.

Without that route statement, if I ping a host w/ source of VLAN 254 it fails.

With that statement, pings work from VLAN 254 interface.