Problem:
Use Case:
------------
Customer has monitoring tool which pings all the network IP address to verify the device is up.
Issue
-------
Unable to ping the VRRP IP of MM from the client connected to MD
Description
---------------
- Client when tries to ping VRRP-IP of MM, the traffic goes via IPSEC to MM instead of default gateway of client
- if the client tries to ping physical IP of MM then the traffic goes via default gateway of the client
Diagnostics:Topology
------------
- Topology: MM Active, MM Standby, MD
- 2 MM and MD are on VLAN 156
- Client is on VLAN 137 with default gateway on the MD’s uplink switch
- Masterip on MD is set to VRRP-IP of MM
- Client IP: 10.30.137.247
- MM Active IP: 10.30.156.221
- MM VRRP-IP: 10.30.156.232
Datapath of MM when pining controller-ip of MM >>>>>>>> we can see the packet reaches the MM through uplink port 0/0/0
==========================================
Source IP Destination IP Prot SPort DPort Cntr Prio ToS Age Destination TAge Packets Bytes Flags
--------------- --------------- ---- ----- ----- -------- ---- --- --- ----------- ---- --------- --------- ---------------
10.30.137.247 10.30.156.221 1 7914 2048 1/4108 0 0 0 0/0/0 0 1 60 FCI
10.30.137.247 10.30.156.221 1 7913 2048 1/4108 0 0 0 0/0/0 1 1 60 FCI
10.30.137.247 10.30.156.221 1 7912 2048 1/4108 0 0 0 0/0/0 3 1 60 FCI
10.30.137.247 10.30.156.221 1 7911 2048 1/4108 0 0 1 0/0/0 4 1 60 FCI
10.30.137.247 10.30.156.221 1 7910 2048 1/4108 0 0 1 0/0/0 5 1 60 FCI
Datapath of MD when pinging Controller-IP of MM
===========================================
(MD-125) #show datapath session table 10.30.137.247 | include 10.30.156.221
10.30.156.221 10.30.137.247 1 7874 0 0/0 0 0 1 tunnel 14 c 1 60 FI >>>>>>>>>>> tunnel 14 is GRE tunnel from AP on which client connected
10.30.137.247 10.30.156.221 1 7878 2048 0/0 0 0 1 tunnel 14 8 1 60 FCI
10.30.137.247 10.30.156.221 1 7871 2048 0/0 0 0 1 tunnel 14 f 1 60 FCI
10.30.156.221 10.30.137.247 1 7870 0 0/0 0 0 1 tunnel 14 10 1 60 FI
10.30.137.247 10.30.156.221 1 7875 2048 0/0 0 0 1 tunnel 14 b 1 60 FCI
10.30.156.221 10.30.137.247 1 7879 0 0/0 0 0 1 tunnel 14 7 1 60 FI
10.30.156.221 10.30.137.247 1 7885 0 0/0 0 0 0 tunnel 14 0 1 60 FI
10.30.137.247 10.30.156.221 1 7881 2048 0/0 0 0 0 tunnel 14 5 1 60 FCI
Datapath of MM when pinging VRRP IP of MM (we can see the traffic is coming through the IPSCEC tunnel)
=======================================
(DEE-MM-221) *[mm] # show datapath session table 10.30.137.247
Source IP Destination IP Prot SPort DPort Cntr Prio ToS Age Destination TAge Packets Bytes Flags
--------------- --------------- ---- ----- ----- -------- ---- --- --- ----------- ---- --------- --------- ---------------
10.30.137.247 10.30.156.221 1 7092 2048 1/4108 0 0 0 tunnel 13 0 1 60 FCI
10.30.137.247 10.30.156.221 1 7091 2048 1/4108 0 0 0 tunnel 13 1 1 60 FCI
10.30.137.247 10.30.156.221 1 7090 2048 1/4108 0 0 1 tunnel 13 2 1 60 FCI
10.30.137.247 10.30.156.221 1 7089 2048 1/4108 0 0 1 tunnel 13 3 1 60 FCI
10.30.137.247 10.30.156.221 1 7088 2048 1/4108 0 0 1 tunnel 13 4 1 60 FCI
(DEE-MM-221) *[mm] #show datapath tunnel table | include 13
13 SPI8AB68C00 in 10.30.156.221 50 IPSE 1500 0 routeDest 009C 0 2138 0 T 0 0
Datapath of MD when pinging VRRP IP of MM
======================================
10.30.137.247 10.30.156.232 1 7272 2048 0/0 0 0 1 tunnel 14 10 1 60 FNCI >>> we can see the packet is getting Dest-Natd
10.30.137.247 10.30.156.232 1 7287 2048 0/0 0 0 0 tunnel 14 1 1 60 FNCI
10.30.137.247 10.30.156.232 1 7282 2048 0/0 0 0 0 tunnel 14 6 1 60 FNCI
Root Cause
--------------
- it is an inbuilt behavior of the controller to hijack all the traffic destined to the Master-local IPSEC tunnel IP (here vrrp ip of MM) using predefined ACLs
- hence when we ping the VRRP IP from the client on MD, the controller hijack the traffic and pass it through the tunnel
- when the MM response to the ICMP, since the client is in a different VLAN, the MM will send the packet to it’s default gatway
- since MM’s default gateway is not aware of the client VLAN the ICMP response get dropped on the network.
This behavior is same in Master-local setup (6.X code)
SolutionSolution 1
--------------
Make sure the MM default gateway has the route to the client network
Stateful processing of ICMP should be disabled on firewalls since it will drop the ICMP packet due to asymmetric routing.
Solution 2
-------------
Add a static route on the Master controller to the client network with next hop as the master-local ipsec MAP
On MM/Master
#config t
#ip route <client network> <subnet mask> default-local-master-ipsecmap10.30.156.188 >>>>>>>>> this is the default IPSEC mac to the MD
Note: solution 2 will work only on the 6.X code and in 8.X with clustering it is not possible.