Wireless Access

Hi,

I have a strange issue between two sites.  The current setup I have works across multiple sites but one is causing an issue.

Mobility Master is at head office

Managed device on another site

VPN configured to allow connection between both

Added relevant firewall rules

Managed devices added as normal to Mobility Master

Everything comes up and syncs the config

The connection between the two sites drops intermittently for a few seconds every day this is currently being investigated.  However once this happens the MM loses connection to the MD (they show as down on the MM) and they never re-establish unless I do a full reboot of both the primary and secondary controllers on the remote site.  I have no idea why this would work until a drop in connectivity and why the connection doesn't re-establish between MM and MD like it has done on other sites when a drop between sites happens.

Thanks

This is showing in the security logs:

Mar 25 15:41:16 :103103:  <3440> <WARN> |ike|   IKE SA Deletion: IKE2_delSa peer:172.20.xx.xx:4500 id:2375889169 errcode:ERR_IKESA_EXPIRED saflags:0x41000005 arflags:0x20
Mar 25 15:41:58 :103103:  <3440> <WARN> |ike|   IKE SA Deletion: IKE2_delSa peer:172.20.xx.xx:4500 id:2375889170 errcode:ERR_IKESA_EXPIRED saflags:0x41000005 arflags:0x20

Also showing this when searching logs for cfmg:

Mar 25 14:24:28  cfgm[3375]: <399816> <3375> <ERRS> |cfgm|  handle_read: State(READY:UPDATE SUCCESSFUL:CFGID-869:PEND-0:INITCFGID:0) FD=33:Failure receiving heartbeat response header information Result=-1 Err=Connection timed out
Mar 25 14:24:37  cfgm[3375]: <399838> <3375> <WARN> |cfgm|  LmsHeartBeatResultAction: State(CONNECTINPROGRESS:UPDATE SUCCESSFUL:CFGID-869:PEND-0:INITCFGID:0) FD=33:Cannot heartbeat with the master.
Mar 25 14:24:58  cfgm[3375]: <399838> <3375> <WARN> |cfgm|  LmsHeartBeatResultAction: State(READY:UPDATE SUCCESSFUL:CFGID-869:PEND-0:INITCFGID:0) FD=33:Cannot heartbeat with the master.

What firmware version are you running? Also, does the VPN connection between the two sites NAT address space between the sites?

Thanks for your reply.

Firmware version is 8.4.0.3

No there are no NATs involved

Has this ever worked before? It sounds like an MTU issue with the MM/MD IPSec tunnel being routed down your local site-to-site VPN tunnel

Yes most definitely in fact it works every time I reboot the two managed devices they come back online - then when the ISP drops out at the remote site temporarily the MM sees the two MDs as down and they never come back up - yet I can get to them through the browser window and they are still up on site, they just lose connection to the MM and the only way to resolve it is to reboot them both - not ideal. 

I have opened the firewall on both sides to allow traffic for testing purposes from the IP of the MMs and the IP of the MDs.

However when even trying to ping the MDs from the MMs the firewall doesn't see it - i guess it looks like it is trying to go down the tunnel that was created between the MM and MD but doesn't get a response so it fails?

Right.  If that tunnel doesn't exist, pings do not work because there is a route setup to reach the MD that goes through the tunnel via the ipsec map.

Correct - that is what i was saying by the above post - i understand why it does not work however what i do not understand is why, like other sites, the connection between the MM and the MD does not re-establish when the VPN between the two sites is back?

Do the other sites have a firewall?  Is there NAT involved? (I think this was asked before).

I would debug the ipsec connection on the MM side using the instructions here : EDIT https://community.arubanetworks.com/t5/Aruba-Solution-Exchange/Troubleshooting-IPsec/ta-p/282677 

https://community.arubanetworks.com/t5/Controller-Based-WLANs/Understanding-and-Troubleshooting-IPSec-issues/ta-p/240527 (the debug syntax is slightly different  in 8.x).  That would show you the attempt the MDs is making to connect.  I would also do "show datapath session table <ip address of md>" repeatedly on the MM side to see what traffic is being sent to the MM from the MD.

Will do the debug stuff now and get back to you.

Yeah all sites have the same setup.

No there is no NAT involved.

Thanks

I would honestly work with TAC in parallel so that you get specific help with this.  It is incredibly painful and slow to do this on a user forum.

Yeah I have a ticket raised just waiting on call back and I like to look at these things myself to understand it better.

Commands on Aruba OS 8 to enable debugging:

logging security process lt2p level debugging
logging security process l2tp level debugging
logging security process crypto level debugging
logging security process authmgr level debugging
logging security process localdb level debugging

Results from MM datapath session table to MD:

Source IP or MAC  Destination IP  Prot SPort DPort Cntr     Prio ToS Age Destination TAge Packets    Bytes      Flags           CPU ID  
----------------- --------------- ---- ----- ----- -------- ---- --- --- ----------- ---- ---------- ---------- --------------- ------- 
192.168.xx.xx      172.20.xx.xx    17   8211  8421   1/4125  0    0   1   0/0/0       19   0          0          FYCI            5        
192.168.xx.xx      172.20.xx.xx    17   8211  8444   1/4123  0    0   0   0/0/0       6    0          0          FYCI            5        
192.168.xx.xx      172.20.xx.xx    17   8211  8344   1/4126  0    0   1   0/0/0       d    0          0          FYCI            5        
172.20.xx.xx      192.168.xx.xx    17   8222  8211   0/0     0    0   1   0/0/0       b    0          0          FYI             5        

172.20.xx.xx      192.168.xx.xx    17   8211  8211   0/0     0    0   55  0/0/0       13e7 0          0          FYI             5        
172.20.xx.xx      192.168.xx.xx    17   8494  8211   0/0     0    0   1   0/0/0       d    0          0          FYI             5        
172.20.xx.xx      192.168.xx.xx    17   8224  8211   0/0     0    0   1   local       5d   64         28032      FCI             4        
192.168.xx.xx      172.20.xx.xx    17   8211  8224   0/0     0    0   5   local       5d   0          0          FYI             4        

172.20.xx.xx      192.168.xx.xx    17   8344  8211   0/0     0    0   1   0/0/0       d    0          0          FYI             5        
192.168.xx.xx      172.20.xx.xx    17   8211  8494   1/4124  0    0   0   0/0/0       d    0          0          FYCI            5        
172.20.xx.xx      192.168.xx.xx    17   8421  8211   0/0     0    0   1   0/0/0       19   0          0          FYI             5        
192.168.xx.xx      172.20.xx.xx    17   8211  8211   0/0     0    0   0   0/0/0       13e7 17820      12162461   FCI             5        

192.168.xx.xx      172.20.xx.xx    17   8211  8222   1/4126  0    0   1   0/0/0       b    0          0          FYCI            5        
172.20.xx.xx      192.168.xx.xx    17   8444  8211   0/0     0    0   0   0/0/0       6    0          0          FYI             5  

Nothing show in logs so far on MM

Have the same issue running 8.5.0.2

Do you have a technical support case open?

I am asking, because it could be a firewall issue.

I'll open a ticket, but it doesn't look like a firewall. Since I can't hit the MM from MD, tried to ping, but it failed. At the same time I can ping MM just fine from the aggregation box the MD is attached to using the source from the MD's subnet. So it looks like the MD has some issues with datapath programming.

If an MD successfully established connectivity with the MM, there is a permanent route between them via the ipsec connection, so if the ipsec connection is down, a ping should not succeed.  Other devices will be able to ping the MD and MM because they don't have connectivity that depends on an ipsec connection.

You might want to start by troubleshooting why the ipsec connection cannot be established or maintained.  TAC  (technical support) can certainly assist with that.

Ok, I see what you are saying. And indeed you are correct.

I've had a thorough look into firewall after your remark and indeed found an issue at our edge Cisco ASA box. Wrong connection state after tunnel flap prevented a proper IKE signaling between MM and MD.

We don't have NAT between MM and MD sites, but for some reason MD use NAT-T to build tunnel, do you know if that's a feature or something?

Thank you

Yes.

The ports needed for connectivity between devices is here:  https://www.arubanetworks.com/techdocs/ArubaOS_86_Web_Help/Content/arubaos-solutions/external-firewallconf/fire-port-conf-arub.htm?Highlight=ports

The solution for this particular issue if you are using Cisco ASA with route-based S2S IPSec is to enable floating connection teardown, for example to teardown the connection after 30s, if more specific route has been added to the routing table:

timeout floating-conn 0:00:30  

Did that fix your specific issue?

Yes, it did.

Thank you

Hi @dmitry.skotnikov - firstly I am so happy someone else has an insight into this issue it has been driving me crazy

It looks like the default of this setting on Cisco ASAs (Which is what we use and have the same issue) is 0:00:00

Why does it need to be set to 00:00:30 to fix this?  I'm not sure I understand why this would make a difference and i want to understand

Ok, let me try to explain.

What I've found on the firewall after the IPSec has been restored between MM and MD sites was the state for a connection UDP MD:4500 -> MM:4500 via outside interface of ASA. It was weird, because I had a more specific route towards the MM subnet via the IPSec VTI tunnel interface, meaning the connection from MD towards MM supposed to be made via tunnel interface instead of the egress outside interface.

ASA has an old feature which prevents the box from clearing the connection even after we installed a more specific route into the routing table. In other words, when the IPSec tunnel restored and routing changed, we still had a connection via the outside interface, which was established when MD was trying to re-connect to MM at the time of IPSec outage, and since at that time we had only a default route via outside, the state has been programmed by the firewall. And since MM/MD uses NAT-T to build the tunnel, the firewall will see a UDP flow, which will never expire, since MD is constantly trying to re-establish the tunnel.  

The configuration knob timeout floating-conn 0:00:00, which is a default for ASA pretty much says never clear the connection I've described above.

Changing to timeout floating-conn 0:00:30, will make sure the connection will be cleared after 30s when the IPSec is restored and we have a better route towards MM.

If the description above is vague, please, check this out to get another take on it:

https://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-generation-firewalls/113592-udp-traffic-fails-00.html

Yes i would just like to say to you - thank you very much

This VTI setup you have is what we have and this is when the major problems happened.

Also your advice led me down the correct path as I was able to do the command on the ASA:

show conn address (ip off mobility master)

This returned two idle UDP connections :

Once I used this command:

clear conn protocol udp address (ip address of the MM)

Straight away the MM to MD connection came back.  So i believe with the timeout command this will fix the problem.

Thank you again.

Yes, you are right clearing the connection manually is the way to restore connectivity immediately, you need to do it for each MC in your cluster if you run more than 1 box.

Awesome, glad it helps!

I realize this is an older post, but I'm curious if you are aware of a similar setting/feature on a Fortigate Firewall?

Thanks!

Dmitry this is a fantastic find - thank you this has solved my issue on all sites! Thank you again