Controllerless Networks

How do I debug IPsec tunnel down between controller 7210 (address 10.229.111.222 below)    and RAP (address 168.228.154.273) ? My tunnel went down couple of days ago, I captured IPSEC traffic on network, see below:

15:20:46.931398 IP 168.228.154.273.13340 > 10.229.111.222.ipsec-nat-t: UDP-encap: ESP(spi=0x8d187300,seq=0xd), length 132
15:20:46.931592 IP 10.229.111.222.ipsec-nat-t > 168.228.154.273.13340: UDP-encap: ESP(spi=0xbdd8f700,seq=0x3c), length 132

But on controller tunnel is down... please advise

(WifiCtr01w) #show crypto ipsec sa peer 168.228.154.273

% No active IPSEC SA for 168.228.154.273 

Do you see any mention of this RAP in your system log?

show log system

This is what I found in 'show log system':

Apr 30 18:41:04 :303022: <WARN> |AP Z-RAP-irek@192.168.0.86 nanny| Reboot Reason: AP rebooted Thu Apr 30 18:39:05 EDT 2015; Unable to set up IPSec tunnel after 85 tries
Apr 30 19:14:16 :303022: <WARN> |AP Z-RAP-irek@192.168.0.87 nanny| Reboot Reason: AP rebooted Thu Apr 30 19:12:16 EDT 2015; Unable to set up IPSec tunnel after 85 tries
Apr 30 19:47:27 :303022: <WARN> |AP Z-RAP-irek@192.168.0.88 nanny| Reboot Reason: AP rebooted Thu Apr 30 19:45:28 EDT 2015; Unable to set up IPSec tunnel after 85 tries
Apr 30 20:20:39 :303022: <WARN> |AP Z-RAP-irek@192.168.0.89 nanny| Reboot Reason: AP rebooted Thu Apr 30 20:18:40 EDT 2015; Unable to set up IPSec tunnel after 85 tries
Apr 30 20:53:51 :303022: <WARN> |AP Z-RAP-irek@192.168.0.90 nanny| Reboot Reason: AP rebooted Thu Apr 30 20:51:52 EDT 2015; Unable to set up IPSec tunnel after 85 tries
Apr 30 21:27:03 :303022: <WARN> |AP Z-RAP-irek@192.168.0.91 nanny| Reboot Reason: AP rebooted Thu Apr 30 21:25:04 EDT 2015; Unable to set up IPSec tunnel after 85 tries
Apr 30 22:00:15 :303022: <WARN> |AP Z-RAP-irek@192.168.0.92 nanny| Reboot Reason: AP rebooted Thu Apr 30 21:58:16 EDT 2015; Unable to set up IPSec tunnel after 85 tries
Apr 30 22:33:27 :303022: <WARN> |AP Z-RAP-irek@192.168.0.93 nanny| Reboot Reason: AP rebooted Thu Apr 30 22:31:27 EDT 2015; Unable to set up IPSec tunnel after 85 tries

I can see it tries all "RAP-pool" addresses ...not sure why IPSec tunnel not setup 

Do you see any output in the following:

show datapath session table | include 4500

show crypto isakmp sa

Here it is what I see when traffic is coming from RAP, then after that tunnel goes down.

(WifiCtr01w) #show crypto isakmp sa

ISAKMP SA Active Session Information
------------------------------------
Initiator IP Responder IP Flags Start Time Private IP
------------ ------------ ----- --------------- ----------
.... -
168.228.154.273 10.229.111.222 r-v2-c-R May 1 14:07:33 192.168.0.122

Flags: i = Initiator; r = Responder
 m = Main Mode; a = Agressive Mode v2 = IKEv2
 p = Pre-shared key; c = Certificate/RSA Signature; e = ECDSA Signature
 x = XAuth Enabled; y = Mode-Config Enabled; E = EAP Enabled
 3 = 3rd party AP; C = Campus AP; R = RAP; Ru = Custom Certificate RAP; I = IAP
 V = VIA; S = VIA over TCP
(WifiCtr01w) #show datapath session table | include 4500
10.229.111.222 168.228.154.273 17 4500 15474 0/0 0 0 2 pc0 16 0 0 F
168.228.154.273 10.229.111.222 17 15474 4500 0/0 0 0 0 pc0 17 0 0 FC
......

So it appears the RAP comes up; establishes its tunnel....then loses it?   Has anything changed within your infrastructure; perhaps firewall configuration changes?   Is this the only RAP having the issue, or is it the only RAP?    Can you try to configure a RAP to connect locally to the controller, to remove any firewall/Internet/etc. issues to narrow down the possible reason?