At the AP side, all appears to be fine (it works to a local VPNC, similarly configured). The customer has shown me their firewall's port-forwarding rules for external-to-internal IP translation for this controller (Ext-IP:udp4500 <-> Int-IP:udp4500) and it appears to be all correct and is what is received by the AP's "show ata endpoint" configuration. They see the UDP4500 traffic in the firewall from these microbranch devices, and I see what I believe is the tunnel trying to negotiate. I have the wired client (IP phone's) pcaps by inserting a wire pcap monitoring device and observing the traffic there, but this just shows problems when the tunnels go down.
In the "show datapath session" I see the traffic hitting the VPNC, and when established it can remain good for minutes or weeks. I've asked if they do any SSL Inspection on their firewall, which they say they do not. We've tried rebooting the VPNC, upgrading it, nothing seems to work. I suppose we could try with just a public IP exposed on the box with no firewall in between and see if that helps?
What would be the appropriate debug/investigative steps to look at why the IPsec tunnel is not forming? Is there a "controlpath" pcap I could obtain on the VPNC for the AP's ike/isakmp/IPsec negotiation?
I've had Aruba troubleshoot other VPNC issues that I would think would depend on this being solid, and they have always come up empty-handed and say my config "looks" good. Perhaps if I could find a smoking gun in a process failure or debug log, then that would help them.
Original Message:
Sent: Feb 13, 2024 04:57 PM
From: ariyap
Subject: VPNC Microbranch connection issues
has there been any packet captures on on the AP or VPNC or the firewall side? that would quickly narrow down the issue.
------------------------------
If my post was useful accept solution and/or give kudos.
Any opinions expressed here are solely my own and not necessarily that of HPE or Aruba.
Original Message:
Sent: Feb 13, 2024 04:46 PM
From: ryh
Subject: VPNC Microbranch connection issues
Also to note, this is a Central-managed 303H on version 10.5.0.1 (others have been tried as well), connecting to a VPNC on 10.4.1.0
-ryh
------------------------------
ryh
Original Message:
Sent: Feb 13, 2024 04:43 PM
From: ryh
Subject: VPNC Microbranch connection issues
Greetings, Wizards!
I have been battling an issue with microbranch-VPNC communications for a customer. What appeared at first to be a end-device issue (ip-phone on a wired port, CL2 to the DataCenter VPNC), now appears to be the microbranch connection to the VPNC itself. It seemed to be just one of the microbranch APs, but I have replicated the issue on my own (303H).
What I see in the logs is the tunnel not establishing, or trying to re-establish often. The wired client is put into a "deny all" role since the VPNC is not responding for the auth-request (show ap debug auth-trace-buf shows "server out-of-service"). The AP "show ata state" information reveals something like the following:
ub_303h# show ata state-transition-history
Tunnel State Transition History
-------------------------------
Timestamp Peer IP UUID Current State Event Next State
--------- ------- ---- ------------- ----- ----------
2024-02-13 20:53:21 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 INIT TUN_RECV CONNECTING
2024-02-13 20:53:27 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 CONNECTING TUN_RECV CONNECTING
2024-02-13 20:53:30 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 CONNECTING PROBE_OK CONNECTED
2024-02-13 20:53:56 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 CONNECTED TUN_REKEY REKEYING
2024-02-13 20:54:27 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 REKEYING HB_TIMEOUT CONNECTING
2024-02-13 20:54:28 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 CONNECTING HB_TIMEOUT INIT
2024-02-13 20:57:00 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 INIT TUN_RECV CONNECTING
2024-02-13 20:58:12 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 CONNECTING TUN_RECV CONNECTING
2024-02-13 20:59:55 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 CONNECTING TUN_RECV CONNECTING
2024-02-13 21:01:58 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 CONNECTING PROBE_TIMEOUT INIT
2024-02-13 21:06:58 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 INIT TUN_RECV_TIMEOUT SURVIVING
2024-02-13 21:07:58 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 SURVIVING TUN_RECV CONNECTING
2024-02-13 21:08:12 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 CONNECTING TUN_RECV CONNECTING
2024-02-13 21:08:14 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 CONNECTING TUN_RECV CONNECTING
2024-02-13 21:08:16 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 CONNECTING TUN_RECV CONNECTING
2024-02-13 21:09:32 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 CONNECTING TUN_RECV CONNECTING
2024-02-13 21:11:35 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 CONNECTING PROBE_TIMEOUT INIT
2024-02-13 21:11:35 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 INIT TUN_RECV_TIMEOUT SURVIVING
2024-02-13 21:11:52 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 SURVIVING TUN_RECV CONNECTING
2024-02-13 21:13:55 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 CONNECTING PROBE_TIMEOUT INIT
2024-02-13 21:18:56 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 INIT TUN_RECV_TIMEOUT SURVIVING
2024-02-13 21:20:01 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 SURVIVING SUR_TUN_ERR SURVIVING
2024-02-13 21:21:07 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 SURVIVING SUR_TUN_ERR SURVIVING
2024-02-13 21:22:14 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 SURVIVING SUR_TUN_ERR SURVIVING
2024-02-13 21:22:45 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 SURVIVING TUN_RECV CONNECTING
2024-02-13 21:24:45 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 CONNECTING PROBE_TIMEOUT INIT
2024-02-13 21:24:45 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 INIT TUN_RECV_TIMEOUT SURVIVING
2024-02-13 21:25:53 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 SURVIVING SUR_TUN_ERR SURVIVING
2024-02-13 21:27:02 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 SURVIVING SUR_TUN_ERR SURVIVING
2024-02-13 21:28:12 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 SURVIVING SUR_TUN_ERR SURVIVING
2024-02-13 21:29:23 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 SURVIVING SUR_TUN_ERR SURVIVING
2024-02-13 21:30:36 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 SURVIVING SUR_TUN_ERR SURVIVING
2024-02-13 21:31:49 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 SURVIVING SUR_TUN_ERR SURVIVING
2024-02-13 21:33:03 38.126.108.243 317de5bb-646c-4f30-adef-744b204dfe92 SURVIVING SUR_TUN_ERR SURVIVING
ub_303h#
However, when it is finally established it may stay good for a long (30-90 minutes) period of time, or it may break before the client device can complete DHCP. When the connection is up, pings reliably pass through the tunnel at a stable latency (~100ms) and no drops over 100 pings.
Some of the microbranch APs are up and stay up. Some do not.
In my lab (no internet) I am able to set up a microbranch-to-VPNC setup with no issues and it is completely stable. Aruba TAC has been no help in the many requests I've made to have them look at it.
Any suggestions? It appears to be an IKE-related issue, but could it be a routing issue or packet drop at the VPNC side?
------------------------------
ryh
------------------------------