A Practical Guide to SSE Agent Troubleshooting

by matt.hum@hpe.com

One of the questions I get asked often is, "How do I debug or troubleshoot the HPE Aruba Networking SSE agent?" This can be especially relevant when a destination or application isn't pulling up or resolving. In this post, we'll explore some common issues and practical solutions to help you navigate these challenges more effectively. Let's dive in and make troubleshooting a bit easier.

A good first step in troubleshooting is checking to ensure the tunnel is up and functioning appropriately. If the agent shows you are connected, then we need to check DNS. All machines should have nslookup on it, so check the response of a DNS lookup. Here are 2 examples, one with a successful agent lookup (1), and one without (2).

Example 1

 Example 2

In example 1, the DNS request was sent to the nearest SSE PoP and resolved with a synthetic IP, thus redirecting the traffic through the tunnel.

In example 2, the request was sent to the locally configured DNS server (in this case Cloudflare’s public DNS server) and resolved to the public IP that someone else owns.

If you are using IP ranges with internal IPs instead of hostnames, check the end system’s routing table and see if the routes are correctly inserted and have a gateway of 100.65.0.1.

If everything is set up correctly, the traffic should be redirected to the PoP as expected. If issues persist, we need to investigate further. You can access all debugging tools from the Debug Tools section in the agent. If a password is required, you can set it in the management portal under Agent Settings.

In the latest version of the agent (3.82)^[1] the capability of a One Time Password (OTP) was introduced to give an agent specific one time unlock into the debug tools. This would allow an administrator or help desk technician to have an end user access the debug screens and gather some information. This can be accessed from the Enrolled Agents section of the management portal.

Once in the debug screens, the main debug screen displays the agent version and tunnel status. The “Start Tunnel” and “Stop Tunnel” options can be used here to manually disconnect and/or reconnect the agent’s tunnel to the PoP.  If there are issues with the agent establishing a connection, the Checking Connectivity option can be helpful.

The "Start Record" and "Stop Record" options provide a pcap file of all traffic traversing the tunnel. Personally, I prefer using Wireshark, as it offers more flexibility with pre and post capture filters. The "Export Logs" option is useful when there is an issue with the agent, allowing you to quickly export logs to your desktop and provide them to the support team. The "Dev Tools" can be ignored, as they are only related to the agent's UI.

Once you have finished debugging, the "End Debugging Session" option will exit the debug screens and return you to the primary agent window.

Of all the options here, we are more interested in the screens under "Advanced" and then "Diagnostic." Here, you should be able to see the ZTNA flow and expanding it will provide a lot of valuable information.

First, the Status will come in 4 different colors, Green, Yellow, Orange and Red.

Green means the flow went through appropriately.

Red indicates a block or other failed communication

Yellow indicates that the Agent to connector flow was ok, but the destination server within the environment did not respond, indicating an upstream issue.

If you see orange, capture a screenshot for support as orange indicates an unknown state. This is extremely rare but keep an eye out for this status.

The second column, “Created at,” shows us the GUID of the flow, the date and time, total bandwidth consumed, the general application latency, and the destination name that was matched in the policy.

The third column, “Advanced info” -> “Name” indicated the PoPs that were traversed.

In the case above, the agent and the connector were connected to the same PoP in Oregon, so there is no backend connector needed.

In the case above, the agent was connected to the PoP in Texas, while the connector was connected to the pop in Northern California. It is important to note that in the meshed infrastructure^[2], PoP selection is based on lowest Round Trip Time (RTT) per flow and is updated regularly. In tenants without the meshing feature, Global Server Load Balancing (GSLB) is used to identify the closest PoP and will not change until reconnect from a different location.

The fourth column shows the latency between hops.

The fifth through ninth column is the duration and traffic statistics of that matching flow, while the last column indicates “next hop.” Note that the next hops within PoPs are internal on an isolated namespace and are not accessible externally.

Next is the Agent ID screen. If you ever need to request support for this specific agent, be sure to include the Agent Id that you get from the Agent Id screen.

The Stats entry has a number of subpages, but you’ll probably only need a few. First the wireguard (wg) “tab” shows all the PoPs the agent is connected to, along with the relevant PoP details.

This can be used to identify why certain PoPs are being used and how traffic could be routed. As with the meshed infrastructure, the agent will connect to the 3 closest PoPs and then calculate the RTT every heartbeat. This is used to identify which PoP every flow should take as part of the overall decision process. Meshed PoPs will say mesh in the name and are different endpoints in the same PoP as the GSLB endpoints.

For tenants without meshing enabled, the Agent will use GSLB to determine the closest PoP and will connect to that.

The dns tab shows internally resolved DNS requests and can be used to troubleshoot specific flows and client-side DNS issues.

The dns method tab provides a DNS decision according to policy for each server.

Here “Intercept” indicates traffic that was redirected to the PoP for TLS interception. External is for traffic that is resolved by an external DNS server, such as the agent’s internal traffic and TLS exceptions. This completely bypasses the PoP and our SSE DNS services.

“CustomAnswer” is reserved for the Agent’s internal testing communications and is hardcoded into the config.

Finally, Bandwidth Limit shows what limits are placed on the traffic from that Agent to/from the PoP.

The last page of interest is the Policy page. These are the destinations as determined by the policy ruleset. Scroll down to the bottom of this page to view a few useful sections, “filteredOutApps” and “userApps.” FilteredOutApps lists all the destinations that the user/agent was assigned to but are not available for some reason. The most common reason is that the corresponding connectors are down or have communication issues. UserApps lists all the destinations that are applied to the agent for use.

After this, there is user information. This will include the groups that the user is a member of, the email, and other information obtained from the SAML assertion, and the version number of the agent.

All this should provide debug information on where to look next and where to continue troubleshooting.

Effectively debugging and troubleshooting the HPE Aruba Networking SSE agent is essential for maintaining business continuity and ensuring smooth operations. By understanding the common issues and applying the practical solutions discussed in this post, you can effectively address any challenges that arise. A systematic approach and the proper use of available tools are key to successful troubleshooting. Happy troubleshooting!