Wireless Access

last person joined: yesterday 

Access network design for branch, remote, outdoor and campus locations with Aruba access points, and mobility controllers.
Expand all | Collapse all

[RAP] Troubleshooting

  • 1.  [RAP] Troubleshooting

    Posted Oct 07, 2020 05:49 AM

    Hi,

    Recently I converted an AP at a remote location to (from IAP) RAP mode but it fails to setup a tunnel.

    It has ping to the controller, it succeeded in setting up the VPN and downloading the firmware from the controller (8.6.0.5-FIPS) but it fails to setup a tunnel after booting into RAP mode.

    What troubleshooting can I do? What am I missing?

    It fails to setup a tunnel even if I put it on the same L2/L3 segment (over a p2p link) so I suspect the issue is in the RAP <> controller interaction.

     

    The error the rapconsole shows is:

     IP [] using Ethernet Aborted: HELLO-TIMEOUT. Bringing tunnel down


  • 2.  RE: [RAP] Troubleshooting

    Posted Oct 07, 2020 06:13 AM

    Our problem sounds a bit similar to this issue:

    https://community.arubanetworks.com/t5/Wireless-Access/IPSEC-tunnel-flapping/td-p/69200

     

    The weird thing is that before switching to FIPS we were not having these issues, non of the underlying network design has changed only the firmware type.



  • 3.  RE: [RAP] Troubleshooting

    Posted Oct 07, 2020 07:10 AM

    Do you have the logs from the controller when the issue is occurring? If you check the datapath session do you see the IPSEC traffic?

     

     

    #show datapath session table | include 4500
    #show crypto isakmp sa
    #show crypto ipsec sa
    #show user ip x.x.x.x (ip address of rap user)

     

    *EDIT* - I might be asking the obvious, but you have created an RAP IP Pool and whitelisted the RAP accordingly?

     



  • 4.  RE: [RAP] Troubleshooting

    Posted Oct 07, 2020 09:46 AM

    At the moment I switched the AP to campus mode (by allowing our AP vlan to run over the p2p link to the other location, not an ideal situation) so that the people at the office can work.

     

    I'll try to run these commands a bit later.

     

    We have an RAP IP pool under VPN (in the PIPA range) and the RAP is whitelisted under Remote APs (as far as I understand it would not be able to switch from instant to remote if it was not whitelisted).



  • 5.  RE: [RAP] Troubleshooting

    Posted Oct 07, 2020 11:22 AM

     

    #show datapath session table | include 4500
    [ap-ip]        [controller-ip]      17   59744 4500   0/0     0    0   0   pc1         6a   103        44624      FC              6
    #show crypto isakmp sa
    Initiator IP   Responder IP   Flags     Start Time   Private IP   Peer ID
    ------------   ------------   -----     ----------   ----------   -------------
    [ap-ip]       [controller-ip] r-v2-c-R  Oct  7 18:06:16   169.254.1.23                            CN=[serial]::[mac]
    IPSEC SA (V2) Active Session Information
    -----------------------------------
    Initiator IP   Responder IP   SPI(IN/OUT)   Flags Start Time   Inner IP   Ipsec-map
    ------------   ------------   ----------------   ----- ---------------   --------   ---------
    [ap-ip]   [controller-ip]   [hex]/[hex]  UT2   Oct  7 18:10:12   169.254.1.24
    #show user ip [ap-ip]
    This operation can take a while depending on number of users. Please be patient ....
    
    
    Datapath Session Table Entries
    ------------------------------
    
    Flags: F - fast age, S - src NAT, N - dest NAT
           D - deny, R - redirect, Y - no syn
           H - high prio, P - set prio, T - set ToS
           C - client, M - mirror, V - VOIP
           Q - Real-Time Quality analysis
           u - Upstream Real-Time Quality analysis
           I - Deep inspect, U - Locally destined
           E - Media Deep Inspect, G - media signal
           r - Route Nexthop, h - High Value
           A - Application Firewall Inspect
           J - SDWAN Default Probe stats used as fallback
           B - Permanent, O - Openflow
           L - Log, o - Openflow config revision mismatched
    
    Source IP or MAC  Destination IP  Prot SPort DPort Cntr     Prio ToS Age Destination TAge Packets    Bytes      Flags           CPU ID
    ----------------- --------------- ---- ----- ----- -------- ---- --- --- ----------- ---- ---------- ---------- --------------- -------
    
    
    Name: , IP: [ap-ip], MAC: 00:00:00:00:00:00, Age: 00:00:14
    Role: logon (how: ROLE_DERIVATION_NONE), ACL: 2/0
    Authentication: No, status: not started, method: , protocol: , server:
    Role Derivation: ROLE_DERIVATION_NONE
    VLAN Derivation: Unknown
    Idle timeout (global): 300 seconds, Age: 00:00:00
    Mobility state: Wireless, HA: Yes, Proxy ARP: No, Roaming: No Tunnel ID: 0 L3 Mob: 0
    Flags: internal=1, trusted_ap=0, l3auth=0, mba=0, vpnflags=2, u_stm_ageout=0
    Flags: innerip=0, outerip=1, vpn_outer_ind:0, download=1, wispr=0
    IP User termcause: 0
    phy_type: b-, l3 reauth: 0, BW Contract: up:0 down:0, user-how: 4
    Vlan default: 0, Assigned: 0, Current: 0 vlan-how: 0 DP assigned vlan:0
    Mobility Messages: L2=0, Move=0, Inter=0, Intra=0, Flags=0x0
    SlotPort=0x0, Port=0x0 (vlan 0)
    Essid: , Bssid: 00:00:00:00:00:00 AP name/group: N/A/default Phy-type: b- Forward Mode: tunnel
    RadAcct sessionID:n/a
    RadAcct Traffic In 604/291680 Out 40/27045 (0:604/0:0:4:29536,0:40/0:0:0:27045)
    Timers: L3 reauth 0, mac reauth 0 (Reason: ), dot1x reauth 0 (Reason: )
    Profiles AAA:, dot1x:, mac: CP:n/a def-role:'logon' via-auth-profile:''
    ncfg flags udr 0, mac 0, dot1x 0, RADIUS interim accounting 0
    IP Born: 1602082930 (Wed Oct  7 18:02:10 2020)
    Core User Born: 1602082930 (Wed Oct  7 18:02:10 2020)
    Upstream AP ID: 0, Downstream AP ID: 0
    User Agent String:
    L3-Auth Session Timeout from RADIUS: 0
    Mac-Auth Session Timeout Value from RADIUS: 0
    Dot1x Session Timeout Value from RADIUS: 0
    Dot1x Session Term-Action Value from RADIUS: N/A
    CaptivePortal Login-Page URL from RADIUS: N/A
    Reauth-interval from role: 0
    Number of reauthentication attempts: mac reauth 0, dot1x reauth 0
    mac auth server: N/A, dot1x auth server: N/A
    Address is from DHCP: no
    ipuser_notify_action:NoAction/NoAction
    Per-user-log pointer 0x23e19bc (id 3722), num logs 0
    RTTS disabled: rtts_throughput 0 rtts_discard 0 rtts_reest 0 rtts_keepalive 0
    User added to cluster bucket-map: No
    
    The phy column shows client's operational capabilities for current association
    
    Flags: A: Active, B: Band Steerable, H: Hotspot(802.11u) client, K: 802.11K client, M: Mu beam formee, R: 802.11R client, W: WMM client, w: 802.11w client, V: 802.11v BSS trans capable, P: Punctured preamble, U: HE UL Mu-mimo, O: OWE client, S: SAE client, E: Enterprise client, m: Agile Multiband client, C: Cellular Data Capable - network available, c: Cellular Data Capable - network unavailable, p: Pending GSM activation, T: Individual TWT client, t: Broadcast TWT client
    
    PHY Details: HT   : High throughput;      20: 20MHz;  40: 40MHz; t: turbo-rates (256-QAM)
                 VHT  : Very High throughput; 80: 80MHz; 160: 160MHz; 80p80: 80MHz + 80MHz
                 HE   : High Efficiency;       80: 80MHz; 160: 160MHz; 80p80: 80MHz + 80MHz
                 <n>ss: <n> spatial streams
    
    Association Table
    -----------------
    Name  bssid  mac  auth  assoc  aid  l-int  essid  vlan-id  tunnel-id  phy  assoc. time  num assoc  Flags  Band steer moves (T/S)  phy_cap
    ----  -----  ---  ----  -----  ---  -----  -----  -------  ---------  ---  -----------  ---------  -----  ----------------------  -------

     



  • 6.  RE: [RAP] Troubleshooting

    Posted 12 days ago
    So far Aruba support has also not been able to resolve the issue, my guess is that this is a bug in the firmware.

    ------------------------------
    Keeper of the Keys
    ------------------------------



  • 7.  RE: [RAP] Troubleshooting

    Posted 11 days ago

    Do you see the AP in 'show ap active'? Or in 'show ap database'? With what flags?

    Can you connect your AP, or a similar AP in RAP mode somewhere close to the controller, so you can determine if this is dependent on the connection that you have to your RAP, or it is configuration or a possible bug?

    In many cases there is something in the path like firewalls, WAN accellerators, that can break the communication from RAP to the controller.



    ------------------------------
    Herman Robers
    ------------------------
    If you have urgent issues, always contact your Aruba partner, distributor or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC.
    ------------------------------



  • 8.  RE: [RAP] Troubleshooting

    Posted 10 days ago
    Hey Herman,

    To test this and absolutely rule out the possibility of firewall interference we took an AP on the same vlan as the controller and converted it to RAP mode with the same results as what was mentioned above.

    The test AP shows in the show ap database with flag "2" (Using IKE version 2) but not in show ap active since it is considered to be down.

    ------------------------------
    Keeper of the Keys
    ------------------------------



  • 9.  RE: [RAP] Troubleshooting

    Posted 10 days ago
    Have you requested escalation of your support ticket already?

    Do you see entries about the RAP entering the controller in the controller logs?
    Did you whitelist the RAP in the RAP whitelist, that is a different whitelist than the standard AP whitelist?

    It seems like the AP is rejected (not whitelisted for RAP), or there is no route back as the datapath session should contain two entries:
    (md7010) #show datapath session | include 4500
    192.168.31.41     192.168.31.15   17   4500  4500   0/0     0    40  0   0/0/14      caa6 3660561    1371287904 FC              6
    192.168.31.43     192.168.31.15   17   4500  4500   0/0     0    0   0   0/0/14      11   16         9680       FC              7
    192.168.31.15     192.168.31.43   17   4500  4500   0/0     0    0   1   0/0/14      11   0          0          FY              7
    192.168.31.15     192.168.31.41   17   4500  4500   0/0     0    0   0   0/0/14      caa6 9395       388905     F               6​

    In this example, the .15 is the controller, the 41 and 43 are APs, and you see a connection in both directions.

    Controller logs should contain the information for TAC Support to find a solution.

    ------------------------------
    Herman Robers
    ------------------------
    If you have urgent issues, always contact your Aruba partner, distributor or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC.
    ------------------------------



  • 10.  RE: [RAP] Troubleshooting

    Posted 10 days ago
    > Have you requested escalation of your support ticket already?

    I have not requested an escalation yet, I kind of assumed that the support guys would do that themselves once they realized this was too much for them (they said they were having issues with lab accessibility due to covid and I had a vacation in the middle so it kind of sat for a while).

    > Do you see entries about the RAP entering the controller in the controller logs?

    Not sure what you mean

    > Did you whitelist the RAP in the RAP whitelist, that is a different whitelist than the standard AP whitelist?

    Yup

    This is the datapath for the RAP:
    #show datapath session | include [ap-ip]
    [controller]         [ap-ip]     17   4500  65034  0/0     0    0   1   pc1         12   0          0          FY              7
    [ap-ip]       [controller]       17   65034 4500   0/0     0    0   0   pc1         12   5          4832       FC              7​


    ------------------------------
    Keeper of the Keys
    ------------------------------



  • 11.  RE: [RAP] Troubleshooting

    Posted 9 days ago
    About the escalation, you can always ask for escalation if you feel there is not enough progress to the case compared to your priority and urgency. TAC engineers are in general very dedicated and like to help you with all they can, but in some cases having someone else look at it speeds up the resolution. You can also always ask to see if there is a different method to find the issue than building it in the lab if that is not possible at the moment.

    On the controller logs, if you have syslog configured to an external log server or after you try to join the RAP to the controller and run 'show log all 10000' (for the last 10000 lines of log) there is all kind of information on what is happening, how the RAP is authenticated (or rejected) and other related information. My experience is that in most cases that logs at least provide an indication of where to look.

    ------------------------------
    Herman Robers
    ------------------------
    If you have urgent issues, always contact your Aruba partner, distributor or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC.
    ------------------------------