Wireless Access

Hi Airheads,

 

I am standing up a PoC for a client and am having trouble getting the RAP to connect. Environment is below:

 

Hardware MM

2 x 7210 MC, clustered

1 x 505 CAP

1 x 305 CAP

1 x 305 that i flick between CAP and RAP to test

Software is 8.6.0.3

 

MC's are behind a firewall, bi directional nat'd. Cluster is formed and each MC knows its public IP address:

 

lc-cluster group-profile "MC-CLUSTER"
controller 10.11.101.12 priority 128 mcast-vlan 0 vrrp-ip 10.11.101.112 vrrp-vlan 101 group 1 rap-public-ip x.x.x.x
controller 10.11.101.13 priority 128 mcast-vlan 0 vrrp-ip 10.11.101.113 vrrp-vlan 101 group 1 rap-public-ip x.x.x.y
active-client-rebalance-threshold 50
standby-client-rebalance-threshold 75
heartbeat-threshold 0

 

RAP Cluster pool is defined at the MM level:

(MM-01) [mynode] #show running-config | include lc-rap
lc-rap-pool rap-pool 172.17.0.1 172.17.2.254

 

(MM-01) [mynode] #show lc-rap-pool

IP addresses used in pool rap-pool
172.17.0.1

Total:-
1 IPs used - 765 IPs free - 766 IPs configured
LC RAP Pool Total Allocs/Deallocs/Reserves : 5/1/0
LC RAP Pool Allocs/Deallocs/Reserves(succ/fail) : 3/1/(0/0)

 

 

When provisioned as a cap all works fine, SSIDs broadcast, can connect. When reprovisioning as a rap pointed at either the public IP or private ip i can see errors on the controller such as:

 

Peer:AUTH_HMAC_SHA1_96 Peer:ESN_0 <-- R Notify: INTERNAL_ADDRESS_FAILURE

 

Doing some searching i came across this link which had the same error: https://www.booches.nl/2019/05/migrate-rap-from-aos-6-x-to-aos-8-x/

 

Cluster pool is definitely set and looks to be allocating an IP by the looks of it. I cant seem to find the rap pool command in the config making its way from the MM to the MC's (maybe by design???).

 

AP is showing as whitelisted on the MM but not on the MC's

 

I am working with a local SE on the issue but hoping others have had something similar and have a solution.

 

Cheers

 

In your cluster configuration, have you defined a RAP public IP for each member? See attached screenshot. When terminating RAPs to a cluster you should define this IP.

Yep, public IP's are defined (see original post config snippet).

 

All documentation 8.4 onwards also says to define the RAP pool at the MM level under "Cluster RAP Pool" settings so that is what has been done. SE also confirmed this to be accurate.

 

Cheers

That never worked for me. MD is where I got it to work.

 

 

 

Also your RAP pool should be configured under your Managed Network level, not the MM Level.

See if your RAP mac is whitelisted on the MD:

Show crypto isakmp clusterMac

 

See if your RAP was assigned an ip address from the pool on the MD:

Show crypto isakmp clusterIP

 

 

 

show crypto isakmp clusterMAC

Cluster RAPMAC Table Entries:

Total RAPMAC Entries: 0

 

show crypto isakmp clusterIP

Cluster RAPIP Table Entries:

Total RAPIP Entries: 0

 

Same on both controllers

 

on the MM show whitelist-db rap shows:

AP-entry Details
----------------
Name AP-Group AP-Name Full-Name Authen-Username Revoke-Text AP_Authenticated Description Date-Added Enabled Remote-IP Remote-IPv6 Cluster-InnerIP Cert-type
---- -------- ------- --------- --------------- ----------- ---------------- ----------- ---------- ------- --------- ----------- --------------- ---------
xxxxxxxxxxxx RAP-TEST xxxx-305 Provisioned Thu Apr 2 11:43:39 2020 Yes 0.0.0.0 :: 172.17.0.1 NA

When you go to configuration> access points> whitelist> Remote AP whitelist do you see your whitelisted RAPs at the highest /MD level?  What about if you navigate down to the MD level?

---

@brettbrown wrote:

Cluster pool is definitely set and looks to be allocating an IP by the looks of it. I cant seem to find the rap pool command in the config making its way from the MM to the MC's (maybe by design???).

---

 

With cluster RAP, the pool itself does not get pushed to MDs, as opposed to the way things work when not using cluster with RAPs. The pool is used (by the MM) to assign an IP (from the pool) to the RAP when it's put into the whitelist database; which essentially becomes a static IP. That IP is then consistent across the MDs within the cluster that it terminates in (indeed across all clusters of the MM).

 

When the RAP arrives in the MD, it will use the auth-server "internal" to try to authenticate, in a cluster this should be pointing to the MM virtual IP, please confirm it using "show aaa authentication-server internal", it should show the IP of the MM VIP (or MM primary if you don't have a backup MM). 

 

If someone has previously set the MD internal auth server to "use-local-switch" then this could easily cause the problem you're seeing.

 

Here is how the security debug should look (warning: don't run log level debugging security in a live RAP network for any extended amount of time). In the below debug, a RAP (9c:8c:d8:09:05:0c) connects to MD (192.168.1.144) from NAT source IP 172.35.0.10, via a RAP cluster public IP (does not appear in debug).

The RAP whitelist entry is not found on the MD so the MD will query the MM VIP (192.168.1.142) and finds a result which has internal IP

168430082 (0xa0a0a02 == 10.10.10.2).

 

 

Apr 5 15:40:30 :124004:  <3368> <DBUG> |authmgr|  RX (sock) message of type 66, len 1028 
Apr 5 15:40:30 :124454:  <3368> <DBUG> |authmgr|  auth_user_query_raw: recvd request user:9c:8c:d8:09:50:0c ip:172.35.0.10 cookie:-2145524411 
Apr 5 15:40:30 :124098:  <3368> <DBUG> |authmgr|  Setting authstate 'started' for user 172.35.0.10, client VPN.
Apr 5 15:40:30 :124099:  <3368> <DBUG> |authmgr|  Setting auth type 'VPN' for user 172.35.0.10, client VPN.
Apr 5 15:40:30 :124004:  <3368> <DBUG> |authmgr|  ncfg_auth_server_group_authtype ip=172.35.0.10, method=VPN vpnflags:2
Apr 5 15:40:30 :124004:  <3368> <DBUG> |authmgr|  ncfg_auth_server_group_authtype vpnflags:2 vpn-profile:default-rap
Apr 5 15:40:30 :124004:  <3368> <DBUG> |authmgr|  ip=172.35.0.10, sg=default
Apr 5 15:40:30 :124004:  <3368> <DBUG> |authmgr|  aal_authenticate aal 0x2b9cc94 username 9c:8c:d8:09:05:0c
Apr 5 15:40:30 :124547:  <3368> <DBUG> |authmgr|  aal_authenticate server_group:default.
Apr 5 15:40:30 :124004:  <3368> <DBUG> |authmgr|  Select server for method=VPN, user=9c:8c:d8:09:05:0c, essid=<>, server-group=default, last_srv <>
Apr 5 15:40:30 :124038:  <3368> <INFO> |authmgr|  Reused server Internal for method=VPN; user=9c:8c:d8:09:05:0c,  essid=<>, domain=<>, server-group=default
Apr 5 15:40:30 :133028:  <3426> <DBUG> |localdb|  executeUSERDBMethod(127.0.0.1:8214 ==> 127.0.0.1:8344 PktType:0x402 SeqNum:37117 MsgCode:62): Received udb_msg with msgtype:62 id:178 reqtype:6 dbtype:13
Apr 5 15:40:30 :133108:  <3426> <DBUG> |localdb|  executeUSERDBMethod: Query for mac:9c:8c:d8:09:05:0c not successful locally with msgtype:62 id:178 reqtype:6 dbtype:13
Apr 5 15:40:30 :133032:  <3426> <DBUG> |localdb|  localdb_send_db_fetch_req: Sending Fetch-Req on WL-entry for mac 9c:8c:d8:09:05:0c to 192.168.1.142:8344 with msgtype:62 id:178 reqtype:9 dbtype:13
Apr 5 15:40:30 :133028:  <3426> <DBUG> |localdb|  executeUSERDBMethod(192.168.1.142:8344 ==> 192.168.1.144:8344 PktType:0x2002 SeqNum:8267 MsgCode:62): Received udb_msg with msgtype:79 id:178 reqtype:10 dbtype:13
Apr 5 15:40:30 :133108:  <3426> <DBUG> |localdb|  executeUSERDBMethod: Received FETCH-RSP for mac:9c:8c:d8:09:05:0c with msgtype:79 id:178 reqtype:10 dbtype:13
Apr 5 15:40:30 :133005:  <3426> <INFO> |localdb|  User 9c:8c:d8:09:05:0c  Successfully Authenticated
Apr 5 15:40:30 :124004:  <3368> <DBUG> |authmgr|  udb_gen_whitelist_avpairs: Added avpair name Remote-IP value 0 
Apr 5 15:40:30 :124004:  <3368> <DBUG> |authmgr|  udb_gen_whitelist_avpairs: Added avpair name Remote-IPv6 value :: 
Apr 5 15:40:30 :124004:  <3368> <DBUG> |authmgr|  udb_gen_whitelist_avpairs: Added avpair name Inner-IP value 168430082  (0xa0a0a02 == 10.10.10.2)
Apr 5 15:40:30 :124004:  <3368> <DBUG> |authmgr|  udb_gen_whitelist_avpairs: Added avpair name Cert_type value 1 
Apr 5 15:40:30 :124003:  <3368> <INFO> |authmgr|  Authentication result=Authentication Successful(0), method=VPN, server=Internal, user=9c:8c:d8:09:05:0c 

 

 

Thanks for all the replies.

 

show aaa authentication-server internal

Internal Server
---------------
Host IP addr Retries Timeout Status
---- ------- ------- ------- ------
Internal 10.11.101.11 3 5 Enabled

 

10.11.101.11 is the MM so that part looks ok.

 

LMB/BLMS is not set, my understanding is once provisioned (pointing at either public IP or private IP), the RAP should pull down the node list and be able to work out how it connects, is this not correct? I don't have them set in the ap-group i was using for then in CAP mode and they connect ok to both controllers and fail over.

 

show profile-errors shows nothing invalid on the mm or mc's.

 

Cheers

yes you're correct, lms/blms should not be configured for this to work and the AP will learn about the RAP public IPs as it comes up on the cluster.

 

Please enable "logging security level debugging" for the MD in question, try to connect your RAP and either compare it to what I posted before, or, post it here.

 

In essence, the RAP connects to the public IP configured as its master IP (via dns or whatever), gets port forwarded to one of the MDs, starts processing, is not found in the MD whitelist DB, the MD then queries the MM, it should return success together with the internal IP, and then the MD will add it to its local whitelist DB and proceeds to do normal RAP bring up.

 

The security debug on the MD that the master IP maps to should tell us 95% of what is wrong.

1000 lines attached, this is with the RAP connecting via the internal private ip.

 

Inner IP doesn't appear to be making it down to the MD:

Apr 5 21:30:18 :103063:  <3593> <DBUG> |ike|   ipc_ikev2_auth_recv_vpn_packet cookie:2236370049 innerip 0.0.0.0 inneripv6
Apr 5 21:30:18 :103063:  <3593> <DBUG> |ike|   ipc_ikev2_auth_recv_vpn_packet removing ctx 206bbe4 from auth-list. auth-cookie 2236370049
Apr 5 21:30:18 :103063:  <3593> <DBUG> |ike|   *** ipc_auth_recv_packet user=xx:xx:xx:xx:xx:xx, pass=******, result=0   ctx:0x206bbe4, ctx-innerip::: l2tp_pool:default-l2tp-pool
Apr 5 21:30:18 :103063:  <3593> <DBUG> |ike|   ipc_ikev2_auth_recv_vpn_packet rsp.cluster_rap_innerip 0.0.0.0 rsp.inner_ip 0.0.0.0
Apr 5 21:30:18 :103063:  <3593> <DBUG> |ike|    Peer:AUTH_HMAC_SHA1_96 Peer:ESN_0  <-- R   Notify: INTERNAL_ADDRESS_FAILURE (ESP spi=2fe2a400)#
Apr 5 21:30:18 :103063:  <3593> <DBUG> |ike|   SEND 80 bytes to 192.168.15.131(57906) (427784.469)

it is not that the inner IP is not making it, it is that the MD is authenticating the RAP in an odd way, such that it never tries to query the MM

 

The clue appears to be the log about skipping the cert CN check

 

Apr 5 21:30:11 :124004:  <3650> <DBUG> |authmgr|  RX (sock) message of type 66, len 1036
Apr 5 21:30:11 :124454:  <3650> <DBUG> |authmgr|  auth_user_query_raw: recvd request user:xx:xx:xx:xx:xx:xx ip:192.168.15.131 cookie:-2058597249
Apr 5 21:30:11 :132218:  <3650> <INFO> |authmgr|  Skipping certificate common name check for username= MAC=00:00:00:00:00:00
Apr 5 21:30:11 :124453:  <3650> <DBUG> |authmgr|  auth_user_query_resp: response user:xx:xx:xx:xx:xx:xx ip:192.168.15.131 cookie:-2058597249
Apr 5 21:30:11 :124198:  <3650> <ERRS> |authmgr|  {00:00:00:00:00:00-??} Missing server in attribute list, auth=VPN, utype=L3.
Apr 5 21:30:11 :124441:  <3650> <DBUG> |authmgr|  auth_user_query_resp: vpnflags:2
Apr 5 21:30:11 :124004:  <3650> <DBUG> |authmgr|  ip=192.168.15.131, sg=internal

 

which you can see is not in the working debug I posted before. I checked into the auth code and it would appear that if the VPN profile has "Check certificate common name against AAA server" disabled, then it circumvents this whole process of querying the local DB (and then the MM).

 

I also see that your RAP has picked up "aaa server-group internal" ( (sg=internal) which is also different from the default setting, which would have been "aaa server-group default"

 

So... please check if any of the following have been modified from their defaults, and/or add them here for discussion.

> show aaa server-group internal
> show aaa server-group default
> show aaa authentication vpn default-rap
> show aaa authentication vpn default

If you find that "Check certificate common name against AAA server" has been set to "disabled", especially in "aaa authentication vpn default-rap", please try changing it back to enabled.

In the case of a cluster, the RAP pool should be configured on the Mobility Master. In AOS 8.6.x.x, the RAP pool can only be configured under the 'Mobility Master' in the hierarchy, which is equivalent to /mm in the CLI.

Leave the LMS/BKUP-LMS empty.

 

Please validate in the WebUI that the RAP pool is configured as per the above. From the CLI, validate it by running the command:

'show configuration committed /mm | include lc-rap-pool'

 

Commands below. I should note, these were all factory default and have been setup from scratch with fairly minimal setup, just what was required to get them clustered and a few ssid's setup. CAP mode, works fine.

 

show aaa server-group internal

Fail Through:No
Load Balance:No

Auth Servers
------------
Name      Server-Type  trim-FQDN  Match-Type  Match-Op  Match-Str
----      -----------  ---------  ----------  --------  ---------
Internal  Internal     No

Role/VLAN derivation rules
---------------------------
Priority  Attribute  Operation  Operand  Type    Action    Value  Validated
--------  ---------  ---------  -------  ----    ------    -----  ---------
1         Role       value-of            String  set role         No

 

show aaa server-group default

Fail Through:No
Load Balance:No

Auth Servers
------------
Name      Server-Type  trim-FQDN  Match-Type  Match-Op  Match-Str
----      -----------  ---------  ----------  --------  ---------
Internal  Internal     No

Role/VLAN derivation rules
---------------------------
Priority  Attribute  Operation  Operand  Type    Action    Value  Validated
--------  ---------  ---------  -------  ----    ------    -----  -------

 

show aaa authentication vpn default-rap

VPN Authentication Profile "default-rap" (Predefined (changed))
---------------------------------------------------------------
Parameter                                         Value
---------                                         -----
Default Role                                      default-vpn-role
Server Group                                      internal
RADIUS Accounting Server Group                    N/A
Max Authentication failures                       0
Check certificate common name against AAA server  Disabled
Export VPN IP address as a route                  Enabled
User idle timeout                                 N/A
PAN Firewall Integration                          Disabled

show aaa authentication vpn default

VPN Authentication Profile "default"
------------------------------------
Parameter                                         Value
---------                                         -----
Default Role                                      default-vpn-role
Server Group                                      default
RADIUS Accounting Server Group                    N/A
Max Authentication failures                       0
Check certificate common name against AAA server  Enabled
Export VPN IP address as a route                  Enabled
User idle timeout                                 N/A
PAN Firewall Integration                          Disabled

Re enabling "Check certificate common name against AAA server" seems to have fixed the issue (i don't recall disabling....)

 

Will re provision to talk to public IP and continue testing.

Thanks for your help!
 

that's good news, if anything else crops up let us know here.

 

@jgoff
Thank you for the detailed explanation. After recreating my cluster with public IPs I ran into some headaches with RAP provisioning. The "RAP solution Guide with Clusters" PDF that has been making its way around SE circles is a great document but it fails to mention most of what you posted and more or less assumes one is creating things like a cluster for the first time. I was able to resolve my issues by making sure the the internal db was referenced and that certificate checking against AAA server was enabled.

@FPU_RB
Glad to know it was helpful - couple of things:

Can you send me a copy of the exact document you're referring to, either a link to it, or, DM me a dropbox link or some such ?

I thought it a bit of a rare case that someone would have changed these defaults, I can potentially look to chase up the authors (if possible) of the doc you're referring to, but can you elaborate a bit how it came to be that your system had non default values here ?

 

thanks. -jeff

@jgoff
Sure thing. I've attached the PDF here.

The geniuses who made changes to the default server group is this case were the professional services outfit that helped us stand up our solution back in early 2019. Make a change on a default profile and then don't reference it. That's smart right? /s I find myself still chasing little things like this down a year and a half later.

When I was having issues, I noticed the APs were hitting our ClearPass Publisher strangely considering we were only planning on using the internal RAP whitelist DB on the MMs. To get it to stop showing up in access tracker, incorrectly, I thought it wise to disable "Check certificate common name against AAA server" in the L3 Authentication/VPN Authentication/default-rap profile. A combination of these 2 items made the RAPs not able to complete joining the cluster.

@FPU_RB,

Thanks for the doc, ok, that's the one I thought it was going to be. I'll reach out to the author and see about putting a note in about this little gotcha (or a quick cross check for it).

 

I hear you regarding twiddling of knobs, it's an unfortunate double edge sword - having so many things to tweak increases the surface area for making changes that appear benign but turn out to have an impact (which is not least on us, as there is little warning or documentation to really deeply explain what might happen). But, if we take those options away or hide them, then sure enough we will find out there are a few customers using them.

 

Hi 

I'm running code 8.7.0.0, have two MC 7010 in cluster 
Have setup with two RAP public IP as described in document Aruba Remote Access Point Solution Guide for Teleworkers and Home Offices 
Provisioning of RAP seemes working fine, AP is converting to RAP. Is seems also that Ap is getting inner IP and establishing Ipsec tunnel ok with first controller and authenticating ok.
But in some point something is going wrong and tunnel is deleted and RAP is not comming up.
I have defined whitelist for the RAP and tried all above suggestion (it seems so) 
If I disable clustering,  RAP is comming up 


I enclose log of debug (ike, localdb, l2tp, authmgr) 

Is there something what I miss in configuration ? Please help

Karol

------------------------------
Karol Karkowski
------------------------------

Original Message:
Sent: Aug 20, 2020 10:54 PM
From: Jeffrey Goff
Subject: RAP not connecting to cluster after provisioning

@FPU_RB,

Thanks for the doc, ok, that's the one I thought it was going to be. I'll reach out to the author and see about putting a note in about this little gotcha (or a quick cross check for it).

 

I hear you regarding twiddling of knobs, it's an unfortunate double edge sword - having so many things to tweak increases the surface area for making changes that appear benign but turn out to have an impact (which is not least on us, as there is little warning or documentation to really deeply explain what might happen). But, if we take those options away or hide them, then sure enough we will find out there are a few customers using them.

 

Did you check the whitelist database on BOTH controllers (MDs)? I've seen weird things in my env where when provisioning, one controller's whitelist database gets updated properly but the other doesn't. I then fixed the one that didnt update properly via the cli.

From GUI I can see this whitelist entry on both MDs 
But from CLI with command
show whitelist-db rap
I can see only entry on first controller, on second is empty, but I have now provisioned RAP without clustering 

(Aruba-01) #show whitelist-db rap


AP-entry Details
----------------
Name AP-Group AP-Name Full-Name Authen-Username Revoke-Text AP_Authenticated Description Date-Added Enabled Remote-IP Remote-IPv6 Cluster-InnerIP Cluster-InnerIPv6 Cert-type
---- -------- ------- --------- --------------- ----------- ---------------- ----------- ---------- ------- --------- ----------- --------------- ----------------- ---------
20:4c:03:d0:9e:94 RAPy RAP1 Provisioned Sun Mar 28 23:19:31 2021 Yes 0.0.0.0 :: 192.168.237.11 0.0.0.0 factory-cert

AP Entries: 1

(Aruba-02) #show whitelist-db rap


AP-entry Details
----------------
Name AP-Group AP-Name Full-Name Authen-Username Revoke-Text AP_Authenticated Description Date-Added Enabled Remote-IP Remote-IPv6 Cluster-InnerIP Cluster-InnerIPv6 Cert-type
---- -------- ------- --------- --------------- ----------- ---------------- ----------- ---------- ------- --------- ----------- --------------- ----------------- ---------

------------------------------
Karol Karkowski
------------------------------

Original Message:
Sent: Mar 29, 2021 05:55 PM
From: Cody Ensanian
Subject: RAP not connecting to cluster after provisioning

Did you check the whitelist database on BOTH controllers (MDs)? I've seen weird things in my env where when provisioning, one controller's whitelist database gets updated properly but the other doesn't. I then fixed the one that didnt update properly via the cli.

------------------------------
Cody Ensanian

Original Message:
Sent: Mar 29, 2021 05:28 PM
From: Karol Karkowski
Subject: RAP not connecting to cluster after provisioning

Hi 

I'm running code 8.7.0.0, have two MC 7010 in cluster 
Have setup with two RAP public IP as described in document Aruba Remote Access Point Solution Guide for Teleworkers and Home Offices 
Provisioning of RAP seemes working fine, AP is converting to RAP. Is seems also that Ap is getting inner IP and establishing Ipsec tunnel ok with first controller and authenticating ok.
But in some point something is going wrong and tunnel is deleted and RAP is not comming up.
I have defined whitelist for the RAP and tried all above suggestion (it seems so) 
If I disable clustering,  RAP is comming up 


I enclose log of debug (ike, localdb, l2tp, authmgr) 

Is there something what I miss in configuration ? Please help

Karol

------------------------------
Karol Karkowski

Original Message:
Sent: Aug 20, 2020 10:54 PM
From: Jeffrey Goff
Subject: RAP not connecting to cluster after provisioning

@FPU_RB,

Thanks for the doc, ok, that's the one I thought it was going to be. I'll reach out to the author and see about putting a note in about this little gotcha (or a quick cross check for it).

 

I hear you regarding twiddling of knobs, it's an unfortunate double edge sword - having so many things to tweak increases the surface area for making changes that appear benign but turn out to have an impact (which is not least on us, as there is little warning or documentation to really deeply explain what might happen). But, if we take those options away or hide them, then sure enough we will find out there are a few customers using them.

 

Hello all,

 

Similar issues here.I have two 7210 lab controllers with internal IPs. They also have public IPs that I have configured in the Cluster. I have the RAPs pointing to the public IPs for their LMS, and they pop up briefly on the MM, but soon fall off again.

 

Can you guys tell me what command you ran to get to get the log output that pointed you to the cert CN check? I'd like to try and pinpoint where it's failing to reach the MM.

Can you post the output of 'show ap database' ? Is there any associated flags next to the AP? What about any errors in your config 'show profile-errors'? Do you also have LMS/BLMS in the AP System Profile?