Wireless Access

Hello,

I'm a new to Aruba network administrator and had to rebuild a mobility master the other day.

I've verified both my vMM have the same IPSec key, and failover is intact, etc.

I've also verified my pMCs have the same IPSec key as each other, and the MM.

I have a group of 20 APs that were rebooted due to replacing UPSes which led me to discover they are no longer receiving profiles (at least they list 0 WLANs in AirWave) and am unsure of the cause.

I can provide log capture immediately, I'm not exactly sure which ones would be beneficial in this context. I used show log all 20 and looked to see what was going on as I watched an AP power cycle, it indicated it couldn't communicate with the controller.

In AirWave, I see they have a controller IP, but the Failover IP is generally 0.0.0.0 on the non-working APs.

Any insight would be appreciated.

What about the MD's (controllers)?  The APS cannot terminate on an mm.  You need to type "show log system 50" on the MDs to see if there is a problem.

When you say that it said it couldn't communicate with the server, what do you see? CPSEC issues, IPSEC Issues, etc... Have you checked your licensing on the MM also for the Controllers/APs since you rebuilt it? Your license passphrase may have changed as well.

I have not checked on licensing, I though because one was still in tact they would just share that when they synced up.

Here's output from show log system 50 on MD

Feb 25 12:48:44 :319001:  <6389> <ERRS> |ARM Process|  Unexpected (arm process) runtime error at handle_sta_management, 481, NO BSSID object (04:bd:88:70:b0:50)for STA 0c:51:01:a2:b6:70

Feb 25 12:48:46 :319001:  <6389> <ERRS> |ARM Process|  Unexpected (arm process) runtime error at handle_sta_management, 481, NO BSSID object (94:b4:0f:89:10:d1)for STA f0:24:75:a5:1d:42

Feb 25 12:48:47 :319001:  <6389> <ERRS> |ARM Process|  Unexpected (arm process) runtime error at handle_sta_management, 481, NO BSSID object (94:b4:0f:88:39:10)for STA a4:34:d9:f3:dc:1b

Feb 25 12:48:47 :319001:  <6389> <ERRS> |ARM Process|  Unexpected (arm process) runtime error at handle_sta_management, 481, NO BSSID object (18:64:72:50:31:60)for STA 74:4a:a4:dd:65:68

Feb 25 12:48:47 :319001:  <6389> <ERRS> |ARM Process|  Unexpected (arm process) runtime error at handle_sta_management, 481, NO BSSID object (90:4c:81:5f:41:f1)for STA d4:61:da:bd:31:b0

Feb 25 12:48:47 :319001:  <6389> <ERRS> |ARM Process|  Unexpected (arm process) runtime error at handle_sta_management, 481, NO BSSID object (18:64:72:50:31:60)for STA 74:4a:a4:dd:65:68

Feb 25 12:48:48 :319001:  <6389> <ERRS> |ARM Process|  Unexpected (arm process) runtime error at handle_sta_management, 481, NO BSSID object (04:bd:88:66:a6:60)for STA 74:b5:87:0f:c4:89

Feb 25 12:48:52 :319001:  <6389> <ERRS> |ARM Process|  Unexpected (arm process) runtime error at handle_sta_management, 481, NO BSSID object (ac:a3:1e:f0:ce:e1)for STA ce:a7:cf:f2:2a:28

Feb 25 12:48:53 :357002:  <6813> <WARN> |cfgdist| handle_read:702 (TID:6813) Status of ::ffff:10.1.32.48(00:1a:1e:01:56:f0) is now UP

Feb 25 12:48:55 :319001:  <6389> <ERRS> |ARM Process|  Unexpected (arm process) runtime error at handle_sta_management, 481, NO BSSID object (04:bd:88:66:a9:61)for STA 5c:70:a3:74:fd:8f

Feb 25 12:48:56 :319001:  <6389> <ERRS> |ARM Process|  Unexpected (arm process) runtime error at handle_sta_management, 481, NO BSSID object (94:b4:0f:88:87:60)for STA 48:43:7c:ca:e7:4d

Feb 25 12:48:58 :319001:  <6389> <ERRS> |ARM Process|  Unexpected (arm process) runtime error at handle_sta_management, 481, NO BSSID object (94:b4:0f:8a:2e:31)for STA 7c:a1:ae:be:9d:07

Feb 25 12:48:58 :319001:  <6389> <ERRS> |ARM Process|  Unexpected (arm process) runtime error at handle_sta_management, 481, NO BSSID object (04:bd:88:70:48:91)for STA 7c:f3:1b:bf:c0:a9

Feb 25 12:48:58 :319001:  <6389> <ERRS> |ARM Process|  Unexpected (arm process) runtime error at handle_sta_management, 481, NO BSSID object (94:b4:0f:8a:40:70)for STA 34:12:98:a8:a2:d7

Feb 25 12:49:01 :319001:  <6389> <ERRS> |ARM Process|  Unexpected (arm process) runtime error at handle_sta_management, 481, NO BSSID object (04:bd:88:70:92:c1)for STA e4:b2:fb:24:27:41

Feb 25 12:49:13 :355002:  <6415> <DBUG> |cert_dwnld|  cert_downld_mgr_papi_recv_cb: Recv from 10.1.32.48:15272

Feb 25 12:49:13 :355002:  <6415> <DBUG> |cert_dwnld|  cert_downld_mgr_papi_recv_cb: MessageCode: 80 len 148 data_len 72 Type 2

Feb 25 12:49:13 :355002:  <6415> <DBUG> |cert_dwnld|  cert_downld_mgr_papi_recv_cb: IP 10.1.32.52 l3_peer_ip 0.0.0.0 role 2 sync_state 0 got_master_ip 1 got_switch_ip 1

Feb 25 12:49:13 :355003:  <6415> <INFO> |cert_dwnld|  cert_downld_mgr_papi_recv_cb: Received cert req from 10.1.32.48:15272

Feb 25 12:49:13 :355002:  <6415> <DBUG> |cert_dwnld|  cert_downld_cert_req_hdlr: Received cert type 67108864 cert name master-ssh-pub-cert cert filename  from switchip 10.1.32.48

Feb 25 12:49:13 :355002:  <6415> <DBUG> |cert_dwnld|  cert_downld_cert_req_hdlr: After conversion cert type 4 cert name master-ssh-pub-cert from switchip 10.1.32.48

Feb 25 12:49:13 :355002:  <6415> <DBUG> |cert_dwnld|  cert_request_to_certmgr: cert_request_to_certmgr:295 cert_type 4, cert_name master-ssh-pub-cert

Feb 25 12:49:13 :355002:  <6415> <DBUG> |cert_dwnld|  cert_request_to_certmgr: cert_request_to_certmgr:312 cnr->certType 4

Feb 25 12:49:13 :355002:  <6415> <DBUG> |cert_dwnld|  cert_downld_cert_req_hdlr: Cert state returned 0

Feb 25 12:49:13 :355002:  <6415> <DBUG> |cert_dwnld|  cert_downld_cert_req_hdlr: Certificate master-ssh-pub-cert found at path /flash/certmgr/PublicCert/master-ssh-pub-cert

Feb 25 12:49:13 :355002:  <6415> <DBUG> |cert_dwnld|  cert_downld_cert_req_hdlr: Sending cert master-ssh-pub-cert size 1123

Feb 25 12:49:13 :355002:  <6415> <DBUG> |cert_dwnld|  cert_downld_send_cert_req_resp: resp.cert_type = 4, resp.cert_state =0, cert_state 0

Feb 25 12:49:13 :355002:  <6415> <DBUG> |cert_dwnld|  cert_downld_send_cert_req_resp: copied certBuf to papi_buffer

Feb 25 12:49:13 :355002:  <6415> <DBUG> |cert_dwnld|  cert_downld_send_papi_message: Sending msg 20736 to 10.1.32.48:15272

Feb 25 12:49:13 :355002:  <6415> <DBUG> |cert_dwnld|  cert_downld_send_papi_message: Sent message 20736 to 10.1.32.48:15272

Feb 25 12:49:13 :355002:  <6415> <DBUG> |cert_dwnld|  cert_downld_send_cert_req_resp: Cert response sent for cert master-ssh-pub-cert type 4 to 10.1.32.48

Feb 25 12:49:15 :305061:  <6036> <WARN> |stm|  sapm_proc_hello_req Ignoring the HELLO message from AP b4:5d:50:c6:4a:ee. MM doesn't support AP redirect or termination.

Feb 25 12:49:50 :305061:  <6036> <WARN> |stm|  sapm_proc_hello_req Ignoring the HELLO message from AP b4:5d:50:c6:4a:ee. MM doesn't support AP redirect or termination.

Feb 25 12:50:37 :357002:  <6814> <WARN> |cfgdist| handle_read:702 (TID:6814) Status of ::ffff:10.1.32.55(00:50:56:8a:fc:db) is now UP

Feb 25 12:50:39 :305061:  <6036> <WARN> |stm|  sapm_proc_hello_req Ignoring the HELLO message from AP b4:5d:50:c6:4a:ee. MM doesn't support AP redirect or termination.

Feb 25 12:50:41 :399838:  <5954> <WARN> |fpapps|  Duplicate MAP_ADD from IKE for default-psk-redundant-master-ipsecmap (gw 10.1.32.55) mapid 0 vlanid 0 flags 0x2 addr 10.1.32.55 mask 255.255.255.255 prio 0

Feb 25 12:50:41 :399838:  <5954> <WARN> |fpapps|  Received TUN_UP from IKE for default-psk-redundant-master-ipsecmap mapid 0x0, vlanid 0, flags = 0x2 uplink_priority 0

Feb 25 12:50:49 :305061:  <6036> <WARN> |stm|  sapm_proc_hello_req Ignoring the HELLO message from AP b4:5d:50:c6:4a:ee. MM doesn't support AP redirect or termination.

Feb 25 12:51:34 :305061:  <6036> <WARN> |stm|  sapm_proc_hello_req Ignoring the HELLO message from AP b4:5d:50:c6:4a:ee. MM doesn't support AP redirect or termination.

Feb 25 12:53:01 :305061:  <6036> <WARN> |stm|  sapm_proc_hello_req Ignoring the HELLO message from AP b4:5d:50:c6:4a:ee. MM doesn't support AP redirect or termination.

Feb 25 12:54:16 :305061:  <6036> <WARN> |stm|  sapm_proc_hello_req Ignoring the HELLO message from AP b4:5d:50:c6:4a:ee. MM doesn't support AP redirect or termination.

Feb 25 13:02:19 :399816:  <6000> <ERRS> |profmgr|  [/sc/mynode

Feb 25 13:04:55 :303022:  <WARN> |AP ADM-NOC-1@10.1.67.2 nanny|  Reboot Reason: AP rebooted Tue Feb 25 13:03:24 CST 2020; SAPD: Reboot requested by controller

Feb 25 13:05:31 :305061:  <6036> <WARN> |stm|  sapm_proc_hello_req Ignoring the HELLO message from AP b4:5d:50:c6:4a:ee. MM doesn't support AP redirect or termination.

Feb 25 13:06:08 :305061:  <6036> <WARN> |stm|  sapm_proc_hello_req Ignoring the HELLO message from AP b4:5d:50:c6:4a:ee. MM doesn't support AP redirect or termination.

Feb 25 13:07:22 :305061:  <6036> <WARN> |stm|  sapm_proc_hello_req Ignoring the HELLO message from AP b4:5d:50:c6:4a:ee. MM doesn't support AP redirect or termination.

Feb 25 13:08:27 :305061:  <6036> <WARN> |stm|  sapm_proc_hello_req Ignoring the HELLO message from AP b4:5d:50:c6:4a:ee. MM doesn't support AP redirect or termination.

Feb 25 13:09:40 :305061:  <6036> <WARN> |stm|  sapm_proc_hello_req Ignoring the HELLO message from AP b4:5d:50:c6:4a:ee. MM doesn't support AP redirect or termination.

Feb 25 13:10:58 :305061:  <6036> <WARN> |stm|  sapm_proc_hello_req Ignoring the HELLO message from AP b4:5d:50:c6:4a:ee. MM doesn't support AP redirect or termination.

Feb 25 13:12:23 :305061:  <6036> <WARN> |stm|  sapm_proc_hello_req Ignoring the HELLO message from AP b4:5d:50:c6:4a:ee. MM doesn't support AP redirect or termination.

From the looks of it, your APs are trying to terminate to the Mobility Masters and not the Controllers. Can you try configuring LMS or any Aruba Discovery Protocol methods to DNS, DHCP Options, or test with setting it static on the AP its self after factory reset (setenv serverip <Controller VIP/IP>)

Would that be something I misconfigured along the way with the MM rebuild?

I'm not sure what you might have configured, or not configured on the MM rebuild, but if it was redundant and added back to your HA MM cluster it should not have changed anything. It would be worth checking the LMS IP configured in the AP Group, under AP System Profile.

Gotcha; I'm checking into the licensing issue.

For the MM setup, I just followed the generic prompts when you run the OVA

Then I used the ASE to generate a L2 redundant config, set the ipsec key; priority, etc.

That's pretty much it.

If it helps, 340~ of my 360 APs that have not been powered off, are fine, I presume because they still have tunnels to the controllers, I'm just not sure where in the rebuild I would have broken that. It seemed pretty simple.

On the MM, type "show ap database" and see if any of the access points have flags.

No flags at mynode#> show ap database

If you don't have the "L" flag, your problem is not licensing.  Like the previous poster said, what is your discovery method for those access points?

ADP

Feb 25 13:45:52 <vrrp 208055>  <6992> <ERRS> |vrrp|  Invalid advertisement received for VRID 20, Mismatched authentication type

Feb 25 13:46:23 <vrrp 208055>  <6992> <ERRS> |vrrp|  Invalid advertisement received for VRID 20, Mismatched authentication type

Feb 25 13:47:24 <vrrp 208055>  <6992> <ERRS> |vrrp|  Invalid advertisement received for VRID 20, Mismatched authentication type

Feb 25 13:48:25 <vrrp 208055>  <6992> <ERRS> |vrrp|  Invalid advertisement received for VRID 20, Mismatched authentication type

Feb 25 13:49:00 <vrrp 208055>  <6992> <ERRS> |vrrp|  Invalid advertisement received for VRID 20, Mismatched authentication type

Feb 25 13:49:31 <vrrp 208055>  <6992> <ERRS> |vrrp|  Invalid advertisement received for VRID 20, Mismatched authentication type

This is from show system log.

Feel free to take a look at the Airheads Broadcasting Channel on how to set things up to check your work.  It is difficult to back into a deployment with limited information.

That message means that you misconfigured the preshared key on your VRRPs so they probably both are answering to the same ip address.

If you can share the configuration I can take a quick look for you and see what may be going on.

I'd appreciate that Dustin.

I resolved the first piece of the issue, it came down to what someone had suggested earlier, with the MM passphrase changing.

The last leg of my issues seems to be AirWave reporting.

If I look in the MM console I see my full client load(3-3500 clients), but Airwave is rarely above 700 or so. On top of that, it indicates the controllers were deleted from the cluster every 10 minutes or so.

But on the MM side, the cluster appears healthy, and balanced.