Wireless Access

Hi, 

I have 7 AP with 1 650 mobility controller. one of the AP goes down. it works for 10 minutes after rebooting, then goes down. with the power led flashing green and the ethernet, wireless led is off.

I am checking the status on the controller, it says that the AP is down. I don't know where to start on troubleshooting the problem, so any help is really appreciated. 

regards

Do the following :

- show log system all | include <apmac address>

- logging level debug ap-debug <apmac address>

- show log all <apmac address>

do a show ap license-usage to make sure you have enough licenses 

Do you have cpsec on ? show control-plane-security 

You could also console into the AP to make sure it's getting the right DHCP information or it can reach the right controller (master)

Make sure you haven't execeed the amount of APs that the controller is able to support :

Make sure that AP is getting PoE on the switch is connected

What LEDs are showing ?

hi, 

here is the output of the commands. 

(Aruba650) #show log system all | include 24:de:c6:c0:17:d5
Jun 20 04:48:44 :305049: <WARN> |stm| Unsecure AP "24:de:c6:c0:17:d5" (MAC 24:de:c6:c0:17:d5, IP 192.168.0.209) has been denied access because Control Plane Security is enabled and the AP is not approved.
Jun 20 04:49:02 :305049: <WARN> |stm| Unsecure AP "24:de:c6:c0:17:d5" (MAC 24:de:c6:c0:17:d5, IP 192.168.0.209) has been denied access because Control Plane Security is enabled and the AP is not approved.
Jun 20 04:49:25 :305049: <WARN> |stm| Unsecure AP "24:de:c6:c0:17:d5" (MAC 24:de:c6:c0:17:d5, IP 192.168.0.209) has been denied access because Control Plane Security is enabled and the AP is not approved.
Jun 20 04:50:32 :305048: <WARN> |stm| Dropping unsecure AP message code 16121 from AP at 192.168.0.209 (MAC address 24:de:c6:c0:17:d5)
Jun 20 04:50:35 :311002: <WARN> |AP 24:de:c6:c0:17:d5@192.168.0.209 sapd| Rebooting: SAPD: Rebooting after installing trust update. Factory Cert present
Jun 20 04:50:35 :303086: <ERRS> |AP 24:de:c6:c0:17:d5@192.168.0.209 nanny| Process Manager (nanny) shutting down - AP will reboot!
Jun 20 04:51:55 :311020: <ERRS> |AP 24:de:c6:c0:17:d5@192.168.0.209 sapd| An internal system error has occurred at file sapd_redun.c function redun_init_tunnel_master line 3048 error Unable to open /tmp/num_ipsec.
Jun 24 06:10:52 :311020: <ERRS> |AP 24:de:c6:c0:17:d5@192.168.0.209 sapd| An internal system error has occurred at file sapd_redun.c function sapd_proc_redun_msg line 4319 error Error: Received RC_OPCODE_ERROR lms 192.168.0.248 tunnel 0.0.0.0 RC_ERROR_ISAKMP_N_VERSION2_SUPPORTED.
Jun 24 06:11:43 :311020: <ERRS> |AP 24:de:c6:c0:17:d5@192.168.0.209 sapd| An internal system error has occurred at file sapd_redun.c function sapd_proc_redun_msg line 4319 error Error: Received RC_OPCODE_ERROR lms 192.168.0.248 tunnel 0.0.0.0 RC_ERROR_ISAKMP_N_VERSION2_SUPPORTED.
Jun 24 06:15:47 :311020: <ERRS> |AP 24:de:c6:c0:17:d5@192.168.0.209 sapd| An internal system error has occurred at file sapd_redun.c function sapd_proc_redun_msg line 4319 error Error: Received RC_OPCODE_ERROR lms 192.168.0.248 tunnel 0.0.0.0 RC_ERROR_ISAKMP_N_VERSION2_SUPPORTED.
Jul 9 20:06:56 :311020: <ERRS> |AP 24:de:c6:c0:17:d5@192.168.0.209 sapd| An internal system error has occurred at file sapd_redun.c function sapd_proc_redun_msg line 4319 error Error: Received RC_OPCODE_ERROR lms 192.168.0.248 tunnel 0.0.0.0 RC_ERROR_ISAKMP_N_VERSION2_SUPPORTED.

(Aruba650) # show control-plane-security

Control Plane Security Profile
------------------------------
Parameter Value
--------- -----
Control Plane Security Enabled
Auto Cert Provisioning Disabled
Auto Cert Allow All Enabled
Auto Cert Allowed Addresses N/A

control plane security is enabled, and there seems a problem with the security of the AP.

You can add the mac address of that AP into the CPSec whitelist but there's other options like a certain range of IPs or auto cert validation , here's the guide :

http://www.arubanetworks.com/techdocs/ArubaOS_60/UserGuide/control_plane.php

Below command would tell you the status of the AP. 

Here is the below command.

(Aruba) #show whitelist-db cpsec

Control-Plane Security Whitelist-entry Details
----------------------------------------------
MAC-Address Enable State Cert-Type Description Revoke Text Last Updated
----------- ------ ----- --------- ----------- ----------- ------------
00:24:6c:c8:68:7f Enabled certified-factory-cert factory-cert Mon Jul 8 21:28:49 2013
00:0b:86:68:bc:01 Enabled unapproved-no-cert switch-cert 00:0b:86:68:bc :01 Tue Mar 12 11:44:13 2013

You can manually add the mac address to see if that helps 

(Aruba) (config) #whitelist-db cpsec add mac-address 00:0b:86:68:bc:01 description 00:0b:86:68:bc:01

If you have console access to the AP, you can also reset the AP to see how it goes.

Thank you,

Sriram S

Techincal Support Engineer

srirams@arubanetworks.com

408.585.1928

The AP point mac address is there in the campus ap whitelist. and cert type is factory-certificate, also state is certified-switch-cert, and revoked is NO. 

I have no Idea why in the logs it says that the AP is not approved. also it says if you check the logs further down "enternal system error has occured"

I noticed this :

Auto Cert Provisioning Disabled

You already manually added the mac to the cpsec whitelist ?

Try to clear it / delete it from the list and readd it again

Hi Fayez,

Please let me know if you still have issues after we reset, clear and re-enable the auto-cert provisioning parameter.

You can also reach me at my desk 408.585.1928

Thank you,

Sriram Subramanian

Technical Support Engineer

srirams@arubanetworks.com

408.585.1928..

Hi all, 

I tried deleting the cert and adding it again manually, it didn't work. I tried enabling aut cert provisioining, it is enabled at the moment no luck. I will try reseting the access point from the console connection next and see how it goes.

below is the log again. it still shows, at todays date that the access point in unapproved, while trying to add the ap to the whitelist gives that the "entry already exits". as you can see from both of the command outputs below.

(Aruba650) #show log system all | include 24:de:c6:c0:17:d5
Jun 20 04:48:44 :305049: <WARN> |stm| Unsecure AP "24:de:c6:c0:17:d5" (MAC 24: de:c6:c0:17:d5, IP 192.168.0.209) has been denied access because Control Plane S ecurity is enabled and the AP is not approved.
Jun 20 04:49:02 :305049: <WARN> |stm| Unsecure AP "24:de:c6:c0:17:d5" (MAC 24: de:c6:c0:17:d5, IP 192.168.0.209) has been denied access because Control Plane S ecurity is enabled and the AP is not approved.
Jun 20 04:49:25 :305049: <WARN> |stm| Unsecure AP "24:de:c6:c0:17:d5" (MAC 24: de:c6:c0:17:d5, IP 192.168.0.209) has been denied access because Control Plane S ecurity is enabled and the AP is not approved.
Jun 20 04:50:32 :305048: <WARN> |stm| Dropping unsecure AP message code 16121 from AP at 192.168.0.209 (MAC address 24:de:c6:c0:17:d5)
Jun 20 04:50:35 :311002: <WARN> |AP 24:de:c6:c0:17:d5@192.168.0.209 sapd| Rebo oting: SAPD: Rebooting after installing trust update. Factory Cert present
Jun 20 04:50:35 :303086: <ERRS> |AP 24:de:c6:c0:17:d5@192.168.0.209 nanny| Proc ess Manager (nanny) shutting down - AP will reboot!
Jun 20 04:51:55 :311020: <ERRS> |AP 24:de:c6:c0:17:d5@192.168.0.209 sapd| An i nternal system error has occurred at file sapd_redun.c function redun_init_tunne l_master line 3048 error Unable to open /tmp/num_ipsec.
Jun 24 06:10:52 :311020: <ERRS> |AP 24:de:c6:c0:17:d5@192.168.0.209 sapd| An i nternal system error has occurred at file sapd_redun.c function sapd_proc_redun_ msg line 4319 error Error: Received RC_OPCODE_ERROR lms 192.168.0.248 tunnel 0.0 .0.0 RC_ERROR_ISAKMP_N_VERSION2_SUPPORTED.
Jun 24 06:11:43 :311020: <ERRS> |AP 24:de:c6:c0:17:d5@192.168.0.209 sapd| An i nternal system error has occurred at file sapd_redun.c function sapd_proc_redun_ msg line 4319 error Error: Received RC_OPCODE_ERROR lms 192.168.0.248 tunnel 0.0 .0.0 RC_ERROR_ISAKMP_N_VERSION2_SUPPORTED.
Jun 24 06:15:47 :311020: <ERRS> |AP 24:de:c6:c0:17:d5@192.168.0.209 sapd| An i nternal system error has occurred at file sapd_redun.c function sapd_proc_redun_ msg line 4319 error Error: Received RC_OPCODE_ERROR lms 192.168.0.248 tunnel 0.0 .0.0 RC_ERROR_ISAKMP_N_VERSION2_SUPPORTED.
Jul 9 20:06:56 :311020: <ERRS> |AP 24:de:c6:c0:17:d5@192.168.0.209 sapd| An in ternal system error has occurred at file sapd_redun.c function sapd_proc_redun_m sg line 4319 error Error: Received RC_OPCODE_ERROR lms 192.168.0.248 tunnel 0.0. 0.0 RC_ERROR_ISAKMP_N_VERSION2_SUPPORTED.
Jul 9 22:03:39 :311020: <ERRS> |AP 24:de:c6:c0:17:d5@192.168.0.209 sapd| An in ternal system error has occurred at file sapd_redun.c function sapd_proc_redun_m sg line 4342 error Error: Received RC_OPCODE_ERROR lms 192.168.0.248 tunnel 0.0. 0.0 RC_ERROR_IKE_XAUTH_AUTHORIZATION_FAILED.
Jul 9 22:03:39 :311020: <ERRS> |AP 24:de:c6:c0:17:d5@192.168.0.209 sapd| An in ternal system error has occurred at file sapd_redun.c function redun_retry_tunne l line 3233 error redun_retry_tunnel: Switching to clear. Error:RC_ERROR_IKE_XAU TH_AUTHORIZATION_FAILED. Ipsec not successful after reboot.
Jul 10 06:08:10 :305049: <WARN> |stm| Unsecure AP "24:de:c6:c0:17:d5" (MAC 24: de:c6:c0:17:d5, IP 192.168.0.209) has been denied access because Control Plane S ecurity is enabled and the AP is not approved.
Jul 10 06:08:16 :305048: <WARN> |stm| Dropping unsecure AP message code 16121
Jul 10 06:08:25 :305049: <WARN> |stm| Unsecure AP "24:de:c6:c0:17:d5" (MAC 24: is enabled and the AP is not approved.
Jul 10 06:09:39 :305048: <WARN> |stm| Dropping unsecure AP message code 16121
Jul 9 22:05:41 :311002: <WARN> |AP 24:de:c6:c0:17:d5@192.168.0.209 sapd| Reboo
Jul 9 22:05:42 :303086: <ERRS> |AP 24:de:c6:c0:17:d5@192.168.0.209 nanny| Proce
Jul 9 22:13:35 :311020: <ERRS> |AP 24:de:c6:c0:17:d5@192.168.0.209 sapd| An in 4342 error Error: Received RC_OPCODE_ERROR lms 192.168.0.248 tunnel 0.0.0.0 RC_E
Jul 9 22:13:35 :311020: <ERRS> |AP 24:de:c6:c0:17:d5@192.168.0.209 sapd| An in 233 error redun_retry_tunnel: Switching to clear. Error:RC_ERROR_IKE_XAUTH_AUTHO
Jul 10 06:18:06 :305048: <WARN> |stm| Dropping unsecure AP message code 16121
Jul 9 22:14:09 :311002: <WARN> |AP 24:de:c6:c0:17:d5@192.168.0.209 sapd| Reboo
Jul 9 22:14:09 :303086: <ERRS> |AP 24:de:c6:c0:17:d5@192.168.0.209 nanny| Proce

(Aruba650) # show whitelist-db cpsec

Control-Plane Security Whitelist-entry Details
----------------------------------------------
MAC-Address Enable State Cert-Type Description Re voke Text Last Updated
----------- ------ ----- --------- ----------- -- --------- ------------
24:de:c6:c0:17:cd Enabled certified-switch-cert factory-cert Tue Jun 11 09:04:06 2013
24:de:c6:c0:17:d2 Enabled certified-switch-cert factory-cert Tue Jun 11 09:04:45 2013
24:de:c6:c0:17:d4 Enabled certified-switch-cert factory-cert Thu Jun 20 11:43:41 2013
24:de:c6:c0:17:d0 Enabled certified-switch-cert factory-cert Thu Jun 20 11:43:57 2013
24:de:c6:c0:17:cf Enabled certified-switch-cert factory-cert Thu Jun 20 11:43:58 2013
24:de:c6:c0:17:d5 Enabled certified-switch-cert factory-cert Wed Jul 10 06:19:31 2013
24:de:c6:c0:17:cb Enabled certified-switch-cert factory-cert Thu Jun 20 12:56:14 2013

Total Entries: 7

(Aruba650) (config) #whitelist-db cpsec add mac-address 24:de:c6:c0:17:d5 descri ption 24:de:c6:c0:17:d5
Entry already exists!

I tried manually entering the cert, didn't work. I tried enabling auto cert provisioning didn't work. I then reset the access point and connect it to console port. 

while trying to reconfigure the access point, I noticed that the ap reboots by it self every 5 min or so. it send sigkill messages to all the processes and reboots.

as you can see for the attached console output file.

any ideas

Attachment(s)

aruba ap2.txt   5 KB 1 version

Hi Fayez,

Can we check if this AP is crashing as all other AP`s are working fine?

Below command will give that info.

show ap debug crash-info ap-name <name of the ap>

Make sure port connected to the AP doesnt show up errors or not going up & down.

Below command will provide what traffic comes in from AP itself.

show datapath session table ap-name <name of the ap>

Reason for reboot:-

==============

show ap debug system-status ap-name <name of the ap>

Could you please post the show log system all | include <ap-name> as this will fetch you more information about AP itself.

Thank you,

Sriram S

Technical Support Engineer

srirams@arubanetworks.com

408.585.1928

I dont understand, what exactly do you mean?

The reason I ask is because I noticed the following on the console output you provided :

The 6.2.0.0 - 3.2.0.2 format is usually seen on the Aruba Instant APs 

Aruba Instant :
Aruba Networks
ArubaOS Version 6.2.0.0-3.2.0.2 (build 37229 / label #37229)

Regular Campus AP:
Aruba Networks
ArubaOS Version 6.2.0.3 (build 38054 / label #38054)

it is instant because i reset it, trying to find if that would help. all the other AP are campus APs.

this is what i tried lately. I took one of the working configured AP and connected it in the place of the faulty one. what happened is that the working AP stopped working. I'm guessing that there is something wrong with the cabling, I'm gonna look into that. the weird thing though, is when i place the working AP back into its old place, it is not working.

I mean are they this fragale that if you take it offline and connect it again it wont work. now I have two AP that are not working, everything is fallinig apart. I have no idea what is happenning.