Wireless Access

I have a 650 controller, 1 RAP5WN, 1 RAP2 (mine for testing), 3 Remote AP's AP124's, 3 Local AP's AP124's.

Both the RAP5 and RAP2 come up no problems, the AP124's though do not. I see the IKE phase one and two complete and when I do a sh user-table verbose (ip address) I see the AP's in the ap-role, same as the working RAP's. If I take the AP's back to the office and purge/save/boot, they show up on the controller as "unprovisioned" as intended.

I did a recent OS upgrade and I am worried that something happened that affects only the AP124's in my "remote" ap-group.

The controller was on 5.0.2.X I upgraded to 5.0.4.5 then upgraded to 6.1.3.1

Any suggestions?

Is it username/password authentication for the AP124 or cert-based authentication for RAP?

Found something else interesting i am seeing two roles when the remote AP hits the controller

(TBC Dell PC W-650) #show user-table verbose | include 108.
192.168.222.16   00:00:00:00:00:00  00:24:6c:c5:ba:8c  ap-role   00:00:02    VPN   108.15.30.217   N/A                                                                     default                   tunnel               Internal  1
108.15.30.217    00:00:00:00:00:00                     logon     00:01:58    VPN                   N/A                                                                     default                   tunnel                         1

Same ap I am assuming but the 192.168.222.16 has the "ap-role" and the 108.15.30.217 is the public IP address of that AP's gateway but it says logon with no MAC listed? Have i hit some but here?

ipsec sa
(TBC Dell PC W-650) #show crypto ipsec sa

IPSEC SA Active Session Information
-----------------------------------
Initiator IP     Responder IP     InitiatorID         ResponderID         Flags    Start Time      Inner IP
------------     ------------     -----------         -----------         -----  ---------------   --------
76.182.232.32    192.168.0.40     192.168.222.7/32    0.0.0.0/0           UT     May 29 20:02:58   192.168.222.7

IPSEC SA (V2) Active Session Information
-----------------------------------
Initiator IP     Responder IP     SPI(IN/OUT)        Flags Start Time        Inner IP
------------     ------------     ----------------   ----- ---------------   --------
108.15.30.217    192.168.0.40     1029a500/5150f000  UT2   May 29 20:36:30   192.168.222.17
173.10.144.118   192.168.0.40     a6e03200/45d72700  UT2   May 29 20:15:06   192.168.222.12

isakmp sa

(TBC Dell PC W-650) #show crypto isakmp sa

ISAKMP SA Active Session Information
------------------------------------
Initiator IP     Responder IP   Flags       Start Time      Private IP
------------     ------------   -----     ---------------   ----------
76.182.232.32    192.168.0.40   r-m-p-x-R May 29 20:02:58   192.168.222.7
108.15.30.217    192.168.0.40   r-v2-c-R  May 29 20:36:30   192.168.222.17
173.10.144.118   192.168.0.40   r-v2-c-R  May 29 20:15:07   192.168.222.12

Flags: i = Initiator; r = Responder
       m = Main Mode; a = Agressive Mode v2 = IKEv2
       p = Pre-shared key; c = Certificate/RSA Signature; e =  ECDSA Signature
       x = XAuth Enabled; y = Mode-Config Enabled; E = EAP Enabled
       3 = 3rd party AP; C = Campus AP; R = RAP
       V = VIA; S = VIA over TCP

Total ISAKMP SAs: 3

sh datapath session table | include 4500

(TBC Dell PC W-650) #show datapath session table | include 4500
192.168.0.40    173.10.144.118  17   4500  4500   0/0     0 0   13  1/5         2129 F
192.168.0.40    108.15.30.217   17   4500  1024   0/0     0 0   0   1/5         2    F
76.182.232.32   192.168.0.40    17   4500  4500   0/0     0 0   0   1/5         481c FC
173.10.144.118  192.168.0.40    17   4500  4500   0/0     0 0   0   1/5         2129 FC
192.168.0.40    76.182.232.32   17   4500  4500   0/0     0 0   134 1/5         481d F
108.15.30.217   192.168.0.40    17   1024  4500   0/0     0 0   0   1/5         3    FC

Tried both username and certificate. It is certifcate at the moment.

let's see the output of "show datapath session table 192.168.222.16" if 192.168.222.16 is the inner ip of the AP, when it comes up to see if anything is being blocked..

It keeps changing ip addresses, i am guessing after it reboots, it gets another ip?

(TBC Dell PC W-650) #show datapath session table | include 192.168.222.19
192.168.0.40    192.168.222.19  1    316   2048   0/0     0 0   0   local       3    FCI
192.168.222.19  192.168.0.40    1    316   0      0/0     0 0   0   local       3    FYI

It actually incrementing by one IP address each time? up to .21 now.

That is because it is probably rebootstrappping.

type "show log system 50" to see why the AP is rebootstrapping.  While it is connected, try to do "show datapath session table <ip address of ap>" to see if any  traffic is being blocked.

(TBC Dell PC W-650) #show datapath session table | include 192.168.222.29
192.168.0.40    192.168.222.29  1    377   2048   0/0     0 0   0   local       9    FCI
192.168.0.40    192.168.222.29  1    376   2048   0/0     0 0   1   local       e    FCI
192.168.0.40    192.168.222.29  1    378   2048   0/0     0 0   0   local       4    FCI
192.168.222.29  192.168.0.40    1    378   0      0/0     0 0   1   local       4    FYI
192.168.222.29  192.168.0.40    1    377   0      0/0     0 0   1   local       9    FYI
192.168.222.29  192.168.0.40    1    376   0      0/0     0 0   1   local       e    FYI

I can only catch it for a matter of seconds and i don't see anything related to it in the system logs.

Let's see the output of "show rights ap-role"

No problem

(TBC Dell PC W-650) #  show rights ap-role

Derived Role = 'ap-role'
 Up BW:No Limit   Down BW:No Limit
 L2TP Pool = default-l2tp-pool
 PPTP Pool = default-pptp-pool
 Periodic reauthentication: Disabled
 ACL Number = 4/0
 Max Sessions = 65535


access-list List
----------------
Position  Name     Location
--------  ----     --------
1         control
2         ap-acl

control
-------
Priority  Source  Destination  Service       Action  TimeRange  Log  Expired  Queue  TOS  8021P  Blacklist  Mirror  DisScan  ClassifyMedia  IPv4/6
--------  ------  -----------  -------       ------  ---------  ---  -------  -----  ---  -----  ---------  ------  -------  -------------  ------
1         user    any          udp 68        deny                             Low                                                           4
2         any     any          svc-icmp      permit                           Low                                                           4
3         any     any          svc-dns       permit                           Low                                                           4
4         any     any          svc-papi      permit                           Low                                                           4
5         any     any          svc-sec-papi  permit                           Low                                                           4
6         any     any          svc-cfgm-tcp  permit                           Low                                                           4
7         any     any          svc-adp       permit                           Low                                                           4
8         any     any          svc-tftp      permit                           Low                                                           4
9         any     any          svc-dhcp      permit                           Low                                                           4
10        any     any          svc-natt      permit                           Low                                                           4
ap-acl
------
Priority  Source  Destination  Service        Action  TimeRange  Log  Expired  Queue  TOS  8021P  Blacklist  Mirror  DisScan  ClassifyMedia  IPv4/6
--------  ------  -----------  -------        ------  ---------  ---  -------  -----  ---  -----  ---------  ------  -------  -------------  ------
1         any     any          svc-gre        permit                           Low                                                           4
2         any     any          svc-syslog     permit                           Low                                                           4
3         any     user         svc-snmp       permit                           Low                                                           4
4         user    any          svc-http       permit                           Low                                                           4
5         user    any          svc-http-accl  permit                           Low                                                           4
6         user    any          svc-smb-tcp    permit                           Low                                                           4
7         user    any          svc-msrpc-tcp  permit                           Low                                                           4
8         user    any          svc-snmp-trap  permit                           Low                                                           4
9         user    any          svc-ntp        permit                           Low                                                           4
10        user    controller   svc-ftp        permit                           Low                                                           4

Expired Policies (due to time constraints) = 0

The thing that gets me on the issue of rights and or access control is the fact that two other remote ap's are working? A RAP2 and RAP5 which are obviously different but they still get the same role.

Looks just fine.

When the ap is up, type "show ap debug-log ap-name <name of ap>"

No worky. I used the MAC address as the name and it just shows "no ap found...."

You might want to open a parallel case with support.  It can be very painful troubleshooting this way.

I am wondering if that is the problem....let me walk through the configuration of the AP's..

So, a new ap gets plugged into my controller directly, it shows up as "unprovisioned"

I go to configuration>ap installation, select the AP and hit "provision"

Then I select the new group for the ap (remote in this case)

Then I enter the radio paremeters for 5 and 2.4GHz

Then I select the authentication method, hit yes for remote AP, use either certificate or user name password

Then under Master Discovery I put in the 209.155.3.XXX address for the public IP of the controller

Then I change the name of the ap at the bottom of the page

Apply and Reboot

After this the AP never shows back up as registered tothe controller because it cannot reach the 209.155 address from the corp lan. So we take the AP to it's home and....all of this starts happening.

Understand on the case/troubleshooting and I have a case but the problem is I have a Dell Controller- Power Connect W 650 so i have to go through Dell. So far they have not been very much help.....

One other interesting thing I found troubleshooting.....there is a difference between a working remote AP and the non-working remote AP when I dig a bit deeper into the ipsec sa messages

WORKING AP

(TBC Dell PC W-650) #show crypto ipsec sa peer 76.182.232.32


 Initiator IP: 76.182.232.32
 Responder IP: 192.168.0.40
 Initiator: No
 Initiator cookie:dc56258e019aca4d Responder cookie:c6356454092b11f5
 SA Creation Date: Wed May 30 08:23:09 2012
 Life secs: 7200
 Initiator Phase2 ID: 192.168.222.7/255.255.255.255
 Responder Phase2 ID: 0.0.0.0/0.0.0.0
 Phase2 Transform: EncAlg:esp-aes256 HMAC:esp-sha-hmac
 Encapsulation Mode:UDP-encapsulated Tunnel
 PFS: No
 OUT SPI 33DA9700, IN SPI 6033D400
 Inner IP 192.168.222.7, internal type C
 AP
 Reference count: 3

NON-WORKING AP

(TBC Dell PC W-650) #show crypto ipsec sa peer 108.15.30.217

 Initiator IP: 108.15.30.217
 Responder IP: 192.168.0.40
 Initiator: No
 SA Creation Date: Wed May 30 09:00:48 2012
 Life secs: 7200
 Exchange Type: IKE_SA (IKEV2)
 Phase2 Transform:Encryption Alg: AES 256 Authentication Alg: SHA1
 Encapsulation Mode Tunnel
 PFS: no
 IN SPI: A5055300, OUT SPI: 7AE9B700
 CFG Inner-IP 192.168.222.28
 Responder IP: 192.168.0.40

So, a coworker of mine found some errors in the logs I did not catch. I have attached the messages file as well.

FYI

Just located three new AP's and configured them for remotes and they all work!

So, I guess we will see what TAC says about getting some new AP's. I have already done a purge and a factory-reset on them.

I have a final update for those reading the post....sorry for the huge let down but the orginal AP's now work as well. waa, waaa, waaaaa. Don't get me wrong, I am very pleased they work, I was still hoping though for some fix that was not so obvious. 

The customer performed another factory_reset on the AP's and we reconfigured them. They now work. Not sure if the first factory reset did not take or if it did not actually happen? I guess I will never know.

Lessons Learned (or should I say reminded of):

1. Give your AP's plenty of time to upgrade themselves before upgrading your OS to the next version on the controller

2. If at all possible, physically be present during an upgrade, remote troubleshooting generally creates other unwanted variables

Until Next time.....

Ai