Wireless Access

Hey Guys,

Just upgraded a 620 Controller to Release 6.3.1.1 and I noticed my RAP-2WG just died.

A "datapath session" command shows the session from my RAP to the Wireless Controller but I don't see it come up.

Wireless Controller was previously on Release 6.1.X.

Any one seen this issue before?

In the datapath session table output, are there any "D" for deny's ?

---

@jfernyc wrote:

In the datapath session table output, are there any "D" for deny's ?

---

I don't believe I saw one.

---

@jfernyc wrote:

In the datapath session table output, are there any "D" for deny's ?

---

(Aruba620BurlingtonMaster) #show datapath session | include 4500
192.168.100.61 99.227.188.4 17 4500 49793 0/0 0 0 2 1/8 1a 0 0 FY
192.168.100.61 99.227.188.4 17 4500 49795 0/0 0 0 0 1/8 3 0 0 FY
99.227.188.4 192.168.100.61 17 49795 4500 0/0 0 0 0 1/8 3 0 0 FC
99.227.188.4 192.168.100.61 17 49793 4500 0/0 0 0 1 1/8 1a 0 0 FC

(Aruba620BurlingtonMaster) #

Start debugging:

config t
logging level debugging security subcat ike
logging level debugging security process aaa
logging level debugging security process authmgr
logging level debugging security subcat l2tp
logging level debugging security subcat vpn

 Connect the RAP, then type "show log security 50" and see if there are any error messages that correspond to that RAP.

To turn of debugging:

config t
no logging level debugging security subcat ike
no logging level debugging security process aaa
no logging level debugging security process authmgr
no logging level debugging security subcat l2tp
no logging level debugging security subcat vpn

---

@cjoseph wrote:

Start debugging:

 

config t
logging level debugging security subcat ike
logging level debugging security process aaa
logging level debugging security process authmgr
logging level debugging security subcat l2tp
logging level debugging security subcat vpn

 Connect the RAP, then type "show log security 50" and see if there are any error messages that correspond to that RAP.

 

To turn of debugging:

config t
no logging level debugging security subcat ike
no logging level debugging security process aaa
no logging level debugging security process authmgr
no logging level debugging security subcat l2tp
no logging level debugging security subcat vpn

 

---

Nov 3 07:50:30 :103061: <ERRS> |ike| 99.227.188.4:49152-> find_listening_transport: virtual transport for address 192.168.100.61 not found in virtual_listen_list
Nov 3 07:50:30 :103063: <DBUG> |ike| 99.227.188.4:49152-> transport_handle_messages Could not find transports for IP-Port:c0a8643d:4500, Re-init the transports
Nov 3 07:50:35 :103061: <ERRS> |ike| 99.227.188.4:49152-> find_listening_transport: virtual transport for address 192.168.100.61 not found in virtual_listen_list
Nov 3 07:50:35 :103063: <DBUG> |ike| 99.227.188.4:49152-> transport_handle_messages Could not find transports for IP-Port:c0a8643d:4500, Re-init the transports
Nov 3 07:50:40 :103061: <ERRS> |ike| 99.227.188.4:49152-> find_listening_transport: virtual transport for address 192.168.100.61 not found in virtual_listen_list
Nov 3 07:50:40 :103063: <DBUG> |ike| 99.227.188.4:49152-> transport_handle_messages Could not find transports for IP-Port:c0a8643d:4500, Re-init the transports

Nov 3 07:59:38 :103061: <ERRS> |ike| 99.227.188.4:49199-> find_listening_transport: virtual transport for address 192.168.100.61 not found in virtual_listen_list
Nov 3 07:59:38 :103063: <DBUG> |ike| 99.227.188.4:49199-> transport_handle_messages Could not find transports for IP-Port:c0a8643d:4500, Re-init the transports
Nov 3 07:59:43 :103061: <ERRS> |ike| 99.227.188.4:49199-> find_listening_transport: virtual transport for address 192.168.100.61 not found in virtual_listen_list
Nov 3 07:59:43 :103063: <DBUG> |ike| 99.227.188.4:49199-> transport_handle_messages Could not find transports for IP-Port:c0a8643d:4500, Re-init the transports
Nov 3 07:59:52 :103061: <ERRS> |ike| 99.227.188.4:49201-> find_listening_transport: virtual transport for address 192.168.100.61 not found in virtual_listen_list
Nov 3 07:59:52 :103063: <DBUG> |ike| 99.227.188.4:49201-> transport_handle_messages Could not find transports for IP-Port:c0a8643d:4500, Re-init the transports

Haven't seen this error type before. Any idea what the above means?

Do you have your RAPs pointing at a VRRP?

---

@cjoseph wrote:

Do you have your RAPs pointing at a VRRP?

 

---

No I don't. This is really strange.

Don't know if resetting the RAP to factory default will help. I doubt cos, as you can see, the RAP is definitely attempting to establish a session to the Controller.

What do you think?

I already have a Ticket opened with TAC. However, I think the TAC Engr is of the opinion that maybe resetting the RAP to factory default might help.

However, one thing I noticed right when I login to the Controller via CLI, I see the below error:

WARNING: This controller has RAP whitelist data stored in pre-6.3 format, which is consuming excess flash space. You will need this data if you ever need to downgrade the software to pre-6.3 release. If you have backed up your flash already, you may delete the pre-6.3 data by running the command 'local-userdb-ap del all'

Please do the following :

show  vpdn l2tp local pool - make sure you have the RAP pool active

show  crypto ipsec sa - ipsec phase 1 

show  crypto isakmp sa - ike phase 1 

show user-table verbose 

show  rights ap-role

How are you configuring this RAPs ?  RAP whitelist ?

show  local-userdb-ap

How are your RAPs reaching the controller ? 

---

@victorfabian wrote:

 

Please do the following :

 

show  vpdn l2tp local pool - make sure you have the RAP pool active

show  crypto ipsec sa - ipsec phase 1 

show  crypto isakmp sa - ike phase 1 

show user-table verbose 

 

show  rights ap-role

 

How are you configuring this RAPs ?  RAP whitelist ?

show  local-userdb-ap

 

How are your RAPs reaching the controller ? 

---

Yes I'm doing a RAP Whitelist.

My RAP is sitting behind a Home Router and assigned to a Public IP Address NATed to the Wireless Controller.

eosuorah,

Please try this:

process restart isakmpd core 
WARNING: Do you really want to create core and restart process: isakmpd (y/n): y
Creating core and restarting: isakmpd

---

@cjoseph wrote:

eosuorah,

 

Please try this:

 

process restart isakmpd core 
WARNING: Do you really want to create core and restart process: isakmpd (y/n): y
Creating core and restarting: isakmpd

 

---

Done. 

I suspect this has something to do with the Whitelist for the APs.

Prior to AOS 6.2 there was one Whitelist right? And my Version before my upgrade was AOS 6.1.

Something must have messed up the database or something. 

---

@eosuorah wrote:

I suspect this has something to do with the Whitelist for the APs.

 

Prior to AOS 6.2 there was one Whitelist right? And my Version before my upgrade was AOS 6.1.

Something must have messed up the database or something. 

---

eosuorah,

That message is just reminding you that the RAP database migrated.  It just says you can reclaim space by deleting the old database if you do not plan to downgrade.

The error message that we see is when the controller receives ISAKMP on an interface or ip address that it does not know about.  Does the RAP controller process traffic on a loopback address?

---

@cjoseph wrote:

---

@eosuorah wrote:

I suspect this has something to do with the Whitelist for the APs.

 

Prior to AOS 6.2 there was one Whitelist right? And my Version before my upgrade was AOS 6.1.

Something must have messed up the database or something. 

---

eosuorah,

 

That message is just reminding you that the RAP database migrated.  It just says you can reclaim space by deleting the old database if you do not plan to downgrade.

 

The error message that we see is when the controller receives ISAKMP on an interface or ip address that it does not know about.  Does the RAP controller process traffic on a loopback address?

 

---

Yeah I figured. Already got that sorted out.

Nope it processes traffic on it's VLAN Assigned IP Address.

---

@victorfabian wrote:

 

Please do the following :

 

show  vpdn l2tp local pool - make sure you have the RAP pool active

show  crypto ipsec sa - ipsec phase 1 

show  crypto isakmp sa - ike phase 1 

show user-table verbose 

 

show  rights ap-role

 

How are you configuring this RAPs ?  RAP whitelist ?

show  local-userdb-ap

 

How are your RAPs reaching the controller ? 

---

(Aruba620BurlingtonMaster) #show vpdn l2tp local pool

IP addresses used in pool VPNRAPPool
none

Total:-
0 IPs used - 11 IPs free - 11 IPs configured
IP pool allocations / de-allocations - L2TP: 0/0 IKE: 0/0

(Aruba620BurlingtonMaster) #

(Aruba620BurlingtonMaster) #show crypto ipsec sa

% No active IPSEC SA

(Aruba620BurlingtonMaster) #

(Aruba620BurlingtonMaster) #show crypto isakmp sa

% No active ISAKMP SA

(Aruba620BurlingtonMaster) #

(Aruba620BurlingtonMaster) #show local-userdb-ap

NOTE: This command has been deprecated. Please use "show whitelist-db rap" command.

AP Entries: 0

(Aruba620BurlingtonMaster) #show whitelist-db rap

AP-entry Details
----------------
Name AP-Group AP-Name Full-Name Authen-Username Revoke-Text AP_Authenticated Description Date-Added Enabled Remote-IP
---- -------- ------- --------- --------------- ----------- ---------------- ----------- ---------- ------- ---------
00:24:6c:cd:38:b5 RAPGroup SavioRAP Provisioned Sat Nov 2 11:48:14 2013 Yes 0.0.0.0
00:24:6c:cd:39:25 RAPGroup 00:24:6c:cd:39:25 Provisioned Sat Nov 2 11:49:58 2013 Yes 0.0.0.0

AP Entries: 2

(Aruba620BurlingtonMaster) #

We are having this same problem after upgrading a 620 from AOS 6.1.3.0 to AOS 6.3.1.0.  Resetting the RAP-2WG to factory defaults does not resolve the issue. The RAPs show as upgrading in their status, then they just reboot and try to upgrade again. My RAP-2WGs have AOS 5.0.0.0 on the backup partition.

I did not see this problem on a 3600 controller we upgraded to 6.3.1.0, but thats a different controller HW platform and image file.

I am planning to open a case as well.

Good thing i saw this topic before upgrading my office 620 to 6.3.1.1, got all the sales department  and most of the engineering with RAP 2 on their houses... this would have been a problem, to me... hope you guys fix this soon, so i can upgrade it :) hehe

Cheers

Carlos

---

@NightShade1 wrote:

Good thing i saw this topic before upgrading my office 620 to 6.3.1.1, got all the sales department  and most of the engineering with RAP 2 on their houses... this would have been a problem, to me... hope you guys fix this soon, so i can upgrade it :) hehe

 

Cheers

Carlos

---

Will keep you all updated.

Thanks that would be a great help :)

---

@stracey wrote:

We are having this same problem after upgrading a 620 from AOS 6.1.3.0 to AOS 6.3.1.0.  Resetting the RAP-2WG to factory defaults does not resolve the issue. The RAPs show as upgrading in their status, then they just reboot and try to upgrade again. My RAP-2WGs have AOS 5.0.0.0 on the backup partition.

 

I did not see this problem on a 3600 controller we upgraded to 6.3.1.0, but thats a different controller HW platform and image file.

 

I am planning to open a case as well.

---

I noticed that my Controller has a Crash Information. I have provided the Crash Information to Aruba and they will provide an update after analysis. The other thing is, my Campus AP 105s will not come up when I have "Control Plane Security" enabled. So right now, I have this disabled and my Campus AP 105s are up and running. However, the ones at our Remote Office which has one of the VAPs in Bridged Mode will not broadcast the associated SSID becuase I have Control Plane Security disabled.

So I have 2 issues to deal with. Control Plane Security impacting my CAPs and secondly, my RAPs not coming up at all.

---

@eosuorah wrote:

---

@stracey wrote:

We are having this same problem after upgrading a 620 from AOS 6.1.3.0 to AOS 6.3.1.0.  Resetting the RAP-2WG to factory defaults does not resolve the issue. The RAPs show as upgrading in their status, then they just reboot and try to upgrade again. My RAP-2WGs have AOS 5.0.0.0 on the backup partition.

 

I did not see this problem on a 3600 controller we upgraded to 6.3.1.0, but thats a different controller HW platform and image file.

 

I am planning to open a case as well.

---

I noticed that my Controller has a Crash Information. I have provided the Crash Information to Aruba and they will provide an update after analysis. The other thing is, my Campus AP 105s will not come up when I have "Control Plane Security" enabled. So right now, I have this disabled and my Campus AP 105s are up and running. However, the ones at our Remote Office which has one of the VAPs in Bridged Mode will not broadcast the associated SSID becuase I have Control Plane Security disabled.

 

So I have 2 issues to deal with. Control Plane Security impacting my CAPs and secondly, my RAPs not coming up at all.

 

---

eosuorah,

It could take up to 15 minutes when enable control plane security for all the access points to come up.  How long did you wait?

---

@cjoseph wrote:

---

@eosuorah wrote:

---

@stracey wrote:

We are having this same problem after upgrading a 620 from AOS 6.1.3.0 to AOS 6.3.1.0.  Resetting the RAP-2WG to factory defaults does not resolve the issue. The RAPs show as upgrading in their status, then they just reboot and try to upgrade again. My RAP-2WGs have AOS 5.0.0.0 on the backup partition.

 

I did not see this problem on a 3600 controller we upgraded to 6.3.1.0, but thats a different controller HW platform and image file.

 

I am planning to open a case as well.

---

I noticed that my Controller has a Crash Information. I have provided the Crash Information to Aruba and they will provide an update after analysis. The other thing is, my Campus AP 105s will not come up when I have "Control Plane Security" enabled. So right now, I have this disabled and my Campus AP 105s are up and running. However, the ones at our Remote Office which has one of the VAPs in Bridged Mode will not broadcast the associated SSID becuase I have Control Plane Security disabled.

 

So I have 2 issues to deal with. Control Plane Security impacting my CAPs and secondly, my RAPs not coming up at all.

 

---

eosuorah,

 

It could take up to 15 minutes when enable control plane security for all the access points to come up.  How long did you wait?

 

---

Good question. Hmm. I did wait awhile, but not sure if it was up to 15mins. I might try out again after hours to see.

---

@eosuorah wrote:

---

@cjoseph wrote:

---

@eosuorah wrote:

---

@stracey wrote:

We are having this same problem after upgrading a 620 from AOS 6.1.3.0 to AOS 6.3.1.0.  Resetting the RAP-2WG to factory defaults does not resolve the issue. The RAPs show as upgrading in their status, then they just reboot and try to upgrade again. My RAP-2WGs have AOS 5.0.0.0 on the backup partition.

 

I did not see this problem on a 3600 controller we upgraded to 6.3.1.0, but thats a different controller HW platform and image file.

 

I am planning to open a case as well.

---

I noticed that my Controller has a Crash Information. I have provided the Crash Information to Aruba and they will provide an update after analysis. The other thing is, my Campus AP 105s will not come up when I have "Control Plane Security" enabled. So right now, I have this disabled and my Campus AP 105s are up and running. However, the ones at our Remote Office which has one of the VAPs in Bridged Mode will not broadcast the associated SSID becuase I have Control Plane Security disabled.

 

So I have 2 issues to deal with. Control Plane Security impacting my CAPs and secondly, my RAPs not coming up at all.

 

---

eosuorah,

 

It could take up to 15 minutes when enable control plane security for all the access points to come up.  How long did you wait?

 

---

Good question. Hmm. I did wait awhile, but not sure if it was up to 15mins. I might try out again after hours to see.

 

---

Waited for over an hour and the CAPs didn't come up with Control Plane Security enabled.

Please open a case for the control plane security issue.

Please also open a separate thread for it as well, because this thread is about RAP-2WG issue with 6.3.11

---

@cjoseph wrote:

Please open a case for the control plane security issue.

 

Please also open a separate thread for it as well, because this thread is about RAP-2WG issue with 6.3.11

---

TAC is currently looking into this as well as this started right after upgrading to 6.3.1.1.

---

@eosuorah wrote:

---

@cjoseph wrote:

Please open a case for the control plane security issue.

 

Please also open a separate thread for it as well, because this thread is about RAP-2WG issue with 6.3.11

---

TAC is currently looking into this as well as this started right after upgrading to 6.3.1.1.

 

 

---

Okay, but this thread about RAP-2WG and 6.3.1.1.  Everyone who does a search for RAP2WG and 6.3.1.1 will pull up this thread, and this part of it is about control plane security, not the subject, so they will be misled.  Please open a separate thread about this specific issue, because people will be misled by the title.  We can then offer troubleshooting steps on that thread.

---

@cjoseph wrote:

---

@eosuorah wrote:

---

@cjoseph wrote:

Please open a case for the control plane security issue.

 

Please also open a separate thread for it as well, because this thread is about RAP-2WG issue with 6.3.11

---

TAC is currently looking into this as well as this started right after upgrading to 6.3.1.1.

 

 

---

Okay, but this thread about RAP-2WG and 6.3.1.1.  Everyone who does a search for RAP2WG and 6.3.1.1 will pull up this thread, and this part of it is about control plane security, not the subject, so they will be misled.  Please open a separate thread about this specific issue, because people will be misled by the title.  We can then offer troubleshooting steps on that thread.

---

I agree. Just opened a separate thread already.

Hello All,

This issue has been finally been resolved with the help from the Engineering/Development Team.

For some reason, it was determined that my Controller had two Wired Oplinks both on separate VLANs (1 and 10). No one knows for sure if the upgrade created this 2nd Wired Uplink.

So, Aruba APs communicate to the Contoller via this Wired Uplink on an internally created VLAN (VLAN ID 4095) for processes. So with the fact that my Controller had two Wired Uplinks on separate VLANs, this created a conflict and thus led to the CAPs and RAPs not establishing sessions back to the Controller.

The minute we removed the 2nd Wired Uplink configuration (for VLAN 10), my RAPs established their ISAKMP session back to the Controller with CPSec enabled. See below for the highlighted issue:

(Aruba620BurlingtonMaster) #show running-config | include vlan

Building Configuration...

controller-ip vlan 1

vlan 10 "Mgmt" 

vlan 101 

vlan 333 

trusted vlan 1-4094

switchport access vlan 101

trusted vlan 1-4094

trusted vlan 1-4094

trusted vlan 1-4094

trusted vlan 1-4094

trusted vlan 1-4094

trusted vlan 1-4094

trusted vlan 1-4094

trusted vlan 1-4094

switchport trunk allowed vlan 1,10,101,221,333

interface vlan 1

interface vlan 10

interface vlan 101

interface vlan 333

uplink wired vlan 1

uplink wired vlan 10

adp igmp-vlan 0

   switchport access vlan 101

   vlan 333

   vlan 333                                       

   vlan 333

   vlan 333

   vlan 1

   vlan 1

   vlan 101

   vlan 1

   vlan 221

   vlan 10

(Aruba620BurlingtonMaster) # 

---

@cjoseph wrote:
Thank you so much for following up.

---

You are very welcome! Can I get a Kudos from everyone? LOL!