Wireless Access

Reply
Occasional Contributor II
Posts: 10
Registered: ‎10-21-2013

Remote AP dropped clients, tunnel IP change

We have been monitoring an Aruba 135 we have set up in the cafeteria of one of our high schools. All of the APs in this building are set up as remote APs as opposed to campus. We wanted to monitor AP and client health during the AP's peak usage time, the students' lunch hour. We have seen upwards of 100 clients on the AP during peak usage. We were unsure of the amount of stress the AP could handle, and we were using the students as a test. While monitoring the AP, I noticed all of the clients dropped for roughly 30 seconds. Most of the clients were able to re-associate fine, but a few seemed to have issues re-associating. We noticed on our controller that the AP also had it's IP tunnel changed during this brief outage.

 

 

 

Here is info I pulled from the debug log for the AP on the controller. At 12:00 is exactly when the outage occurred, and you can even see the AP's tunnel IP changes from 74.122 to 74.35. I believe the font that is bolded could mean something, but I am unsure how to interpret it.

 

Nov 6 11:58:44 stm[1413]: <132094> <WARN> |AP RHS_CAFE_330041@192.168.74.122 stm| ^[msg WPA2 Key Message 2] [mac c8:d1:5e:33:95:c6] [bssid 24:de:c6:f1:f5:40] [apname RHS_CAFE_330041]
Nov 6 11:58:46 stm[1413]: <132094> <WARN> |AP RHS_CAFE_330041@192.168.74.122 stm| ^[msg WPA2 Key Message 2] [mac c8:d1:5e:33:95:c6] [bssid 24:de:c6:f1:f5:40] [apname RHS_CAFE_330041]
Nov 6 11:58:47 stm[1413]: <132094> <WARN> |AP RHS_CAFE_330041@192.168.74.122 stm| ^[msg WPA2 Key Message 2] [mac c8:d1:5e:33:95:c6] [bssid 24:de:c6:f1:f5:40] [apname RHS_CAFE_330041]
Nov 6 11:59:44 stm[1413]: <132093> <ERRS> |AP RHS_CAFE_330041@192.168.74.122 stm| ^[msg WPA2 Key message 2] [mac 04:f7:e4:b1:8f:55] [bssid 24:de:c6:f1:f5:50] [apname RHS_CAFE_330041] [stcnt1 0] [stcnt2 1] [apcnt1 0] [apcnt2 2]
Nov 6 11:59:59 sapd[1392]: sapd_redun_config_dnsmasq, rewrite dnsmasq config file
Nov 6 12:00:25 sapd[1392]: PAPI_Send: sendto RAPPER_PORT2 failed: No such file or directory Message Code 9 Sequence Num is 27853
Nov 6 12:00:27 sapd[1392]: PAPI_Send: To: 7f000001:8424 Type:0x3 Timed out.
Nov 6 12:00:30 KERNEL(RHS_CAFE_330041@172.16.12.220): asap_station_add: WARNING: !
Nov 6 12:00:41 stm[1413]: <132094> <WARN> |AP RHS_CAFE_330041@192.168.74.35 stm| ^[msg WPA2 Key Message 2] [mac b0:9f:ba:35:97:b0] [bssid 24:de:c6:f1:f5:40] [apname RHS_CAFE_330041]
Nov 6 12:00:42 stm[1413]: <132094> <WARN> |AP RHS_CAFE_330041@192.168.74.35 stm| ^[msg WPA2 Key Message 2] [mac b0:9f:ba:35:97:b0] [bssid 24:de:c6:f1:f5:40] [apname RHS_CAFE_330041]
Nov 6 12:00:44 stm[1413]: <132094> <WARN> |AP RHS_CAFE_330041@192.168.74.35 stm| ^[msg WPA2 Key Message 2] [mac b0:9f:ba:35:97:b0] [bssid 24:de:c6:f1:f5:40] [apname RHS_CAFE_330041]

 

 

 

 

I ran a show ap remote debug association-failure ap-name on the AP right after the outage, and it returned a few clients that did not seem to reassociate properly:

 

b0:9f:ba:35:97:b0 RHS_CAFE_330041 24:de:c6:f1:f5:50 OpenVVSDWiFi auth 802.11a 1m:54s Did not attempt to associate
b0:9f:ba:35:97:b0 RHS_CAFE_330041 24:de:c6:f1:f5:40 OpenVVSDWiFi 802.11g 1m:54s Sapcp Ageout (internal ageout)
84:38:35:eb:92:a7 RHS_CAFE_330041 24:de:c6:f1:f5:50 OpenVVSDWiFi auth 802.11a 2m:14s Did not attempt to associate
c0:63:94:4e:b5:6e RHS_CAFE_330041 24:de:c6:f1:f5:50 OpenVVSDWiFi auth 802.11a 2m:14s Did not attempt to associate
c0:63:94:4e:b5:6e RHS_CAFE_330041 24:de:c6:f1:f5:40 OpenVVSDWiFi auth 802.11g 2m:14s Did not attempt to associate
8c:58:77:1c:39:c6 RHS_CAFE_330041 24:de:c6:f1:f5:40 OpenVVSDWiFi auth 802.11g 2m:14s Did not attempt to associate
Num Association Failures:6

 

 

I finally did a show ap debug system-status ap-name on the AP and saw that the CPU and Memory peaked right at the same time:

 

Peak CPU Util in the last one hour
----------------------------------
Timestamp CPU Util(%) Memory Util(%)
--------- ----------- --------------
2013-11-06 12:00:31 48 28

 

 

 

My original thought was maybe the AP had too many clients on it, as it had I believe 102 cilents right before the outage. If anyone has any insight or suggestions, please share. Thanks.

MVP
Posts: 1,422
Registered: ‎10-25-2011

Re: Remote AP dropped clients, tunnel IP change

How many APs are terminating IPSEC tunnels to this controller? This creates unnecessary overhead. Why no local controller?

Also, with RAPs all terminating to your controller, ARM is not working as designed, the RAPs see each other as neighbours/interferers unless you specifically marked them all as valid on the controller.

Have you confirmed that you did not suffer an internet outage or link issue? Have you check the switch logs as well?

What version of controller are you using?

Pasquale Monardo | Senior Network Solutions Consultant
ACDX #420 | ACMP
[If you found my post helpful, please give kudos!]
Occasional Contributor II
Posts: 10
Registered: ‎10-21-2013

Re: Remote AP dropped clients, tunnel IP change

 

Re: Remote AP dropped clients, tunnel IP change
 
How many APs are terminating IPSEC tunnels to this controller? Close to 100 APs, all of which are 135s
 
 
This creates unnecessary overhead. Why no local controller? We have over 20 buidlings located on our campus. All of our buildings are connected to our administration center through our own WAN. Our administration center houses the controller which is a W-7240. We were told that in our scenario, the single controller would be sufficient.

Also, with RAPs all terminating to your controller, ARM is not working as designed, the RAPs see each other as neighbours/interferers unless you specifically marked them all as valid on the controller. We were having issues in a few of our buildings where clients would randomly disconnect throughout the day. After countless hours of troubleshooting with Aruba phone support, we were told to try provisioning the APs as remote AP as opposed to campus. We were told that campus APs actually create two tunnels back to the controller (CPsec and IPsec) where as remote AP only creates the one IPsec. We then converted the APs to Remote AP in the buildings where we were having issues and have had great results.
 
We were not told that converting the APs to RAP would not have any issues with ARM, and from what I see, ARM is working properly.

Have you confirmed that you did not suffer an internet outage or link issue? We have a network monitoring system and did not see any issues with outages. No pings were lost to any of the switches or the building's router.
 
 Have you check the switch logs as well? We have not looked at switch logs yet, but we do have a few other APs connected to that switch that did not experience the same outage.

What version of controller are you using? Our controller is a W-7240 and is running OS 6.2.1.3.
 
Thank you for taking the time to respond and look forward to hearing back.
Search Airheads
Showing results for 
Search instead for 
Did you mean: