Wireless Access

Hi All,

I'm running an eval of Airwave.. one of the problems it's uncovered is a ton of radius time outs - specifically "Authentication server request timed out for XX-SERVER"

In trying to correct this issue I setup a second NPS server to serve a smaller site (<100 devices). It's generating time out errors too..

So that's got me wondering if Aruba/Airwave isn't reporting this data correctly, or wondering if NPS is just poorly suited to serve up radius for a wireless network.

What's your experience/design been in setting up a NPS server(s) to accommodate 3500ish wireless clients across 9 controllers? Is there a better radius product that will authenticate against MS AD for machine and user authentication?

Thanks

Depending on the server configuration (hardware and other services) NPS can handle hundreds of requests per second.  Now, if this your DC, then it is obviously doing other things as well.     I have many customers using NPS for RADIUS, mainly for its ease of integration, and of course price.   If you need to stick with NPS, you could look at using an NPS Proxy to balance the requests across multiple servers.   But, since you asked, I'll answer:  ClearPass Policy Manager would be a good option to look at for an alternative RADIUS solution.

Getting back to your RADIUS timeouts; have you troubleshot it any further?  Are your clients complaining?  Do you have a lot of Apple/iOS devices?

Just asking the  nps server is locally on the site where you authenticating servers?

Or you got for example a wireless controller on a remote site and you got the NPS servers on like a data center or central site?

I ask you this because you can do EAP termination on the controller is is recommended in situations where the radisu server is not local to the controller....

EAP process is terminated on the controller and only radius request are send to the server...  Its good like i said when radius server is not local to the WLAN.

>Just asking the  nps server is locally on the site where you authenticating servers?

Yes and no.. I setup a second NPS server local to one of the controllers and the problem didn't go away.

>Or you got for example a wireless controller on a remote site and you got the NPS servers on like a data center or central site?

Yes...

>I ask you this because you can do EAP termination on the controller is is recommended in situations where the radisu server is not local to the controller....EAP process is terminated on the controller and only radius request are send to the server...  Its good like i said when radius server is not local to the WLAN.

Interesting... I believe I looked into that awhile back and it didn't fit well in our environment. I can't remeber why but I'll take another look...

Did you ever find a resolution to your problem? I'm having a very similar problem with RADIUS timeouts that I cannot get to the bottom of but I have a LOT less clients than you do.

Running 2008R2 NPS on an unloaded server connected to the same switch that my Aruba controller is on. EAP termination at the RADIUS server.

The RADIUS server is only getting hit by 5-6 clients per minute so you definitely have a much busier network than I do.

The Aruba controller complains of the RADIUS server timing out. From the other side, I don't see any errors or network distruptions regarding RADIUS - it simply just isn't seeing the traffic.

When you configure the second NPS  which is local to the controller, did you put that one as primary right on that controller?

Im sure you did but i still ask

Also like the other forum guy said

Did you ever found resolution to this?

I have setup some of those and never had issue with this kind of thing...

No resolution yet...

>When you configure the second NPS which is local to the controller, did you put that one as primary right on that controller?

Yeap...

> I have setup some of those and never had issue with this kind of thing...

What was the specs on the servers you used? hardware/os/hypervisor??

So I suspect these timeouts are a result of a group of misconfigured clients.

Here's what I'm seeing in the Windows Event Log

Log Name: Security
Source: Microsoft-Windows-Security-Auditing
Date: 11/28/2012 11:36:21 AM
Event ID: 6274
Task Category: Network Policy Server
Level: Information
Keywords: Audit Failure
User: N/A
Computer: SERVERNAME.dom.lan
Description:
Network Policy Server discarded the request for a user.

Contact the Network Policy Server administrator for more information.

User:
Security ID: S-1-5-21-547700318-1172196121-2737236298-41244
Account Name: loginname
Account Domain: DOM
Fully Qualified Account Name: DOM\loginname

Client Machine:
Security ID: NULL SID
Account Name: -
Fully Qualified Account Name: -
OS-Version: -
Called Station Identifier: 000B86041A80
Calling Station Identifier: 1474116FD51C

NAS:
NAS IPv4 Address: 172.25.197.2
NAS IPv6 Address: -
NAS Identifier: 172.22.197.5
NAS Port-Type: Wireless - IEEE 802.11
NAS Port: 19

RADIUS Client:
Client Friendly Name: Aruba
Client IP Address: 172.22.197.5

Authentication Details:
Connection Request Policy Name: Secure Wireless Connections
Network Policy Name: Secure Wireless Connections
Authentication Provider: Windows
Authentication Server: SERVERNAME.dom.lan
Authentication Type: EAP
EAP Type: -
Account Session Identifier: -
Reason Code: 1
Reason: An internal error occurred. Check the system event log for additional information.

I can't find a way to make NPS send an ACCESS-REJECT message to the client when this happens - so the controller sees this as a timeout.

As I mentioned earlier in this thread I had a similar problem. Unfortunately I do not have a resolution right now, but through troubleshooting with Aruba support I've come to the same conclusion as you did.

Traffic on my network this week is unusually light because most people are out on holiday. I saw some RADIUS timeouts and there were only a handful of clients inside my building. I checked on my RADIUS logs and saw that one user was failing to auth right before the timeout occured.

Now I have this user's iPad 4 running iOS 6.0.1 in my hands. When it's asleep, everything is well. As soon as I wake it up, it tries to connect to my wireless network and since it's not set up right it tries to connect, then fails, then tries again, etc.

While this happens, I can watch the timeout value climb from #show aaa authentication-server radius statistics. Checking on my NPS server, I get the exact same type of error you see. 

I suspect this user tried to set up her iPad on the wireless network - she input her domain credentials then it is my theory that she did not click the ACCEPT button my my self-signed certificate. So the iPad is throwing the saved credentials at my RADIUS server but it's not trusting the certificate I'm using with RADIUS.

My response to Aruba support was the exact question you posed - why isn't NPS reporting back to the controller with a failure? I understand that the user *does* in fact meet the conditions I specified in my policy...but doesn't the RADIUS standard say that it should answer back with something??

I was able to narrow down my issue to iPhones/iPads with outdated cached Active Directory credentials. Here is how I was able to reproduce the issue:

1. Join iWhatever to the wireless network using valid AD credentials.
2. Change the AD password.
3. Turn off the iWhatever (or just turn off the WiFi) and then turn it back on.
4. If you let it sit at the home screen, it will try to connect with the outdated AD credentials over and over. For some reason instead of REJECTING the connection attempt, NPS decides to discard it.
5. The Aruba controller rightfully doesn't see a response from NPS so it marks it as down.

Now if you try to use any data on the device at step #4 (for instance you load Safari or the Mail app) you *are* prompted for new credentials. One of my users incorrectly assumed it would be her Apple ID credentials and did not notify anyone when it didn't take her password so she gave up using data on my wireless network. Unfortunately the device would try to join the wireless network over and over every morning when she brought it in.

I am still confused as to why NPS is just discarding the request instead of answering one way or another. I do not know if the finger should be pointed at Apple or Microsoft. If the iWhatever actually prompted for credentials after it failed to join a network x number of times I think I wouldn't be having this issue...

I have the same situation where the NPS is discarding EAP silently and causing the controller to mark the server as down when there has been no 3x10 reply. Has anyone found out why no reply is given? I have Googled this one to exhaustion.

Was just wondering if anyone got a resolution to this issue? Seeing something similar...

We have found in our environment that it is due to NPS 2008 silently discarding non PEAP authentication requests. The logs of the NPS server show:

Authentication Details:

                Connection Request Policy Name:           1-Secure Wireless Connections Aruba

                Network Policy Name:                   Secure Wireless Connections Aruba London

                Authentication Provider:                              Windows

                Authentication Server:                  MISRAD1.xxxx.domain-name.com

                Authentication Type:                     EAP

                EAP Type:                                            -

                Account Session Identifier:                          -

                Reason Code:                                    1

                Reason:                                                                An internal error occurred. Check the system event log for additional information.

A PEAP requiest shows:

Authentication Details:

            Connection Request Policy Name:          1-Secure Wireless Connections Aruba

            Network Policy Name:                     Secure Wireless Connections Aruba London

            Authentication Provider:                 Windows

            Authentication Server:                    MISRAD1.xxxx.domain-name.com

            Authentication Type:                       PEAP

            EAP Type:                             Microsoft: Secured password (EAP-MSCHAP v2)

            Account Session Identifier:                        -

Quarantine Information:

            Result:                                               Full Access

            Extended-Result:                             -

            Session Identifier:                            -

            Help URL:                             -

            System Health Validator Result(s):

                  -

The server admins are still working with Microsoft to ascertain why it is not rejecting the request as opposed to just discarding the request. A WS capture on the controller will show the 3x10 rules and timeout. This is mainly caused by BYOD clients that are not policy enforced. I have also had lengthy conversations with an Aruba TAC engineer about this. The controller will mark any server down on a 3x10 rule *even* if there is other radius traffic passing (request/challenge/approve/reject) which to me does not make sense. Apprently this has been the source of some debate within Aruba.

What has made matters worse is that in 6.3.1.5 SNMP has been updated to send these traps out. I have since disabled them:

wlsxAuthServerReqTimedOut                  Yes           Disabled

wlsxNAuthServerTimedOut                    Yes           Disabled

....and also set my dead timers to 0

Global User idle timeout = 15300 seconds
Auth Server dead time = 0 minutes
Logon user lifetime = 5 minutes
User Interim stats frequency = 300 seconds

It's not ideal, but stops the reporting and automatic ticket generation.

The Radius RFS states:

http://www.ietf.org/rfc/rfc3579.txt

“On receiving a valid Access-Request packet containing EAP-Message

attribute(s), a RADIUS server compliant with this specification and

wishing to authenticate with EAP MUST respond with an

Access-Challenge packet containing EAP-Message attribute(s).  If the

RADIUS server does not support EAP or does not wish to authenticate

with EAP, it MUST respond with an Access-Reject.”

We continue to work with Microsoft.

Thanks for the detailed update!

Keep us posted. :)

More often than not, it is also a Blackberry that does it.

We're seeing the same issues, and some time has passed on this ticket.  Were you ever able to come up with a better resolution?

Great thread, good details.

Just FYI, we're using Server 2012 NPS and seeing the same thing.

Is this causing your users AD accounts to be locked out or is the request discarded before it's processed to the domain?

I remember with older NPS servers we had this issue and could resolve this by using NPS lockout method described here:

http://technet.microsoft.com/en-us/library/dd197529(v=ws.10).aspx

With reset timers etc timed correctly with regards to your GPO of failed authentication attempts and lockout policy you'll achieve a state where the accounts can get "NPS locked out" but not domain locked. This is useful in some situations for example if you're wired or machine authenticated with your primary workstation and only use PEAP with user accounts with BYOD devices.

Not sure if this applies to your problems but might be worth a shot.

Hi - No, we don't get user account lockout, just local controllers flipping back and forth between NPS servers when the 3x10 timeout is reached. Supposedly the high timeout counter issue is resolved in 6.4, we are running 6.3.1.9. I was informed yesterday that Microsoft want to make some more tests. So, not out of the woods yet. It gets quite frustrating that as technically I am a user of an Aruba product, I am having to act as a mediator for Aruba and Microsoft when really I think the two companies should just be working together rather than letting the customer undertake all the work to resolve their issue.

Has there been any update to this? I just came into this issue with a customer that deployed a local controller (7010) into an exsiting master-local architecture. The other controllers (3600's) never had an issue with the NPS that was deployed on Server 2012. It wasn't until users at the new location with the 7010 were there any authentication request time outs. 

If anyone has any insight, I would appreciate it. Thanks!!

Shane

Any updates on this topic as we are seeing this as well with NPS running on Windows Server 2008.

The user in the post before deployed a controller with much more capacity than the first, so it is possible that his radius server is overburdened; either the cpu, the storage or the network.  If you have timeouts, you can assume the same.  Please open a case so that they can look into your specific situation to make sure there is nothing configured on the Aruba controller that would make this worse..  There is no magic bullet to deal with server timeouts.

How can EAP termination be done on the controller  ?

---

@mahesh_shirke wrote:

How can EAP termination be done on the controller  ?

---

Mahesh_Shirke,

Please search the knowledgebase here: http://support.arubanetworks.com/KNOWLEDGEBASE/tabid/133/Default.aspx before posting in the forum.  It will save you a great deal of time.

NPS is running on a DC that I installed to handle radius requests. When NPS services are offline it runs somewhere between 0%-1% utilization. When NPS is running it doesn't go above 10% utilization - with the exception of the occasional spike hear and there. Airwave reports 3400 clients

I've sniffed the traffic hitting the primary NPS box, and I'm guesstimating that it's getting about 300 or so requests per second.

>Are your clients complaining?

Yes.. that's why I started an Airwave eval. I'm getting reports of sporadic authentication issues... like when a teacher starts up a class set of laptops. Out of 30ish devices 2-3 of them wont get online on the first attempt. This is true of both our Chomebooks and Win7 Laptops.

>Do you have a lot of Apple/iOS devices?

Yes we do. The bulk of them are personal devices.

I've been hitting the internet pretty hard looking for answers about NPS performance. So far as I've read a single NPS server can handle 200 requests per second and/or 5000 wireless devices. I'm having a hard time believing this when my smallest site is having issues with 144 wireless devices.