Security

Hi all. We have a case open, but I figured I'd check for ideas here as well.

There's a longer story behind this, but the short version is, the config on our controllers was overwritten by a VERY old config which made wireless inoperable. We restored to a config from about a week and a half ago, and everything now works, except 802.1x, even though things were fine a week and a half ago, and prior. 

We took a packet capture on our NPS and only saw RADIUS requests and challenges. No accepts or denies. We have about 3 hours under our belt with Support, and so far they've come up with enabling termination on the controllers, which appears to be working for now, except that's not how we had it configured before. So while we can do that, there's clearly an underlying issue that neither we, nor support were able to see right off the bat. 

Any additional thoughts or suggestions would be GREATLY appreciated.

amoreno,

You are probably in the best hands with support.  The information that you would have to give us publicly for us to narrow down what could have happened, I am assuming support already has.

The configuration overwrite in addition to restoring to an old backup makes this even more difficult.  If this is a production network, TAC has to tread lightly to get you back where you were before.

There are so many ways that this could break....Our guessing here would only be reviewing things that TAC has tried...just hours later.  Please keep us up to date with your progress here.

With that being said, you should see the traffic coming in in the eventviewer on your NPS.  There might be a message about an invalid radius client that you might see:  https://technet.microsoft.com/en-us/library/cc735406%28v=ws.10%29.aspx

cjoseph,

we'll definitely continue working with TAC on this. my supervisor and I just wanted to see if there was anything else that could be suggested, and possibly help us interprete what we're seeing, since TAC is focused on resolving the issue.

With termination disabled, show auth-tracebuf gave us the following (two different devices):

Feb 2 16:30:15 station-up * XX:XX:XX:XX:1c:6b 6c:f3:7f:XX:XX:92 - - wpa2 aes
Feb 2 16:30:15 eap-id-req <- XX:XX:XX:XX:1c:6b 6c:f3:7f:XX:XX:92 1 5
Feb 2 16:30:15 eap-id-resp -> XX:XX:XX:XX:1c:6b 6c:f3:7f:XX:XX:92 1 14 E00###689
Feb 2 16:30:15 rad-req -> XX:XX:XX:XX:1c:6b 6c:f3:7f:XX:XX:92 251 199
Feb 2 16:30:15 rad-resp <- XX:XX:XX:XX:1c:6b 6c:f3:7f:XX:XX:92/NPS_server 251 90
Feb 2 16:30:15 eap-req <- XX:XX:XX:XX:1c:6b 6c:f3:7f:XX:XX:92 2 6
Feb 2 16:30:15 eap-resp -> XX:XX:XX:XX:1c:6b 6c:f3:7f:XX:XX:92 2 152
Feb 2 16:30:15 station-down * XX:XX:XX:XX:1c:6b 6c:f3:7f:XX:XX:92 - -

Feb 2 16:31:23 eap-start -> XX:XX:XX:XX:d5:b9 6c:f3:7f:XX:XX:f0 - -
Feb 2 16:31:23 eap-id-req <- XX:XX:XX:XX:d5:b9 6c:f3:7f:XX:XX:f0 1 5
Feb 2 16:31:28 eap-id-req <- XX:XX:XX:XX:d5:b9 6c:f3:7f:XX:XX:f0 1 5
Feb 2 16:31:34 eap-id-req <- XX:XX:XX:XX:d5:b9 6c:f3:7f:XX:XX:f0 2 5
Feb 2 16:31:39 eap-id-req <- XX:XX:XX:XX:d5:b9 6c:f3:7f:XX:XX:f0 2 5
Feb 2 16:31:44 eap-id-req <- XX:XX:XX:XX:d5:b9 6c:f3:7f:XX:XX:f0 2 5
Feb 2 16:31:50 eap-id-req <- XX:XX:XX:XX:d5:b9 6c:f3:7f:XX:XX:f0 3 5
Feb 2 16:31:56 eap-id-req <- XX:XX:XX:XX:d5:b9 6c:f3:7f:XX:XX:f0 3 5
Feb 2 16:32:01 eap-id-req <- XX:XX:XX:XX:d5:b9 6c:f3:7f:XX:XX:f0 3 5
Feb 2 16:32:08 eap-id-req <- XX:XX:XX:XX:d5:b9 6c:f3:7f:XX:XX:f0 4 5
Feb 2 16:32:14 eap-id-req <- XX:XX:XX:XX:d5:b9 6c:f3:7f:XX:XX:f0 4 5
Feb 2 16:32:20 eap-id-req <- XX:XX:XX:XX:d5:b9 6c:f3:7f:XX:XX:f0 4 5
Feb 2 16:32:25 eap-id-req <- XX:XX:XX:XX:d5:b9 6c:f3:7f:XX:XX:f0 5 5
Feb 2 16:32:30 eap-id-req <- XX:XX:XX:XX:d5:b9 6c:f3:7f:XX:XX:f0 5 5
Feb 2 16:32:35 eap-id-req <- XX:XX:XX:XX:d5:b9 6c:f3:7f:XX:XX:f0 5 5
Feb 2 16:32:41 station-down * XX:XX:XX:XX:d5:b9 6c:f3:7f:XX:XX:f0 - -

Coupled with that are Access-Requests and Access-Challenges, but never anything else. We also searched for the Event ID 13 as suggested and didn't find it in Event Viewer. 

Also when doing a show log, there are tons of the following for many users:

"Maximum number of retries was attempted for station [User Name] [station MAC] [AP MAC], authenticating the station"

"Dropping the radius packet for Station [station MAC] [AP MAC] doing 802.1x"

"Failed to send the radius request for Station [station MAC] [AP MAC]"

And an occasional "Dropping the radius packet for Station [station MAC] [AP MAC]"

We'll keep waiting for TAC's response, but just wanted to see if we could get some insight while we wait. If not, then not a big deal.

Thanks

Amoreno,

That is the controller's side of the issue.  Are you saying that you do not see any responses in the NPS eventviewer?  The Auth-tracebuf shows that the client is not responding in the second part...but...it could mean quite a few things...

On the NPS side, were there any changes made?  Was the certificate that is tied to the remote access policy changed on the NPS?  If not, you should just be able to point back to the same radius server and everything works.  Did you change any of the client trust parameters in group policy or on the physical client?  Did you try to just have a mobile client like an iphone connect?  Mobile clients are much more forgiving than windows clients and if they can connect, you can work from there.  If the mobile client cannot connect, you have bigger issues.....

The big question is, what caused this is the first place, and how can we roll back those changes?

cjoseph,

Before TAC was contacted, when termination was off, in Windows Logs > Security > we were seeing some Audit Successes and some Audit Failures, which is strange. So I guess I should say that 802.1x was working for some users, but not for others. We were also seeing a lot more Machine Auth attempts than we were before, which is strange since we don't do machine auth. 

No changes on NPS. Only change was on the Aruba controllers. We tested with an iMac, Windows 7 laptop, and three different Android phones. None of them worked with the three different accounts we tried for the three different user roles we have - Admin, Staff, and Student, even though aaa test-server authenticated them just fine from the controllers.

So the cause was me trying to schedule the controllers to reboot via Airwave to push out the 6.4.2.4 code, as was recommended by some users here. I put Airwave into read/write mode, and didn't anticipate pushing the old mismatched config immediately, which was a mistake. I thought it would just grant r/w access and then we could make changes later.

After I realized that wireless was broken, I put Airwave back in monitor only, and then started attempting to fix things manually. Captive portal wasn't working, but WPA2 and 802.1x was working. This was on Friday. Then on Sunday I remembered that when I uploaded the firmware to the standby partition, a save config was taken. So yesterday, I booted the master controller to that, but things seemed to be the same, at which point I booted to another config from 2 weeks ago. After which, CP and WPA2 worked, but now not .1x, and supposedly some, not all, RAP client's network ports.

So that's what happened.

Amoreno,

On the NPS, you should be looking in Eventviewer> Roles> NPS to see any activity there.  That is the spot where actionable events for NPS exist.

Domain devices that are at the ctrl-alt-delete screen will by default  try a machine authentication.  That is a client-side configuration...

The aaa test server only tests for username and password connectivity.  It does not test to see if the client trusts the Radius Server's certificate, etc.  You might want to uncheck "Validate server Certificate" on one of your clients to see if it works.  In addition, you should be able to test an i-device (ipad or iphone) to see if they work.  If they do not, find the corresponding event in the NPS log and expand it to get an idea what is really going on.  If username and password is working, theoretically and i-device should be working, because that is the least common denominator. 

On the NPS server, do you not see any authentication attempts for dot1x wireless at all?    Please make sure the server is set to log successes and failures:

auditpol /get /subcategory:"Network Policy Server"

System audit policy
Category/Subcategory                      Setting
Logon/Logoff
  Network Policy Server                   Success and Failure

If Failures are not set; run the following on the NPS server:

auditpol /set /subcategory:"Network Policy Server" /success:enable /failure:enable

The NPS logs may be your best bet on digging up the issue.

Doublecheck your NPS policies are set and that you hvae the proper server certificte defined under the PEAP settings of the policy.

I'm working with amoreno on this issue.

We've made no changes to our NPS configuration. Our cert is still valid, and our policies haven't changed and work properly. We went from working to immediately not working and the only change was to our Aruba controller config.

We are logging both rejected and successful authentication attempts, but the majority of requests do not get to that point. The Access-Request is received by NPS, NPS responds with an Access-Challenge which is then ignored. Because the request isn't accepted or rejected, it doesn't end up in event viewer at all.

Well,

Do a config Diff between a working and non-working config and see what is the issue.  If you cannot, your other alternative is to create a new config for a 802.1x WLAN from scratch and test that.  If that works, change the ESSID to the old, non-working WLAN.

There are tons of things that could have happened.

We created a new RADIUS server entry with 100% identical configuration to the existing server entry.

We replaced the old server with the new server in our existing RADIUS server group.

Everything works.

How can this be? 

Both servers are successful with "aaa test-server", but only one works with termination disabled.

Was that the only server in that server group?  If it was not, it might have been marked out of service.

It was the only server in the group.