Higher Education

Reply
This is an open group. Sign in and click the "Join Group" button to become a group member and start posting.
Highlighted
Frequent Contributor I
Posts: 96
Registered: ‎04-09-2007
MacBooks - wpa2-eap-peap reauth delays...

oh my!

 

I've come accross an odd issue with [appears to be newer] macbooks and how they reauthenticate to a  wpa2-aes-peap-mschapv2 networks.... ie roam to a new bssid.   I first noticed this with a colleagues new macbook-retina running 10.8.4 - his desk is located righly right between 2 AP's and likely on the edge for eithers 5Ghz - in anycase it would not be unusal for his macbook to roam - without actually moving.    He would experience apparent random disconnects from the network - macbook would be stuck in the authenticating state - toggle wifi off/on and he'd be back online.... until it happened again.
 

I come to find out that - specifically when reauthneticating - I'll see a 10-20second delay in response from the macbook to the TLS exchange for setting up the PEAP tunnel.  (using wiresharp to monitor eap on the macbook)   the client starts the TLS handshake with a Client Hello - the controller responds with a Server Hello - then the macbook is silent for 10-20 seconds before responding with certificate, client key exchange, change ciper etc.... then authentication proceeds normally.

 

on 6.1.3.2 and 6.3.0.0 that 10-20 delay is too long for the default timers on the aruba controller.  extending the ir request timeout to 30 - give enough delay for the macbook to do whatever it needs to.... but still that's 10-20 seconds of network outage - just for roaming?!  Is anyone else seeing this?   It does appear to be related to the certificate in use.   I currentlyam using a cert via incommon.org an intermediate for addtrust.com - and its based on a 2048 bit key.   If i use my expired 1024bit based geotrust cert... I do not see this delay - also see this delay with aruba's selfsigned 2048 bit key on 6.3   I do note with my current key - the TLS Server Hello is large so fragmented into 3 packets to accomadate 2048 bit certs and intermediates etc...

 

is this just an issue that macos does not like 2048 bit certs or perhaps fragmented server hellos? (though on initial wifi turn up - this delay is not seen and the same certificate is in use).

 

This can be replicated at will, by issueing a "aaa user delete mac <macbook>" on the controller - and captureing the eap traffic via wireshark on the macbook when it attempts to rejoin the network it was just kicked from.

 

I see this for our eduroam ssid as well - so for networks where peap is terminated on the controller and ones that it isn't

 

ssid's are wpa2 with aes.   using eap-peap with mschapv2 for authentication.

 

Anyone else see this?  perhaps I can just tweak how I present my certificates - don't relish changing back to another CA for 1024 bit certs  (if I could find a CA still issuing those)

 

Travis

MVP
Posts: 501
Registered: ‎04-03-2007
Re: MacBooks - wpa2-eap-peap reauth delays...
This is insanely interesting. I've observed this via user complaints as well as with my own MacBook. Everything is exactly as you describe. I know 10.8.5 is on the horizon and rumors are there are wifi fixes. I hope you can verify that against your tests.

Yeah, not sure if its certificate related since its used on initial connect too (as you pointed out). Is there the same amount of fragmenting on initial connect as there is during the roaming event?

And the aaa dot1x timer...you say increasing allows for an eventual connect without bouncing the radio? If so, to which timer specifically are you referring?
==========
Ryan Holland, ACDX #1 ACMX #1
The Ohio State University
Frequent Contributor I
Posts: 96
Registered: ‎04-09-2007
Re: MacBooks - wpa2-eap-peap reauth delays...

initial connect and and a re-auth appear the same - just that added delay between the server hello and the macbook responding with the encrypted handshake message.

 

In my 802.1x auth provide I changed the "Interval between Identity Requests" from the default of 5 to 30 seconds.

!

aaa authentication dot1x "<profile-name>"
  timer idrequest_period 30

!

 

 

 

 

 

If anyone's interested here's what I see capturing eap packets on the macbook itself:

### inline comments highlighted the delay seen

### I start the capture - then on the controller "aaa user delete mac 28:cf:e9:1a:83:09"

1 2013-08-07 10:27:00.887552000 00:1a:1e:14:a1:f4 28:cf:e9:1a:83:09 EAP Request, Identity [RFC3748]
2 2013-08-07 10:27:00.887832000 28:cf:e9:1a:83:09 00:1a:1e:14:a1:f4 EAP Response, Identity [RFC3748]
3 2013-08-07 10:27:00.889222000 00:1a:1e:14:a1:f4 28:cf:e9:1a:83:09 EAP Request, PEAP [Palekar]
4 2013-08-07 10:27:00.890602000 28:cf:e9:1a:83:09 00:1a:1e:14:a1:f4 TLSv1 Client Hello
5 2013-08-07 10:27:00.891929000 00:1a:1e:14:a1:f4 28:cf:e9:1a:83:09 TLSv1 Server Hello, Certificate, Certificate Request, Server Hello Done
6 2013-08-07 10:27:00.892440000 28:cf:e9:1a:83:09 00:1a:1e:14:a1:f4 EAP Response, PEAP [Palekar]
7 2013-08-07 10:27:00.893285000 00:1a:1e:14:a1:f4 28:cf:e9:1a:83:09 TLSv1 Server Hello, Certificate, Certificate Request, Server Hello Done
8 2013-08-07 10:27:00.893694000 28:cf:e9:1a:83:09 00:1a:1e:14:a1:f4 EAP Response, PEAP [Palekar]
9 2013-08-07 10:27:00.894490000 00:1a:1e:14:a1:f4 28:cf:e9:1a:83:09 TLSv1 Server Hello, Certificate, Certificate Request, Server Hello Done

###notice the delay here - particualy bad 20 seconds before macbook responds
10 2013-08-07 10:27:20.913454000 28:cf:e9:1a:83:09 00:1a:1e:14:a1:f4 TLSv1 Certificate, Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message
11 2013-08-07 10:27:20.921292000 00:1a:1e:14:a1:f4 28:cf:e9:1a:83:09 TLSv1 Change Cipher Spec, Encrypted Handshake Message
12 2013-08-07 10:27:20.922331000 28:cf:e9:1a:83:09 00:1a:1e:14:a1:f4 EAP Response, PEAP [Palekar]
13 2013-08-07 10:27:20.923792000 00:1a:1e:14:a1:f4 28:cf:e9:1a:83:09 TLSv1 Application Data
14 2013-08-07 10:27:20.923967000 28:cf:e9:1a:83:09 00:1a:1e:14:a1:f4 TLSv1 Application Data
15 2013-08-07 10:27:20.925785000 00:1a:1e:14:a1:f4 28:cf:e9:1a:83:09 TLSv1 Application Data
16 2013-08-07 10:27:20.925988000 28:cf:e9:1a:83:09 00:1a:1e:14:a1:f4 TLSv1 Application Data
17 2013-08-07 10:27:20.971770000 00:1a:1e:14:a1:f4 28:cf:e9:1a:83:09 TLSv1 Application Data
18 2013-08-07 10:27:20.972316000 28:cf:e9:1a:83:09 00:1a:1e:14:a1:f4 TLSv1 Application Data
19 2013-08-07 10:27:20.978457000 00:1a:1e:14:a1:f4 28:cf:e9:1a:83:09 TLSv1 Application Data
20 2013-08-07 10:27:20.978710000 28:cf:e9:1a:83:09 00:1a:1e:14:a1:f4 TLSv1 Application Data
21 2013-08-07 10:27:20.980302000 00:1a:1e:14:a1:f4 28:cf:e9:1a:83:09 EAP Success

### so I do get connected.... no I toggle the macbooks wifi off and back on:
22 2013-08-07 10:27:48.763879000 00:1a:1e:14:a1:f4 28:cf:e9:1a:83:09 EAP Request, Identity [RFC3748]
23 2013-08-07 10:27:48.769827000 28:cf:e9:1a:83:09 00:1a:1e:14:a1:f4 EAP Response, Identity [RFC3748]
24 2013-08-07 10:27:48.771249000 00:1a:1e:14:a1:f4 28:cf:e9:1a:83:09 EAP Request, PEAP [Palekar]
25 2013-08-07 10:27:48.772942000 28:cf:e9:1a:83:09 00:1a:1e:14:a1:f4 TLSv1 Client Hello
26 2013-08-07 10:27:48.774251000 00:1a:1e:14:a1:f4 28:cf:e9:1a:83:09 TLSv1 Server Hello, Certificate, Certificate Request, Server Hello Done
27 2013-08-07 10:27:48.774718000 28:cf:e9:1a:83:09 00:1a:1e:14:a1:f4 EAP Response, PEAP [Palekar]
28 2013-08-07 10:27:48.775540000 00:1a:1e:14:a1:f4 28:cf:e9:1a:83:09 TLSv1 Server Hello, Certificate, Certificate Request, Server Hello Done
29 2013-08-07 10:27:48.775877000 28:cf:e9:1a:83:09 00:1a:1e:14:a1:f4 EAP Response, PEAP [Palekar]
30 2013-08-07 10:27:48.776672000 00:1a:1e:14:a1:f4 28:cf:e9:1a:83:09 TLSv1 Server Hello, Certificate, Certificate Request, Server Hello Done

#no delay here! transaction appears the same....
31 2013-08-07 10:27:48.800952000 28:cf:e9:1a:83:09 00:1a:1e:14:a1:f4 TLSv1 Certificate, Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message
32 2013-08-07 10:27:48.807489000 00:1a:1e:14:a1:f4 28:cf:e9:1a:83:09 TLSv1 Change Cipher Spec, Encrypted Handshake Message
33 2013-08-07 10:27:48.808353000 28:cf:e9:1a:83:09 00:1a:1e:14:a1:f4 EAP Response, PEAP [Palekar]
34 2013-08-07 10:27:48.809816000 00:1a:1e:14:a1:f4 28:cf:e9:1a:83:09 TLSv1 Application Data
35 2013-08-07 10:27:48.809980000 28:cf:e9:1a:83:09 00:1a:1e:14:a1:f4 TLSv1 Application Data
36 2013-08-07 10:27:48.811567000 00:1a:1e:14:a1:f4 28:cf:e9:1a:83:09 TLSv1 Application Data
37 2013-08-07 10:27:48.811928000 28:cf:e9:1a:83:09 00:1a:1e:14:a1:f4 TLSv1 Application Data
38 2013-08-07 10:27:48.853859000 00:1a:1e:14:a1:f4 28:cf:e9:1a:83:09 TLSv1 Application Data
39 2013-08-07 10:27:48.854216000 28:cf:e9:1a:83:09 00:1a:1e:14:a1:f4 TLSv1 Application Data
40 2013-08-07 10:27:48.856809000 00:1a:1e:14:a1:f4 28:cf:e9:1a:83:09 TLSv1 Application Data
41 2013-08-07 10:27:48.856991000 28:cf:e9:1a:83:09 00:1a:1e:14:a1:f4 TLSv1 Application Data
42 2013-08-07 10:27:48.858440000 00:1a:1e:14:a1:f4 28:cf:e9:1a:83:09 EAP Success

Regular Contributor I
Posts: 241
Registered: ‎04-03-2007
Re: MacBooks - wpa2-eap-peap reauth delays...

HI Travis,

 

Curious, what is the controller model you are using, M3 or 72xx? And what are the models of APs?

 

I'm wondering if 2048 bit certs are causing a lag because of the crypto engine peformance. We also use an incommon.org cert with an intermediate for addtrust.com. I recall it took significantly longer creating the 2048 CSR over a 1024 CSR. We use EAP-TTLS/PAP..

 

Mike

Frequent Contributor I
Posts: 96
Registered: ‎04-09-2007
Re: MacBooks - wpa2-eap-peap reauth delays...

We are using M3 controllers - and this behavior has been seen with AP125, AP105, AP135 - even RAP5's

 

I'm sure the 2048 certs have more overhead - but the odd part is that from a state where the wifi was off then turned on - the authentication has no delay - and similar for other devices - ie win7 laptop - its only when the macbook is rejoining an ssid that this pause is seen.

 

Travis

Contributor II
Posts: 49
Registered: ‎04-09-2010
Re: MacBooks - wpa2-eap-peap reauth delays...

We're using 2048 bit certificates as well and we've been experiencing this problem around our campus on Mac's only. 

 

Any workaround you guys know of?

Frequent Contributor I
Posts: 96
Registered: ‎04-09-2007
Re: MacBooks - wpa2-eap-peap reauth delays...

I did contact Apple about this - they did acknowledge that it appears to be an issue in macOSx - that's causing a cert validation delay.  So it could be possible to tweak cert settings/trusts on macos itself to mitigate issue - but a special config on each client is not something I see as a viable long-term solution - so I have not really tested that angle.   So far - just tweaking the timers as listed above - has allowed the macbooks to self-recover - which is a big plus... but there's still a signifigant break in connectivity when roaming.

 

I need to check up to see if Apple has any update.   We don't actually have any support contact with apple - so I haven't been pushing them too hard -  been monitoring for system updates - haven't seen anything to specifically address this and most recent patches have not altered behavior.   Apple was interested in having me test Mavericks-beta to see if issue persisted... but I'm not an active developer member....

 

Please feel free to contact Apple and reference my case (#  480081631) - just to keep the pressure on!

 

And if anyone else reading this thread has found any better work-arounds/mitigations, please share!

 

 

Contributor II
Posts: 49
Registered: ‎04-09-2010
Re: MacBooks - wpa2-eap-peap reauth delays...

Setting the timer to 30 seconds works, to a point. They still end up losing connection for a time which disrupts Airplay or any other constant connection program. It seems to happen when they get bounced from one AP to another as well. 

 

Doing testing revealed this happening on 10.8.4 and 10.8.5. Near consistent 22 second time to get authenticated. Hence the 30 second window for re-auth working. 

Frequent Contributor I
Posts: 96
Registered: ‎04-09-2007
Re: MacBooks - wpa2-eap-peap reauth delays...

I've found that on macosx if you go into the keychain and for the cert used for 802.1x auth - if you modify the trust settings to also always trust for SSL - it eliminates the delay I've been seeing when a macbook re-auths to the ssid.

 

The cert gets set to always trust for eap and x.509 when you first trust it....  guessing the apple bug has something to do with 2048 bit cert and not seeing this as an eap transaction...etc... but at the core its an ssl transaction - so explicitly trusting the cert for ssl - makes validation really quick :)

 

So far at least for my test station... Not a great solution as the users need to update settings... and obscure security certificate ones at that.... but should ease the pain for users that are willing to attempt - need to see if I can get any of mine to try this.

 

 

Guru Elite
Posts: 8,637
Registered: ‎09-08-2010
Re: MacBooks - wpa2-eap-peap reauth delays...
If you use QuickConnect for supplicant configuration, I believe the
profile will trigger the root CA to be trusted for the connection in the
key chain.

You have to include the root CA cert in the config profile and select it as
trusted under the PEAP options.

I assume it would be the same with other products like XpressConnect.

Tim Cappalli | Aruba Security TME
@timcappalli | timcappalli.me | ACMX #367 / ACCX #480
Search Airheads
Showing results for 
Search instead for 
Did you mean: