Wireless Access

I've been struggling with an issue since upgrading our controller to v. 6.1.3.1 from 5.0.4.0 a few weeks ago. We have recently deployed a bunch of 2011 MacBook Airs running the latest version of OS X Lion (10.7.3). They appear to get disconnected from the wireless several times throughout the day, which resets their connections to our email system and Filemaker server which ultimitely causes them to lose work and get very frustrated that they have to log back into these systems.

 

I've done a ton of research on the Apple forums and on here and while many people reported issues with Lion and wireless, I haven't found anyone with issues while using Aruba specificially. The thing is, these clients all appeared to work fine before we upgraded to the v.6 code. I've been working with Aruba support for the past 2 weeks and they've gone into the controller and tweaked a bunch of settings in accordance to best practices, but nothing has seemed to help.

 

One thing I've notied is if I ping a client, the response times are all over the place unless they are downloading or doing something on the Internet. That makes it sound like the wireless card is going into power save mode, but there's no way to modify those settings on the Mac like you can on Windows. I've also set up a little Belkin router to test with and when I connect one of the clients to that wirelessly, the ping times are >1 ms, right where they should be. I've tried all of the suggestions on the client site including replacing kernel extensions for wireless with those from an old OS, but the ping times are still all over the place. Older Mac clients, running on 10.6.8, and our many iPads and iPhones are working just fine. I know it sounds like an Apple specific issue, but it's a combination of them and Aruba v.6. 

 

I've called Apple this morning and of course they are pointing the finger at the wireless. I've actually been on hold with them twice this mornig and gotten disconnected twice, so I'm at my wit's end! Any suggestions would be greatly appreciated! TIA!

 

I'm afraid I don't have an answer for you and haven't yet received such reports from Apple laptop users but I can relate.  We did the upgrade from 5.0 to 6.1.3.1 last week and it tripped up a few systems.  We had a number of Dell laptops running XP with earlier generation 802.11n dual-band chipsets that all needed wireless driver updates to work again.  Also we had one case of an iPhone 4 running iOS 4.2 that refused to stay connected to wireless until it was upgraded to the latest iOS 5 release.

 

If you have the spare controllers and APs around and are motivated enough I'd suggest comparing what an AP beacon looks like coming from an ArubaOS 5 and a 6.1.3.1 infrastructure and seeing what new functionality is being broadcast that could be triggering a power save bug, etc.  I've been able to resolve such issues here without resorting to that but it might be worth it for you.  Just an idea.

Thank you for the reply. It certainly sounds like a driver incompatibility issue here too. I do have some extra APs that I've tested with, but unfortunately not a spare controller. I set up a wide open SSID yesterday and tested a client on that and I see similair behavior. The Aruba tech wanted me to test this way to see if it was a TKIP issue as I guess that's a fairly common issue with Macs. I'm hoping 10.7.4 will be out soon with new drivers, but we'll see. I'm going to try caling Apple back again today too. :smileyindifferent:

I just wanted to bump this back up with an update and to see if anyone else has been experiencing this. I've gotten both Aruba and Apple to admit that this is a known issue and to look for an upcoming software update to help fix it, but that's been a month ago now. I've installed 10.7.4 and a beta of 10.7.5, even the 10.8 beta but I still see the same behavior on a client. I've also updated the controller to 6.1.3.2 but that didn't help either. Is anyone else seeing similar behavior and/or have any fixes/workarounds? TIA!

I can tell you that we have quite a few of the laptops and the version of Lion you mention running well:

 

What is your encryption?

How many clients are on your WLAN? (total)

Do the wireless clients share a subnet with wired clients?

 

Thanks for the quick reply! We are using WPA2 Personal encryption to join the network and MAC radius authentication from there using a Bradford Network Sentry. During the school year we are right around 400 clients, but during the summer it's just over 100 and we are seeing the same issues. We split students YOG into subnets and for employees, wired and wireless clients do share the same subnet. We only have a handful of wired employee clients though.

What is the known issue?  What happens and what is the sequnce of events?

 

They're really just getting crappy performance on the wireless network. If I ping a Lion client that's just sitting on my desk, the response times are all over the place and you will eventually see big chunks of timeouts. When there's about 15+ timeouts in a row, the clients will lose connection to their email client (FirstClass) and Filemaker. This can lead to them losing work and/or having to sign in multiple times a day and it drives them nuts. If the clients are actively downloading something from the Internet though, the ping times are consistently right at 1ms. It's something to do with the wireless card going into some sort of sleep mode, I'm guessing to save battery life.

 

Aruba has been able to replicate the issue in their test environment, but believes that the APs are working as they should and it's the clients that are not behaving correctly. The last email I recieved from support:

 

"We did some testing using the same clients and found out that the clients are sending the PS frame very frequently and then when AP’s are scanning other channel to find a better channel, that’s the time when the frame goes missing and AP thinks that client is still awake and keep sending data.

 

One of the option we tested is to disable Scanning to validate the above theory and it helped the situation, however Disabling Scanning will not be recommended since it means asking ARM not to perform.

 

So as of now the AP’s are behaving normal and scanning other channel for the defined scan time hence we cannot change the scan algorithm because it’s a huge change and might bring up another issue with the ARM functionality.

 

However, in order to understand the behavioral change with these new MacBook Version I am also checking on the possibility of providing some data to Apple support and get their views on the same. Will keep you updated on the same."

 

Apple said that they know about this issue with Aruba hardware and to look for a future OS release to fix it. 10.7.5 was supposed to be focused on network reliability and performance but the golden master beta doesn't appear to address it.

Are you dropping broadcast traffic on that Virtual AP?

 

Yes we are. I've had our VAR and the Aruba techs go over everything in the config as well and they say it all looks good.

Okay.  I will wait for someone else to comment, because I have not seen that..

 

Wow - anything further on this - I'm having the same issue with some of the macs I have here - this one is still running snow leopard however - just drops randomly, has issues actually getting connected to an ap. The interesting thing I see on the controller is that their connection time keeps incrementing when they disconnect - so the user will report getting disconnected 3 times in the last hour, but age of the connection is 4 hours. Also seeing it on PCs with Linux and Windows 7 - working with an aruba engineer  right now trying different drivers (for windows) to see if there is any change.

 

Right now I'm running my system fully wirelessly to see if I see the same issues - so far it's ok....

 

Lirria

---

@Lirria wrote:

Wow - anything further on this - I'm having the same issue with some of the macs I have here - this one is still running snow leopard however - just drops randomly, has issues actually getting connected to an ap. The interesting thing I see on the controller is that their connection time keeps incrementing when they disconnect - so the user will report getting disconnected 3 times in the last hour, but age of the connection is 4 hours. Also seeing it on PCs with Linux and Windows 7 - working with an aruba engineer  right now trying different drivers (for windows) to see if there is any change.

 

Right now I'm running my system fully wirelessly to see if I see the same issues - so far it's ok....

 

Lirria

---

Lirria,

 

We have to make sure that we are dealing with the same issue.  Lets us know the result of the troubleshooting.

 

 

Of course - it's just nice to know what resolved it for somebody else so it gives you things to look at to see if that is also another issue in your environment. We are waiting on the results from the lastest debug log review - hopefully there will be something in there.

 

thank you

 

Lirria

I wanted to let you know that we have a similar problem in our office.  10.6, 10.7, 10.8 and we get clients roaming from one AP to the Next and intermittently getting knocked off.  The Airport menulet shows the antenna signal animation moving up and down like it's trying to connect.  If you click on the Airport Menulet it will say Authenticating.  I have a ticket open with Aruba for over 2 months now, they have tried best practices fixes, Upgrading the OS on the 650 controller to the latest and still not resolved.  

 

We do not run any wireless PC's in our office to test with.  Seems to be an Aruba Mac issue?

 

 

I would have to say that since I worked with the engineers my issues have mostly resolved (or people just aren't reporting them) I have a total of 27 aps, 11 are local 16 are remote. All have a variety of OS's (Linux, Mac, Windows) and we are what I would classify as stable.  I usually spend 1 week a quarter running on nothing but wireless to make sure there are no issues. I also reboot the controller and all AP's monthly - hasn't seemed to cause any issues and thing seem to be running better - plus it clears out the bootstrap log so you can actually tell when you have an issue.

 

RIght at the moment I have 229 connections to the aps - yes some are passive, but some are active and are working well - I just completed a skype meeting with 2 other people in the same room - 2 of the 3 of us were on wireless with no issues.

 

Please continue working with the tech support group to get the issue resolved - they really can help.

 

Lirria

---

@DJMagicMelliott wrote:

I wanted to let you know that we have a similar problem in our office.  10.6, 10.7, 10.8 and we get clients roaming from one AP to the Next and intermittently getting knocked off.  The Airport menulet shows the antenna signal animation moving up and down like it's trying to connect.  If you click on the Airport Menulet it will say Authenticating.  I have a ticket open with Aruba for over 2 months now, they have tried best practices fixes, Upgrading the OS on the 650 controller to the latest and still not resolved.  

 

We do not run any wireless PC's in our office to test with.  Seems to be an Aruba Mac issue?

 

 

---

Do you know if you have OKC (Opportunistic Key Caching) on your 802.1x profile Unchecked? MACs do not support this and it definitely could cause issues with roaming.  Look under configuration> Security> Authentication> L2 Authentication> 802.1x profiles.  Find the one that corresponds to your WLAN and make sure OKC is unchecked under the Advanced tab.

 

Update on our situation:

 

We are using 802.1x. The certificate that's sent over as part of the 802.1x authentication process is signed by a CA, and the signing chain makes the cert long enough that it doesn't fit into a single packet. When OSX 10.8.2 Macs roamed between APs, they'd get this cert sent to them, the roam event would usually take upwards of 15 seconds, and potentially not work at all. Disabling OCSP and CRL checks on the Mac clients had no effect.

 

Switching to a self-signed cert, which does fit into a single packet, pretty much eliminated the roaming problem. We're still having other issues with Mac clients, but the self-signed cert fixed our most annoying and reproduceable issue.

Good tip - glad things are working better for you!

 

Lirria

We have started deploying 10.8.3 to our Mac clients, and things seem to be better. The drops/disconnects seem to come in waves now. It will be fine for a week or two, and then all of a sudden one day or night I'll get a bunch of reports of people not being able to connect, or getting dropped a bunch. It seems to be limited to areas where we still have some AP61s in production, but I've also gotten reports in areas where we have 105s deployed.

 

On a related note, I did upgrade a lot of areas from older AP61s to 105s recently, and some of the reports I've gotten indicate that the coverage isn't as good in some spots with the newer APs, mounted in the same locations that the 61s were. I know they probably don't have the same coverage pattern, but I wouldn't think it would be any worse! 

 

Not sure if you are doing this but one of the things that what suggested by one of the Aruba engineers was to enable validate-pmkid when you have OKC enabled under the 802.1x profile since the Mac devices don't work properly when roaming if that is not enabled

 

Please read this post :

http://community.arubanetworks.com/t5/Mobile-Device-Configuration-and/MAC-WLAN-Client-quot-Nuances-quot/td-p/1461/page/2

 

 

That is currently enabled and OKC is disabled so it probably doesn't even matter, right?

 

OKC is good for the other devices (Non - Mac) since it speeds up the roaming process and doesn't have to do the whole full 802.1x process everytime the device roams within the ESS .

 

 

We are about 90-95% Mac/iPad so I'm not to worried about the few Windoze clients out there! :smileywink:

 

 

Good deal

We were having a terrible time with our Mac clients roaming to different access points, even when the computers were sitting still, and dropping connections to our email server. After a lot of troubleshooting, the technician we were working with suggested running the following command on the Macs:

 

sudo defaults write /Library/Preferences/com.apple.airport.opproam disabled -bool true

 

The difference has been like night and day. It stops the Macs from aggressively roaming, which in turn stops 802.1X re-authentication and dropped connections. The clients that we've done this to have reported that their connections now rarely (if ever) are dropped while sitting still. If you've got Mac clients, are using 802.1X, and are dropping connections frequently, I think it's worth looking into.

Thank you very much for sharing, I'm going to try it out on a few clients as we still see disconnects from time to time!

We are seeing the same issue occur on our RAP environment (6.1.3.0) -- our production wirless is 3.4.2.4/5.0.4.2 so I do not know how this fares in the office. From what I have been told, this is only affecting Macs. After a random period of time thier connection just "dies". They can see other SSIDs available, but not the one they are currently connected to in airport. To fix they must either connect to a different SSID and connect back, or run the Apple wireless network repair tool (can't recall the actual name of the utility). I have an open case with TAC but have not gotten anywhere as of yet.

i also encountered same issue where is wifi connection suddenly drop and very difficulties to connect. i've reported and login case to Aruba, and the Aruba Engineer has did a remote site support to our controller to solve this issue. But unfortunately that my problem still remain and couldnt be solve by Aruba Engineer. and the Aruba engineer told me that this issue is come from Apple instead of Aruba.

 

what is a good answer and solution from Aruba.

Although we are still monitoring the issue and do not have a resolution, we have noticed fewer complaints from the Mac users that have been upgraded to lion (not mountain lion) We will continue to work with Aruba to find a solution. Lirria

Unfortuantely, Macs have had connectivity issues for some time. I've experienced this myself as a Mac only user. 

 

I have noticed that the last Lion update and going into Mountain Lion that there are issue with the Keychain. I've found myself having to manually delete certs and passwords and then reauth to get connected.

 

This is a know issue and not an issue with Aruba. The finger should be pointed at Apple.

After spending 3 weeks monitoring the macs (win7 and linux too) on the network we found a couple of things.

 

1. Upgrading the OS to Lion (not mountain lion yet) improved connectivity.

2. Changing the settings for 802.1x for Delay between WPA/WPA2 Unicast Key and Group Key Exchange and Delay between EAP-Success and WPA2 Unicast Key Exchange which apparently have to deal with Macs.

 

Between those 2 things our one Mac user that had been dropping 6-7 times a day has dropped once in 7 days, and added bonus has been the linux systems are also staying connected better.

 

Working with the support staff was time consuming but they did find a least some config settings that had to be changed to make things better.

 

Definitely think Apple is not providing a current driver for snow leopard and that is part of the issue.

 

Lirria

have you tried diabling IPv6?  This can only done at the terminal.

 

>networksetup -setv6off Wi-Fi

 

 

We normally disable ip v6 on all systems (network and desktops)  Not the best long term solution - but it keeps many of the issues down we have seen. The Macs and Linux systems definately are more likely to have issues with ip v6 enabled.

 

Lirria

Wow - sorry I haven't checked this thread in a while, I thought I was getting email alerts when there were responses, but I guess not.

 

Lirria - we are definitely not having the same issue if Lion fixes it for you. We have Snow Leopard clients working flawlessly, but it's ONLY our Lion and Mountain Lion computers that are having issues. Unfortunately most of our production machines shipped with Lion so we have to deal with it.

 

We haven't seen any improvements in performance. I had a user just today say that they dropped their connection 10 times! Aruba says that Apple knows of the issue and is working on a fix in a future release, but 10.7.5 and 10.8.2 haven't fixed it yet. I'm not so sure that a 10.7.6 is even going to come out!

Sorry to hear that - hopefully it will get all worked out for you. I know with Lion things did get better which does point to something in the OS (most likely driver based) that's the issue. Hopefully some patches will fix the issue.

 

Best of luck!

 

Lirria

Interesting read, we've upgraded from 5.0.3.x to 6.1.3.2 and are now starting to receive reports of the same type as reported earlier in this thread.  Interesting as we are also getting more reports from Apple-based devices than others.

 

We were running pretty stable on the 5.x codetrain and only reason we were going to 6.x was to support the AP-135's that were to be deployed at the end of the year.

 

Looks like we might have to roll-back to 5.x code train...

---

@jasonmathew1 wrote:

Interesting read, we've upgraded from 5.0.3.x to 6.1.3.2 and are now starting to receive reports of the same type as reported earlier in this thread.  Interesting as we are also getting more reports from Apple-based devices than others.

 

We were running pretty stable on the 5.x codetrain and only reason we were going to 6.x was to support the AP-135's that were to be deployed at the end of the year.

 

Looks like we might have to roll-back to 5.x code train...

---

Jasonmatthew1,

 

You should open a TAC case to ensure that your specific issue is being troubleshot....  Please do not assume you have the same issue.  Let's see what we can do for your situation.

 

---

@jasonmathew1 wrote:

Interesting read, we've upgraded from 5.0.3.x to 6.1.3.2 and are now starting to receive reports of the same type as reported earlier in this thread.  Interesting as we are also getting more reports from Apple-based devices than others.

 

We were running pretty stable on the 5.x codetrain and only reason we were going to 6.x was to support the AP-135's that were to be deployed at the end of the year.

 

Looks like we might have to roll-back to 5.x code train...

---

Do you have any more specific information about your issue? Right now we're being told from Aruba that they've talked to engineers at Apple and they have confirmed that it's an issue with some Broadcom wifi cards and will be working on a fix in a future firmware release. I haven't heard a peep about 10.7.6 coming out any time soon, so I think we are SOL. Meanwhile we still have tons of clients complaining about poor wifi performance. :smileyfrustrated:

This sounds incredibly similar to the issue I am facing.

 

Mac laptops affected more than any other device. (iPads seem fine. Windows is fine.) 10.6 not really affected, 10.7 and10.8 are very bad. Users report random disconnects while sitting at their desks. Mac roaming performance is generally poor, with 15+ seconds of downtime when switching between APs.

 

 

Running ArubaOS 6.1.3.3 and 135 APs.

 

Hello rmcleod

 

Hid you ever get a resolution. I have been battling this exact issue for over a year. Only our Mac clients are experiencing WiFi issues and it causes Firstclass email to disconnect

Hi Brad,

Bill's command to disable roaming has been the fix for us. As long as the clients have strong signal strength, and we run that command on their laptops, they are no longer getting disconnected all the time. I've also been setting the MTU to 1493 as I read that somewhere else, but not sure if it's helping much.

We're experiencing this same issue. Only certain users. Tried disabling Client Match and Band Steering, but some Mac clients just keep getting dropped from Firstclass again and again. 

 

Tried running the command mentioned in this thread, we'll see if it helps. 

theiglu,

 

Have you opened a case to see if there is anything underlying that could be causing your issue?

 

Yes I have. We disabled Client Match, but beyond that there isn't much they can do. I can't reproduce the issue for them and my users are very hard to get any feedback from. I actually just closed the case this morning after but then heard about more users experiencing the problem. 

 

So far the debug logs show nothing that correlates to the drops for particular users. The timestamps are hard to go by since the drops usually don't appear until a few minutes after Firstclass loses connection. It sits there and doesn't say it has disconnected for a while until it errors out, sometimes this can go on for 4-5 minutes. 

I did notice that OKC was enabled on our 802.1X profile. I have disabled it for now since the majority of our faculty are on OSX devices. 

theiglu,

 

That is a big one.

Just had a device roam to another AP while she was sitting at her desk. This is a user that I just this afternoon cleaned her entire Wifi settings and performed the roaming command mentioned earlier. Just heard back from her that is occurring again. Here is the debug from the last few minutes, MDNS stuff excluded:

 

Apr 22 15:27:46 :501109: <NOTI> |AP JRCAC-203@10.10.161.70 stm| Auth request: a8:bb:cf:04:0d:98: AP 10.10.161.70-00:24:6c:80:84:d9-JRCAC-203 auth_alg 0
Apr 22 15:27:46 :501093: <NOTI> |AP JRCAC-203@10.10.161.70 stm| Auth success: a8:bb:cf:04:0d:98: AP 10.10.161.70-00:24:6c:80:84:d9-JRCAC-203
Apr 22 15:27:46 :501109: <NOTI> |AP JRCAC_202@10.10.161.87 stm| Auth request: a8:bb:cf:04:0d:98: AP 10.10.161.87-00:1a:1e:89:fa:e1-JRCAC_202 auth_alg 0
Apr 22 15:27:46 :501093: <NOTI> |AP JRCAC_202@10.10.161.87 stm| Auth success: a8:bb:cf:04:0d:98: AP 10.10.161.87-00:1a:1e:89:fa:e1-JRCAC_202
Apr 22 15:27:46 :501095: <NOTI> |stm| Assoc request @ 15:27:46.565546: a8:bb:cf:04:0d:98 (SN 79): AP 10.10.161.87-00:1a:1e:89:fa:e1-JRCAC_202
Apr 22 15:27:46 :522036: <INFO> |authmgr| MAC=a8:bb:cf:04:0d:98 Station DN: BSSID=00:24:6c:80:84:d9 ESSID=Academic8021X VLAN=75 AP-name=JRCAC-203
Apr 22 15:27:46 :522234: <DBUG> |authmgr| Setting idle timer for user a8:bb:cf:04:0d:98 to 300 seconds (idle timeout: 300 ageout: 0).
Apr 22 15:27:46 :522152: <DBUG> |authmgr| station free: bssid=00:24:6c:80:84:d9, @=0x0x1570e2c.
Apr 22 15:27:46 :501095: <NOTI> |AP JRCAC_202@10.10.161.87 stm| Assoc request @ 15:27:51.739606: a8:bb:cf:04:0d:98 (SN 79): AP 10.10.161.87-00:1a:1e:89:fa:e1-JRCAC_202
Apr 22 15:27:46 :501100: <NOTI> |AP JRCAC_202@10.10.161.87 stm| Assoc success @ 15:27:51.741375: a8:bb:cf:04:0d:98: AP 10.10.161.87-00:1a:1e:89:fa:e1-JRCAC_202
Apr 22 15:27:46 :501080: <NOTI> |stm| Deauth to sta: a8:bb:cf:04:0d:98: Ageout AP 10.10.161.70-00:24:6c:80:84:d9-JRCAC-203 STA has roamed to another AP
Apr 22 15:27:46 :501100: <NOTI> |stm| Assoc success @ 15:27:46.569271: a8:bb:cf:04:0d:98: AP 10.10.161.87-00:1a:1e:89:fa:e1-JRCAC_202
Apr 22 15:27:46 :522035: <INFO> |authmgr| MAC=a8:bb:cf:04:0d:98 Station UP: BSSID=00:1a:1e:89:fa:e1 ESSID=Academic8021X VLAN=75 AP-name=JRCAC_202
Apr 22 15:27:46 :522077: <DBUG> |authmgr| MAC=a8:bb:cf:04:0d:98 ingress 0x0x10324 (tunnel 804), u_encr 64, m_encr 64, slotport 0x0x2100 , type: local, FW mode: 0, AP IP: 0.0.0.0 mdie 0 ft_complete 0
Apr 22 15:27:46 :522078: <DBUG> |authmgr| MAC=a8:bb:cf:04:0d:98, wired: 0, vlan:75 ingress:0x0x10324 (tunnel 804), ingress:0x0x10324 new_aaa_prof: Academic8021X-AAA, stored profile: Academic8021X-AAA stored wired: 0 stored essid: Academic8021X, stored-ingress: 0x0x1001f
Apr 22 15:27:46 :522247: <DBUG> |authmgr| User idle timer removed for user with MAC a8:bb:cf:04:0d:98.
Apr 22 15:27:46 :501105: <NOTI> |AP JRCAC-203@10.10.161.70 stm| Deauth from sta: a8:bb:cf:04:0d:98: AP 10.10.161.70-00:24:6c:80:84:d9-JRCAC-203 Reason STA has roamed to another AP
Apr 22 15:27:46 :522258: <DBUG> |authmgr| "VDR - Add to history of user user a8:bb:cf:04:0d:98 vlan 0 derivation_type Reset VLANs for Station up index 14.
Apr 22 15:27:46 :501000: <DBUG> |AP JRCAC-203@10.10.161.70 stm| Station a8:bb:cf:04:0d:98: Clearing state
Apr 22 15:27:46 :522255: <DBUG> |authmgr| "VDR - set vlan in user for a8:bb:cf:04:0d:98 vlan 75 fwdmode 0 derivation_type Default VLAN.
Apr 22 15:27:46 :522258: <DBUG> |authmgr| "VDR - Add to history of user user a8:bb:cf:04:0d:98 vlan 75 derivation_type Default VLAN index 15.
Apr 22 15:27:46 :522255: <DBUG> |authmgr| "VDR - set vlan in user for a8:bb:cf:04:0d:98 vlan 75 fwdmode 0 derivation_type Current VLAN updated.
Apr 22 15:27:46 :522258: <DBUG> |authmgr| "VDR - Add to history of user user a8:bb:cf:04:0d:98 vlan 75 derivation_type Current VLAN updated index 16.
Apr 22 15:27:46 :522246: <DBUG> |authmgr| Idle timeout should be driven by STM for MAC a8:bb:cf:04:0d:98.
Apr 22 15:27:46 :522254: <DBUG> |authmgr| VDR - mac a8:bb:cf:04:0d:98 rolename logon fwdmode 0 derivation_type Initial Role Contained vp not present.
Apr 22 15:27:46 :522258: <DBUG> |authmgr| "VDR - Add to history of user user a8:bb:cf:04:0d:98 vlan 0 derivation_type Reset Role Based VLANs index 17.
Apr 22 15:27:46 :522083: <DBUG> |authmgr| Skip User-Derivation, mba:0 udr_exist:0,default_role:logon,pDefRole:0x0xdafcc4
Apr 22 15:27:46 :524124: <DBUG> |authmgr| dot1x_supplicant_up(): MAC:a8:bb:cf:04:0d:98, pmkid_present:False, pmkid:N/A
Apr 22 15:27:46 :522243: <DBUG> |authmgr| MAC=a8:bb:cf:04:0d:98 Station Updated Update MMS: BSSID=00:1a:1e:89:fa:e1 ESSID=Academic8021X VLAN=75 AP-name=JRCAC_202
Apr 22 15:27:46 :527000: <DBUG> |mdns| mdns_parse_auth_userapname_message 447 Auth->MDNS User APNAME: MAC:a8:bb:cf:04:0d:98, NEW AP NAME:JRCAC_202
Apr 22 15:27:46 :522038: <INFO> |authmgr| username=cmerrill MAC=a8:bb:cf:04:0d:98 IP=0.0.0.0 Authentication result=Authentication Successful method=802.1x server=CushDCSrv02
Apr 22 15:27:46 :522044: <INFO> |authmgr| MAC=a8:bb:cf:04:0d:98 Station authenticate(start): method=802.1x, role=faculty/faculty//logon, VLAN=75/75, Derivation=2/1, Value Pair=1, flags=0x8
--More-- (q) quit (u) pageup (/) search (n) repeat Apr 22 15:27:46 :522017: <INFO> |authmgr| MAC=a8:bb:cf:04:0d:98 IP=?? Derived role 'faculty' from server rules: server-group=CushTest, authentication=802.1x
Apr 22 15:27:46 :522127: <DBUG> |authmgr| {L2} Update role from faculty to faculty for IP=0.0.0.0.
Apr 22 15:27:46 :522049: <INFO> |authmgr| MAC=a8:bb:cf:04:0d:98,IP=N/A User role updated, existing Role=faculty/faculty, new Role=faculty/faculty, reason=Station Authenticated with auth type: 4
Apr 22 15:27:46 :522128: <DBUG> |authmgr| download-L2: acl=55/0 role=faculty, tunl=0x0x10324, PA=0, HA=1, RO=0, VPN=0 L3MOB=0.
Apr 22 15:27:46 :522050: <INFO> |authmgr| MAC=a8:bb:cf:04:0d:98,IP=N/A User data downloaded to datapath, new Role=faculty/55, bw Contract=0/0, reason=Download driven by user role setting, idle-timeout=300
Apr 22 15:27:46 :522258: <DBUG> |authmgr| "VDR - Add to history of user user a8:bb:cf:04:0d:98 vlan 0 derivation_type Reset Dot1x VLANs index 18.
Apr 22 15:27:46 :522254: <DBUG> |authmgr| VDR - mac a8:bb:cf:04:0d:98 rolename NULL fwdmode 0 derivation_type Dot1x Aruba VSA vp present.
Apr 22 15:27:46 :527000: <DBUG> |mdns| mdns_parse_auth_userrole_message 287 Auth User ROLE: MAC:a8:bb:cf:04:0d:98, NAME:cmerrill, ROLE_NAME:faculty
Apr 22 15:27:46 :522254: <DBUG> |authmgr| VDR - mac a8:bb:cf:04:0d:98 rolename NULL fwdmode 0 derivation_type Dot1x MSFT Attributes vp present.
Apr 22 15:27:46 :522254: <DBUG> |authmgr| VDR - mac a8:bb:cf:04:0d:98 rolename NULL fwdmode 0 derivation_type Dot1x Server Rule vp present.
Apr 22 15:27:46 :522259: <DBUG> |authmgr| "VDR - Do Role Based VLAN Derivation user a8:bb:cf:04:0d:98 role faculty authtype 4 rolehow Matched server rule.
Apr 22 15:27:46 :522254: <DBUG> |authmgr| VDR - mac a8:bb:cf:04:0d:98 rolename faculty fwdmode 0 derivation_type Dot1x Server Rule Role Contained vp not present.
Apr 22 15:27:46 :522258: <DBUG> |authmgr| "VDR - Add to history of user user a8:bb:cf:04:0d:98 vlan 0 derivation_type Reset Role Based VLANs index 19.
Apr 22 15:27:46 :522161: <DBUG> |authmgr| Valid Dot1xct, remote:0, assigned:75, default:75, current:75,termstate:0, wired:0, dot1x enabled:1, psk:0 static:0 bssid=00:1a:1e:89:fa:e1.
Apr 22 15:27:46 :522255: <DBUG> |authmgr| "VDR - set vlan in user for a8:bb:cf:04:0d:98 vlan 75 fwdmode 0 derivation_type Current VLAN updated.
Apr 22 15:27:46 :522258: <DBUG> |authmgr| "VDR - Add to history of user user a8:bb:cf:04:0d:98 vlan 75 derivation_type Current VLAN updated index 20.
Apr 22 15:27:46 :522260: <DBUG> |authmgr| "VDR - Cur VLAN updated a8:bb:cf:04:0d:98 mob 0 inform 1 remote 0 wired 0 defvlan 75 exportedvlan 0 curvlan 75.
Apr 22 15:27:46 :522029: <INFO> |authmgr| MAC=a8:bb:cf:04:0d:98 Station authenticate: method=802.1x, role=faculty/faculty//logon, VLAN=75/75, Derivation=2/1, Value Pair=1
Apr 22 15:27:46 :522008: <NOTI> |authmgr| User Authentication Successful: username=cmerrill MAC=a8:bb:cf:04:0d:98 IP=10.10.75.23 role=faculty VLAN=75 AP=JRCAC_202 SSID=Academic8021X AAA profile=Academic8021X-AAA auth method=802.1x auth server=CushDCSrv02
Apr 22 15:27:46 :522053: <DBUG> |authmgr| PMK Cache getting updated for a8:bb:cf:04:0d:98, (def, cur, vhow) = (75, 75, 1) with vlan=0 vlanhow=0 essid=Academic8021X role=faculty rhow=2

That mainly happens when the power is too high.  When you have time, please constrain your Min and Max to 12 and 18 when you feel comfortable.  Do the same to your 2.4ghz and 5ghz to start.

---

@theiglu wrote:

Yes I have. We disabled Client Match, but beyond that there isn't much they can do. I can't reproduce the issue for them and my users are very hard to get any feedback from. I actually just closed the case this morning after but then heard about more users experiencing the problem. 

 

So far the debug logs show nothing that correlates to the drops for particular users. The timestamps are hard to go by since the drops usually don't appear until a few minutes after Firstclass loses connection. It sits there and doesn't say it has disconnected for a while until it errors out, sometimes this can go on for 4-5 minutes. 

---

 

 

theiglu,

 

I would say that rarely is Clientmatch a problem.  You should re-enable it.  Clientmatch can only do so much in a poor environment.  To get some ideas, please look at the post here:  http://community.arubanetworks.com/t5/Technology-Blog/Removing-the-Bottleneck-in-Wireless/ba-p/77978

 

Client Match being disabled was under direct instruction by TAC. Of course after that, everything seemed ok, then it didn't again. That's the problem, everything is so intermittent with these Mac issues. 

 

 

Right now our AP's are set to the following in the default ARM profile:

 

Max Tx EIRP:127

Min Tx EIRP: 9

 

If I bring down the max transmit power, do you think less AP dense areas of our campus will suffer?

Theiglu,

This is an open forum and it is just my opinion about Client match only because you said it did not make a difference.

If you make the 5ghz band 12 and 18, you still have coverage on the 2.4ghz. I only arrive at that number because this is where you start with VoIP handsets to make them roam well.

When you say less dense, how far apart are they?

I understand about the client match, I'm not trying to discredit anything. Just that sometimes these settings help, other times they don't. Of course, if I had a machine I could duplicate the problem at will with, I could tell much sooner if these changes helped or hurt. Your input is very welcome, I'm no expert at this and this forum allows me to talk to people about it as I get responses from faculty. I can't do that with an engineer from TAC on the phone since I can't duplicate the issue during the call. 

 

Most of our dorms have AP's in every other room or hallway. I'd say most are within 70 feet of each other at most. We do have a lot of brick and concrete to contend with. 

If your dorms are brick and concrete, your APs should be in every other room where it can service possibly the next room over and possibly the room above and below.  The hallway AP normally does not provide good in-room coverage and causes interference due to great line of sight with other hallway APs....

 

Please see the  Next Generation Wireless Architecture for Multimedia-Grade Residence Halls VRD here: http://www.arubanetworks.com/resources/reference-design-guides/

 

---

@theiglu wrote:

I understand about the client match, I'm not trying to discredit anything. Just that sometimes these settings help, other times they don't. Of course, if I had a machine I could duplicate the problem at will with, I could tell much sooner if these changes helped or hurt. Your input is very welcome, I'm no expert at this and this forum allows me to talk to people about it as I get responses from faculty. I can't do that with an engineer from TAC on the phone since I can't duplicate the issue during the call. 

 

Most of our dorms have AP's in every other room or hallway. I'd say most are within 70 feet of each other at most. We do have a lot of brick and concrete to contend with. 

---

Did you ever make any progress on this? We just deployed some brand new MacBook Airs running 10.9.5, with roaming disabled, and we are hearing initial reports that the FirstClass disconnects are back. I've been unable to replicate in my testing so far.

It appears as though Yosemite may have resolved the Firstclass dropping
issue, but unfortunately the delays in reauthentication on 802.1X now mean
it disconnects the machine completely fairly often which is even worse
than just FC dropping. If the machine stays in the same space and doesn't
roam to another AP, Firstclass is far more reliable than in
Mavericks/Lion. I did notice that the firmware/driver version in Yosemite
was a major release newer than in Mavericks/Lion. I estimate that to be
the difference as it seems to only be the Broadcom 43xx cards doing it.
The Mavericks version of the driver/firmware is 5.xxx... while Yosemite is
6.xxx if I recall correctly.

I was able to get a number of machines to replicate the issue
consistently. Even clean wipe Mavericks would do it. It appears to be
traffic density related. If we have an empty campus, there are very few
Firstclass disconnects compared to while the campus was busy. TAC and I
did a number of FC packet captures that showed the response packets coming
back from the FC server making it to the controller, but never making it
to the machine. So somewhere in between the controller and the OS X
client, the packet disappears.

Once Yosemite gets the authentication chain delay fixed (same as early
versions of Mavericks), then we'll likely deploy out Yosemite to a wider
user base and get a more thorough test of Firstclass.

theiglu,

 

We have another account tracking the Yosemite roaming/auth delay issue.  Do you have any details to share on that?

I probably don't have much info compared to a lot of customers, but from my vantage point it seems very similar to the authentication delays in early Mavericks on 802.1X. The workaround then was to set all the certificates in the authentication chain to "Always Trust" in Keychain for EAP. I'm testing that now on two machines to see if it makes any difference. I'll trying to get a definite one way or the other today to see if it helped. 

We've had the same wireless drop's with our macbook air's.  Further investigation points to roaming.  I can force roam(walking) and see that the air's disconnect and most times won't even reconnect to the new ap.  Macbook pro's were disconnecting when roaming but would reconnect after 10-15 sec.  Pro's sitting between two ap's were not as sensitive to roaming when stationary.  Both notebooks were fine when using open SSID and also PSK, so I can verify that it was an 802.1x issue for us.  The fix that works for us after testing this am is going to the eap cert in the keychain and selecting a trust all for all settings(when using this cert).  Thx for all the input, this was a real pain point for us!

Glad I could help!

 

Now to wait and see when Apple releases the next 10.10 update to see if it is resolved (again). 

"I probably don't have much info compared to a lot of customers, but from my vantage point it seems very similar to the authentication delays in early Mavericks on 802.1X. The workaround then was to set all the certificates in the authentication chain to "Always Trust" in Keychain for EAP. I'm testing that now on two machines to see if it makes any difference. I'll trying to get a definite one way or the other today to see if it helped. "

 

Hello,

 

We are having the random drops as well - 6.4.3.4-4.2.1.2_53307.   We've probabaly put in 40 hours now with support and have made no progress.  We've even turned off authentication.    Any suggestions would be great.   I'm not as familar with OSX.   What should I do about the certs and always trust or is this applicable if we turned authentication off?

I think you should open a new thread, so that your issue gets the visibility it deserves.

 

with that being said, did TAC look at the RF at the site with the issue?

OH sure - We keep getting escallated and they keep making the same changes - first they'll ask why they got changed, then they'll change them back, then the next tech will make the same changes and so on and so forth.   They've pretty much said the only thing they can do is if we call them BEFORE someone loses connectivity and they can watch as they lose connectivity because for whatever reasone even if we run all the reports they ask for seconds after they lose connectivity it is too late.

Those are the most difficult issues to deal with - intermittent connectivity, so it could involve alot of guessing because they do not know what the problem is.  To make progress, they need specific data to be collected when the problem occurs.  If you don't know when it will occur, you won't know how to collect the information so they have to try different things to see if things improve to determine what is happening.

 

Do you have a ticket number?

Sure - I'm not complaining - they've been trying for ages and I'm starting to lose hope.  It isn't that they aren't trying but I'm not sure if there is anything left that hasn't been tried.   I got the logs for them today just minutes after it occurred, a show tech-support.  

 

1. Enable debug for client.

                        debug pkt mac <client MAC address>

                        debug pkt match mac

                        debug pkt type all

                        debug pkt dump

           

Type "q" to exit the console packet dump

 

1. #show datapath session | include <client-IP>
2. #show clock
3. #show log user-debug | include <client MAC address>
4. #show ap debug mgmt-frames | include <client-MAC address>

We've also been gathering the syslogs for weeks and sending them at the time of a disconnect.   If something doesn't jump out at them by now I'm not sure what is left.

 

What is the ticket number?

1828967.   They are going to call me in the morning to review their findings.   Hopefully they've figured something out.  

 

The issue for me is I've dissappointed a customer and I charge by the hour but I certainly can't charge a dime for those many many many many hours I've spent and solved nothing.   I'm sure it has been no picnic for the engineers I've worked with as it is frustrating to work on something and not be able to figure it out.

Two things:

1.  You are bonding channels on the 2.4ghz band, which is a no no, because it causes so much interference.  That is why your 2.4ghz channels look like 7+ and 11-, which should not be done.

2.  You are running DFS channels, which not all devices support, so devices that cannot see those channels will end up on the already congested 2.4ghz band.

 

Do this:

Go to RF >Advanced and make sure Wide Channel Bands is "None" and 80 mhz support is "Disabled".  Make sure "client aware" is set to "Disabled" so that the APs can adjust properly:

Click on the Edit button next to 5GHZ channels and only enable the channels on the right in the "selected" column like you see below.

 

Click on ok, ok, ok to get out of the dialogs.  See if your problem continues to happen.

 

Remove that "Test" SSID, because it adds more management traffic to the air.  

The ******Guest  SSID should have "Broadcast Filter ARP" under Advanced

Set  "local probe request threshold" to 0, on the "**rive", and "TT" SSIDs.

Under the TT SSID, the G-max-tx-rate should be 54, and NOT 1

 

 

 

 

 

Thanks so very much!!!

 

Are 1 & 2 the same issue - removing the +7 +1 for 2.4 or am I looking in the wrong place.

 

 

I've got the rest adjusted.

Setting the wide bands to None fixes the 7+1 issue.
Only selecting the channels in the screenshot fixes the DFS issue.

Thanks.   I know everyone's been working on trying to figure this out and I certainly appreciate your time as well.   Hopefully we'll get some feedback today.

The settings I suggested are only to make the RF better for the clients that are there.  It is quite possible that all your problems are not solved, but reduced.  That might allow the clients with the real problems to surface.