Hello,
I want to pick up this discussion for further documentation, if you hit the issue that clients will not connect to an AP-505 at 8.6.x release.
Because this thread will be an google hit if you search for this issue...
Actually I was informed by an customer, that the connection to an SSID with WPA2 PSK authentication at one particular AP-505 fails (mobility conductor, md cluster environment).
The issue was reproducable, and shows up as follows:
- Older devices, like Galaxy S5 NEO, will try to connect but will not be successfully without any error message.
- If an older AP, AP-305 with an open ssid and captive portal for instance, is around, the device will connect there
- Newer devices supporting AX, for example IPhone 12, will throw an connection error
If you issue auth-tracebuf, you will see messages until wpa2-key4 but the user never will show up at the user table.
It's interesting, that in some cases the user will be shown on the user list at mobility conductor GUI and entrys for active and standby controller but without IP address.
But as wrote before, if you issue show user-table, the client will not show up at cli.
Of course the device will also get no IP address and will not shown at the ARP table at default GW.
In this case, we got nearly 40 AP-505's purchased at quater 3 of 2020.
All AP's of that batch are working fine and are not affected.
A couple of months later, we added one AP-505.
Only this one AP-505 is effected by the issue.
I was able to fix the issue with disabeling high efficiency for that particular SSID.
This workaround was necessary at version 8.6.0.16.
A previous post says that a driver update at 8.6.0.7 (released at Dec 14, 2020) and 8.7.1.1 (released at Dec 15, 2020) will bring more stability for AP-505, because it brings an driver update for AP-505.
I can not confirm that for the 8.6 release.
There is no notice of an driver update at 8.6.0.7 release notes and I found also no notice at any higher 8.6.x releases for that at INSTANT or AOS version.
The problem was existing in that installation also under 8.6.0.16, it was only fixable by disabeling HE for that SSID.
But the release notes of
8.7.1.1 (AOS controller based, I did not find it yet for instant version) does contain a driver update for AP-505.
The bug ID's AOS-207318, AOS-207996 were fixed by that release with the description: "Clients experienced poor performance with AP-505
access points running ArubaOS 8.7.0.0 or later
versions. Enhancements to the wireless driver
resolved this issue.".
Because of that, I decided to update the whole infrasructure (conductors, MD's, AP's) to 8.7.1.8.
And, yes, with AOS 8.7.1.8 the issue is gone.
With enabled HE, the connection to that one AP was successfully again!
So may, a batch of AP-505's will have this issue, others not.
May some little differences in the hardware, or something else?
However, for this setup (AOS based), the issue was only resolvable by updating to 8.7.1.8 (not tested with 8.7.1.1 - 8.7.1.7 or instant releases).
Unfortunately, 8.7.x is still not at conservative state....
------------------------------
Best regards, mom
------------------------------
Original Message:
Sent: Mar 07, 2021 08:26 PM
From: Steve Yuroff
Subject: Aruba AP 505 Random Disconnects
Hi Marcel,
When I look at the image in your tweet, I think the key message I'm supposed to get from it is "increased consistency". Do you agree?
I have not tested using WPA personal, as it's not a suitable fix. Any data gathered about it differing from 802.1x wouldn't be used to execute a solution.
The behavior has only been observed on macOS devices, as that's 98% of my client base.
Has been documented on Intel and M1
I can make no comment on Windows behavior.
When the issue occurs, the association with the previously used AP remains, there is not a roam. My roaming experience has been consistently successful.
I have done OTA captures on another device to gather frames, but as there's no certain way to induce this behavior, and it seems to have subsided by disabling .ax, I haven't been able to get a capture with frames.
If the issue has occurred, inducing a RSSI-based roam to another AP will restore connectivity. Clients have had this experience while stationary, so minimal to no reason to expect a roaming threshold to be hit.
During the capture that was graphed above, I did not have pings happening, so don't try to match them there.
My roles are as basic as possible: I'm enforcing no ACLs or restrictions within the Aruba hardware.
------------------------------
Steve Yuroff
Original Message:
Sent: Mar 07, 2021 02:27 PM
From: marcel koedijk
Subject: Aruba AP 505 Random Disconnects
Hi Steve,
The minimum firmware for a AP-505 is ArubaOS 8.6.x.x and it's little younger product than it;s bigger brother the AP-515 that can run at 8.5.x.x with some 802.11ax restrictions.
The AP-505 WLAN chipset driver gets an update in the releases 8.6.0.7 and 8.7.1.1 which i have much better overall stability. I test it a couple of times by downgrading/upgrading the firmware and measure the results in a UXI sensor and a WLANpi Wiperf sensor. The measurements where done at a very close distance of 1-2meter. One small different is that i have controllerbased controllers and your use aruba instant virtual cluster. I did post my findings on twitter.
https://twitter.com/Marcel_koedijk/status/1339656755631022081?s=20
https://twitter.com/Marcel_koedijk/status/1339667989822631949?s=20
Some suggestions:
- What kind of authentications did you use, your graphic shown some eap-traffic in the middle. Does the issue resolved when using wpa2-personal for testing purpose? Maybe try to enable 802.11k?
- Is this issue only exist on Apple devices? Are the all M1 chipset or mixed? Did you try a windows devices and have this the same issues or not?
- Doest the AP roam to another bssid when the issue occur?
- Could you try to make an 802.11 frame packet trace, mac's can do that;)
- I see some issue in the past. What happend when you have an the issue occuring and walk with your macbook to another access points, does the problem take away when you roam back and forward?
- While you say icmp had no issues, but dig will not resolve dns its sounds like an uper layer issue, but i cant match that with your graphic showing. What role/acl do your clients get?
------------------------------
Marcel Koedijk | MVP Guru 2021 | ACMP | ACCP | Ekahau ECSE | Not an HPE Employee | Opionions are my own
Original Message:
Sent: Mar 07, 2021 11:22 AM
From: Steve Yuroff
Subject: Aruba AP 505 Random Disconnects
I think this graph and the key tell a story. Captured on a client while the failure condition happened, during a Teams call.
------------------------------
Steve Yuroff
Original Message:
Sent: Mar 07, 2021 09:23 AM
From: Steve Yuroff
Subject: Aruba AP 505 Random Disconnects
> When google on Apple's Big Sure release and Wi-Fi issues your are defintly not alone.
This is not a Big Sur specific problem. I've documented it in both BS and fully current 10.15.7
>My personal experience with the AP-505 is that 8.6.0.7 and 8.7.1.1 give a much more stable performance what i measured <snip>
Much more stable than what? What are you comparing to?
Other exploration of this topic has led me to learn that the 8.5 firmware is considered quite stable and drama -free, "conservative" release in Aruba official terms. Since I have a 505 and 505H in my deployment, I cannot use that, and have to be in the 8.5/8.7 category, which is under the "standard" release category. What Aruba calls "standard", others might call "developmental" or even "beta", and I think it shows.
My MOS scores and other measures are quite good... until the sudden moment that data stops flowing and connections drop.
Yes, I have a TAC open. I feel like I'm often reiterating overlooked facts with my contact, and correcting misinterpretations.
Detailed description: Tickets/calls come to us as "My Teams (or Zoom) call failed". As this problem does not yet have a way to induce on demand, data gathering requires luck and patience, but I've been able to document that at times, most data flow to the affected client randomly stops:
-if present, UDP data flow of a videoconference falls off
-ARP is not seen by client
-TCP error rate jumps
-UDP and TCP communications fail, documented by
-- ssh -vvv user@dotted.quad.ip.address will not start an SSH handshake
-- dig 8.8.8.8 aruba.com doesn't return any traffic
--but ping 8.8.8.8 does get icmp replies
I've found 3 steps to regain connection:
-wait long enough. I do not know what action occurs to "unbreak" the data flow
-manually off/on WiFi at client device
-induce a roam to require fresh authentication
During this, 802.1x authentication timer continues to count up on client, no feedback of signal loss
My reporting devices have included 13" macbook pro on 10.15.7 and M1 Air on big Sur 11.2.2.
Has happened on multiple APs
2.4 20Mhz and 5 40Mhz are both available
I have experienced it at probably -70 (not documented at the moment, average of that desk), noise floor at -90, 270-300Mbps
I will not have documented channel utilization values, but the office is nearly empty, because pandemic. The network is not being pushed to capacity.
Controlled by Aruba Instant virtual controller
Logging- this has been troublesome. Per my TAC, I've been doing debug packet dumps via ssh to an AP. My observation is that "debug pkt type all" does not dump all packets as expected. I'm mostly getting just DNS data out of it.
------------------------------
Steve Yuroff
Original Message:
Sent: Mar 05, 2021 05:11 PM
From: marcel koedijk
Subject: Aruba AP 505 Random Disconnects
Hi Steve,
There could be many things that have impact on your experience. First of all 802.11ax is still a amendment and is in the last state of ratification to be the new standard. I agree that when you keep a close look at your client drivers enable the HE SSID shoud be possible those days. Still i like to turn it of for a while in most environment because there are still a lot of different clients to support with old drivers that sucks. The other problem is that apple don't like to certified there new products by the Wi-Fi Alliance which can cause issues in intercompatibility. When google on Apple's Big Sure release and Wi-Fi issues your are defintly not alone.
https://www.ieee802.org/11/Reports/802.11_Timelines.htm
https://www.wi-fi.org/product-finder-results?sort_by=default&sort_order=desc&keywords=macbook
https://www.digitaltrends.com/computing/common-macos-big-sur-problems-how-to-fix-them/
https://forums.macrumors.com/threads/wifi-issues-m1-macbook-pro.2270884/page-4
My personal experience with the AP-505 is that 8.6.0.7 and 8.7.1.1 give a much more stable performance what i measured with a Aruba UXI and WLANPi WiPerf sensor seperatly in the same time. At home i use two AP-505 with the 8.6.0.7 controllerbased release and work daily hours with Microsoft Teams Video Call while my girlfriend use Teams the full day on another floor to teach students from home. My VOIP MOS values are 4.5 what is quit excellent and also my UCC value's that are continuelessly monitored are real great. (iam a using microsoft/iphone/and some older iMACs) No i don't think you got the wrong product.
Because all kind of WiFi issues are very annoying and hard to troubleshoot i understand your concerns. Even if it's hard to help you here on the forum the best thing you allready did is to open a support case, be patient and assist with the required log files and questionaries is the best for now, understand that issues could be complex and give TAC support the time to do investigation in there LAB. When they don't call you back in time pick up your phone and call them, ia'm sure they will give you a fast response of the status.
Feel free to share us some details of your issue, like;
- Detailed problem description
- Have different type of users (mac/phones/windows) have the same issues?
- Is the issue related to a specific area?
- Is there a validation survey rapport? (highly recommended)
- What are you radio settings? (2.4Ghz, 5Ghz or both)
- Channel widht
- Signal strength (RSSI and SNR)
- Bitrates
- Channel Utilization
- Frame retry rates
- EIRP tx-power
- Controller or controllerless environment, what type of controllers?
- Share some logging
- Share "show user mac ...." details
- etc.etc
Ok you get the point, there is a lot information to share to get a picture of your problem. That's why TAC support is your best bid. Feel free to share us some information and keep update the community if you find a solution.
------------------------------
Marcel Koedijk | MVP Guru 2021 | ACMP | ACCP | Ekahau ECSE | Not an HPE Employee | Opionions are my own
Original Message:
Sent: Mar 04, 2021 09:43 AM
From: Steve Yuroff
Subject: Aruba AP 505 Random Disconnects
Oh yes, FULL AGREE that I shouldn't have to disable .ax, but if I'm pushed to a corner where I either
-leave .ax on and have an office raging at me because their videoconferences randomly drop (unacceptable)
-turn it off, run as a .ac network, and be grumpy that I can't get the full feature set
-go back to my 7 year old Ruckus network and try to get Aruba to take this deployment back and give me my money back
I'm going with option 2.
Our network is dominated by macOS, so drivers are defined by Apple. My clients are either on the latest patch of 10.15.7 or a few on BigSur. Either way, as current as possible.
Maintenance: Firmware states I'm on 8.7.1.1_78245 (Digitally Signed - Production Build) and that it's current.
I have a case open with TAC on this.
My expectations in this situation is that I'll accept disabling .ax and running as a .ac network for now, but TAC does what's necessary to gather logs and data to characterize the problem and take it to engineering for fix. I haven't heard from my TAC contact since Tuesday PM, and I disabled .ax about 2 hours after talking to him, based on the suggestion in this post.
------------------------------
Steve Yuroff
Original Message:
Sent: Mar 04, 2021 05:03 AM
From: Herman Robers
Subject: Aruba AP 505 Random Disconnects
You should not need to disable 11ax. Make sure you run recent firmware for your AP/controllers, as specifically for the 50x and 51x there have been fixes in the chipset drivers that caused a few issues like these.
Also, some client devices had issues in their drivers, like some infamous Intel drivers that just did not show 11ax SSIDs. Updating the driver on the client solved that.
If you still experience disconnects or other issues, please reach out to Aruba TAC support. They can see if that is a configuration or a possible unfixed known issue in the drivers.
------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.
Original Message:
Sent: Mar 02, 2021 05:50 PM
From: Steve Yuroff
Subject: Aruba AP 505 Random Disconnects
Trying to see how much overlap this has with a problem I'm pursuing: were you seeing the AP rejecting all users at once or specific devices at a time?
It kills me to turn off HE, having .ax was a serious data point in buying the 500 series over other distributed controller, SMB quality options (Ruckus Unleashed).
------------------------------
Steve Yuroff
Original Message:
Sent: Feb 02, 2021 10:22 PM
From: David Adino
Subject: Aruba AP 505 Random Disconnects
Hi,
Wouldn't disabling the High Efficiency also remove 802.11AX as well?
Me too are facing performance issue with AP 505, though the purpose of customer buying 505 is for AX, but there are just too many people getting it fix by disabling this. :/
BR
David
------------------------------
David Adino
Original Message:
Sent: Sep 17, 2020 07:30 AM
From: david gavigan
Subject: Aruba AP 505 Random Disconnects
HI
Issue is resolved High Effiency was the cause of the problem.
Thanks so much for the help