Controllerless Networks

last person joined: 26 minutes ago 

Aruba Instant Wi-Fi: Meet the controllerless Wi-Fi solution that's easy to set-up, is loaded with security and smarts, and won't break your budget.
Expand all | Collapse all

Troubleshooting intermittent poor throughput on AP-515

This thread has been viewed 106 times
  • 1.  Troubleshooting intermittent poor throughput on AP-515

    Posted May 27, 2021 12:47 PM
    Hi all
    This is my first post, so be gentle with me :-)
    Please feel free to point me in the direction of a troubleshooting guide so that I can try and nail the basics before troubling the rest of you with my issues.

    We've just started to plan the roll out of 30 AP-515s and I have installed one unit in a small office as a trial. At first, performance was excellent and everyone was happy. However, as soon as I left for the day (of course) the users started to report performance issues. I have now experienced this for myself and I would like to ask for some advice from those more experienced with the Instant system. For background, I have a reasonable amount of wired and wireless knowledge, but I would not necessarily consider myself an expert. We were using Meru/Fortinet for many years and I also have a lot of experience with Ubiquiti equipment, but we are moving away from those two systems to standardise on Aruba. 
    The basic symptom here is that everything is fine one moment, clients are working happily and a speed test reveals almost 300Mbps to the Internet. Then suddenly and without any warning everything will grind to a halt and a speed test will reveal about 1Mbps. After reading some of the manual and searching on here I have tried looking at `show clients` which reports signal and speed as 'good'. From a client (Macs and Chromebooks mainly) I am seeing RSSI figures of about -45 - -40 (the AP is only 3-10 metres from the clients in this office). So raw signal strength and transmit power should hopefully not be the issue here. 
    The AP was initially powered by an 802.3af PoE switch and I had IPM enabled. I thought that may have been the cause of this poor behaviour and so I switched it to an 802.3at switch and disabled IPM, but that has not made any difference. 

    I've tried disabling 80MHz channels in config -> RF -> ARM -> Access point control.
    I've tried enabling Client Match (all settings left at default values) 

    I have the output of `show tech-support` at various times when the problem is occurring, just let me know what portions will be helpful to post. I've also attached some screenshots from the gui.

    Also, is it advisable to strip out MAC addresses and serial numbers of APs? 

    Thanks for your assistance.


  • 2.  RE: Troubleshooting intermittent poor throughput on AP-515

    Posted May 28, 2021 12:47 PM
    Hi,

    Is the problem happening all the time or is it randomly happening? Are the clients connecting on 2.4 GHz or 5Ghz band?  While the problem is occurring on Wi-Fi, are you experiencing the same slowness if you test from a wired device? Can you do iPerf test to a local machine instead of speedtest to the internet to check if it is really Wi-Fi issue or internet issue?

    To decide if it is a real Wi-Fi issue, you need to check Radio 0 or Radio 1 depending on which band the client is using.
    You need to check

    • Channel Utilization (Check Utilization)
    • Retry Rate (Check Drops / Errors)
    For both of the above, compare the output from Radio 0 or Radio 1 once the issue is happening and once the network is operating properly.


    • MCS Rate - If you have a MAC , use the Option and click Apple's Wi-Fi menu icon to find the MCS index displayed as part of the Wi-Fi connection details



    ------------------------------
    Ayman Mukaddam
    ------------------------------



  • 3.  RE: Troubleshooting intermittent poor throughput on AP-515

    Posted Jun 06, 2021 11:29 AM
    Thank you, Ayman. 

    I do remember that the channel utilisation figures were green and very low. I will get more detailed info when this next happens and also check the retry rate values. 
    The issue definitely relates to the 515, as anyone in the office that is hardwired reports no issues. They're all in the same switch. 
    If I swap out the 515 with the existing Unifi AP then everything is stable and remains so. 

    The problem occurs every hour or two, so it's not constant but it is consistent throughout the day. 

    Also, we receive notifications from an app that runs on an iPad in the same office and which is used for a visitors signing in system. If the app loses connectivity with its cloud hosted portal then it sends out an alert. We are getting these alerts multiple times a day and this seems to coincide when the rest of the office users who are on wifi report issues.

    I'll be back with more info as soon as I have it (undoubtedly, tomorrow, when everyone is back at work) 

    Thank you



  • 4.  RE: Troubleshooting intermittent poor throughput on AP-515

    Posted 29 days ago
    I am also experiencing a transient speed issue with the AP515, running FW 8.7.1.3. Symptoms are as you describe, transient and random, yet consistent throughout the network of 44 AP's across the day. 

    It doesn't appear to be interference related, although the speeds appear to be lower on those AP's where more than 2 or 3 clients are connected. Running a speedtest on the client often shows less than 10Mbps with a WiFI link speed in excess of 400Mbps.

    Network is running 40Mhz bands, HE disabled, ARM running the channels/power.

    Tonight I have upgraded to 8.7.1.4 and will report any change tomorrow when we have 300 people or so back on the network.

    JC

    ------------------------------
    John Cirocco
    ------------------------------



  • 5.  RE: Troubleshooting intermittent poor throughput on AP-515

    Posted 29 days ago
    Hi, I'm also experiencing a similar issue on the 515's. Seems transient yet consistent over the course of a day, not specific to an individual AP or channel. I am experiencing this on 8.7.1.3, tonight have upgraded to 8.7.1.4, not sure if we should upgrade to 8.8 yet.

    We're running 40Mhz bands, HE disabled. 

    JC

    ------------------------------
    John Cirocco
    ------------------------------



  • 6.  RE: Troubleshooting intermittent poor throughput on AP-515

    Posted 29 days ago
    I think I might be seeing the same... in troubleshooting unreliable videoconferencing experiences through a relatively new Instant deployment, I've been observing the behavior of a consistent 12Mbps iperf stream.  It will typically hold a consistent 1.5MB/sec... until it sometimes just pauses, then unleashes a backlog of queued data.  This is the kind of thing that might feel like a slight glitch in your webpage load, and is a giant smack in the face for videoconferencing.

    [  5]  61.00-62.00  sec  1.50 MBytes  12.6 Mbits/sec                  

    [  5]  62.00-63.00  sec  1.50 MBytes  12.6 Mbits/sec                  

    [  5]  63.00-64.00  sec  1.38 MBytes  11.5 Mbits/sec                  

    [  5]  64.00-65.00  sec  1.50 MBytes  12.6 Mbits/sec                  

    [  5]  65.00-66.00  sec  1.14 MBytes  9.54 Mbits/sec                  

    [  5]  66.00-67.00  sec  0.00 Bytes  0.00 Mbits/sec                  

    [  5]  67.00-68.00  sec  0.00 Bytes  0.00 Mbits/sec                  

    [  5]  68.00-69.00  sec  0.00 Bytes  0.00 Mbits/sec                  

    [  5]  69.00-70.00  sec  0.00 Bytes  0.00 Mbits/sec                  

    [  5]  70.00-71.00  sec  0.00 Bytes  0.00 Mbits/sec                  

    [  5]  71.00-72.00  sec   802 KBytes  6.57 Mbits/sec                  

    [  5]  72.00-73.00  sec  4.17 MBytes  35.0 Mbits/sec                  

    [  5]  73.00-74.00  sec  4.15 MBytes  34.8 Mbits/sec                  

    [  5]  74.00-75.00  sec  4.76 MBytes  39.9 Mbits/sec                  

    [  5]  75.00-76.00  sec  1.38 MBytes  11.5 Mbits/sec                  

    [  5]  76.00-77.00  sec  1.50 MBytes  12.6 Mbits/sec                  

    [  5]  77.00-78.00  sec  1.50 MBytes  12.6 Mbits/sec                  

    [  5]  78.00-79.00  sec  1.50 MBytes  12.6 Mbits/sec   



    ------------------------------
    Steve Yuroff
    ------------------------------



  • 7.  RE: Troubleshooting intermittent poor throughput on AP-515

    Posted 29 days ago
    Apologies for the late update. I forgot that I was due to take a week off. While I was away one of my colleagues looked into the issue. He tried various things that were recommended but nothing helped. In the end he bit the bullet and downgraded from 8.8 to 8.6.0.9 and it has been absolutely rock solid with not a single complaint in over two weeks. I would definitely try a downgrade if you have not given this version of firmware a try.

    Fingers crossed that this may help you.

    Stuart


  • 8.  RE: Troubleshooting intermittent poor throughput on AP-515

    Posted 29 days ago
    I have a mix of 515, and one 505 and a 505H.  I know there are minimum firmware versions for the chipset that the 505 family runs on... where do we look up if 8.6.0.9 is a compatible release for them?

    And is the downgrade procedure any more complicated than providing the right firmware files and executing?

    ------------------------------
    Steve Yuroff
    ------------------------------



  • 9.  RE: Troubleshooting intermittent poor throughput on AP-515

    Posted 29 days ago
    look in the firmware section, the 515 is Draco, the 505 might be the same. Ive done lots of lab testing going up and down FW versions, no problems with that.

    ------------------------------
    John Cirocco
    ------------------------------



  • 10.  RE: Troubleshooting intermittent poor throughput on AP-515

    Posted 29 days ago
    FWIW, a 515 is Draco, 505 and 505H are Gemini.

    ------------------------------
    Steve Yuroff
    ------------------------------



  • 11.  RE: Troubleshooting intermittent poor throughput on AP-515

    Posted 28 days ago
    All of my experience is with 515s. I've got some 535s coming in the coming weeks, as soon as distributors can actually secure stock!
    Changing firmware in either direction appears to be a smooth and simple process. I didn't need to do anything special, but of course took a backup of the running config first.

    I've read many discussions on here that talk of 802.11ax basically being fairly useless at the moment because there are so many problems with chipsets and drivers. One suggestion that seems to be popular for when stability issues are mentioned is to try turning off 'high efficiency', which disables 802.11ax (someone please correct me if I'm wrong here, I'm no expert yet!). This is something I only found out after the downgrade was performed and so I'm not sure if that would have also resolved my issues, at least temporarily until we receive a load of new client devices and want to actually start reaping the benefits of 802.11ax!

    Stuart


  • 12.  RE: Troubleshooting intermittent poor throughput on AP-515

    Posted 28 days ago
    Yes, High Efficiency = 802.11ax.  I disabled that a while ago in my troubleshooting, it wasn't a solution for me.  Moderately frustrating to be deploying .ax laptops but disabling that function because my wireless vendor's code isn't stable regarding it.  It's still a questionable efficiency gain until a critical mass of the airtime is being used by .ax clients... I'm sure my client base isn't there yet, but it's moving that way.

    Glad to hear that any firmware change seems to be trivial.  I'll run whatever it takes to be stable, because both my coworkers and I are more than tired of interruptions to videoconferences.  I'll hold onto hope that someday I can use my .ax feature set in the 515 and 505

    ------------------------------
    Steve Yuroff
    ------------------------------



  • 13.  RE: Troubleshooting intermittent poor throughput on AP-515

    Posted 29 days ago
    Updating to 8.7.1.4 caused the AP's to go into a reboot loop. The AP's appeared to crash randomly (not all at once) so impact to client was not great, after a couple of hours all AP's became stable and I pushed 8801.

    8801 has been OK for a few hours this morning, will provide a further update over the coming days.

    TAC have advised that all feature updates and fixes in 87 are also in 88.


    ------------------------------
    John Cirocco
    ------------------------------



  • 14.  RE: Troubleshooting intermittent poor throughput on AP-515

    Posted 28 days ago
    Performance & stability has vastly improved on 8.8.0.1, even on AP's with higher user counts. So far so good.

    ------------------------------
    John Cirocco
    ------------------------------



  • 15.  RE: Troubleshooting intermittent poor throughput on AP-515

    Posted 28 days ago
    Funny, I have a ticket/day on the Wifi running 8.8.x, with only a skeleton crew in the office.

    I have an open ticket with TAC, and am getting frustrated with the latency of getting assistance from them.

    ------------------------------
    Steve Yuroff
    ------------------------------



  • 16.  RE: Troubleshooting intermittent poor throughput on AP-515

    Posted 27 days ago
    Hi Steve, what firmware are you running? 

    On 8711 I experienced regular random pauses of data flow, some as long as 2 minutes. Zoom/Teams calls would drop completely. Mac's would not resume data flow after sleep. I proved this in the lab back and forth, 8713 resolved both the sleep and data pause issue, essentially they were the same symptom. Device stays connected to Wifi yet has no data flow. 8400 did not have these problems. I remedied this by going to 8713, which had transient speed issues, remedied by going to 8801.

    JC


    ------------------------------
    John Cirocco
    ------------------------------



  • 17.  RE: Troubleshooting intermittent poor throughput on AP-515

    Posted 27 days ago
    Hi JC, Oh yes: multiple minutes of data flow pause, completely breaking Zoom/Teams calls is EXACTLY what's causing me stress right now.  I AM NOT ALONE. 
    I've documented this behavior under both 8.7.1.1_78245, and on 6/20 I upgraded to 8.8.0.1_80393, and the problems persist.
     All I want is drama-free, boring as paint firmware. 

     




  • 18.  RE: Troubleshooting intermittent poor throughput on AP-515

    Posted 26 days ago
    Was on with TAC early this AM, agreed to downgrade to 8.6.0.9.  Really hoping I get the same success out of that as sram (Stuart) got above.​

    ------------------------------
    Steve Yuroff
    ------------------------------



  • 19.  RE: Troubleshooting intermittent poor throughput on AP-515

    Posted 26 days ago
    I understand your frustration, please let me know how 8609 works for you.

    JC

    ------------------------------
    John Cirocco
    ------------------------------



  • 20.  RE: Troubleshooting intermittent poor throughput on AP-515

    Posted 20 days ago
    Good news: we've gone 3 days with only 1 user complaint about wifi connectivity, and even received kudos when feedback was solicited.

    Bad news: the one user complaint was about coverage.  That happened because one can't run a 505H on firmware 8.6.0.9, the minimum system requirements for a 505H is 8.7.0.0, leading to a predictable coverage gap.

    The Aruba TAC did not advise that we'd loose an AP from our cluster by making this change... not impressed.  TAC should note the incompatibility of the plan vs hardware.

    So I think I have a stable system... and a paperweight. That feels a little short of success.

    ------------------------------
    Steve Yuroff
    ------------------------------



  • 21.  RE: Troubleshooting intermittent poor throughput on AP-515

    Posted 20 days ago
    Matterafact, I now realize there is no such thing as a conservative release firmware that can be run on the 505H.  It's been only supported in Standard (aka beta) releases for its whole history.  This might be worth knowing for someone else interested in the 505H AND a stable system.


    ------------------------------
    Steve Yuroff
    ------------------------------



  • 22.  RE: Troubleshooting intermittent poor throughput on AP-515

    Posted 20 days ago
    just note that version 8.6.0.10 has been released
    https://www.arubanetworks.com/techdocs/Aruba-Instant-8.x-Books/Release-Notes/86/Aruba-Instant-8.6.0.10-Release-Notes.pdf

    ------------------------------
    Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba.
    ------------------------------



  • 23.  RE: Troubleshooting intermittent poor throughput on AP-515

    Posted 20 days ago
    Released the day after I downgraded to 8.6.0.9.  What timing we have!

    But thanks for the heads up, I had missed the fact that 8.6.0.10 has gone Conservative Release.

    ------------------------------
    Steve Yuroff
    ------------------------------



  • 24.  RE: Troubleshooting intermittent poor throughput on AP-515

    Posted 20 days ago

    Hi,

    you could force that 505H to run as "standalone" AP utilizing a 8.7.x firmware version (AP can run then even in same mgmt-vlan as the "cluster AP's")

    Of course - you would need to manually copy the config to that device and keep the configs manually in sync.

    And as soon as a stable working 8.7.x or higher firmware will be available join that AP to the cluster again.

    Groetjes

    Jochem



    ------------------------------
    Jochem Knoben
    ------------------------------



  • 25.  RE: Troubleshooting intermittent poor throughput on AP-515

    Posted 20 days ago
    Hi Jochem,

    I appreciate knowing this is an option, thanks for pointing it out.
    My reluctance to follow through on this option is that I don't trust 8.7.1.1, because that's where I started this cluster at, and it's had problems since deployment.  I'm cautiously optimistic that 8.6.0.[9|10] is a happy place, and feel like that's an uncomfortable risk to have one standalone on a firmware that has proven itself to lead to Ticket Generating Events.

    I'm rather surprised that Aruba firmware has so many troubling behaviors, and product that I bought over 6 months ago can't actually be used on proven, stable "Conservative" firmware.

    ------------------------------
    Steve Yuroff
    ------------------------------



  • 26.  RE: Troubleshooting intermittent poor throughput on AP-515

    Posted 10 days ago
    Hi SWY, how has performance been on 8609, has it resolved the intermittent slowness?

    ------------------------------
    John Cirocco
    ------------------------------



  • 27.  RE: Troubleshooting intermittent poor throughput on AP-515

    Posted 10 days ago
    Hi John,

    I am pleased to say that since moving to 8.6.0.9, we've ended the inflow of "my videoconference suddenly dropped" tickets.

    Only bummer is that it renders my 505H standalone, and that device can only run a firmware where we'll still have this experience.  I've repositioned APs to adapt to removing it from the floorplan, and am pinning my hope that someday it will get a fully stable firmware, exempt from all drama.

    ------------------------------
    Steve Yuroff
    ------------------------------



  • 28.  RE: Troubleshooting intermittent poor throughput on AP-515

    Posted 9 days ago
    Thats great to hear. 8801 has been an improvement over 8713 however we still have regular transient speed degradation. I'll be moving the network(s) to 8609 in the hope I too can stem the stream of complaints.

    ------------------------------
    John Cirocco
    ------------------------------