Wireless Access

 View Only
last person joined: 21 hours ago 

Access network design for branch, remote, outdoor, and campus locations with HPE Aruba Networking access points and mobility controllers.
Expand all | Collapse all

MSM720 + 3500yl + HP560s performance issues

This thread has been viewed 35 times
  • 1.  MSM720 + 3500yl + HP560s performance issues

    Posted Jan 30, 2023 09:57 AM
    Hi there,

    I recently installed the following configuration:
    Net DiagramThe ISP is a fiber gigabit, which goes through a router (temporary - will be replaced by a Checkpoint 4400 running pfsense pretty soon), then goes into the default VLAN of a 3500yl-48G-POE, which then goes into the Internet network (ports 5&6) of an MSM720 Premium (with just the two networks, one VSC and a few SSIDs), which then goes to the Access Network (ports 1 through 4) and through a cable into a second untagged VLAN of the 3500yl, which then I use to power on a bunch of HP560 APs. Almost all cabling is CAT6 SFTP to reduce RF and power cabling interference. I get pretty solid coverage across the building, though the speed is dubiously slow - can't get more than 160-170Mbits even though the 560's are capable of 1300Mbits 802.11ac on paper. If I were to connect to the ISP router (some vodafone-branded box, dubbed "gigabox" - still only 802.11ac, though) I get upwards of 550Mbits without a hitch, so it must be an issue with the rest of the network.

    Now the caveats that I should point are (and if anyone has advice how to fix those, that'd be terrific):
    - I couldn't trunk the ports between the 3500yl and the msm720 - I wanted to create either a static trunk or an LACP bond to ensure I have higher throughput, but for some reason the two switches just don't allow me to setup a bond, and if I were to set up a static trunk the link breaks - I believe it detects the multiple links as an IP loop and just disables the port. If I unplug three of the 4 cables, everything starts working again.
    - I am not exceeding the AP licenses I have on the MSM720 - I have license for 10, I'm using only 6 so far.
    - There is also a 2900-24G between the 3500yl and the ISP router, though, I'm using it only as a pass-through. Happy to remove it if it's the culprit, though it should be capable of doing 2x1Gbit links @ standard 1500 MTUs without a hitch, even if i were sending packets through it constantly 24x7. There is nothing else connected to the 2900-24G.
    - I'm using the 3500yl also to power on the APs, but I've tried using POE injectors instead directly to the MSM720 with very similar results.
    - The config of the 3500yl is literally 2 untagged VLANs (ports 1-12; POE enabled and ports13-48; POE disabled) post factory-reset and that is all.
    - The config of the MSM720 is 1VSC, 6APs, 3-4WLANs (each with its own DHCP and CIDR for client isolation) and no RADIUS authentication nor captive portal setup.
    - The 2900 is post factory reset, without any configuration whatsoever.
    - All APs are post factory reset and are in sync with the controller;

    I'm happy to run whatever tests you can point me to in order to figure out where's the choke in the setup above. Everything should be capable of well over what I'm using atm, so I suspect I might've screwed the pooch with the config at one place or another.


  • 2.  RE: MSM720 + 3500yl + HP560s performance issues

    Posted Jan 30, 2023 11:28 AM

    This is the MSM configuration guide for v6.6.2. On page 73 you can find chapter Port Trunking which explains the detail the configuration of link-aggregation (trunking) with some examples

    You wrote: can't get more than 160-170Mbits even though the 560's are capable of 1300Mbits 802.11ac on paper. If I were to connect to the ISP router (some vodafone-branded box, dubbed "gigabox" - still only 802.11ac, though) I get upwards of 550Mbits without a hitch, so it must be an issue with the rest of the network

    What exactly is 160/170Mbps. Is this the data rate displayed by the WLAN client utility? Or it is the result of a throughput test done via speedtest.net or maybe iPerf?
    What are the capabilities of the device used for the test? WiFi standard, spatial streams, channel-width?
    Was it connected to 2.4GHz or 5GHz?
    Wat is the channel width you are using in 2.4 and 5GHz?
    How exactly were measured the 550Mbps when connected to the ISP router? Did you connect using a cable or also wifi? If WiFi was this date rate or the result of a throughput test? For the wifi test did you use the same device which you used to test the 560 APs?

    If 160/170Mbps is real throughput than this can be normal depending on the device type. The 560 has a maximum data rate of 1300Mbps but this is  under the best conditions which are not realistic and also assuming that the client device has the same capabilities. You need to use 80MHz channels (which is not recommended in 5GHz in enterprise environment) and the end device has also to support 80MHz, 3 spatial streams. The device has to be very close to the AP because otherwise it cannot use 256 QAM which is required for 1300Mbps.  Most client devices support 1 or 2 streams and will achieve max data rates coming near the figures you provided.
    1300 is the max rate of the 5GHz radio because 802.11ac is a 5GHz technology. The 2.4 GHz radio is still working with 802.11n and has a max data rate of 450Mbps (using 40MHz channels which is also not recommended)
    You are using a switch which supports most probably the first PoE Implementation 802.3af. According to specs of the AP.

    • With 802.3af PoE, the 2.4 GHz radio will operate in 2 x 2:2 MIMO mode < 12.9 W.
    • With 802.3at PoE+, Both radios will operate in 3 x 3:3 MIMO mode < 14 W.
    That means the 2.4GHz radio will only support 2 spatial streams which will reduce the max achievable data rate further.

    Data rate in WLAN is generally not equal to real throughput. It is assumed that the real throughput measured with testing tools can be between 50 and 60% of the data rate. 

    I am not sure if it would be possible to test the internet speed from the MSM720. Maybe you can connect a laptop directly to port 5 (in the Internet network) have it obtain an IP from the ISP router or configure it with a static IP and perform a speed test. The result could tell you if the problem is in the path between the MSM and the ISP router or between the MSM720 and the wireless clients.

  • 3.  RE: MSM720 + 3500yl + HP560s performance issues

    Posted Jan 31, 2023 09:29 AM
    Hi Emil, thanks for replying and looking into this!

    To answer your questions and address your statements:
    - I believe I've been through the guide, but I'll have another look at the trunking section. Thanks!
    - All speeds stated above are from speedtests. I ran a 100M iperf test between a phone (moto g100) situated abt a foot from one of the APs and a computer plugged into the 3500yl and got 130Mbits on average. I'm happy to run a longer test if you think that would help us more. For the record the speedtest the computer got roughly at the same time was 650/100Mbits (the ISP service is 1000/100Mbps).
    - the wireless devices I used for tests are:
      - moto g100 - it's ax-capable, but I think the ac-band is capped at 866Mbps, not sure about the channel widths and spatial streams, though it should be MIMO-capable, so at least 2 streams.
      - iPhone 13 Max Pro - https://support.apple.com/en-ie/guide/deployment/dep268652e6c/web
      - 16" MacBookPro '21 M1Max - https://support.apple.com/en-ie/guide/deployment/dep2ac3e3b51/web
    - I'm using as far as I can say only 5GHz (or rather the controller switches the devices if possible to 5GHz automagically, though it is transparent to me)
    - As far as I can say it uses auto-channel width:

    I'm unsure how to select a static channel width, as the UI allows me to set up only auto-values:
    Here is the complete config of the radios:
    - The ISP router test was a speedtest, through wifi, from the moto g100, capped at 866Mbps (I assume this is the max the radio of the phone is capable of), so yes, same device, same distance from AP (roughly 1-2ft).
    - I'm aware that the specs on paper are seldom achievable in real world scenarios, so I'm not looking to get anywhere near that. The thing is, I replace a consumer-grade TP-Link One Mesh (Archer C7 router + 1300Mbps PLC adapters with APs) on which I was getting 150Mbps, so I'm disappointed an enterprise setup gives us slightly higher than that throughput. I would be happy with 4-500Mbps or anywhere near that, but 170Mbps is just too low and considering I'm not proficient with the devices, I'm sure I messed up some of the configurations.
    - Correct, the 3500yl is 802.3af capable, not 802.3at. To prove a point I isolated the 3500yl, by dropping two APs and connecting 4 directly to the MSM720 through the suppoted POE injectors (J9867B), but hadn't realized they were 802.3af too. That said, the throughput dropped to 80-90Mbps in this setup. I reverted the config to using the 3500yl again and it went back up to 170Mbps.
    - 50-60% of the datarate would be brilliant - 50% of 866Mbps (the max link rate the clients seem to be capable of) is still 2.5 times higher than what I'm getting otherwise @5GHz.  
    - I didn't test the internet speed from the MSM720, but I tested it from the 3500yl as stated above and I got 650/100. I will try testing from the MSM720 too.

    Thanks again and looking forward to your further advices.

  • 4.  RE: MSM720 + 3500yl + HP560s performance issues

    Posted Jan 31, 2023 02:08 PM
    The MSM720 is setup as follows: VLAN 1 (Internet network) - ports 5&6 untagged, port 1 through 4 - (Access network) untagged;
    The uplink from the 3500yl is on port 5, downlink to the APs is port 4;

    I tried with the computer hooked up to port 6 (Internet network) - speedtest averages 820/100, I'm assuming it's because the MSM is not doing any routing.
    The computer hooked up to port 1 (Access net) - 150/100
    Hooked up to 3500yl in the port group for APs - 150/100
    Decided that may be the dual-personality ports 5 &6 may be are somehow linked to the ASIC with higher throughput, so moved port 6 to the access net vlan - averaged again 150/100.

    Every SSID in the VSC does also DHCP and in essence some form of routing and wonder whether that is the culprit? I'll set up a dummy network with no DHCP and test the speed on it too.

  • 5.  RE: MSM720 + 3500yl + HP560s performance issues

    Posted Jan 31, 2023 02:17 PM
    I tried that - averaged 150/100 again. Here is the VSC profile I created for the test:

  • 6.  RE: MSM720 + 3500yl + HP560s performance issues

    Posted Jan 31, 2023 03:27 PM
    Hello Vladislav, 
    Yes, the tests look OK. So to summarize:  you have good throughput when you are connected by cable to the Internet Network on the MSM720 and bad throughput when connected to ports of the Access Network. So the issue should lie within the controller and be caused by the routing, access control or firewall functions of the MSM720.

    For your test you created an SSID without DHCP however as far as I can see it is still a VSC which requires the APs to tunnel the traffic to the controller and the controller to route it out of the internet network.

    A better test would be to bypass the controller entirely and make the APs bridge the wireless traffic directly to the 3500. But of course you will need to have an external DHCP server and configure routing on the 3500 or the ISP router. The AP port of the 3500 should also be tagged with the egress client VLAN.
    You can check the configuration guide but as far as I remember you achieve this by disabling "Use Controller for Access Control" in the VSC menu. Then in the VSC binding you specify the egress VLAN. This type of deployment is called distributed forwarding (also local breakout) and it was the recommended deployment for employee access. The VSC with User Controller for Access Control were primarily intended for guest access.

    Similar test can be done if you switch one or several APs to autonomous mode. This will also allow you to exclude the MSM720 from the datapath.

    Maybe you should check if the controller is configured with some bandwidth restrictions, for example under Controller >> Network > Bandwidth control. Or check if disabling the firewall (Controller >> Security > Firewall) will affect the throughput. Or some other features which I have already forgotten.

    There are also some things in the wireless configuration but since your test is showing that even wired clients in the Access Network have the issue, I don't think that they are causing it.

    I understand your disappointment that enterprise equipment is not meeting your expectations. But you have to keep in mind that all of the listed equipment is at least 10 years old. To my knowledge all the devices are declared end of sales and end of support by HPE. Some of them may have many years of productive operation and HW issues also cannot be excluded.

  • 7.  RE: MSM720 + 3500yl + HP560s performance issues

    Posted Jan 31, 2023 05:32 PM
    Hi Emil,

    Yeah, I'm completely aware of the caveats of running old enterprise equipment in my homelab and that is fine. Again - I'm not delusional and hoping for the RF testing room speeds, but a fraction of what they're supposed to be capable of. If I get 3-400Mbps I'd be happy as a clam.

    Now with that out of the way, I tried disabling the firewall and saw no change. Figured any sort of packet inspection or manipulation would reduce throughput, so turned off IDS too and saw marginal to no improvement. And there is no bandwidth restriction in the menu you mentioned.

    I have another 4-5 APs lying around, so I will try an autonomous mode AP tomorrow. If you can point me to a guide for the distributed setup, I'd be happy to try that too.

    In the meantime I set the testnet vsc to not use the controller for neither access control, nor for auth and even though the averages are relatively the same, it started peaking to abt 210Mbps downlink, so I feel we're defo on the right path.


  • 8.  RE: MSM720 + 3500yl + HP560s performance issues

    Posted Feb 09, 2023 06:29 PM

    Hi Emil, 

    so some great news - finally got the time to dig through settings again and managed to disable access control globally (after first disabling it on all VSCs manually) and post-sync the throughput via speedtest jumped to a solid 300-350-380Mbps down, 100 up (my uplink is capped at 1G down/100Mbit up, so the ISP is clipping my wings there). It does occasionally spike to 400-430Mbps down, though I'm assuming this is hitting some CDN or cache partially and then fetching resources from the original server or something. Either way, I couldn't be happier. Especially since the link speeds are capped at 780-866Mbps, so basically 50% of the max throughput.

    I've got a couple more questions that probably won't be a clearcut yes/no's so I hope you'd be able to give me your "best guess" based on your experience: 

    • As we discussed the POE switch is just 802.3af and caps out at 13W per device. I could replace it for a smaller 802.3at one to unblock that extra watt or two out of the built-in injector, but the question is would this help in any way in reality? Like, if I read the specs of the 560's correctly the only downside of using the af-standard is that the 2.4GHz radio works at 2x2:2 MIMO instead of it's full 3x3:3, but the 2.4GHz radio is capped at 802.11n, which I guess means max 450Mbps link speed, so realistically I don't think I'll ever get any higher practical throughput. Is there any point in looking for like a 2910 or something similar, besides potentially cutting some on the electricity bill (the 3500yl idles at like 350watts when POE is enabled...)
    • I'm surprized how much the throughput increased by dropping the access control entirely from the MSM720.
      I'm wondering if I were to set up a RADIUS server (which probably means I'd need to set up a full-time running AD domain with an NPS and all that shebang supporting the RADIUS) but if I did, would there be any reason the throughput could increase any further? I might try it out in the future regardless so I play again with the tech from back in the day of my MCSEs, but nonetheless knowing it might have some real benefit could motivate me to do so sooner rather than later.
    • To the previous point I noticed there is an option for fast wireless roaming, which, however, requires me to set the key on the VSCs to be dynamic, which means I guess RADIUS or AD+NPS. Is there any other way to set up dynamic key and not have neither RADIUS nor AD running for simplicity's sake?

    • All things said and done, I'm still not entirely sure my setup is currently a local breakout or distirbuted - does it suffice to just not use the controller for access control? If I understand correctly using the controller for AC forces traffic to be routed through the controller, which in turn capped the throughput? Or am I missing something else?

    Thanks again for all your help!



  • 9.  RE: MSM720 + 3500yl + HP560s performance issues

    Posted Feb 12, 2023 06:03 AM

    Hello Vladislav, <o:p></o:p>

    Regarding 802.3af PoE. <o:p></o:p>

    Yes, according to the specifications this the only restriction. With 3x3:3 MIMO you can achieve a maximum data rate of 450Mbps however only if you are using 40MHz channels. With 2x2:2 MIMO and 40MHz channels the max data rate is 300Mbps.<o:p></o:p>

    It is an industry accepted best practice to not use 40MHz in 2.4GHz band in enterprise environment because there are only 3x non-overlapping 20MHz channels and bonding any 2 of them causes co-channel interference (CCI). In home environments 40MHz channels in 2.4GHz are mostly used although CCI can also be an issue and will also affect the throughput. <o:p></o:p>

    When using 20MHz channels and 3 spatial streams (3x3:3) the maximum data rate is 216Mbps and when using 20MHz and 2 spatial streams (2x2:2) it is 144Mbps. Again, this are data rates, and the real throughput is only about 50 to max 70 percent of the data rate. So if you want/need to use 20MHz channels there will be a throughput cap. With 40MHz channels the data rate may be acceptable but the throughput may be affected by CCI.<o:p></o:p>

    I think it will depend on the capabilities of your client population; do you have a lot of 2.4GHz only clients. How many spatial streams do they support. The majority of the client devices have a 2x2:2 radios. Only high-end enterprise class laptops may have 3x3:3 but they will also for sure also support 5GHz.<o:p></o:p>

    <o:p> </o:p>

    Would the throughput increase if you setup an external RADIUS server? <o:p></o:p>

    Assuming that you are using WPA2-Personal (called Preshared Key here) now and Access Control is disabled so that the APs are bridging the traffic directly to the switch, there will be no increase of the throughput when using 802.1x and RADIUS server instead. There will be certainly a huge increase of the security level and that's why this mode is called WPA2-Enterprise. The forwarding path will remain the same. <o:p></o:p>

    With 802.1x and RADIUS server, the encryption keys (PMKs) are generated dynamically at the client and RADIUS as a byproduct of the 801.1x EAP authentication. For every user and every new authentication new dynamic keys are generated. With WPA2-Preshared Key on the contrary, the controller and all the clients must be preconfigured with the same static preshared key. This introduces the risk that the preshared key can be compromised. Of course dynamic pairwise transient keys are also created for every new association but they are based on the static PMK. That's why with WPA2-Preshared key, captured wireless traffic can be decrypted if the intruder knows the static preshared key and can capture the 4 way handshake. This is not possible or much more difficult with 802.1x/RADIUS. But as I said, this will not affect the throughput in any way. The encryption by itself is the same it is WPA2 AES.<o:p></o:p>

    Is there any other way to set up dynamic key and not have neither RADIUS nor AD running for simplicity's sake?<o:p></o:p>

    As explained above the EAP protocol requires the interaction with an EAP capable RADIUS server and as a by-product of this interaction (apart from authentication) dynamic encryption keys are created. If I remember correctly the controller has also an internal user database which can also be used for EAP. However I am not sure if it can work without "Use Controller for Access Control". "Use controller for Authentication" has to be enabled for sure. Maybe you can test if you are curious and have time.<o:p></o:p>

    But I don't see any benefits for the throughput by enabling "Fast wireless roaming/WPA2 opportunistic key caching". This option is only useful when using 802.1x/RADIUS and not intended for PSK. The time that it takes for a wireless client to connect is different with WPA2 PSK and WPA2 Enterprise. With WPA2-PSK there is a limited number of frames which are exchanged only between AP and client, and it takes less than 50ms.  With WPA2 Enterprise, the client starts an EAPOL multiple frame exchange with the AP, then the AP (or the controller) starts a RADIUS interaction with a RADIUS server. These exchanges can take about 700ms to more than 1 second (depending on latency to the RADIUS server). If a client has to do the full EAP/RADIUS exchange with every AP it roams to, this will severely affect the roaming because there is no traffic for about 1 second and time sensitive application will be affected. The goal of WPA2 opportunistic key caching is to avoid the full RADIUS authentication with every AP while roaming. The client must perform only a single full authentication, then the controller will distribute the key created at the first authentication to all APs and subsequent roams to other APs will require only a short handshake instead of the full authentication. <o:p></o:p>

    WPA2-PSK has by default connection and roaming times which are similar to the roaming times with WPA2 opportunistic key caching. So I don't see any increase in throughput or even in roaming by using WPA2 with 802.1x/RADIUS instead of WPA2 Preshared Key. The only increase would be in security but of course it also adds complexity.<o:p></o:p>

    I'm still not entirely sure my setup is currently a local breakout or distributed?<o:p></o:p>

    Based on your description the traffic should be distributed now. All traffic of wireless clients is decrypted and translated to 802.3 Ethernet frames by the AP. The AP then forwards the Ethernet frames directly to the switch where it is switched or routed. <o:p></o:p>

    Maybe you can connect a client to a certain AP, note the wireless MAC of the client and check with show mac-address <wireless-mac> in the switch CLI on which port exactly the MAC was learned. If it is the port where the AP is connected, this confirms that we have distributed forwarding. In centralized forwarding (use controller for AC) the MAC should appear on the MSM720 internet port.<o:p></o:p>

    You could even configure a port-mirroring and mirror all the traffic of the AP port to another switch port. If you connect a laptop with Wireshark to the other port, you would be able to see if the packets of the client are captured there. You should be able to see DHCP, ARP, DNS etc.<o:p></o:p>

    With "use controller for Access Control" basically the AP is tunneling all wireless frames to the controller. The controller is doing decryption, translation to Ethernet and routing or switching.<o:p></o:p>