Wireless Access

• With 802.3af PoE, the 2.4 GHz radio will operate in 2 x 2:2 MIMO mode < 12.9 W.
• With 802.3at PoE+, Both radios will operate in 3 x 3:3 MIMO mode < 14 W.

Original Message:
Sent: Jan 29, 2023 06:55 PM
From: tachev
Subject: MSM720 + 3500yl + HP560s performance issues

Hi there,

I recently installed the following configuration:

The ISP is a fiber gigabit, which goes through a router (temporary - will be replaced by a Checkpoint 4400 running pfsense pretty soon), then goes into the default VLAN of a 3500yl-48G-POE, which then goes into the Internet network (ports 5&6) of an MSM720 Premium (with just the two networks, one VSC and a few SSIDs), which then goes to the Access Network (ports 1 through 4) and through a cable into a second untagged VLAN of the 3500yl, which then I use to power on a bunch of HP560 APs. Almost all cabling is CAT6 SFTP to reduce RF and power cabling interference. I get pretty solid coverage across the building, though the speed is dubiously slow - can't get more than 160-170Mbits even though the 560's are capable of 1300Mbits 802.11ac on paper. If I were to connect to the ISP router (some vodafone-branded box, dubbed "gigabox" - still only 802.11ac, though) I get upwards of 550Mbits without a hitch, so it must be an issue with the rest of the network.

Now the caveats that I should point are (and if anyone has advice how to fix those, that'd be terrific):
- I couldn't trunk the ports between the 3500yl and the msm720 - I wanted to create either a static trunk or an LACP bond to ensure I have higher throughput, but for some reason the two switches just don't allow me to setup a bond, and if I were to set up a static trunk the link breaks - I believe it detects the multiple links as an IP loop and just disables the port. If I unplug three of the 4 cables, everything starts working again.
- I am not exceeding the AP licenses I have on the MSM720 - I have license for 10, I'm using only 6 so far.
- There is also a 2900-24G between the 3500yl and the ISP router, though, I'm using it only as a pass-through. Happy to remove it if it's the culprit, though it should be capable of doing 2x1Gbit links @ standard 1500 MTUs without a hitch, even if i were sending packets through it constantly 24x7. There is nothing else connected to the 2900-24G.
- I'm using the 3500yl also to power on the APs, but I've tried using POE injectors instead directly to the MSM720 with very similar results.
- The config of the 3500yl is literally 2 untagged VLANs (ports 1-12; POE enabled and ports13-48; POE disabled) post factory-reset and that is all.
- The config of the MSM720 is 1VSC, 6APs, 3-4WLANs (each with its own DHCP and CIDR for client isolation) and no RADIUS authentication nor captive portal setup.
- The 2900 is post factory reset, without any configuration whatsoever.
- All APs are post factory reset and are in sync with the controller;

I'm happy to run whatever tests you can point me to in order to figure out where's the choke in the setup above. Everything should be capable of well over what I'm using atm, so I suspect I might've screwed the pooch with the config at one place or another.

Cheers,
V

Hi Emil, 

so some great news - finally got the time to dig through settings again and managed to disable access control globally (after first disabling it on all VSCs manually) and post-sync the throughput via speedtest jumped to a solid 300-350-380Mbps down, 100 up (my uplink is capped at 1G down/100Mbit up, so the ISP is clipping my wings there). It does occasionally spike to 400-430Mbps down, though I'm assuming this is hitting some CDN or cache partially and then fetching resources from the original server or something. Either way, I couldn't be happier. Especially since the link speeds are capped at 780-866Mbps, so basically 50% of the max throughput.

I've got a couple more questions that probably won't be a clearcut yes/no's so I hope you'd be able to give me your "best guess" based on your experience: 

• As we discussed the POE switch is just 802.3af and caps out at 13W per device. I could replace it for a smaller 802.3at one to unblock that extra watt or two out of the built-in injector, but the question is would this help in any way in reality? Like, if I read the specs of the 560's correctly the only downside of using the af-standard is that the 2.4GHz radio works at 2x2:2 MIMO instead of it's full 3x3:3, but the 2.4GHz radio is capped at 802.11n, which I guess means max 450Mbps link speed, so realistically I don't think I'll ever get any higher practical throughput. Is there any point in looking for like a 2910 or something similar, besides potentially cutting some on the electricity bill (the 3500yl idles at like 350watts when POE is enabled...)
• I'm surprized how much the throughput increased by dropping the access control entirely from the MSM720.
  I'm wondering if I were to set up a RADIUS server (which probably means I'd need to set up a full-time running AD domain with an NPS and all that shebang supporting the RADIUS) but if I did, would there be any reason the throughput could increase any further? I might try it out in the future regardless so I play again with the tech from back in the day of my MCSEs, but nonetheless knowing it might have some real benefit could motivate me to do so sooner rather than later.
• To the previous point I noticed there is an option for fast wireless roaming, which, however, requires me to set the key on the VSCs to be dynamic, which means I guess RADIUS or AD+NPS. Is there any other way to set up dynamic key and not have neither RADIUS nor AD running for simplicity's sake?
  
  
  
  
  
• All things said and done, I'm still not entirely sure my setup is currently a local breakout or distirbuted - does it suffice to just not use the controller for access control? If I understand correctly using the controller for AC forces traffic to be routed through the controller, which in turn capped the throughput? Or am I missing something else?

Thanks again for all your help!

Cheers,

Vlad

Hello Vladislav,

Regarding 802.3af PoE.

Yes, according to the specifications this the only restriction. With 3x3:3 MIMO you can achieve a maximum data rate of 450Mbps however only if you are using 40MHz channels. With 2x2:2 MIMO and 40MHz channels the max data rate is 300Mbps.

It is an industry accepted best practice to not use 40MHz in 2.4GHz band in enterprise environment because there are only 3x non-overlapping 20MHz channels and bonding any 2 of them causes co-channel interference (CCI). In home environments 40MHz channels in 2.4GHz are mostly used although CCI can also be an issue and will also affect the throughput.

When using 20MHz channels and 3 spatial streams (3x3:3) the maximum data rate is 216Mbps and when using 20MHz and 2 spatial streams (2x2:2) it is 144Mbps. Again, this are data rates, and the real throughput is only about 50 to max 70 percent of the data rate. So if you want/need to use 20MHz channels there will be a throughput cap. With 40MHz channels the data rate may be acceptable but the throughput may be affected by CCI.

I think it will depend on the capabilities of your client population; do you have a lot of 2.4GHz only clients. How many spatial streams do they support. The majority of the client devices have a 2x2:2 radios. Only high-end enterprise class laptops may have 3x3:3 but they will also for sure also support 5GHz.

Would the throughput increase if you setup an external RADIUS server?

Assuming that you are using WPA2-Personal (called Preshared Key here) now and Access Control is disabled so that the APs are bridging the traffic directly to the switch, there will be no increase of the throughput when using 802.1x and RADIUS server instead. There will be certainly a huge increase of the security level and that's why this mode is called WPA2-Enterprise. The forwarding path will remain the same.

With 802.1x and RADIUS server, the encryption keys (PMKs) are generated dynamically at the client and RADIUS as a byproduct of the 801.1x EAP authentication. For every user and every new authentication new dynamic keys are generated. With WPA2-Preshared Key on the contrary, the controller and all the clients must be preconfigured with the same static preshared key. This introduces the risk that the preshared key can be compromised. Of course dynamic pairwise transient keys are also created for every new association but they are based on the static PMK. That's why with WPA2-Preshared key, captured wireless traffic can be decrypted if the intruder knows the static preshared key and can capture the 4 way handshake. This is not possible or much more difficult with 802.1x/RADIUS. But as I said, this will not affect the throughput in any way. The encryption by itself is the same it is WPA2 AES.

Is there any other way to set up dynamic key and not have neither RADIUS nor AD running for simplicity's sake?

As explained above the EAP protocol requires the interaction with an EAP capable RADIUS server and as a by-product of this interaction (apart from authentication) dynamic encryption keys are created. If I remember correctly the controller has also an internal user database which can also be used for EAP. However I am not sure if it can work without "Use Controller for Access Control". "Use controller for Authentication" has to be enabled for sure. Maybe you can test if you are curious and have time.

But I don't see any benefits for the throughput by enabling "Fast wireless roaming/WPA2 opportunistic key caching". This option is only useful when using 802.1x/RADIUS and not intended for PSK. The time that it takes for a wireless client to connect is different with WPA2 PSK and WPA2 Enterprise. With WPA2-PSK there is a limited number of frames which are exchanged only between AP and client, and it takes less than 50ms.  With WPA2 Enterprise, the client starts an EAPOL multiple frame exchange with the AP, then the AP (or the controller) starts a RADIUS interaction with a RADIUS server. These exchanges can take about 700ms to more than 1 second (depending on latency to the RADIUS server). If a client has to do the full EAP/RADIUS exchange with every AP it roams to, this will severely affect the roaming because there is no traffic for about 1 second and time sensitive application will be affected. The goal of WPA2 opportunistic key caching is to avoid the full RADIUS authentication with every AP while roaming. The client must perform only a single full authentication, then the controller will distribute the key created at the first authentication to all APs and subsequent roams to other APs will require only a short handshake instead of the full authentication.

WPA2-PSK has by default connection and roaming times which are similar to the roaming times with WPA2 opportunistic key caching. So I don't see any increase in throughput or even in roaming by using WPA2 with 802.1x/RADIUS instead of WPA2 Preshared Key. The only increase would be in security but of course it also adds complexity.

I'm still not entirely sure my setup is currently a local breakout or distributed?

Based on your description the traffic should be distributed now. All traffic of wireless clients is decrypted and translated to 802.3 Ethernet frames by the AP. The AP then forwards the Ethernet frames directly to the switch where it is switched or routed.

Maybe you can connect a client to a certain AP, note the wireless MAC of the client and check with show mac-address <wireless-mac> in the switch CLI on which port exactly the MAC was learned. If it is the port where the AP is connected, this confirms that we have distributed forwarding. In centralized forwarding (use controller for AC) the MAC should appear on the MSM720 internet port.

You could even configure a port-mirroring and mirror all the traffic of the AP port to another switch port. If you connect a laptop with Wireshark to the other port, you would be able to see if the packets of the client are captured there. You should be able to see DHCP, ARP, DNS etc.

With "use controller for Access Control" basically the AP is tunneling all wireless frames to the controller. The controller is doing decryption, translation to Ethernet and routing or switching.