Controllerless Networks

Reply
Frequent Contributor I

Client throughput capped at 40M instead of +200M occationally

A customer of mine have reported odd performance issues running with IAP-305 or IAP-315 on 5Ghz, 40Mhz VHT channels.

 

In an attempt to reproduce the problem I've deployed a single IAP-315 connected to a 2530-24G-PoEP switch. Connected to the switch is a wired desktop running Linux and iperf3 in server mode.

 

The IAP broadcasts a single SSID "MathiasPSK" on 5GHz, 40MHz and VHT enabled, and there are no other 5GHz APs or devices in the neighbourhood:

 

a8:bd:27:c0:44:fc# show ap spectrum channel-details

Channel Summary Table
---------------------
Channel  Quality(%)  Utilization(%)  WiFi(%)  Bluetooth(%)  Microwave(%)  Cordless Phone(%)  Total nonwifi(%)  KnownAPs  UnknownAPs  Noise Floor(dBm)  MaxAPSignal(dBm)  Max AP SSID  Max AP BSSID  MaxInterference(dBm)  SNIR(dB)
-------  ----------  --------------  -------  ------------  ------------  -----------------  ----------------  --------  ----------  ----------------  ----------------  -----------  ------------  --------------------  --------
52+      100         0               0        0             0             0                  0                 1         0           -94               -                 -            -             -                     0

While only having a single laptop or two connected, I can both send and receive 200+ Mbps from my laptop "IT-089":

 

C:\Users\mathias.sundman\Downloads\iperf-3.1.3-win64>iperf3 -c 192.168.88.99
Connecting to host 192.168.88.99, port 5201
[  4] local 192.168.88.60 port 50537 connected to 192.168.88.99 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  25.5 MBytes   213 Mbits/sec
[  4]   1.00-2.00   sec  22.6 MBytes   190 Mbits/sec
[  4]   2.00-3.01   sec  24.6 MBytes   205 Mbits/sec
[  4]   3.01-4.00   sec  24.2 MBytes   204 Mbits/sec


C:\Users\mathias.sundman\Downloads\iperf-3.1.3-win64>iperf3 -c 192.168.88.99 -R
Connecting to host 192.168.88.99, port 5201
Reverse mode, remote host 192.168.88.99 is sending
[  4] local 192.168.88.60 port 49826 connected to 192.168.88.99 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  27.3 MBytes   229 Mbits/sec
[  4]   1.00-2.00   sec  29.4 MBytes   247 Mbits/sec
[  4]   2.00-3.00   sec  29.7 MBytes   249 Mbits/sec
[  4]   3.00-4.00   sec  29.3 MBytes   246 Mbits/sec

The client-table while everything is good looks like this:

a8:bd:27:c0:44:fc# show clients

Client List
-----------
Name             IP Address     MAC Address        OS       ESSID       Access Point       Channel  Type  Role        IPv6 Address               Signal    Speed (mbps)
----             ----------     -----------        --       -----       ------------       -------  ----  ----        ------------               ------    ------------
IT-089           192.168.88.60  a4:34:d9:63:f9:0c  Windows  MathiasPSK  a8:bd:27:c0:44:fc  52+      AC    MathiasPSK  fe80::7455:924e:6276:4868  62(good)  400(good)
Mathiass-MBP     192.168.88.80  f4:0f:24:19:b0:f9  Win XP   MathiasPSK  a8:bd:27:c0:44:fc  52+      AC    MathiasPSK  --                         47(good)  600(good)

a8:bd:27:c0:44:fc# show ap debug client-table

Client Table
------------
MAC                ESSID       BSSID              Assoc_State  HT_State    AID  PS_State    UAPSD            Tx_Pkts  Rx_Pkts  PS_Qlen  Tx_Retries  Tx_Rate  Rx_Rate  Last_ACK_SNR  Last_Rx_SNR  TX_Chains  Tx_Timestamp              Rx_Timestamp              MFP Status (C,R)  Idle time  Client health (C/R)  Tx_Bytes   Rx_Bytes
---                -----       -----              -----------  --------    ---  --------    -----            -------  -------  -------  ----------  -------  -------  ------------  -----------  ---------  ------------              ------------              ----------------  ---------  -------------------  --------   --------
f4:0f:24:19:b0:f9  MathiasPSK  a8:bd:27:84:4f:d0  Associated   AWvSsEeBbM  0x1  Power-save  (0,0,0,0,N/A,0)  243115   205125   0        23          600      48       48            51           4[0xf]     Thu Apr 13 22:03:43 2017  Thu Apr 13 22:03:43 2017  (0,0)             29         100/93               480550541  529986917
a4:34:d9:63:f9:0c  MathiasPSK  a8:bd:27:84:4f:d0  Associated   AWvSsEBbM   0x2  Awake       (0,0,0,0,N/A,0)  39       219      0        0           400      400      67            55           4[0xf]     Thu Apr 13 22:04:06 2017  Thu Apr 13 22:04:06 2017  (0,0)             6          100/93               15574      25012

A little later, most likely after I had joined also my iPhone to the network, my laptop "IT-089" suddenly can only transmit about 40Mbps. It can still receive +200M though:

 

C:\Users\mathias.sundman\Downloads\iperf-3.1.3-win64>iperf3 -c 192.168.88.99
Connecting to host 192.168.88.99, port 5201
[  4] local 192.168.88.60 port 50684 connected to 192.168.88.99 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  4.88 MBytes  40.8 Mbits/sec
[  4]   1.00-2.00   sec  5.12 MBytes  43.0 Mbits/sec
[  4]   2.00-3.00   sec  5.12 MBytes  43.0 Mbits/sec
[  4]   3.00-4.00   sec  5.12 MBytes  43.0 Mbits/sec

C:\Users\mathias.sundman\Downloads\iperf-3.1.3-win64>iperf3 -c 192.168.88.99 -R
Connecting to host 192.168.88.99, port 5201
Reverse mode, remote host 192.168.88.99 is sending
[  4] local 192.168.88.60 port 50754 connected to 192.168.88.99 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  24.0 MBytes   201 Mbits/sec
[  4]   1.00-2.00   sec  25.5 MBytes   214 Mbits/sec
[  4]   2.00-3.00   sec  25.3 MBytes   213 Mbits/sec
[  4]   3.00-4.00   sec  24.9 MBytes   208 Mbits/sec

Examining the client-table again still says that IT-089 has a Tx_Rate and Rx_Rate of 400:

 

a8:bd:27:c0:44:fc# show clients

Client List
-----------
Name             IP Address     MAC Address        OS       ESSID       Access Point       Channel  Type  Role        IPv6 Address               Signal    Speed (mbps)
----             ----------     -----------        --       -----       ------------       -------  ----  ----        ------------               ------    ------------
Mathiass-iPhone  192.168.88.77  b4:8b:19:db:38:b1           MathiasPSK  a8:bd:27:c0:44:fc  52+      AC    MathiasPSK  --                         56(good)  360(good)
SELT-0046        192.168.88.59  e8:b1:fc:08:39:41  Win 7    MathiasPSK  a8:bd:27:c0:44:fc  52+      AC    MathiasPSK  fe80::a06e:2ac6:fdaa:9aec  56(good)  400(good)
IT-089           192.168.88.60  a4:34:d9:63:f9:0c  Windows  MathiasPSK  a8:bd:27:c0:44:fc  52+      AC    MathiasPSK  fe80::7455:924e:6276:4868  62(good)  400(good)
Mathiass-MBP     192.168.88.80  f4:0f:24:19:b0:f9  Win XP   MathiasPSK  a8:bd:27:c0:44:fc  52+      AC    MathiasPSK  --                         47(good)  600(good)
Number of Clients   :4
Info timestamp      :6737


a8:bd:27:c0:44:fc# show ap association

The phy column shows client's operational capabilities for current association

Flags: A: Active, B: Band Steerable, H: Hotspot(802.11u) client, K: 802.11K client, M: Mu beam formee, R: 802.11R client, W: WMM client, w: 802.11w client V: 802.11v BSS trans capable

PHY Details: HT   : High throughput;      20: 20MHz;  40: 40MHz; t: turbo-rates (256-QAM)
             VHT  : Very High throughput; 80: 80MHz; 160: 160MHz; 80p80: 80MHz + 80MHz
             <n>ss: <n> spatial streams

Association Table
-----------------
Name               bssid              mac                auth  assoc  aid  l-int  essid       vlan-id  tunnel-id  phy              assoc. time  num assoc  Flags  DataReady
----               -----              ---                ----  -----  ---  -----  -----       -------  ---------  ---              -----------  ---------  -----  ---------
a8:bd:27:c0:44:fc  a8:bd:27:84:4f:d0  b4:8b:19:db:38:b1  y     y      4    20     MathiasPSK  1        0x0        a-VHT-40sgi-2ss  1m:53s       1          WV     Yes (Implicit)
a8:bd:27:c0:44:fc  a8:bd:27:84:4f:d0  e8:b1:fc:08:39:41  y     y      3    250    MathiasPSK  1        0x0        a-VHT-40sgi-2ss  23s          1          W      Yes (Implicit)
a8:bd:27:c0:44:fc  a8:bd:27:84:4f:d0  a4:34:d9:63:f9:0c  y     y      2    250    MathiasPSK  1        0x0        a-VHT-40sgi-2ss  14s          1          W      Yes (Implicit)
a8:bd:27:c0:44:fc  a8:bd:27:84:4f:d0  f4:0f:24:19:b0:f9  y     y      1    10     MathiasPSK  1        0x0        a-VHT-40sgi-3ss  53m:54s      1          W      Yes (Implicit)
Num Clients:4
a8:bd:27:c0:44:fc# show ap debug client-table

Client Table
------------
MAC                ESSID       BSSID              Assoc_State  HT_State    AID  PS_State    UAPSD            Tx_Pkts  Rx_Pkts  PS_Qlen  Tx_Retries  Tx_Rate  Rx_Rate  Last_ACK_SNR  Last_Rx_SNR  TX_Chains  Tx_Timestamp              Rx_Timestamp              MFP Status (C,R)  Idle time  Client health (C/R)  Tx_Bytes   Rx_Bytes
---                -----       -----              -----------  --------    ---  --------    -----            -------  -------  -------  ----------  -------  -------  ------------  -----------  ---------  ------------              ------------              ----------------  ---------  -------------------  --------   --------
f4:0f:24:19:b0:f9  MathiasPSK  a8:bd:27:84:4f:d0  Associated   AWvSsEeBbM  0x1  Power-save  (0,0,0,0,N/A,0)  245028   209414   0        26          600      540      48            47           4[0xf]     Thu Apr 13 22:29:57 2017  Thu Apr 13 22:29:57 2017  (0,0)             7          97/94                481564301  530766840
b4:8b:19:db:38:b1  MathiasPSK  a8:bd:27:84:4f:d0  Associated   AWvSsEe     0x4  Power-save  (0,0,0,0,N/A,0)  26       347      0        0           360      360      57            56           4[0xf]     Thu Apr 13 22:29:58 2017  Thu Apr 13 22:29:58 2017  (0,0)             6          100/94               4622       3713
e8:b1:fc:08:39:41  MathiasPSK  a8:bd:27:84:4f:d0  Associated   WvSsEbM     0x3  Awake       (1,1,1,1,2,0)    66       140      0        0           400      270      61            56           4[0xf]     Thu Apr 13 22:29:59 2017  Thu Apr 13 22:30:02 2017  (0,0)             2          100/94               21497      55268
a4:34:d9:63:f9:0c  MathiasPSK  a8:bd:27:84:4f:d0  Associated   AWvSsEBbM   0x2  Awake       (0,0,0,0,N/A,0)  10813    37469    0        231         400      400      77            63           4[0xf]     Thu Apr 13 22:30:02 2017  Thu Apr 13 22:30:03 2017  (0,0)             1          90/94                1692430    56257386

How can I further diagnose what is capping the traffic?

 

PS: During the day while troublshooting this behaviour in the customers network the problem moved between different clients (HP Latops, Lenovo laptops, Linux Laptops, Windows 10 laptops) and sometimes it was TX that was capped and sometimes it was RX.

 

PS: I posted this in another thread yesterday, but for some reason I can't see in on the board today. Appolgize if it will now appear twice.

Frequent Contributor I

Re: Client throughput capped at 40M instead of +200M occationally

While performing the same tests today, the problem seemed to start when another laptop with not-so-good signal joined the network. My test-laptop (which is localted 20cm from the AP) was then suddenly capped to 40M upload again.

 

Even after disconnecting all other client, and using "disconnect-user mac xxx" to clean the associations in the AP, my laptop could still only push 40M.

 

Turning off the clients wifi-adater and on again did not resolve the issue.

 

After rebooting the client, it was back to normal again.

 

I've noticed that during the problem when I try to upload from the client, the Tx_Retries value increases quickly:

 

a8:bd:27:c0:44:fc# show ap debug client-table

Client Table
------------
MAC                ESSID       BSSID              Assoc_State  HT_State   AID  PS_State  UAPSD            Tx_Pkts  Rx_Pkts  PS_Qlen  Tx_Retries  Tx_Rate  Rx_Rate  Last_ACK_SNR  Last_Rx_SNR  TX_Chains  Tx_Timestamp              Rx_Timestamp              MFP Status (C,R)  Idle time  Client health (C/R)  Tx_Bytes  Rx_Bytes
---                -----       -----              -----------  --------   ---  --------  -----            -------  -------  -------  ----------  -------  -------  ------------  -----------  ---------  ------------              ------------              ----------------  ---------  -------------------  --------  --------
a4:34:d9:63:f9:0c  MathiasPSK  a8:bd:27:84:4f:d0  Associated   AWvSsEBbM  0x1  Awake     (0,0,0,0,N/A,0)  173611   610345   0        4193        400      400      57            43           4[0xf]     Fri Apr 14 12:56:55 2017  Fri Apr 14 12:56:55 2017  (0,0)             0          92/92                18447895  926498265


a8:bd:27:c0:44:fc# show ap debug client-table

Client Table
------------
MAC                ESSID       BSSID              Assoc_State  HT_State   AID  PS_State  UAPSD            Tx_Pkts  Rx_Pkts  PS_Qlen  Tx_Retries  Tx_Rate  Rx_Rate  Last_ACK_SNR  Last_Rx_SNR  TX_Chains  Tx_Timestamp              Rx_Timestamp              MFP Status (C,R)  Idle time  Client health (C/R)  Tx_Bytes  Rx_Bytes
---                -----       -----              -----------  --------   ---  --------  -----            -------  -------  -------  ----------  -------  -------  ------------  -----------  ---------  ------------              ------------              ----------------  ---------  -------------------  --------  --------
a4:34:d9:63:f9:0c  MathiasPSK  a8:bd:27:84:4f:d0  Associated   AWvSsEBbM  0x1  Awake     (0,0,0,0,N/A,0)  222316   782369   0        5116        400      400      58            47           4[0xf]     Fri Apr 14 12:57:43 2017  Fri Apr 14 12:57:43 2017  (0,0)             0          92/92                23476096  1187675233

Which can also be seen on this graph the IntantUI:

wifi_retransmissions.png

 

Guru Elite

Re: Client throughput capped at 40M instead of +200M occationally

I don't know the exact specifics of your situation, but in general, adding a client that is further away degrades a connection.  The client that is near also will de-rate or lower its speed to attempt to connect better.  It could take some time for that client to up-rate or send frames at higher rates, after.  This all depends on the client and it can vary greatly.

 

A packet capture, looking at the rates as a column might confirm this...



Colin Joseph
Aruba Customer Engineering

Looking for an Answer? Search the Community Knowledge Base Here: Community Knowledge Base

Frequent Contributor I

Re: Client throughput capped at 40M instead of +200M occationally

Thanks for your reply,

 

But if that's the case, shouldn't I see that my close-by client has lowered his Tx or Rx bitrate in the "show ap debug client-table" output?

 

It still says my client is using 400Mbps. I also confirmed with show ap debug client-stats that it is only the 450Mbps counters that are increasing:

 

a8:bd:27:c0:44:fc#  show ap debug client-stats a4:34:d9:63:f9:0c a8:bd:27:84:4f:d0 | inc "Rx Data Bytes"
Rx Data Bytes                    294732358
Rx Data Bytes   12 Mbps  (Mon)   300
Rx Data Bytes   54 Mbps  (Mon)   25800
Rx Data Bytes  108 Mbps  (Mon)   0
Rx Data Bytes  300 Mbps  (Mon)   20518
Rx Data Bytes  450 Mbps  (Mon)   294686040
Rx Data Bytes  1300 Mbps  (Mon)  0
Rx Data Bytes  1300 Mbps+ (Mon)  0
a8:bd:27:c0:44:fc#  show ap debug client-stats a4:34:d9:63:f9:0c a8:bd:27:84:4f:d0 | inc "Rx Data Bytes"
Rx Data Bytes                    299091164
Rx Data Bytes   12 Mbps  (Mon)   300
Rx Data Bytes   54 Mbps  (Mon)   25800
Rx Data Bytes  108 Mbps  (Mon)   0
Rx Data Bytes  300 Mbps  (Mon)   20518
Rx Data Bytes  450 Mbps  (Mon)   299044846
Rx Data Bytes  1300 Mbps  (Mon)  0
Rx Data Bytes  1300 Mbps+ (Mon)  0

a8:bd:27:c0:44:fc#  show ap debug client-stats a4:34:d9:63:f9:0c a8:bd:27:84:4f:d0 | inc "Tx Data Bytes"
Tx Data Bytes Transmitted        3608817
Tx Data Bytes                    4346481
Tx Data Bytes   12 Mbps  (Mon)   0
Tx Data Bytes   54 Mbps  (Mon)   344
Tx Data Bytes  108 Mbps  (Mon)   0
Tx Data Bytes  300 Mbps  (Mon)   2558
Tx Data Bytes  450 Mbps  (Mon)   3605915
Tx Data Bytes  1300 Mbps  (Mon)  0
Tx Data Bytes  1300 Mbps+ (Mon)  0
a8:bd:27:c0:44:fc#  show ap debug client-stats a4:34:d9:63:f9:0c a8:bd:27:84:4f:d0 | inc "Tx Data Bytes"
Tx Data Bytes Transmitted        3739429
Tx Data Bytes                    4497801
Tx Data Bytes   12 Mbps  (Mon)   0
Tx Data Bytes   54 Mbps  (Mon)   344
Tx Data Bytes  108 Mbps  (Mon)   0
Tx Data Bytes  300 Mbps  (Mon)   2558
Tx Data Bytes  450 Mbps  (Mon)   3736527
Tx Data Bytes  1300 Mbps  (Mon)  0
Tx Data Bytes  1300 Mbps+ (Mon)  0

Isn't that proof enough that the client is trying to use the high bitrate, but something else is blocking it causing re-transmissions?

 

PS: Now suddently the client started to cap to 40M again without any other client joinging the network.

 

 

Guru Elite

Re: Client throughput capped at 40M instead of +200M occationally

If you do a packet capture of both channels, and look at the "rate" column in the packet capture, that might give you an idea of what is happening.

 

Also the "rate" in that output is not updated more than once per minute, so it does not give you a realtime idea of what is happening...



Colin Joseph
Aruba Customer Engineering

Looking for an Answer? Search the Community Knowledge Base Here: Community Knowledge Base

Frequent Contributor I

Re: Client throughput capped at 40M instead of +200M occationally

Hmm, I must be missing something.

 

I downloaded the Aruba version of Wireshark (1.4.1) from the support site, and configured it to listen to ARUBA_ERM on port 5555.

 

Then started pcap from the IAP:

a8:bd:27:c0:44:fc# pcap start a8:bd:27:84:4f:d0 192.168.88.54 5555 0 5000

Packet capture has started for pcap-id:3

In wireshart I filter on "wlan" and see loads of stuff, but I'm unable to find the "Rate column" you talk about. I tried modifying the Display Columns and added "IEEE 802.11 TX rate", but that column remains empty for all packets.

 

I also tried the latest official version of Wireshark, 2.2.6, but that didn't decode the Aruba encapsulated packets.

 

What have I missed?

Frequent Contributor I

Re: Client throughput capped at 40M instead of +200M occationally

Using wireshark on my MAC in monitor mode I were able to capture what's going on in the air.

 

AFAICS my client is both sending and receiving the actual data frames (+1500bytes) using a data rate of 400M. Wireshark doesn't show any rate in the column for those packes though, but looking inside them shows a headers saying Data Rate: 400M:

 

Screen Shot 2017-04-14 at 16.13.58.png

 

Screen Shot 2017-04-14 at 16.14.25.png

Guru Elite

Re: Client throughput capped at 40M instead of +200M occationally

You are only seeing the rates seen in management frames.  You should right-click on the line with 400.0 mb/s and left click "Apply as Column"



Colin Joseph
Aruba Customer Engineering

Looking for an Answer? Search the Community Knowledge Base Here: Community Knowledge Base

Frequent Contributor I

Re: Client throughput capped at 40M instead of +200M occationally

Ahh, cool - thanks.

 

So I guess that means we have now concluded that my client *is* sending the data with a rate of 400M so there must be something else capping the traffic, right?

 

Screen Shot 2017-04-14 at 16.48.21.png

Guru Elite

Re: Client throughput capped at 40M instead of +200M occationally

How many of those frames are retries?

 



Colin Joseph
Aruba Customer Engineering

Looking for an Answer? Search the Community Knowledge Base Here: Community Knowledge Base

Search Airheads
cancel
Showing results for 
Search instead for 
Did you mean: