I have a 10Gbps network containing 2 servers (DellPowerEdge 4700r) and a hardware switch ( ARUBA 2930F) in between. The two servers are connected via 10G SFP+ ports to the switch. I send TCP traffic at 10Gbps from one server to another. In the switch configuration, I can either choose to use the OpenFlow pipeline by configuring the OpenFlow instance on the switch or use default L2 switch behavior. When I am using default switch behavior (no OpenFlow), I get a throughput of ~9.4Gbps from one server to another. Also, I don’t see any Rx discards on the ingress port. However, when I use OpenFlow instead (enable OpenFlow instance), and send traffic again, I see a lot of Rx discards on the switch. As a result, the throughput fluctuates between 2Gbps to 8Gbps (RTO hits many times).
Is it the issue with slower packet processing in the OpenFlow pipeline? Has someone else observed it? Does this seem like an issue with the configuration? I have made sure that the controller code and OpenFlow switch configuration is correct.
There are *standard* and *custom* pipelines for OpenFlow. I have been using the *standard* pipeline of OpenFlow. I have read in the documents of the switch here under OpenFlow custom pipeline -> Performance in custom pipeline model (added picture reference as well) about this behavior regarding the OpenFlow *custom* pipeline. It says that in OpenFlow custom pipeline: “packets are dropped at the ingress if the pipeline is busy processing too many packets. You can check Rx discards on the switch to obtain such data”. Maybe a similar issue can arise in the standard pipeline as well. It is known that if packets are processed in software, they won't be processed at the wire-rate, i.e. it would take longer to process them and Rx discards might occur. Now, In the OpenFlow pipeline (both standard and custom) we have an option of choosing where we place our flow rules, either in the software table (SRAM) or hardware table (TCAM). All my flow rules of OpenFlow are in the hardware table (TCAM). Shouldn't they be processed at the line rate now and not show any Rx discards?
The server sending traffic is connected to the switch at port 25, the server receiving the traffic is connected at port 27. The configuration on the switch with the OpenFlow standard pipeline setup is as follows:
#To show running-configuration on the switch:
leaf1(config)# sh running-config
Running configuration:
; JL253A Configuration Editor; Created on release #WC.16.05.0007
; Ver #12:08.1d.9b.3f.bf.bb.ef.7c.59.fc.6b.fb.9f.fc.ff.ff.37.ef:ba
hostname "leaf1"
module 1 type jl253a
max-vlans 2048
snmp-server community "public" unrestricted
openflow
controller-id 1 ip 192.168.0.1 controller-interface vlan 2
instance "l1"
listen-port
member vlan 3
controller-id 1
version 1.3 only
limit software-rate 2000
max-backoff-interval 10
pipeline-model standard-match
packet-in vlan-tagging input-form
enable
exit
enable
exit
vlan 1
name "DEFAULT_VLAN"
no untagged 1-3,11-13,19-28
untagged 4-10,14-18
no ip address
exit
vlan 2
name "c-vlan"
untagged 19-24
ip address 192.168.1.1 255.255.0.0
exit
vlan 3
name "s-vlan"
untagged 1-3,11-13,25-28
ip address 10.1.0.1 255.0.0.0
exit
There are 3 tables in the pipleine: Start table, policy table and sw-table-1. Start table and policy table are in hardware (TCAM), while sw-table-1 is in software. Start table with 1 flow rule, policy table with 5 flow rules and then sw-table-1 with 1 flow rule. All packets go to start table directly. The one flow in start table has a go-to-table rule which forwards all packets to the policy table. I have all my rules in the policy table. Then, the sw-table-1 has a rule which drops the packets. This sw-table-1 is never called in the pipeline.
#To show flow tables in the standard pipeline:
leaf1(config)# sh openflow instance l1 flow-table
OpenFlow Instance Flow Table Information
Table Flow Available Free Miss
ID Table Name Count Flow Count Count Goto Table
----- --------------------- -------- -------------- ------------- -------------
0 Start 1 NA 0 100
100 Policy Table 5 NA 722 200
200 SW Table 1 1 NA 0 *
* Denotes that the pipeline could end here.
#To show flow rules in start table:
leaf1(of-inst-l1)# sh openflow instance l1 flows flow-table 0
OpenFlow Flow Table
Flow 1
Match
Incoming Port : Any Ethernet Type : Any
Source MAC : Any Destination MAC : Any
Source MAC Mask : 000000-000000
Destination MAC Mask : 000000-000000
VLAN ID : Any VLAN Priority : Any
Source IP Address : Any
Destination IP Address : Any
IP Protocol : Any
IP ECN : Any IP DSCP : Any
Source Port : Any Destination Port : Any
Source Port Range : NA
Destination Port Range : NA
TCP Flags : NA
Custom Match One : Any
Custom Match One Mask : Any
Custom Match Two : Any
Custom Match Two Mask : Any
Custom Match Three : Any
Custom Match Three Mask : Any
Custom Match Four : Any
Custom Match Four Mask : Any
Attributes
Priority : 0 Duration : 16 seconds
Hard Timeout : 0 seconds Idle Timeout : 0 seconds
Byte Count : NA Packet Count : NA
Flow Table ID : 0 Controller ID : 1
Cookie : 0x0
Hardware Index: NA
Instructions
Goto Table ID : 100
#To show flow rules in Policy Table:
leaf1(config)# sh openflow instance l1 flows flow-table 100
OpenFlow Flow Table
Flow 1
Match
Incoming Port : 27 Ethernet Type : ARP
Source MAC : Any Destination MAC : Any
Source MAC Mask : 000000-000000
Destination MAC Mask : 000000-000000
VLAN ID : Any VLAN Priority : Any
Source Protocol Address : Any
Target Protocol Address : Any
Source Hardware Address : Any
Source Hardware Address Mask : 000000-000000
Target Hardware Address : Any
Target Hardware Address Mask : 000000-000000
ARP Opcode : Any
IP ECN : Any IP DSCP : Any
Source Port : Any Destination Port : Any
Source Port Range : NA
Destination Port Range : NA
TCP Flags : NA
TCP Mask : NA
Custom Match One : Any
Custom Match One Mask : Any
Custom Match Two : Any
Custom Match Two Mask : Any
Custom Match Three : Any
Custom Match Three Mask : Any
Custom Match Four : Any
Custom Match Four Mask : Any
Attributes
Priority : 1 Duration : 5239 seconds
Hard Timeout : 0 seconds Idle Timeout : 0 seconds
Byte Count : NA Packet Count : 0
Flow Table ID : 100 Controller ID : 1
Cookie : 0x0
Hardware Index: 0
Instructions
Apply Actions
Output : 25
Flow 2
Match
Incoming Port : 25 Ethernet Type : ARP
Source MAC : Any Destination MAC : Any
Source MAC Mask : 000000-000000
Destination MAC Mask : 000000-000000
VLAN ID : Any VLAN Priority : Any
Source Protocol Address : Any
Target Protocol Address : Any
Source Hardware Address : Any
Source Hardware Address Mask : 000000-000000
Target Hardware Address : Any
Target Hardware Address Mask : 000000-000000
ARP Opcode : Any
IP ECN : Any IP DSCP : Any
Source Port : Any Destination Port : Any
Source Port Range : NA
Destination Port Range : NA
TCP Flags : NA
TCP Mask : NA
Custom Match One : Any
Custom Match One Mask : Any
Custom Match Two : Any
Custom Match Two Mask : Any
Custom Match Three : Any
Custom Match Three Mask : Any
Custom Match Four : Any
Custom Match Four Mask : Any
Attributes
Priority : 1 Duration : 5239 seconds
Hard Timeout : 0 seconds Idle Timeout : 0 seconds
Byte Count : NA Packet Count : 0
Flow Table ID : 100 Controller ID : 1
Cookie : 0x0
Hardware Index: 0
Instructions
Apply Actions
Output : 27
Flow 3
Match
Incoming Port : 27 Ethernet Type : Any
Source MAC : Any Destination MAC : Any
Source MAC Mask : 000000-000000
Destination MAC Mask : 000000-000000
VLAN ID : Any VLAN Priority : Any
Source IP Address : Any
Destination IP Address : Any
IP Protocol : Any
IP ECN : Any IP DSCP : Any
Source Port : Any Destination Port : Any
Source Port Range : NA
Destination Port Range : NA
TCP Flags : NA
TCP Mask : NA
Custom Match One : Any
Custom Match One Mask : Any
Custom Match Two : Any
Custom Match Two Mask : Any
Custom Match Three : Any
Custom Match Three Mask : Any
Custom Match Four : Any
Custom Match Four Mask : Any
Attributes
Priority : 65500 Duration : 5239 seconds
Hard Timeout : 0 seconds Idle Timeout : 0 seconds
Byte Count : NA Packet Count : 17202875
Flow Table ID : 100 Controller ID : 1
Cookie : 0x0
Hardware Index: 0
Instructions
Apply Actions
Output : 25
Flow 4
Match
Incoming Port : 25 Ethernet Type : Any
Source MAC : Any Destination MAC : Any
Source MAC Mask : 000000-000000
Destination MAC Mask : 000000-000000
VLAN ID : Any VLAN Priority : Any
Source IP Address : Any
Destination IP Address : Any
IP Protocol : Any
IP ECN : Any IP DSCP : Any
Source Port : Any Destination Port : Any
Source Port Range : NA
Destination Port Range : NA
TCP Flags : NA
TCP Mask : NA
Custom Match One : Any
Custom Match One Mask : Any
Custom Match Two : Any
Custom Match Two Mask : Any
Custom Match Three : Any
Custom Match Three Mask : Any
Custom Match Four : Any
Custom Match Four Mask : Any
Attributes
Priority : 65500 Duration : 5239 seconds
Hard Timeout : 0 seconds Idle Timeout : 0 seconds
Byte Count : NA Packet Count : 498479217
Flow Table ID : 100 Controller ID : 1
Cookie : 0x0
Hardware Index: 0
Instructions
Apply Actions
Output : 27
Flow 5
Match
Incoming Port : Any Ethernet Type : Any
Source MAC : Any Destination MAC : Any
Source MAC Mask : 000000-000000
Destination MAC Mask : 000000-000000
VLAN ID : Any VLAN Priority : Any
Source IP Address : Any
Destination IP Address : Any
IP Protocol : Any
IP ECN : Any IP DSCP : Any
Source Port : Any Destination Port : Any
Source Port Range : NA
Destination Port Range : NA
TCP Flags : NA
TCP Mask : NA
Custom Match One : Any
Custom Match One Mask : Any
Custom Match Two : Any
Custom Match Two Mask : Any
Custom Match Three : Any
Custom Match Three Mask : Any
Custom Match Four : Any
Custom Match Four Mask : Any
Attributes
Priority : 0 Duration : 5239 seconds
Hard Timeout : 0 seconds Idle Timeout : 0 seconds
Byte Count : NA Packet Count : 726
Flow Table ID : 100 Controller ID : 1
Cookie : 0x0
Hardware Index: NA
Instructions
Apply Actions
Controller Port
#To show flow rules in sw-table-1:
leaf1(of-inst-l1)# sh openflow instance l1 flows flow-table 200
OpenFlow Flow Table
Flow 1
Match
Incoming Port : Any Ethernet Type : Any
Source MAC : Any Destination MAC : Any
Source MAC Mask : 000000-000000
Destination MAC Mask : 000000-000000
VLAN ID : Any VLAN Priority : Any
Source IP Address : Any
Destination IP Address : Any
IP Protocol : Any
IP ECN : Any IP DSCP : Any
Source Port : Any Destination Port : Any
Source Port Range : NA
Destination Port Range : NA
TCP Flags : NA
TCP Mask : NA
Custom Match One : Any
Custom Match One Mask : Any
Custom Match Two : Any
Custom Match Two Mask : Any
Custom Match Three : Any
Custom Match Three Mask : Any
Custom Match Four : Any
Custom Match Four Mask : Any
Attributes
Priority : 0 Duration : 56 seconds
Hard Timeout : 0 seconds Idle Timeout : 0 seconds
Byte Count : 0 Packet Count : 0
Flow Table ID : 200 Controller ID : NA
Cookie : 0x0
Hardware Index: NA
Instructions
Drop
# Now showing packet statistics at each port 25 and 27. You will see thousands of Rx Discards at port 25 when I ran TCP flow for 40 seconds:
leaf1(of-inst-l1)# sh int 25
Status and Counters - Port Counters for port 25
Name :
MAC Address : 548028-5c79a7
Link Status : Up
Port Enabled : Yes
Totals (Since boot or last clear) :
Bytes Rx : 2,260,064,239 Bytes Tx : 64,489,408
Unicast Rx : 26,948,680 Unicast Tx : 918,509
Bcast/Mcast Rx : 46 Bcast/Mcast Tx : 2
Errors (Since boot or last clear) :
FCS Rx : 0 Drops Tx : 0
Alignment Rx : 0 Collisions Tx : 0
Runts Rx : 0 Late Colln Tx : 0
Giants Rx : 0 Excessive Colln : 0
Total Rx Errors : 0 Deferred Tx : 0
Others (Since boot or last clear) :
Discard Rx : 5,704 Out Queue Len : 0
Unknown Protos : 0
Rates (5 minute weighted average) :
Total Rx(Kbps) : 2,149,080 Total Tx(Kbps) : 4,320
Unicast Rx (Pkts/sec) : 174,967 Unicast Tx (Pkts/sec) : 6,433
B/Mcast Rx (Pkts/sec) : 0 B/Mcast Tx (Pkts/sec) : 0
Utilization Rx : 21.49 % Utilization Tx : 00.04 %
leaf1(of-inst-l1)# sh int 27
Status and Counters - Port Counters for port 27
Name :
MAC Address : 548028-5c79a5
Link Status : Up
Port Enabled : Yes
Totals (Since boot or last clear) :
Bytes Rx : 64,489,042 Bytes Tx : 2,251,406,709
Unicast Rx : 918,509 Unicast Tx : 26,948,680
Bcast/Mcast Rx : 2 Bcast/Mcast Tx : 47
Errors (Since boot or last clear) :
FCS Rx : 0 Drops Tx : 0
Alignment Rx : 0 Collisions Tx : 0
Runts Rx : 0 Late Colln Tx : 0
Giants Rx : 0 Excessive Colln : 0
Total Rx Errors : 0 Deferred Tx : 0
Others (Since boot or last clear) :
Discard Rx : 0 Out Queue Len : 0
Unknown Protos : 0
Rates (5 minute weighted average) :
Total Rx(Kbps) : 4,256 Total Tx(Kbps) : 2,118,904
Unicast Rx (Pkts/sec) : 6,342 Unicast Tx (Pkts/sec) : 172,523
B/Mcast Rx (Pkts/sec) : 0 B/Mcast Tx (Pkts/sec) : 0
Utilization Rx : 00.04 % Utilization Tx : 21.18 %
My flow rules are all in hardware, the switch should be able to process packets at the wire-rate, i.e. 10Gbps, and should not show any Rx discards. Any help will be appreciated!
