SD-WAN

Hi,

I am using an Aruba 2930F switch. I want to add an experimenter match field, and I am trying to follow the example in the guide of TCP sequence number. If I understand correctly the header length would be 112 bits as the following:

0xffff050a00002481000300040004

it made me confused since the Experimenter filed header should be 64 bits according to the OpenFlow documentation!

Is this how the OFPTFPT_MATCH array  in the table features message should look like:

OFTableFeaturePropMatchVer13(oxmIds=[0x80000004, 0x80000706, 0x80000906 .....(any other basic 32bit oxmids),0xffff050a00002481000300040004]) 

Can you please provide me with an example of a correct  OFPTFPT_MATCH array that has an experimenter field in addition to the basic oxm fields, and which controller can be used to achieve this?

Thanks,

Enas

Hello Enas,

It is exciting to see that you are trying to use the customizable match fields feature on our switches.

When the OXM class type in the OXM header of a OFPTFPT_MATCH field is OFPXMC_EXPERIMENTER, the switch will parse the information in the OXM header as follows:

The OpenFlow Admin Guide talk in detail about how the OXM header needs to be setup for a customizable match field both in the OFPMP_TABLE_FEATURES message and the OFPT_FLOW_MOD message.

I am attaching a sample packet trace that defines a table (0) with 2 custom match fields TCP Sequence and Acknowledgement numbers along with the standard OXM fields ETH_TYPE and IP_PROTO. (Please rename the extension back to .pcap to open it with a packet analyzer tool like wireshark)

The packet trace also has a sample flow-mod request that adds an entry to this table with the following match,

ETH_TYPE=0x800, IP_PROTO=6, TCP_ACK_NUM(Field=6)=0x98765432,TCP_SEQ_NUM(Field=5)=0x12345678.

Hope this helps you better understand how the experimenter fields are encoded.

We used sample VAN applications to send both the OFPMP_TABLE_FEATURES and OFP_FLOW_MOD messages with experimenter match fields.

Any OpenFlow controller that supports libraries to encode an OXM_MATCH field with class type OFPXMC_EXPERIMENTER can be used. (Ex: HPE VAN, RYU etc.,)

Let me know if you have further questions on this topic.

Thanks!

Abhay

Thank you Abhay for your reply, you have been very helpful!

Ok so it seems you split the Experimenter header into 32bit segments. I have tried sending the same set of match fields (TCP Sequence and Acknowledgement numbers along with the standard OXM fields ETH_TYPE and IP_PROTO) to table 0, however, I get the following error:

0010:01:39:43.01 OPFL eOFNetTask:RX from tcp:10.0.0.1:6653: 0: Table 0 received 
   has match 5 which cannot be supported

I have attached the .pcap file that has the table features request. If I simply remove the two custom fields the request is executed successfully and the tables are created.

Do you know what I might be doing wrong? 

Note: I found it strange that you used 0a and 0c for the (OXM_FIELD) instead of 05 and 06 like it is described in the guide.

Thank you  in advance,

Enas

Hello Enas,

The OXM field is the upper 7 bits of the byte and the lower bit is used to determine if the field is maskable.

Hence when the field is 5 with no masking capability, it translates to a value of 0xa for the byte and 0xc for field 6.

Regarding the error you see, I don't see anything wrong in the TABLE_FEATURES message you are pushing to the switch.

I take it you are currently running the 16.02 firmware. Can you please upgrade your switch firmware to 16.03.0003 and re-run your test? It is possible that the error you see was a switch firmware bug that was fixed in 16.03.

Thanks!

Abhay

I take it you are currently running the 16.02 firmware. Can you please upgrade your switch firmware to 16.03.0003 and re-run your test? It is possible that the error you see was a switch firmware bug that was fixed in 16.03.

Thanks again Abhay for your reply. You are right when I upgraded to 16.03 the error disappeared, and the match fields were added.  

The OXM field is the upper 7 bits of the byte and the lower bit is used to determine if the field is maskable. Hence when the field is 5 with no masking capability, it translates to a value of 0xa for the byte and 0xc for field 6.

Aha I see, I hope the guide gets updated to include this in defining a custom field instead of using 0x05 and 0x6 (I believe with masking capability it would be 0xb and 0xd).

Ok, now I am trying to understand the format of the flow mod message. Let's take the TCP_ACK_NUM(Field=6)=0x98765432 match example. According to the flow-mod example you sent the match for the TCP_ACK_NUM should be :

ffff 0c 18 00002481 00000004 98765432

I am a bit confused about the 0x18. which in the guide explained as the OXM_LEN: The length of the experimenter header excluding the OXM header. So why is it 0x18 since we set the oxm_len to 0x0a in the table features? 

I am trying to define a custom field of 128bits (16 bytes) that is parsed after the UDP header (8 bytes). I defined the field as the following in the table feature: 

ffff 0a 0a 00002481 0003 0008 0010  

The field is created successfully on the switch.

However, when I create a flow-mod to match against this field I get an error of OFBadMatchErrorMsgVer13(xid=46, code=BAD_FIELD, and sometimes the switch suddenly reboots!

Is this how the match of my field in the flow-mod should look like ??

ffff 0a 14 00002481 00000010 3133 5444 4538 3930 3631 3433 3638 3730

where 0x31335444453839303631343336383730 is my match data.

Hello Enas,

Glad to know moving to 16.03 helped you progress.

I will take note of clarifiying the values associated with the OXM_FIELD in the Admin Guide.

With masking set, yes the fields would be 0xb and 0xd instead of 0xa and 0xc respectively.

Regarding your following query -

"I am a bit confused about the 0x18. which in the guide explained as the OXM_LEN: The length of the experimenter header excluding the OXM header. So why is it 0x18 since we set the oxm_len to 0x0a in the table features? "

Remember, the OXM_LEN is for the current OXM header.

In case of the TABLE_FEATURES request message, the OXM Experimenter header has only the following data post the OXM header itself:

HPE OUI ID (4 bytes) + START_TYPE (2 bytes) + OFFSET_START (2 bytes) + NUM_BYTES (2 bytes) = 10 bytes (0xa)

In case of the FLOW_MOD message, the OXM Experimenter header has the following data post the OXM header itself:

HPE OUI ID (4 bytes) + LENGTH_OF_MATCH_DATA (4 bytes) + MATCH_DATA (16bytes always) = 24 bytes (0x18)

Coming to your example,

Is this how the match of my field in the flow-mod should look like ??

ffff 0a 14 00002481 00000010 3133 5444 4538 3930 3631 3433 3638 3730

If you observe closely, the OXM_LEN in your example is set incorrectly to 20 bytes (0x14) instead of 24 bytes (0x18).

Thus the switch is rejecting the flow since it doesn't contain the 16 bytes of data it expects based on the definition of the field in TABLE_FEATURES request message.

Also, the OXM_LEN for a custom match OXM_EXPERIMENTER field which is non maskable should always be 24 bytes (0x18), the switch will only consider the relevant match data based on the LENGTH_OF_MATCH_DATA set.

Hope this helps.

Thanks!

Abhay

Hi Abhay, 

I really appreciate your help,

Ok so when I correct the length and I set my match field to:

ffff 0a 18 00002481 00000010 3133 5444 4538 3930 3631 3433 3638 3730

I no longer get the bad field error, however, I get the error OFBadMatchErrorMsgVer13(xid=20, code=BAD_WILDCARDS. The first pcap file attached has the packets capture. Any idea?

Moreover, I actually aim to have this same field with mask capabilities (I was trying the basic field first before moving to the more complex case). I define my field as the following in the table feature msg:

ffff 0b 0a 00002481 0003 0008 0010  

and as the following in the match msg:

ffff 0b 28 00002481 00000020 3133 5444 4538 3930 3631 3433 3638 3730 FFFF FFFF FFFF FF00 0000 FFFF 0000 0000

Where FFFF FFFF FFFF FF00 0000 FFFF 0000 0000 is my mask.

However, I get the error OFBadMatchErrorMsgVer13(xid=46, code=BAD_FIELD. The second pcap file attached has the packets capture.

I have also tried:

ffff 0b 28 00002481 00000010 3133 5444 4538 3930 3631 3433 3638 3730 FFFF FFFF FFFF FF00 0000 FFFF 0000 0000

and

ffff 0b 18 00002481 00000010 3133 5444 4538 3930 3631 3433 3638 3730 FFFF FFFF FFFF FF00 0000 FFFF 0000 0000

But got the same error. I didn't find any information that explains how to define a custom match field with masking in the guide, so any help would be useful.

Thanks a bunch!

Enas

Ok so I have figured the answer to my first question: I was getting code=BAD_WILDCARDS because I needed to add all the the match fields that are not set in my flow-mod to the OFTableFeaturePropWildcards of the table, including the custom fields. the trick was that I had to add the custom fields in the wildcard list in the following format 

ffff0a0a 00002481 only and not ffff0a0a 00002481 0003 0008 0010 !!

So I will wait for your answer on my second question regarding defining a custom field with mask capabilities :) 

Thanks,

Enas

Hello Enas,

Yes the first error was because of the missing wildcards definition for the first table.

Regarding the second error, can you please enable debug for OpenFlow and share the error string returned by the switch for the flow which fails.

debug destination session

debug openflow error

It will help us understand the reason for failure much better.

Thanks!

Abhay

If I enable the debug for openflow errors and the destination session and then send out the flow-mod with the masked custom field the switch reboots! So I am unable to use these debug features.

However, this is the error I get in the message statistics:

OFPBMC_BAD_FIELD 1
The table doesn't support match on at least one field in the rule.

It seems I am not defining the custom match field with mask correctly. Can you please let me know what is the correct header format (in the feature table match prop and the flow-mod match entries) for a custom field with mask?

I defined it this way in the match prop. I set it to 0b instead of 0a to set the wildcard bit.

ffff 0b 0a 00002481 0003 0008 0010  

And this way in the flow-mod match:

ffff 0b 28 00002481 00000020 3133 5444 4538 3930 3631 3433 3638 3730 FFFF FFFF FFFF FF00 0000 FFFF 0000 0000

Where FFFF FFFF FFFF FF00 0000 FFFF 0000 0000 is my mask.

I set the length to 28 instead of 18 to add the 16 bytes of the mask (24+16=40 == 0x28). And I set the length to 20 instead of 10 for the same reason (16 + 16 = 32 == 0x20). However, this seems incorrect because the table does not identify the field. The pcap file (BAD_FIELD) is attached to my previous msg.

Hello Enas,

It seems like a switch firmware bug is causing the device to reboot when you have debug turned on while pushing a flow with a custom match field. We will investigate it.

Regarding the usage of custom match field with masks in the flow, I don't see anything obviously wrong in the format of the OXM header in your example. It is again possible that it is a switch firmware bug that is causing the flow to error out.

We will try to recreate this locally in our lab and get back to you on it.

Thanks!

Abhay

PS: This might take a while since I am currently out of office through next week.

Thanks Abhay,

Aha I see. Can you or one of your colleagues meanwhile just send me a pcap of a custom field with a mask (similar to the one you sent me of the TCP sequence number only with a mask), that would be very useful for me!

I am starting to feel that the switch does not support a custom match field with mask, because I have tried every possible header format and it didn't work :)

If a pcap is hard to get, maybe someone can just provide me with a written format for a custom field with mask headers ?

Hello Enas,

Thanks for your patience with this request.

The design of the solution accomodates for the support of masking with custom match fields. The most logical explanation for it to not work currently is a possible switch firmware bug. I am working with the team to recreate the issue you reported and confirm the same.

If we conclude that it is indeed a switch firmware bug, we will fix this in one of the upcoming releases of the firmware.

Meanwhile, can you please share the pcap of the TableFeatureRequest message which has the tables with masked custom matches in it. The previous pcaps you have shared don't have that particular message.

Thanks!

Abhay

Hello Enas,

I just verified programming a table with maskable custom match field and subsequently programming a flow with a masked custom match field on a 2930F running the 16.03 firmware is successful.

I have attached a pcap of the flow-mod with a masked custom match field.

And here is the show command output of the programmed flow.

Aruba-2930F-24G-4SFP(openflow)# show openflow instance aggregate flows flow-table 6

OpenFlow Flow Table

Flow 1
Match
Incoming Port : Any Ethernet Type : IP
Source MAC : Any Destination MAC : Any
Source MAC Mask : 000000-000000
Destination MAC Mask : 000000-000000
VLAN ID : Any VLAN Priority : Any
Source IP Address : Any
Destination IP Address : Any
IP Protocol : TCP
IP ECN : Any IP DSCP : Any
Source Port : Any Destination Port : Any
Source Port Range : NA
Destination Port Range : NA
TCP Flags : NA
Custom Match One : 0x12345678
Custom Match One Mask : 0xff00ffff
Custom Match Two : Any
Custom Match Two Mask : Any
Custom Match Three : Any
Custom Match Three Mask : Any
Custom Match Four : Any
Custom Match Four Mask : Any
Attributes
Priority : 35000 Duration : 44 seconds
Hard Timeout : 0 seconds Idle Timeout : 0 seconds
Byte Count : 0 Packet Count : 0
Flow Table ID : 6 Controller ID : listen-port
Cookie : 0x0
Hardware Index: NA
Instructions
Write Actions
Output : 10

And the hex dump of the OXM Experimenter field in the flow-mod. The custom match field in my case is a 4 byte field.

0000           ff ff 0b 2c 00 00 24 81 00 00 00 04 12 34 56 78
0010            00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 04
0020           ff 00 ff ff 00 00 00 00 00 00 00 00 00 00 00 00

At this point, I don't believe there is any bug in the switch firmware and the functionality should work as expected.

Please let us know if you still face any troubles to get the feature to work.

Hope this helps.

Thanks!

Abhay

Hello Enas,

I took another look at the flow-mod encoding of the experimenter field in your example and I see a problem.

ffff 0b 28 00002481 00000020 3133 5444 4538 3930 3631 3433 3638 3730 FFFF FFFF FFFF FF00 0000 FFFF 0000 0000

The encoding is setting the length for both the value and mask in one go. Instead, it should be 2 different length fields each for value and mask. That is,

ffff 0b 2c 00002481 00000010 3133 5444 4538 3930 3631 3433 3638 3730 00000010 FFFF FFFF FFFF FF00 0000 FFFF 0000 0000

I do understand this wasn't clarified in the Admin Guide and hence the confusion. I will get the Admin Guide updated with these details in the next release.

I hope with this change, you will be able to successfully program a flow with a masked custom match field.

Thanks!

Abhay

Yes it finally worked! Thank you for following up, and yes I urge you to update the guide and include the complete description, it would save a lot of time for folks like me!

I have one last question regarding the custom fields and I thought to ask it on this same thread before closing it.

In the table feature message I had to create 2 custom fields in order for the message to succeed (Although, I only need one custom field and as you know each custom field is a group by it self to it becomes limiting to have this extra custom field that I don't really need).  But when I create only one custom field the table feature message fails.

The OFTableFeaturePropMatch takes a list of 32 bits. The experimenter match field header in the HP switch is 112 bits (However, as I mentioned in my first post it is 64 bits in the OpenFlow standard!). so 112/32=3.5, thus, you would get 16 bits at the end. However, if you have two custom fields things work out great because you will get 112+112= 224, and 224/32=7 and the message succeeds. 

I have attached a pcap file with my attempt to define only one custom field.

I get the following error: 

OFTableFeaturesFailedErrorMsgVer13(xid=20, code=BAD_ARGUMENT 

OFPTFFC_BAD_ARGUMENT 1
An unsupported match type was part of the request.

Am I doing something wrong? How can I correctly define one custom match field?

Thanks,

Enas

Hello Enas,

The problem is with how the OFPTFPT_MATCH is encoded in the message. Per the OpenFlow Specification, the OFPTFPT_MATCH property should end at an 8 byte boundary. If the data in the property doesn't end at an exact 8 byte boundary, it should be padded with 0's at the end to make it end at an 8 byte boundary.

From the OpenFlow 1.3.2 specification (page 83)

/* Match, Wildcard or Set-Field property */
struct ofp_table_feature_prop_oxm {
uint16_t type; /* One of OFPTFPT_MATCH,
OFPTFPT_WILDCARDS,
OFPTFPT_WRITE_SETFIELD,
OFPTFPT_WRITE_SETFIELD_MISS,
OFPTFPT_APPLY_SETFIELD,
OFPTFPT_APPLY_SETFIELD_MISS. */
uint16_t length; /* Length in bytes of this property. */
/* Followed by:
* - Exactly (length - 4) bytes containing the oxm_ids, then
* - Exactly (length + 7)/8*8 - (length) (between 0 and 7)
* bytes of all-zero bytes */
uint32_t oxm_ids[0]; /* Array of OXM headers */
};

In your example, the padding is incorrect after the OFPTFPT_MATCH (6 bytes instead of 4 bytes). Thus, the switch is attempting to decode it as a match field instead of as the next property. Just correct the padding to end at an 8 byte boundary and it should work.

Which controller are you using in your setup?

Thanks!

Abhay

Hi Abhay,

I am using the floodlight controller.  

Ok I think this is the problem, the floodlight controller expects an oxm_id list of 4 bytes (32 bits), and after that it has paded the property with another 4 zero bytes to end at the 8 byte boundary. However, the last part of the custom match field is 2 bytes and not 4 (112/8=14, 14/4=3.5), so it is already padded with 2 zero bytes (to match the format of 4 bytes oxm id list). Thus, we would get 2+4=6 bytes of zero padding and the decoding problem that you have described happens. 

From the same openflow 1.3.1 specs, page 81:

/* Match, Wildcard or Set-Field property */
struct ofp_table_feature_prop_oxm {
uint16_t type; /* One of OFPTFPT_MATCH,
OFPTFPT_WILDCARDS,
OFPTFPT_WRITE_SETFIELD,
OFPTFPT_WRITE_SETFIELD_MISS,
OFPTFPT_APPLY_SETFIELD,
OFPTFPT_APPLY_SETFIELD_MISS. */
uint16_t length; /* Length in bytes of this property. */
/* Followed by:
* - Exactly (length - 4) bytes containing the oxm_ids, then
* - Exactly (length + 7)/8*8 - (length) (between 0 and 7)
* bytes of all-zero bytes */
uint32_t oxm_ids[0]; /* Array of OXM headers */
};
OFP_ASSERT(sizeof(struct ofp_table_feature_prop_oxm) == 4);
The oxm_ids is the list of OXM types for the feature (see A.2.3.2). The elements of that list are 32-bit
OXM headers or 64-bit OXM headers for experimenter OXM fields.

So if the HP switch used a 64 bit header for the experimenter field instead of 112 bits, then I think there will be no problem because there will be no extra 2 bytes padding, everything would be a multiple of 32-bits.

Have you tried to create a single custom field using a controller other than HPE VAN?

Just out of curiosity, I have tried to create a list of match fields that won't require extra zero padding from floodlight (that already ends at the 8 byte padding). However, the switch still rejects the request, maybe the 2 zero padding is still causing the problem ?? I have attached the pcap. 

This is the error I get:

OFPTFFC_BAD_ARGUMENT 1
An unsupported match type was part of the request.

Thanks,

Enas

Hello Enas,

What we have done with the experimenter header for custom matches in TABLE_FEATURES_REQUEST message is to treat the OXM header as a complete TLV rather than just as an OXM ID. Treating it as a TLV gives us the much needed flexibility to allow adding a payload to the header that defines the property of the custom match field (start, offset, length). With just a 64 bit header, it wouldn't be possible anyways.

I am not a 100% sure if we are violating the OpenFlow SPEC by adding a payload to the header for these fields but alteast wireshark is able to dissect the TABLE_FEATURES_REQUEST message with custom match field headers correctly. It does treat the Experimenter header as a TLV and displays the payload as Experimenter Value.

OXM ID
Class: OFPXMC_EXPERIMENTER (0xffff)
0000 101. = Field: 5
.... ...0 = Has mask: False
Length: 10
Experimenter: 0x00002481
Experimenter Value: 000300040004
OXM ID
Class: OFPXMC_EXPERIMENTER (0xffff)
0000 110. = Field: 6
.... ...0 = Has mask: False
Length: 10
Experimenter: 0x00002481
Experimenter Value: 000300080004

I have not tried programming custom match fields via any other controller other than HPE VAN.

In the other example that you have shared, I see that the OXM Length associated with the OFPTFP_MATCH field seems incorrect. That is the reason the message is being rejected.

Can you check if the FloodlightAPI can be forced to program experimenter headers with payloads and calculate the length accordingly.

Thanks!

Abhay

Hello Enas,

If you are happy with the response, can you please mark this thread as solved? That would be really helpful.

Thanks!

Abhay

Hi Abhay,

I have not marked this as solved yet because I was expecting your team to try to program a custom match field using an OpenFlow controller other than HPE VAN. 

Thanks,

Enas