Wireless Access

Have you used the packet capture facility of an Aruba Access Point to ship packets to Wireshark on your desktop?  Have you wondered why Wireshark reports so many of the packets as “malformed?”  Part of the problem is due to an Aruba bug and part of the problem is due to inconsistent processing by the Wireshark dissectors.

First, we must remember that the wireless medium is half-duplex.  The AP where you are capturing packets cannot really “capture” the frames it is transmitting.  The AP makes a copy of the frame at some point on the frame’s journey down the protocol stack so that the AP can ship the copy to your desktop.  This necessarily means that what it ships to you will not be a true copy of what it sends out on the air because there are some values that get filled in after the point where the copy is made.  And if you have specified a file format that includes radiotap information, much of that information will be missing because it didn’t really “capture” it over its radio.

Aruba’s bug occurs when the capturing AP makes a copy of a frame it is transmitting.  It fails to append the four-byte “frame check sequence” (FCS).  This shows up in Wireshark pretty clearly but it gets very confusing very quickly because it manifests itself differently based on the file format you chose when you set up the capture.

An Aruba packet capture supports six possible formats but only three of those formats are supported by Wireshark.  The formats available to you also depend on whether you are doing an interactive or a raw capture and they depend on whether you are using the GUI or the CLI.  The formats are identified by an ERM Type.  ERM is Aruba’s “Encapsulated Remote Mirroring” protocol.  This chart summarizes the feature set:

On the Wireshark side we must also be aware of how the ERM dissectors work.  The ERM Type 1 dissector assumes the last four bytes of the frame are to be used as the FCS.  The ERM Type 0 and Type 3 dissectors assume the FCS is not included in the frame.  Unfortunately, the “Assume packets have FCS” checkbox at “Edit | Preferences | Protocols | IEEE 802.11” has no impact.  It is unclear why these dissectors work differently since there is no difference in the way Aruba handles the FCS.

Let’s talk about ERM Type 1 first because the problem is easy to see in Wireshark.  Because Wireshark uses the last four bytes of the frame as the FCS, and because Aruba includes the FCS on the frames received by the AP but not on the frames transmitted by the AP, Wireshark sees the frames that were transmitted by the capturing AP as “malformed.”  This shows up very clearly in the Management frames transmitted by the capturing AP.  Wireshark uses the last four bytes of the frame as the FCS and then can’t decode the final field correctly.

The situation is pretty much reversed for ERM Type 0 and Type 3.  Because Wireshark does not think the frame has an FCS, it tries to decode the last four bytes of frames received by the AP as part of the frame.  For received Control frames, those last four bytes are ignored by Wireshark, probably because a Control frame is a fixed length (though it seems to me that Wirehsark should report the frame as malformed).  For received Data frames, Wireshark just assumes those last four bytes are part of the data payload.  Where we see the problem is in received Management frames.  Wireshark tries to decode those last four bytes as a “Tagged parameter” and fails.

Of course some of those packets may legitimately be malformed but most occurrences can be safely ignored.  I would like to thank Kevin Schoenfeld for his fine analysis of this issue and his help in identifying the problems.  For additional details, his 5-part blog on this issue starts here: http://kjspgd.net/?p=30.

hi Doug

Thanks for the post, coincidentally I have just been looking at problems with rate encoding in the various formats (leading up to either a bug or a fix to ARUBA_ERM); I think it's safe to say all the capture types have got a few bugs and quirks.

I have just reached out to Kevin and will have a chat about what can and cannot be fixed (i.e. as he correctly alludes, some data is not available when the copy is made by AM) and to get some thoughts on what we should try to fix first, where possible.

Another example of this is the ACK frames (and friends) which are generated after the packet capture, we never see the AP's own tx acked frames. The only really viable workaround here is to use a separate AP as the capture AP. Certainly this is not necessarily a luxury all can afford, but it is an alternative that does solve some of the limitations of capturning on the client serving AP.

Do let me know (email @ my forum username) or in this thread if you have any thoughts on what we should try to fix up first.

Disclaimer: of course I cannot guarantee that all issues can or will be fixed :) but I'll do what I can over the next couple of weeks.

regards

-jeff

Hi,

i'm the original author of Aruba ERM dissector...
If you have more detail about some undecoded type... (or known bug with possilibity of fix it), i can help you to fix the dissector !

Thank you, Alexis, I understand that you and Jeff are in touch now so I assume you guys are running with this.  I really appreciate the response.  Jeff just needs to tell you whether he is shipping the captured packets with or without the FCS and then you need to adjust the dissectors accordingly.  My suggestion is that Jeff modify his end to always include the FCS, then you just need to modify the Type 0 and 3 dissectors to assume the last four bytes are the FCS.

Jeff - Thank you for reaching out.  From the perspective of analyzing packet captures from Wireshark, I would prioritize it like this:

1.  Treat all frames forwarded in a packet capture in a consistent manner.  We understand that some data is not available at the time a copy is made of a frame transmitted by the capturing AP.  But the frame still needs a FCS.  The alternative is to strip the FCS from the received frames before forwarding them.  All forwarded frames in a packet capture should either have the FCS or not.  Ideally, you would compute the FCS based on the available data at the time of the capture.  We know this is not what actually went out in the air but we don't want to see "invalid FCS" in Wireshark, which will be the case if all you do is throw out four null bytes for the FCS.

2. Are you responsible for the Wireshark ERM dissectors?  If so, they need to be modified to be in sync with your handling of the FCS for the various file formats.  If not, you need to coordinate your changes with whoever does for the obvious reasons.  Right now Type 1 assumes the the last four bytes are for the FCS; Type 0 and 3 ignore the FCS.    And it would also be nice if these dissectors respected the "Assume packets have FCS" checkbox (IEEE 802.11 protocol) or provided their own.

3. Documentation.  The ArubaOS packet capture is a great feature but it has been painful to figure it out because of what I perceive to be a lack of documentation.

Your interest is much appreciated!

Doug

---

@doug.fsu wrote:

Jeff - Thank you for reaching out.  From the perspective of analyzing packet captures from Wireshark, I would prioritize it like this:

2. Are you responsible for the Wireshark ERM dissectors?  If so, they need to be modified to be in sync with your handling of the FCS for the various file formats.  If not, you need to coordinate your changes with whoever does for the obvious reasons.  Right now Type 1 assumes the the last four bytes are for the FCS; Type 0 and 3 ignore the FCS.    And it would also be nice if these dissectors respected the "Assume packets have FCS" checkbox (IEEE 802.11 protocol) or provided their own.

Hi Doug,

there is no responsible of Aruba ERM dissector (any body can propose a patch to fix a bug...)

the better solution to fix problem, is attach same pcap sample on bug tracker ( https://bugs.wireshark.org ) and say what the correct display/dissection...

I have no real check for FCS stuff when i write the dissector

Excellent thought.  I have just submitted my first Wireshark bug, #11204, and it includes screen shots showing that Type 1 dissects the FCS and Type 3 does not.  I hope I did everything correctly.  I would be happy to get more involved, even to the point of helping with the dissector.  You would have to provide me with some guidance, though, I have not done anything like this before.

the problem is not really about type 1 or 3... The FCS is present in type 3 for received packets (you can see it when you pick a non data packet recevied by the capturing AP, it's malformed because wireshark tries to parse the last 4 bytes as some other IE etc).

the reason it doesnt spray FCS errors everywhere is because in the aruba erm dissector, type 3 is passed onto

        ieee80211_handle = find_dissector("wlan_withoutfcs");

whereas peek traffic (type 1 or type 5) is passed to

        peek_handle = find_dissector("peekremote");

which then inside the peek dissector, it calls

       ieee80211_handle = find_dissector("wlan_withfcs");

I think the problem is generically to be stated as such - an Aruba AP that is capturing and serving traffic, cannot capture it's own FCS nor does it capture the rate (nor a bunch of other stuff generated way down the stack like ACK packets, RTS/CTS etc).

As for the solution, in progress - first stop is asking my R&D how they feel about adding the FCS for us, or at least adding a 0x00000000 pad so we can fake it out at the wireshark side.

regards

-jeff

Jeff -

Totally agree with your "first stop," much appreciated.  But I do see an issue on the dissector side in regard to Type 1 vs Type 3 (and 0).  We agree the FCS is present in type 3, which then means that:

ieee80211_handle = (find_dissector("wlan_withoutfcs") would be incorrect.  It should be changed to 

ieee80211_handle = (find_dissector("wlan_withfcs") so that it correctly interprets the FCS.

Would it be even better is if it would honor the "Assume packets have FCS" checkbox?

If you are able to include the Control frames being sent by the capturing AP that would be a bonus but I am not going to complain because those are missing.  I get why you might not be able to provide those.

---

@doug.fsu wrote:

Jeff -

Totally agree with your "first stop," much appreciated.  But I do see an issue on the dissector side in regard to Type 1 vs Type 3 (and 0).  We agree the FCS is present in type 3, which then means that:

ieee80211_handle = (find_dissector("wlan_withoutfcs") would be incorrect.  It should be changed to 

ieee80211_handle = (find_dissector("wlan_withfcs") so that it correctly interprets the FCS.

 

 

 

 

Would it be even better is if it would honor the "Assume packets have FCS" checkbox?

 

If you are able to include the Control frames being sent by the capturing AP that would be a bonus but I am not going to complain because those are missing.  I get why you might not be able to provide those.

---

 For using Assume packet have FCS, you need to have the information... and the information is on header... and for type 3, there is no information about FCS is present or not..

but yes, it is possible to change the type 3 to use wlan_withfcs

---

@doug.fsu wrote:

Jeff -

Totally agree with your "first stop," much appreciated.  But I do see an issue on the dissector side in regard to Type 1 vs Type 3 (and 0).  We agree the FCS is present in type 3, which then means that:

ieee80211_handle = (find_dissector("wlan_withoutfcs") would be incorrect.  It should be changed to 

ieee80211_handle = (find_dissector("wlan_withfcs") so that it correctly interprets the FCS.

[-jeff] sure, it can be done, but, then its going to start spewing the same error as type 1, because now it will be right for RX frames, but not TX frames - which is the same issue as type 1.

The FCS is always there for a received frame, no matter the capture type. It;'s a function of how the AM part of the AP hands off the packet to the capture part. The FCS is missing for any tx frame - ideally we should have padded it out or 'calculated' it. Let's see how far I get with false calculation at the AP, but I suspect when we come full circle it will be 0 pad + fixup at wireshark.

Anything we touch here also has a knock on effect for Omnipeek, so that needs to be checked and tested too (on my todo list, they have some plugin that eats the Aruba ERM, but I havent loaded it up to check yet - since peek signals bad FCS as a flag in the peek header, they may not care/re-calculate on the actual packet, not 100% sure right now)

 Would it be even better is if it would honor the "Assume packets have FCS" checkbox?

such a toggle could be added to the aruba-erm plugin, but it's just going to make type3 look like type1 in terms of colouration of bad FCS as far as I can tell.

 

If you are able to include the Control frames being sent by the capturing AP that would be a bonus but I am not going to complain because those are missing.  I get why you might not be able to provide those.

[-jeff] as far as I know, there is no way for that to work due to where the packet copy is made in the stack  - in those cases, time to bust out a spare AP as a mobile sniffer, or we have to resort to capturing with a suitable external device/laptop. The more common cases for that are MPDU agg issues; but those never happen.... :| !!!!!

---

Jeff - Are you aware that Types 3 and 0 are also reporting errors? The errors are different though because those Types handle the FCS differently than Type 1.  It was a devil to diagnose because we had the Aruba bug coupled with the inconsistent behavior of the dissectors.

We agree the FCS is always there and that if we change Type 3 and 0 as suggested that those types will then report the same error as Type 1.  This is what we want.  Since the FCS is always there, no matter the Type, we want the dissector to handle it in the same way.   What we have now is Type 1 "malformed packets" due to missing FCS in transmitted frames, and Type 3 "malformed packets"  due to extra bytes in received frames, even though there is no difference in the way those FCS bytes are being delivered!  Does this make sense?  If not, how about a phone conversation?  BTW, I totally forgot to mention that I opened a TAC case on this last week before publishing this article.  The Case # is1691653.  

Doug, see inline - I think we are back on the same page now :) - I am going offline now, will pickup the thread in the morning.

@doug.fsu wrote:

Jeff - Are you aware that Types 3 and 0 are also reporting errors? The errors are different though because those Types handle the FCS differently than Type 1.  It was a devil to diagnose because we had the Aruba bug coupled with the inconsistent behavior of the dissectors.

 

[-jeff] yes am aware, this is what I was trying to mention in my last post :) I understand the time lost, I have just thrown away days of pcaps due to problems in type 1 and type 3 - my drama was more at the rates being messed up though, esp in type 3.

 

 

We agree the FCS is always there and that if we change Type 3 and 0 as suggested that those types will then report the same error as Type 1.  This is what we want.  Since the FCS is always there, no matter the Type, we want the dissector to handle it in the same way.

 

  What we have now is Type 1 "malformed packets" due to missing FCS in transmitted frames, and Type 3 "malformed packets"  due to extra bytes in received frames, even though there is no difference in the way those FCS bytes are being delivered!  

 

[-jeff] yes, we are saying the same thing. I was rambling on about the same-same result in terms of just flipping the with_fcs vs. your initial opposition to the problems with the tx frames. After a few re-reads, I see where we diverged (sorry, 1.30am here) - see my next point. 

Does this make sense?  If not, how about a phone conversation?  BTW, I totally forgot to mention that I opened a TAC case on this last week before publishing this article.  The Case # is1691653.  

 

 

[-jeff] yes, we are in effect saying the same thing. As a by product of dealing with this at the AP side, the type 3 needs to be flipped to with_fcs, absolutely!

To reiterate so there is clarity (and there are more than just these issues, this is just about FCS for the moment) - right this moment we have 2 issues to deal with:

issue 1:  aruba erm type 3 needs to be changed to with_fcs - this needs to be done as a matter of course and at any time.

Issue 2:  tx frames need to have FCS inserted at the AP side - likely either by:

a> padding it for "tx" frames and handling the padding in the Wireshark side either by adding a FCS into the payload (havent looked into the feasibility), or, by simply calling the 802.11 decoder with no FCS (easy). Note that the AP is aware of when its dealing with a TX frame, it's already stuffing predetermined values in some fields (i.e. signal = 100%), but as it stands right now, Wireshark has no way to determine this.

b> calculating a fake-but-correct-at-the-time FCS and dropping it into the buffer before sending the packet out to the capture IP. R&D will push back on this, it's CPU work that they don't want to be doing. Will raise a bug on it tomorrow.

 

---

Yes!

Issue 1: Need "with_fcs" for Type 0 too.

Issue 2: a) This is a viable solution except that we don't want to call the decoder "withoutFCS" when the packets actually have an FCS because then we get the "malformed packets" we currently see with Type 0 & 3.  Yes, this solution will still give us "malformed packets" due to invalid FCS but this is the best outcome you can deliver.  Then it would be up to the ieee80211 dissector to provide the ability to turn off FCS validation, just like the dissectors for IP, TCP and UDP can turn off validation for the CRC.

b) Yes, this is the preferred solution but I defer to your engineers on whether or not they can spare the CPU cycles.

hi Doug

inline below - pardon the code boxes, seems the easiest way to add to quoted text :)

regards

-jeff

---

@doug.fsu wrote:

Jeff - Thank you for reaching out.  From the perspective of analyzing packet captures from Wireshark, I would prioritize it like this:

1.  Treat all frames forwarded in a packet capture in a consistent manner.  We understand that some data is not available at the time a copy is made of a frame transmitted by the capturing AP.  But the frame still needs a FCS.  The alternative is to strip the FCS from the received frames before forwarding them.

 

 

[-jeff] I would like to avoid stripping the FCS, the case here is "packet capturing APs transmitting a frame", we don't want to sacrifice function for the other cases. We already have a number of limitations for the case of serving-but-packet-capturing that make it not 100% useful in all cases. 

 

 All forwarded frames in a packet capture should either have the FCS or not.  Ideally, you would compute the FCS based on the available data at the time of the capture.  We know this is not what actually went out in the air but we don't want to see "invalid FCS" in Wireshark, which will be the case if all you do is throw out four null bytes for the FCS.

 

[-jeff] agreed, I plan to ask R&D for their assessment on adding the FCS, but I am not holding my breath (adding FCS == CPU load). At least if we pad it with 4 null bytes we can catch it on the Wireshark side and handle it there (be it ignoring it, or, maybe even forcing the correct FCS in)

 

2. Are you responsible for the Wireshark ERM dissectors?  If so, they need to be modified to be in sync with your handling of the FCS for the various file formats.  If not, you need to coordinate your changes with whoever does for the obvious reasons.  Right now Type 1 assumes the the last four bytes are for the FCS; Type 0 and 3 ignore the FCS.    And it would also be nice if these dissectors respected the "Assume packets have FCS" checkbox (IEEE 802.11 protocol) or provided their own.

 

[-jeff] I am in touch with the author Alexis already, I am working on the IAP dissector this week as I am still getting all my ducks in a row regarding the aruba_erm. For the dissector stuff I will just submit updates as I get to them. 

 

3. Documentation.  The ArubaOS packet capture is a great feature but it has been painful to figure it out because of what I perceive to be a lack of documentation.

 

[-jeff] The doc section is pretty hard reading and a bit confusing - I can't speak for others, but mostly I am using the raw-start and will deal with filtering myself in whatever I capture to. Please highlight in particular what needs some cleanup or that gave you some grief ? Does it need more examples, should the raw-start and interactive be split into two sections etc?

 

---