Wired Intelligent Edge

 View Only
Back to discussions
Expand all | Collapse all

CX switches drops ip packets if checksum is not good...

This thread has been viewed 85 times

  • 1.  CX switches drops ip packets if checksum is not good...

    Posted Jan 05, 2023 07:24 AM
    Hello,

    End of previous year I did installation of CX6100 to my client who was running QSC QLAN system (proprietary 48khz/24bit HQ audio over lan stuff...) using installed switch. QLAN audio quality was really bad (robotic sound) while everythig else running in that same switch was just normal.
    After days debugging I tested to replace 6100 with old 2540 and also QLAN started to work fine (whaaat?!?!?) after change back to 6100 robotic sound was back...

    Local HPE guys found problem: There was 50% packet drop in port which sent audio stream to processor comparing to port which was inputting data from AD converter...
    QLAN analog to digital converter (model QIO) was sending every other UDP packet with wrong L3 packet checksum (0xFFFFFF) and 6100 made decision drop all of these packets (while older L2-level AOS/Cisco/Extreme just passes them over without any checksum inspection)
    Same new feature exists also in 6300M so this is un-documented AOS-CX feature which could cause unexpected problems for users.

    There is ticket open about this to get more information about this and some configuration and logging features available to control this feature.

    If somebody has additional information about this, I would be glad to hear about that.

    ------------------------------
    Jori Luoto
    ------------------------------


  • 2.  RE: CX switches drops ip packets if checksum is not good...

    Posted Jan 06, 2023 02:58 AM
    Hi
    If the checksum is 0xFFFF it is a legal checksum. Are you sure the reason for the packet drops is the UDP checksum and not somethin else? If you can do a packet capture on both ingress and egress of the switch and compare to find if its only the UDP 0xFFFF which are dropped. 

    ------------------------------
    Arne Opdal
    ------------------------------

    Original Message


  • 3.  RE: CX switches drops ip packets if checksum is not good...

    Posted Jan 06, 2023 03:20 AM
    Leagal checksum yes but not right one. Wireshark also reports checksum to be wrong in every other packet and in simultanious packet capture from both ports it's easy to see packet drop between in and out ports.
    Also QSC has done problem fixing beta fw to this bug but I think Aruba also should create documentation about this new feature, create separate "bad header checksum" counter and possible to give user a possibility to disable such feature.

    ------------------------------
    Jori Luoto
    ------------------------------

    Original Message


  • 4.  RE: CX switches drops ip packets if checksum is not good...

    Posted Jan 06, 2023 10:38 AM
    If you have a TAC case open, please push for this. Let me known in a PM if you get stuck and the local HPE guys can't help.
    While this is 'grey area' and may cause issue with L3/firewall/security devices, I would personally think that for L2 switching there should be no L3 checksum checking, and for sure no dropping, or have the option to do that. If TAC has all the data and captures, that is the right path to get this fixed in product and documentation.

    ------------------------------
    Herman Robers
    ------------------------
    If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

    In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.
    ------------------------------

    Original Message


  • 5.  RE: CX switches drops ip packets if checksum is not good...

    Posted Jun 10, 2024 09:18 PM

    Did you get this resolved with new firmware? Having this issue now on l2 wan vlan on switch causing global upload slowness.


    Original Message


  • 6.  RE: CX switches drops ip packets if checksum is not good...

    Posted Jun 11, 2024 12:41 AM

    Yes my problem was originally fixed with special fw and now fixing feature behavior change is implemented to version 10.13->

    Are you sure that you are suffering this problem? I mean this was caused by 3rd party product and switch just react to it...



    ------------------------------
    Jori Luoto
    AV-IT Specialist
    ------------------------------

    Original Message


  • 7.  RE: CX switches drops ip packets if checksum is not good...

    Posted Jun 17, 2024 08:09 AM

    I understand it is a unique problem. We believe our Verizon ONT (Nokia) is source of bad checksum and the Verizon plugs into WAN VLAN on CX Switch. Opened tickets with Aruba and got them closed. Introducing a dumb switch between Verizon ONT and Aruba CX "fixes" the issues. 


    Original Message


  • 8.  RE: CX switches drops ip packets if checksum is not good...

    Posted Feb 01, 2023 09:25 AM
    There will be fixing production level service release for 6100 and 6300 series in end of february where cx not drop bad checksum packets anymore👌

    ------------------------------
    Jori Luoto
    ------------------------------

    Original Message


  • 9.  RE: CX switches drops ip packets if checksum is not good...

    Posted Jun 26, 2024 03:34 AM

    Hi Jori - May i know what firmware version you where running, when you had the issue?


    Original Message


  • 10.  RE: CX switches drops ip packets if checksum is not good...

    Posted Jun 26, 2024 03:46 AM

    10.12 and below...Special version which I got from development was 10.10.1031.

    This is not yet fact but it seems that 10.13.1010 has (possibly) same "feature" than before because QSC QiO module seems to act same way than before so I'm not sure if it's fixed on that version either(?!?!)

    Due to QSC software features needed, we had no change to update QSC to problem fixing level so we are still running "broken version" there.



    ------------------------------
    Jori Luoto
    AV-IT Specialist
    ------------------------------

    Original Message


  • 11.  RE: CX switches drops ip packets if checksum is not good...

    Posted Jul 10, 2024 11:54 AM

    Oh my,

    Feature which drops faulty checksum packets still exists in 10.13.1010.... I'm now running with 6300 series which is L3 device but still I really would like to switch this one off.



    ------------------------------
    Jori Luoto
    AV-IT Specialist
    ------------------------------

    Original Message


  • 12.  RE: CX switches drops ip packets if checksum is not good...

    Posted Jul 11, 2024 04:02 AM

    Please work with Aruba Support or with your local Aruba partner to get this addressed with the product teams. Probably everyone reading this agrees with that this needs to be addressed, but it has to be sent to engineering in order to actually get it done.



    ------------------------------
    Herman Robers
    ------------------------
    If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

    In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.
    ------------------------------

    Original Message


  • 13.  RE: CX switches drops ip packets if checksum is not good...

    Posted Jul 11, 2024 04:58 AM

    Will do.

    Yesterday I got info from ERT guys that fix should be inserted to fw 10.13.1000-> but today I did tests with recommended version 10.13.1030 but no success with that either.

    Let's see how this goes.



    ------------------------------
    Jori Luoto
    AV-IT Specialist
    ------------------------------

    Original Message


  • 14.  RE: CX switches drops ip packets if checksum is not good...
    Best Answer

    Posted Jul 15, 2024 01:40 AM

    Resolution arrived!

    In 10.13.1000 and later firmware in L2 vlan context there is configuration "ip header-error-ignore" which seems to fix my issue.

    Small downside is that you cannot have SVI attached to vlan while using this configuration. With this component combination I can see few possible use scenarios where there is need to have SVI and header-error-ignore enabled at same time.



    ------------------------------
    Jori Luoto
    AV-IT Specialist
    ------------------------------

    Original Message


Related Content