Wired Intelligent Edge

Hello,

End of previous year I did installation of CX6100 to my client who was running QSC QLAN system (proprietary 48khz/24bit HQ audio over lan stuff...) using installed switch. QLAN audio quality was really bad (robotic sound) while everythig else running in that same switch was just normal.
After days debugging I tested to replace 6100 with old 2540 and also QLAN started to work fine (whaaat?!?!?) after change back to 6100 robotic sound was back...

Local HPE guys found problem: There was 50% packet drop in port which sent audio stream to processor comparing to port which was inputting data from AD converter...
QLAN analog to digital converter (model QIO) was sending every other UDP packet with wrong L3 packet checksum (0xFFFFFF) and 6100 made decision drop all of these packets (while older L2-level AOS/Cisco/Extreme just passes them over without any checksum inspection)
Same new feature exists also in 6300M so this is un-documented AOS-CX feature which could cause unexpected problems for users.

There is ticket open about this to get more information about this and some configuration and logging features available to control this feature.

If somebody has additional information about this, I would be glad to hear about that.

------------------------------
Jori Luoto
------------------------------

Hi
If the checksum is 0xFFFF it is a legal checksum. Are you sure the reason for the packet drops is the UDP checksum and not somethin else? If you can do a packet capture on both ingress and egress of the switch and compare to find if its only the UDP 0xFFFF which are dropped. 

------------------------------
Arne Opdal
------------------------------

Original Message:
Sent: Jan 05, 2023 07:24 AM
From: Jori Luoto
Subject: CX switches drops ip packets if checksum is not good...

Hello,

End of previous year I did installation of CX6100 to my client who was running QSC QLAN system (proprietary 48khz/24bit HQ audio over lan stuff...) using installed switch. QLAN audio quality was really bad (robotic sound) while everythig else running in that same switch was just normal.
After days debugging I tested to replace 6100 with old 2540 and also QLAN started to work fine (whaaat?!?!?) after change back to 6100 robotic sound was back...

Local HPE guys found problem: There was 50% packet drop in port which sent audio stream to processor comparing to port which was inputting data from AD converter...
QLAN analog to digital converter (model QIO) was sending every other UDP packet with wrong L3 packet checksum (0xFFFFFF) and 6100 made decision drop all of these packets (while older L2-level AOS/Cisco/Extreme just passes them over without any checksum inspection)
Same new feature exists also in 6300M so this is un-documented AOS-CX feature which could cause unexpected problems for users.

There is ticket open about this to get more information about this and some configuration and logging features available to control this feature.

If somebody has additional information about this, I would be glad to hear about that.

------------------------------
Jori Luoto
------------------------------

Leagal checksum yes but not right one. Wireshark also reports checksum to be wrong in every other packet and in simultanious packet capture from both ports it's easy to see packet drop between in and out ports.
Also QSC has done problem fixing beta fw to this bug but I think Aruba also should create documentation about this new feature, create separate "bad header checksum" counter and possible to give user a possibility to disable such feature.

------------------------------
Jori Luoto
------------------------------

Original Message:
Sent: Jan 06, 2023 02:58 AM
From: Arne Opdal
Subject: CX switches drops ip packets if checksum is not good...

Hi
If the checksum is 0xFFFF it is a legal checksum. Are you sure the reason for the packet drops is the UDP checksum and not somethin else? If you can do a packet capture on both ingress and egress of the switch and compare to find if its only the UDP 0xFFFF which are dropped. 

------------------------------
Arne Opdal

Original Message:
Sent: Jan 05, 2023 07:24 AM
From: Jori Luoto
Subject: CX switches drops ip packets if checksum is not good...

Hello,

End of previous year I did installation of CX6100 to my client who was running QSC QLAN system (proprietary 48khz/24bit HQ audio over lan stuff...) using installed switch. QLAN audio quality was really bad (robotic sound) while everythig else running in that same switch was just normal.
After days debugging I tested to replace 6100 with old 2540 and also QLAN started to work fine (whaaat?!?!?) after change back to 6100 robotic sound was back...

Local HPE guys found problem: There was 50% packet drop in port which sent audio stream to processor comparing to port which was inputting data from AD converter...
QLAN analog to digital converter (model QIO) was sending every other UDP packet with wrong L3 packet checksum (0xFFFFFF) and 6100 made decision drop all of these packets (while older L2-level AOS/Cisco/Extreme just passes them over without any checksum inspection)
Same new feature exists also in 6300M so this is un-documented AOS-CX feature which could cause unexpected problems for users.

There is ticket open about this to get more information about this and some configuration and logging features available to control this feature.

If somebody has additional information about this, I would be glad to hear about that.

------------------------------
Jori Luoto
------------------------------

If you have a TAC case open, please push for this. Let me known in a PM if you get stuck and the local HPE guys can't help.
While this is 'grey area' and may cause issue with L3/firewall/security devices, I would personally think that for L2 switching there should be no L3 checksum checking, and for sure no dropping, or have the option to do that. If TAC has all the data and captures, that is the right path to get this fixed in product and documentation.

------------------------------
Herman Robers
------------------------
If you have urgent issues, always contact your Aruba partner, distributor, or Aruba TAC Support. Check https://www.arubanetworks.com/support-services/contact-support/ for how to contact Aruba TAC. Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.

In case your problem is solved, please invest the time to post a follow-up with the information on how you solved it. Others can benefit from that.
------------------------------

Original Message:
Sent: Jan 06, 2023 03:20 AM
From: Jori Luoto
Subject: CX switches drops ip packets if checksum is not good...

Leagal checksum yes but not right one. Wireshark also reports checksum to be wrong in every other packet and in simultanious packet capture from both ports it's easy to see packet drop between in and out ports.
Also QSC has done problem fixing beta fw to this bug but I think Aruba also should create documentation about this new feature, create separate "bad header checksum" counter and possible to give user a possibility to disable such feature.

------------------------------
Jori Luoto

Original Message:
Sent: Jan 06, 2023 02:58 AM
From: Arne Opdal
Subject: CX switches drops ip packets if checksum is not good...

Hi
If the checksum is 0xFFFF it is a legal checksum. Are you sure the reason for the packet drops is the UDP checksum and not somethin else? If you can do a packet capture on both ingress and egress of the switch and compare to find if its only the UDP 0xFFFF which are dropped. 

------------------------------
Arne Opdal

Original Message:
Sent: Jan 05, 2023 07:24 AM
From: Jori Luoto
Subject: CX switches drops ip packets if checksum is not good...

Hello,

End of previous year I did installation of CX6100 to my client who was running QSC QLAN system (proprietary 48khz/24bit HQ audio over lan stuff...) using installed switch. QLAN audio quality was really bad (robotic sound) while everythig else running in that same switch was just normal.
After days debugging I tested to replace 6100 with old 2540 and also QLAN started to work fine (whaaat?!?!?) after change back to 6100 robotic sound was back...

Local HPE guys found problem: There was 50% packet drop in port which sent audio stream to processor comparing to port which was inputting data from AD converter...
QLAN analog to digital converter (model QIO) was sending every other UDP packet with wrong L3 packet checksum (0xFFFFFF) and 6100 made decision drop all of these packets (while older L2-level AOS/Cisco/Extreme just passes them over without any checksum inspection)
Same new feature exists also in 6300M so this is un-documented AOS-CX feature which could cause unexpected problems for users.

There is ticket open about this to get more information about this and some configuration and logging features available to control this feature.

If somebody has additional information about this, I would be glad to hear about that.

------------------------------
Jori Luoto
------------------------------

There will be fixing production level service release for 6100 and 6300 series in end of february where cx not drop bad checksum packets anymore👌

------------------------------
Jori Luoto
------------------------------

Original Message:
Sent: Jan 05, 2023 07:24 AM
From: JL24
Subject: CX switches drops ip packets if checksum is not good...

Hello,

End of previous year I did installation of CX6100 to my client who was running QSC QLAN system (proprietary 48khz/24bit HQ audio over lan stuff...) using installed switch. QLAN audio quality was really bad (robotic sound) while everythig else running in that same switch was just normal.
After days debugging I tested to replace 6100 with old 2540 and also QLAN started to work fine (whaaat?!?!?) after change back to 6100 robotic sound was back...

Local HPE guys found problem: There was 50% packet drop in port which sent audio stream to processor comparing to port which was inputting data from AD converter...
QLAN analog to digital converter (model QIO) was sending every other UDP packet with wrong L3 packet checksum (0xFFFFFF) and 6100 made decision drop all of these packets (while older L2-level AOS/Cisco/Extreme just passes them over without any checksum inspection)
Same new feature exists also in 6300M so this is un-documented AOS-CX feature which could cause unexpected problems for users.

There is ticket open about this to get more information about this and some configuration and logging features available to control this feature.

If somebody has additional information about this, I would be glad to hear about that.

------------------------------
Jori Luoto
------------------------------