Seems we are hitting a bug found in version 8.10.0.5 causing auth process crash. After upgrading to version 8.10.0.7, the issue is resolved.
Original Message:
Sent: Jul 03, 2023 01:23 AM
From: timyuaruba
Subject: AOS 8.10 7280 Cluster issue
There is no ping drop between the Conductors and the MDs, basic checks have been done and there is no clues showing any poacket loss. The cluster is working well in version 8.7 but after upgrade we start to see auth process crashes and cluster disconnect events.
x.x.x.x ip address is the other controller in the cluster.
Original Message:
Sent: Jul 02, 2023 10:47 PM
From: cjoseph
Subject: AOS 8.10 7280 Cluster issue
You have a connectivity issue between your cluster members and your conductor.
I would check to make sure you don't have a duplicate ip address in your network.
I would do a basic ping to your conductor from the wired network and your Mds from the wired network to see if any pings drop. If they do, you have to troubleshoot a basic connectivity issue like, duplicate ip address or bad patch cabling. I would also look at the port statistics on the interfaces where your controllers connect to your switches.
What is the device at the x.x.x.x ip address?
------------------------------
Any opinions expressed here are solely my own and not necessarily that of Hewlett Packard Enterprise or Aruba Networks.
HPE Design and Deploy Guides: https://community.arubanetworks.com/support/migrated-knowledge-base?attachments=&communitykey=dcc83c62-1a3a-4dd8-94dc-92968ea6fff1&pageindex=0&pagesize=12&search=&sort=most_recent&viewtype=card
Original Message:
Sent: Jul 02, 2023 08:57 AM
From: timyuaruba
Subject: AOS 8.10 7280 Cluster issue
Hello Airheads,
We are seeing a weird behaviour on a 7280 cluster. We have upgraded from code 8.7.1.11 ,which is working fine, to 8.10.0.6 as 8.7 is going eol. After Upgrade I found that there is random disconnection in the 7280 cluster. I can see some crash event matching the timestamp. Any suggestions?
-------------------------------------------------------------------------------------------------------------
#show lc-cluster history
-------------LEGEND-----------------
HRPC - HBT RCVD PEER CONN
HMPD - HBT MISS PEER DEAD
LMRPD - LINK MAP RCVD PEER DEAD
IDPD - IPSEC DOWN PEER DEAD
CPDPD - CRIT PROCESS DOWN PEER DEAD
CDPD - CLUSTER DISABLED PEER DEAD
------------------------------------
Cluster History
---------------------------------------
TIME IPv4 Address STATUS REASON
------------------------ --------------- ------------ -------
Sat Jul 1 15:30:23 2023 x.x.x.x CONNECTED HRPC
Sun Jul 2 09:56:04 2023 x.x.x.x DISCONNECTED CPDPD
Sun Jul 2 09:57:50 2023 x.x.x.x CONNECTED HRPC
Sun Jul 2 18:41:12 2023 x.x.x.x DISCONNECTED CPDPD
Sun Jul 2 18:42:57 2023 x.x.x.x CONNECTED HRPC
#show crashinfo
Crash Info Table
----------------
Crash Time Process Name
---------- ------------
July 02 09:56 auth
July 02 18:41 auth
Jul 2 09:56:16 2023 stm[15462]: PAPI_Security: Denying message 0 received from unauthenticated source x.x.x.x:8225 for PAPI port 8419
Jul 2 09:56:16 2023 stm[15462]: PAPI RxPacketFromSibyte: ACK to invalid packet SN = 0x0000da8f opcode=0x17
Jul 2 09:56:16 2023 stm[15462]: PAPI_Security: Denying message 0 received from unauthenticated source x.x.x.x:8225 for PAPI port 8419
Jul 2 09:56:16 2023 stm[15462]: PAPI RxPacketFromSibyte: ACK to invalid packet SN = 0x0000da92 opcode=0x17
Jul 2 09:56:16 2023 stm[15462]: PAPI_Security: Denying message 0 received from unauthenticated source x.x.x.x:8225 for PAPI port 8419
Jul 2 09:56:16 2023 stm[15462]: PAPI RxPacketFromSibyte: ACK to invalid packet SN = 0x0000da95 opcode=0x17
Jul 2 09:56:16 2023 stm[15462]: PAPI_Security: Denying message 0 received from unauthenticated source x.x.x.x:8225 for PAPI port 8419
Jul 2 09:56:16 2023 stm[15462]: PAPI RxPacketFromSibyte: ACK to invalid packet SN = 0x0000da98 opcode=0x17
Jul 2 09:56:16 2023 stm[15462]: PAPI_Security: Denying message 0 received from unauthenticated source x.x.x.x:8225 for PAPI port 8419
Jul 2 09:56:16 2023 stm[15462]: PAPI RxPacketFromSibyte: ACK to invalid packet SN = 0x0000da9b opcode=0x17
Jul 2 09:56:16 2023 stm[15462]: PAPI_Security: Denying message 0 received from unauthenticated source x.x.x.x:8225 for PAPI port 8419
Jul 2 09:56:16 2023 stm[15462]: PAPI RxPacketFromSibyte: ACK to invalid packet SN = 0x0000da9e opcode=0x17
Jul 2 09:56:16 2023 stm[15462]: PAPI_Security: Denying message 0 received from unauthenticated source x.x.x.x:8225 for PAPI port 8419
Jul 2 09:56:16 2023 stm[15462]: PAPI RxPacketFromSibyte: ACK to invalid packet SN = 0x0000daa1 opcode=0x17
Jul 2 09:56:16 2023 stm[15462]: PAPI_Security: Denying message 0 received from unauthenticated source x.x.x.x:8225 for PAPI port 8419
Jul 2 09:56:16 2023 stm[15462]: PAPI RxPacketFromSibyte: ACK to invalid packet SN = 0x0000daa4 opcode=0x17
Jul 2 09:56:16 2023 stm[15462]: PAPI_Security: Denying message 0 received from unauthenticated source x.x.x.x:8225 for PAPI port 8419
Jul 2 09:56:16 2023 stm[15462]: PAPI RxPacketFromSibyte: ACK to invalid packet SN = 0x0000daa7 opcode=0x17
Jul 2 09:56:16 2023 stm[15462]: PAPI_Security: Denying message 0 received from unauthenticated source x.x.x.x:8225 for PAPI port 8419
Jul 2 09:56:16 2023 stm[15462]: PAPI RxPacketFromSibyte: ACK to invalid packet SN = 0x0000daab opcode=0x17
Jul 2 09:56:16 2023 stm[15462]: PAPI_Security: Denying message 0 received from unauthenticated source x.x.x.x:8225 for PAPI port 8419
Jul 2 09:56:16 2023 stm[15462]: PAPI RxPacketFromSibyte: ACK to invalid packet SN = 0x0000daaf opcode=0x17
Jul 2 09:56:16 2023 stm[15462]: PAPI_Security: Denying message 0 received from unauthenticated source x.x.x.x:8225 for PAPI port 8419
Jul 2 18:41:21 2023 stm[15462]: PAPI RxPacketFromSibyte: ACK to invalid packet SN = 0x000055c9 opcode=0x17
Jul 2 18:41:21 2023 stm[15462]: PAPI_Security: Denying message 0 received from unauthenticated source x.x.x.x:8225 for PAPI port 8419
Jul 2 18:41:21 2023 stm[15462]: PAPI RxPacketFromSibyte: ACK to invalid packet SN = 0x000055cc opcode=0x17
Jul 2 18:41:21 2023 stm[15462]: PAPI_Security: Denying message 0 received from unauthenticated source x.x.x.x:8225 for PAPI port 8419
Jul 2 18:41:21 2023 stm[15462]: PAPI RxPacketFromSibyte: ACK to invalid packet SN = 0x000055cf opcode=0x17
Jul 2 18:41:21 2023 stm[15462]: PAPI_Security: Denying message 0 received from unauthenticated source x.x.x.x:8225 for PAPI port 8419
Jul 2 18:41:21 2023 stm[15462]: PAPI RxPacketFromSibyte: ACK to invalid packet SN = 0x000055d3 opcode=0x17
Jul 2 18:41:21 2023 stm[15462]: PAPI_Security: Denying message 0 received from unauthenticated source x.x.x.x:8225 for PAPI port 8419
Jul 2 18:41:21 2023 stm[15462]: PAPI RxPacketFromSibyte: ACK to invalid packet SN = 0x000055d6 opcode=0x17
Jul 2 18:41:21 2023 stm[15462]: PAPI_Security: Denying message 0 received from unauthenticated source x.x.x.x:8225 for PAPI port 8419
Jul 2 18:41:21 2023 stm[15462]: PAPI RxPacketFromSibyte: ACK to invalid packet SN = 0x000055d9 opcode=0x17
Jul 2 18:41:21 2023 stm[15462]: PAPI_Security: Denying message 0 received from unauthenticated source x.x.x.x:8225 for PAPI port 8419
Jul 2 18:41:21 2023 stm[15462]: PAPI RxPacketFromSibyte: ACK to invalid packet SN = 0x000055dc opcode=0x17
Jul 2 18:41:21 2023 stm[15462]: PAPI_Security: Denying message 0 received from unauthenticated source x.x.x.x:8225 for PAPI port 8419
Jul 2 18:41:21 2023 stm[15462]: PAPI RxPacketFromSibyte: ACK to invalid packet SN = 0x000055df opcode=0x17
Jul 2 18:41:21 2023 stm[15462]: PAPI_Security: Denying message 0 received from unauthenticated source x.x.x.x:8225 for PAPI port 8419
Jul 2 18:41:21 2023 stm[15462]: PAPI RxPacketFromSibyte: ACK to invalid packet SN = 0x000055e2 opcode=0x17
-------------------------------------------------------------------------------------------------------------