Setting the Scene
For a while I had been having inexplicable, intermittent issues with the access point plugged into my switch. The switch is a CX6300M that supports SmartRate (aka Multigig). This is a good thing, because the local AP is an AP515, with support for a 2.5Gb connection.
The Components
⦁ Aruba 6300M 48SR5 CL6 PoE 4SFP56 Switch (JL659A)
⦁ Aruba AP-515 (RW) Unified AP (Q9H62A)
⦁ Central is used to manage all switches and APs, including these
Symptoms
Some of the symptoms that I noticed:
- Wireless devices would occasionally have connectivity of throughput issues
- Devices would sometimes be unexpectedly connected to APs other than the 515
- AI insights reported higher than normal errors (like DHCP or DNS)
- Problems were intermittent, and not readily reproducible
These switch event log entries were the most consistent indicator that something unusual was going on. These are all from port 1/1/26 - the AP515:
2021-07-03T11:28:25.291053+10:00 bvcore-6300 lldpd[3329]: Event|106|LOG_INFO|MSTR|1|LLDP neighbor 24:62:ce:cc:6d:22 deleted on 1/1/26
2021-07-03T11:26:45.347769+10:00 bvcore-6300 hpe-policyd[3602]: Event|5511|LOG_WARN|AMM|-|Interface 1/1/26 Link gone down
2021-07-03T11:26:41.573630+10:00 bvcore-6300 poe-protod[3319]: Event|7905|LOG_INFO|MSTR|1|Powered device disconnected on interface 1/1/26
2021-07-03T11:26:41.210722+10:00 bvcore-6300 intfd[652]: Event|404|LOG_INFO|UKWN|1|Link status for interface 1/1/26 is down
2021-07-03T11:26:37.267789+10:00 bvcore-6300 intfd[652]: Event|403|LOG_INFO|UKWN|1|Link status for interface 1/1/26 is up
2021-07-03T11:26:32.396348+10:00 bvcore-6300 hpe-policyd[3602]: Event|5511|LOG_WARN|AMM|-|Interface 1/1/26 Link came up
2021-07-03T11:26:32.142903+10:00 bvcore-6300 intfd[652]: Event|404|LOG_INFO|UKWN|1|Link status for interface 1/1/26 is down
2021-07-03T11:26:25.309276+10:00 bvcore-6300 hpe-policyd[3602]: Event|5511|LOG_WARN|AMM|-|Interface 1/1/26 Link gone down
2021-07-03T11:26:14.017325+10:00 bvcore-6300 intfd[652]: Event|403|LOG_INFO|UKWN|1|Link status for interface 1/1/26 is up
2021-07-03T11:26:09.177683+10:00 bvcore-6300 intfd[652]: Event|404|LOG_INFO|UKWN|1|Link status for interface 1/1/26 is down
Event ID 5511 is a good example for the link up/down.
bvcore-6300# show event -a -r -e 5511 | in 1/1/26
2021-07-03T11:26:45.347769+10:00 bvcore-6300 hpe-policyd[3602]: Event|5511|LOG_WARN|AMM|-|Interface 1/1/26 Link gone down
2021-07-03T11:26:32.396348+10:00 bvcore-6300 hpe-policyd[3602]: Event|5511|LOG_WARN|AMM|-|Interface 1/1/26 Link came up
2021-07-03T11:26:25.309276+10:00 bvcore-6300 hpe-policyd[3602]: Event|5511|LOG_WARN|AMM|-|Interface 1/1/26 Link gone down
2021-07-03T11:00:53.314973+10:00 bvcore-6300 hpe-policyd[3602]: Event|5511|LOG_WARN|AMM|-|Interface 1/1/26 Link came up
2021-07-03T11:00:45.299778+10:00 bvcore-6300 hpe-policyd[3602]: Event|5511|LOG_WARN|AMM|-|Interface 1/1/26 Link gone down
2021-07-03T10:58:03.307378+10:00 bvcore-6300 hpe-policyd[3602]: Event|5511|LOG_WARN|AMM|-|Interface 1/1/26 Link came up
2021-07-03T10:57:45.364306+10:00 bvcore-6300 hpe-policyd[3602]: Event|5511|LOG_WARN|AMM|-|Interface 1/1/26 Link gone down
2021-07-03T10:36:28.301364+10:00 bvcore-6300 hpe-policyd[3602]: Event|5511|LOG_WARN|AMM|-|Interface 1/1/26 Link came up
2021-07-03T10:36:20.304822+10:00 bvcore-6300 hpe-policyd[3602]: Event|5511|LOG_WARN|AMM|-|Interface 1/1/26 Link gone down
2021-07-03T10:35:28.314502+10:00 bvcore-6300 hpe-policyd[3602]: Event|5511|LOG_WARN|AMM|-|Interface 1/1/26 Link came up
2021-07-03T10:35:20.313240+10:00 bvcore-6300 hpe-policyd[3602]: Event|5511|LOG_WARN|AMM|-|Interface 1/1/26 Link gone down
2021-07-03T10:33:48.293762+10:00 bvcore-6300 hpe-policyd[3602]: Event|5511|LOG_WARN|AMM|-|Interface 1/1/26 Link came up
2021-07-03T10:33:40.318406+10:00 bvcore-6300 hpe-policyd[3602]: Event|5511|LOG_WARN|AMM|-|Interface 1/1/26 Link gone down
2021-07-03T10:33:08.318124+10:00 bvcore-6300 hpe-policyd[3602]: Event|5511|LOG_WARN|AMM|-|Interface 1/1/26 Link came up
2021-07-03T10:33:00.298939+10:00 bvcore-6300 hpe-policyd[3602]: Event|5511|LOG_WARN|AMM|-|Interface 1/1/26 Link gone down
Troubleshooting
I found a new Cat6 cable, and decided to install that instead - maybe the higher quality/increased frequency spec would work better with the 2.5Gb connection. And it did!
With the AP515 now on port 1/1/25 for the past six weeks:
bvcore-6300# show event -r -e 5511 | in 1/1/25
No errors!
Apart from the regular LLDP neighbour messages, nothing at all!
bvcore-6300# show event -r | in 1/1/25
:
2021-07-03T11:28:34.493665+10:00 bvcore-6300 lldpd[3329]: Event|105|LOG_INFO|MSTR|1|LLDP neighbor 24:62:ce:cc:6d:22 updated on 1/1/25
2021-07-03T11:28:11.209651+10:00 bvcore-6300 lldpd[3329]: Event|105|LOG_INFO|MSTR|1|LLDP neighbor 24:62:ce:cc:6d:22 updated on 1/1/25
2021-07-03T11:27:40.405618+10:00 bvcore-6300 lldpd[3329]: Event|104|LOG_INFO|MSTR|1|LLDP neighbor 24:62:ce:cc:6d:22 added on 1/1/25
2021-07-03T11:27:35.480595+10:00 bvcore-6300 hpe-mvrpd[3290]: Event|3101|LOG_INFO|MSTR|1|MVRP enabled on port 1/1/25
2021-07-03T11:27:34.403729+10:00 bvcore-6300 intfd[652]: Event|403|LOG_INFO|UKWN|1|Link status for interface 1/1/25 is up
2021-07-03T11:26:47.564129+10:00 bvcore-6300 poe-hald[3341]: Event|7902|LOG_INFO|MSTR|1|Powered device power delivery on interface 1/1/25
2021-07-03T11:26:47.559817+10:00 bvcore-6300 poe-protod[3319]: Event|7901|LOG_INFO|MSTR|1|Detected powered device on interface 1/1/25. Type:3, Class:5
Conclusion
Higher speed connections like 2.5 and 5Gb for APs are becoming more common now - don't skimp on the patch leads.
Network troubleshooting should always start at Layer 1 - the physical layer - and work up from there. Jumping on the CLI or NMS is nearly always where people start, and with good reason. However, then it is easy to forget Layer 1, especially if it has worked for a long time. However, you will occasionally have to go back and verify it. Don't forget Layer 1!
------------------------------
Richard Litchfield
Airheads MVP 2020, 2021
------------------------------