Wireless Access

Hi Airheads, 

First post here. 

We are beginning a rather large deployment and upgrade of our wireless and NAC. We have entered our pilot stage and installing the first 30 of ~7.5k APs and have hit a concerning issue with our 635 APs on our Cat 3650 switches (around 500 of these). 

These APs are rebooting every 50 mintes or so with current analysis looking like a PoE failure, or PoE negotiation problem. Interestingly we also 518s, 565s, and 655s connected to the same switches and not exhibiting the same issue as well as a number of other PoE devices including Cisco 3802 APs that are not failing. We also have a some of these 635s staying up for longer than 50 minutes on the same switch. 

We isolated an AP via power injector and it also rebooted while we had power inline enabled on the Cisco port it was connected to. 

Has anyone encountered anything similar to this before? 

We have engaged TAC, account manager and service delivery manager but sometimes the community have the best answers!

Cheers, 

Paddy

Have you enabled LLDP?

Switch (config)# lldp run

By Default Cisco Switches are doing Power Classification by CDP

Which version of code are you running?

Original Message:
Sent: Mar 09, 2024 09:28 PM
From: paddy
Subject: AP 635 PoE issues and random reboots.

Hi Airheads, 

First post here. 

We are beginning a rather large deployment and upgrade of our wireless and NAC. We have entered our pilot stage and installing the first 30 of ~7.5k APs and have hit a concerning issue with our 635 APs on our Cat 3650 switches (around 500 of these). 

These APs are rebooting every 50 mintes or so with current analysis looking like a PoE failure, or PoE negotiation problem. Interestingly we also 518s, 565s, and 655s connected to the same switches and not exhibiting the same issue as well as a number of other PoE devices including Cisco 3802 APs that are not failing. We also have a some of these 635s staying up for longer than 50 minutes on the same switch. 

We isolated an AP via power injector and it also rebooted while we had power inline enabled on the Cisco port it was connected to. 

Has anyone encountered anything similar to this before? 

We have engaged TAC, account manager and service delivery manager but sometimes the community have the best answers!

Cheers, 

Paddy

Hi Cordless, 

Yep LLDP is on. 

APs are running 10.4.0.2_87241. 

We have an interesting phenomenon occurring here. Overview of last weeks TS is as follows: 

-Saturday, 3 floors of ~10xAPs each, going to 3x different 3650s - All APs were rebooting ~50 minutes. 

-Sunday, rolled back 2 floors and migrated one floor to an aruba 6300, these worked

-Monday, plugged ~10 APs into our PoC environment (same switches and switch versions) these worked, so plugged them into Cisco switch that was affected on weekend, continued to work well. Migrated the affected APs on Aruba switch back to Cisco during Monday night, these have now been up since then

-Tuesday, attempted to re-migrate another problem floor and issue reintroduced itself for these APs. 

We cannot effectively replicate the issue and are at a loss as to why it appears to be occurring. 

Have you enabled LLDP?

Switch (config)# lldp run

By Default Cisco Switches are doing Power Classification by CDP

Which version of code are you running?

Original Message:
Sent: Mar 09, 2024 09:28 PM
From: paddy
Subject: AP 635 PoE issues and random reboots.

Hi Airheads, 

First post here. 

We are beginning a rather large deployment and upgrade of our wireless and NAC. We have entered our pilot stage and installing the first 30 of ~7.5k APs and have hit a concerning issue with our 635 APs on our Cat 3650 switches (around 500 of these). 

These APs are rebooting every 50 mintes or so with current analysis looking like a PoE failure, or PoE negotiation problem. Interestingly we also 518s, 565s, and 655s connected to the same switches and not exhibiting the same issue as well as a number of other PoE devices including Cisco 3802 APs that are not failing. We also have a some of these 635s staying up for longer than 50 minutes on the same switch. 

We isolated an AP via power injector and it also rebooted while we had power inline enabled on the Cisco port it was connected to. 

Has anyone encountered anything similar to this before? 

We have engaged TAC, account manager and service delivery manager but sometimes the community have the best answers!

Cheers, 

Paddy

We have identified the problem!!!

A very old script function to repair Cisco APs was detecting these as down and trying to bounce them!

Hi Cordless, 

Yep LLDP is on. 

APs are running 10.4.0.2_87241. 

We have an interesting phenomenon occurring here. Overview of last weeks TS is as follows: 

-Saturday, 3 floors of ~10xAPs each, going to 3x different 3650s - All APs were rebooting ~50 minutes. 

-Sunday, rolled back 2 floors and migrated one floor to an aruba 6300, these worked

-Monday, plugged ~10 APs into our PoC environment (same switches and switch versions) these worked, so plugged them into Cisco switch that was affected on weekend, continued to work well. Migrated the affected APs on Aruba switch back to Cisco during Monday night, these have now been up since then

-Tuesday, attempted to re-migrate another problem floor and issue reintroduced itself for these APs. 

We cannot effectively replicate the issue and are at a loss as to why it appears to be occurring. 

Have you enabled LLDP?

Switch (config)# lldp run

By Default Cisco Switches are doing Power Classification by CDP

Which version of code are you running?

Original Message:
Sent: Mar 09, 2024 09:28 PM
From: paddy
Subject: AP 635 PoE issues and random reboots.

Hi Airheads, 

First post here. 

We are beginning a rather large deployment and upgrade of our wireless and NAC. We have entered our pilot stage and installing the first 30 of ~7.5k APs and have hit a concerning issue with our 635 APs on our Cat 3650 switches (around 500 of these). 

These APs are rebooting every 50 mintes or so with current analysis looking like a PoE failure, or PoE negotiation problem. Interestingly we also 518s, 565s, and 655s connected to the same switches and not exhibiting the same issue as well as a number of other PoE devices including Cisco 3802 APs that are not failing. We also have a some of these 635s staying up for longer than 50 minutes on the same switch. 

We isolated an AP via power injector and it also rebooted while we had power inline enabled on the Cisco port it was connected to. 

Has anyone encountered anything similar to this before? 

We have engaged TAC, account manager and service delivery manager but sometimes the community have the best answers!

Cheers, 

Paddy