Wireless Access

Hi!

This might not be the right forum. We have 800 AP's deployed in different buildings, controlled by a cluster of 3 controllers (2x7210, 1x7220). We are not using ANY IoT features, and have no IoT config at all.

16th of May we upgraded from 8.10.0.10 to 8.10.0.12 because of the recent CVE's. This was rather uneventful, except at one particular building, where they use Assa Abloy Zigbee locks. 

The door lock system is not in anyway connected to the Aruba AP's (even though I have read that they could be) - they have their own zigbee hubs. This has worked for the past two years, since the building was new. (This is a care home for elderly people with dementia - so they need lots of locks...)

When the vendor of the lock system started to debug this, there was insane amounts of noise in the Zigbee-channels. They asked if it would be somehow related to wifi, which I denied. Unfortunately they disconnected an AP, and then it started working immediately around that AP. Same happened with other locks - disconnect the nearest AP, locks started working.

As I wrote, before this we had absolutely no IoT config. I have now tried different combinations with radio mode none and transmit power -40 - but it doesn't seem to fix it. I have also created a support case, but really not impressed by the commitment there...

Do anyone have any ideas? Or have a way to disable/verify that zigbee is disabled?

Regards,

Stian

Hi Stian,

1. What models are the APs and how does the Channel Utilization look for 2.4GHz on the APs in the building with the locks? Assuming these zigbee locks operate on 2.4GHz - (know zigbee also has some 800MHz and 900MHz ranges too).
2. Do you happen to have Airwave for historical channel utilization data?
3. If you plug the AP back in - does it immediately kill the zigbee locks or does it take time?
4. I was curious if you were seeing anything like in my attachment.

------------------------------
Chris

Original Message:
Sent: May 24, 2024 08:20 AM
From: Stian Jordet
Subject: Zigbee noise after upgrade

Hi!

This might not be the right forum. We have 800 AP's deployed in different buildings, controlled by a cluster of 3 controllers (2x7210, 1x7220). We are not using ANY IoT features, and have no IoT config at all.

16th of May we upgraded from 8.10.0.10 to 8.10.0.12 because of the recent CVE's. This was rather uneventful, except at one particular building, where they use Assa Abloy Zigbee locks. 

The door lock system is not in anyway connected to the Aruba AP's (even though I have read that they could be) - they have their own zigbee hubs. This has worked for the past two years, since the building was new. (This is a care home for elderly people with dementia - so they need lots of locks...)

When the vendor of the lock system started to debug this, there was insane amounts of noise in the Zigbee-channels. They asked if it would be somehow related to wifi, which I denied. Unfortunately they disconnected an AP, and then it started working immediately around that AP. Same happened with other locks - disconnect the nearest AP, locks started working.

As I wrote, before this we had absolutely no IoT config. I have now tried different combinations with radio mode none and transmit power -40 - but it doesn't seem to fix it. I have also created a support case, but really not impressed by the commitment there...

Do anyone have any ideas? Or have a way to disable/verify that zigbee is disabled?

Regards,

Stian

Hi Chris!

Thank you for your reply! I have the feeling we have hit the same thing here.

1: It's AP-635 (I was sure I had wrote that, sorry about that!) 

2: Sorry, no Airwave. 

3: This I haven't really tested properly, but after I thought I had disabled Zigbee (by making an IoT profile with radio mode "none", we replugged the AP's, and the doors still worked. I was very happy, untill the next morning when they called me and said it was just like the day before!

4: Yup, I can confirm this.

I disabled the 2,4 GHz radio, but that made no difference. But that was no big surprise, of course. 

I'm having a support session tomorrow morning, but my expectations are extremely low. Unless they have a immediate solution, I will downgrade tomorrow night.

Regards,

Stian

Original Message:
Sent: May 25, 2024 04:46 PM
From: cbjohns
Subject: Zigbee noise after upgrade

Hi Stian,

1. What models are the APs and how does the Channel Utilization look for 2.4GHz on the APs in the building with the locks? Assuming these zigbee locks operate on 2.4GHz - (know zigbee also has some 800MHz and 900MHz ranges too).
2. Do you happen to have Airwave for historical channel utilization data?
3. If you plug the AP back in - does it immediately kill the zigbee locks or does it take time?
4. I was curious if you were seeing anything like in my attachment.

------------------------------
Chris

Original Message:
Sent: May 24, 2024 08:20 AM
From: Stian Jordet
Subject: Zigbee noise after upgrade

Hi!

This might not be the right forum. We have 800 AP's deployed in different buildings, controlled by a cluster of 3 controllers (2x7210, 1x7220). We are not using ANY IoT features, and have no IoT config at all.

16th of May we upgraded from 8.10.0.10 to 8.10.0.12 because of the recent CVE's. This was rather uneventful, except at one particular building, where they use Assa Abloy Zigbee locks. 

The door lock system is not in anyway connected to the Aruba AP's (even though I have read that they could be) - they have their own zigbee hubs. This has worked for the past two years, since the building was new. (This is a care home for elderly people with dementia - so they need lots of locks...)

When the vendor of the lock system started to debug this, there was insane amounts of noise in the Zigbee-channels. They asked if it would be somehow related to wifi, which I denied. Unfortunately they disconnected an AP, and then it started working immediately around that AP. Same happened with other locks - disconnect the nearest AP, locks started working.

As I wrote, before this we had absolutely no IoT config. I have now tried different combinations with radio mode none and transmit power -40 - but it doesn't seem to fix it. I have also created a support case, but really not impressed by the commitment there...

Do anyone have any ideas? Or have a way to disable/verify that zigbee is disabled?

Regards,

Stian

Hi Stian,

Happy to help and thanks for the information. We had upgraded from 8.10.0.5 to 8.10.0.11 a couple weeks ago to resolve CVEs [8.10.0.12 is essentially the same as 8.10.0.11 - but only patched a webUI issue - AOS-253146 and AOS-254328. We don't have  zigbee door locks (well that's questionable), but I started seeing high channel utilization on 2.4GHz Radios on AP-635s - in locations that were upgraded from AP-225s. About 20 AP-635s out of 200 currently [in the middle of a hardware upgrade] - airwave helped show that this problem started when equipment was swapped - AND was not a problem on AP-635s prior to 8.10.0.11.

That led me to find AOS-245433 - when you speak with support tomorrow, make mention of that bug -  it was documented that it only affected 8.11 and 8.12 - but seems it is also affecting the 8.10 code now too.


My symptoms with AP-635s in this state were the following:

• The SSIDs are not beaconing properly on 2.4GHz Radio from doing an air capture/wi-fi analyzer.
• The 2.4GHz Radio are in a constant state of "reset" of 2 resets per second, essentially rendering the radio in-operable - show ap debug radio-stats ap-name <ap-name> radio 1
  
• If they check via "show ap arm rf-summary ap-name <ap-name" - the AP will most likely report no interference in the intf_idx - but you'll noticed that the util(Qual) displays high utilization on 1, 6, and 11.
  
  
  
• The issue seemed to affect AP-635s that may be on the perimeter of a building, but that may just be a red herring. I feel that some of the AP-635s may be reacting to something - but I've had a couple AP-635s that were practically in the middle of no where in a house that hit this problem.
• The last symptom we found - which was interesting. We were hunting for interference that a nearby AP was seeing with our fluke meter. Finally, we tried unplugging the AP-635 nearby that was exhibiting this high utilization, and the problem went away. We believe that when the AP-635 enters the "constant radio reset" state - it ends up generating it's own interference. The fluke meter is of our AP-635 with 0 clients on 2.4GHz - with no beacons being seen - and that drop is when we unplugged it. When we plugged the AP back in, all was well - for about 2 days.
  
  

I've been monitoring our AP-635s over the past few days. Rebooting the AP resolves the issue temporarily, but I've seen this issue crop up on APs within an hour or half a day. In fact, Thursday, I was doing a live site-survey, wasn't seeing my 2.4GHz SSID, and sure enough, the AP had only been online for about 5-10 minutes, entered into this bad state [which sounds awfully like your situation]. And this state can last for up to several days. Since the utilization seems to be real - it's definitely possible the the AP is saturating the spectrum and disrupting other devices on 2.4GHz.

We're currently awaiting word of a patch for 8.10 code train. Something I will mention. When you disabled the 2.4GHz radio, did you try rebooting the AP? Something I noticed, if I switch an affected AP-635s dot11g-radio-profile  to an "am-mode" or fully disabled the radio - and then back to "ap-mode" or re-enabled the radio - the utilization returns almost immediately - and the "radio resets" have incremented a large amount - almost as if it's not taking full affect. Note - while the am-mode was enabled - the reset counter DID NOT increment - till after I set it back to "ap-mode". And of course while the radio was disabled - "Command Failed: radio 1 down" - but when i re-enabled the radio - the reset counter had incremented over a 100. Note the rate of reset seems to be 2 resets per second - so about 110 resets a minute. You'll normally see radio resets, but not at this rate. I haven't had a chance to take a fluke or spectrum analyzer to one of these affected APs while the radio was disabled - I'll add that to my to-do list this week.

Hope this helps. And I thought about cutting out a bunch of the info, but thought I should share some of the info that led me down this route.

Good luck!

Hi Chris!

Thank you for your reply! I have the feeling we have hit the same thing here.

1: It's AP-635 (I was sure I had wrote that, sorry about that!) 

2: Sorry, no Airwave. 

3: This I haven't really tested properly, but after I thought I had disabled Zigbee (by making an IoT profile with radio mode "none", we replugged the AP's, and the doors still worked. I was very happy, untill the next morning when they called me and said it was just like the day before!

4: Yup, I can confirm this.

I disabled the 2,4 GHz radio, but that made no difference. But that was no big surprise, of course. 

I'm having a support session tomorrow morning, but my expectations are extremely low. Unless they have a immediate solution, I will downgrade tomorrow night.

Regards,

Stian

Hi Stian,

1. What models are the APs and how does the Channel Utilization look for 2.4GHz on the APs in the building with the locks? Assuming these zigbee locks operate on 2.4GHz - (know zigbee also has some 800MHz and 900MHz ranges too).
2. Do you happen to have Airwave for historical channel utilization data?
3. If you plug the AP back in - does it immediately kill the zigbee locks or does it take time?
4. I was curious if you were seeing anything like in my attachment.

------------------------------
Chris

Original Message:
Sent: May 24, 2024 08:20 AM
From: Stian Jordet
Subject: Zigbee noise after upgrade

Hi!

This might not be the right forum. We have 800 AP's deployed in different buildings, controlled by a cluster of 3 controllers (2x7210, 1x7220). We are not using ANY IoT features, and have no IoT config at all.

16th of May we upgraded from 8.10.0.10 to 8.10.0.12 because of the recent CVE's. This was rather uneventful, except at one particular building, where they use Assa Abloy Zigbee locks. 

The door lock system is not in anyway connected to the Aruba AP's (even though I have read that they could be) - they have their own zigbee hubs. This has worked for the past two years, since the building was new. (This is a care home for elderly people with dementia - so they need lots of locks...)

When the vendor of the lock system started to debug this, there was insane amounts of noise in the Zigbee-channels. They asked if it would be somehow related to wifi, which I denied. Unfortunately they disconnected an AP, and then it started working immediately around that AP. Same happened with other locks - disconnect the nearest AP, locks started working.

As I wrote, before this we had absolutely no IoT config. I have now tried different combinations with radio mode none and transmit power -40 - but it doesn't seem to fix it. I have also created a support case, but really not impressed by the commitment there...

Do anyone have any ideas? Or have a way to disable/verify that zigbee is disabled?

Regards,

Stian

Hi again, Chris!

Really thankful for your insight! Really very valuable! Appreciate you took your time! :)

I can confirm your findings about radio 1 resets and also about no noise in intf_idx but high utilization.

We did not try to restart the AP's after we disabled 2,4GHz, maybe that would help? The 2,4GHz LED was off, and we were happy with that. Does not own any wifi-tester. 

But regarding AOS-245433, are you sure about the number? When googling I get one hit in the release notes for 10.4.0.3 - and it doesn't seem relevant? Besides it's way too old? 

Other than that, having support session with Aruba in one hour. Will update after :)

Regards,

Stian

Hi Stian,

Happy to help and thanks for the information. We had upgraded from 8.10.0.5 to 8.10.0.11 a couple weeks ago to resolve CVEs [8.10.0.12 is essentially the same as 8.10.0.11 - but only patched a webUI issue - AOS-253146 and AOS-254328. We don't have  zigbee door locks (well that's questionable), but I started seeing high channel utilization on 2.4GHz Radios on AP-635s - in locations that were upgraded from AP-225s. About 20 AP-635s out of 200 currently [in the middle of a hardware upgrade] - airwave helped show that this problem started when equipment was swapped - AND was not a problem on AP-635s prior to 8.10.0.11.

That led me to find AOS-245433 - when you speak with support tomorrow, make mention of that bug -  it was documented that it only affected 8.11 and 8.12 - but seems it is also affecting the 8.10 code now too.


My symptoms with AP-635s in this state were the following:

• The SSIDs are not beaconing properly on 2.4GHz Radio from doing an air capture/wi-fi analyzer.
• The 2.4GHz Radio are in a constant state of "reset" of 2 resets per second, essentially rendering the radio in-operable - show ap debug radio-stats ap-name <ap-name> radio 1
  
• If they check via "show ap arm rf-summary ap-name <ap-name" - the AP will most likely report no interference in the intf_idx - but you'll noticed that the util(Qual) displays high utilization on 1, 6, and 11.
  
  
  
• The issue seemed to affect AP-635s that may be on the perimeter of a building, but that may just be a red herring. I feel that some of the AP-635s may be reacting to something - but I've had a couple AP-635s that were practically in the middle of no where in a house that hit this problem.
• The last symptom we found - which was interesting. We were hunting for interference that a nearby AP was seeing with our fluke meter. Finally, we tried unplugging the AP-635 nearby that was exhibiting this high utilization, and the problem went away. We believe that when the AP-635 enters the "constant radio reset" state - it ends up generating it's own interference. The fluke meter is of our AP-635 with 0 clients on 2.4GHz - with no beacons being seen - and that drop is when we unplugged it. When we plugged the AP back in, all was well - for about 2 days.
  
  

I've been monitoring our AP-635s over the past few days. Rebooting the AP resolves the issue temporarily, but I've seen this issue crop up on APs within an hour or half a day. In fact, Thursday, I was doing a live site-survey, wasn't seeing my 2.4GHz SSID, and sure enough, the AP had only been online for about 5-10 minutes, entered into this bad state [which sounds awfully like your situation]. And this state can last for up to several days. Since the utilization seems to be real - it's definitely possible the the AP is saturating the spectrum and disrupting other devices on 2.4GHz.

We're currently awaiting word of a patch for 8.10 code train. Something I will mention. When you disabled the 2.4GHz radio, did you try rebooting the AP? Something I noticed, if I switch an affected AP-635s dot11g-radio-profile  to an "am-mode" or fully disabled the radio - and then back to "ap-mode" or re-enabled the radio - the utilization returns almost immediately - and the "radio resets" have incremented a large amount - almost as if it's not taking full affect. Note - while the am-mode was enabled - the reset counter DID NOT increment - till after I set it back to "ap-mode". And of course while the radio was disabled - "Command Failed: radio 1 down" - but when i re-enabled the radio - the reset counter had incremented over a 100. Note the rate of reset seems to be 2 resets per second - so about 110 resets a minute. You'll normally see radio resets, but not at this rate. I haven't had a chance to take a fluke or spectrum analyzer to one of these affected APs while the radio was disabled - I'll add that to my to-do list this week.

Hope this helps. And I thought about cutting out a bunch of the info, but thought I should share some of the info that led me down this route.

Good luck!

Hi Chris!

Thank you for your reply! I have the feeling we have hit the same thing here.

1: It's AP-635 (I was sure I had wrote that, sorry about that!) 

2: Sorry, no Airwave. 

3: This I haven't really tested properly, but after I thought I had disabled Zigbee (by making an IoT profile with radio mode "none", we replugged the AP's, and the doors still worked. I was very happy, untill the next morning when they called me and said it was just like the day before!

4: Yup, I can confirm this.

I disabled the 2,4 GHz radio, but that made no difference. But that was no big surprise, of course. 

I'm having a support session tomorrow morning, but my expectations are extremely low. Unless they have a immediate solution, I will downgrade tomorrow night.

Regards,

Stian

Hi Stian,

1. What models are the APs and how does the Channel Utilization look for 2.4GHz on the APs in the building with the locks? Assuming these zigbee locks operate on 2.4GHz - (know zigbee also has some 800MHz and 900MHz ranges too).
2. Do you happen to have Airwave for historical channel utilization data?
3. If you plug the AP back in - does it immediately kill the zigbee locks or does it take time?
4. I was curious if you were seeing anything like in my attachment.

------------------------------
Chris

Original Message:
Sent: May 24, 2024 08:20 AM
From: Stian Jordet
Subject: Zigbee noise after upgrade

Hi!

This might not be the right forum. We have 800 AP's deployed in different buildings, controlled by a cluster of 3 controllers (2x7210, 1x7220). We are not using ANY IoT features, and have no IoT config at all.

16th of May we upgraded from 8.10.0.10 to 8.10.0.12 because of the recent CVE's. This was rather uneventful, except at one particular building, where they use Assa Abloy Zigbee locks. 

The door lock system is not in anyway connected to the Aruba AP's (even though I have read that they could be) - they have their own zigbee hubs. This has worked for the past two years, since the building was new. (This is a care home for elderly people with dementia - so they need lots of locks...)

When the vendor of the lock system started to debug this, there was insane amounts of noise in the Zigbee-channels. They asked if it would be somehow related to wifi, which I denied. Unfortunately they disconnected an AP, and then it started working immediately around that AP. Same happened with other locks - disconnect the nearest AP, locks started working.

As I wrote, before this we had absolutely no IoT config. I have now tried different combinations with radio mode none and transmit power -40 - but it doesn't seem to fix it. I have also created a support case, but really not impressed by the commitment there...

Do anyone have any ideas? Or have a way to disable/verify that zigbee is disabled?

Regards,

Stian

Hi Stian,

I apologize, I had a typo in the bug ID - AOS-245443 - I've included a couple screenshots. This seems to also exist in 10 code as well (hence that variant in release notes where it's grouped in three). But the AOS-245443 - "High channel utilization is observed in AP-635 access points running ArubaOS 8.11.0.0 or later versions. This issue is observed on 2.4 GHz radios." is the relevant one.

Hi again, Chris!

Really thankful for your insight! Really very valuable! Appreciate you took your time! :)

I can confirm your findings about radio 1 resets and also about no noise in intf_idx but high utilization.

We did not try to restart the AP's after we disabled 2,4GHz, maybe that would help? The 2,4GHz LED was off, and we were happy with that. Does not own any wifi-tester. 

But regarding AOS-245433, are you sure about the number? When googling I get one hit in the release notes for 10.4.0.3 - and it doesn't seem relevant? Besides it's way too old? 

Other than that, having support session with Aruba in one hour. Will update after :)

Regards,

Stian

Hi Stian,

Happy to help and thanks for the information. We had upgraded from 8.10.0.5 to 8.10.0.11 a couple weeks ago to resolve CVEs [8.10.0.12 is essentially the same as 8.10.0.11 - but only patched a webUI issue - AOS-253146 and AOS-254328. We don't have  zigbee door locks (well that's questionable), but I started seeing high channel utilization on 2.4GHz Radios on AP-635s - in locations that were upgraded from AP-225s. About 20 AP-635s out of 200 currently [in the middle of a hardware upgrade] - airwave helped show that this problem started when equipment was swapped - AND was not a problem on AP-635s prior to 8.10.0.11.

That led me to find AOS-245433 - when you speak with support tomorrow, make mention of that bug -  it was documented that it only affected 8.11 and 8.12 - but seems it is also affecting the 8.10 code now too.


My symptoms with AP-635s in this state were the following:

• The SSIDs are not beaconing properly on 2.4GHz Radio from doing an air capture/wi-fi analyzer.
• The 2.4GHz Radio are in a constant state of "reset" of 2 resets per second, essentially rendering the radio in-operable - show ap debug radio-stats ap-name <ap-name> radio 1
  
• If they check via "show ap arm rf-summary ap-name <ap-name" - the AP will most likely report no interference in the intf_idx - but you'll noticed that the util(Qual) displays high utilization on 1, 6, and 11.
  
  
  
• The issue seemed to affect AP-635s that may be on the perimeter of a building, but that may just be a red herring. I feel that some of the AP-635s may be reacting to something - but I've had a couple AP-635s that were practically in the middle of no where in a house that hit this problem.
• The last symptom we found - which was interesting. We were hunting for interference that a nearby AP was seeing with our fluke meter. Finally, we tried unplugging the AP-635 nearby that was exhibiting this high utilization, and the problem went away. We believe that when the AP-635 enters the "constant radio reset" state - it ends up generating it's own interference. The fluke meter is of our AP-635 with 0 clients on 2.4GHz - with no beacons being seen - and that drop is when we unplugged it. When we plugged the AP back in, all was well - for about 2 days.
  
  

I've been monitoring our AP-635s over the past few days. Rebooting the AP resolves the issue temporarily, but I've seen this issue crop up on APs within an hour or half a day. In fact, Thursday, I was doing a live site-survey, wasn't seeing my 2.4GHz SSID, and sure enough, the AP had only been online for about 5-10 minutes, entered into this bad state [which sounds awfully like your situation]. And this state can last for up to several days. Since the utilization seems to be real - it's definitely possible the the AP is saturating the spectrum and disrupting other devices on 2.4GHz.

We're currently awaiting word of a patch for 8.10 code train. Something I will mention. When you disabled the 2.4GHz radio, did you try rebooting the AP? Something I noticed, if I switch an affected AP-635s dot11g-radio-profile  to an "am-mode" or fully disabled the radio - and then back to "ap-mode" or re-enabled the radio - the utilization returns almost immediately - and the "radio resets" have incremented a large amount - almost as if it's not taking full affect. Note - while the am-mode was enabled - the reset counter DID NOT increment - till after I set it back to "ap-mode". And of course while the radio was disabled - "Command Failed: radio 1 down" - but when i re-enabled the radio - the reset counter had incremented over a 100. Note the rate of reset seems to be 2 resets per second - so about 110 resets a minute. You'll normally see radio resets, but not at this rate. I haven't had a chance to take a fluke or spectrum analyzer to one of these affected APs while the radio was disabled - I'll add that to my to-do list this week.

Hope this helps. And I thought about cutting out a bunch of the info, but thought I should share some of the info that led me down this route.

Good luck!

Hi Chris!

Thank you for your reply! I have the feeling we have hit the same thing here.

1: It's AP-635 (I was sure I had wrote that, sorry about that!) 

2: Sorry, no Airwave. 

3: This I haven't really tested properly, but after I thought I had disabled Zigbee (by making an IoT profile with radio mode "none", we replugged the AP's, and the doors still worked. I was very happy, untill the next morning when they called me and said it was just like the day before!

4: Yup, I can confirm this.

I disabled the 2,4 GHz radio, but that made no difference. But that was no big surprise, of course. 

I'm having a support session tomorrow morning, but my expectations are extremely low. Unless they have a immediate solution, I will downgrade tomorrow night.

Regards,

Stian

Hi Stian,

1. What models are the APs and how does the Channel Utilization look for 2.4GHz on the APs in the building with the locks? Assuming these zigbee locks operate on 2.4GHz - (know zigbee also has some 800MHz and 900MHz ranges too).
2. Do you happen to have Airwave for historical channel utilization data?
3. If you plug the AP back in - does it immediately kill the zigbee locks or does it take time?
4. I was curious if you were seeing anything like in my attachment.

------------------------------
Chris

Original Message:
Sent: May 24, 2024 08:20 AM
From: Stian Jordet
Subject: Zigbee noise after upgrade

Hi!

This might not be the right forum. We have 800 AP's deployed in different buildings, controlled by a cluster of 3 controllers (2x7210, 1x7220). We are not using ANY IoT features, and have no IoT config at all.

16th of May we upgraded from 8.10.0.10 to 8.10.0.12 because of the recent CVE's. This was rather uneventful, except at one particular building, where they use Assa Abloy Zigbee locks. 

The door lock system is not in anyway connected to the Aruba AP's (even though I have read that they could be) - they have their own zigbee hubs. This has worked for the past two years, since the building was new. (This is a care home for elderly people with dementia - so they need lots of locks...)

When the vendor of the lock system started to debug this, there was insane amounts of noise in the Zigbee-channels. They asked if it would be somehow related to wifi, which I denied. Unfortunately they disconnected an AP, and then it started working immediately around that AP. Same happened with other locks - disconnect the nearest AP, locks started working.

As I wrote, before this we had absolutely no IoT config. I have now tried different combinations with radio mode none and transmit power -40 - but it doesn't seem to fix it. I have also created a support case, but really not impressed by the commitment there...

Do anyone have any ideas? Or have a way to disable/verify that zigbee is disabled?

Regards,

Stian

Hi Stian,

Another update on the behavior. I confirmed that disabling the 2.4GHz radio on 2 of our AP-635s, was not sufficient to resolve the channel utilization/interference issue. I took my fluke meter shortly after disabling the 2.4GHz Radio - confirmed that CLI reported radio 1 (2.4GHz) was down - and the LED reported the same as well - but the Channel Utilization remained high until I power cycled the AP. I then did the same on a 2nd AP-635 that was exhibiting this issue - but gave it about 10 minutes after disabling the radio to clear up - still had to power cycle the AP.

Hi Stian,

I apologize, I had a typo in the bug ID - AOS-245443 - I've included a couple screenshots. This seems to also exist in 10 code as well (hence that variant in release notes where it's grouped in three). But the AOS-245443 - "High channel utilization is observed in AP-635 access points running ArubaOS 8.11.0.0 or later versions. This issue is observed on 2.4 GHz radios." is the relevant one.

Hi again, Chris!

Really thankful for your insight! Really very valuable! Appreciate you took your time! :)

I can confirm your findings about radio 1 resets and also about no noise in intf_idx but high utilization.

We did not try to restart the AP's after we disabled 2,4GHz, maybe that would help? The 2,4GHz LED was off, and we were happy with that. Does not own any wifi-tester. 

But regarding AOS-245433, are you sure about the number? When googling I get one hit in the release notes for 10.4.0.3 - and it doesn't seem relevant? Besides it's way too old? 

Other than that, having support session with Aruba in one hour. Will update after :)

Regards,

Stian

Hi Stian,

Happy to help and thanks for the information. We had upgraded from 8.10.0.5 to 8.10.0.11 a couple weeks ago to resolve CVEs [8.10.0.12 is essentially the same as 8.10.0.11 - but only patched a webUI issue - AOS-253146 and AOS-254328. We don't have  zigbee door locks (well that's questionable), but I started seeing high channel utilization on 2.4GHz Radios on AP-635s - in locations that were upgraded from AP-225s. About 20 AP-635s out of 200 currently [in the middle of a hardware upgrade] - airwave helped show that this problem started when equipment was swapped - AND was not a problem on AP-635s prior to 8.10.0.11.

That led me to find AOS-245433 - when you speak with support tomorrow, make mention of that bug -  it was documented that it only affected 8.11 and 8.12 - but seems it is also affecting the 8.10 code now too.


My symptoms with AP-635s in this state were the following:

• The SSIDs are not beaconing properly on 2.4GHz Radio from doing an air capture/wi-fi analyzer.
• The 2.4GHz Radio are in a constant state of "reset" of 2 resets per second, essentially rendering the radio in-operable - show ap debug radio-stats ap-name <ap-name> radio 1
  
• If they check via "show ap arm rf-summary ap-name <ap-name" - the AP will most likely report no interference in the intf_idx - but you'll noticed that the util(Qual) displays high utilization on 1, 6, and 11.
  
  
  
• The issue seemed to affect AP-635s that may be on the perimeter of a building, but that may just be a red herring. I feel that some of the AP-635s may be reacting to something - but I've had a couple AP-635s that were practically in the middle of no where in a house that hit this problem.
• The last symptom we found - which was interesting. We were hunting for interference that a nearby AP was seeing with our fluke meter. Finally, we tried unplugging the AP-635 nearby that was exhibiting this high utilization, and the problem went away. We believe that when the AP-635 enters the "constant radio reset" state - it ends up generating it's own interference. The fluke meter is of our AP-635 with 0 clients on 2.4GHz - with no beacons being seen - and that drop is when we unplugged it. When we plugged the AP back in, all was well - for about 2 days.
  
  

I've been monitoring our AP-635s over the past few days. Rebooting the AP resolves the issue temporarily, but I've seen this issue crop up on APs within an hour or half a day. In fact, Thursday, I was doing a live site-survey, wasn't seeing my 2.4GHz SSID, and sure enough, the AP had only been online for about 5-10 minutes, entered into this bad state [which sounds awfully like your situation]. And this state can last for up to several days. Since the utilization seems to be real - it's definitely possible the the AP is saturating the spectrum and disrupting other devices on 2.4GHz.

We're currently awaiting word of a patch for 8.10 code train. Something I will mention. When you disabled the 2.4GHz radio, did you try rebooting the AP? Something I noticed, if I switch an affected AP-635s dot11g-radio-profile  to an "am-mode" or fully disabled the radio - and then back to "ap-mode" or re-enabled the radio - the utilization returns almost immediately - and the "radio resets" have incremented a large amount - almost as if it's not taking full affect. Note - while the am-mode was enabled - the reset counter DID NOT increment - till after I set it back to "ap-mode". And of course while the radio was disabled - "Command Failed: radio 1 down" - but when i re-enabled the radio - the reset counter had incremented over a 100. Note the rate of reset seems to be 2 resets per second - so about 110 resets a minute. You'll normally see radio resets, but not at this rate. I haven't had a chance to take a fluke or spectrum analyzer to one of these affected APs while the radio was disabled - I'll add that to my to-do list this week.

Hope this helps. And I thought about cutting out a bunch of the info, but thought I should share some of the info that led me down this route.

Good luck!

Hi Chris!

Thank you for your reply! I have the feeling we have hit the same thing here.

1: It's AP-635 (I was sure I had wrote that, sorry about that!) 

2: Sorry, no Airwave. 

3: This I haven't really tested properly, but after I thought I had disabled Zigbee (by making an IoT profile with radio mode "none", we replugged the AP's, and the doors still worked. I was very happy, untill the next morning when they called me and said it was just like the day before!

4: Yup, I can confirm this.

I disabled the 2,4 GHz radio, but that made no difference. But that was no big surprise, of course. 

I'm having a support session tomorrow morning, but my expectations are extremely low. Unless they have a immediate solution, I will downgrade tomorrow night.

Regards,

Stian

Hi Stian,

1. What models are the APs and how does the Channel Utilization look for 2.4GHz on the APs in the building with the locks? Assuming these zigbee locks operate on 2.4GHz - (know zigbee also has some 800MHz and 900MHz ranges too).
2. Do you happen to have Airwave for historical channel utilization data?
3. If you plug the AP back in - does it immediately kill the zigbee locks or does it take time?
4. I was curious if you were seeing anything like in my attachment.

------------------------------
Chris

Original Message:
Sent: May 24, 2024 08:20 AM
From: Stian Jordet
Subject: Zigbee noise after upgrade

Hi!

This might not be the right forum. We have 800 AP's deployed in different buildings, controlled by a cluster of 3 controllers (2x7210, 1x7220). We are not using ANY IoT features, and have no IoT config at all.

16th of May we upgraded from 8.10.0.10 to 8.10.0.12 because of the recent CVE's. This was rather uneventful, except at one particular building, where they use Assa Abloy Zigbee locks. 

The door lock system is not in anyway connected to the Aruba AP's (even though I have read that they could be) - they have their own zigbee hubs. This has worked for the past two years, since the building was new. (This is a care home for elderly people with dementia - so they need lots of locks...)

When the vendor of the lock system started to debug this, there was insane amounts of noise in the Zigbee-channels. They asked if it would be somehow related to wifi, which I denied. Unfortunately they disconnected an AP, and then it started working immediately around that AP. Same happened with other locks - disconnect the nearest AP, locks started working.

As I wrote, before this we had absolutely no IoT config. I have now tried different combinations with radio mode none and transmit power -40 - but it doesn't seem to fix it. I have also created a support case, but really not impressed by the commitment there...

Do anyone have any ideas? Or have a way to disable/verify that zigbee is disabled?

Regards,

Stian

This has been a stubborn, inconsistent and hard to fix problem for some time, but as I understand it a fix is finally available and will be introduced in 8.12.0.2 and 10.6.0.1

Hi Stian,

Another update on the behavior. I confirmed that disabling the 2.4GHz radio on 2 of our AP-635s, was not sufficient to resolve the channel utilization/interference issue. I took my fluke meter shortly after disabling the 2.4GHz Radio - confirmed that CLI reported radio 1 (2.4GHz) was down - and the LED reported the same as well - but the Channel Utilization remained high until I power cycled the AP. I then did the same on a 2nd AP-635 that was exhibiting this issue - but gave it about 10 minutes after disabling the radio to clear up - still had to power cycle the AP.

Hi Stian,

I apologize, I had a typo in the bug ID - AOS-245443 - I've included a couple screenshots. This seems to also exist in 10 code as well (hence that variant in release notes where it's grouped in three). But the AOS-245443 - "High channel utilization is observed in AP-635 access points running ArubaOS 8.11.0.0 or later versions. This issue is observed on 2.4 GHz radios." is the relevant one.

Hi again, Chris!

Really thankful for your insight! Really very valuable! Appreciate you took your time! :)

I can confirm your findings about radio 1 resets and also about no noise in intf_idx but high utilization.

We did not try to restart the AP's after we disabled 2,4GHz, maybe that would help? The 2,4GHz LED was off, and we were happy with that. Does not own any wifi-tester. 

But regarding AOS-245433, are you sure about the number? When googling I get one hit in the release notes for 10.4.0.3 - and it doesn't seem relevant? Besides it's way too old? 

Other than that, having support session with Aruba in one hour. Will update after :)

Regards,

Stian

Hi Stian,

Happy to help and thanks for the information. We had upgraded from 8.10.0.5 to 8.10.0.11 a couple weeks ago to resolve CVEs [8.10.0.12 is essentially the same as 8.10.0.11 - but only patched a webUI issue - AOS-253146 and AOS-254328. We don't have  zigbee door locks (well that's questionable), but I started seeing high channel utilization on 2.4GHz Radios on AP-635s - in locations that were upgraded from AP-225s. About 20 AP-635s out of 200 currently [in the middle of a hardware upgrade] - airwave helped show that this problem started when equipment was swapped - AND was not a problem on AP-635s prior to 8.10.0.11.

That led me to find AOS-245433 - when you speak with support tomorrow, make mention of that bug -  it was documented that it only affected 8.11 and 8.12 - but seems it is also affecting the 8.10 code now too.


My symptoms with AP-635s in this state were the following:

• The SSIDs are not beaconing properly on 2.4GHz Radio from doing an air capture/wi-fi analyzer.
• The 2.4GHz Radio are in a constant state of "reset" of 2 resets per second, essentially rendering the radio in-operable - show ap debug radio-stats ap-name <ap-name> radio 1
  
• If they check via "show ap arm rf-summary ap-name <ap-name" - the AP will most likely report no interference in the intf_idx - but you'll noticed that the util(Qual) displays high utilization on 1, 6, and 11.
  
  
  
• The issue seemed to affect AP-635s that may be on the perimeter of a building, but that may just be a red herring. I feel that some of the AP-635s may be reacting to something - but I've had a couple AP-635s that were practically in the middle of no where in a house that hit this problem.
• The last symptom we found - which was interesting. We were hunting for interference that a nearby AP was seeing with our fluke meter. Finally, we tried unplugging the AP-635 nearby that was exhibiting this high utilization, and the problem went away. We believe that when the AP-635 enters the "constant radio reset" state - it ends up generating it's own interference. The fluke meter is of our AP-635 with 0 clients on 2.4GHz - with no beacons being seen - and that drop is when we unplugged it. When we plugged the AP back in, all was well - for about 2 days.
  
  

I've been monitoring our AP-635s over the past few days. Rebooting the AP resolves the issue temporarily, but I've seen this issue crop up on APs within an hour or half a day. In fact, Thursday, I was doing a live site-survey, wasn't seeing my 2.4GHz SSID, and sure enough, the AP had only been online for about 5-10 minutes, entered into this bad state [which sounds awfully like your situation]. And this state can last for up to several days. Since the utilization seems to be real - it's definitely possible the the AP is saturating the spectrum and disrupting other devices on 2.4GHz.

We're currently awaiting word of a patch for 8.10 code train. Something I will mention. When you disabled the 2.4GHz radio, did you try rebooting the AP? Something I noticed, if I switch an affected AP-635s dot11g-radio-profile  to an "am-mode" or fully disabled the radio - and then back to "ap-mode" or re-enabled the radio - the utilization returns almost immediately - and the "radio resets" have incremented a large amount - almost as if it's not taking full affect. Note - while the am-mode was enabled - the reset counter DID NOT increment - till after I set it back to "ap-mode". And of course while the radio was disabled - "Command Failed: radio 1 down" - but when i re-enabled the radio - the reset counter had incremented over a 100. Note the rate of reset seems to be 2 resets per second - so about 110 resets a minute. You'll normally see radio resets, but not at this rate. I haven't had a chance to take a fluke or spectrum analyzer to one of these affected APs while the radio was disabled - I'll add that to my to-do list this week.

Hope this helps. And I thought about cutting out a bunch of the info, but thought I should share some of the info that led me down this route.

Good luck!

Hi Chris!

Thank you for your reply! I have the feeling we have hit the same thing here.

1: It's AP-635 (I was sure I had wrote that, sorry about that!) 

2: Sorry, no Airwave. 

3: This I haven't really tested properly, but after I thought I had disabled Zigbee (by making an IoT profile with radio mode "none", we replugged the AP's, and the doors still worked. I was very happy, untill the next morning when they called me and said it was just like the day before!

4: Yup, I can confirm this.

I disabled the 2,4 GHz radio, but that made no difference. But that was no big surprise, of course. 

I'm having a support session tomorrow morning, but my expectations are extremely low. Unless they have a immediate solution, I will downgrade tomorrow night.

Regards,

Stian

Hi Stian,

1. What models are the APs and how does the Channel Utilization look for 2.4GHz on the APs in the building with the locks? Assuming these zigbee locks operate on 2.4GHz - (know zigbee also has some 800MHz and 900MHz ranges too).
2. Do you happen to have Airwave for historical channel utilization data?
3. If you plug the AP back in - does it immediately kill the zigbee locks or does it take time?
4. I was curious if you were seeing anything like in my attachment.

------------------------------
Chris

Original Message:
Sent: May 24, 2024 08:20 AM
From: Stian Jordet
Subject: Zigbee noise after upgrade

Hi!

This might not be the right forum. We have 800 AP's deployed in different buildings, controlled by a cluster of 3 controllers (2x7210, 1x7220). We are not using ANY IoT features, and have no IoT config at all.

16th of May we upgraded from 8.10.0.10 to 8.10.0.12 because of the recent CVE's. This was rather uneventful, except at one particular building, where they use Assa Abloy Zigbee locks. 

The door lock system is not in anyway connected to the Aruba AP's (even though I have read that they could be) - they have their own zigbee hubs. This has worked for the past two years, since the building was new. (This is a care home for elderly people with dementia - so they need lots of locks...)

When the vendor of the lock system started to debug this, there was insane amounts of noise in the Zigbee-channels. They asked if it would be somehow related to wifi, which I denied. Unfortunately they disconnected an AP, and then it started working immediately around that AP. Same happened with other locks - disconnect the nearest AP, locks started working.

As I wrote, before this we had absolutely no IoT config. I have now tried different combinations with radio mode none and transmit power -40 - but it doesn't seem to fix it. I have also created a support case, but really not impressed by the commitment there...

Do anyone have any ideas? Or have a way to disable/verify that zigbee is disabled?

Regards,

Stian

Excellent news! Also, since it seems they managed to back-port this bug to 8.10.0.12LSR, hope they are planning on applying the fix back here as well!

Any kind of time frame on when this fix may be available? 

Students are gone for the Summer, but come fall, it will be an issue if our newest wifi areas (Wi-Fi6E!) have non-functional 2.4 GHz coverage. Our summer plans include updating two more dorms from AP-2xx to AP-635, so I will be spreading this bug to all of our residential areas. (Unfortunately, I won't quite get through all of our installed AP-215s this Summer, so we are stuck at 8.10.x for now.)

This has been a stubborn, inconsistent and hard to fix problem for some time, but as I understand it a fix is finally available and will be introduced in 8.12.0.2 and 10.6.0.1

Hi Stian,

Another update on the behavior. I confirmed that disabling the 2.4GHz radio on 2 of our AP-635s, was not sufficient to resolve the channel utilization/interference issue. I took my fluke meter shortly after disabling the 2.4GHz Radio - confirmed that CLI reported radio 1 (2.4GHz) was down - and the LED reported the same as well - but the Channel Utilization remained high until I power cycled the AP. I then did the same on a 2nd AP-635 that was exhibiting this issue - but gave it about 10 minutes after disabling the radio to clear up - still had to power cycle the AP.

Hi Stian,

I apologize, I had a typo in the bug ID - AOS-245443 - I've included a couple screenshots. This seems to also exist in 10 code as well (hence that variant in release notes where it's grouped in three). But the AOS-245443 - "High channel utilization is observed in AP-635 access points running ArubaOS 8.11.0.0 or later versions. This issue is observed on 2.4 GHz radios." is the relevant one.

Hi again, Chris!

Really thankful for your insight! Really very valuable! Appreciate you took your time! :)

I can confirm your findings about radio 1 resets and also about no noise in intf_idx but high utilization.

We did not try to restart the AP's after we disabled 2,4GHz, maybe that would help? The 2,4GHz LED was off, and we were happy with that. Does not own any wifi-tester. 

But regarding AOS-245433, are you sure about the number? When googling I get one hit in the release notes for 10.4.0.3 - and it doesn't seem relevant? Besides it's way too old? 

Other than that, having support session with Aruba in one hour. Will update after :)

Regards,

Stian

Hi Stian,

Happy to help and thanks for the information. We had upgraded from 8.10.0.5 to 8.10.0.11 a couple weeks ago to resolve CVEs [8.10.0.12 is essentially the same as 8.10.0.11 - but only patched a webUI issue - AOS-253146 and AOS-254328. We don't have  zigbee door locks (well that's questionable), but I started seeing high channel utilization on 2.4GHz Radios on AP-635s - in locations that were upgraded from AP-225s. About 20 AP-635s out of 200 currently [in the middle of a hardware upgrade] - airwave helped show that this problem started when equipment was swapped - AND was not a problem on AP-635s prior to 8.10.0.11.

That led me to find AOS-245433 - when you speak with support tomorrow, make mention of that bug -  it was documented that it only affected 8.11 and 8.12 - but seems it is also affecting the 8.10 code now too.


My symptoms with AP-635s in this state were the following:

• The SSIDs are not beaconing properly on 2.4GHz Radio from doing an air capture/wi-fi analyzer.
• The 2.4GHz Radio are in a constant state of "reset" of 2 resets per second, essentially rendering the radio in-operable - show ap debug radio-stats ap-name <ap-name> radio 1
  
• If they check via "show ap arm rf-summary ap-name <ap-name" - the AP will most likely report no interference in the intf_idx - but you'll noticed that the util(Qual) displays high utilization on 1, 6, and 11.
  
  
  
• The issue seemed to affect AP-635s that may be on the perimeter of a building, but that may just be a red herring. I feel that some of the AP-635s may be reacting to something - but I've had a couple AP-635s that were practically in the middle of no where in a house that hit this problem.
• The last symptom we found - which was interesting. We were hunting for interference that a nearby AP was seeing with our fluke meter. Finally, we tried unplugging the AP-635 nearby that was exhibiting this high utilization, and the problem went away. We believe that when the AP-635 enters the "constant radio reset" state - it ends up generating it's own interference. The fluke meter is of our AP-635 with 0 clients on 2.4GHz - with no beacons being seen - and that drop is when we unplugged it. When we plugged the AP back in, all was well - for about 2 days.
  
  

I've been monitoring our AP-635s over the past few days. Rebooting the AP resolves the issue temporarily, but I've seen this issue crop up on APs within an hour or half a day. In fact, Thursday, I was doing a live site-survey, wasn't seeing my 2.4GHz SSID, and sure enough, the AP had only been online for about 5-10 minutes, entered into this bad state [which sounds awfully like your situation]. And this state can last for up to several days. Since the utilization seems to be real - it's definitely possible the the AP is saturating the spectrum and disrupting other devices on 2.4GHz.

We're currently awaiting word of a patch for 8.10 code train. Something I will mention. When you disabled the 2.4GHz radio, did you try rebooting the AP? Something I noticed, if I switch an affected AP-635s dot11g-radio-profile  to an "am-mode" or fully disabled the radio - and then back to "ap-mode" or re-enabled the radio - the utilization returns almost immediately - and the "radio resets" have incremented a large amount - almost as if it's not taking full affect. Note - while the am-mode was enabled - the reset counter DID NOT increment - till after I set it back to "ap-mode". And of course while the radio was disabled - "Command Failed: radio 1 down" - but when i re-enabled the radio - the reset counter had incremented over a 100. Note the rate of reset seems to be 2 resets per second - so about 110 resets a minute. You'll normally see radio resets, but not at this rate. I haven't had a chance to take a fluke or spectrum analyzer to one of these affected APs while the radio was disabled - I'll add that to my to-do list this week.

Hope this helps. And I thought about cutting out a bunch of the info, but thought I should share some of the info that led me down this route.

Good luck!

Hi Chris!

Thank you for your reply! I have the feeling we have hit the same thing here.

1: It's AP-635 (I was sure I had wrote that, sorry about that!) 

2: Sorry, no Airwave. 

3: This I haven't really tested properly, but after I thought I had disabled Zigbee (by making an IoT profile with radio mode "none", we replugged the AP's, and the doors still worked. I was very happy, untill the next morning when they called me and said it was just like the day before!

4: Yup, I can confirm this.

I disabled the 2,4 GHz radio, but that made no difference. But that was no big surprise, of course. 

I'm having a support session tomorrow morning, but my expectations are extremely low. Unless they have a immediate solution, I will downgrade tomorrow night.

Regards,

Stian

Hi Stian,

1. What models are the APs and how does the Channel Utilization look for 2.4GHz on the APs in the building with the locks? Assuming these zigbee locks operate on 2.4GHz - (know zigbee also has some 800MHz and 900MHz ranges too).
2. Do you happen to have Airwave for historical channel utilization data?
3. If you plug the AP back in - does it immediately kill the zigbee locks or does it take time?
4. I was curious if you were seeing anything like in my attachment.

------------------------------
Chris

Original Message:
Sent: May 24, 2024 08:20 AM
From: Stian Jordet
Subject: Zigbee noise after upgrade

Hi!

This might not be the right forum. We have 800 AP's deployed in different buildings, controlled by a cluster of 3 controllers (2x7210, 1x7220). We are not using ANY IoT features, and have no IoT config at all.

16th of May we upgraded from 8.10.0.10 to 8.10.0.12 because of the recent CVE's. This was rather uneventful, except at one particular building, where they use Assa Abloy Zigbee locks. 

The door lock system is not in anyway connected to the Aruba AP's (even though I have read that they could be) - they have their own zigbee hubs. This has worked for the past two years, since the building was new. (This is a care home for elderly people with dementia - so they need lots of locks...)

When the vendor of the lock system started to debug this, there was insane amounts of noise in the Zigbee-channels. They asked if it would be somehow related to wifi, which I denied. Unfortunately they disconnected an AP, and then it started working immediately around that AP. Same happened with other locks - disconnect the nearest AP, locks started working.

As I wrote, before this we had absolutely no IoT config. I have now tried different combinations with radio mode none and transmit power -40 - but it doesn't seem to fix it. I have also created a support case, but really not impressed by the commitment there...

Do anyone have any ideas? Or have a way to disable/verify that zigbee is disabled?

Regards,

Stian