Cloud Managed Networks

Hi all,

We've experienced some very weird things from Central in the past couple of days and I was wondering if anyone else has?

We have 70-odd AP's (almost all 315's) that are managed through Central which has been great for the last 4 years we've had them but yesterday they seem to have almost all started misbehaving.  We're still investigating - with the help of HPE/Aruba - but it's a slow process because the behaviour is so odd.

Currently, no AP's receive config updates.  They're still showing as online in Central and we've confirmed they can communicate from our network to Central.  You can change anything in Central and it's just not pushed to the AP's, not even something simple like blinking the locator LED.

To make it more odd, if we power-cycle an AP they lose all config, can't reconnect to Central and are stuck with what looks like a default config.  Admittedly we've only seen that with 2 AP's out of the cluster but we're too afraid to test another one because we can't afford to have any more down.

I was wondering if anyone else has seen anything like this?  We're starting to think it's down to a bug or something weird in Central rather than anything we've done, partly because we've eliminated almost every possibility on our end, and partly because we discovered another bug with Central about 2 weeks ago with AP configuration.

Anyone else having problems like this?  Also - I'm open to thoughts on how to troubleshoot further.

L8r.

were there any changes on the LAN side or your firewall lately?

also what firmware are you running? is it AOS10?

Hi all,

We've experienced some very weird things from Central in the past couple of days and I was wondering if anyone else has?

We have 70-odd AP's (almost all 315's) that are managed through Central which has been great for the last 4 years we've had them but yesterday they seem to have almost all started misbehaving.  We're still investigating - with the help of HPE/Aruba - but it's a slow process because the behaviour is so odd.

Currently, no AP's receive config updates.  They're still showing as online in Central and we've confirmed they can communicate from our network to Central.  You can change anything in Central and it's just not pushed to the AP's, not even something simple like blinking the locator LED.

To make it more odd, if we power-cycle an AP they lose all config, can't reconnect to Central and are stuck with what looks like a default config.  Admittedly we've only seen that with 2 AP's out of the cluster but we're too afraid to test another one because we can't afford to have any more down.

I was wondering if anyone else has seen anything like this?  We're starting to think it's down to a bug or something weird in Central rather than anything we've done, partly because we've eliminated almost every possibility on our end, and partly because we discovered another bug with Central about 2 weeks ago with AP configuration.

Anyone else having problems like this?  Also - I'm open to thoughts on how to troubleshoot further.

L8r.

Hi,

@ariyap - the firewall was one of the first places we looked (the other being recent changes to the AP group). There had been no changes in or around the timeframe of the problem beginning. We can see traffic going through the firewall so it seems okay (we also have made no recent changes on the AP group either).

Firmware version is 8.6.0.9_79813 on all of them.

Thanks.

were there any changes on the LAN side or your firewall lately?

also what firmware are you running? is it AOS10?

Hi all,

We've experienced some very weird things from Central in the past couple of days and I was wondering if anyone else has?

We have 70-odd AP's (almost all 315's) that are managed through Central which has been great for the last 4 years we've had them but yesterday they seem to have almost all started misbehaving.  We're still investigating - with the help of HPE/Aruba - but it's a slow process because the behaviour is so odd.

Currently, no AP's receive config updates.  They're still showing as online in Central and we've confirmed they can communicate from our network to Central.  You can change anything in Central and it's just not pushed to the AP's, not even something simple like blinking the locator LED.

To make it more odd, if we power-cycle an AP they lose all config, can't reconnect to Central and are stuck with what looks like a default config.  Admittedly we've only seen that with 2 AP's out of the cluster but we're too afraid to test another one because we can't afford to have any more down.

I was wondering if anyone else has seen anything like this?  We're starting to think it's down to a bug or something weird in Central rather than anything we've done, partly because we've eliminated almost every possibility on our end, and partly because we discovered another bug with Central about 2 weeks ago with AP configuration.

Anyone else having problems like this?  Also - I'm open to thoughts on how to troubleshoot further.

L8r.

Hi again all,

Further to this - we have verified it's nothing to do with our firewall or network.  Here's what we did:

• Connect one of the failed AP's to the ethernet port on a 4G hotspot (so it's not connected to our network at all, not behind our firewall or even using the same ISP).
• Connect a console cable to the AP mentioned above.
• Factory reset the AP mentioned above using the reset hole on the back.
• Watch what happens.

The console shows a reset is confirmed.  My understanding is that the AP should then contact Central and get a config, but it doesn't. It in fact goes into a reboot loop.

Any thoughts?

Thanks.

Hi,

@ariyap - the firewall was one of the first places we looked (the other being recent changes to the AP group). There had been no changes in or around the timeframe of the problem beginning. We can see traffic going through the firewall so it seems okay (we also have made no recent changes on the AP group either).

Firmware version is 8.6.0.9_79813 on all of them.

Thanks.

were there any changes on the LAN side or your firewall lately?

also what firmware are you running? is it AOS10?

Hi all,

We've experienced some very weird things from Central in the past couple of days and I was wondering if anyone else has?

We have 70-odd AP's (almost all 315's) that are managed through Central which has been great for the last 4 years we've had them but yesterday they seem to have almost all started misbehaving.  We're still investigating - with the help of HPE/Aruba - but it's a slow process because the behaviour is so odd.

Currently, no AP's receive config updates.  They're still showing as online in Central and we've confirmed they can communicate from our network to Central.  You can change anything in Central and it's just not pushed to the AP's, not even something simple like blinking the locator LED.

To make it more odd, if we power-cycle an AP they lose all config, can't reconnect to Central and are stuck with what looks like a default config.  Admittedly we've only seen that with 2 AP's out of the cluster but we're too afraid to test another one because we can't afford to have any more down.

I was wondering if anyone else has seen anything like this?  We're starting to think it's down to a bug or something weird in Central rather than anything we've done, partly because we've eliminated almost every possibility on our end, and partly because we discovered another bug with Central about 2 weeks ago with AP configuration.

Anyone else having problems like this?  Also - I'm open to thoughts on how to troubleshoot further.

L8r.

When an AP in factory-default mode powers on and connects to the Internet, it will automatically check into Aruba Activate using Websocket. 

It is activate that will direct the AP to connect to Central by providing the correct URL.

do you use activate service and do you have any rules in it?

if not the "show ap debug cloud-server" could provide more clues.

Hi again all,

Further to this - we have verified it's nothing to do with our firewall or network.  Here's what we did:

• Connect one of the failed AP's to the ethernet port on a 4G hotspot (so it's not connected to our network at all, not behind our firewall or even using the same ISP).
• Connect a console cable to the AP mentioned above.
• Factory reset the AP mentioned above using the reset hole on the back.
• Watch what happens.

The console shows a reset is confirmed.  My understanding is that the AP should then contact Central and get a config, but it doesn't. It in fact goes into a reboot loop.

Any thoughts?

Thanks.

Hi,

@ariyap - the firewall was one of the first places we looked (the other being recent changes to the AP group). There had been no changes in or around the timeframe of the problem beginning. We can see traffic going through the firewall so it seems okay (we also have made no recent changes on the AP group either).

Firmware version is 8.6.0.9_79813 on all of them.

Thanks.

were there any changes on the LAN side or your firewall lately?

also what firmware are you running? is it AOS10?

Hi all,

We've experienced some very weird things from Central in the past couple of days and I was wondering if anyone else has?

We have 70-odd AP's (almost all 315's) that are managed through Central which has been great for the last 4 years we've had them but yesterday they seem to have almost all started misbehaving.  We're still investigating - with the help of HPE/Aruba - but it's a slow process because the behaviour is so odd.

Currently, no AP's receive config updates.  They're still showing as online in Central and we've confirmed they can communicate from our network to Central.  You can change anything in Central and it's just not pushed to the AP's, not even something simple like blinking the locator LED.

To make it more odd, if we power-cycle an AP they lose all config, can't reconnect to Central and are stuck with what looks like a default config.  Admittedly we've only seen that with 2 AP's out of the cluster but we're too afraid to test another one because we can't afford to have any more down.

I was wondering if anyone else has seen anything like this?  We're starting to think it's down to a bug or something weird in Central rather than anything we've done, partly because we've eliminated almost every possibility on our end, and partly because we discovered another bug with Central about 2 weeks ago with AP configuration.

Anyone else having problems like this?  Also - I'm open to thoughts on how to troubleshoot further.

L8r.

When an AP in factory-default mode powers on and connects to the Internet, it will automatically check into Aruba Activate using Websocket. 

It is activate that will direct the AP to connect to Central by providing the correct URL.

do you use activate service and do you have any rules in it?

if not the "show ap debug cloud-server" could provide more clues.

Hi again all,

Further to this - we have verified it's nothing to do with our firewall or network.  Here's what we did:

• Connect one of the failed AP's to the ethernet port on a 4G hotspot (so it's not connected to our network at all, not behind our firewall or even using the same ISP).
• Connect a console cable to the AP mentioned above.
• Factory reset the AP mentioned above using the reset hole on the back.
• Watch what happens.

The console shows a reset is confirmed.  My understanding is that the AP should then contact Central and get a config, but it doesn't. It in fact goes into a reboot loop.

Any thoughts?

Thanks.

Hi,

@ariyap - the firewall was one of the first places we looked (the other being recent changes to the AP group). There had been no changes in or around the timeframe of the problem beginning. We can see traffic going through the firewall so it seems okay (we also have made no recent changes on the AP group either).

Firmware version is 8.6.0.9_79813 on all of them.

Thanks.

were there any changes on the LAN side or your firewall lately?

also what firmware are you running? is it AOS10?

Hi all,

We've experienced some very weird things from Central in the past couple of days and I was wondering if anyone else has?

We have 70-odd AP's (almost all 315's) that are managed through Central which has been great for the last 4 years we've had them but yesterday they seem to have almost all started misbehaving.  We're still investigating - with the help of HPE/Aruba - but it's a slow process because the behaviour is so odd.

Currently, no AP's receive config updates.  They're still showing as online in Central and we've confirmed they can communicate from our network to Central.  You can change anything in Central and it's just not pushed to the AP's, not even something simple like blinking the locator LED.

To make it more odd, if we power-cycle an AP they lose all config, can't reconnect to Central and are stuck with what looks like a default config.  Admittedly we've only seen that with 2 AP's out of the cluster but we're too afraid to test another one because we can't afford to have any more down.

I was wondering if anyone else has seen anything like this?  We're starting to think it's down to a bug or something weird in Central rather than anything we've done, partly because we've eliminated almost every possibility on our end, and partly because we discovered another bug with Central about 2 weeks ago with AP configuration.

Anyone else having problems like this?  Also - I'm open to thoughts on how to troubleshoot further.

L8r.

Hi all,

We've experienced some very weird things from Central in the past couple of days and I was wondering if anyone else has?

We have 70-odd AP's (almost all 315's) that are managed through Central which has been great for the last 4 years we've had them but yesterday they seem to have almost all started misbehaving.  We're still investigating - with the help of HPE/Aruba - but it's a slow process because the behaviour is so odd.

Currently, no AP's receive config updates.  They're still showing as online in Central and we've confirmed they can communicate from our network to Central.  You can change anything in Central and it's just not pushed to the AP's, not even something simple like blinking the locator LED.

To make it more odd, if we power-cycle an AP they lose all config, can't reconnect to Central and are stuck with what looks like a default config.  Admittedly we've only seen that with 2 AP's out of the cluster but we're too afraid to test another one because we can't afford to have any more down.

I was wondering if anyone else has seen anything like this?  We're starting to think it's down to a bug or something weird in Central rather than anything we've done, partly because we've eliminated almost every possibility on our end, and partly because we discovered another bug with Central about 2 weeks ago with AP configuration.

Anyone else having problems like this?  Also - I'm open to thoughts on how to troubleshoot further.

L8r.

Did you check if subscriptions are still valid?

Best, Gorazd

Hi all,

We've experienced some very weird things from Central in the past couple of days and I was wondering if anyone else has?

We have 70-odd AP's (almost all 315's) that are managed through Central which has been great for the last 4 years we've had them but yesterday they seem to have almost all started misbehaving.  We're still investigating - with the help of HPE/Aruba - but it's a slow process because the behaviour is so odd.

Currently, no AP's receive config updates.  They're still showing as online in Central and we've confirmed they can communicate from our network to Central.  You can change anything in Central and it's just not pushed to the AP's, not even something simple like blinking the locator LED.

To make it more odd, if we power-cycle an AP they lose all config, can't reconnect to Central and are stuck with what looks like a default config.  Admittedly we've only seen that with 2 AP's out of the cluster but we're too afraid to test another one because we can't afford to have any more down.

I was wondering if anyone else has seen anything like this?  We're starting to think it's down to a bug or something weird in Central rather than anything we've done, partly because we've eliminated almost every possibility on our end, and partly because we discovered another bug with Central about 2 weeks ago with AP configuration.

Anyone else having problems like this?  Also - I'm open to thoughts on how to troubleshoot further.

L8r.

Hi again all,

I think we've worked out what was wrong - just not how it went wrong (hopefully Aruba support can shed some light there).

The VC was in some sort of semi-offline state.  It was up and we could ping it or ssh/web to it but working with someone from Aruba, they showed me the papi-test command and it failed the test between a member AP and the VC AP.  Rebooting the VC fixed it.

There are some pretty strange things out of this though. First is that the VC was able to fail like that in the first place, but also that the behaviour of the APs was pretty surprising. That is, if they were rebooted they come up with no running config (well a basic config with only a few lines).

Then some also got stuck in a reboot-loop. At the moment they're still stuck, even if you power them on with no network - just power - they get stuck in the loop.  I'd say they were bricked except three times now we've seen them spontaneously recover for no obvious reason.

I understand that this may not have occurred with AOS10, and we may look at that but we're also some versions behind with v8 so I guess there's a chance there's a fix for this in a more recent 8.x release too.

In any case, if anyone's interested I'll report back what support find.

Thanks for your input.

L8r.

Did you check if subscriptions are still valid?

Best, Gorazd

Hi all,

We've experienced some very weird things from Central in the past couple of days and I was wondering if anyone else has?

We have 70-odd AP's (almost all 315's) that are managed through Central which has been great for the last 4 years we've had them but yesterday they seem to have almost all started misbehaving.  We're still investigating - with the help of HPE/Aruba - but it's a slow process because the behaviour is so odd.

Currently, no AP's receive config updates.  They're still showing as online in Central and we've confirmed they can communicate from our network to Central.  You can change anything in Central and it's just not pushed to the AP's, not even something simple like blinking the locator LED.

To make it more odd, if we power-cycle an AP they lose all config, can't reconnect to Central and are stuck with what looks like a default config.  Admittedly we've only seen that with 2 AP's out of the cluster but we're too afraid to test another one because we can't afford to have any more down.

I was wondering if anyone else has seen anything like this?  We're starting to think it's down to a bug or something weird in Central rather than anything we've done, partly because we've eliminated almost every possibility on our end, and partly because we discovered another bug with Central about 2 weeks ago with AP configuration.

Anyone else having problems like this?  Also - I'm open to thoughts on how to troubleshoot further.

L8r.

by any chance did you have "preferred master or conductor " configured?

Hi again all,

I think we've worked out what was wrong - just not how it went wrong (hopefully Aruba support can shed some light there).

The VC was in some sort of semi-offline state.  It was up and we could ping it or ssh/web to it but working with someone from Aruba, they showed me the papi-test command and it failed the test between a member AP and the VC AP.  Rebooting the VC fixed it.

There are some pretty strange things out of this though. First is that the VC was able to fail like that in the first place, but also that the behaviour of the APs was pretty surprising. That is, if they were rebooted they come up with no running config (well a basic config with only a few lines).

Then some also got stuck in a reboot-loop. At the moment they're still stuck, even if you power them on with no network - just power - they get stuck in the loop.  I'd say they were bricked except three times now we've seen them spontaneously recover for no obvious reason.

I understand that this may not have occurred with AOS10, and we may look at that but we're also some versions behind with v8 so I guess there's a chance there's a fix for this in a more recent 8.x release too.

In any case, if anyone's interested I'll report back what support find.

Thanks for your input.

L8r.

Did you check if subscriptions are still valid?

Best, Gorazd

Hi all,

We've experienced some very weird things from Central in the past couple of days and I was wondering if anyone else has?

We have 70-odd AP's (almost all 315's) that are managed through Central which has been great for the last 4 years we've had them but yesterday they seem to have almost all started misbehaving.  We're still investigating - with the help of HPE/Aruba - but it's a slow process because the behaviour is so odd.

Currently, no AP's receive config updates.  They're still showing as online in Central and we've confirmed they can communicate from our network to Central.  You can change anything in Central and it's just not pushed to the AP's, not even something simple like blinking the locator LED.

To make it more odd, if we power-cycle an AP they lose all config, can't reconnect to Central and are stuck with what looks like a default config.  Admittedly we've only seen that with 2 AP's out of the cluster but we're too afraid to test another one because we can't afford to have any more down.

I was wondering if anyone else has seen anything like this?  We're starting to think it's down to a bug or something weird in Central rather than anything we've done, partly because we've eliminated almost every possibility on our end, and partly because we discovered another bug with Central about 2 weeks ago with AP configuration.

Anyone else having problems like this?  Also - I'm open to thoughts on how to troubleshoot further.

L8r.