Wireless Access

Hi,

 

I faced the issues about APs (model 305) in VC hang and freeze. They occur many times a day. Now I am investigating log-files like:-

1.cli[2754]: <341014> <INFO> AP rebooting Gateway unreachable.

2.cli[2754]: <341014> <INFO> AP rebooting System cmd at uptime 0D 16H 18M 25S: Gateway unreachable.

3.nanny[2675]: <303086> <ERRS> Process Manager (nanny) shutting down - AP will reboot!

4.KERNEL [ 98.575880] VAP device aruba101 created osifp: (dc7fa5c0) os_if: (d7df0000)
sapd[2771]: Error opening /proc/sys/net/aruba101/amsdu_rx_max : No such file or directory

During 1 AP freezed, another APs also freezed.

In VC, It should be another APs handle clients from which one down.

 

The currently solution is unplug and plug POE to manual restart APs.

Please advise.

It looks like from the logs you posted that the APs are rebooting via the nanny process since they cannot reach the gateway. The nanny process will also reboot an AP if it cannot obtain an DHCP IP address after a certain time. Ideally, it would be good to have a console cable connected to confirm if the AP does start its boot process or does actually hang.

The first item we should be looking at is, why is the default gateway suddenly unavailable to the AP? Is there a DHCP issues, switch port issue, STP, POE budget issue etc?

 Hi zalion0,

 

Could you please explain more about nanny process?

I did not understand it.

 

All APs configured in static IP. While it happend with APs, another network devices work fine.

For console access, it is impossible to do because installed on the ceiling and no one to take it.

 

I think not on DHCP and switch port issues but did not check STP.

Please advise or any idea.

Hi, did you manage to solve this issue ?

Hello, could you solve this?

 

We are having the same issue with one of our customers.

AP rebooting System cmd at uptime 8D 0H 52M 37S: Gateway unreachable.

 

Thanks in advance!!

Hi,

 

Did you check the connection?

Make sure the connection is alway reach to each other.

 Hi, what you mean by checking connection ? ALL AP's are on the same switch and even if one lost connection this should no cause reboot of all of them. But this is happening .

We did try to disable preemption function ale all fallback options in uplink configuration and now it seems to be more stable.
https://community.arubanetworks.com/t5/Controller-less-WLANs/Static-IP-switch-to-DHCP-for-IAPs/ta-p/483080

 

I mean interface VLAN might go down that AP in the subnet.

TAC confirmed it is normal case if the gateway unreachable. I tried to explain that it does not make sense.

I tried all configured and upgrade firmware but still happen.

Preemption is disabled in my case.

---

@tatwftw wrote:

Hello, could you solve this?

 

We are having the same issue with one of our customers.

AP rebooting System cmd at uptime 8D 0H 52M 37S: Gateway unreachable.

 

Thanks in advance!!

---

What FW do you use ?

Tried FW below but cannot solve the issue.

6.5.2.x

6.5.4.x

Hi.

 

It failed with 8.3.0.6, previously it had 6.5.4.4 and it also failed.

Hi,

I am facing the very same issue. Contacted the TAC and no help.

They are blaming the network, DHCP reservation, power issue etc...The truth is network is working fine.

Did anyone manage to sove this issue??

 

Thanks

Nik

You did not say in your post what access points and what version of code your are running. You might be able to get help if you open that in a new post.

There are many networks of varying size that are running without issues, so I don't think size and scale are necessarily an issue. You could try to turn on broadcast multicast optimization on all of your clan interfaces if that has not been done already.

What do the system logs indicate?

What are the last transactions between the sapd and stm modules? Check for any specific AP Mac address that is problematic.

Have you tried enabling sapd debug on the Problematic APs?

Are any modules swamped with over utilization on the controller ?

What is the controller model, ap models , and base user count?

And what do you mean by hang? Are you not able to ping them or are you unable to connect to a SSID that they broadcast?

The WLANS are dissapperaing and then coming back again in 10 mins randomly--means all the IAPs were getting restarted ( Due to some issue : gateway unreachable) automatically when the issue was exhibiting. Also we have tested that the gateway was indeed reachable from another wired client in the same VLAN. The VC and all IAPs are all in the same VLAN. The VC will be unreachable and i belive this is why all the APS are getting restarted.

Reboot Time and Cause: AP rebooted Sat Sep 28 08:16:11 UTC 2019; System cmd at uptime 9D 23H 58M 19S: Gateway unreachable

Was an interface configuration change made to the gateway on a switch or any hops in between IAP and VC?

 

Are the packet interface counters incrementing regularly? or are the counters unchanged ? if they are unchanged when executing the command consecutively , for how long are they in an unchanged state till you see a visible change in the counters?

 

 

--Give Kudos: found something helpful, important, or cool? Click Kudos Star in a post.
--Problem Solved? Click "Accepted Solution" in a post.

Hi All,

 

There was no change in configurations in the VLAN or even in the network.
The VC and IAPs are all in the same dedicated VLAN. Like I mentioned before, I had connected a wired client from the same VLAN only to test the gateway connectivity when the VC disconnection became too frequent. This client had no packet drops when the IAPs were getting restarted saying the gateway unreachable.
I have come across the VC IP address configuration from the techdocs as below and in my case under the Virtual Controller network settings its selected 'Default',

 

"The IP configured for the Virtual Controller can be in the same subnet as IAP or can be in a different subnet. If the Virtual Controller IP is in a different subnet, configure the Virtual Controller mask, gateway, and VLAN as described in the following steps "

 

Now I happen to check the IAP User Guide to find the below

 

"If the Virtual Controller IP address is in the same
subnet as the IAP, ensure that you select Custom
from the drop-down list and configure the following
details:
l Virtual Controller Netmask—Enter subnet
mask details
l Virtual Controller Gateway—Enter a gateway --"

So here I am confused whether i need to keep the same settings or should i go ahead and change the Virtual Controller settings to "Custom" and enter the following??

Virtual Controller Netmask—

Virtual Controller Gateway—

Virtual Controller DNS— (???? is it required)

Virtual Controller VLAN—

 

Thanks

 

I think everyone who is having a problem in this thread should open an individual case with TAC so that their specific situation can be troubleshot.  It is very confusing to deal with multiple people who share a single message, but have different setups.

Hi cjoseph,

 

The reason i have posted this issue here is becasue I had already opened a case with the TAC and there is no significant change in the status of these issues yet. I can see there are others facing similar problems and i was seriously hoping to get some kind of solution to this.

 

We have already worked on our network infra to checkup on this issue and found nothing that points the issue to network side.

 

Thanks

 

 

There is nothing wrong with posting here.  Opening a case with TAC simultaneously would allow information that cannot be shared here to be collected.  It is entirely possible that users are having problems that look the same, but are different.

Hi cjoseph,

Totally agreeing to your points.
I was able to fix issues related to IAP and CP from the various community threads before when the TAC or even our partner support was of no help. We have already had multiple remote sessions with the TAC over a period of 1 month and like i mentioned before, the issue still persists.
Anyways always values inputs and suggestions from experts like you.

 

Thanks

Hi Nikhil,

 

This is what I understand of your issue thus far,

 

IAPs reboot multiple times over a period of time with the reboot cause being 'Gateway unreachable'.

 

Let me give you a backdrop on the reboot cause

 

1. When the gateway(Typically your master/uplink) is unreachable for a specific amount of time (~5-6 minutes)

 

2. When the uplink check (Between VC-master and IAP-slave)fails for the same amount of time (The uplink check fails if the packet stats (for the uplink interface) for Rx do not increment/stay stagnant for the above mentioned time)

 

The IAP will reboot if the above scenarios occur with the reboot cause: Gateway Unreachable

 

The gateway may be unreachable for a number of reasons.

 

To list a few,

 

1. There may be a new master election

2. The master may be down for some reason

3. Any hop inbetween the VC and the IAP might be facing connectivity issues

 

The IAPs then try to reconnect to the master like how they would when you add them anew to a network.

 

They try to arp for the master's IP that they get after a master election.

 

Now,

 

1. What is the instant version being used.
2. How is the master election taking place? Is it Manual or Automatic?

3. How many hops are there between the IAP-slave and the VC-master? 

4. Do any network devices in between the VC and IAP have DHCP Snooping enabled?

5. Is ARP allowed or is there any layer 2 mechanism blocking ARP messages?
6. Are you using any other network device with other mechanisms that may possibly block ARP requests?
7. ARP is broadcast, so has any feature that blocks/drops broadcast messages enabled?

 

--Give Kudos: found something helpful, important, or cool? Click Kudos Star in a post.
--Problem Solved? Click "Accepted Solution" in a post.

---

@nikhildask wrote:

The WLANS are dissapperaing and then coming back again in 10 mins randomly--means all the IAPs were getting restarted ( Due to some issue : gateway unreachable) automatically when the issue was exhibiting. Also we have tested that the gateway was indeed reachable from another wired client in the same VLAN. The VC and all IAPs are all in the same VLAN. The VC will be unreachable and i belive this is why all the APS are getting restarted.

Reboot Time and Cause: AP rebooted Sat Sep 28 08:16:11 UTC 2019; System cmd at uptime 9D 23H 58M 19S: Gateway unreachable

---

I would make sure that you are not accidentally giving out the gateway ip address in your dhcp scope to a client.

Hi Cjoseph,

 

In my customer environment, APs are in static IP and dedicated management subnet for them. I think it might be ignored about DHCP issue.

 

Hi all,

 

As Nikhildask said, this issue was point to the network as TAC talked to me in the ticket. 

After TAC recommended to upgrade version as I said in previous posted.

He accused that it is the network issue not AP issue.

I focused on wired network in same subnet reached the gateway but AP not.

I monitoring utilization port connected between AP and core switch as the gateway. The result was good not error or packet drop.

I tried to split AP out of the cluster but all AP randomly faced the issue.

 

Current status does not solve and priority on the network.

Thanks for the reply!
To give you an overview about the issue, we have close to 30 IAP 305s (6.5.4.3) in our environment. Ever since we had implemented, we have been facing random wireless disconnections on our wireless networks . After checking, its been noted that the VC isnt reachable and the APs are getting restarted hence the WLANs were disconnecting.
The logs on the APs indicated AP rebooted : Gateway unreachable and after contacting the TAC, we were asked to check for any issue with gateway reachability or any IP duplicate issue.
We were able to successfully ping the default gateway (from a wired client) when the issue was happening (VC unreachable) and there were other clients in the same VLAN which were working fine.

The following changes were done with no luck

 

1) Changed the preffered master multiple times.

2) Checked switch logs for any errors or high utilization.

3) Updated Switch Fw and even changed the PoE switch.

4) Reserved IPs for all the APs in DHCP

 

AP uptime is 1 day 3 hours 18 minutes 38 seconds
Reboot Time and Cause: AP rebooted Sat Sep 28 08:16:11 UTC 2019; System cmd at uptime 9D 23H 58M 19S: Gateway unreachable

 

As you had advised, I have turned On multicast optimization on all our WLANS. 
Any help on this would be highly appreciated.

 

Hello there.

After several tests with no results (changed gateway IP address, changed gateway device, change switches involved) we finally removed the 4 IAP cluster.

We installed another cluster of 7 IAPs (different IAPs) and as for today, not one reboot happened so we assume that it was a hardware or some kind of configuration issue. Not we or TAC knows why this happened, so, my advice is to remove that cluster or reset default config.

 

Best regards.

Gonzalo.

Hi,

 

I am not sure this case solve or not. Another engineer handle this.

But TAC confirmed it is network issue.