Wireless Access

I've been battling with this for weeks without a solution, so I'm turning to you :catsad:

 

We have a remote location connected via MPLS where we run two access points in campus mode. Both are connected to a managed HP Procurve PoE layer 3 switch.

 

One of the access points is operating perfectly, with no missed hearbeats or any other issues whatsoever.

The other access point is not recieving any heartbeats at all, and is rebootstraping extremely often (30 hearbeats missed). As a result the users gets connected/disconnected all the time when they're near this AP.

 

I've checked the following:

• MTU (set to 1400 due to the MPLS, verified with non-fragmented 1400 byte pings)
• IP address conflict (I've assigned the AP to three different IP's with no change in behaviour)
• Cables (every cable in the patch chain has been switched at least twice. I even tried to connect the AP directly to the switch)
• Switch (switch has been replaced, no change.)
• Switchport (zero errors on the port)
• Controller reboot: No help
• Wiped the cert from the AP: No help
• Grand finale: Changed the access point. Everything worked fine for one hour, and then the AP went back to not getting any heartbeats.

I'm at a loss here.. I've attached logs from the "show ap debug system-status" against the AP.

 

Any ideas?

Have you opened a case with TAC?

Hi,

 

No. We're using Dell branded Aruba gear, and TAC refuses to help us. Going through Dell support with this is pointless, as they don't know what I'm talking about half the time.

when you say replace AP did you take a new one, or did you swap them or such?

 

could always be a bug, are you on recent firmware?

Two things.

 

Firstly, can you post up a tech-support dump from a controller log please?

 

Secondly, what global region are you in? I do quite a bit with Dell OEM stuff. I have sensible contacts within the UK region, but if you're outside there, you might get routed to a team who aren't quite as, let's say "au-fait". Recent calls I've logged have been ok, now that I've been through a sensible "learning process" with them.

 

First of all, sorry for the late replies. Been sick.

 

@boneyard We replaced the AP with a new one, and a used one (so three AP's tested, all fail at this location)

 

@The.racking.monkey I'm in Norway, so I really have no idea where I end up when I call ProSupport. If you have technical contacts inside Dell who are good with Aruba gear then I'd be really happy if you could share their contact details in a PM.

 

I'm noticing something weird with our firewall. It's a Cisco ASA5510, which is in the middle between the AP's and the Controller. When I try to ping an AP from the VLAN where my workstation resides I can't get a reply from the AP. If I do the same from the vlan where the controller resides I do get a reply. When I check the firewall, the following messages pop up:

 

Denied ICMP type=0, from laddr 10.101.8.15 on interface mgmt-lim to 10.50.10.20: no matching session

 mgmt-lim is the interface where the controller resides, not the AP (10.101.18.15). Why would the ICMP reply come from the controller? We're running these AP's in campus mode with bridged VAP's to local MPLS routers.. is the controller acting as a VPN tunnel for the management IP of the AP's?

 

For some reason the forum doesn't allow me to upload attachments.. I'll try from another browser in a second.

 

 

Trying attachment again..

And now I'm getting these from several AP's that are also struggling with heartbeats:

 

Mar 12 14:01:37	 sapd[2361]: <311020> <ERRS> |AP TRM-JZ-AP01@192.168.40.15 sapd| An internal system error has occurred at file sapd_sysctl.c function sapd_sysctl_write_param line 102 error Error opening /proc/sys/dev/wifi1/tx_ed_threshold : No such file or directory.

 

Sorry for the reply spam. I just thought it would be a good idea to make a visio of the logical network design here.

There are much, much more vlans and firewall interfaces involved, but the design in the attached PDF is enough to get a overview of how I've done things.

Mar 12 14:01:37	 sapd[2361]: <311020> <ERRS> |AP TRM-JZ-AP01@192.168.40.15 sapd| An internal system error has occurred at file sapd_sysctl.c function sapd_sysctl_write_param line 102 error Error opening /proc/sys/dev/wifi1/tx_ed_threshold : No such file or directory.

These errors are likely related to a bug that will be fixed in 6.3.1.4. You can ignore these if your APs are up and servicing clients.

 

I'm having same issue.

 

Is there an estimated date for this new version?

Enrique, what issue specifically?  We have not identified the solution to the OP's problem, so there is no solution.

I mean the log message:

 

Mar 12 14:01:37	 sapd[2361]: <311020> <ERRS> |AP TRM-JZ-AP01@192.168.40.15 sapd| An internal system error has occurred at file sapd_sysctl.c function sapd_sysctl_write_param line 102 error Error opening /proc/sys/dev/wifi1/tx_ed_threshold : No such file or directory.

 

Since I updated to from 6.1 to 6.3.1.3 version, I see this message continuously.

And this thread it's the only reference about this message I could found.

---

@pauska wrote:

I'm noticing something weird with our firewall. It's a Cisco ASA5510, which is in the middle between the AP's and the Controller. When I try to ping an AP from the VLAN where my workstation resides I can't get a reply from the AP. If I do the same from the vlan where the controller resides I do get a reply. When I check the firewall, the following messages pop up:

 

Denied ICMP type=0, from laddr 10.101.8.15 on interface mgmt-lim to 10.50.10.20: no matching session

 mgmt-lim is the interface where the controller resides, not the AP (10.101.18.15). Why would the ICMP reply come from the controller? We're running these AP's in campus mode with bridged VAP's to local MPLS routers.. is the controller acting as a VPN tunnel for the management IP of the AP's?

 

---

that is expected behaviour. Aruba APs create a tunnel to there controller and traffic send to the AP is picked up and send trough this tunnel.

 

you seem to be able to trace this down to one location, i would focus on checking what is different between that location and the others.

 

which ArubaOS version is this, would an upgrade be possible?

Enrique,

The error message is cosmetic and should be fixed when 6.3.1.4 when it comes out. The OP has a connectivity issue with a few access points rebooting. Do you have the same issue?

No, I don't have that issue.

 

Sorry for the confusion.

No problem.

I am having a similar issue at a remote site.

 

Currently, the controller, 3200 6.3.1.3, resides at another location with multiple AP's. They are running fine and have users currently on them.

 

I then have another site running over an MPLS line with again, many AP's. ALl of a sudden a few days ago, the AP's stopped accepting clients. At this point, no one can connect to any of the AP's, although they are still broadcasting our SSID's.

 

As of right now, Aruba tried looking into this and have only determined that they are missing or not able to transmit any heartbeats.

 

I believe these AP's at the second site are in campus mode. They are all connected to Cisco switches. Both sites have a Sonicwall as a firewall/router.

 

Any ideas would be greatly appreciated.

any update on this? I have a customer with the same issue on T1 links. 

Josh,

 

Why don't you go ahead and open your own case?  If they figure out what the problem is, it will save you some time...especially if the OP is having a different issue than you are...

 

Yeah they had a case open but all TAC wanted them to do was purge the AP and try again. 

 

I've encouraged them to escalate the case. I'm not actively helping them but this was more for my own curiosity ;-) 

Josh,  got it.  Thanks.

 

Josh - did TAC solve this for you?

 

We're still experiencing the issue from time to time.

honestly i haven't worked with the customer any more on it. I've extended the heartbeat timers before to help with slow links with success.

We needed up upgrading the firmware of the controller to the latest version and we haven't had the same issue since. I read somewhere else that this also fixed the issue and in my companies case, this seems to have also resolved it.

On a side note, we did however switch providers and had severe issues due to misconfigured QoS on the routers. Either way, related or not, the issue hasn't happened in a good while.
---------------------------------------------------------------------------------------
This email has been scanned for email related threats and delivered safely by Mimecast.
For more information please visit http://www.mimecast.com
---------------------------------------------------------------------------------------