Wireless Access

Since upgrading to 6.4.2.10 we've seen very odd behaviour from some of our AP125 units. Symptoms are that most users can't connect. We suspect the AP doesn't allow any new connections and any users still active just haven't left since the problem occurred. 

 

We use 802.1x authentication and users are asked for their password, even though it's correct. The correct password isn't accepted. 

 

A dead giveaway of this occurring is a low number of clients in the university library, which is always busy. As soon as a problem AP is rebooted, clients associate and authenticate successfully. A rebooted AP will be fine for a couple of days before it fails again. 

 

Auth-tracebuf seems to show the client repeatedly disconnecting during the auth process however we can be confident this is not a client issue.

 

I've raised a case with TAC but wondered if anyone else has seen this behaviour.

Ultimate Fish,

You did the right thing opening a TAC case. Did you turn in user debugging to have a clue what could be going on? If it is intermittent and random, there could be quite a few reasons. Is this a 100% ap-125 deployment or are there other access points?

It's really weird. It isn't really intermittent in that once an AP stops working it stays in that state, so there's the opportunity to do stuff and see how it responds. I turned on user debugging but didn't see much there, not entirely what I'm looking for.

 

We have about 450 125s but we also have 134/5, 105, 205, 225, 93h, 103h.

I've just got off the phone with the good people of the TAC. Turns out it's a known bug in 6.2.4.10 with client match. It apparently affects all APs, we're probably just seeing it on our 125s because they're often in busy areas such as the library.

 

Workaround is to disable client match. A fix should be out by the weekend.

Hello,

 

we 've hit this bug as well.

Aruba has added a remark on the software download page:

"If you have deployed AP-9x,10x,11x,12x,13x,17x, RAP-3,10x,15x and Client Match is enabled, please do not upgrade to this release."

 

If you have the above O/S and APs on your estate you can identify APs that have issues by running:

#show log system all | include "STM failed"

you will see messages like:

Aug 21 11:56:55 :311020:  <ERRS> |xxx@A.B.C.D sapd|  An internal system error has occurred at file sapd_msg.c function sapd_papi_snd_cb line 1410 error Message to 127.0.0.1:AP STM failed: err Connection timed out msgcode 0 arg (nil).

 

You could reboot APs if they are just a handfull, but contact TAC, as they can support further.

 

 

 

Thanks Gem, that's really helpful. We only had two APs left that were reported as problematic. I'd already been through every AP with 0 clients and rebooted those (about 400 APs) so nearly got them all ;)

 

We've scheduled an upgrade to the fixed version on Tuesday.

Hi, thanks :)

 

Did you have any update on this one from TAC?

We still the same issue, but on 6.4.12, I don't see any fix yet.

No... TAC haven't been very good on this. We've installed .12 but only by chance happened to notice that .11 breaks everything so, no thanks to an absence of updates from the TAC, we nearly installed another broken version of code. i've re-enabled client match and so far things are quiet so maybe all is well. However I'm out of the office, so I'll let my colleagues know the problem potentially continues.

 

Bit concerned about all this... Come on Aruba, don't you test releases any more?

Thanks, there has been a small issue with the updates this time indeed, but we discussed this with TAC and we have a clear view.

64.2.11 come out and went away very quickly as it had a major bug as you said. It's the only bug on the release notes of 6.4.2.11. To be honest I never even noticed the 6.4.2.11 being available for download.

 

However, looking at the archives, I found 6.4.2.11, which has a fix for our issue:

BUG ID: 122542, 121740, 122374, 122539,122914
Symptom: The station management process of the AP was busy, that resulted in clients failing to
associate with the AP. This issue is resolved by enhancing the WLAN driver.
Scenario: This issue was observed when client match was enabled and the controller moved the client
from one AP to another. This issue was observed on AP-110 Series, AP-120 Series, and AP-130 Series
access points associated with controllers running ArubaOS 6.3.1.x, 6.4.2.x, or 6.4.3.x.
Platform: AP-110 Series, AP-120 Series, and AP-130 Series access points.
Reported Version: ArubaOS 6.4.2.9

 

So as per standard practice, 6.4.2.12 contains all those fixes as well.

 

As the download page for current releace has the 6.4.2.10 and 6.4.2.12, I asked them to add a remark on the .12 that bug fixes compared to the .10 are on the archive folder, release .11

 

If that makes sence...

:)

In any case it appears to be fixed.

6.4.2.11 was available quite quickly after I'd made contact with the TAC but it has a bug that makes everything crash all the time (or words to that effect) so was pulled.

 

Our change management procedure requires we at least attempt to plan software upgrades, so we very nearly installed 6.4.2.11 which would have been a disaster. When we have campus wide wifi problems it makes us look bad, I don't like looking bad... Aruba keeps making us look bad :(

We have been having this problem for the past few days. We have two controllers, one with 6.3.x code and one with 6.4.x code. This is so that we can keep older AP's online untill they are replaced. Initally we were running 6.3.1.17 on one and 6.4.2.4 on the other. We turned client match off on all our controllers after find out this problem the hard way. We have upgraded as of this morning to 6.3.1.18 and 6.4.2.12 on the controllers. We haven't had as many reports of the new client connects but we are still getting them. I have an open case with TAC on this as well.

Do post the outcome, or any progress, here. 

 

Since upgrading to 6.2.4.12 we seem to be ok, but there are occasional niggles with some folk not being able to connect. Don't think it's related to the client match issue though as it seems to only affect our 802.1X authed ssid. 

 

Have to say that Aruba's code testing is a bit worrying... I've been in this job eight months and have seen quite a few show stoppers in that time. It makes our department look bad, and I don't like that.

I have to say the same as far as what I have seen in the last two years. We have been hit at least 3 times with major code level issues that has begun to make me wonder about any new code upgrade is how much going to be broken instead of fixed. I have to say I might be dating my self but I remember the days when the code was rock solid. It might have a feature that or command that didn't work the way you wanted to but at least controllers didn't reboot or AP's didn't stop responding.

It's a worry isn't it. Of our 2000 APs, 500 are AP125 which we know won't be supported after 6.2.4.x so we need to be thinking about upgrading them (in fact we've already started). But the number of problems we've seen make it tempting to look at trialling another vendor for some of these buildings. That would be painful (especially as we make use of Airwave) but it's more important the network actually works. 

 

Even if you don't need any of the fixes, you have to upgrade to support the new APs. We've been in a situation before where we couldn't use APs we'd purchased because the running OS didn't support them but the upgrade wouldn't support the old APs in production. So if you get to the stage you're scared to upgrade for fear of wrecking the network and incurring the wrath of thousands of users but you can't get new APs without upgrading.... Another vendor has to be considered.

Gilmorr and Ultimate fish, this probably goes without saying, but I hope you are working with your local Aruba representative on any systemic problems you have. If you have a large or diverse environment, no upgrade can be done without reviewing release notes thoroughly and working with your account team to figure out what is the best version of code you should be running. Some issues take a good amount of time to develop or even appear so working with your account team and soliciting input on the forum here will go a long way to covering your bases. There is no such thing as bug free code, but there are things to do to decrease the chance that you will run into systemic issues. Working with your account team closely provides that opportunity.

That's fair comment. We only do upgrades when we need to because a fix is required or AP support is required. We've experienced system problems that, according to our Aruba partner, nobody else in the world is experiencing... yet a fix appears in the next Aruba OS release. 

 

The most worry example is the recent ClientMatch is broken bug in 6.2.4.10, should be fixed in 6.2.4.11 followed by "woah... don't install that, it breaks everything... here's 6.2.4.12 which works fine, honestly". 

 

Big code doesn't come without bugs, I get that. But.... we're not trigger happy with updates and the show stoppers have been a problem for us. It 'feels' like some of the updates have been rushed out without adequate testing. I may be doing a hard working team a great disservice there, but when it goes down our users shout and swear at us, so we'll get a bit grumpy with Aruba.

Ultimate_Fish,

All valid comments. I just want to make sure that someone is at least advising you about the direction you are taking so that you do not go this alone. You have every right to complain when things don't go as planned. We do not know what to improve unless people complain.

Thanks Colin, it's good to know Aruba takes seriously comments made here ;)

Thanks for looking in to this. This should probably be moved to another thread but I wanted to make one final comment that might better frame my thoughts and that will hopefully will find its way to the correct management team.

I've worked with you on campus in the past and we have had Aruba for quite some time. So I've known Aruba Code for quite some time. What worries me is not the problem right now but the trend I have been seeing in its coded development. I know that issues need to be fixed ASAP but a lot of the new code is not as hardened as I think it should be. I can think of at least 4 issues right off the top of my head that have caused campus wide problems there were extended as we took time to debugged them. Now I know the system is of a good design and the support is doing its job by collecting as much info as possible and creating workarounds until code is released to fix the problem. But it's the things that get me is when its core coded level problems that can't really be fixed by anything other than a code upgrade.  From my software development days I know that there will always be bugs that come up as this feature or that module interacts with this one or that one. But at the basis there must be a core code that is rock solid. It used to be there, but I'm feeling that a lot of the code debugging is being done at the customer site instead of the development lab where it should be. Now eventually we do get the problem fixed but not until after I've been called to my boss's boss and had to explain why we have had yet another code level issue. At that point it makes no difference, it's my fault and I have to "own" the problem. As another user pointed out those are the ones that I really don't like because it makes us all look bad and there is nothing I can do to take steps to prevent it that case.  

The question on my mind is what can be done to at least minimize this from occurring the next time around? Not upgrade and stay with stable code that's tried and true. That can't be done for security and AP support reasons. So we are pushed to newer code for support reasons to stay current. I don't have the magic bullet for this problem's fix but hopefully I've spurred some conversation in to a problem that I think a few of us have been seeing.