Wireless Access

Hi Guys,

I have another interesting one for you.

We have a MIS package that is used by 1200 users, both wired and wireless.

Every day (in the AM) all our wireless users suddenly get kicked out of the package, with a network error.

From a wireless client, I can no longer ping the application server, nor can I access any of the shares from this particular server.

However I can ping all other servers, and access all other shares.

All the wired clients continue as normal and are not affected by this issue at all.

The only way to get things working again for the wireless clients is to reboot both controllers.

I've not been able to do much in the way of diagnostics, as the second this system goes offline, I get flooded with phone calls, so to minimise downtime I‘ve just had to reboot the controllers.

What I have been able to ascertain is that it is effecting both controllers, but I haven’t tried pinging the effected server from the diagnostics page yet, I’ll be doing that the next time we see this issue.

I can only assume that the server is being flagged as a bad client or something due to the amount of connection requests first thing in the morning, but I don’t know where to look to prove / disprove this theory - or better still, how to stop it :(

- Has this ever worked?

- When did it start happening?

- We do not have enough information to determine what is going wrong

- You can save some time by opening a TAC case in parallel, so that they can gather all of your non-public information and start to work on it.

Hi CJ,

Yes this worked issue free for a good few months, although this is relatively a new installation (8-9 months)

It seemed to start happening when our support contractor updated the OS on the controllers to solve a voice vlan issue.

As an attempt to resolve the prevalent server dropping issue, they stopped all the traffic from going through the controllers which indeed stopped the server from dropping off, but introduced other issues with the captive portal.

I got them to switch it back, and everything was ok for a few days, then every day last week and so far this week we have had the issue.

The support contractor is adamant that is it just the SQL traffic that is being "interrupted" however given that we can’t access shares or ping the server when it drops off, I’d say it was the host itself that being blocked for whatever reason.

I've got logs from the controllers from before and after a reboot for comparison, but no idea which file to look at. There are umpteen.

Really daft question, how do I open a TAC case?

7Cups,

Send an email to support@arubanetworks.com to determine what your options are.  You need a current support contract to open a TAC case and they will let you know your status.

Your problem does not seem straighforward, and if a contractor has been doing work on your system it is going to be very difficult to determine what is wrong through this forum.  Please send an email to support to see what your options are.

We can try to help here, but we would not want to make things worse if we do not know the consequences of our advice.  We can certainly give you general information, however.

Let's start from scratch:

- What version of ArubaOS is this?

- Are your clients using encryption?

- Is the server on the same VLAN as the clients?

- When you have the disconnect issue, can you reach anything at all?

Hi CJ,

Thanks for the speedy reply - your quickly becoming my one to one support agent - ha!

I'll blast an email over to them now and see where we stand.

In answer to your questions:

- We are using ArubaOS: 3.1.6.8

- Clients are not encrypted

- The server is on the same VLAN as the clients

- When being denied access to the application server, everything else remains connected and accessible. Domain access, mapped drives internet, etc, etc. It is limited to the one application server.

Hi,

I just want to throw out one theory here, beacause I saw something like this before.

CJ is right though, this could be massively complex to troubleshoot.

What I have seen in the past, is certain Microsoft servers using multicast addresses for client comms, after the initial session started as unicast. Depending on your Aruba config, this might result in all sorts of unusual behavior (no traffic, some traffic etc).

If it was this, it would explain why the client connectivity looked ok from all other perspectives. And why wired clients were ok too.

Although time consuming, if I suspected it was something like this, I'd first need to establish if the server was using mcast. Sniffing the server port (although a bit of a faff) for interesting traffic is usually the first way to find out.

Hi Jake!

You could be onto something there!

I vaguely remember us opting to block multicast traffic on the controllers.

Fireing up wireshark as we speak...

---

@7cups wrote:

Hi Jake!

You could be onto something there!

I vaguely remember us opting to block multicast traffic on the controllers.

 

Fireing up wireshark as we speak...

 

---

7cups,

Good move.

The only thing that I am worried about is that you cannot even ping the server, which is NOT multicast, so it should not be dropped.  I would check to make sure that the switch port the server is connected to has negotiated to the correct speed and duplex and that there are no errors on that interface (layer 1).  Next, I would make sure that the server is populated with ARP entries when there is a problem (arp -a).  I would then go to the controller and make sure there is an ARP entry for the server (show arp | include <ip address of server).

There is no way to blacklist a wired client dynamically on the Aruba controller.  It would have to be blocked in the user role, really.  Even then it would be permanent, not temporary.

Ok so its just dropped again now, while sniffing.

I can ping the server that is "down" from both controllers, so the conrollers can still see the server.

Controller ARP

(TSA00974) #show arp | include 192.168.0.7
Internet        192.168.0.7     56:B5:97:DB:5B:FE       vlan1

Server ARP

C:\Users\config\Desktop>arp -a

Interface: 192.168.0.7 --- 0xb
  Internet Address      Physical Address      Type
  ... // Removed irrelevant
  192.168.0.2           7e-80-e7-5e-d9-9a     dynamic
  192.168.0.3           ce-b5-af-ea-32-89     dynamic
  ... // Removed irrelevant
  192.168.80.1          00-0b-86-6e-2a-f4     dynamic
  192.168.80.2          00-0b-86-6e-2a-a0     dynamic
  ... // Removed irrelevant
  192.168.255.255       ff-ff-ff-ff-ff-ff     static
  224.0.0.252           01-00-5e-00-00-fc     static
  239.255.255.250       01-00-5e-7f-ff-fa     static
  255.255.255.255       ff-ff-ff-ff-ff-ff     static

 Core Switch Port Status

===========================- TELNET - MANAGER MODE -============================
                       Status and Counters - Port Status

                     Intrusion                               MDI   Flow  Bcast
   Port     Type       Alert    Enabled  Status     Mode     Mode  Ctrl  Limit
  ------  ---------  ---------  -------  ------  ----------  ----  ----  -----
  C1      100/1000T  No         Yes      Up      1000FDx     MDIX  off   0
  C24     100/1000T  No         Yes      Up      1000FDx     MDIX  off   0

Hope this is of use. 

You said this is a flat network yes? No routing?

Whether the controller has an ARP entry for the client is only relevant if the "convert arp" is configured in the VAP.

If you can post a copy of your config, that might help? Take out anything sensitive.

Ok. So .7 is the server?

What I would try is as follows (when you get the issue again, sorry!).

Check the client ARP entry on the server, and the server entry on the client. Do they both exist? And look right?

Very first thing to do next (whilst the issue exists), is take off "Drop Broadcast and Multicast" and "Convert Broadcast ARP requests to unicast" from the VAP.

In the short term (depending how big your flat network is), it will likely run slower, but it might restore the session. And if it does, we will know what the root cause is.

Then, we can work out what to do about it (will need more info).

Well this particular part of the network could be considered flat yes.

The controllers, DHCP clients (120) and this particular server are all on the same network currently.

While the number of devices aren't all that many, the broadcast domain is pretty big which I'm looking to segment once our users go off over the Christmas break.

.7 is indeed the server in question

I've attached the config as per your request.

Just before a change the mcast / ucast settings on the controllers - will there be any downtime from the change of settings or will it be instant?

Interestingly enough, today is the first day that we haven't had the MIS server disappear to the wireless clients. It usually happens around 08:30-45 am.

Best regards,

Attachment(s)

tsaAruba.txt   20 KB 1 version

Right, the watched washing machine never breaks when the engineer visits...

These changes should not impact service. You might notice a minor blip, but I would not expect anything significant. Of course you'll understand I can't promise that, but if I was doing it, I'd just go ahead.

You've got "firewall broadcast-filter arp" enabled globally in the stateful firewall config (I normally do this per VAP rather than global, doing both is redundant). I don't think it shows up in the individual VAP configs as it's on by default in your version (pretty sure of that).

So, you'll need to go into the firewall global config and turn that off, then make the changes to each VAP I described.

Hi guys,

Id like to appologise for leaving this thred hanging for so long, id also like to thank everyone for their help and suggestions.

After examining the ARP logs over and over again, I finally noticed what the issue was.

The logs shown that the MAC address of the host configured for that particualr IP address changed  - so essentially a wireless device was configured with the same IP Address as the server that kept dropping for wireless clients.

So every time this device joined the wireless network the controlers directed all the traffic for the server to the wireless device instead.

:)