Wireless Access

After deleting a blacklist entry for a client (in an environment with ~1600 WAPs), I see the following error in my controller logs for about 4 of the WAPs...

stm[1409]: <501116> <WARN> |AP TESTWAP-1@192.168.1.50 stm| Blacklist del: 11:22:33:aa:bb:cc: timeout

stm[1409]: <501116> <WARN> |AP TESTWAP-2@192.168.1.69 stm| Blacklist del: 11:22:33:aa:bb:cc: timeout

stm[1409]: <501116> <WARN> |AP TESTWAP-3@192.168.1.47 stm| Blacklist del: 11:22:33:aa:bb:cc: timeout

stm[1409]: <501116> <WARN> |AP TESTWAP-4@192.168.1.33 stm| Blacklist del: 11:22:33:aa:bb:cc: timeout

What exactly does this timeout mean? I believe the blacklist database is maintained on the controllers, correct? I dont think a local blacklist exists on each AP? What has timed out?

This client will become blacklisted, I will then clear the blacklist, but if Im tracking my logging correctly, the client is getting blacklisted by one of these 4 WAPs (if they associate to any other WAP, they dont get blacklisted).

Trying to figure out the exact worklfow under the hood for the blacklisting.

- What version of ArubaOS are you using?

- What method did you use to remove the blacklist?

- AOS 8.7.0.0

- stm remove-blacklist-client xx:xx:xx:xx:xx:xx

when you type "show ap blacklist-clients" does the client still show up there?

Yes, client was in the blacklist before clearing (as verified by running "show ap blacklist-clients | include <client_mac>")

Recap of events...

1. client is blacklisted due to arp spoof

2. verified client is in blacklist (with above command)

2a. had client disconnect from ssid

3. clear blacklist entry (stm remove-blacklist-client <client_mac>)

4. watch controller logs and see the "del timeout" errors for only a few specific APs (but they are APs nearby the user, ones they would likely associate to)

5. have user reconnect

6. client is not back in the blacklist (verified via "show ap blacklist-clients") but connected

I think the log message is misleading.  That log message is usually shown when a client's blacklist timer expires and the client is removed from the blacklist.  From what you described, it worked the way it should, but it showed that message when you removed it and NOT when it expired.

Do I have that correct?

I think what you're saying makes sense. (and in a way, yes, the logging is misleading then.) Our expiry time is set at 0 (permanent), so its not from ageing out / timing out. 

Guess I'm still trying to figure out what these log entries really mean and why the log entries are tied to APs, if when manually removing a blacklist, its only removing it from the blacklist db on the controllers. (it shouldnt care about APs, right?) 

The controller does keep a list of blacklisted clients, but that list is also pushed to APs so that clients can be stopped before association.  Initial "association" is handled by the APs, is why they also need a copy of this list.

I think the log is misleading.

Figured the APs must have a local copy, since as you mentioned they handle the initial association.

Have you ever seen/heard of issues where after clearing a blacklist entry on a controller, it fails to push out to APs/update the APs local list? (failure could be due to poor performance on the controller, or maybe network issues/congestion causing that update not reach all APs in the system, etc)?

Is there any advance debugging I could enable on an AP to see blacklist messages between controllers and the AP? For example, something like this? (or a different "process" setting?)

---

Have you ever seen/heard of issues where after clearing a blacklist entry on a controller, it fails to push out to APs/update the APs local list? (failure could be due to poor performance on the controller, or maybe network issues/congestion causing that update not reach all APs in the system, etc)?

I have not.

Is there any advance debugging I could enable on an AP to see blacklist messages between controllers and the AP? For example, something like this? (or a different "process" setting?)

 

---

You can see the blacklist on each AP by running "show ap remote blacklist-clients ap-name <name of ap>".  But, all APs are not immediately sent a copy of the blacklist every time something is added or removed to the blacklist, because that would be a huge issue in a large network.  Some APs receive it immediately, others request it the next time there is an association.  I don't think you have anything to worry about, though. 

Thanks for that command, I have been looking for something like this. Thanks for all of your input!

I'm digging into this as I am still fighting weird issues regarding the iOS14 ARP spoof issue. Just gathering more tools do debug and dive in, so appreciate your help!

I think I found part of my problem. I noticed with this client I would delete him from one MD, but it was still in the blacklist on the other MD. As far as I am aware, the lists should replicate between my two clustered MDs, however I think some sort of error caused the blacklist entry to be removed from one (where I ran the command), but failed to remove from the other.

So the client happened to be fine temporarily because he was terminating on the MD that didnt have the entry. But if they roamed to an AP and started to terminate on that other MD, he was getting blacklisted due to the entry being there.

Like I said, I believe running the "stm remove-blacklist-client" command on a single MD in a cluster should propagate to the other MD. But in this case, something caused that to fail.

Try blacklisting and removing from the MM GUI if possible.

I had thought of that, but dont have the command on the MM

What version of ArubaOS are you running?  You should be able to click on the garbage can shield next to a client on the MM to blacklist it..

To blacklist from the MM, it appears to be a hidden command:

blmgr add-blacklist-client 20:4c:03:4a:41:23

To remove:

blmgr remove-blacklist-client 20:4c:03:4a:41:23

Ahhhhh bingo!!   I'll give this a try!  Thanks!!!   Super MVP over here!

Is there a MM hidden equivalent for "show ap blacklist-clients?" Or can this only be run on the MDs?

There is not.  You can certainly see and manag the blacklisted clients on the MM using Dashboard> Security> Blacklist Clients.

We're running 8.7.0.0.

I will have to check our GUI for that ability. Wish it was available via CLI.