Controllerless Networks

Directly after completing a recent update to AP firmware 8.6.0.5-8.6.0.5_75979 - we've started experiencing an issue where after an indeterminate period, arp requests for the gateway of our entire network will return the MAC of one of the APs.

Disabling the AP in question will cause arp requests for the gateway to return to the MAC of our core switch (after a minute or so of major network connectivity issues across the site). However every few hours a different AP will somehow become the new gateway and all traffic starts flowing through that AP.

We've spent about 12 hours on TAC phone calls over the weekend, and we still aren't any closer to resolving this. At one point it appeared that removing the VC ip address corrected the issue, and we had about 24 hours of normal behavior, but this morning the issue has started occurring again. With a further TAC call resulting in "get some wireshark logs" 

Currently we're considering either, trying a downgrade of the firmware to an earlier version (or all the way back to 8.3 where we updated from)

Or just turning off all of our APs on site as this issues is impacting production in unpredictable ways.

What is the make/model of the core switch?

Also, are you doing any DHCP from the VC?

Thanks for the reply.

Core switches are HPE 8212zl

No we're not doing DHCP via the VC.

Last night we tried a downgrade of the AP firmware from 8.6 to 8.4, the issue appeared to go away for about 13 hours, but it's just now started re-occuring

Hi wrightmt, I highly suggest you do keep working with TAC on this one. I know these longer running ones are a pain. What you're experiencing is pretty odd though so having multiple eyes on it will be helpful.

I've seen something like this occur on a different switch (non-Aruba) where it made a change to the ARP table after an ARP probe. In that case the switch wasn't doing the right thing and required a software fix.

I will see what I can do about a capture in my lab to see if I see any ARP probes coming from the VC. It's probably not related, but I'm curious.

So to be clear... You're seeing the ARP table update with the MAC address of the VC (on ever the VC?) on the core switch IP address? Or is the gateway of the network another device?

It would be very useful to determine what device is actually responding to that ARP request. A capture when the problem occurs from the client will be very useful.

What we're seeing is that the ARP table of the core and distribution switches will display our gateway address, with the MAC of a seemingly random AP. Initially we thought it was just the VC, but now it appears that any AP will become the gateway.

10.190.78.1 is our gateway and 00005e-000101 is the core

Heres what it should look like when its working normally:

 IP ARP table

  IP Address       MAC Address       Type    Port
  ---------------  ----------------- ------- ----
  10.190.66.25     005056-9c1847     dynamic F1 
  10.190.66.86     98be94-29e1bb     dynamic F1 
  10.190.66.93     005056-9c1791     dynamic F1 
  10.190.66.140    98fa9b-825ab3     dynamic D20
  10.190.78.1      00005e-000101     dynamic F1 

Here's what we get when it fails:

  IP Address       MAC Address       Type    Port
  ---------------  ----------------- ------- ----
  10.190.66.25     005056-9c1847     dynamic F1 
  10.190.66.86     98be94-29e1bb     dynamic F1 
  10.190.66.93     005056-9c1791     dynamic F1 
  10.190.66.140    98fa9b-825ab3     dynamic D20
  10.190.78.1      20a6cd-ca2330     dynamic F1 

or

  IP Address       MAC Address       Type    Port
  ---------------  ----------------- ------- ----
  10.190.66.25     005056-9c1847     dynamic F1 
  10.190.66.86     98be94-29e1bb     dynamic F1 
  10.190.66.93     005056-9c1791     dynamic F1 
  10.190.66.140    98fa9b-825ab3     dynamic D20
  10.190.78.1      c8b5ad-c33a30      dynamic F1

So if we then track down which AP that is - and disable it, we start geting the correct response when pinging the gateway, the arp table will update with the core MAC

Thanks for that detail. That's super helpful to better understand the symptoms. 

So it is plausible here that it's the 8212 doing the wrong thing rather than the APs. Have you checked the software version for that switch is the latest? Has it been updated recently?

So I took a quick look at the Access Security Guide for the 8200 series switch. Have a look at page 444 at the arp validation checks. It might be that setting something like this gets rid of the issue:

arp-protect validate ip

However!!! Having one of the offending ARP frames (if that's what is causing this) would really help to identify what's happening. There are a couple of other arp-protect validate options: dest-mac and src-mac.

Thank you! That looks very promising, yes I've got wireshark setup and waiting for the next failure, and I'll try to capture some arp frames. I tried that yesterday when the issue was occuring but I only got arp requests and no replies.

It can be fun waiting for the intermittent event. In a previous case I was able to provoke the issue by sending (forging) an ARP packet that was crafted to be similar enough to the one that caused a problem.

Outbound ARP Probe, where the sender IP is 0.0.0.0 was enough to do it. This may not be the same in your case, but it's easy enough to test while you're waiting for the live environment to do it again.

Check out https://scapy.net/

By using something like this as your packet you would easily be able to watch for the packet in Wireshark with a filter for 'eth.addr == 00:11:22:33:44:55'. If that changes the ARP table in your switch then the hunt is over. I will be surprised if it's exactly the same issue though.

em=Ether(src="00:11:22:33:44:55", dst="ff:ff:ff:ff:ff:ff")/ARP(psrc="0.0.0.0", pdst="10.190.78.1", hwsrc="00:11:22:33:44:55")

Thanks again, I've never seen scapy before. Much appreciated!

Maybe a stupid suggestion, but could it be that your DHCP server is not configured to exclude the IP of your default gateway from the scope?

Also, do you have enough free IP addresses in your DHCP scope?

The reason for asking is that in the table of your switch, I see the MAC address only once.

Thanks for your reply - yes that occured to us too during our initial troubleshooting, but it's definitely outside of the scope.

I may be able to close this question shortly, one of the Aruba engineers pointed out an issue with our IAP Uplink configuration that seems to have kicked in after the firmware update. We've made a change and waiting to see if that has resolved the issue, so far so good!

Solution: removing IAP Uplink configuration