We have a device that began having intermittent connectivity. We noticed that when it was broken, it had an all-zeros MAC in its ARP table for the gateway. After further investigation, we saw it receiving ARP replys for the gateway address from an all-zero MAC source saying that the gateway was at 00:00:00:00:00:00. After extensive troubleshooting and packet captures separately on ingress and egress at each interface on the local network, we determined that the all-zero sourced replies were coming from our wireless controller's interface that is in the same LAN as the device having issues.
Short and sweet:
1. A device is sending ARP requests to 00:00:00:00:00:00 for its gateway.
2. Our active Aruba 3400 on the same VLAN is replying to those ARP requests from a source MAC of 00:00:00:00:00:00 with an ARP reply that says the gateway is at 00:00:00:00:00:00.
3. In addition to this being invalid, the gateway IP is NOT even on the Aruba box.
4. This behavior was sudden. No changes were made on any of these devices leading up to the behavior starting.
We will put in a ticket, but has anyone else seen this before? It's pretty baffling.