Controllerless Networks

We have 1 conductor and 5 controllers. No cluster. Been running fine for years. Monday only one online in the conductor. After several days of trouble shooting we found each controller says it owns all 5 ip addresses.
The conductor arp shows the same Mac for all 5 controller IPs. It happens to be the first controller in the network path to receive an ARP request from the controller.
Software version 8.11.2.0 Ssr
If I disable the 0/0/0 interface on that controller, the next one in the network path responds to the conductor arp request saying it owns all 5 ip address’s.
A packet capture on the 0/0/0 interface of any of the controllers while clearing the arp table does not show any aro request for any of the 5 controllers ip address’s yet the controllers arp table is immediately populated with the 5 ip address’s and that controllers MAC address as the owner. But plenty of ARP request for other devices on this subnet.
Conductor and all 5 controllers are on same subnet.
No layer 3 on the vlan, all switches show the correct mac and link direction for each controller.
At the moment each controller believes it owns all the ip addresses for all the controllers according to their aro tables.
The only thing I see in the controllers configs is a IP NAT POOL created at the top level that lists all 5 ip addresses pointing to a ip address in our clearpass data interface. This means all five controllers would have a ip nat address for each of the 5 ip address.
This ip nat pool has been in place since January.
This ip nat pool is the only area in the controllers config where is could get the 5 ips fast enough to populate its sep table without doing an sep request out the interface.

Thoughts please???? Please be kind, it’s been a rough couple of days.
And yes, I have a ticket with TAC. Been on a remote session with them for 8+ hours over the last few days.

Issue solved.
IP NAT POOL created at the public wireless level
Was created in January when clearpass was installed, had a ip nat for each controller ip address and so was pushed to all controllers. No issues with it until Monday and all controllers thought they owned all five up address so they all added their MAC addresses to all five in their respective arp tables.
For some reason the conductor would not allow us to delete the stating they were in use, also were created at the root level somehow, so I edited them and added bogus iOS to each one. Then added a single one to each controller for its respective ip address and once applied all controllers came back online.
Very strange indeed.

---------------------------------
james
---------------------------------



Original Message:
Sent: Mar 07, 2024
From: 2185908404
Subject: 5 Controllers And All Five Say the own All 5 Mgmt Ip Address

We have 1 conductor and 5 controllers. No cluster. Been running fine for years. Monday only one online in the conductor. After several days of trouble shooting we found each controller says it owns all 5 ip addresses.
The conductor arp shows the same Mac for all 5 controller IPs. It happens to be the first controller in the network path to receive an ARP request from the controller.
Software version 8.11.2.0 Ssr
If I disable the 0/0/0 interface on that controller, the next one in the network path responds to the conductor arp request saying it owns all 5 ip address’s.
A packet capture on the 0/0/0 interface of any of the controllers while clearing the arp table does not show any aro request for any of the 5 controllers ip address’s yet the controllers arp table is immediately populated with the 5 ip address’s and that controllers MAC address as the owner. But plenty of ARP request for other devices on this subnet.
Conductor and all 5 controllers are on same subnet.
No layer 3 on the vlan, all switches show the correct mac and link direction for each controller.
At the moment each controller believes it owns all the ip addresses for all the controllers according to their aro tables.
The only thing I see in the controllers configs is a IP NAT POOL created at the top level that lists all 5 ip addresses pointing to a ip address in our clearpass data interface. This means all five controllers would have a ip nat address for each of the 5 ip address.
This ip nat pool has been in place since January.
This ip nat pool is the only area in the controllers config where is could get the 5 ips fast enough to populate its sep table without doing an sep request out the interface.

Thoughts please???? Please be kind, it’s been a rough couple of days.
And yes, I have a ticket with TAC. Been on a remote session with them for 8+ hours over the last few days.