Hello everyone,
I have a cluster of 35 IAP 215's. They use a virtual controller and one AP is set via the "Preferred Master" setting. They are currently on 8.5.0.6_74058 firmware. This problem has been happening for about a month now and I have run out of ideas.
This is the string of errors that happen leading up to the misbehaving AP reboot.
To add some context to the posted log. RM-26-Ceiling is NOT supposed to be the master its RM-01
Error Log:
2020-02-27 00:06:22,Local1.Error,,Feb 27 00:06:22 2020 AP:MS-AP-RM26-Ceiling < B4:5D:50:C0:7B:32> KERNEL(MS-AP-RM26-Ceiling @): [ 580.804391] (08:06:22) I am MASTER. recv-ed a master normal-beacon.
2020-02-27 00:06:22,Local1.Error,,"Feb 27 00:06:22 2020 AP:MS-AP-RM01 < B4:5D:50:C0:77:1C> KERNEL(MS-AP-RM01@): [1757609.516251] (08:06:22) OOPS. someone else thinks he is the master too, beacon version 4 from b4:5d:50:c0:7b:32 bond0"
2020-02-27 00:06:25,Local1.Error,,"Feb 27 00:06:25 2020 AP:MS-AP-RM01 < B4:5D:50:C0:77:1C> KERNEL(MS-AP-RM01@): [1757612.578859] (08:06:25) OOPS. someone else thinks he is the master too, beacon version 4 from b4:5d:50:c0:7b:32 bond0"
2020-02-27 00:06:25,Local1.Error,,Feb 27 00:06:25 2020 AP:MS-AP-RM01 < B4:5D:50:C0:77:1C> KERNEL(MS-AP-RM01@): [1757612.717430] (08:06:25) I am MASTER. recv-ed a master normal-beacon.
2020-02-27 00:06:25,Local1.Error,,"Feb 27 00:06:25 2020 AP:MS-AP-RM01 < B4:5D:50:C0:77:1C> KERNEL(MS-AP-RM01@): [1757612.794536] (08:06:25) master provision, 1 vs. 0"
2020-02-27 00:06:25,Local1.Error,,"Feb 27 00:06:25 2020 AP:MS-AP-RM01 < B4:5D:50:C0:77:1C> KERNEL(MS-AP-RM01@): [1757612.851850] (08:06:25) master provision, 1 vs. 0"
2020-02-27 00:06:25,Local1.Error,,"Feb 27 00:06:25 2020 AP:MS-AP-RM01 < B4:5D:50:C0:77:1C> KERNEL(MS-AP-RM01@): [1757612.909163] (08:06:25) !!! Election result, 1"
2020-02-27 00:06:26,Local1.Error,,"Feb 27 00:06:26 2020 AP:MS-AP-RM01 < B4:5D:50:C0:77:1C> KERNEL(MS-AP-RM01@): [1757613.516277] (08:06:26) OOPS. someone else thinks he is the master too, beacon version 4 from b4:5d:50:c0:7b:32 bond0"
2020-02-27 00:06:26,Local1.Error,,Feb 27 00:06:26 2020 AP:MS-AP-RM01 < B4:5D:50:C0:77:1C> KERNEL(MS-AP-RM01@): [1757613.654923] (08:06:26) I am MASTER. recv-ed a master normal-beacon.
2020-02-27 00:06:26,Local1.Error,,"Feb 27 00:06:26 2020 AP:MS-AP-RM01 < B4:5D:50:C0:77:1C> KERNEL(MS-AP-RM01@): [1757613.732029] (08:06:26) master provision, 1 vs. 0"
2020-02-27 00:06:26,Local1.Error,,"Feb 27 00:06:26 2020 AP:MS-AP-RM01 < B4:5D:50:C0:77:1C> KERNEL(MS-AP-RM01@): [1757613.789342] (08:06:26) master provision, 1 vs. 0"
2020-02-27 00:06:26,Local1.Error,,"Feb 27 00:06:26 2020 AP:MS-AP-RM01 < B4:5D:50:C0:77:1C> KERNEL(MS-AP-RM01@): [1757613.846655] (08:06:26) !!! Election result, 1"
2020-02-27 00:06:28,Local1.Error,,"Feb 27 00:06:28 2020 AP:MS-AP-RM26-Ceiling < B4:5D:50:C0:7B:32> KERNEL(MS-AP-RM26-Ceiling @): [ 586.726256] (08:06:27) OOPS. someone else thinks he is the master too, beacon version 4 from b4:5d:50:c0:77:1c bond0"
2020-02-27 00:06:28,Local1.Error,,Feb 27 00:06:28 2020 AP:MS-AP-RM26-Ceiling < B4:5D:50:C0:7B:32> KERNEL(MS-AP-RM26-Ceiling @): [ 586.863781] (08:06:28) I am MASTER. recv-ed a master normal-beacon.
2020-02-27 00:06:28,Local1.Error,,"Feb 27 00:06:28 2020 AP:MS-AP-RM26-Ceiling < B4:5D:50:C0:7B:32> KERNEL(MS-AP-RM26-Ceiling @): [ 586.938803] (08:06:28) master provision, 0 vs. 1"
2020-02-27 00:06:28,Local1.Error,,"Feb 27 00:06:28 2020 AP:MS-AP-RM26-Ceiling < B4:5D:50:C0:7B:32> KERNEL(MS-AP-RM26-Ceiling @): [ 586.994040] (08:06:28) !!! Election result, -5"
2020-02-27 00:06:29,Local1.Error,,"Feb 27 00:06:29 2020 AP:MS-AP-RM26-Ceiling < B4:5D:50:C0:7B:32> KERNEL(MS-AP-RM26-Ceiling @): [ 587.666811] (08:06:28) OOPS. someone else thinks he is the master too, beacon version 4 from b4:5d:50:c0:77:1c bond0"
2020-02-27 00:06:29,Local1.Error,,Feb 27 00:06:29 2020 AP:MS-AP-RM26-Ceiling < B4:5D:50:C0:7B:32> KERNEL(MS-AP-RM26-Ceiling @): [ 587.804402] (08:06:29) I am MASTER. recv-ed a master normal-beacon.
2020-02-27 00:06:29,Local1.Error,,"Feb 27 00:06:29 2020 AP:MS-AP-RM26-Ceiling < B4:5D:50:C0:7B:32> KERNEL(MS-AP-RM26-Ceiling @): [ 587.879428] (08:06:29) master provision, 0 vs. 1"
2020-02-27 00:06:29,Local1.Error,,"Feb 27 00:06:29 2020 AP:MS-AP-RM26-Ceiling < B4:5D:50:C0:7B:32> KERNEL(MS-AP-RM26-Ceiling @): [ 587.934660] (08:06:29) !!! Election result, -5"
The show ver command always shows this entry:
Reboot Time and Cause: AP rebooted Thu Feb 27 00:30:41 UTC 2020; System cmd at uptime 0D 0H 4M 56S: Preempted by provisioned master (b4:5d:50:c0:77:1c 10.23.1.81) uptime from boot: 4 minutes 55 seconds; uptime from being master: 40 seconds
I have tried downgrading to the 6.5 firmware train to see if it was a problem with the firmware. That was not the case.
I have factory reset the AP and added back into the cluster along with outright replacing it. I tested all physical variables, wiring, placement, switch port. The APs are connected to Cisco Catalyst Switches. There has been no power interruptions from the PoE side of things either.
I hope this is enough information to work with. Thanks in advance.