Controllerless Networks

After firmware update (from 6.3->8.3->8.6 and downgrade to 8.5..) 13pcs of iap315 in one VC system started to act weird and now random amount of ap's "un-joins" from VC every now and then. "un-joined" ap's seems to stay functional (connectivity via ssh and port 4343 works) and they seems to be aware about which ap is master in vc but connectivity to network and vc connectivity does not work properly (drops wlan connectivity off from clients every now and then)

Tech-Support dump has two lines about non-working ap's from exact same time when ap drops off from virtual controller... Any idea what's wrong?

2020-01-19 23:14:01 cli_rap_gre_ep_request(6440) action: del, 10.10.0.110, controller all
2020-01-20 06:23:20 cli_rap_gre_ep_request(6440) action: del, 10.10.0.111, controller all

What version of 8.5 are you running on the 315s? Have you confirmed that all 315s are running the same image?

Version in all ap's are 8.5.0.5 and "current missing" ap is running 8.5.0.5 according "show version".

Next error seems to exists in system log just before first messages deletion log entries.. EDIT: But seem like there is couple AP Missed Poll messages also from access points which does NOT drop off from vc.. (This is really weird?!?)

Jan 20 19:49:28  cli[5990]: <341004> <WARN> |AP ap-102-ab:cc@10.10.0.102 cli|  AP Missed Poll - ap-106-ab:94 Radio 0 Cur Time 4-94167 Prev Time 3-94076

...bit more info: ap which were disassociated yesterday evening has been now associated back to vc and other ap has been disassociated form vc.

Wired network has not been changed since last summer and all of those 13 pcs ap's are connected to two switches and there has been problems with ap's connected to both switches so I assume there is no problems in switch network.

EDIT: I added remote syslog yesterday to vc and today between 16:01 and 16:24 I got HUGE "throwup" of log messages (total of 348Mb) from two ap's where most of the log entries are like this:

2020-01-21 16:01:10 - 2020(err) AP:ap-107-aa:b6 <10.10.0.107 34:FC:B9:C8:AA:B6> cli[5990]: PAPI_Send: sendto AP CLI failed: (null) Message Code 0 Sequence Num is 28799
2020-01-21 16:01:10 - 2020(err) AP:ap-107-aa:b6 <10.10.0.107 34:FC:B9:C8:AA:B6> cli[5990]: sendAck: sendto 6c000a0a:24635 failed: Resource temporarily unavailable

Sounds like there could be a network event or problem with the AP accessing the network that is intermittently preventing the cluster communications to reach all APs.

How are these APs interconnected with each other? Do you have a network diagram you can share?

Forgot to answer this question but seems like problem was firmware itself. When I downgrade iap's down to version 6.5.4.16 problems were gone.

There have to be something else involved here in this problem because I have couple of another sites which also has iap315's and there is no problems at all.