Airwave server accessibility is slow, full swap memory being utilized, although the server meets the recommended hardware specs.
If we see the below errors in async_logger_client.log in /var/log/async_logger_client.log
_do_updates: Can't go back in time (yet)! 1359139598 < 1769432174
_do_updates: Can't go back in time (yet)! 1359139939 < 1769432174
_do_updates: Can't go back in time (yet)! 1359140281 < 1769432174
_do_updates: Can't go back in time (yet)! 1359140694 < 1769432174
_do_updates: Can't go back in time (yet)! 1359141024 < 1769432174
_do_updates: Can't go back in time (yet)! 1359141397 < 1769432174
_do_updates: Can't go back in time (yet)! 1359141711 < 1769432174
_do_updates: Can't go back in time (yet)! 1359142062 < 1769432174
_do_updates: Can't go back in time (yet)! 1359142375 < 1769432174
_do_updates: Can't go back in time (yet)! 1359142690 < 1769432174
_do_updates: Can't go back in time (yet)! 1359143046 < 1769432174
This can mean Airwave is trying to update the time on the same devices over and over, coz of the duplicate devices. We need to find out if there are any duplicate devices on Airwave and either delete them or ignore them.
We could do the following from the airwave CLI to find the duplicate devices:
# dbc"SELECT id, name, apparent_ip FROM ap WHERE apparent_ip IN (SELECT apparent_ip FROM ap GROUP BY apparent_ip HAVING count(*)>1) ORDER BY apparent_ip;"
id | name | apparent_ip
-------+-------------------------+----------------
34414 | IDC12-NAE-LAB-AP17 | 0.0.0.0
34413 | APa493.4cc1.12ae | 0.0.0.0
34756 | AP-4floor-temp1 | 10.125.171.45
33687 | kh14b-5f7ap27 | 10.125.171.45
34853 | Akhil-IAP | 10.17.168.230
34854 | 00:24:6c:c8:68:98 | 10.17.168.230
34541 | AP7cad.74ff.31be | 10.185.194.7
34806 | AP1c6a.7ae2.bb50 | 10.185.194.7
34351 | AP4c4e.3560.424e | 10.185.32.159
32942 | Idc73a-wap06s | 10.185.32.159
The above command from the DB shows us if we have any devices with the same IP address. We could either delete those devices or if they come up again leave them in new devices we could move them to ignore devices list. Once we do that the server performance will drastically change.
Note: This KB doesn’t fix all the performance issues, this for the specific crash we see in the async_logger_client, mentioned above:
Please find the below db command which will help you find duplicate devices with the same mac address:
# dbc "SELECT id, name, lan_mac FROM ap WHERE lan_mac IN (SELECT lan_mac FROM ap GROUP BY lan_mac HAVING count(*)>1) ORDER BY lan_mac; "
Why do we see the duplicate devices in the first place?
That is because of the known defects in pre 7.7.x versions and or we upgrade directly to 8.0.x from 7.6.x, there are two work arounds, we could use:
1. Delete those duplicate devices [ Most likely will come up again in New devices, we need to leave them in new devices, and move to ignore if your airwave is on pre 7.7.x versions].
Deleting the devices helps in versions 7.7.x and above, as the duplicate device will not show up in these versions.
2. Ignore those devices [ best option], airwave will not talk to them.
if your airwave is on Pre 7.7.x versions, moving those devices to the ignored devices is the best option.
Note: When considering the device to either delete or move to ignore state, move the duplicate device which has no data. DO NOT move the one's that has data for example: devices showing ap count or client counts, should not be moved.