Airwave hard disk was 100% utilized and GUI stopped working. cleared old files/temporary files to get some space back. However, still unable to load the GUI.
Environment : Airwave versions 8.0 and above.
We could see the below errors in the service_watcher logs related to Redis, located at /var/log/service_watcher.
bootstrap.pl(15747): Falling through to CORE::exit (15747 - Daemon::ServiceWatcher): Could not connect to Redis server at 127.0.0.1:6 379: Connection refused at /opt/airwave/local/lib/perl5/site_perl/5.10.1/Redis.pm line 423
Redis::__build_sock('Redis=HASH(0x7ffa6bd7c7a8)') called at /opt/airwave/local/lib/perl5/site_perl/5.10.1/Redis.pm line 402
Redis::__connect('Redis=HASH(0x7ffa6bd7c7a8)') called at /opt/airwave/local/lib/perl5/site_perl/5.10.1/Redis.pm line 68
Redis::new('Redis', 'encoding', undef, 'server', '127.0.0.1:6379', 'password', '0npmtRNFI/3J0zGxuuU1cEEb') called at /usr/local/airwave/lib/perl/Mercury/Redis/Hash/Base.pm line 64
Mercury::Redis::Hash::Base::get_new_redis('Mercury::Redis::Hash::ClusterStatus=HASH(0x7ffa6bccb1f8)') called at /usr/local/airwave/lib/perl/Mercury/Redis/Hash/Base.pm line 55
Mercury::Redis::Hash::Base::connect_to_redis('Mercury::Redis::Hash::ClusterStatus=HASH(0x7ffa6bccb1f8)') called at /usr/local/airwave/lib/perl/Mercury/Redis/Hash/Base.pm line 49
Mercury::Redis::Hash::Base::redis('Mercury::Redis::Hash::ClusterStatus=HASH(0x7ffa6bccb1f8)') called at /usr/local/airwave/lib/perl/Mercury/Redis/Hash/Base.pm line 88Mercury::Redis::Hash::Base::put('Mercury::Redis::Hash::ClusterStatus=HAS H(0x7ffa6bccb1f8)', 'amc.service_watcher', 'HASH(0x7ffa6bd37808)') called at /usr/local/airwave/lib/perl/Mercury/Daemon/ServiceWatcher.pm line 246
This above error, will occur due to Redis not being able to start, If we look into redis.log, located at /var/log/redis/redis.log, we could see the below error:
 11 Nov 12:05:16.064 # Server started, Redis version 2.6.16
 11 Nov 12:05:16.064 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.con
f and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
 11 Nov 12:05:16.064 # Short read or OOM loading DB. Unrecoverable error, aborting now.
 11 Nov 12:19:55.761 # Unable to set the max number of files limit to 10032 (Operation not permitted), setting the max clients configuration to 3984
Your disk is full so when redis tries to create a db file it fails because there is no space left on the disk and it creates zero sized db file. Starting redis fails even after we getting some space back, because of zero sized db file, which got created in CentOS at the following file path "/var/lib/redis/dump.rdb". This is an issue with Redis versions currently airwave has. However, In newer versions of redis this bug is fixed, if you use older version of redis simply removing dump.rdb will work. But do this if dump.rdb file size is zero, otherwise do not because you might lose data.
To fix this issue, all we need to do is, move the dump.rdb from /var/lib/redis/dump.rdb to /tmp, by using the command below:
# mv /var/lib/redis/dump.rdb /tmp/
and execute the below command to restart the redis server.
# service redis restart
Note: we need to execute the above commands, only after confirmation that we have free space on the server.