Solution
When the number of rogues on the system is plenty then the size of rogues_cache [/var/lib/airwave/cache/fastmmap/] file is expected to be large as show below.
Run the following command to confirm the issue:
# ls -lha /var/lib/airwave/cache/fastmmap/rogues_cache
-rw-r----- 1 apache apache 5G May 6 05:26 /var/lib/airwave/cache/fastmmap/rogues_cache
The problem will re-appear any time httpd is restarted until the number of rogues has been reduced. Please make sure that we have some number other 0 for rogue’s data retention under rapids setup page to avoid this trouble in the future.
Deleting the fastmmap.size file (/var/airwave/cache/fastmmap.sizes) while the AMP is disabled should allow httpd and all the other services to complete the normal startup process when the AMP is re-enabled.
Run the following command from Airwave command line resolve the issue:
# amp_disable
# rm -rf /var/airwave/cache/fastmmap.sizes
# amp_enable
As we remove the /var/airwave/cache/fastmmap.sizes and enable amp services, the server will be restored successfully without any crash as show below.
Verify the /var/log/httpd/error_log to confirm httpd hand-off the control to Service watcher:
bootstrap.pl(11661): Starting...
bootstrap.pl(11661): Last apache startup (PPID: 1)
bootstrap.pl(11661): Preloading perl code...
bootstrap.pl(11661): Killing any existing daemons...
bootstrap.pl(11661): Initializing Cache::FastMmap cache files.
bootstrap.pl(11661): Checking AWRRD cache file.
bootstrap.pl(11661): Forked off ServiceWatcher starter pid 11684
bootstrap.pl(11684): Forked off ServiceWatcher pid 11691
[notice] Apache/2.2.15 (Unix) mod_apreq2-20051231/2.6.1 mod_perl/2.0.4 Perl/v5.10.1 configured -- resuming normal operations
bootstrap.pl(11691): ServiceWatcher 11691 running