Monitoring, Management & Location Tracking

Airwave GUI fails to load, HTTPD service crashes
Problem:

Airwave WebUI fails to load, HTTPD service keeps on crashing, when we do #wd in the Airwave CLI



Diagnostics:

When we try to do # wd from airwave CLI, we could see HTTPD daemon keeps on restarting.

When we look at /var/log/httpd/error_log, we can see the following:

[Mon Oct 10 19:05:10 2016] [error] fork: Cannot allocate memory at /usr/local/airwave/lib/perl/Mercury/Utility/Processes.pm line 95.\nCompilation failed in require at (eval 2) line 1.\n
[Mon Oct 10 19:05:10 2016] [error] Can't load Perl file: /etc/httpd/conf/bootstrap.pl for server localhost.localdomain:0, exiting...
[Mon Oct 10 19:05:29 2016] bootstrap.pl(13503): Starting...
[Mon Oct 10 19:05:29 2016] bootstrap.pl(13503): Not the last run yet (PPID: 13502)
[Mon Oct 10 19:05:56 2016] bootstrap.pl(13533): Starting...
[Mon Oct 10 19:06:21 2016] bootstrap.pl(13533): Last apache startup (PPID: 1)
[Mon Oct 10 19:06:21 2016] bootstrap.pl(13533): Preloading perl code...
[Mon Oct 10 19:07:24 2016] bootstrap.pl(13533): Killing any existing daemons...
[Mon Oct 10 19:07:24 2016] bootstrap.pl(13533): Initializing Cache::FastMmap cache files.
[Mon Oct 10 19:12:25 2016] bootstrap.pl(18423): Starting...
[Mon Oct 10 19:12:25 2016] bootstrap.pl(18423): Not the last run yet (PPID: 18422)
[Mon Oct 10 19:12:50 2016] bootstrap.pl(18444): Starting...
[Mon Oct 10 19:13:13 2016] bootstrap.pl(18444): Last apache startup (PPID: 1)
[Mon Oct 10 19:13:13 2016] bootstrap.pl(18444): Preloading perl code...
[Mon Oct 10 19:14:18 2016] bootstrap.pl(18444): Killing any existing daemons...
[Mon Oct 10 19:14:18 2016] bootstrap.pl(18444): Initializing Cache::FastMmap cache files.
[Mon Oct 10 19:17:43 2016] bootstrap.pl(18444): Checking AWRRD cache file.
Use of uninitialized value $arch in scalar chomp at /usr/local/airwave/lib/perl/Mercury/Utility/OS.pm line 134.
Use of uninitialized value in string eq at /usr/local/airwave/lib/perl/Mercury/Scripts/CheckAWRRDCache.pm line 40.
[Mon Oct 10 19:17:50 2016] [error] fork: Cannot allocate memory at /usr/local/airwave/lib/perl/Mercury/Utility/Processes.pm line 95.\nCompilation failed in require at (eval 2) line 1.\n

[Mon Oct 10 19:17:50 2016] [error] Can't load Perl file: /etc/httpd/conf/bootstrap.pl for server localhost.localdomain:0, exiting...
[Mon Oct 10 19:18:03 2016] bootstrap.pl(18940): Starting...

we could see the above error, the httpd/error logs, Cannot Allocate memory in the processess.pm file.

We could also do strace on the HTTPD daemon to understand the reason of the crash. we can see the PID of the httpd daemon from #wd output as below:

PID  PPID  PGID %CPU %MEM      RSS      ELAPSED CMD
18323     1 18323  2.5  5.8  3800964   2-23:37:45 Daemon::httpd_
18860     1 18860  0.1  5.7  3725756   2-23:36:51 Daemon::AsyncLoggerClient
18846     1 18846  0.1  5.7  3727060   2-23:36:56 Daemon::AsyncLoggerClient
18855     1 18855  0.1  5.7  3729872   2-23:36:53 Daemon::AsyncLoggerClient
18842     1 18842  0.1  5.7  3726916   2-23:36:57 Daemon::AsyncLoggerClient

When we do the strace, we can see the below output:

# strace -p 18323

open("/tmp/sharefile-22647-1476115281-4957", O_RDWR) = 7
mmap(NULL, 522715136, PROT_READ|PROT_WRITE, MAP_SHARED, 7, 0) = 0x7fa00aa47000
munmap(0x7fa00aa47000, 522715136)     = 0
mmap(NULL, 522715136, PROT_READ|PROT_WRITE, MAP_SHARED, 7, 0) = 0x7fa00aa47000
pipe([13, 14])                    = 0
pipe([15, 16]) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fa05b5caa70) = -1 ENOMEM (Cannot allocate memory)
close(13) = 0
close(14) = 0
close(15) = 0
close(16)= 0
write(2, "Use of uninitialized value $arch"..., 112) = 112
write(2, "Use of uninitialized value in st"..., 115) = 115
stat("/var/lib/airwave/cache/fastmmap/awrrdtool_write_cache", 0x7fa05d8d0790) = -1 ENOENT (No such file or directory)
close(7) = 0
munmap(0x7fa00aa47000, 522715136)= 0
lstat("/tmp/sharefile-22647-1476115281-4957", {st_mode=S_IFREG|0640, st_size=522715136, ...}) = 0
unlink("/tmp/sharefile-22647-1476115281-4957") = 0
sendto(6, "X\0\0\0\4", 5, MSG_NOSIGNAL, NULL, 0) = 5
close(6) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fa05b5caa70) = -1 ENOMEM (Cannot allocate memory)
write(2, "[Mon Oct 10 21:31:25 2016] [erro"..., 186) = 186
write(2, "[Mon Oct 10 21:31:25 2016] [erro"..., 133) = 133
exit_group(1)

We could see the same error, cannot Allocate memory

 

 

 

 

 

 



Solution

As per the analysis, this server needs more memory (RAM), for the HTTPD daemon to run.

Please make sure the server meets the minimum system requirements, we can find the Hardware server sizing guide at support.arubanetworks.com.

Version history
Revision #:
2 of 2
Last update:
‎03-17-2017 05:58 AM
Updated by:
 
Labels (1)
Contributors
Search Airheads
Showing results for 
Search instead for 
Did you mean: 
Is this a frequent problem?

Request an official Aruba knowledge base article to be written by our experts.