Monitoring, Management & Location Tracking

Issue: After upgrade Apache (httpd -- the web server) keeps restarting; other processes (daemons) not coming up

After upgrading to 7.1 some customers have encountered a problem where the web server (Apache / httpd) does not come up all the way. We've seen this particularly on systems with limited hardware resources.

You will see messages similar to the following in /var/log/low_level_service_watcher:

Mon Sep 20 10:22:50 2010: Started
Mon Sep 20 10:24:22 2010: Apache alive (for 304 seconds), but no ServiceWatcher; starting 35s grace period...
Mon Sep 12 10:25:25 2010: Apache alive but no ServiceWatcher after 35 seconds; restarting.
Stopping httpd: [FAILED]
Starting httpd: [ OK ]

* * * * * * * * *
WHAT'S HAPPENING...
* * * * * * * * *
The message "Apache alive (for [300+] seconds)" indicates that Low Level Service Watcher is hitting the 5 minute timeout for starting up Apache. It then gives up and tries to restart Apache again.

If you check the status of the web server you will likely see that only one process has started. This is indicated by a single process ID number after "pid" --there should be several:

# service httpd status
httpd______________________________________ (pid 25492) is running...

You can also check to see how many Apache processes are running with this command:

# psg httpd

You should see several processes listed. If you're only seeing one instance, that is probably an indication of this problem.

* * * * * * * * *
SOLUTION
* * * * * * * * *
You can increase the timeout in LowLevelServiceWatcher by editing the following file:

/root/svn/mercury/lib/perl/Mercury/Daemon/LowLevelServiceWatcher.pm

Search for the following line:


sub apache_uptime_threshold { 5 * 60 }


The line above indicates a timeout of 5 minutes. Try bumping that up to something like 15 minutes:


sub apache_uptime_threshold { 15 * 60 }


...then do a make:

# root
# make

Watch the daemons to see if they come up:

# wd

...the order that processes start up is:

LowLevelServiceWatcher
Apache (httpd)
ServiceWatcher
(the rest of the processes)

When Apache comes up again, wait up to the full timeout (threshold) period that you specified to see if ServiceWatcher comes up. All the processes should eventually come up as expected shortly after that.

* * * * * * * * *
DOCUMENTATION
* * * * * * * * *
Also, please check the AWMS Hardware Sizing Guide on the Support Download page for the latest hardware requirements:
http://www.airwave.com/download