Environment Information : Airwave version : 7.7.7
Symptoms : As Symptoms for this issue we can see the httpd daemons restarting. We can see the daemons by executing below command,
# wd
Also, when we tail low_level_ service watcher we would be able to see pound in stopped state and httpd daemon restarting.
To confirm the pound status we can check the status of Pound by executing the below command,
[root@localhost mercury]# service pound status
pound is stopped
Since, pound is in stopped state we can check whether postgresql is running or not. (In this scenario it would be in stopped state).
[root@localhost mercury]# service postgresql status
postmaster is stopped
When we try to restart it, it would fail and would give us the following error message,
[root@localhost mercury]# service postgresql restart
Starting postgresql service: [Failed]
postmaster dead but pid file exists
Cause : To get a better idea on why postgres is not starting we can check the pgsql log under /var/log
# cd /var/log
# less pgsql
Jan 15 10:24:06 rgppaws001 postgres[29137]: [1-2] LOCATION: CreateLockFile, miscinit.c:919
Jan 15 10:24:52 rgppaws001 postgres[29256]: [1-1] FATAL: 53100: could not write lock file "/tmp/.s.PGSQL.5432.lock": No space left
on device
Jan 15 10:24:52 rgppaws001 postgres[29256]: [1-2] LOCATION: CreateLockFile, miscinit.c:919
Jan 15 10:26:31 rgppaws001 postgres[29532]: [1-1] FATAL: 53100: could not write lock file "/tmp/.s.PGSQL.5432.lock": No space left
on device
The above message states that database is not able to write any more to /tmp/s.PGSQL.5432.lock since there is no space left in "/tmp"
We can do df -h to confirm the space.
Resolution : Clear up the old files in "/tmp" directory. We can use rm -rf command to clear files. Once we get free space under "/tmp" we can try restarting the postgres service.
[root@localhost log]# service postgresql restart
Stopping postgresql service: [ OK ]
Starting postgresql service: [ OK ]
Now the database would be up, as a result Pound and httpd daemons would be up.
Now we would be able to access the web interface.