Problem:
Server GUI not accessible due to file corruption
Diagnostics:
When we check the daemons using the below command, we will see all the daemons related to Airwave is up and running.
# wd
We can try to access the DB of the server using the below command, and we will see the below error message,
[root@cusqui001rawb4 mercury]# db
psql: could not connect to server: Connection refused
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
If we try to restart the postgres service manually we will see the below permission related message,
[root@cusqui001rawb4 mercury]# service postgresql restart
/etc/init.d/postgresql: line 431: /var/lib/pgsql/9.4/pgstartup.log: Read-only file system
Stopping postgresql service: [FAILED]
Starting postgresql service: /etc/init.d/postgresql: line 228: /var/lib/pgsql/9.4/pgstartup.log: Read-only file system
touch: cannot touch `/var/lock/subsys/postgresql.5432': Read-only file system
/etc/init.d/postgresql: line 145: /var/lib/pgsql/9.4/pgstartup.log: Read-only file system
/etc/init.d/postgresql: line 146: /var/lib/pgsql/9.4/pgstartup.log: Read-only file system
If we try to access any logs or try initiating reboot from CLI we will be getting "bash permission denied" message.
Solution
The above errors and symptoms shows the file system on the server is corrupted. We can fix this by unmounting the partition and doing a power cycle for the server. For unmount you can follow the below command,
# mount -o remount,rw /
Post that do a power recycle. If its a hardware box you can do using the iLO or on physical server.
Post reboot the server should be coming up without any issue and the DB will be accessible. In case if there are still more files to be fixed it will prompt for file system check which we can do using live CD with linux rescue mode. Below are the steps for the same.
We can fix this issue by running the file system check from Linux rescue mode. To do this first mount the ISO for Airwave or centos to the server. When it boots up with the ISO on the boot prompt type "Linux rescue"
boot: linux rescue
Once done run the below commands for doing an LVM scan if you have used Linux LVM to create your partition
lvm pvscan
lvm vgscan
lvm lvchange -ay VolGroup00
lvm lvscan
Once done we can umount "/" so that we can do the file system check
chroot /mnt/sysimage
umount /
We can execute the below command to do the file system check
fsck -f -y /dev/VolGroup00/LogVol00
Once done we can reboot the server using the below command,
reboot
Now the server should be booting up fine.