Airwave server GUI down due to file corruption

By esupport posted Oct 18, 2019 03:00 AM

  
Problem:


Server GUI not accessible due to file corruption



Diagnostics:


When we check the daemons using the below command, we will see all the daemons related to Airwave is up and running.

# wd

 

We can try to access the DB of the server using the below command, and we will see the below error message,

[root@cusqui001rawb4 mercury]# db

psql: could not connect to server: Connection refused

        Is the server running locally and accepting

        connections on Unix domain socket "/tmp/.s.PGSQL.5432"?


If we try to restart the postgres service manually we will see the below permission related message,
 

[root@cusqui001rawb4 mercury]# service postgresql restart

/etc/init.d/postgresql: line 431: /var/lib/pgsql/9.4/pgstartup.log: Read-only file system

Stopping postgresql service:                               [FAILED]

Starting postgresql service: /etc/init.d/postgresql: line 228: /var/lib/pgsql/9.4/pgstartup.log: Read-only file system

touch: cannot touch `/var/lock/subsys/postgresql.5432': Read-only file system

/etc/init.d/postgresql: line 145: /var/lib/pgsql/9.4/pgstartup.log: Read-only file system

/etc/init.d/postgresql: line 146: /var/lib/pgsql/9.4/pgstartup.log: Read-only file system


If we try to access any logs or try initiating reboot from CLI we will be getting "bash permission denied" message.



Solution


The above errors and symptoms shows the file system on the server is corrupted. We can fix this by unmounting the partition and doing a power cycle for the server. For unmount you can follow the below command,
 

# mount -o remount,rw /

Post that do a power recycle. If its a hardware box you can do using the iLO or on physical server.

Post reboot the server should be coming up without any issue and the DB will be accessible. In case if there are still more files to be fixed it will prompt for file system check which we can do using live CD with linux rescue mode. Below are the steps for the same.

We can fix this issue by running the file system check from Linux rescue mode. To do this first mount the ISO for Airwave or centos to the server. When it boots up with the ISO on the boot prompt type "Linux rescue"

boot: linux rescue

Once done run the below commands for doing an LVM scan if you have used Linux LVM to create your partition

lvm pvscan
lvm vgscan
lvm lvchange -ay VolGroup00
lvm lvscan


Once done we can umount "/" so that we can do the file system check

chroot /mnt/sysimage
umount /

We can execute the below command to do the file system check

fsck -f -y /dev/VolGroup00/LogVol00


Once done we can reboot the server using the below command,

reboot

Now the server should be booting up fine.

0 comments
1 view