Network Management

So. Today my Airwave (AirWave Management Platform 8.2.10.1) looks like this (PIC1)

When checking the logs for troubleshooting (PIC2)

When checking performance. I can see my /var/log i full (PIC3)

Does the "down" services relates to the /var/log

Where do i empty the /var/log? or increase the size?

Hi, it looks similar to what was already discussed here (at least looking at the complete filling of /var/log/ folder and also considering the Airwave version too). Generally services are affected by the saturation of such an important filesystem's mount point (/var/log/).

Services are stopped due to either system performance or log directory is full.

We have a known issue where kernal log grown in GB size which filled complete /var/log directory, not sure if your server hitting this, need to check all file sizes in that directory to confirm.

You can login in to AMPCLI and go to Advanced > Enter commands > Cleanup 

To clear log files which are older to  bring all services up for now.

so the cleanup actually worked! the services came up again! thank you for the tip!

It might be a temporary fix, if you noticed log directory growing quickly open TAC ticket.

Yeah. the TAC is already on. Thx

Has this been resolved in .11? 

This issue not noticed on all customer servers, to fix we have to comment line in iptable file.

We have added new line in iptables related to packet limit in 8.2.10 which we can comment and their is no harm in it.

I do not think they fix it. we found a solution though.

The partition for /var/log is 20GB.

The problem show itself, as the Kernel file in /var/log was expanding out of proportions. i went from a couple of MB,to 18GB in a couple of weeks, and the rotation did now work.

so what we did was:

root@localhost CentOS7 /var/log # vi /etc/logrotate.d/kernel
/var/log/kernel {
su root root
copytruncate
missingok
create 0600 root root
maxsize 1M
rotate 4
compress
}

/var/log/kernel {
su root root
copytruncate
missingok
create 0600 root root
"/etc/logrotate.d/kernel" 10L, 133C written

and now it works like a smotth operator. 

NB when the /var/log reach max capacity, it shut downs services