Network Management

Hey Guys,

We have Airwave monitoring around 450 devices and the nightly backup is scheduled for the default 4:15am. The backups seem to take extraordinarily long to complete, on average of 7 hours.

These are the last 4 backups:

total 3.5G
-rw-r----- 1 root apache 796M Oct 14 10:06 nightly_data001.tar.gz
-rw-r----- 1 root apache 788M Oct 13 09:25 nightly_data002.tar.gz
-rw-r----- 1 root apache 787M Oct 12 09:08 nightly_data003.tar.gz
-rw-r----- 1 root apache 777M Oct 11 08:45 nightly_data004.tar.gz

The files aren't quite large as you can see but should they be taking this long? If not, what could be the issue here?

Thanks.

The main contributor to backups taking a long time is the tar process.

You'll get a good idea of how long that takes from these:

Check size of used disk space:

         # df -h

Check the size of the RRD files:

          # du -hs /var/airwave/rrd

Count of how many files are in the folder

         # cd /var/airwave/rrd; find . -type f | wc -l

Client behavior is typically a good sign to gauge this.  If you have a log of unique clients daily, then I'd expect backups to take longer since there'd be more files to compress and backup.  If you maintain long periods of historical data retention, this would also factor against time to backup.

Another good thing to track would be how the nightly backup process is doing and where it's spending the most time.  This can be glossed from several logs in /var/log: nightly_backup, nightly_maintenance.

Additionally, you can open a support case to have a TAC member take a closer look with you.

Hey Rob

Thanks for the info.

[root@localhost rrd]# df -h

Filesystem           Size Used Avail Use% Mounted on

/dev/mapper/VolGroup00-LogVol00            455G   35G 397G   9% /

tmpfs                 7.8G     0 7.8G   0% /dev/shm

/dev/sda1             97M   36M   57M 39% /boot

[root@localhost ~]# du -hs /var/airwave/rrd

13G     /var/airwave/rrd

[root@localhost ~]# cd /var/airwave/rrd; find . -type f | wc -l

512493

From the log files it seems the process takes about 5 hours doing the tar file.

Tue Oct 14 05:01:17 EDT 2014
+ tar -hczf /var/airwave-backup/nightly_data.tar.gz --exclude-from=/tmp/exclude_from_backup /var/log /tftpboot /var/airwave /alternative/postgres_dump_29386 /etc/ntp.conf /etc/resolv.conf /etc/pki/tls/aw-ca-bundle.pem /tmp/amp-backup-version /etc/postfix/main.cf /var/airwave-backup/visualrf/visualrf_backup_localhost.localdomain.zip
tar: Removing leading `/' from member names
tar: /var/log/visualrf/visualrf.log: file changed as we read it
tar: /var/airwave/rrd/client_bandwidth_A3: file changed as we read it
tar: /var/airwave/rrd/client_bandwidth_9E: file changed as we read it
+ set +x
Tue Oct 14 10:06:47 EDT 2014
(finished tar)

Is there any way we could get this time down?

Thanks

Looks like the timing is due to the number of RRDs.  Tar is slow, and we've got tasks scheduled to look into other backup / compression packaging protocols to possibly replace it in the future.  For now, your closest solution would depend on how long you are keeping client data.

Try fine tuning the inactive client and client association variables under AMP Setup -> General -> Historical Data Retention.  These 2 settings directly correlate to the number of RRD files on your AirWave system.

Thanks again Rob,

I'm assuming a restoral would take a similar time?

I typically expect time to restore a backup to be less than time to create the backup.  Main reason being that there's no clean up process prior to a restore.

Rob is correct with regards to the RRDs, the more you have the longer the backup will take to complete.

I've had my fair share of issues with backups before and I still going through some pain but TAC, Rob and Dan have been very helpful with regards to that.

Also, corrupts RRDs can also cause this behavior. You can run the following commands (in screen mode) to clean out corrupt RRDs...

type 'scripts'

[root@localhost scripts]# ./identify_broken_rrds > /tmp/corruptrrd.txt

In that text file, it will tell you which RRDs are corrupt.

You can simply eliminate one by one or probably the use the method below

rm -rf /var/airwave/cache/rrdcached/rrd.journal* 

rm -rf /var/lib/airwave/cache/fastmmap/awrrdtool_write_cache 
service rrdcached restart 

After the above commands are complete execute the command

cat /var/airwave/rrd/rrd_check 

If the output of the above command shows any files then proceed with executing the below command.

touch /tmp/base 

perl -e 'utime time+600, time+600, "/tmp/base"' 
/usr/bin/find /var/airwave/rrd -newer /tmp/base -type f > /var/airwave/rrd/rrd_check 

see if this helps...