Monitoring, Management & Location Tracking

Introduction : This article explains the process involved in identifying corrupt RRD files and how to get rid of them.

Due to presence of corrupt RRD files many web pages on Airwave might end up with a page crash.
A sample error message is shown below.

The server has encountered an error while performing your request. 
Please try again or email a copy of this message along with the 
<https://120.126.79.246/display_log?log=/var/log/httpd/error_log> web server 
log to customer support. Contact Aruba Technical Support at 
<mailto:support@arubanetworks.com> support@arubanetworks.com or 
1-800-943-4526 (800-WiFi-LAN). 

Time: Tue Dec 17 13:47:06 2013 
URL: /nf/client_list 
Arguments: limit_to_connected=1 

Error: 

[Tue Dec 17 13:47:06 2013] [error] [client 127.0.0.1] 
AWRRDtool last error: (last: Failed to read any or all header bytes: 
Successlast: see prior output for details) 
One or more of your RRD files seems to be corrupted.
Please run: scripts/identify_broken_rrds at Mercury/Utility/Assert.pm line 45 
at Mercury/Utility/Assert.pm line 45 
59 ( Mercury/AWRRD/Base.pm: 151) ASSERT 
58 ( Mercury/AWRRD/Base.pm: 121) Mercury::AWRRD::Base::check_for_errors([arg 
list truncated]) 
57 ( Mercury/AWRRD/BandwidthBase.pm: 92) Mercury::AWRRD::Base::_tool([arg 
list truncated]) 
56 ( Mercury/AWRRD/BandwidthBase.pm: 153) 
Mercury::AWRRD::BandwidthBase::last('Mercury::AWRRD::ClientBandwidth=HASH(0x 
7fa750c15c58)', undef) 
55 ( Mercury/AWRRD/BandwidthBase.pm: 140) 
Mercury::AWRRD::BandwidthBase::five_minute_average_bps('Mercury::AWRRD::Clie 
ntBandwidth=HASH(0x7fa750c15c58)') 
54 ( Mercury/Client.pm: 520) 
Mercury::AWRRD::BandwidthBase::five_minute_average_kbps('Mercury::AWRRD::Cli 
entBandwidth=HASH(0x7fa750c15c58)') 
53 ( Mercury/Client.pm: 523) 

From the error message one can see that the script to identify broken RRD's is to be executed to identify broken RRD files.

Answer :

Step1:
Before we start identifying the corrupt or broken RRD files it is always good to take a backup of the server in case something goes wrong during the process mentioned below.

To take a manual backup execute the command
#amp_backup 
Once the backup process is complete extract the backup file named as databackup.tar.gz from the /alternative directory.

Step2:
To run the script to identify broken RRD files, execute the following command.

./scripts/identify_broken_rrds > /tmp/rrd.txt 

The above command will save the list of broken RRD files in rrd.txt file located in the /tmp directory.

Step3a:
Create a folder corruptedfiles in /tmp using mkdir /tmp/corruptedfiles

Step3b:
Move the broken RRD files to the /tmp/corruptedrrdfiles

Once all the broken RRD files are replaced with known good files the page crash would not occur anymore.