Monitoring, Management & Location Tracking

Airwave - Identify and delete corrupt RRD files

Aruba Employee

Introduction : This article explains the process involved in identifying corrupt RRD files and how to get rid of them.

Due to presence of corrupt RRD files many web pages on Airwave might end up with a page crash.
A sample error message is shown below.

The server has encountered an error while performing your request. 
Please try again or email a copy of this message along with the 
<https://120.126.79.246/display_log?log=/var/log/httpd/error_log> web server 
log to customer support. Contact Aruba Technical Support at 
<mailto:support@arubanetworks.comsupport@arubanetworks.com or 
1-800-943-4526 (800-WiFi-LAN). 

Time: Tue Dec 17 13:47:06 2013 
URL: /nf/client_list 
Arguments: limit_to_connected=1 

Error: 

[Tue Dec 17 13:47:06 2013] [error] [client 127.0.0.1] 
AWRRDtool last error: (last: Failed to read any or all header bytes: 
Successlast: see prior output for details) 
One or more of your RRD files seems to be corrupted.
Please run: scripts/identify_broken_rrds at Mercury/Utility/Assert.pm line 45 
at Mercury/Utility/Assert.pm line 45 
59 ( Mercury/AWRRD/Base.pm: 151) ASSERT 
58 ( Mercury/AWRRD/Base.pm: 121) Mercury::AWRRD::Base::check_for_errors([arg 
list truncated]) 
57 ( Mercury/AWRRD/BandwidthBase.pm: 92) Mercury::AWRRD::Base::_tool([arg 
list truncated]) 
56 ( Mercury/AWRRD/BandwidthBase.pm: 153) 
Mercury::AWRRD::BandwidthBase::last('Mercury::AWRRD::ClientBandwidth=HASH(0x 
7fa750c15c58)', undef) 
55 ( Mercury/AWRRD/BandwidthBase.pm: 140) 
Mercury::AWRRD::BandwidthBase::five_minute_average_bps('Mercury::AWRRD::Clie 
ntBandwidth=HASH(0x7fa750c15c58)') 
54 ( Mercury/Client.pm: 520) 
Mercury::AWRRD::BandwidthBase::five_minute_average_kbps('Mercury::AWRRD::Cli 
entBandwidth=HASH(0x7fa750c15c58)') 
53 ( Mercury/Client.pm: 523) 


From the error message one can see that the script to identify broken RRD's is to be executed to identify broken RRD files.

 

Answer :

 

Step1:
Before we start identifying the corrupt or broken RRD files it is always good to take a backup of the server in case something goes wrong during the process mentioned below.

To take a manual backup execute the command
#amp_backup 
Once the backup process is complete extract the backup file named as databackup.tar.gz from the /alternative directory.

Step2:
To run the script to identify broken RRD files, execute the following command.

./scripts/identify_broken_rrds > /tmp/rrd.txt 

The above command will save the list of broken RRD files in rrd.txt file located in the /tmp directory.

Step3a:
Create a folder corruptedfiles in /tmp using mkdir /tmp/corruptedfiles

Step3b:
Move the broken RRD files to the /tmp/corruptedrrdfiles

mv var/airwave/rrd/client_bandwidth_1A/04_46_65_03_CB_1A.in_bps.awrrd /tmp/corruptedrrdfiles/corruptedrrd.txt

Step4:
Next identify any known good rrd file and replace the broken rrd file with the known good file using the copy command.
To identify a known good RRD file, one can pick a file which is not in the list extracted in Step2.
For Example: 
04_46_65_46_FD_1A.in_bps.awrrd is a non corrupt file in the folder client_bandwidth_1A. This file should be over written on the corrupt file which is 04_46_65_03_CB_1A.in_bps.awrrd using the command below
 
cp var/airwave/rrd/client_bandwidth_1A/04_46_65_46_FD_1A.in_bps.awrrd var/airwave/rrd/client_bandwidth_1A/04_46_65_03_CB_1A.in_bps.awrrd

It is best to move into the same directory where the broken RRD file is present and pick a file which is not in the list obtained from Step2.


Once all the broken RRD files are replaced with known good files the page crash would not occur anymore.

Version history
Revision #:
1 of 1
Last update:
‎07-08-2014 04:27 PM
Updated by:
 
Labels (1)
Contributors
Comments
Gerry_uottawa

Thanks , very useful !

jonathan.starkie

Hi, my server chap has run the above and encountered the below error message, can anyone offer advice on how to resolve this? And/or what the error is referring to?

 

 

./scripts/identify_broken_rrds > /tmp/rrd.txt
Failed to create AWRRD lasts list at ./scripts/identify_broken_rrds line 59.

 

 

I had a similar situation that was fixed by recompiling the code after a fresh install of 8.2.0.3, could be extreme but it's worth opening a tac case for it.

root; make
jonathan.starkie

Recompliling the code after a fresh install?

Why would that need to be done? 

 

Forgive me if this is something obvious, my linux knowledge is basic... 

It fixed my issue, for some reason
TAC and I could not get the issues resolved with patches, etc..
Search Airheads
cancel
Showing results for 
Search instead for 
Did you mean: 
Is this a frequent problem?

Request an official Aruba knowledge base article to be written by our experts.