Monitoring, Management & Location Tracking

What to do when graphs are not updating (cache file corruption or rrdtool segfault)

If all AirWave graphs are not updating at all, especially after a power failure or other hardware problem, it's possible that there are corrupt temporary caches of graph data.

How to identify the problem:

If the problem is occuring during/after an upgrade you may see something similar to the below message in the upgrade log:

/var/lib/airwave/cache/fastmmap/client_count_triggers'
UpdateFS: Skipping 'migrate Apache SSL key/cert to pound'
bin/update_db.pl
RRDTool restarted: flush_all
awrrdtool unexpectedly exited at /usr/local/airwave/lib/perl/Mercury/RRDTool.pm line 179.
make[1]: *** [updatedb] Error 255
make[1]: Leaving directory `/root/svn/mercury'
make: *** [upgrade] Error 2
DISABLED - 1!!

You can confirm the cause of the rrdtool crash by examining the /var/log/messages log file where you may find error messages similar to the following output:

Aug 3 17:50:29 static-173-49-223-99 rrdcached[10056]: replaying from journal: /var/airwave/cache/rrdcached/rrd.journal.1312399808.915255
Aug 3 17:50:29 static-173-49-223-99 rrdcached[10056]: Replayed 0 entries (0 failures)
Aug 3 17:50:29 static-173-49-223-99 rrdcached[10056]: replaying from journal: /var/airwave/cache/rrdcached/rrd.journal.1312407784.110793
Aug 3 17:50:29 static-173-49-223-99 rrdcached[10056]: Replayed 0 entries (0 failures)
Aug 3 17:50:29 static-173-49-223-99 rrdcached[10056]: journal processing complete
Aug 3 17:50:29 static-173-49-223-99 rrdcached[10056]: listening for connections
Aug 3 17:50:44 static-173-49-223-99 kernel: awrrdtool[10423]: segfault at 00002afe9c728181 rip 0000000000404a3c rsp 00007fff607c1ef0 error 4
Aug 3 17:50:44 static-173-49-223-99 kernel: awrrdtool[10427]: segfault at 00002acb40dba181 rip 0000000000404a3c rsp 00007fff22eaf810 error 4

The rrd.journal cache file read and subsequent awrrdtool segfault indicates a possible problem with the rrdcache.

These commands would clear those caches and allow them to be rebuilt:

# rm -f /var/airwave/cache/rrdcached/rrd.journal*
# rm -rf /var/lib/airwave/cache/fastmmap/awrrdtool_write_cache
# service rrdcached restart

ADDITIONAL TROUBLESHOOTING STEPS:

Verify that the awrrd cache lock file exists, is owned by apache, and has the right file permissions. Create this file if it does not using touch, chown, chgrp, chmod, etc. The file should look like:

# ll /var/lib/airwave/cache/fastmmap/awrrdtool_write_cache.lock
-rw-r--r-- 1 apache apache 0 Mar 25 16:47 /var/lib/airwave/cache/fastmmap/awrrdtool_write_cache.lock

Version History
Revision #:
1 of 1
Last update:
‎06-25-2014 02:04 PM
Updated by:
 
Labels (1)
Contributors
Search Airheads
Showing results for 
Search instead for 
Did you mean: 
Is this a frequent problem?

Request an official Aruba knowledge base article to be written by our experts.