This is best troubleshooted through a support case with TAC.
In the interim, some data points to gather:
1) dbsize (this lists the size of data tables)
2) current Historical Data Retention values (AMP Setup -> General)
3) Which report is failing? What report widgets are included in that report? (see Reports -> Definitions for that particular failed report type)
4) disk space (df -h)
5) available memory (free -m)