A couple of things:
1. On the AMP Setup -> General page, you can configure the number of monitoring processes. Each of these processes will use a decent amount of memory, so it's important not to have too many when you're bumping up against a RAM limitation. For your size of deployment, I recommend setting that to 1.
2. According to our latest (7.4) hardware sizing guide, the minimum RAM for monitoring 100 devices is 6GB. Recommended is 8GB. Increasing RAM could help too. More is essentially always better.