Monitoring, Management & Location Tracking

* Why do we use computer RAM?

Processors execute code. They take a specific input stream and produce an output stream based on the input. This stream of data comes from the processor cache and RAM (memory).

Both memory and hard drives are a form of data storage. However, processors operate at a high speed in terms of how fast they process data. Memory can supply data at a similarly fast rate. Hard drives are much slower but are stable in the sense that you can remove power and they will retain their stored data (nonvolatile storage). Due to this speed disparity between our stable storage device (hard drive) and the processor we use memory as an intermediate step. Another way to think of this is that hard drives are long term memory, RAM is short term memory.

* What happens if memory runs out?

Computer memory acts as a buffer between stable non-volatile storage and the processor. This buffer can overflow. When this occurs computers typically use a swap partition on the disk to begin using the disk as extended memory. Unfortunately, this means the processor is now stuck waiting for the slow speeds of the disk when it wants to access that memory. This typically causes a server to slow to the point that it is unusable. It will typically be inaccessible via SSH/HTTP but will respond to pings.

Once a Linux server actually runs out of available memory and is stuck swapping running process memory to disk, it will invoke the Out of Memory killer. Evidence of this can be seen in /var/log/messages such as:

Oct 13 17:56:42 localhost kernel: Out of memory: Kill process 5458 (apache2) score 30 or sacrifice child
Oct 13 17:56:42 localhost kernel: Killed process 5458 (apache2) total-vm:274536kB, anon-rss:17224kB, file-rss:0kB
Oct 13 18:02:42 localhost kernel: Adding 524284k swap on /dev/xvdb. Priority:-1 extents:1 across:524284k SS

This indicates Linux actually ran out of memory and started killing processes to recover.

* How much memory is actually available?

On AirWave the “AMP” memory usage from the System->Performance page is the actual used memory. This is because kernel buffer and cache can be purged if necessary. Another way to view this from the command line is the “free –m” command which includes a “-/+ buffers/cache” line which shows the actual used/available memory if you eliminate buffers and cache.

[root@localhost mercury]# free -m
             total       used       free     shared    buffers     cached
Mem:          7872       7441        430          0        152       4959
-/+ buffers/cache:       2329       5542
Swap:         4095          0       4095

In this example 7441/7872 is used, but if you remove buffers/cache, only 2329/7872 is used. This 2329/7872 value is what customers should pay attention to.

* Why is swap being used if there is free memory?

There is a Linux kernel parameter known as Swappiness which determines how a Linux system will use swap space: http://en.wikipedia.org/wiki/Swappiness

By default, including on AirWave, this is set to 60. A value of 0 means swap will only be used if there no available memory, meaning the total used memory - (buffers + cache) has exceeded the available memory on the system. A value of 100 indicates that the kernel will attempt to swap any inactive memory pages to disk as often as possible. There is a drawback to setting this value too low or too high as explained in the Wikipedia article.

What this means for AirWave is that customers will see swap being used regularly if their system is active, even if they have more than adequate free memory available.

* When is memory or swap usage actually a problem?

Swap is typically not a concern. The only concern customers should have is if total used memory - (buffers + cache) is greater than the total memory. This can be viewed in the AMP System->Performance page historical memory usage for “AMP” memory. If “AMP” memory is not exceeding or close to exceeding total memory the customer should not be concerned.