Questions:
Following questions or issues will be covered here:
- My memory is used permanently at 100%
- My memory usage does not decrease
- My memory usage is low after reboot but increases permanently
- Do I need to configure swap space?
- My swap is used even shortly after reboot
Answer
To explain the issues arising here we need to have a look at the memory management of Linux and the Linux kernel.
Level 1: Programs and Applications
When applications start they usually tell the kernel how many memory they will need to run. Assume they request 1GB of memory. Frequently the apps request a maximum value and therefore the Linux kernel does not reserve the full 1GB of memory as it might never be used by the application. So the kernel confirms the request and might reserve only a percentage (let’s say 25%) of the requested memory. The remaining 75% are “virtual memory” as it is requested but not yet allocated. In case the application uses more than the already allocated 25% of memory the kernel will allocate what is needed. This completely transparent for the application which does not know about the not-yet allocated memory.
This behaviour is meant to optimize memory usage for multiple applications. By this mechanism you can run more applications at the same time as the not-yet assigned memory is available for other applications.
Level 2: Operating System/ Kernel
Now what is done for the applications is done in a similar way for the operating system including the apps. Assume we have multiple apps which all use currently only 25% of their requested memory. We assume having five apps running and all 25% of their memory together indeed fills up the available physical memory for close to 100%. Now what happens when one of the applications starts to consume more than their 25% of memory? The app does not know there is no more physical memory available and wants to use the already confirmed (virtual) memory.
This is the event when swap comes into play. By default operating system (“OS”) (Linux, Windows, MacOS) preallocate some slot on the storage as so called swap or swapfile. When the memory is full and one of the apps requests its already assigned memory the kernel of the OS allocates parts of the swap file and stores the data from the application requesting memory. So this would mean the application will have 25% of its memory in physical memory and the other requested memory will be stored in swap.
Unfortunately using swap as memory extension has one big disadvantage: it is ways slower than physical memory. So an application consuming all of it’s memory would run to 25% in memory and with 75% in swap. This would mean a hughe performance degration for this app.
Therefore we have:
Level 3: Swap
Imagine our scenario as described: the app using almost all of it’s memory is very slow because all memory is already preallocated to the other apps. This makes sense to a certain level. But what if the other applications are not needed currently? In example a reorganization tool which only runs nightly? So this application still has its 25% of memory allocated in physical but does not need it! While our active application runs very slow because of 75% of its memory being in swap.
The solution for the OS is to mark regions of the memory (called “pages”) if they have been accessed recently. If not, the kernel assumes they are not used. Being not used means it does not matter if access to this page is fast (physical memory) or slow (swap). The kernel then “swaps out” this page- it copies its content to the swap file and releases the physical memory to be used by other apps.
When doing this for all unused pages and allocating the released memory to the application currently using it we have applications being “swaped out” but no one cares if they are slow or not as they are not needed in the moment. And we have the busy app which now can have nearly 100% of its allocated memory in fast physical memory. So it is not a bad sign when swap is used!
Simplifications used for swap:
We assume up to now the apps can run directly with their memory allocated (swapped-out) in swap. This is not the case, instead before being able to access the memory the kernel needs to copy the page from swap to physical memory to be available for the app. However, this process is transparent for the app. It is called “swap-in”.
Level 4: Swap-In
Now when our busy application runs in memory it is fast and we have used our memory closed to its optimum. When the nightly app now starts running their pages are in swap at the beginning. This means the kernel will swap-in the pages the app requests to access (meaning some other pages of other apps have to be swaped-out in order to get free memory pages). Once back in physical memory the application can run. To swap-in the kernel needs to have some free pages which will be archived by swapping-out pages from other apps just as before. The app running during the day is now not needed and therefore does not access its pages. The kernel will then swap out the pages from this application and use the freed pages to swap-in the pages from the nightly app.
Now we have the first possible bottleneck: once too many applications are running at the same time the kernel will be very busy just by swapping-out and swapping-in. This will be a performance degration! In this case we have to few memory ressources for the workload we assigned to the server. On the other side, if only a few apps run and allocate only 50% of the memory it appears to be a waste of money paying for the double amount of memory needed. Solution herefore is: use available memory as cache.
Level 5: Cache
Now we have different possible scenarios:
- Applications not running in parallel
Kernel optimizations and swap usage optimizes the performance of these applications. Just as described above: The app running during day is fast as the nightly-app is swapped out. The start of the nightly app is delayed a little bit due to swap-out of the daily app and the swap-in of the nightly app. But once in memory the app runs fast and the daily app does not need performance (hence does not need physical memory during thee night). - Applications running in parallel
For optimized performance there has to be enough physical memory available to keep the active pages of both (or multiple) apps in memory. If there is not enough the kernel has permanently swap-in and swap-out used pages which will cause a performance degration. We could give the system the exact amount of memory to run nearly all time in physical memory. But this is difficult to archive due to the dynamic memory usage of apps. - Application not consuming all physical memory
Now of course in this case there is no need to use swap because the applications fit into the physical memory. But this is still not optimal as we might have expensive physical memory which is not used!
The first two scenarios are covered above so we will step further and have a look at the third scenario.
Apps might not only run in memory they will access files as well. To read and/ or store data. So it accesses the (slow) disk to read certain blocks. The app might need these blocks at a later stage so it will read them again. And later again. Accessing the disk all the time. Again, performance will be low.
Now to optimize this the OS uses the memory not allocated by the apps to act as a cache for the storage. In the above case the app requests some blocks from disks. The kernel will read them and copy the blocks to the app and additionally to a free memory page. When the app later requests again to read these blocks the kernel does not need to access the disk. It is just sufficent to copy the stored blocks from memory to the app. Ways faster as you can imagine.
Similar this happens for disk writes. One app writes a block and a little bit later a second app might write adjacent blocks on the disk. Instead of writing both junks separately the kernel keeps the writes for a while in memory to see if there are further requests coming in. And now needs to access the disk only once to write both operations.
Level 6: Memory Usage Overall
Now it obviously makes no sense to keep not-used pages from apps in physical memory while other apps (or the cache) might have need using this memory. Therefore the kernel performs swap-out of unused pages in advance in order to have memory available for other apps or the cache.
Thus, all physical memory which is not used by apps will be used for cache to optimize disk access.
Level 7: Putting it all together
- Apps will only have physical memory assigned which they are actively using
- Thus, memory for other applications is available
- Memory allocated but not used can be moved to swap to free physical memory for other apps
- When accessed by an app these pages are copied back from swap to memory
- Swap usage is not bad at all
- High swap-in and swap-out rates all the time are bad and will cause performance degradation
- Memory not used by apps will be used for caching purpose
- Unused memory does not make sense therefore the kernel will over time always use nearly 100% of the memory
Summary
After a reboot there will be plenty of memory as the apps have just been started and have not yet allocated all memory (Level 1). As there have not been many disk requests the cache is not filled up, so memory is really free. And swap is not used at all.
When time goes by, apps start running and allocating so swap usage will be increasing. Same applies to apps not running at all- their pages will be swapped-out to free pages for other usage (already in advance). Additionally the amount of free memory will decrease as the cache gets populated by every disk access.
After a while you will notice your system does not have any memory “free” and swap will be used. when swap-in/ swap-out rates are low this is a sign of the system having optimized performance. Not used pages are in swap, the available pages are used for other apps and cache. Perfect!