If you issue the top or free command, you may be spooked and lead to believe you are out of memory and you start digging about checking the processes memory usage without getting any wiser. The reason being you and Linux using different terminology with regards to free memory, which I will try to explain this in this post.
Consider the output of the following Linux-commands
Expert@chkp-fw01]# free -m total used free shared buffers cached Mem: 5958 5767 191 0 286 4122 -/+ buffers/cache: 1357 4600 Swap: 13311 0 13311
Only 191MB of the physical memory are listed as free.
[Expert@chkp-fw01]# top top - 10:11:47 up 76 days, 13:03, 1 user, load average: 0.57, 0.62, 0.60 Tasks: 155 total, 1 running, 154 sleeping, 0 stopped, 0 zombie Cpu(s): 2.4%us, 0.6%sy, 0.0%ni, 91.5%id, 3.5%wa, 0.1%hi, 1.9%si, 0.0%st Mem: 6101524k total, 5905804k used, 195720k free, 293136k buffers Swap: 13631272k total, 140k used, 13631132k free, 4221868k cached
The same is reported by top.
Check Point software claims everything is ok.
SmartView Monitor reports no abnormal situation with regards to memory (Seems to be same numbers as cpstat -f old_memory os).
And the same goes for the following CLI-command:
[Expert@chkp-fw01]# cpstat -f memory os Total Virtual Memory (Bytes): 20206383104 Active Virtual Memory (Bytes): 1423405056 Total Real Memory (Bytes): 6247960576 Active Real Memory (Bytes): 1423261696 Free Real Memory (Bytes): 4824698880 Memory Swaps/Sec: - Memory To Disk Transfers/Sec: -
Disk Cache borrows memory
What we are looking at here is the OS “claiming” you are out of memory and Check Point claims you are not.
And the reason people thinks their systems is out of memory is because any available memory is borrowed by Linux for Disk cache and moved from the free-pool to the cached-pool.
This enables applications to access frequently used files directly from RAM rather than reading from disk, improving performance accordingly.
In our example this means 4122MB of the memory is used for disk cache.
- If an application requries more memory than available in the free-pool, it will simply take back the memory from the cached-pool.
So the numbers you should focus on are
Expert@chkp-fw01]# free -m total used free shared buffers cached Mem: 5958 5767 191 0 286 4122 -/+ buffers/cache: 1357 4600 [Expert@chkp-fw01]# cpstat -f memory os Total Virtual Memory (Bytes): 20206383104 Active Virtual Memory (Bytes): 1423405056 Total Real Memory (Bytes): 6247960576 Active Real Memory (Bytes): 1423261696 Free Real Memory (Bytes): 4824698880 Memory Swaps/Sec: - Memory To Disk Transfers/Sec: -
Double check if you have experienced any memory problems
If you want to double check you have not experienced any memory exhaustion, refer to the “fw ctl pstat”-command
[Expert@chkp-fw01]# fw ctl pstat Machine Capacity Summary: Memory used: 9% (406MB out of 4349MB) - below low watermark Concurrent Connections: 63% (31876 out of 49900) - below low watermark Aggressive Aging is disabled Hash kernel memory (hmem) statistics: Total memory allocated: 456097792 bytes in 111352 4KB blocks using 1 pool Initial memory allocated: 456097792 bytes (Hash memory extended by 0 bytes) Memory allocation limit: 3647995904 bytes using 512 pools Total memory bytes used: 43932104 unused: 412165688 (90.37%) peak: 81898532 Total memory blocks used: 12513 unused: 98839 (88%) peak: 20760 Allocations: 2600341503 alloc, 0 failed alloc, 2600027312 free System kernel memory (smem) statistics: Total memory bytes used: 669813704 peak: 674592492 Total memory bytes wasted: 2111250 Blocking memory bytes used: 951732 peak: 1803108 Non-Blocking memory bytes used: 668861972 peak: 672789384 Allocations: 682142 alloc, 0 failed alloc, 680932 free, 0 failed free vmalloc bytes used: 666632524 expensive: no Kernel memory (kmem) statistics: Total memory bytes used: 256392060 peak: 291448252 Allocations: 2601018382 alloc, 0 failed alloc, 2600703628 free, 0 failed free External Allocations: 34880 for packets, 114673920 for SXL
In this example you will see the failed allocations are 0, so there is no memory problem on this machine.
Failed Allocation explainations
- hmem means: Hash Kernel memory – Typically a configuration issue and adjustment to concurrent connections may be required.
- smem means: System Memory – OS Memory was exhausted. Memory shortage.
- kmem means: Kernel Memory – Some application did not get memory. Memory shortage.
Hopefully this explains how you and Linux used different terminology with regards to free/used memory and enabled to verify correct operation of your systems.
Gos
Great write up and still applicable to quickly determine if there is a memory problem or not.