Check Point Memory consumption – Out of Memory

If you issue the top or free command, you may be spooked and lead to believe you are out of memory and you start digging about checking the processes memory usage without getting any wiser. The reason being you and Linux using different terminology with regards to free memory, which I will try to explain this in this post.


Consider the output of the following Linux-commands

Expert@chkp-fw01]# free -m
             total       used       free     shared    buffers     cached
Mem:          5958       5767        191          0        286       4122
-/+ buffers/cache:       1357       4600
Swap:        13311          0      13311

Only 191MB of the physical memory are listed as free.

[Expert@chkp-fw01]# top
top - 10:11:47 up 76 days, 13:03,  1 user,  load average: 0.57, 0.62, 0.60
Tasks: 155 total,   1 running, 154 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.4%us,  0.6%sy,  0.0%ni, 91.5%id,  3.5%wa,  0.1%hi,  1.9%si,  0.0%st
Mem:   6101524k total,  5905804k used,   195720k free,   293136k buffers
Swap: 13631272k total,      140k used, 13631132k free,  4221868k cached

The same is reported by top.



Check Point software claims everything is ok.

SmartView Monitory - memory usage
SmartView Monitor reports no abnormal situation with regards to memory (Seems to be same numbers as cpstat -f old_memory os).
And the same goes for the following CLI-command:

[Expert@chkp-fw01]# cpstat -f memory os

Total Virtual Memory (Bytes):  20206383104
Active Virtual Memory (Bytes): 1423405056
Total Real Memory (Bytes):     6247960576
Active Real Memory (Bytes):    1423261696
Free Real Memory (Bytes):      4824698880
Memory Swaps/Sec:              -
Memory To Disk Transfers/Sec:  -



Disk Cache borrows memory
What we are looking at here is the OS “claiming” you are out of memory and Check Point claims you are not.
And the reason people thinks their systems is out of memory is because any available memory is borrowed by Linux for Disk cache and moved from the free-pool to the cached-pool.
This enables applications to access frequently used files directly from RAM rather than reading from disk, improving performance accordingly.
In our example this means 4122MB of the memory is used for disk cache.

  • If an application requries more memory than available in the free-pool, it will simply take back the memory from the cached-pool.



So the numbers you should focus on are

Expert@chkp-fw01]# free -m
            total       used       free     shared    buffers     cached
Mem:          5958       5767        191          0        286       4122
-/+ buffers/cache:       1357       4600

[Expert@chkp-fw01]# cpstat -f memory os

Total Virtual Memory (Bytes):  20206383104
Active Virtual Memory (Bytes): 1423405056
Total Real Memory (Bytes):     6247960576
Active Real Memory (Bytes):    1423261696
Free Real Memory (Bytes):      4824698880
Memory Swaps/Sec:              -
Memory To Disk Transfers/Sec:  -



Double check if you have experienced any memory problems
If you want to double check you have not experienced any memory exhaustion, refer to the “fw ctl pstat”-command

[Expert@chkp-fw01]# fw ctl pstat

Machine Capacity Summary:
  Memory used: 9% (406MB out of 4349MB) - below low watermark
  Concurrent Connections: 63% (31876 out of 49900) - below low watermark
  Aggressive Aging is disabled

Hash kernel memory (hmem) statistics:
  Total memory allocated: 456097792 bytes in 111352 4KB blocks using 1 pool
  Initial memory allocated: 456097792 bytes (Hash memory extended by 0 bytes)
  Memory allocation  limit: 3647995904 bytes using 512 pools
  Total memory bytes  used: 43932104   unused: 412165688 (90.37%)   peak: 81898532
  Total memory blocks used:    12513   unused:    98839 (88%)   peak:    20760
  Allocations: 2600341503 alloc, 0 failed alloc, 2600027312 free

System kernel memory (smem) statistics:
  Total memory  bytes  used: 669813704   peak: 674592492
  Total memory bytes wasted:  2111250
    Blocking  memory  bytes   used:   951732   peak:  1803108
    Non-Blocking memory bytes used: 668861972   peak: 672789384
  Allocations: 682142 alloc, 0 failed alloc, 680932 free, 0 failed free
  vmalloc bytes  used: 666632524 expensive: no

Kernel memory (kmem) statistics:
  Total memory  bytes  used: 256392060   peak: 291448252
        Allocations: 2601018382 alloc, 0 failed alloc, 2600703628 free, 0 failed free
        External Allocations: 34880 for packets, 114673920 for SXL

In this example you will see the failed allocations are 0, so there is no memory problem on this machine.


Failed Allocation explainations

  • hmem means: Hash Kernel memory – Typically a configuration issue and adjustment to concurrent connections may be required.
  • smem means: System Memory – OS Memory was exhausted. Memory shortage.
  • kmem means: Kernel Memory – Some application did not get memory. Memory shortage.



Hopefully this explains how you and Linux used different terminology with regards to free/used memory and enabled to verify correct operation of your systems.


Gos

5.00 avg. rating (98% score) - 1 vote

Leave a Reply

Your email address will not be published. Required fields are marked *