oom killer
Dealing with the Linux OOM Killer at the program level
The OOM killer uses the process level metric called oom_score_adj to decide if/when to kill a process. This file is present in /proc/$pid/oom_score_adj. The oom_score_adj can vary from -1000 to 1000, by default it is 0.
ou can add a large negative score to this file to reduce the probability of your process getting picked and terminated by OOM killer. When you set it to -1000, it can use 100% memory and still avoid getting terminated by OOM killer.
On the other hand, if you assign 1000 to it, the Linux kernel will keep killing the process even if it uses minimal memory.
Run this command sudo echo -1000 > /proc/<PID>/oom_score_adj
Dealing with it at the OS level
y default Linux has a somewhat brain-damaged concept of memory management: it lets you allocate more memory than your system has, then randomly shoots a process in the head when it gets in trouble. (The actual semantics of what gets killed are more complex than that – Google “Linux OOM Killer” for lots of details and arguments about whether it’s a good or bad thing).
To disable this behaviour:
- Disable the OOM Killer (Put
vm.oom-kill = 0in/etc/sysctl.conf) - Disable memory overcommit (Put
vm.overcommit_memory = 2in/etc/sysctl.conf) Note that this is a trinary value: 0 = “estimate if we have enough RAM”, 1 = “Always say yes”, 2 = “say no if we don’t have the memory”)
These settings will make Linux behave in the traditional way (if a process requests more memory than is available malloc() will fail and the process requesting the memory is expected to cope with that failure).
Reboot your machine to make it reload /etc/sysctl.conf, or use the proc file system to enable right away, without reboot:
echo 2 > /proc/sys/vm/overcommit_memory
https://gist.github.com/t27/ad5219a7cdb7bcb977deccbc48a480d5
Display the current value of /proc/sys/vm/panic_on_oom.
root@murali:~# cat /proc/sys/vm/panic_on_oom
0
#CentOS
grep -i “out of memory” /var/log/messages
#Debian / Ubuntu
grep -i “out of memory” /var/log/kern.log
root@murali:~# grep -i “out of memory” /var/log/kern.log
Aug 28 03:34:17 murali kernel: [1064779.556009] Out of memory: Killed process 1019968 (mysqld) total-vm:1838484kB, anon-rss:992940kB, file-rss:0kB, shmem-rss:0kB, UID:113 pgtables:2732kB oom_score_adj:0
Aug 28 03:34:29 murali kernel: [1064792.074689] Out of memory: Killed process 1264481 (mysqld) total-vm:1791164kB, anon-rss:839876kB, file-rss:0kB, shmem-rss:0kB, UID:113 pgtables:2120kB oom_score_adj:0
Aug 28 03:34:43 murali kernel: [1064805.215661] Out of memory: Killed process 1264605 (mysqld) total-vm:1803084kB, anon-rss:818264kB, file-rss:0kB, shmem-rss:0kB, UID:113 pgtables:2048kB oom_score_adj:0
Aug 28 03:34:57 murali kernel: [1064819.431296] Out of memory: Killed process 1264715 (mysqld) total-vm:1846932kB, anon-rss:842272kB, file-rss:0kB, shmem-rss:0kB, UID:113 pgtables:2112kB oom_score_adj:0
Aug 28 03:35:27 murali kernel: [1064849.931777] Out of memory: Killed process 1264822 (mysqld) total-vm:1786008kB, anon-rss:811192kB, file-rss:0kB, shmem-rss:0kB, UID:113 pgtables:2032kB oom_score_adj:0
Aug 28 07:23:37 murali kernel: [1078539.566272] Out of memory: Killed process 1264974 (mysqld) total-vm:1847508kB, anon-rss:959296kB, file-rss:0kB, shmem-rss:0kB, UID:113 pgtables:2420kB oom_score_adj:0
Aug 28 20:19:15 murali kernel: [1125076.276146] Out of memory: Killed process 1283191 (mysqld) total-vm:1838936kB, anon-rss:900864kB, file-rss:0kB, shmem-rss:0kB, UID:113 pgtables:2308kB oom_score_adj:0
Aug 28 20:19:23 murali kernel: [1125084.858991] Out of memory: Killed process 1332123 (mysqld) total-vm:1833944kB, anon-rss:467192kB, file-rss:0kB, shmem-rss:0kB, UID:113 pgtables:1388kB oom_score_adj:0
Aug 28 20:19:30 murali kernel: [1125091.104251] Out of memory: Killed process 1332225 (mysqld) total-vm:1813128kB, anon-rss:440428kB, file-rss:0kB, shmem-rss:0kB, UID:113 pgtables:1356kB oom_score_adj:0
Aug 28 20:19:55 murali kernel: [1125116.200865] Out of memory: Killed process 1332352 (mysqld) total-vm:1756944kB, anon-rss:483996kB, file-rss:0kB, shmem-rss:0kB, UID:113 pgtables:1380kB oom_score_adj:0
Aug 28 20:20:02 murali kernel: [1125123.562417] Out of memory: Killed process 1332439 (mysqld) total-vm:1753024kB, anon-rss:441800kB, file-rss:0kB, shmem-rss:0kB, UID:113 pgtables:1312kB oom_score_adj:0
Aug 28 20:20:04 murali kernel: [1125125.271016] Out of memory: Killed process 1332104 (php-fpm8.1) total-vm:715396kB, anon-rss:88816kB, file-rss:2080kB, shmem-rss:57056kB, UID:33 pgtables:500kB oom_score_adj:0
root@murali:~#