Raju Ginni

wordpress tutorials seo hosting etc

You are here: Home / Linux sysadmin tutorials linux system administrator / linux performance tuning inode limit file descriptors tco, kernel etc

linux performance tuning inode limit file descriptors tco, kernel etc

 

 

Tune Linux for Performance

Backlog queue /Connection Queue
consider increasing net.core.somaxconn, the maximum number of connections that can be queued awaiting attention from NGINX.
it will be queued in the operating system before they are being handed over to Nginx.

etc/sysctl.conf

# Increase size of file handles and inode cache
fs.file-max = 2097152
 net.core.somaxconn = 65536
net.ipv4.tcp_max_tw_buckets = 1440000
//If there are error messages in the kernel log, increase the value until errors stop.
If you set the somaxconn key to a value greater than 512, change the backlog parameter of the NGINX listen directive to match:
server {
listen 80 backlog=65536;
# The rest of server configuration
}

File descriptors/ max of open files

sys.fs.file_max defines the system wide limit for file descriptors.
nofile defines the user file descriptor limit,
set in the /etc/security/limits.conf file.
system wide  FD Limits

/etc/sysctl.conf

# Increase size of file handles and inode cache
fs.file-max = 2097152
fs.file-max = 100000
sysctl -p

Show all system parameters with their values (default or changed)

sysctl -A
Show values of parameters modified by you

sysctl -p

what is ulimit and how to change it permanently?

to use enough system resources
Setting       Soft/Hard
NoFiles       32768/65536
Filesize      unlimited/unlimited
Data          unlimited/unlimited
Stack         unlimited/unlimited
Memory        unlimited/unlimited
Coredump      unlimited/unlimited

2. set permanent ulimit values at /etc/security/limits.conf 

Restart the system:

esadmin system stopall
esadmin system startall

ulimit -u unlimited
ulimit -u
31878
root@lemp-s-2vcpu-4gb-blr1-01:~# ulimit
unlimited
root@lemp-s-2vcpu-4gb-blr1-01:~# ulimit -u unlimited
root@lemp-s-2vcpu-4gb-blr1-01:~# ulimit -u
unlimited
cat /proc/sys/fs/file-max
813940
root@lemp-s-2vcpu-4gb-blr1-01:~# ulimit -Hn
1048576
root@lemp-s-2vcpu-4gb-blr1-01:~# ulimit -Sn
1024
root@lemp-s-2vcpu-4gb-blr1-01:~# cat /proc/sys/fs/file-max
813940
root@lemp-s-2vcpu-4gb-blr1-01:~# sysctl fs.file-max
fs.file-max = 813940
root@instance-1:~# sysctl fs.file-max
fs.file-max = 9223372036854775807
root@instance-1:~#
root@instance-1:~# ulimit -aS
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31814
max locked memory (kbytes, -l) 65536
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 31814
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
root@instance-1:~# ulimit -aH
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31814
max locked memory (kbytes, -l) 65536
max memory size (kbytes, -m) unlimited
open files (-n) 1048576
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 31814
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
root@instance-1:~#

what is ulimit in linux?

command that can set or report the resource limit of the current user.
ulimit
unlimited
ulimit -a
max process 31814

Difference between hard & soft ulimits in linux?

Hard limits are maintained by the kernel while the soft limits are enforced by the shell.
A soft limit is like a warning and hard limit is a real max limit
“hard” and “soft”. Hard resource limit defines the physical limit that the user can reach. The “soft” resource limit is manageable by the user. Its value can go up to the “hard” limit.
To check the value of max “core file size”, use the flag “-c”.
ulimit -c
root@instance-1:~# ulimit -c
0
For max “data seg size” (in kilobytes)
root@instance-1:~# ulimit -d
unlimited
ulimit -u – maximum number of process
ulimit -u
31814
linux check how many processes are running
Command to count the number of processes running in Linux
root@instance-1:~# ps aux | wc -l
118
ps -U user1 | wc -l
ps -U www-data | wc -l
ps -U mysql | wc -l
ps -U root | wc -l

ps -C httpd | wc -l

 

root@instance-1:~# ps -U www-data | wc -l
12
root@instance-1:~# ps -U mysql | wc -l
2
root@instance-1:~# ps -U root | wc -l
98

To know many file descriptors per process*?

ulimit -n
1024
ulimit -t – time limit to each process run
unlimited
ulimit –help
sudo nano  /etc/security/limits.conf
core: Core file size (in KB).
data: Max data size (in KB)
fsize: Max file size (in KB).
memlock: Max locked-in-memory address space (in KB).
nofile: Max number of file descriptors
stack: Max size of the stack (in KB).
cpu: Maximum CPU time (in MIN).
maxlogins: Maximum number of logins for the current user/group
priority: Set the priority of the user’s processes
rtprio: Max priority in real-time.

do changes in /etc/security/limits.conf require a reboot?

No but you should close all active sessions windows.

 

fs.file-max vs ulimit

ulimit shows the per-process maximum open files by user

ulimit -n
1024

(system wide limit) file-max is the maximum File Descriptors (FD) enforced on a kernel level, which cannot be surpassed by all processes without increasing.

root@instance-1:~# cat /proc/sys/fs/file-max
9223372036854775807

temporary change ulimits for user until restart

-n processes

root@instance-1:~# ulimit -n
1024

ulimit -n 2048

root@instance-1:~# ulimit -n 2048
root@instance-1:~# ulimit -n
2048
root@instance-1:~#

 

What;s the difference between file descriptor and inode limits?

Files are represented by inodes
All information needed by the filesystem to handle a file is included in a data structure called an inode.
Max files for user account in a system ex: 1,50,000 files in shared server account. //general “file on a disk”
essentially an index into an array of open files kept by the kernel. files can be open at a time.

file descriptor: it is just an integer number that uniquely represents an opened file in operating system

 

User Level FD Limits

/etc/security/limits.conf
www-data soft nproc 100000
www-data soft nofile 100000
1
2
soft nofile 4096
hard nofile 4096
two file descriptors for each connection increase no file to support highload by os.
File Descriptors (FD)
worker_rlimit_nofile 100000;

nano /etc/sysctl.conf

 

### IMPROVE SYSTEM MEMORY MANAGEMENT ###

# Increase size of file handles and inode cache
fs.file-max = 2097152
# Do less swapping
vm.swappiness = 10
vm.dirty_ratio = 60
vm.dirty_background_ratio = 2

### GENERAL NETWORK SECURITY OPTIONS ###

# Number of times SYNACKs for passive TCP connection.
net.ipv4.tcp_synack_retries = 2
# Allowed local port range
net.ipv4.ip_local_port_range = 2000 65535
# Protect Against TCP Time-Wait
net.ipv4.tcp_rfc1337 = 1
# Decrease the time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout = 15
# Decrease the time default value for connections to keep alive
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_intvl = 15

### TUNING NETWORK PERFORMANCE ###

# Default Socket Receive Buffer
net.core.rmem_default = 31457280
# Maximum Socket Receive Buffer
net.core.rmem_max = 12582912
# Default Socket Send Buffer
net.core.wmem_default = 31457280
# Maximum Socket Send Buffer
net.core.wmem_max = 12582912
# Increase number of incoming connections
net.core.somaxconn = 65535
# Increase number of incoming connections backlog
net.core.netdev_max_backlog = 65535
# Increase the maximum amount of option memory buffers
net.core.optmem_max = 25165824
# Increase the maximum total buffer-space allocatable
# This is measured in units of pages (4096 bytes)
net.ipv4.tcp_mem = 65535 131072 262144
net.ipv4.udp_mem = 65535 131072 262144
# Increase the read-buffer space allocatable
net.ipv4.tcp_rmem = 8192 87380 16777216
net.ipv4.udp_rmem_min = 16384
# Increase the write-buffer-space allocatable
net.ipv4.tcp_wmem = 8192 65535 16777216
net.ipv4.udp_wmem_min = 16384

# Increase the tcp-time-wait buckets pool size to prevent simple DOS attacks

net.ipv4.tcp_max_tw_buckets = 1440000
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
Paste this (at the end of the file) to /etc/security/limits.conf (using nano /etc/security/limits.conf) and save it,
nginx           soft    nofile          2097152
nginx           hard    nofile          2097152
www-data        soft    nofile          2097152
www-data        hard    nofile          2097152
Paste this (at the end of the file) to /etc/pam.d/common-session (using nano /etc/pam.d/common-session) and save it,
session required pam_limits.so
Change listen.backlog in /etc/php5/fpm/pool.d/www.conf (using nano /etc/php5/fpm/pool.d/www.conf) and save it,
listen.backlog = 65535 //default to system
Change worker_rlimit_nofile in /etc/nginx/nginx.conf (using nano /etc/nginx/nginx.conf) and save it,
worker_rlimit_nofile 99999;
And finally reboot 🙂
Ephemeral ports – When used as a proxy

Tune file descriptor limits on Linux

php-fpm, also uses user as nginx server
www-data
nginx uses 2 file descriptor per connection. max connections 512 with default linux configuration.
apache uses one process per 1 connection.
 default limit 1024 files for one process.
nginx is single threaded /single process.
ulimit-aS
ulimit-aH (you cannot increase above this limit without increasing kernel parameters)
to view current user limits
su www-data
and run above commands.

Updating Default Kernel Parameter Settings

to increase the number of simultaneously connection to db server
msgmni
kernel.shmmax=268435456 for 32-bit
kernel.shmmax=1073741824 for 64-bit
kernel.msgmni=1024
fs.file-max=8192
kernel.sem=”250 32000 32 1024″
useful commands
too, htop, vimsat,iosat, sar
sudo apt-get clean
sudo apt-get autoremove

ubuntu 20.04 server optimization

Tuning Nginx to handle more than 50.000 reqs/s
linux sysctl conf linux security limits conf
Disable unnecessary daemons to save memory & cpu

Tuned – Automatic Performance Tuning of CentOS/RHEL Servers

how to apply changes to /etc/security/limits.conf immediately

 

applies after reboot

without reboot, modify /etc/pam.d/common-session by adding this line at the end of file:

session required pam_limits.so

logout & login

maxfiles limit in linux unix

max files for a process default 1024MB., my mysql server running on 8GB RAM and 100 connections or process 100*2files=200opened files

you may increase when server error says max open files reached.  for tuning you can increse max open fiels based on availablity of RAM.

The maximum number of file descriptors available to a process is controlled by user limit parameters. The “nofiles” parameter is the number of file descriptors available to a process. The default maximum number of open files in a Linux system is 1024

Inode limit:

maximum inode count maxfiles

The maximum number of inodes and maxfiles depends on the volume capacity and the type of file system:

he default maxfiles limit is roughly one file per 32 KiB of volume capacity

fs.file-max  counts max opened file in the system. ideally it should be 10% RAM in the server.

df -i: This command displays the number of inodes available

root@srv443573:~# df -i

inodex check command in linux
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/root 25804800 426222 25378578 2% /
tmpfs 2050188 1 2050187 1% /dev/shm
tmpfs 819200 525 818675 1% /run
tmpfs 2050188 3 2050185 1% /run/lock
/dev/sda15 0 0 0 – /boot/efi
tmpfs 410037 26 410011 1% /run/user/0