Some day you wake up and everything is broken. Your lighttpd log is filled with:

(response.c.544) file not found ... or so:  Too many open files
(mod_proxy.c.744) socket failed:  Too many open files 
(network_linux_sendfile.c.143) open failed:  Too many open files 
(response.c.544) file not found ... or so:  Too many open files /favicon.ico ->

I'm sure every admin running some popular service (or poorly coded software ) has faced this problem.

No more file handlers

The Linux OS protect himself from abusive resources usage by setting limits. A limit may be system-wide or specific to a user. The default limits suit most hardware and installation. However if you're hosting some resource consuming services you may want to raise these limits (or rewrite this hungry program ) .

System limits: sysctl.conf

Let's have a look at the system limits:

~$ cat /proc/sys/fs/file-nr
5504         0        	206401
|         		|          		|
|         		|          		|
|          |         	maximum number of file handlers
|         	number of allocated but unused file handlers
number of allocated file handlers

File descriptors are allocated in a dynamic way. The amount of opened file descriptors equals column 1 - column 2. If this amount is close to column 3, you should raise the limit to avoid running out of file handlers. To do so, add to your /etc/sysctl.conf file:

fs.file-max = 131072

This new setting will be applied at next startup. To apply it immediatly, run:

~$ sysctl -p

To check the value has been updated:

~$ cat /proc/sys/fs/file-max
131072

User limits: /etc/security/limits.conf

You can also set custom limits for each user. Add these lines to /etc/security/limits.conf to change the number of file handlers that www-data can use:

www-data soft nofile 1024
www-data hard nofile 65535

These limits are used by PAM. They will be used at next user login or next user cron task. To make sure these values are used you may want to check that pam_limits is included in /etc/pam.d/cron. Its content should be:

# Sets up user limits, please define limits for cron tasks
# through /etc/security/limits.conf
session    required   pam_limits.so

Caution: startup services can't use these limits. Startup scripts are run by root so there is no user login and PAM is not used.

Lighttpd limits: server.max-fds

An extreme way to solve the problem would be to edit your /etc/init.d/ startup script to add some ulimit commands. To avoid this,  some servers like Lighttpd support changing the file handler limit in their configuration file. Add the following line to Lighttpd's configuration file:

server.max-fds = 65535