What to do when the file descriptor limit is reached

When the file descriptor limit is reached on your server, you may see problems right away. You might notice high CPU usage, failed connections, or issues that do not have clear causes. These symptoms can disrupt your workflow and affect users. Quick action helps you restore normal server behavior and prevent data loss.

• High CPU usage
• Failed connections
• Hard-to-trace symptoms

With the right steps, you can solve these problems and keep your server running smoothly.

Immediate Steps for File Descriptor Limit Issues

Common Symptoms and Errors

You may notice your server acting strangely when it hits the file descriptor limit. Applications can stop responding. Users may see connection failures. Sometimes, you will find error messages in your logs that point to resource problems. Here are some of the most frequent errors you might encounter:

You should check your logs for these messages. They help you confirm that the number of open file descriptors has reached the system-wide limit.

Freeing Up Descriptors Quickly

When you face this problem, you need to act fast. Start by finding which processes use the most open file descriptors. You can use the following command to list all open files:

lsof | less

To see which process uses the most, try:

lsof | awk '{print $2}' | sort | uniq -c | sort -nr | head

This command shows you the process IDs with the highest counts. If you find a process that should not use so many files, you can stop or restart it. You can also close unused applications or connections. This action frees up resources and helps your server recover.

Tip: Sometimes, a single application can leak file descriptors. Check for runaway logs or stuck network connections.

Restarting Services or Processes

If freeing up resources does not solve the issue, you may need to restart services. Restarting a service releases all file descriptors it holds. Use the following command to restart a service (replace service_name with the actual name):

sudo systemctl restart service_name

If you do not know which service causes the problem, you can restart the most common ones, such as web servers or database servers. In some cases, you may need to restart the entire server. This step should be your last resort.

You can also check the current maximum open file descriptors for your shell session with:

ulimit -n

If you need to raise the limit temporarily, use:

ulimit -n 4096

This command sets the new limit for the current session. Remember, this change does not affect other users or the system-wide limit.

By following these steps, you can restore normal operation quickly. You will also prevent further issues while you investigate the root cause.

Check Number of Open File Descriptors

Using lsof and Other Tools

You need to know how many open file descriptors your server uses. The lsof command gives you a clear view of all open files and helps you spot problems fast. You can use these commands to check and monitor usage:

• lsof: Lists all open files, including sockets and pipes.
• sudo lsof | head -20: Shows the first 20 open files. Use sudo for full results.
• sudo lsof | awk 'NR>1 {print $1, $2}' | sort | uniq -c | sort -rn | head -20: Counts open file descriptors for each process and sorts them.
• lsof -p $(pgrep myapp) | wc -l: Counts open file descriptors for a specific process.
• watch -n 2 "lsof -p $(pgrep myapp) | wc -l": Monitors the count for a process every two seconds.

The lsof tool helps you find which processes use the most resources. You can also spot locked or unreleased files that may cause issues. If you see one process with a high count, you may have found the cause of your problem.

Viewing Limits with ulimit and System Files

You should also check the current file descriptor limit for your session. The ulimit -n command shows the maximum number of open file descriptors allowed. If you want to change this limit for your session, use ulimit -n <number>. For more control, use ulimit -Sn <number> for the soft limit or ulimit -Hn <number> for the hard limit.

To make changes permanent, you need to edit system files. Update /etc/security/limits.conf for user limits or /etc/systemd/system.conf for service limits. After you change system service limits, run systemctl daemon-reload and restart the service.

Here is a table showing default limits on popular Linux systems:

You should check and modify system limits if you run large applications or many services. This step helps you avoid hitting the file descriptor limit again.

Increase File Descriptor Limit

When you reach the file descriptor limit, you need to act fast to keep your server stable. You can raise this limit in two main ways: temporarily for your current session or permanently for the whole system. You should choose the method that fits your needs and the type of application you run.

Temporary Changes with ulimit

You can use the ulimit command to change the maximum open file descriptors for your current shell session. This method works well when you need a quick fix or want to test changes before making them permanent. High-load environments, like busy web servers or database servers, often need a higher limit during peak times. Backup tools, file sync programs, and applications with file descriptor leaks also benefit from a temporary increase.

Common situations where you need to raise the limit include:

• Applications with many worker threads or goroutines
• Programs that open many sockets or files at once
• Servers that handle heavy traffic or many connections
• Log files that do not rotate or close properly

To check your current limit, run:

ulimit -n

To set a new limit for your session, use:

ulimit -n 65536

This command sets the number of open file descriptors to 65,536 for your shell. If you close the shell or restart your server, the limit returns to its default value. You may need superuser rights to set very high values.

If you set the limit too low, you may see errors like Nginx refusing new connections or MongoDB failing to read and write data. Other applications can also stop working if they cannot open enough files.

Permanent Changes in sysctl.conf and Limits.conf

For a long-term solution, you should update your system configuration files. The sysctl.conf file controls the system-wide limit for file handles. The fs.file-max parameter sets the highest number of file handles your kernel can use. This setting is important for servers that run many applications or handle lots of users at once.

If you do not raise the system-wide limit, you may see “Too many open files” errors. These errors can cause crashes and service outages. You should also update /etc/security/limits.conf to set user-specific limits. This file lets you control how many files each user or group can open.

To change the system-wide limit, add this line to /etc/sysctl.conf:

fs.file-max = 100000

Apply the change with:

sudo sysctl -p

To set user limits, add lines like these to /etc/security/limits.conf:

* soft nofile 65536
* hard nofile 65536

The asterisk (*) means the rule applies to all users. You can replace it with a username if you want. The soft limit is what the shell uses by default. The hard limit is the maximum you can set. You may need to log out and log back in for these changes to take effect.

Applying Changes for systemd Services

Many modern Linux systems use systemd to manage services. You must update the service configuration to change the file descriptor limit for these services. Follow these steps:

1. Create an override file for your service:

   sudo systemctl edit your-service-name.service

2. Add this line under the [Service] section:

   LimitNOFILE=65536

3. Reload the systemd daemon:

   sudo systemctl daemon-reload

4. Restart your service:

   sudo systemctl restart your-service-name.service

5. Check that the new limit is active. Find the main process ID:

   systemctl status your-service-name.service

   Then run:

   cat /proc/[PID]/limits | grep "Max open files"

You can also use systemd-delta --type=extended to confirm your changes. The command systemctl status your-service-name.service shows the current settings.

Tip: Always reload the systemd daemon after making changes. Restart the service to apply the new settings.

By following these steps, you can control the number of open file descriptors for your applications. This helps you avoid service failures and keeps your server running smoothly.

Prevent File Descriptor Exhaustion

Monitoring with Netdata or Zabbix

You can prevent file descriptor exhaustion by using monitoring tools that give you real-time data and alerts. Netdata and Zabbix both help you track open files and sockets without manual checks. These tools make it easier to spot problems before they affect your users.

Netdata provides real-time visibility, so you see changes as they happen. Zabbix lets you monitor many devices and applications, and it stores data for long-term analysis. Both tools reduce the time and effort you spend on manual monitoring. You can set up alerts to warn you when usage nears the file descriptor limit.

Tip: Automated monitoring allows you to focus on other tasks while staying confident that you will catch issues early.

Optimizing Logging and Application Settings

You should review your application settings to avoid unnecessary file descriptor use. Many applications open files or sockets for logging, network connections, or caching. If you do not optimize these settings, you may run into errors like “Too many open files.”

You should close file descriptors right after reading or writing configuration files. Try to keep the number of open file descriptors below 50. Restart your application regularly to clear unused resources. Always check your ulimit settings to make sure your application can handle the expected load.

Regular Review and Team Awareness

You can prevent future problems by making regular audits part of your routine. Review your server’s resource usage and configuration files often. Load test your server to understand its limits and plan for growth. Use rate limiting to control how many requests your server accepts.

• Train your team on resource management patterns.
• Share post-mortems of incidents involving resource leaks.
• Add resource management to your coding standards.
• Create checklists for high-risk code areas.

Regular reviews and team training help everyone spot and fix issues before they become critical.

You can resolve file descriptor limit issues by following these steps:

1. Test your server at twice the expected peak load.
2. Monitor file descriptor counts and TCP states.
3. Set ulimits during server setup.

For high-concurrency servers, watch your metrics and adjust limits carefully. If you set limits too high, you risk system instability.

Stay proactive and keep learning to protect your server’s health.

FAQ

What is a file descriptor?

A file descriptor is a number that your operating system uses to track open files, sockets, or pipes. Each running process gets its own set of file descriptors.

How do I know if my server hit the file descriptor limit?

You may see errors like “Too many open files” in your logs. Use lsof or ulimit -n to check usage and limits.

Can I increase the file descriptor limit without restarting my server?

You can raise the limit for your current session using ulimit -n <number>. For permanent changes, you must update system files and restart affected services.

What causes file descriptor leaks?

Applications may not close files or sockets properly. Poor logging practices or bugs in code often lead to leaks. Regular monitoring helps you catch these issues early.