You deleted a large log file on your Linux server, but df -h still shows the disk is almost full. The file is gone from the directory, but the disk space is still full. A common cause for this is a deleted-open file, which is a running process that still has the deleted file open. Linux removes the filename, but the file’s storage remains allocated until the last process releases it.
In this guide, you will learn how to find these files with lsof, identify the application responsible, and reclaim disk space without restarting the entire server.
Prerequisites
Before you start, make sure to have these things ready:
- A Linux server with root or sudo access.
- The
lsof utility.
- Permission to manage the affected application.
- A maintenance window if recovering the space requires a service restart.
In this guide, the service-management examples use systemd. Replace example process IDs and service names with the values found on your server.
Note: Do not delete additional application files or restart unfamiliar services before identifying what is holding the space.
Why Deleting a File Does Not Always Free Disk Space on Linux
Linux stores a file’s name separately from its data. When an application opens a file, it receives a file descriptor, which is a reference that allows it to read or write the file. Running rm removes the file’s directory entry, but an existing file descriptor can remain valid.
If the file has no remaining hard links, its storage becomes reclaimable once its final open reference is released. For example:
- A web server opens an access log.
- The log grows to several gigabytes.
- An administrator deletes the log.
- The web server continues writing through its existing file descriptor.
- The deleted file continues consuming disk space.
This explains why deleting an active log file may not immediately solve a full-disk problem. The application must close the old file, usually through its supported log-reopen mechanism or a controlled restart.
Step 1: Check the Affected Filesystem
You can start by checking filesystem usage. To do this, run the following df command:
To check the filesystem containing your log directory, run the command below:
Filesystem Size Used Avail Use% Mounted on/dev/vda1 80G 75G 1.0G 99% /
This is an example output. Your device names and available space may be different.
df reports usage for the filesystem containing the specified path. If /var/log belongs to a separate filesystem, deleting files elsewhere will not free space on that filesystem.
Compare Filesystem Usage with Directory Usage
If /var/log is on the root filesystem, you must run this command to check the main directories:
If /var is on a separate filesystem, check it with this command instead:
-x: Stay on the filesystem being checked.
-h: Display readable sizes.
-d1: Show the size of each directory directly inside the given path.
df shows how much space the filesystem uses, while du counts the files it can find in directories. A deleted file that is still open can use space shown by df but not by du. However, different results do not always mean a deleted-open file is the cause.
Step 2: Install lsof Utility
The lsof utility lists files opened by running processes. Its +L1 option selects open files whose link count is less than one, identifying unlinked files. Install it with:
sudo apt updatesudo apt install lsof -y sudo dnf install lsof -y
If the disk is full and lsof will not install, check whether it is already installed. If not, free some space by removing files you know are safe to delete, then try again.
Step 3: Find Deleted Files That Are Still Open
You can use lsof to find deleted files that are still open by running processes. These files may still use disk space. Run:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAMEmyapp 2481 root 5w REG 252,1 5368709120 0 123456 /var/log/myapp/app.log (deleted)
This example shows an application that still has a deleted log file open. The important columns are:
| Column |
Meaning |
COMMAND |
Process command name |
PID |
Process ID |
FD |
File descriptor; 5w means descriptor 5 is open for writing |
TYPE |
File type; REG means a regular file |
DEVICE |
Device containing the file |
SIZE/OFF |
File size or offset, depending on the entry |
NLINK |
Number of hard links |
NODE |
Inode number |
NAME |
Reported file path, often followed by (deleted) |
In this example, the file size is 5 GiB (5368709120 bytes). The file may use less actual disk space, especially if it is a sparse file.
Avoid Counting Duplicate Files
Several processes can have the same deleted file open. If rows show the same device and inode numbers, they refer to the same file. Do not count its size more than once.
Not every result needs action. Focus on large files on the filesystem that are running out of space.
Step 4: Identify the Process and Its Service
Now you must find out which process is keeping the deleted file open and which service it belongs to. This helps you choose the right way to release the disk space. For example, if its PID is 2481:
ps -p 2481 -o pid,ppid,user,etime,args
On a systemd server, check which service contains that PID:
sudo systemctl status 2481
Review the application name, command line, and service information before taking action. To list only the deleted files this process still has open, run:
sudo lsof -nP -a -p 2481 +L1
Here, -a combines the process filter and the unlinked-file filter, so the output must match both conditions. Before you continue, check the following:
- Whether the file is a disposable log or application data.
- Whether the application supports reopening its logs.
- Whether a restart would interrupt users or background jobs.
- Whether other processes also hold the same deleted file open.
Do not assume the COMMAND value is the systemd service name.
Step 5: Free the Disk Space Safely
To free the disk space, the application must close the deleted file. Some applications can reopen their logs without stopping, making this the first option to try. If the application does not support this, you may need to restart its service. Before restarting, check whether it will interrupt users or running tasks.
Option A: Ask the Application to Reopen Its Logs
For applications that support it, reopening logs avoids a full service restart. For example, if Nginx is installed directly on your server, run:
It tells Nginx to open its current log paths and close the old log files without restarting the service. This uses Nginx’s USR1 log-reopen operation. Then, check again:
sudo lsof -nP +L1df -h /var/log
Make sure the command targets the right Nginx instance. You may need a different command if you use a custom config, multiple instances, or containers.
Option B: Restart the Affected Service
If the application cannot reopen the file, restarting its service may free the space. For an example service named myapp.service, you can run:
sudo systemctl restart myapp.service
Check its status and review recent service logs:
sudo systemctl status myapp.service --no-pagersudo journalctl -u myapp.service -n 50 --no-pager
Restarting a service can interrupt users, connections, or running tasks. For databases and other apps that store data, follow their restart instructions.
Disk space is freed only when all processes close the deleted file. Restarting one process may not be enough if another still has it open.
Avoid Forcing Processes to Stop or Clearing Open Files
Do not use kill -9 as your first solution. It stops the application without letting it shut down properly.
Also, do not clear files through /proc/PID/fd/FD without knowing what they contain. This deletes their contents and may cause application problems.
Instead, for a general troubleshooting workflow, use the application’s log-reopen command or plan a safe service restart.
Step 6: Confirm the Space Is Free
After reopening the logs or restarting the service, check the disk space again. Make sure the deleted file is no longer open and the application is working normally.
df -h /var/logsudo lsof -nP +L1
You must confirm:
- Available space increased on the affected filesystem.
- The large deleted file no longer appears.
- The application remains healthy.
- The application is writing to its current log file.
For the example application, run the commands below:
sudo ls -lh /var/log/myapp/sudo tail -n 20 /var/log/myapp/app.log
The space you recover may not match the total sizes shown by lsof. The same file may appear more than once, some files use less space than their listed size, and applications may still be writing data.
If you have enough free space but the disk is still slow, check this guide on disk I/O monitoring with iostat and atop. It helps you find disk performance problems.
Prevent Disk Space Problems with Log Rotation
Instead of deleting active log files, you can use log rotation to manage them. It moves older logs aside and lets the application write to a new file. You can also set rules to compress or delete old logs so they do not fill the disk.
Check Your Existing Rotation Configuration
Check the main configuration and application-specific rules with the commands below:
sudo less /etc/logrotate.confsudo ls -l /etc/logrotate.d/
If you installed Nginx with a package manager, its log rotation file may be here:
sudo less /etc/logrotate.d/nginx
Review:
- Rotation frequency or size conditions.
- How many rotated files are saved.
- Compression settings.
- Ownership and permissions for new logs.
- The application-specific
postrotate action.
Note: Do not add another rule for a log file that already has one.
Validate Without Rotating Files
You can use debug mode to check your log rotation settings without changing any files. It shows what logrotate would do when it runs.
sudo logrotate --debug /etc/logrotate.conf
It is useful for reviewing configuration, but it does not prove that a postrotate command will succeed during a real rotation.
How copytruncate Can Lose Log Entries
The copytruncate option copies a log and then clears the original file in place. It can help when an application cannot reopen its logs, but log entries written between the copy and truncation can be lost.
It should not be treated as a safe replacement for application-supported rotation.
For more ways to prevent log problems, you can check this guide on managing and rotating production logs.
Check Other Causes of Full Linux Disk Space
If lsof does not show any large deleted files, something else may be using the space. You must check the affected filesystem and look for other causes before restarting any services.
First, check which filesystem is running out of space:
Then, check the files on that filesystem. For the root filesystem, run:
Also, check whether the filesystem has run out of inodes:
If the filesystem runs out of inodes, it cannot create new files even if free disk space remains. Use df -i to check inode usage.
Other causes include filesystem data, snapshots, shared storage blocks, or files hidden under mounted directories. Different results from df and du do not tell you the exact cause.
If you ran lsof inside a container, run it again on the host with sudo to check the host’s processes.
Conclusion
If deleting a file does not free disk space, a running process may still have it open. Use lsof to find it, then reopen the logs or restart the service safely. Check the free space afterward and set up log rotation to help prevent the problem from returning.
Whether you use a PerLod Linux VPS or another Linux server, regular disk checks and proper log management help keep storage under control.