A full disk can stop the writing of a database, session, log and temporary file and suddenly shut down the site. The first reaction should not be to delete anonymous files or delete Docker and database directories. First, it is necessary to find out which filesystem is full, space or inode, what has grown, and whether the process still holds the deleted file open.
Quick answer:Withdf -hThe filesystem is full and full.df -iFind the status of the inode. Then just check the consumption of directories from top to bottom inside the same mount. Separate the log, backup, upload, database, Docker and open deleted files. Before deleting, verify the data owner, retention and recovery capabilities.
First, figure out what's finished.
df -h
df -i
findmnt
lsblk -fThese commands show the file system layout, capacity, and inode unchanged. The root may be full but the database volume may have enough space, or vice versa.
Space or Inode?
Very small files can complete the inode while a few gigabytes of space is left. In this case, the creation of the new file is broken and a message appears to be disk full. Cache session, mail queue, or temporary directory with millions of files from scripts can be checked.
Measure the directory usage within the same file system.
sudo du -xhd1 /var
sudo du -xhd1 /srvThe option.-xSelect the path that fits the full mount. RunduIt creates I/O on the big tree; run at peak times with limited priority and range. Don't start with a full-server blind scan.
Why isn't your sum equal to df?
dfIt sees the file system consumption andduThe deleted file that the process still kept open, induIt's not visible but keeps the space until the descriptor is closed.
The file was deleted but opened.
If the toollsofInstalled, the display of zero link-count files can find the process holder. Analyze the output by size and PID. Killing the process or truncating the descriptor can corrupt the data; the safe method is usually rotation/reload or controlled restart of the same service after effect checking.
Logs without rotation
Access/error log, application log and debug log may grow rapidly with duplicate errors. Just don't empty the file; fix the cause of the flood, the rotation policy, compression and retention.
Systemd Journal
The journal also has a disk share and its maintenance policy should be consistent with the audit and capacity requirements. First, consider consumption and timeframe. Reducing retention is a decision to keep evidence; in a security incident, log removal can disrupt the audit.
Backup on the same server.
Daily backups without retention are one of the reasons for the continued growth. Worse, backups are not only recoverable on the same disk in storage failure. Check files for history, accuracy, and out-of-server versions. Do not delete any backups before validating the healthy version and retention policy.
Docker and Overlay Storage
Old images, build layers, stopped containers, volume and logs can take up space. Do not manually delete the Docker internal directory; its metadata is managed with the storage driver. First, specify consumption with your Docker tool to sort and volume owner.
For Docker, you must report image, volume, build cache, and log separately; the total alone does not indicate which part is recoverable.
Database and WAL/Binary Log
The growth of table/index, temporary file, PostgreSQL WAL or MySQL binary log must be managed with the same database tool and retention. Direct deletion of files within the data directory can cause the database to be unlaunched. Replication, backup, and point-in-time recovery determine which logs can be recovered.
Upload and file the program
Check media growth, artifact deploy, export, and temporary file. User files may have legal or business obligations. Extension and file name are not sufficient parameters to delete. Lifecycle storage, quota, and transfer to object storage must be designed to be accessed and backed up.
Cache packages and old kernels
The package manager can keep cache and kernel versions are required for rollback. Use the official distribution command and preview changes. Manual deletion of package database files or boot can spoil the next update and boot. Before cleaning the kernel, specify the running state and boot space.
Temporary file and program cache
The name temp or cache does not allow blind deletion. It may depend on active job, session, or lock. Owner, find the last use time and the official method of clearing the program.
Has the disk really grown?
Sometimes the partition or volume after the disc has not yet been resized, the mount is not expected or the data under mount point is written incorrectly.findmntAnd thelsblkResizing the filesystem is a sensitive operation and must be done with filesystem type documents, snapshots, and rollback programs.
When the site just went down.
- Record the file system and inode full.
- Limit unnecessary write and job-generating according to the runbook.
- Manage the least risky recoverable data with owner approval.
- Control the affected service after the headroom is cleared.
- Measure the health of the database and queue before you open traffic completely.
- Monitor the growth from minute to minute.
The goal is to free up a small amount of headroom for controlled recovery, not to quickly remove the largest volume. In a database or queue, a faulty shutdown may require recovery.
The proposed diagnostic order
df -hAnd thedf -iTo determine the type of shortage- Mount and file system accurate with
findmnt du -xA step on the same mountain.- Check deleted-open files and logs
- Classifying backup, docker, database and upload
- Adapting growth timeline to deployment, job or traffic
- Cleaning only with retention and recovery.
- Definition of quota, rotation and preventive alert
What things can't we remove manually?
- File in the data directory database
- Docker or container runtime internal directory
- Backup without validating another healthy version.
- Log file needed incident without archive
- kernel and boot file without package manager
- User file or upload without policy
- Unknown file, just because of the high volume.
Why does deleting a file sometimes not free up space?
If the process descriptor is kept open, the inode remains until it is closed. If the file is deleted on another mount, the file system will remain unchanged.dfAnd re-check the metric and don't follow the success message of the command.
Prevention of the disease
For percent and byte free, define inode, growth rate, log rate, backup age, database growth, and Docker storage alert. The alert is 100% late; the time to load is more important. Test rotation and retention and consider burst capacity when deploying, backuping, and restoring.
The door.Checklist of initial configuration of the serverLocation of log, independent backup and monitoring must be specified prior to production.
When do you need special assistance?
If youdfAnd theduThere are many differences, the database or Docker data directory is the main factor, the hasty removal of the risk of data loss.Monthly management of the serverIt can design a sustainable growth source without blindly manipulating detection and retention, rotation and alert.
Common Questions
Why didn't you clear the space by deleting the large file?
The process may still hold the deleted file open or the file may be on another filesystem.
Can you clean the Docker folder manually?
No, it's going to ruin metadata and container data. Use the official inventory and runtime tools.
What's the difference between space filling and inode filling?
The first is the lack of storage blocks and the second is the lack of the ability to create new files.df -hAnd thedf -iThey're showing them off.
How much space do we have?
There is no public fixed number; growth rates, burst, backup/restore and database needs are determinants.