Disk Full Recovery: Emergency Response
Emergency response steps when disk space runs out: finding large files with du and ncdu, log cleanup, inode checks, and automatic cleanup configuration.
Table of Contents
Introduction
Running out of disk space can cause service crashes, database errors, and system instability on your server. In this guide, we'll cover emergency response steps and permanent solutions when disk space runs out.
Emergency Detection
Checking Disk Usage
# General disk usage
df -h
# Inode usage
df -i
# With filesystem types
df -Th
When disk usage exceeds 95%, immediate action is required. At 100%, services start crashing.
Finding Large Files and Directories
Analysis with du Command
# Largest directories in root
sudo du -sh /* 2>/dev/null | sort -rh | head -20
# /var directory analysis (usually takes most space)
sudo du -sh /var/* 2>/dev/null | sort -rh | head -10
# /var/log analysis
sudo du -sh /var/log/* 2>/dev/null | sort -rh | head -10
# Find files larger than 100MB
sudo find / -type f -size +100M -exec ls -lh {} \; 2>/dev/null | sort -k5 -rh | head -20
Interactive Analysis with ncdu
# Install ncdu
sudo apt install ncdu -y
# Root directory analysis
sudo ncdu /
# Specific directory analysis
sudo ncdu /var/log
Emergency Cleanup Steps
1. Clean Log Files
# Check systemd journal size
journalctl --disk-usage
# Limit journal to 100MB
sudo journalctl --vacuum-size=100M
# Delete logs older than 7 days
sudo journalctl --vacuum-time=7d
# Truncate large log files (shrink without deleting)
sudo truncate -s 0 /var/log/syslog
sudo truncate -s 0 /var/log/kern.log
# Delete old rotated logs
sudo find /var/log -name '*.gz' -delete
sudo find /var/log -name '*.1' -delete
sudo find /var/log -name '*.old' -delete
The truncate command resets file size to zero without deleting it. This doesn't require restarting services using the file.
2. Clean APT Cache
# Check APT cache size
sudo du -sh /var/cache/apt/archives/
# Clean downloaded package files
sudo apt clean
# Clean old package versions
sudo apt autoclean
# Remove unused dependencies
sudo apt autoremove -y
3. Clean Old Kernels
# Current kernel version
uname -r
# List installed kernels
dpkg --list | grep linux-image
# Remove old kernels (except current)
sudo apt autoremove --purge -y
4. Clean Tmp Directory
# Check /tmp size
sudo du -sh /tmp
# Delete tmp files older than 7 days
sudo find /tmp -type f -atime +7 -delete
# Clean /var/tmp
sudo find /var/tmp -type f -atime +30 -delete
5. Find Deleted But Open Files
Even if a file is deleted, disk space won't be freed if a process is still using it:
# Find deleted but open files
sudo lsof +L1 | grep deleted
# Find large deleted files
sudo lsof +L1 | awk '$7 > 100000000 {print $1, $2, $7, $9}'
# Restart the related process to free space
sudo systemctl restart SERVICE_NAME
Especially when log files are deleted, disk space won't be freed without restarting the related service (nginx, apache, mysql, etc.).
Inode Exhaustion
Even if disk space appears free, inodes can be exhausted:
# Check inode usage
df -i
# Find directories with most files
sudo find / -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head -20
# Clean small file accumulations (e.g., session files)
sudo find /tmp -type f -name 'sess_*' -delete
sudo find /var/spool -type f -mtime +30 -delete
Automatic Log Rotation Configuration
# /etc/logrotate.d/custom-logs
/var/log/myapp/*.log {
daily
missingok
rotate 7
compress
delaycompress
notifempty
create 0640 www-data www-data
}
Journal Size Limit
# /etc/systemd/journald.conf
[Journal]
SystemMaxUse=200M
SystemMaxFileSize=50M
MaxRetentionSec=7day
sudo systemctl restart systemd-journald
Disk Usage Monitoring Script
#!/bin/bash
THRESHOLD=90
PARTITION="/"
USAGE=$(df -h $PARTITION | awk 'NR==2 {print $5}' | tr -d '%')
if [ "$USAGE" -gt "$THRESHOLD" ]; then
echo "WARNING: Disk usage $USAGE% ($PARTITION)"
echo "Largest directories:"
du -sh $PARTITION* 2>/dev/null | sort -rh | head -10
fi
Conclusion
Disk full is a critical issue requiring immediate action. Log files, APT cache, and old kernels are the most common space consumers. By configuring regular log rotation and disk monitoring, you can prevent this issue from recurring.
Related Articles
SSH Connection Problem: Can't Connect to Server
SSH connection troubleshooting guide: Path.net firewall rules, Default Block, creating a filter rule for SSH port, and step-by-step troubleshooting.
RDP Connection Problem: Can't Connect to Windows Server
RDP connection troubleshooting guide: Path.net firewall rules, Default Block, creating a filter rule for RDP port, and step-by-step troubleshooting.
How to Run an MTR Test and Submit Results to Support
Guide to running MTR network tests with WinMTR and Linux MTR, creating ICMP filter rules, and submitting results to support. Diagnose packet loss and latency.