Skip to main content
Back to Category

Disk Full Recovery: Emergency Response

Emergency response steps when disk space runs out: finding large files with du and ncdu, log cleanup, inode checks, and automatic cleanup configuration.

Read time: 13 min Troubleshooting
diskfullduncduloginodecleanuprecovery
Author
REXE Teknoloji Network & Security Team
Editor
REXE Teknoloji Technical Editorial
First published
Last updated

Introduction

Running out of disk space can cause service crashes, database errors, and system instability on your server. In this guide, we'll cover emergency response steps and permanent solutions when disk space runs out.

Emergency Detection

Checking Disk Usage

hljs bash
# General disk usage
df -h

# Inode usage
df -i

# With filesystem types
df -Th

When disk usage exceeds 95%, immediate action is required. At 100%, services start crashing.

Finding Large Files and Directories

Analysis with du Command

hljs bash
# Largest directories in root
sudo du -sh /* 2>/dev/null | sort -rh | head -20

# /var directory analysis (usually takes most space)
sudo du -sh /var/* 2>/dev/null | sort -rh | head -10

# /var/log analysis
sudo du -sh /var/log/* 2>/dev/null | sort -rh | head -10

# Find files larger than 100MB
sudo find / -type f -size +100M -exec ls -lh {} \; 2>/dev/null | sort -k5 -rh | head -20

Interactive Analysis with ncdu

hljs bash
# Install ncdu
sudo apt install ncdu -y

# Root directory analysis
sudo ncdu /

# Specific directory analysis
sudo ncdu /var/log

Emergency Cleanup Steps

1. Clean Log Files

hljs bash
# Check systemd journal size
journalctl --disk-usage

# Limit journal to 100MB
sudo journalctl --vacuum-size=100M

# Delete logs older than 7 days
sudo journalctl --vacuum-time=7d

# Truncate large log files (shrink without deleting)
sudo truncate -s 0 /var/log/syslog
sudo truncate -s 0 /var/log/kern.log

# Delete old rotated logs
sudo find /var/log -name '*.gz' -delete
sudo find /var/log -name '*.1' -delete
sudo find /var/log -name '*.old' -delete

The truncate command resets file size to zero without deleting it. This doesn't require restarting services using the file.

2. Clean APT Cache

hljs bash
# Check APT cache size
sudo du -sh /var/cache/apt/archives/

# Clean downloaded package files
sudo apt clean

# Clean old package versions
sudo apt autoclean

# Remove unused dependencies
sudo apt autoremove -y

3. Clean Old Kernels

hljs bash
# Current kernel version
uname -r

# List installed kernels
dpkg --list | grep linux-image

# Remove old kernels (except current)
sudo apt autoremove --purge -y

4. Clean Tmp Directory

hljs bash
# Check /tmp size
sudo du -sh /tmp

# Delete tmp files older than 7 days
sudo find /tmp -type f -atime +7 -delete

# Clean /var/tmp
sudo find /var/tmp -type f -atime +30 -delete

5. Find Deleted But Open Files

Even if a file is deleted, disk space won't be freed if a process is still using it:

hljs bash
# Find deleted but open files
sudo lsof +L1 | grep deleted

# Find large deleted files
sudo lsof +L1 | awk '$7 > 100000000 {print $1, $2, $7, $9}'

# Restart the related process to free space
sudo systemctl restart SERVICE_NAME

Especially when log files are deleted, disk space won't be freed without restarting the related service (nginx, apache, mysql, etc.).

Inode Exhaustion

Even if disk space appears free, inodes can be exhausted:

hljs bash
# Check inode usage
df -i

# Find directories with most files
sudo find / -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head -20

# Clean small file accumulations (e.g., session files)
sudo find /tmp -type f -name 'sess_*' -delete
sudo find /var/spool -type f -mtime +30 -delete

Automatic Log Rotation Configuration

hljs bash
# /etc/logrotate.d/custom-logs
/var/log/myapp/*.log {
    daily
    missingok
    rotate 7
    compress
    delaycompress
    notifempty
    create 0640 www-data www-data
}

Journal Size Limit

hljs bash
# /etc/systemd/journald.conf
[Journal]
SystemMaxUse=200M
SystemMaxFileSize=50M
MaxRetentionSec=7day
hljs bash
sudo systemctl restart systemd-journald

Disk Usage Monitoring Script

hljs bash
#!/bin/bash
THRESHOLD=90
PARTITION="/"

USAGE=$(df -h $PARTITION | awk 'NR==2 {print $5}' | tr -d '%')

if [ "$USAGE" -gt "$THRESHOLD" ]; then
  echo "WARNING: Disk usage $USAGE% ($PARTITION)"
  echo "Largest directories:"
  du -sh $PARTITION* 2>/dev/null | sort -rh | head -10
fi

Conclusion

Disk full is a critical issue requiring immediate action. Log files, APT cache, and old kernels are the most common space consumers. By configuring regular log rotation and disk monitoring, you can prevent this issue from recurring.

Frequently Asked Questions

Disk is 100% full and I can't connect via SSH, what should I do?

Access the server via VNC console from the REXE customer panel. From the console, free up space with 'sudo journalctl --vacuum-size=50M' and 'sudo apt clean'.

How do I find which directory takes the most space?

Use sudo du -sh /* | sort -rh | head -10 to see the largest directories in root. Usually /var/log, /var/cache, and /tmp are the biggest space consumers.

I deleted files but disk space wasn't freed, why?

A process may still be using the deleted file. Find these files with lsof +L1 | grep deleted and restart the related service.

df shows free space but I can't create files?

Inodes may be exhausted. Check inode usage with df -i. Many small files (sessions, cache) can cause inode exhaustion.

How can I prevent disk from filling up?

Configure log rotation with logrotate. Set journal size limits. Run regular cleanup with cron jobs. Set up a disk usage monitoring script with alerts.

Related Articles

SSH Connection Problem: Can't Connect to Server

SSH connection troubleshooting guide: Path.net firewall rules, Default Block, creating a filter rule for SSH port, and step-by-step troubleshooting.

8 min
sshconnection problemtroubleshooting
Network TrafficInbound— GbpsOutbound— Gbps