Skip to main content
Back to Category

Server Hacked: Emergency Response and Recovery

Emergency response steps when your server is hacked: detection, isolation, forensic analysis, and cleanup guide.

Read time: 18 min Troubleshooting
hacksecurityforensicisolationcleanuprecoveryincident response

Table of Contents

Introduction

Discovering that your server has been hacked is a stressful situation, but by following the correct steps, you can minimize damage and recover your system. In this guide, we will cover the entire process from hack detection to cleanup and security hardening.

Signs of a Hack

Common indicators that your server has been compromised:

  • Unknown processes or services running
  • Abnormal CPU/RAM usage (cryptocurrency mining)
  • Unknown user accounts created
  • Unknown keys in SSH authorized_keys files
  • Unknown tasks in crontab
  • Modified system files
  • Abnormal outbound network traffic
  • Website defacement or redirects
  • Spam email being sent

Step 1: Don't Panic and Document

The first and most important step is to stay calm and document everything:

hljs bash
# Save current state
date > /tmp/incident_$(date +%Y%m%d_%H%M%S).log
who >> /tmp/incident_*.log
w >> /tmp/incident_*.log
last -20 >> /tmp/incident_*.log
ps auxf >> /tmp/incident_*.log
netstat -tlnp >> /tmp/incident_*.log
ss -tlnp >> /tmp/incident_*.log

Do not immediately shut down or reboot the server! Forensic evidence may be lost. Document the current state first.

Step 2: Isolation

Isolate the server to prevent the attacker from causing further damage:

Network Isolation

hljs bash
# Block all outgoing connections (except SSH)
sudo iptables -A OUTPUT -p tcp --dport 22 -j ACCEPT
sudo iptables -A OUTPUT -j DROP

# Or remove all rules from REXE Path Panel
# This allows access only via VNC/IPMI

Terminate Active Sessions

hljs bash
# Check active SSH sessions
who
w

# Terminate suspicious sessions
sudo pkill -u suspicious_user

# Terminate all SSH sessions (except yours)
sudo pkill -9 -u root sshd

Step 3: Forensic Analysis

User Account Check

hljs bash
# List all users
cat /etc/passwd | grep -v nologin | grep -v false

# Check for UID 0 (root privilege) users
awk -F: '($3 == 0) {print}' /etc/passwd

# Check recently created users
ls -lt /home/

# Sudo-enabled users
grep -v '^#' /etc/sudoers
ls -la /etc/sudoers.d/

# SSH authorized_keys check
for user_home in /home/*; do
    if [ -f "$user_home/.ssh/authorized_keys" ]; then
        echo "=== $user_home ==="
        cat "$user_home/.ssh/authorized_keys"
    fi
done
cat /root/.ssh/authorized_keys

Process Analysis

hljs bash
# Show all running processes in tree format
ps auxf

# Search for suspicious processes
ps aux | grep -E '(mine|xmr|crypto|kinsing|kdevtmpfsi)'

# High CPU usage processes
top -b -n 1 | head -20

# Check for hidden processes
ls -la /proc/*/exe 2>/dev/null | grep deleted

# Check process file paths
ls -la /proc/[0-9]*/exe 2>/dev/null | grep -v '/usr\|/bin\|/sbin\|/lib'

Crontab Check

hljs bash
# Root crontab
crontab -l

# All users' crontabs
for user in $(cut -f1 -d: /etc/passwd); do
    crontab -l -u $user 2>/dev/null | grep -v '^#'
done

# System cron directories
ls -la /etc/cron.d/
ls -la /etc/cron.daily/
ls -la /etc/cron.hourly/
cat /etc/crontab

# Systemd timers
systemctl list-timers --all

Network Connection Analysis

hljs bash
# Check active connections
ss -tanp
netstat -tanp

# Processes connecting outbound
ss -tanp | grep ESTABLISHED | grep -v '127.0.0.1'

# Listening ports
ss -tlnp

# Connections to suspicious IPs
netstat -an | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head -20

File System Analysis

hljs bash
# Files modified in the last 24 hours
find / -mtime -1 -type f -not -path '/proc/*' -not -path '/sys/*' 2>/dev/null

# Files created in the last 7 days
find / -ctime -7 -type f -not -path '/proc/*' -not -path '/sys/*' 2>/dev/null

# Check SUID/SGID files
find / -perm -4000 -type f 2>/dev/null
find / -perm -2000 -type f 2>/dev/null

# Suspicious files in /tmp and /var/tmp
ls -la /tmp/
ls -la /var/tmp/
ls -la /dev/shm/

# Hidden files
find / -name ".*" -type f -not -path '/proc/*' -not -path '/sys/*' 2>/dev/null | head -50

Log Analysis

hljs bash
# SSH login attempts
grep 'Failed password' /var/log/auth.log | tail -50
grep 'Accepted' /var/log/auth.log | tail -50

# Successful SSH logins
last -20
lastlog

# System logs
journalctl --since "24 hours ago" | grep -i -E '(error|warning|fail|denied)'

# Web server logs
tail -100 /var/log/nginx/access.log
tail -100 /var/log/apache2/access.log

Step 4: Cleanup

Stop Malicious Processes

hljs bash
# Kill suspicious processes
sudo kill -9 <PID>

# Find and kill crypto miner processes
sudo pkill -9 -f 'xmr\|mine\|crypto\|kinsing'

# Disable suspicious services
sudo systemctl stop suspicious_service
sudo systemctl disable suspicious_service

Remove Malicious Users

hljs bash
# Lock suspicious user
sudo usermod -L suspicious_user

# Delete user
sudo userdel -r suspicious_user

# Change root password
sudo passwd root

# Force password reset for all users
sudo passwd -e username

Clean SSH Keys

hljs bash
# Check and clean all authorized_keys files
find / -name authorized_keys -exec cat {} \;

# Keep only your own key
echo "ssh-rsa YOUR_KEY" > /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys

Step 5: Security Hardening

SSH Security

hljs bash
# /etc/ssh/sshd_config
Port 2222                          # Change default port
PermitRootLogin prohibit-password   # Disable root password login
PasswordAuthentication no           # Disable password login
MaxAuthTries 3                      # Maximum attempts
AllowUsers username                 # Only specific users

sudo systemctl restart sshd

Install Fail2ban

hljs bash
sudo apt install fail2ban -y

# /etc/fail2ban/jail.local
[sshd]
enabled = true
port = 2222
maxretry = 3
bantime = 3600
findtime = 600

sudo systemctl enable fail2ban
sudo systemctl start fail2ban

System Update

hljs bash
sudo apt update && sudo apt upgrade -y
sudo apt dist-upgrade -y
sudo apt install unattended-upgrades -y
sudo dpkg-reconfigure -plow unattended-upgrades

Step 6: Restore from Backup (If Needed)

If cleanup is insufficient, you may need a clean installation:

hljs bash
# 1. Backup important data
tar -czf /tmp/backup_data.tar.gz /var/www/ /etc/ /home/

# 2. Reinstall OS from REXE panel
# my.rexe.tr > Services > OS Reinstallation

# 3. Apply security hardening after clean install
# 4. Restore only data files from backup (NOT configuration files)

When restoring from backup, do NOT restore configuration files! The attacker may have added backdoors. Create configurations from scratch.

Conclusion

Rapid and systematic response is critical in server hack incidents. Follow the steps of documentation, isolation, forensic analysis, cleanup, and security hardening in order. To prevent similar incidents in the future, perform regular security audits, keep up with updates, and implement strong access controls.

Related Articles

SSH Connection Problem: Can't Connect to Server

SSH connection troubleshooting guide: Path.net firewall rules, Default Block, creating a filter rule for SSH port, and step-by-step troubleshooting.

8 min
sshconnection problemtroubleshooting

RDP Connection Problem: Can't Connect to Windows Server

RDP connection troubleshooting guide: Path.net firewall rules, Default Block, creating a filter rule for RDP port, and step-by-step troubleshooting.

7 min
rdpconnection problemtroubleshooting

How to Run an MTR Test and Submit Results to Support

Guide to running MTR network tests with WinMTR and Linux MTR, creating ICMP filter rules, and submitting results to support. Diagnose packet loss and latency.

8 min
mtrwinmtrnetwork test