Skip to main content
Back to Category

Service Auto-Restart Configuration with systemd

Configure automatic service restart with systemd for crash recovery, health checks, and monitoring in production environments.

Read time: 11 min Troubleshooting
systemdserviceauto restartwatchdoglinuxuptimereliability
Author
REXE Teknoloji Network & Security Team
Editor
REXE Teknoloji Technical Editorial
First published
Last updated

Introduction

In production environments, services crashing unexpectedly is inevitable. Properly configured auto-restart mechanisms minimize downtime and improve system reliability. This guide covers configuring service auto-restart with systemd.

Why Auto-Restart?

  • Service crashes due to memory leak or unexpected error
  • Temporary network issues affect the service
  • Dependent service starts late and main service fails at startup
  • Kernel OOM killer terminates the service

Step 1: Check Current Service Status

hljs bash
# Service status
systemctl status service_name

# Service logs
journalctl -u service_name -n 50

# Recent crashes
journalctl -u service_name --since "24 hours ago" | grep -i -E '(fail|error|crash|killed|exit)'

# Restart count
systemctl show service_name | grep -E '(NRestarts|ActiveState|SubState)'

Step 2: systemd Auto-Restart Configuration

hljs ini
[Unit]
Description=My Service
After=network.target

[Service]
Type=simple
User=www-data
ExecStart=/usr/bin/myapp
Restart=always
RestartSec=5
StartLimitIntervalSec=60
StartLimitBurst=5

[Install]
WantedBy=multi-user.target

Restart Options

OptionDescription
Restart=noNo auto-restart (default)
Restart=alwaysRestart in all cases
Restart=on-failureRestart only on failure
Restart=on-abnormalRestart on abnormal exit
Restart=on-abortRestart when killed by signal
hljs bash
# Reload and start
sudo systemctl daemon-reload
sudo systemctl enable service_name
sudo systemctl start service_name

Step 3: Advanced Restart Configuration

Restart Limit

hljs ini
[Service]
Restart=on-failure
RestartSec=10
# Stop if fails more than 5 times in 5 minutes
StartLimitIntervalSec=300
StartLimitBurst=5

Memory Limit with Auto-Restart

hljs ini
[Service]
Restart=always
RestartSec=5
MemoryMax=512M
MemoryHigh=400M

Watchdog Health Check

hljs ini
[Service]
Restart=on-failure
RestartSec=5
WatchdogSec=30
NotifyAccess=main

Step 4: Application-Specific Configurations

Node.js Application

hljs ini
[Unit]
Description=My Node.js App
After=network.target

[Service]
Type=simple
User=nodeuser
WorkingDirectory=/var/www/app
ExecStart=/usr/bin/node /var/www/app/index.js
Restart=always
RestartSec=5
Environment=NODE_ENV=production
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Docker Container

hljs yaml
# docker-compose.yml
services:
  app:
    image: myapp
    restart: unless-stopped
hljs bash
# Docker restart policy
docker update --restart=unless-stopped container_name

Step 5: Service Dependencies

hljs ini
[Unit]
Description=My Web App
After=network.target mysql.service
Requires=mysql.service

[Service]
Restart=on-failure
RestartSec=10
ExecStartPre=/bin/sh -c 'until mysqladmin ping -h localhost; do sleep 2; done'

Step 6: Troubleshooting

hljs bash
# Why isn't the service starting?
journalctl -u service_name -n 100 --no-pager

# Check restart loop
systemctl show service_name | grep NRestarts

# List all failed services
systemctl --failed

# Reset failed counter
systemctl reset-failed service_name

When StartLimitBurst is reached, systemd stops auto-restarting. Use systemctl reset-failed service_name to reset the counter.

Conclusion

Systemd auto-restart configuration significantly improves service reliability in production. Use Restart=always or Restart=on-failure for basic configuration, set the restart interval with RestartSec, and prevent infinite loops with StartLimitBurst. Monitor service status continuously with tools like Uptime Kuma.

Frequently Asked Questions

What is the difference between Restart=always and Restart=on-failure?

'always' restarts the service in all cases — including normal exit, error, or being killed by a signal. 'on-failure' only restarts on non-zero exit code or being killed by a signal. For production services, 'on-failure' is generally preferred.

The service keeps restarting but won't run, what should I do?

First check error logs with 'journalctl -u service_name -n 100'. The issue is usually a missing dependency, wrong configuration, or permission problem. If StartLimitBurst is reached, reset the counter with 'systemctl reset-failed service_name'.

How do I set restart policy for Docker containers?

In docker-compose.yml, 'restart: unless-stopped' is the most common option — it always restarts until the container is manually stopped. 'on-failure:5' restarts a maximum of 5 times.

How can I get notified when a service crashes?

systemd's OnFailure directive can trigger a notification service when a crash occurs. This service can send email, Telegram, or Slack notifications. Monitoring tools like Uptime Kuma can also monitor service status and send alerts.

Related Articles

SSH Connection Problem: Can't Connect to Server

SSH connection troubleshooting guide: Path.net firewall rules, Default Block, creating a filter rule for SSH port, and step-by-step troubleshooting.

8 min
sshconnection problemtroubleshooting
Network TrafficInbound— GbpsOutbound— Gbps