Service Startup Issues After Reboot
Diagnosing and fixing service startup issues after server reboot: systemctl, failed services, boot log analysis, and systemd configuration.
Table of Contents
Introduction
After a server reboot, some services may fail to start automatically, causing downtime and service disruptions. This guide covers how to diagnose and fix service startup issues after a reboot.
Checking Failed Services
List Failed Services
# List all failed services
sudo systemctl list-units --type=service --state=failed
# Check specific service status
sudo systemctl status SERVICE_NAME
# Show all services and their states
sudo systemctl list-units --type=service --all
Reading Boot Logs
# View current boot logs
journalctl -b
# View previous boot logs
journalctl -b -1
# Filter by service
journalctl -b -u SERVICE_NAME
# Show only errors
journalctl -b -p err
# Show boot time analysis
systemd-analyze
# Show service startup order
systemd-analyze blame
# Show critical chain
systemd-analyze critical-chain
systemd-analyze blame shows which services took the longest to start. This helps identify bottlenecks in the boot process.
Enabling Services for Auto-Start
Enable Service at Boot
# Enable service to start at boot
sudo systemctl enable SERVICE_NAME
# Enable and start immediately
sudo systemctl enable --now SERVICE_NAME
# Check if service is enabled
systemctl is-enabled SERVICE_NAME
# List all enabled services
systemctl list-unit-files --type=service --state=enabled
Common Services to Enable
# SSH server
sudo systemctl enable ssh
# Nginx web server
sudo systemctl enable nginx
# Apache web server
sudo systemctl enable apache2
# MySQL/MariaDB
sudo systemctl enable mysql
sudo systemctl enable mariadb
# Docker
sudo systemctl enable docker
# Fail2Ban
sudo systemctl enable fail2ban
# UFW firewall
sudo systemctl enable ufw
Service Dependencies
Understanding Dependencies
# Show service dependencies
systemctl list-dependencies SERVICE_NAME
# Show reverse dependencies
systemctl list-dependencies --reverse SERVICE_NAME
# Show service unit file
systemctl cat SERVICE_NAME
Configuring Dependencies in Unit Files
# /etc/systemd/system/myapp.service
[Unit]
Description=My Application
After=network.target mysql.service
Requires=mysql.service
Wants=redis.service
[Service]
Type=simple
User=www-data
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/start.sh
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
Key directives:
- After=: Start after these units
- Requires=: Hard dependency (fails if dependency fails)
- Wants=: Soft dependency (continues if dependency fails)
- Before=: Start before these units
Boot Order Analysis
# Full boot time
systemd-analyze time
# Service startup times (sorted)
systemd-analyze blame | head -20
# Critical chain visualization
systemd-analyze critical-chain
# Generate SVG boot chart
systemd-analyze plot > boot-chart.svg
If boot time is too long, consider disabling unnecessary services with sudo systemctl disable SERVICE_NAME.
Creating Custom Systemd Service
# Create service file
sudo nano /etc/systemd/system/myapp.service
[Unit]
Description=My Custom Application
After=network.target
[Service]
Type=simple
User=appuser
Group=appuser
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/start
ExecStop=/opt/myapp/bin/stop
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
# Reload systemd daemon
sudo systemctl daemon-reload
# Enable and start service
sudo systemctl enable --now myapp
# Check status
sudo systemctl status myapp
Troubleshooting Common Issues
Service Fails with Exit Code
# Check detailed error
journalctl -u SERVICE_NAME -n 50 --no-pager
# Check service configuration
systemctl cat SERVICE_NAME
# Verify binary exists and is executable
ls -la /path/to/binary
Service Starts Too Early
# Add network dependency
[Unit]
After=network-online.target
Wants=network-online.target
# Enable network-online target
sudo systemctl enable systemd-networkd-wait-online
Service Keeps Restarting
# Check restart count
systemctl show SERVICE_NAME -p NRestarts
# View recent logs
journalctl -u SERVICE_NAME --since "5 minutes ago"
# Temporarily stop restart loop
sudo systemctl stop SERVICE_NAME
sudo systemctl reset-failed SERVICE_NAME
rc.local Alternative
For simple startup commands, create a systemd service instead of rc.local:
# /etc/systemd/system/rc-local.service
[Unit]
Description=RC Local Compatibility
After=network.target
[Service]
Type=oneshot
ExecStart=/etc/rc.local
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
# Make rc.local executable
sudo chmod +x /etc/rc.local
# Enable the service
sudo systemctl enable rc-local
Conclusion
Service startup issues after reboot are common but easily preventable. Ensure critical services are enabled with systemctl enable, configure proper dependencies in unit files, and regularly check boot logs for errors. Creating custom systemd service files gives you full control over service startup behavior.
Related Articles
SSH Connection Problem: Can't Connect to Server
SSH connection troubleshooting guide: Path.net firewall rules, Default Block, creating a filter rule for SSH port, and step-by-step troubleshooting.
RDP Connection Problem: Can't Connect to Windows Server
RDP connection troubleshooting guide: Path.net firewall rules, Default Block, creating a filter rule for RDP port, and step-by-step troubleshooting.
How to Run an MTR Test and Submit Results to Support
Guide to running MTR network tests with WinMTR and Linux MTR, creating ICMP filter rules, and submitting results to support. Diagnose packet loss and latency.