Mail Server Setup: Postfix and Dovecot

Setting up your own mail server gives you full control over your email communication. This guide covers Postfix MTA (Mail Transfer Agent), Dovecot IMAP/POP3 server, SSL/TLS configuration, virtual domain support, and Docker Mail Server setup.

Mail server management is complex and requires ongoing maintenance. IP reputation, spam filtering, and security updates must be regularly monitored.

Prerequisites

DNS Records

Create the following DNS records before setting up your mail server:

# A record - mail server IP
mail.example.com    A       185.x.x.x

# MX record - mail routing
example.com         MX  10  mail.example.com

# SPF record
example.com         TXT     "v=spf1 mx ip4:185.x.x.x -all"

# Reverse DNS (PTR) record
185.x.x.x           PTR     mail.example.com

PTR (Reverse DNS) record can be set from the REXE customer panel or via support request. Emails sent without a PTR record may be marked as spam.

Hostname Setup

hljs bash

sudo hostnamectl set-hostname mail.example.com
echo "185.x.x.x mail.example.com mail" | sudo tee -a /etc/hosts
hostname -f

Postfix Installation (MTA)

Postfix is a reliable and high-performance Mail Transfer Agent.

Installation

hljs bash

sudo apt update
sudo apt install postfix postfix-mysql -y
# Select "Internet Site" during installation
# System mail name: example.com

Basic Configuration

hljs ini

# /etc/postfix/main.cf
myhostname = mail.example.com
mydomain = example.com
myorigin = $mydomain
mydestination = $myhostname, localhost.$mydomain, localhost

inet_interfaces = all
inet_protocols = ipv4
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128

home_mailbox = Maildir/
mailbox_size_limit = 0
recipient_delimiter = +

# TLS settings
smtpd_tls_cert_file = /etc/letsencrypt/live/mail.example.com/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/mail.example.com/privkey.pem
smtpd_use_tls = yes
smtpd_tls_auth_only = yes

# SASL authentication
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes

# Restrictions
smtpd_recipient_restrictions =
    permit_sasl_authenticated,
    permit_mynetworks,
    reject_unauth_destination,
    reject_invalid_hostname,
    reject_non_fqdn_hostname,
    reject_non_fqdn_sender,
    reject_unknown_sender_domain,
    reject_rbl_client zen.spamhaus.org

message_size_limit = 52428800

Submission Port (587)

hljs ini

# /etc/postfix/master.cf
submission inet n       -       y       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_tls_auth_only=yes
  -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject

Dovecot Installation (IMAP/POP3)

hljs bash

sudo apt install dovecot-core dovecot-imapd dovecot-pop3d dovecot-lmtpd -y

Configuration

hljs bash

# /etc/dovecot/dovecot.conf
protocols = imap pop3 lmtp
listen = *, ::

# /etc/dovecot/conf.d/10-mail.conf
mail_location = maildir:~/Maildir

# /etc/dovecot/conf.d/10-auth.conf
disable_plaintext_auth = yes
auth_mechanisms = plain login

# /etc/dovecot/conf.d/10-ssl.conf
ssl = required
ssl_cert = </etc/letsencrypt/live/mail.example.com/fullchain.pem
ssl_key = </etc/letsencrypt/live/mail.example.com/privkey.pem
ssl_min_protocol = TLSv1.2

# /etc/dovecot/conf.d/10-master.conf
service auth {
  unix_listener /var/spool/postfix/private/auth {
    mode = 0666
    user = postfix
    group = postfix
  }
}

SSL/TLS Certificate

hljs bash

sudo apt install certbot -y
sudo certbot certonly --standalone -d mail.example.com

# Auto-renewal
sudo crontab -e
# Add: 0 3 * * * certbot renew --post-hook "systemctl restart postfix dovecot"

Docker Mail Server (Alternative)

Docker Mail Server provides all components in a single container.

hljs yaml

# docker-compose.yml
version: '3.8'
services:
  mailserver:
    image: ghcr.io/docker-mailserver/docker-mailserver:latest
    container_name: mailserver
    hostname: mail.example.com
    ports:
      - "25:25"
      - "465:465"
      - "587:587"
      - "993:993"
    volumes:
      - ./docker-data/dms/mail-data/:/var/mail/
      - ./docker-data/dms/mail-state/:/var/mail-state/
      - ./docker-data/dms/mail-logs/:/var/log/mail/
      - ./docker-data/dms/config/:/tmp/docker-mailserver/
      - /etc/letsencrypt:/etc/letsencrypt:ro
    environment:
      - ENABLE_SPAMASSASSIN=1
      - ENABLE_CLAMAV=1
      - ENABLE_FAIL2BAN=1
      - SSL_TYPE=letsencrypt
    restart: always

hljs bash

# Add user
docker exec -it mailserver setup email add user@example.com

# Generate DKIM key
docker exec -it mailserver setup config dkim

# Start
docker compose up -d

Spam Filtering

hljs bash

sudo apt install spamassassin spamc -y
sudo systemctl enable spamassassin
sudo systemctl start spamassassin

Testing and Verification

hljs bash

sudo systemctl status postfix
sudo systemctl status dovecot
sudo ss -tlnp | grep -E '25|587|993|995'
echo "Test mail" | mail -s "Test" user@example.com
sudo tail -f /var/log/mail.log
openssl s_client -connect mail.example.com:587 -starttls smtp
openssl s_client -connect mail.example.com:993

After setting up your mail server, check your spam score at mail-tester.com. Ensure SPF, DKIM, and DMARC records are correctly configured to achieve a 10/10 score.

Conclusion

Setting up your own mail server provides full control but requires regular maintenance. The Postfix + Dovecot combination offers a reliable solution, while Docker Mail Server provides easier management. Make sure to complete SPF, DKIM, and DMARC configurations.

Mail Server Setup: Postfix and Dovecot

Table of Contents