path: root/content/articles/2019-email-server-tutorial.md

+++
title = "How to set up an email server in 2019"
+++

# How to set up an email server in 2019

This is a guide to set up your own email server in 2019.
This is a messy topic, and the available documentation is even messier,
so I compiled this guide in an attempt to make it all more accessible.

So why run your own mail server? There are many reasons,
ranging from privacy concerns about popular providers,
to simply wanting to learn about this stuff for fun.
If you've arrived here, then I assume you've already found a reason.

But beware that this is not for the faint of heart:
email is a horribly designed system, and even if you manage
to set up your own server, you may find that GMail flags all
your messages as spam, with little you can do about it.
This guide tries to implement all the security measures
that GMail seems to like, but *your mileage may vary*.



## How email works

The program on your device that you use to read and send emails
contains both a Mail User Agent (MUA) and a Mail Retrieval Agent (MRA),
although in practice the MRA is also often referred to as an MUA.
These mail agents can be in a dedicated app like [Thunderbird](https://www.thunderbird.net/en-US/)
or Microsoft Outlook, or it can simply be a web interface.

When you check your inbox, your MRA asks for any new emails
from the server, which then sends them via the [IMAP](https://en.wikipedia.org/wiki/Internet_Message_Access_Protocol) protocol
(we ignore [POP3](https://en.wikipedia.org/wiki/Post_Office_Protocol), since it is rather [outdated](https://www.pop2imap.com/)).
The server-side program here is simply called the IMAP server,
which watches over the server's copy of your inbox.

When you send an email, your MUA passes it along to the server's
Mail Submission Agent (MSA), which then immediately hands it over
to the Mail Transfer Agent (MTA) running on that same server.
In practice, the MSA's functionality is often regarded as part of the MTA.

The MTA then works out where your email needs to go,
and sends it over a protocol called [SMTP](https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol)
to the MTA of the server responsible for the recipient of your message.
In complex networks there may be multiple MTAs in a chain.

The final receiving MTA then hands over your email to the
Mail Delivery Agent (MDA) running on that same server.
The MDA checks whether it's spam, and then writes it to the server's storage,
which is being watched over by an IMAP server, thus completing the cycle.
Most MTA programs also have MDA features for simplicity,
although in this guide we'll be using our IMAP server's MDA helper.

Long story short, the server we want to set up
must at least be running both an MTA for you to send emails
and an IMAP server for you to check your inbox,
one of which should also include MDA functionality.



## Email security

This messy base email system is horribly insecure on its own,
so we still need to tack on some more messy features.
In this context, "security" refers to both spam and privacy protection.
This guide covers all techniques mentioned here.

Spam protection also means two things:
defending yourself against spammers, and
preventing that your emails' recipients' filters don't flag *you* as a spammer.
The former is optional, but the latter is not.

Let's start with how to come across as a trustworthy mail server:

The first feature is [Sender Policy Framework](https://en.wikipedia.org/wiki/Sender_Policy_Framework) (SPF),
which stops a spammer from filling in your email address
in the "sender" field of their message.
You do this by publishing a policy in a DNS record for how your domain
should relate to the IP address of your email server.
A filter enforcing SPF will use that to check that your server
is authorized to send messages with your email address.

Then there's [DomainKeys Identified Mail](https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail) (DKIM),
which is a more comprehensive form of SPF.
It adds a cryptographic signature to all your emails,
which the receiver's spam filter will check against
the contents and a public key you need to publish in a DNS record.
You should implement *both* SPF and DKIM, despite their overlap.

Lastly, we have [Domain-based Message Authentication, Reporting and Conformance](https://en.wikipedia.org/wiki/DMARC).
which uses another DNS record to specify whether you're using SPF and/or DKIM,
and whether super- and subdomains should be regarded as valid for authentication.
It also gives a contact address to which the receiver should send failure reports.
I highly recommend implementing this too.

To defend yourself against spammers your server should enforce
the SPF, DKIM and DMARC policies of senders, in addition to some other checks.
All of that will be handled by your spam filter.

There's one final problem to bring up:
your emails will be sent across the Internet unencrypted.

For client-server communication, this is effectively solved
by IMAPS and SMTPS, which wrap IMAP and SMTP in [TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security).
To be clear, SMTPS only encrypts email submission by an MUA to an MTA (MSA).

For server-server relaying of messages over SMTP, we have
STARTTLS, which is a system for opportunistic TLS, meaning that
communication is only encrypted if both parties agree
after a short unencrypted discussion.
That last part is vulnerable to [MitM](https://en.wikipedia.org/wiki/Man-in-the-middle_attack) attacks,
where anyone along the path of the email servers' discussion
can alter the exchange to block the use of encryption,
which sometimes actually [happens](https://www.eff.org/deeplinks/2014/11/starttls-downgrade-attacks) in practice.

The only way to make sure that STARTTLS is used
is to refuse any emails unless the TLS handshake succeeds.
That's a risky approach, because fewer mail servers support
STARTTLS than you might expect or hope: I've had airline
booking confirmations, full of personal details,
and made with multi-billion-dollar companies,
sent across the Internet without any protection.
I'll show you how, but do it at your own risk.



## Preparation

Are you still here? Good, let's get started. Here's what you'll need:
* SSH root access to a 24/7-accessible server running Linux or *BSD
* an associated domain name for which you can create or edit DNS records
* a TLS certificate for encryption

The TLS certificate is technically optional,
but you *really* should use one; it's 2019 for goodness' sake.
You may get away with using a self-signed certificate,
but instead I highly recommend getting a proper one from
[Let's Encrypt](https://letsencrypt.org/) (it's free!).

In the rest of this guide I'll assume that
you have your public TLS certificate at `/etc/ssl/certs/example.com.pem`,
and your private encryption key at `/etc/ssl/private/example.com.pem`.
You should also create a file with [DH parameters](https://weakdh.org/sysadmin.html)
using the `openssl` tool, included in OpenSSL and LibreSSL:
```sh
openssl dhparam -out /etc/ssl/dhparam.pem 4096
```
This will take a while, because it's looking for sources
of randomness in your (very non-random) computer.
Just keep doing stuff with it running in the background.
The resulting file is public and expendable.



There exist many options for most of the software roles described earlier.
This guide covers multiple options for most of these,
given here from most to least recommended by me:
* IMAP server + MDA: [Dovecot](https://www.dovecot.org/).
* MTA: [OpenSMTPD](https://opensmtpd.org/), [Postfix](http://www.postfix.org/), or [Exim](https://exim.org/).
* Spam filtering: [Rspamd](https://www.rspamd.com/), or [SpamAssassin](https://spamassassin.apache.org/).
* DKIM signing: [OpenDKIM](http://www.opendkim.org/), or [DKIMproxy](http://dkimproxy.sourceforge.net/).

The MTA feeds arriving emails through the spam filter
before giving them to the MDA, which puts them in an inbox folder.
Likewise, it feeds outgoing emails through the DKIM signer.
The IMAP server then delivers your inbox contents to your MUA.

All these MTAs include an MDA, but nevertheless
I recommend using Dovecot's one because it gives the most control.
This is also important because of how spam filtering works:
according to the email specification, an MTA assumes full responsibility
over any emails it accepts for delivery or relaying,
and promises that it will do its best to get it to its destination.

The takeaway from this is that an MTA must not magically
swallow any emails it doesn't like, so if it uses a spam filter,
that filter may tag emails as "spam" or "not spam",
but the MTA is not allowed to act based on that verdict.
The MDA does not have this restriction, so only there
can we make the decision what to do with spam,
and Dovecot's MDA is by far the best equipped for that.

When setting up an email server, you have the choice between
attaching email addresses to system users or virtual users.
With system users, if an email arrives for `foo@example.com`,
then MTA, MDA and IMAP server will all expect that there
exists a `foo` Unix user on the server to deliver it to.
With virtual users, this is not the case.

This guide only covers virtual users, which I recommend
because it's more flexible, and unless you're using some
ancient mainframe there is no advantage to using system users.

Therefore, create a global email user and group,
which are traditionally both called `vmail`.
Do *not* pass the "system user/group" flags,
because that will get you into trouble with Dovecot:
```sh
# GNU CoreUtils:
groupadd vmail
useradd -M -g vmail vmail
# BusyBox:
addgroup vmail
adduser -HD -G vmail vmail
# *BSD:
no clue, you can work it out
```




## DNS records

First things first, you should create an RSA keypair for DKIM,
which you do using the `openssl` utility again.
The minimum size is 1024 bits, but I recommend using 2048 bits.
Bigger is better, but because DNS is involved we can't stretch it
to 4096 bits without risking causing discomfort to [some](https://serverfault.com/questions/747185/dkim-can-i-use-a-rsa-key-larger-than-2048bit-i-e-4096) servers.
```sh
$ openssl genrsa -out /TODO/private.key 2048
$ openssl rsa -in /path/private.key -out /path/public.key
```

Now we must set up all the necessary DNS records.
It may take up to a day for these to propagate over the Internet,
so I recommend doing this section now and the rest tomorrow.

Now, this might sound obvious, but you must have an A and/or AAAA record
to associate your domain `example.com` with the server's IP address.
For email you also must have [reverse DNS](https://en.wikipedia.org/wiki/Reverse_DNS_lookup) set up correctly.
If you're renting your server remotely, you can often do this from
the provider's configuration tool, or you can just create a PTR-type DNS record.

To inform the rest of the Internet that your server is an email server,
create an MX (Mail eXchanger) DNS record for your domain.
Note the dot at the end of the domain name:
```
MX 42 example.com.
```
When a message is sent to an email address ending in `@example.com`,
the sending server will query DNS for any MX records for `example.com`.
There it will find a domain name (in this case `example.com` again),
for which it will look up the IP address, and send the email to.
The domain name in the record must *not* have an associated CNAME record;
it must be directly translatable to an IP address.

You may have multiple MX records, which must contain different
domain names, each with an associated preference number (`42` here).
The sending server will use all MX records with lower numbers first,
and if those servers are all unavailable, it will try a higher number.
If you have multiple mail servers (which is good for availability),
you can therefore declare those as follows:
```
MX 13 mx1.example.com.
MX 42 mx2.example.com.
```
Here, a server sending an email to your domain `example.com`
will try to send it to the IP address of `mx1.example.com` first,
and if that fails, it will move on to the higher numbered `mx2.example.com`.
If both `mx1` and `mx2` have the same number, then the sender
will randomly choose one, which is useful for load balancing.

Next, publish an SPF policy record so you don't look like a spammer.
This is a TXT (*not* SPF!) record with the following contents:
```
TXT "v=spf1 mx -all"
```
Everything after the version `v=spf1` is a list of *mechanisms*
for a spam filter to try out in the given order.
The `-all` at the end says to reject your email
if all of the previous mechanisms fail verification.
I recommend only using the `mx` mechanism, which tells the verifier
to look at the A/AAAA addresses of the domains in your MX records.
This allows you to add, remove, or change your servers
without needing to update this record.

Your DKIM policy must also be published in a TXT record
as follows, where `<public.key>` is the public RSA key `MI...AB`
stored in `/TODO/public.key`, with the newlines removed:
```
TXT "v=DKIM1; t=s; h=sha256; p=<public.key>"
```
Here, `v=DKIM1` is the version and must be the first tag.
The flag `t=s` enables strict mode, as recommended by the [DKIM spec](https://tools.ietf.org/html/rfc6376#page-29),
meaning emails sent from subdomains immediately fail verification.
The optional tag `h=sha256` blocks the use of the old SHA1 algorithm.

Finally, there is the DMARC policy, which, unlike the others,
will need to be updated at the end of this guide.
For now, create another TXT record with these contents,
where `<admin>` is an email address of your choosing,
which may belong to the domain you're setting up for:
```
TXT "v=DMARC1; aspf=s; adkim=s; p=none; sp=reject; fo=1; ruf=mailto:<admin>"
```
The version tag `v=DMARC1` is required and must come first.
Next, `aspf=s` and `adkim=s` enable strict mode for both SPF and DKIM,
which once again blocks subdomains from passing the test.
Then `p=none` and `sp=reject` control what to do to failed messages
coming from the main domain and subdomains, respectively.
Unsurprisingly, `reject` means that delivery should be refused,
`none` asks to let it through anyway, and `quarantine` tells
the filter to take a closer look at the email or to put it in a spam folder.
Finally, `fo=1` asks the filter to create a forensic report
if any type of verification fails, and `ruf=` gives an address to send it to.



## IMAP server: Dovecot

[Dovecot](https://www.dovecot.org/) is a very popular IMAP server,
focused on being lightweight, simple, and secure.
Its documentation, while extensive and accurate,
is an absolute mess: the information you need is
usually either spread across multiple pages,
or buried somewhere two internal links deep.

If you installed Dovecot via your package manager,
you'll probably have a lot of configuration files
in the `/etc/dovecot` directory and its subdirectories.
I want you to delete all of them. Yes, `rm -rf` that crap.

Dovecot is simple to configure, and doesn't care where
you put its settings, so having all that chaos in `/etc/dovecot`
just makes things unnecessarily confusing.

Create a new blank configuration file `/etc/dovecot/dovecot.conf`,
and start by filling in the details about your TLS certificate,
and making clear that unencrypted connections are unacceptable:
```sh
ssl = required
ssl_cert = </etc/ssl/certs/example.com.pem
ssl_key  = </etc/ssl/private/example.com.key
ssl_dh   = </etc/ssl/dhparam.pem

ssl_prefer_server_ciphers = yes

disable_plaintext_auth = yes
```
The final `disable_plaintext_auth` option tells Dovecot
to reject any passwords that were sent unencrypted.
This means it must be [hashed](https://en.wikipedia.org/wiki/Cryptographic_hash_function)
or sent over a TLS connection, ideally both.

Next, tell Dovecot which protocols to use
and where to expect them as follows:
```sh
protocols = lmtp imap

service lmtp {
	inet_listener lmtp {
		address = 127.0.0.1
		port = 39999
	}
	user = vmail
}
```
LMTP is the Local Mail Transport Protocol, which is basically SMTP
but intended for exchange within a single server or cluster.
When an email is received, Dovecot will start a child process
under the `vmail` Unix user to deliver the message to its recipient.

Since we set `ssl = required` earlier, MUAs (MRAs) will only get their mail
if the STARTTLS handshake was successful during the IMAP exchange,
or if they connect via IMAPS to force the use of encryption.
If you want to, you can set Dovecot to only accept IMAPS
by neutralizing the STARTTLS IMAP login listener:
```sh
service imap-login {
	inet_listener imap {
		port = 0
	}
}
```

Then we need to inform Dovecot which email users it should handle,
and what to do with their messages. Create a file `/etc/dovecot/users` for this,
which describes users in the same format as `/etc/passwd`:
```
user:password:uid:gid::homedir
```
You can leave `uid` and `gid` fields blank, since we're
using our `vmail` virtual mail user for all accounts.
You can even omit `homedir` if all users' directories
follow the same pattern, for example `/var/vmail/<user>/`.
The `user` name may contain the ending `@example.com`, but doesn't need to.
Create the password hash to put in the `password` field as follows:
```sh
# If your server is beefy and has lots of RAM:
doveadm pw -s ARGON2ID-CRYPT
# If you're using an old potato:
doveadm pw -s SHA512-CRYPT
```
After you've entered your password, simply copy-paste the entire
hash string outputted by the program into the `password` field.

Dovecot needs a file describing user accounts on two occasions:
* To check whether a user logging into the IMAP server
  is valid and has given the right password.
  This is handled by the `passdb` block(s) in the configuration.
* To know which email addresses the server is responsible for.
  This is given by the `userdb` block(s) in the configuration.
These functions aren't necessarily fulfilled by the same file:
you can map multiple email addresses to one user acccount,
or multiple user accounts to one email address.
For simplicity, though, we'll use the same file:
```sh
passdb {
	driver = passwd-file
	args = scheme=ARGON2ID-CRYPT username_format=%n /etc/dovecot/users
	#args = scheme=SHA512-CRYPT username_format=%n /etc/dovecot/users
}

userdb {
	driver = passwd-file
	args = username_format=%n /etc/dovecot/users
	# Use vmail for all users, and if all users' inboxes have the same form:
	override_fields = uid=vmail gid=vmail home=/var/vmail/%n
}
```
The `driver` option sets the kind of table Dovecot should expect.
We tell it to use a file in the `passwd`-like format described above,
but other common choices include SQL or the actual `/etc/passwd` file.
The options available in `args` depend on the chosen `driver`.

The password hashing algorithm is given by `scheme`, while
`username_format` desribes the format of user names in the table:
if Dovecot receives an email for `someone@example.com`,
then for `%n` it will expect `someone` in `/etc/dovecot/users`,
but if you use `%u` it will want the full email address instead.

In the `userdb` block we force the use of `vmail:vmail`
for all users, and tell Dovecot to put their mail in `/var/vmail/%n`,
where `%n` means the same as before. You can also use `%u`
for the full address again, or `%d` for the domain name.

Optionally, you can create a catch-all inbox that will accept
all emails sent to your domain that don't match anyone in `/etc/dovecot/users`.
Just add this second `userdb` block after the first:
```sh
userdb {
	driver = static
	args = uid=vmail gid=vmail home=/var/vmail/catchall allow_all_users=yes
}
```
The `static` driver means there is no table file:
all configuration is directly within this `userdb` block.
If we didn't specify `allow_all_users=yes`, then Dovecot
would check whether users existed using the `passdb` table.

Finally, now that we've told Dovecot where the virtual mail users' `home`
folders are, we need to specify how to store the emails within them:
```sh
mail_location = maildir:~:LAYOUT=fs
```
The two standard mailbox formats to choose from are `maildir` and `mbox`.
I highly recommend `maildir`; it's faster and more flexible than `mbox`
because it uses folders, whereas the older `mbox` uses a single database file.
The `~` tells Dovecot to directly put all mails in the account's home folder,
and `LAYOUT=fs` to user filesystem directories for you email folders. TODO?





## MTA: Postfix


## MTA: Exim



## MTA: OpenSMTPD

OpenSMTPD is an MTA by the [OpenBSD](https://www.openbsd.org/) project,
who are known for their focus on security and minimalism.
I really like OpenSMTPD's flexible configuration format,
and its simple and intuitive way of passing around emails.

It doesn't implement milter or any other filtering interface,
so DKIM signing and spam checking must be shoehorned in.
As such, only DKIMproxy can be used for DKIM signing.

To begin, write your email domain on a single line in `/etc/smtpd/mailname`:
```
example.com
```
Next, you must fill in the `/etc/smtpd/aliases` file
which maps recipient addresses to system users:
```
mailuser@example.com    systemuser
```
You can create a wildcard (catch-all) address for your domain
by omitting `mailuser`, such that any unknown recipients end up there.
You may also specify multiple system users as `systemuser1,systemuser2`.

The first part of the actual config `/etc/smtpd/smtpd.conf` is as follows,
with DKIMproxy listening on port 19999 and outputting to 20000:
```sh
# Register TLS certificate to encrypt email transport:
pki "example.com" certificate "/etc/ssl/certs/example.com.pem"
pki "example.com" key "/etc/ssl/private/example.com.key"
pki "example.com" dhe auto

table users "/etc/smtpd/aliases"

# Relay emails submitted by client to DKIM signer on port 19999:
listen on eth0 port 465 smtps       pki "example.com" tag SEND
listen on eth0 port 587 tls-require pki "example.com" tag SEND
accept tagged SEND from any for any relay via smtp://127.0.0.1:19999

# Relay outgoing DKIM-signed emails to their intended destination:
listen on lo port 20000 tag DKIM
accept tagged DKIM from local for any relay
```
The lines starting with `pki` enable [DHE](https://en.wikipedia.org/wiki/Diffie%E2%80%93Hellman_key_exchange)
and tell where the public certificate and private key are for the domain.
We listen for client emails on both ports 465 (SMTPS) and 587 (SMTP),
although for the latter we refuse the submission if STARTTLS fails.

If no spam filtering is desired, let OpenSMTPD deliver the email
by appending the following to the configuration file:
```sh
# Receive incoming emails and write them to the inbox:
listen on eth0 port 25 tls pki "example.com"
accept from any for domain "example.com" virtual <users> deliver to lmtp rcpt-to as vmail
```
Here, we look up which system user corresponds to the recipient in `<users>`,
and deliver it to their maildir `mail` in their home folder.


### With SpamAssassin UNTESTED

For SpamAssassin, we'll need to use [spampd](https://github.com/mpaperno/spampd) as an SMTP shim,
which we set to listen on port 29999 and relay to port 30000.
Add the `--nodetach` option in an init script or systemd service,
but leave it out when starting `spampd` by hand for testing:
```sh
$ spampd --host=127.0.0.1:29999 --relayhost=127.0.0.1:30000 [--nodetach]
```
The rest of `smtpd`'s configuration is then as follows:
```sh
# Receive incoming emails and send them to spampd on port 29999:
listen on eth0 port 25 tls pki "example.com" tag SCAN
accept tagged SCAN from any for domain "example.com" relay via smtp://127.0.0.1:29999

# Receive filtered incoming emails and write them to the inbox:
listen on lo port 30000 tag DONE
accept tagged DONE from local for any virtual <users> deliver to maildir "~/mail"
```
TODO

### With Rspamd









## DKIM signer: DKIMproxy

[DKIMproxy](http://dkimproxy.sourceforge.net/) is a mature and simple SMTP proxy
that signs or verifies all emails passed through it
using the Perl Mail::DKIM module.

In this guide we only use its signing feature,
which is controlled by `/etc/dkimproxy/dkimproxy_out.conf`.
This configuration file is pretty self-explanatory:
```sh
listen    127.0.0.1:19999
relay     127.0.0.1:20000

domain    example.com
selector  mail

keyfile   /etc/dkim/private.key
signature dkim(c=relaxed/simple,a=rsa-sha256)
```
This last line clearly tells it to create a DKIM signature,
because it also supports DKIM's predecessor, Yahoo! DomainKey.
The tag `a=rsa-sha256` specifies the signing algorithm,
while `c=relaxed/simple` gives the [canonicalization](https://tools.ietf.org/html/rfc6376#section-3.4) method.

Canonicalization is a way to prevent the signature from becoming invalid
as the email might get reformatted by the MTAs on its journey.
Our choice `relaxed/simple` allows the header to be reformatted (`relaxed`),
but not the actual body of the message (`simple`).




## DKIM signer: OpenDKIM

[OpenDKIM](http://www.opendkim.org/) is a milter program
that can both create and verify DKIM signatures.




## Spam filter: Rspamd



## Spam filter: SpamAssassin