Add incomplete email server tutorial

1 files changed, 612 insertions, 0 deletions

diff --git a/content/articles/2019-email-server-tutorial.md b/content/articles/2019-email-server-tutorial.md
new file mode 100644
index 0000000..cca8379
--- /dev/null
+++ b/content/articles/2019-email-server-tutorial.md
@@ -0,0 +1,612 @@
++++
+title = "How to set up an email server in 2019"
++++
+
+# How to set up an email server in 2019
+
+This is a guide to set up your own email server in 2019.
+This is a messy topic, and the available documentation is even messier,
+so I compiled this guide in an attempt to make it all more accessible.
+
+So why run your own mail server? There are many reasons,
+ranging from privacy concerns about popular providers,
+to simply wanting to learn about this stuff for fun.
+If you've arrived here, then I assume you've already found a reason.
+
+But beware that this is not for the faint of heart:
+email is a horribly designed system, and even if you manage
+to set up your own server, you may find that GMail flags all
+your messages as spam, with little you can do about it.
+This guide tries to implement all the security measures
+that GMail seems to like, but *your mileage may vary*.
+
+
+
+## How email works
+
+The program on your device that you use to read and send emails
+contains both a Mail User Agent (MUA) and a Mail Retrieval Agent (MRA),
+although in practice the MRA is also often referred to as an MUA.
+These mail agents can be in a dedicated app like [Thunderbird](https://www.thunderbird.net/en-US/)
+or Microsoft Outlook, or it can simply be a web interface.
+
+When you check your inbox, your MRA asks for any new emails
+from the server, which then sends them via the [IMAP](https://en.wikipedia.org/wiki/Internet_Message_Access_Protocol) protocol
+(we ignore [POP3](https://en.wikipedia.org/wiki/Post_Office_Protocol), since it is rather [outdated](https://www.pop2imap.com/)).
+The server-side program here is simply called the IMAP server,
+which watches over the server's copy of your inbox.
+
+When you send an email, your MUA passes it along to the server's
+Mail Submission Agent (MSA), which then immediately hands it over
+to the Mail Transfer Agent (MTA) running on that same server.
+In practice, the MSA's functionality is often regarded as part of the MTA.
+
+The MTA then works out where your email needs to go,
+and sends it over a protocol called [SMTP](https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol)
+to the MTA of the server responsible for the recipient of your message.
+In complex networks there may be multiple MTAs in a chain.
+
+The final receiving MTA then hands over your email to the
+Mail Delivery Agent (MDA) running on that same server.
+The MDA checks whether it's spam, and then writes it to the server's storage,
+which is being watched over by an IMAP server, thus completing the cycle.
+Most MTA programs also have MDA features for simplicity,
+although in this guide we'll be using our IMAP server's MDA helper.
+
+Long story short, the server we want to set up
+must at least be running both an MTA for you to send emails
+and an IMAP server for you to check your inbox,
+one of which should also include MDA functionality.
+
+
+
+## Email security
+
+This messy base email system is horribly insecure on its own,
+so we still need to tack on some more messy features.
+In this context, "security" refers to both spam and privacy protection.
+This guide covers all techniques mentioned here.
+
+Spam protection also means two things:
+defending yourself against spammers, and
+preventing that your emails' recipients' filters don't flag *you* as a spammer.
+The former is optional, but the latter is not.
+
+Let's start with how to come across as a trustworthy mail server:
+
+The first feature is [Sender Policy Framework](https://en.wikipedia.org/wiki/Sender_Policy_Framework) (SPF),
+which stops a spammer from filling in your email address
+in the "sender" field of their message.
+You do this by publishing a policy in a DNS record for how your domain
+should relate to the IP address of your email server.
+A filter enforcing SPF will use that to check that your server
+is authorized to send messages with your email address.
+
+Then there's [DomainKeys Identified Mail](https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail) (DKIM),
+which is a more comprehensive form of SPF.
+It adds a cryptographic signature to all your emails,
+which the receiver's spam filter will check against
+the contents and a public key you need to publish in a DNS record.
+You should implement *both* SPF and DKIM, despite their overlap.
+
+Lastly, we have [Domain-based Message Authentication, Reporting and Conformance](https://en.wikipedia.org/wiki/DMARC).
+which uses another DNS record to specify whether you're using SPF and/or DKIM,
+and whether super- and subdomains should be regarded as valid for authentication.
+It also gives a contact address to which the receiver should send failure reports.
+I highly recommend implementing this too.
+
+To defend yourself against spammers your server should enforce
+the SPF, DKIM and DMARC policies of senders, in addition to some other checks.
+All of that will be handled by your spam filter.
+
+There's one final problem to bring up:
+your emails will be sent across the Internet unencrypted.
+
+For client-server communication, this is effectively solved
+by IMAPS and SMTPS, which wrap IMAP and SMTP in [TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security).
+To be clear, SMTPS only encrypts email submission by an MUA to an MTA (MSA).
+
+For server-server relaying of messages over SMTP, we have
+STARTTLS, which is a system for opportunistic TLS, meaning that
+communication is only encrypted if both parties agree
+after a short unencrypted discussion.
+That last part is vulnerable to [MitM](https://en.wikipedia.org/wiki/Man-in-the-middle_attack) attacks,
+where anyone along the path of the email servers' discussion
+can alter the exchange to block the use of encryption,
+which sometimes actually [happens](https://www.eff.org/deeplinks/2014/11/starttls-downgrade-attacks) in practice.
+
+The only way to make sure that STARTTLS is used
+is to refuse any emails unless the TLS handshake succeeds.
+That's a risky approach, because fewer mail servers support
+STARTTLS than you might expect or hope: I've had airline
+booking confirmations, full of personal details,
+and made with multi-billion-dollar companies,
+sent across the Internet without any protection.
+I'll show you how, but do it at your own risk.
+
+
+
+## Preparation
+
+Are you still here? Good, let's get started. Here's what you'll need:
+* SSH root access to a 24/7-accessible server running Linux or *BSD
+* an associated domain name for which you can create or edit DNS records
+* a TLS certificate for encryption
+
+The TLS certificate is technically optional,
+but you *really* should use one; it's 2019 for goodness' sake.
+You may get away with using a self-signed certificate,
+but instead I highly recommend getting a proper one from
+[Let's Encrypt](https://letsencrypt.org/) (it's free!).
+
+In the rest of this guide I'll assume that
+you have your public TLS certificate at `/etc/ssl/certs/example.com.pem`,
+and your private encryption key at `/etc/ssl/private/example.com.pem`.
+You should also create a file with [DH parameters](https://weakdh.org/sysadmin.html)
+using the `openssl` tool, included in OpenSSL and LibreSSL:
+```sh
+openssl dhparam -out /etc/ssl/dhparam.pem 4096
+```
+This will take a while, because it's looking for sources
+of randomness in your (very non-random) computer.
+Just keep doing stuff with it running in the background.
+The resulting file is public and expendable.
+
+
+
+There exist many options for most of the software roles described earlier.
+This guide covers multiple options for most of these,
+given here from most to least recommended by me:
+* IMAP server + MDA: [Dovecot](https://www.dovecot.org/).
+* MTA: [OpenSMTPD](https://opensmtpd.org/), [Postfix](http://www.postfix.org/), or [Exim](https://exim.org/).
+* Spam filtering: [Rspamd](https://www.rspamd.com/), or [SpamAssassin](https://spamassassin.apache.org/).
+* DKIM signing: [OpenDKIM](http://www.opendkim.org/), or [DKIMproxy](http://dkimproxy.sourceforge.net/).
+
+The MTA feeds arriving emails through the spam filter
+before giving them to the MDA, which puts them in an inbox folder.
+Likewise, it feeds outgoing emails through the DKIM signer.
+The IMAP server then delivers your inbox contents to your MUA.
+
+All these MTAs include an MDA, but nevertheless
+I recommend using Dovecot's one because it gives the most control.
+This is also important because of how spam filtering works:
+according to the email specification, an MTA assumes full responsibility
+over any emails it accepts for delivery or relaying,
+and promises that it will do its best to get it to its destination.
+
+The takeaway from this is that an MTA must not magically
+swallow any emails it doesn't like, so if it uses a spam filter,
+that filter may tag emails as "spam" or "not spam",
+but the MTA is not allowed to act based on that verdict.
+The MDA does not have this restriction, so only there
+can we make the decision what to do with spam,
+and Dovecot's MDA is by far the best equipped for that.
+
+When setting up an email server, you have the choice between
+attaching email addresses to system users or virtual users.
+With system users, if an email arrives for `foo@example.com`,
+then MTA, MDA and IMAP server will all expect that there
+exists a `foo` Unix user on the server to deliver it to.
+With virtual users, this is not the case.
+
+This guide only covers virtual users, which I recommend
+because it's more flexible, and unless you're using some
+ancient mainframe there is no advantage to using system users.
+
+Therefore, create a global email user and group,
+which are traditionally both called `vmail`.
+Do *not* pass the "system user/group" flags,
+because that will get you into trouble with Dovecot:
+```sh
+# GNU CoreUtils:
+groupadd vmail
+useradd -M -g vmail vmail
+# BusyBox:
+addgroup vmail
+adduser -HD -G vmail vmail
+# *BSD:
+no clue, you can work it out
+```
+
+
+
+
+## DNS records
+
+First things first, you should create an RSA keypair for DKIM,
+which you do using the `openssl` utility again.
+The minimum size is 1024 bits, but I recommend using 2048 bits.
+Bigger is better, but because DNS is involved we can't stretch it
+to 4096 bits without risking causing discomfort to [some](https://serverfault.com/questions/747185/dkim-can-i-use-a-rsa-key-larger-than-2048bit-i-e-4096) servers.
+```sh
+$ openssl genrsa -out /TODO/private.key 2048
+$ openssl rsa -in /path/private.key -out /path/public.key
+```
+
+Now we must set up all the necessary DNS records.
+It may take up to a day for these to propagate over the Internet,
+so I recommend doing this section now and the rest tomorrow.
+
+Now, this might sound obvious, but you must have an A and/or AAAA record
+to associate your domain `example.com` with the server's IP address.
+For email you also must have [reverse DNS](https://en.wikipedia.org/wiki/Reverse_DNS_lookup) set up correctly.
+If you're renting your server remotely, you can often do this from
+the provider's configuration tool, or you can just create a PTR-type DNS record.
+
+To inform the rest of the Internet that your server is an email server,
+create an MX (Mail eXchanger) DNS record for your domain.
+Note the dot at the end of the domain name:
+```
+MX 42 example.com.
+```
+When a message is sent to an email address ending in `@example.com`,
+the sending server will query DNS for any MX records for `example.com`.
+There it will find a domain name (in this case `example.com` again),
+for which it will look up the IP address, and send the email to.
+The domain name in the record must *not* have an associated CNAME record;
+it must be directly translatable to an IP address.
+
+You may have multiple MX records, which must contain different
+domain names, each with an associated preference number (`42` here).
+The sending server will use all MX records with lower numbers first,
+and if those servers are all unavailable, it will try a higher number.
+If you have multiple mail servers (which is good for availability),
+you can therefore declare those as follows:
+```
+MX 13 mx1.example.com.
+MX 42 mx2.example.com.
+```
+Here, a server sending an email to your domain `example.com`
+will try to send it to the IP address of `mx1.example.com` first,
+and if that fails, it will move on to the higher numbered `mx2.example.com`.
+If both `mx1` and `mx2` have the same number, then the sender
+will randomly choose one, which is useful for load balancing.
+
+Next, publish an SPF policy record so you don't look like a spammer.
+This is a TXT (*not* SPF!) record with the following contents:
+```
+TXT "v=spf1 mx -all"
+```
+Everything after the version `v=spf1` is a list of *mechanisms*
+for a spam filter to try out in the given order.
+The `-all` at the end says to reject your email
+if all of the previous mechanisms fail verification.
+I recommend only using the `mx` mechanism, which tells the verifier
+to look at the A/AAAA addresses of the domains in your MX records.
+This allows you to add, remove, or change your servers
+without needing to update this record.
+
+Your DKIM policy must also be published in a TXT record
+as follows, where `<public.key>` is the public RSA key `MI...AB`
+stored in `/TODO/public.key`, with the newlines removed:
+```
+TXT "v=DKIM1; t=s; h=sha256; p=<public.key>"
+```
+Here, `v=DKIM1` is the version and must be the first tag.
+The flag `t=s` enables strict mode, as recommended by the [DKIM spec](https://tools.ietf.org/html/rfc6376#page-29),
+meaning emails sent from subdomains immediately fail verification.
+The optional tag `h=sha256` blocks the use of the old SHA1 algorithm.
+
+Finally, there is the DMARC policy, which, unlike the others,
+will need to be updated at the end of this guide.
+For now, create another TXT record with these contents,
+where `<admin>` is an email address of your choosing,
+which may belong to the domain you're setting up for:
+```
+TXT "v=DMARC1; aspf=s; adkim=s; p=none; sp=reject; fo=1; ruf=mailto:<admin>"
+```
+The version tag `v=DMARC1` is required and must come first.
+Next, `aspf=s` and `adkim=s` enable strict mode for both SPF and DKIM,
+which once again blocks subdomains from passing the test.
+Then `p=none` and `sp=reject` control what to do to failed messages
+coming from the main domain and subdomains, respectively.
+Unsurprisingly, `reject` means that delivery should be refused,
+`none` asks to let it through anyway, and `quarantine` tells
+the filter to take a closer look at the email or to put it in a spam folder.
+Finally, `fo=1` asks the filter to create a forensic report
+if any type of verification fails, and `ruf=` gives an address to send it to.
+
+
+
+## IMAP server: Dovecot
+
+[Dovecot](https://www.dovecot.org/) is a very popular IMAP server,
+focused on being lightweight, simple, and secure.
+Its documentation, while extensive and accurate,
+is an absolute mess: the information you need is
+usually either spread across multiple pages,
+or buried somewhere two internal links deep.
+
+If you installed Dovecot via your package manager,
+you'll probably have a lot of configuration files
+in the `/etc/dovecot` directory and its subdirectories.
+I want you to delete all of them. Yes, `rm -rf` that crap.
+
+Dovecot is simple to configure, and doesn't care where
+you put its settings, so having all that chaos in `/etc/dovecot`
+just makes things unnecessarily confusing.
+
+Create a new blank configuration file `/etc/dovecot/dovecot.conf`,
+and start by filling in the details about your TLS certificate,
+and making clear that unencrypted connections are unacceptable:
+```sh
+ssl = required
+ssl_cert = </etc/ssl/certs/example.com.pem
+ssl_key  = </etc/ssl/private/example.com.key
+ssl_dh   = </etc/ssl/dhparam.pem
+
+ssl_prefer_server_ciphers = yes
+
+disable_plaintext_auth = yes
+```
+The final `disable_plaintext_auth` option tells Dovecot
+to reject any passwords that were sent unencrypted.
+This means it must be [hashed](https://en.wikipedia.org/wiki/Cryptographic_hash_function)
+or sent over a TLS connection, ideally both.
+
+Next, tell Dovecot which protocols to use
+and where to expect them as follows:
+```sh
+protocols = lmtp imap
+
+service lmtp {
+	inet_listener lmtp {
+		address = 127.0.0.1
+		port = 39999
+	}
+	user = vmail
+}
+```
+LMTP is the Local Mail Transport Protocol, which is basically SMTP
+but intended for exchange within a single server or cluster.
+When an email is received, Dovecot will start a child process
+under the `vmail` Unix user to deliver the message to its recipient.
+
+Since we set `ssl = required` earlier, MUAs (MRAs) will only get their mail
+if the STARTTLS handshake was successful during the IMAP exchange,
+or if they connect via IMAPS to force the use of encryption.
+If you want to, you can set Dovecot to only accept IMAPS
+by neutralizing the STARTTLS IMAP login listener:
+```sh
+service imap-login {
+	inet_listener imap {
+		port = 0
+	}
+}
+```
+
+Then we need to inform Dovecot which email users it should handle,
+and what to do with their messages. Create a file `/etc/dovecot/users` for this,
+which describes users in the same format as `/etc/passwd`:
+```
+user:password:uid:gid::homedir
+```
+You can leave `uid` and `gid` fields blank, since we're
+using our `vmail` virtual mail user for all accounts.
+You can even omit `homedir` if all users' directories
+follow the same pattern, for example `/var/vmail/<user>/`.
+The `user` name may contain the ending `@example.com`, but doesn't need to.
+Create the password hash to put in the `password` field as follows:
+```sh
+# If your server is beefy and has lots of RAM:
+doveadm pw -s ARGON2ID-CRYPT
+# If you're using an old potato:
+doveadm pw -s SHA512-CRYPT
+```
+After you've entered your password, simply copy-paste the entire
+hash string outputted by the program into the `password` field.
+
+Dovecot needs a file describing user accounts on two occasions:
+* To check whether a user logging into the IMAP server
+  is valid and has given the right password.
+  This is handled by the `passdb` block(s) in the configuration.
+* To know which email addresses the server is responsible for.
+  This is given by the `userdb` block(s) in the configuration.
+These functions aren't necessarily fulfilled by the same file:
+you can map multiple email addresses to one user acccount,
+or multiple user accounts to one email address.
+For simplicity, though, we'll use the same file:
+```sh
+passdb {
+	driver = passwd-file
+	args = scheme=ARGON2ID-CRYPT username_format=%n /etc/dovecot/users
+	#args = scheme=SHA512-CRYPT username_format=%n /etc/dovecot/users
+}
+
+userdb {
+	driver = passwd-file
+	args = username_format=%n /etc/dovecot/users
+	# Use vmail for all users, and if all users' inboxes have the same form:
+	override_fields = uid=vmail gid=vmail home=/var/vmail/%n
+}
+```
+The `driver` option sets the kind of table Dovecot should expect.
+We tell it to use a file in the `passwd`-like format described above,
+but other common choices include SQL or the actual `/etc/passwd` file.
+The options available in `args` depend on the chosen `driver`.
+
+The password hashing algorithm is given by `scheme`, while
+`username_format` desribes the format of user names in the table:
+if Dovecot receives an email for `someone@example.com`,
+then for `%n` it will expect `someone` in `/etc/dovecot/users`,
+but if you use `%u` it will want the full email address instead.
+
+In the `userdb` block we force the use of `vmail:vmail`
+for all users, and tell Dovecot to put their mail in `/var/vmail/%n`,
+where `%n` means the same as before. You can also use `%u`
+for the full address again, or `%d` for the domain name.
+
+Optionally, you can create a catch-all inbox that will accept
+all emails sent to your domain that don't match anyone in `/etc/dovecot/users`.
+Just add this second `userdb` block after the first:
+```sh
+userdb {
+	driver = static
+	args = uid=vmail gid=vmail home=/var/vmail/catchall allow_all_users=yes
+}
+```
+The `static` driver means there is no table file:
+all configuration is directly within this `userdb` block.
+If we didn't specify `allow_all_users=yes`, then Dovecot
+would check whether users existed using the `passdb` table.
+
+Finally, now that we've told Dovecot where the virtual mail users' `home`
+folders are, we need to specify how to store the emails within them:
+```sh
+mail_location = maildir:~:LAYOUT=fs
+```
+The two standard mailbox formats to choose from are `maildir` and `mbox`.
+I highly recommend `maildir`; it's faster and more flexible than `mbox`
+because it uses folders, whereas the older `mbox` uses a single database file.
+The `~` tells Dovecot to directly put all mails in the account's home folder,
+and `LAYOUT=fs` to user filesystem directories for you email folders. TODO?
+
+
+
+
+
+## MTA: Postfix
+
+
+## MTA: Exim
+
+
+
+## MTA: OpenSMTPD
+
+OpenSMTPD is an MTA by the [OpenBSD](https://www.openbsd.org/) project,
+who are known for their focus on security and minimalism.
+I really like OpenSMTPD's flexible configuration format,
+and its simple and intuitive way of passing around emails.
+
+It doesn't implement milter or any other filtering interface,
+so DKIM signing and spam checking must be shoehorned in.
+As such, only DKIMproxy can be used for DKIM signing.
+
+To begin, write your email domain on a single line in `/etc/smtpd/mailname`:
+```
+example.com
+```
+Next, you must fill in the `/etc/smtpd/aliases` file
+which maps recipient addresses to system users:
+```
+mailuser@example.com    systemuser
+```
+You can create a wildcard (catch-all) address for your domain
+by omitting `mailuser`, such that any unknown recipients end up there.
+You may also specify multiple system users as `systemuser1,systemuser2`.
+
+The first part of the actual config `/etc/smtpd/smtpd.conf` is as follows,
+with DKIMproxy listening on port 19999 and outputting to 20000:
+```sh
+# Register TLS certificate to encrypt email transport:
+pki "example.com" certificate "/etc/ssl/certs/example.com.pem"
+pki "example.com" key "/etc/ssl/private/example.com.key"
+pki "example.com" dhe auto
+
+table users "/etc/smtpd/aliases"
+
+# Relay emails submitted by client to DKIM signer on port 19999:
+listen on eth0 port 465 smtps       pki "example.com" tag SEND
+listen on eth0 port 587 tls-require pki "example.com" tag SEND
+accept tagged SEND from any for any relay via smtp://127.0.0.1:19999
+
+# Relay outgoing DKIM-signed emails to their intended destination:
+listen on lo port 20000 tag DKIM
+accept tagged DKIM from local for any relay
+```
+The lines starting with `pki` enable [DHE](https://en.wikipedia.org/wiki/Diffie%E2%80%93Hellman_key_exchange)
+and tell where the public certificate and private key are for the domain.
+We listen for client emails on both ports 465 (SMTPS) and 587 (SMTP),
+although for the latter we refuse the submission if STARTTLS fails.
+
+If no spam filtering is desired, let OpenSMTPD deliver the email
+by appending the following to the configuration file:
+```sh
+# Receive incoming emails and write them to the inbox:
+listen on eth0 port 25 tls pki "example.com"
+accept from any for domain "example.com" virtual <users> deliver to lmtp rcpt-to as vmail
+```
+Here, we look up which system user corresponds to the recipient in `<users>`,
+and deliver it to their maildir `mail` in their home folder.
+
+
+### With SpamAssassin UNTESTED
+
+For SpamAssassin, we'll need to use [spampd](https://github.com/mpaperno/spampd) as an SMTP shim,
+which we set to listen on port 29999 and relay to port 30000.
+Add the `--nodetach` option in an init script or systemd service,
+but leave it out when starting `spampd` by hand for testing:
+```sh
+$ spampd --host=127.0.0.1:29999 --relayhost=127.0.0.1:30000 [--nodetach]
+```
+The rest of `smtpd`'s configuration is then as follows:
+```sh
+# Receive incoming emails and send them to spampd on port 29999:
+listen on eth0 port 25 tls pki "example.com" tag SCAN
+accept tagged SCAN from any for domain "example.com" relay via smtp://127.0.0.1:29999
+
+# Receive filtered incoming emails and write them to the inbox:
+listen on lo port 30000 tag DONE
+accept tagged DONE from local for any virtual <users> deliver to maildir "~/mail"
+```
+TODO
+
+### With Rspamd
+
+
+
+
+
+
+
+
+
+## DKIM signer: DKIMproxy
+
+[DKIMproxy](http://dkimproxy.sourceforge.net/) is a mature and simple SMTP proxy
+that signs or verifies all emails passed through it
+using the Perl Mail::DKIM module.
+
+In this guide we only use its signing feature,
+which is controlled by `/etc/dkimproxy/dkimproxy_out.conf`.
+This configuration file is pretty self-explanatory:
+```sh
+listen    127.0.0.1:19999
+relay     127.0.0.1:20000
+
+domain    example.com
+selector  mail
+
+keyfile   /etc/dkim/private.key
+signature dkim(c=relaxed/simple,a=rsa-sha256)
+```
+This last line clearly tells it to create a DKIM signature,
+because it also supports DKIM's predecessor, Yahoo! DomainKey.
+The tag `a=rsa-sha256` specifies the signing algorithm,
+while `c=relaxed/simple` gives the [canonicalization](https://tools.ietf.org/html/rfc6376#section-3.4) method.
+
+Canonicalization is a way to prevent the signature from becoming invalid
+as the email might get reformatted by the MTAs on its journey.
+Our choice `relaxed/simple` allows the header to be reformatted (`relaxed`),
+but not the actual body of the message (`simple`).
+
+
+
+
+## DKIM signer: OpenDKIM
+
+[OpenDKIM](http://www.opendkim.org/) is a milter program
+that can both create and verify DKIM signatures.
+
+
+
+
+## Spam filter: Rspamd
+
+
+
+## Spam filter: SpamAssassin
+
+
+