diff options
| -rw-r--r-- | content/articles/2019-email-server-tutorial.md | 612 |
1 files changed, 612 insertions, 0 deletions
diff --git a/content/articles/2019-email-server-tutorial.md b/content/articles/2019-email-server-tutorial.md new file mode 100644 index 0000000..cca8379 --- /dev/null +++ b/content/articles/2019-email-server-tutorial.md @@ -0,0 +1,612 @@ ++++ +title = "How to set up an email server in 2019" ++++ + +# How to set up an email server in 2019 + +This is a guide to set up your own email server in 2019. +This is a messy topic, and the available documentation is even messier, +so I compiled this guide in an attempt to make it all more accessible. + +So why run your own mail server? There are many reasons, +ranging from privacy concerns about popular providers, +to simply wanting to learn about this stuff for fun. +If you've arrived here, then I assume you've already found a reason. + +But beware that this is not for the faint of heart: +email is a horribly designed system, and even if you manage +to set up your own server, you may find that GMail flags all +your messages as spam, with little you can do about it. +This guide tries to implement all the security measures +that GMail seems to like, but *your mileage may vary*. + + + +## How email works + +The program on your device that you use to read and send emails +contains both a Mail User Agent (MUA) and a Mail Retrieval Agent (MRA), +although in practice the MRA is also often referred to as an MUA. +These mail agents can be in a dedicated app like [Thunderbird](https://www.thunderbird.net/en-US/) +or Microsoft Outlook, or it can simply be a web interface. + +When you check your inbox, your MRA asks for any new emails +from the server, which then sends them via the [IMAP](https://en.wikipedia.org/wiki/Internet_Message_Access_Protocol) protocol +(we ignore [POP3](https://en.wikipedia.org/wiki/Post_Office_Protocol), since it is rather [outdated](https://www.pop2imap.com/)). +The server-side program here is simply called the IMAP server, +which watches over the server's copy of your inbox. + +When you send an email, your MUA passes it along to the server's +Mail Submission Agent (MSA), which then immediately hands it over +to the Mail Transfer Agent (MTA) running on that same server. +In practice, the MSA's functionality is often regarded as part of the MTA. + +The MTA then works out where your email needs to go, +and sends it over a protocol called [SMTP](https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol) +to the MTA of the server responsible for the recipient of your message. +In complex networks there may be multiple MTAs in a chain. + +The final receiving MTA then hands over your email to the +Mail Delivery Agent (MDA) running on that same server. +The MDA checks whether it's spam, and then writes it to the server's storage, +which is being watched over by an IMAP server, thus completing the cycle. +Most MTA programs also have MDA features for simplicity, +although in this guide we'll be using our IMAP server's MDA helper. + +Long story short, the server we want to set up +must at least be running both an MTA for you to send emails +and an IMAP server for you to check your inbox, +one of which should also include MDA functionality. + + + +## Email security + +This messy base email system is horribly insecure on its own, +so we still need to tack on some more messy features. +In this context, "security" refers to both spam and privacy protection. +This guide covers all techniques mentioned here. + +Spam protection also means two things: +defending yourself against spammers, and +preventing that your emails' recipients' filters don't flag *you* as a spammer. +The former is optional, but the latter is not. + +Let's start with how to come across as a trustworthy mail server: + +The first feature is [Sender Policy Framework](https://en.wikipedia.org/wiki/Sender_Policy_Framework) (SPF), +which stops a spammer from filling in your email address +in the "sender" field of their message. +You do this by publishing a policy in a DNS record for how your domain +should relate to the IP address of your email server. +A filter enforcing SPF will use that to check that your server +is authorized to send messages with your email address. + +Then there's [DomainKeys Identified Mail](https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail) (DKIM), +which is a more comprehensive form of SPF. +It adds a cryptographic signature to all your emails, +which the receiver's spam filter will check against +the contents and a public key you need to publish in a DNS record. +You should implement *both* SPF and DKIM, despite their overlap. + +Lastly, we have [Domain-based Message Authentication, Reporting and Conformance](https://en.wikipedia.org/wiki/DMARC). +which uses another DNS record to specify whether you're using SPF and/or DKIM, +and whether super- and subdomains should be regarded as valid for authentication. +It also gives a contact address to which the receiver should send failure reports. +I highly recommend implementing this too. + +To defend yourself against spammers your server should enforce +the SPF, DKIM and DMARC policies of senders, in addition to some other checks. +All of that will be handled by your spam filter. + +There's one final problem to bring up: +your emails will be sent across the Internet unencrypted. + +For client-server communication, this is effectively solved +by IMAPS and SMTPS, which wrap IMAP and SMTP in [TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security). +To be clear, SMTPS only encrypts email submission by an MUA to an MTA (MSA). + +For server-server relaying of messages over SMTP, we have +STARTTLS, which is a system for opportunistic TLS, meaning that +communication is only encrypted if both parties agree +after a short unencrypted discussion. +That last part is vulnerable to [MitM](https://en.wikipedia.org/wiki/Man-in-the-middle_attack) attacks, +where anyone along the path of the email servers' discussion +can alter the exchange to block the use of encryption, +which sometimes actually [happens](https://www.eff.org/deeplinks/2014/11/starttls-downgrade-attacks) in practice. + +The only way to make sure that STARTTLS is used +is to refuse any emails unless the TLS handshake succeeds. +That's a risky approach, because fewer mail servers support +STARTTLS than you might expect or hope: I've had airline +booking confirmations, full of personal details, +and made with multi-billion-dollar companies, +sent across the Internet without any protection. +I'll show you how, but do it at your own risk. + + + +## Preparation + +Are you still here? Good, let's get started. Here's what you'll need: +* SSH root access to a 24/7-accessible server running Linux or *BSD +* an associated domain name for which you can create or edit DNS records +* a TLS certificate for encryption + +The TLS certificate is technically optional, +but you *really* should use one; it's 2019 for goodness' sake. +You may get away with using a self-signed certificate, +but instead I highly recommend getting a proper one from +[Let's Encrypt](https://letsencrypt.org/) (it's free!). + +In the rest of this guide I'll assume that +you have your public TLS certificate at `/etc/ssl/certs/example.com.pem`, +and your private encryption key at `/etc/ssl/private/example.com.pem`. +You should also create a file with [DH parameters](https://weakdh.org/sysadmin.html) +using the `openssl` tool, included in OpenSSL and LibreSSL: +```sh +openssl dhparam -out /etc/ssl/dhparam.pem 4096 +``` +This will take a while, because it's looking for sources +of randomness in your (very non-random) computer. +Just keep doing stuff with it running in the background. +The resulting file is public and expendable. + + + +There exist many options for most of the software roles described earlier. +This guide covers multiple options for most of these, +given here from most to least recommended by me: +* IMAP server + MDA: [Dovecot](https://www.dovecot.org/). +* MTA: [OpenSMTPD](https://opensmtpd.org/), [Postfix](http://www.postfix.org/), or [Exim](https://exim.org/). +* Spam filtering: [Rspamd](https://www.rspamd.com/), or [SpamAssassin](https://spamassassin.apache.org/). +* DKIM signing: [OpenDKIM](http://www.opendkim.org/), or [DKIMproxy](http://dkimproxy.sourceforge.net/). + +The MTA feeds arriving emails through the spam filter +before giving them to the MDA, which puts them in an inbox folder. +Likewise, it feeds outgoing emails through the DKIM signer. +The IMAP server then delivers your inbox contents to your MUA. + +All these MTAs include an MDA, but nevertheless +I recommend using Dovecot's one because it gives the most control. +This is also important because of how spam filtering works: +according to the email specification, an MTA assumes full responsibility +over any emails it accepts for delivery or relaying, +and promises that it will do its best to get it to its destination. + +The takeaway from this is that an MTA must not magically +swallow any emails it doesn't like, so if it uses a spam filter, +that filter may tag emails as "spam" or "not spam", +but the MTA is not allowed to act based on that verdict. +The MDA does not have this restriction, so only there +can we make the decision what to do with spam, +and Dovecot's MDA is by far the best equipped for that. + +When setting up an email server, you have the choice between +attaching email addresses to system users or virtual users. +With system users, if an email arrives for `foo@example.com`, +then MTA, MDA and IMAP server will all expect that there +exists a `foo` Unix user on the server to deliver it to. +With virtual users, this is not the case. + +This guide only covers virtual users, which I recommend +because it's more flexible, and unless you're using some +ancient mainframe there is no advantage to using system users. + +Therefore, create a global email user and group, +which are traditionally both called `vmail`. +Do *not* pass the "system user/group" flags, +because that will get you into trouble with Dovecot: +```sh +# GNU CoreUtils: +groupadd vmail +useradd -M -g vmail vmail +# BusyBox: +addgroup vmail +adduser -HD -G vmail vmail +# *BSD: +no clue, you can work it out +``` + + + + +## DNS records + +First things first, you should create an RSA keypair for DKIM, +which you do using the `openssl` utility again. +The minimum size is 1024 bits, but I recommend using 2048 bits. +Bigger is better, but because DNS is involved we can't stretch it +to 4096 bits without risking causing discomfort to [some](https://serverfault.com/questions/747185/dkim-can-i-use-a-rsa-key-larger-than-2048bit-i-e-4096) servers. +```sh +$ openssl genrsa -out /TODO/private.key 2048 +$ openssl rsa -in /path/private.key -out /path/public.key +``` + +Now we must set up all the necessary DNS records. +It may take up to a day for these to propagate over the Internet, +so I recommend doing this section now and the rest tomorrow. + +Now, this might sound obvious, but you must have an A and/or AAAA record +to associate your domain `example.com` with the server's IP address. +For email you also must have [reverse DNS](https://en.wikipedia.org/wiki/Reverse_DNS_lookup) set up correctly. +If you're renting your server remotely, you can often do this from +the provider's configuration tool, or you can just create a PTR-type DNS record. + +To inform the rest of the Internet that your server is an email server, +create an MX (Mail eXchanger) DNS record for your domain. +Note the dot at the end of the domain name: +``` +MX 42 example.com. +``` +When a message is sent to an email address ending in `@example.com`, +the sending server will query DNS for any MX records for `example.com`. +There it will find a domain name (in this case `example.com` again), +for which it will look up the IP address, and send the email to. +The domain name in the record must *not* have an associated CNAME record; +it must be directly translatable to an IP address. + +You may have multiple MX records, which must contain different +domain names, each with an associated preference number (`42` here). +The sending server will use all MX records with lower numbers first, +and if those servers are all unavailable, it will try a higher number. +If you have multiple mail servers (which is good for availability), +you can therefore declare those as follows: +``` +MX 13 mx1.example.com. +MX 42 mx2.example.com. +``` +Here, a server sending an email to your domain `example.com` +will try to send it to the IP address of `mx1.example.com` first, +and if that fails, it will move on to the higher numbered `mx2.example.com`. +If both `mx1` and `mx2` have the same number, then the sender +will randomly choose one, which is useful for load balancing. + +Next, publish an SPF policy record so you don't look like a spammer. +This is a TXT (*not* SPF!) record with the following contents: +``` +TXT "v=spf1 mx -all" +``` +Everything after the version `v=spf1` is a list of *mechanisms* +for a spam filter to try out in the given order. +The `-all` at the end says to reject your email +if all of the previous mechanisms fail verification. +I recommend only using the `mx` mechanism, which tells the verifier +to look at the A/AAAA addresses of the domains in your MX records. +This allows you to add, remove, or change your servers +without needing to update this record. + +Your DKIM policy must also be published in a TXT record +as follows, where `<public.key>` is the public RSA key `MI...AB` +stored in `/TODO/public.key`, with the newlines removed: +``` +TXT "v=DKIM1; t=s; h=sha256; p=<public.key>" +``` +Here, `v=DKIM1` is the version and must be the first tag. +The flag `t=s` enables strict mode, as recommended by the [DKIM spec](https://tools.ietf.org/html/rfc6376#page-29), +meaning emails sent from subdomains immediately fail verification. +The optional tag `h=sha256` blocks the use of the old SHA1 algorithm. + +Finally, there is the DMARC policy, which, unlike the others, +will need to be updated at the end of this guide. +For now, create another TXT record with these contents, +where `<admin>` is an email address of your choosing, +which may belong to the domain you're setting up for: +``` +TXT "v=DMARC1; aspf=s; adkim=s; p=none; sp=reject; fo=1; ruf=mailto:<admin>" +``` +The version tag `v=DMARC1` is required and must come first. +Next, `aspf=s` and `adkim=s` enable strict mode for both SPF and DKIM, +which once again blocks subdomains from passing the test. +Then `p=none` and `sp=reject` control what to do to failed messages +coming from the main domain and subdomains, respectively. +Unsurprisingly, `reject` means that delivery should be refused, +`none` asks to let it through anyway, and `quarantine` tells +the filter to take a closer look at the email or to put it in a spam folder. +Finally, `fo=1` asks the filter to create a forensic report +if any type of verification fails, and `ruf=` gives an address to send it to. + + + +## IMAP server: Dovecot + +[Dovecot](https://www.dovecot.org/) is a very popular IMAP server, +focused on being lightweight, simple, and secure. +Its documentation, while extensive and accurate, +is an absolute mess: the information you need is +usually either spread across multiple pages, +or buried somewhere two internal links deep. + +If you installed Dovecot via your package manager, +you'll probably have a lot of configuration files +in the `/etc/dovecot` directory and its subdirectories. +I want you to delete all of them. Yes, `rm -rf` that crap. + +Dovecot is simple to configure, and doesn't care where +you put its settings, so having all that chaos in `/etc/dovecot` +just makes things unnecessarily confusing. + +Create a new blank configuration file `/etc/dovecot/dovecot.conf`, +and start by filling in the details about your TLS certificate, +and making clear that unencrypted connections are unacceptable: +```sh +ssl = required +ssl_cert = </etc/ssl/certs/example.com.pem +ssl_key = </etc/ssl/private/example.com.key +ssl_dh = </etc/ssl/dhparam.pem + +ssl_prefer_server_ciphers = yes + +disable_plaintext_auth = yes +``` +The final `disable_plaintext_auth` option tells Dovecot +to reject any passwords that were sent unencrypted. +This means it must be [hashed](https://en.wikipedia.org/wiki/Cryptographic_hash_function) +or sent over a TLS connection, ideally both. + +Next, tell Dovecot which protocols to use +and where to expect them as follows: +```sh +protocols = lmtp imap + +service lmtp { + inet_listener lmtp { + address = 127.0.0.1 + port = 39999 + } + user = vmail +} +``` +LMTP is the Local Mail Transport Protocol, which is basically SMTP +but intended for exchange within a single server or cluster. +When an email is received, Dovecot will start a child process +under the `vmail` Unix user to deliver the message to its recipient. + +Since we set `ssl = required` earlier, MUAs (MRAs) will only get their mail +if the STARTTLS handshake was successful during the IMAP exchange, +or if they connect via IMAPS to force the use of encryption. +If you want to, you can set Dovecot to only accept IMAPS +by neutralizing the STARTTLS IMAP login listener: +```sh +service imap-login { + inet_listener imap { + port = 0 + } +} +``` + +Then we need to inform Dovecot which email users it should handle, +and what to do with their messages. Create a file `/etc/dovecot/users` for this, +which describes users in the same format as `/etc/passwd`: +``` +user:password:uid:gid::homedir +``` +You can leave `uid` and `gid` fields blank, since we're +using our `vmail` virtual mail user for all accounts. +You can even omit `homedir` if all users' directories +follow the same pattern, for example `/var/vmail/<user>/`. +The `user` name may contain the ending `@example.com`, but doesn't need to. +Create the password hash to put in the `password` field as follows: +```sh +# If your server is beefy and has lots of RAM: +doveadm pw -s ARGON2ID-CRYPT +# If you're using an old potato: +doveadm pw -s SHA512-CRYPT +``` +After you've entered your password, simply copy-paste the entire +hash string outputted by the program into the `password` field. + +Dovecot needs a file describing user accounts on two occasions: +* To check whether a user logging into the IMAP server + is valid and has given the right password. + This is handled by the `passdb` block(s) in the configuration. +* To know which email addresses the server is responsible for. + This is given by the `userdb` block(s) in the configuration. +These functions aren't necessarily fulfilled by the same file: +you can map multiple email addresses to one user acccount, +or multiple user accounts to one email address. +For simplicity, though, we'll use the same file: +```sh +passdb { + driver = passwd-file + args = scheme=ARGON2ID-CRYPT username_format=%n /etc/dovecot/users + #args = scheme=SHA512-CRYPT username_format=%n /etc/dovecot/users +} + +userdb { + driver = passwd-file + args = username_format=%n /etc/dovecot/users + # Use vmail for all users, and if all users' inboxes have the same form: + override_fields = uid=vmail gid=vmail home=/var/vmail/%n +} +``` +The `driver` option sets the kind of table Dovecot should expect. +We tell it to use a file in the `passwd`-like format described above, +but other common choices include SQL or the actual `/etc/passwd` file. +The options available in `args` depend on the chosen `driver`. + +The password hashing algorithm is given by `scheme`, while +`username_format` desribes the format of user names in the table: +if Dovecot receives an email for `someone@example.com`, +then for `%n` it will expect `someone` in `/etc/dovecot/users`, +but if you use `%u` it will want the full email address instead. + +In the `userdb` block we force the use of `vmail:vmail` +for all users, and tell Dovecot to put their mail in `/var/vmail/%n`, +where `%n` means the same as before. You can also use `%u` +for the full address again, or `%d` for the domain name. + +Optionally, you can create a catch-all inbox that will accept +all emails sent to your domain that don't match anyone in `/etc/dovecot/users`. +Just add this second `userdb` block after the first: +```sh +userdb { + driver = static + args = uid=vmail gid=vmail home=/var/vmail/catchall allow_all_users=yes +} +``` +The `static` driver means there is no table file: +all configuration is directly within this `userdb` block. +If we didn't specify `allow_all_users=yes`, then Dovecot +would check whether users existed using the `passdb` table. + +Finally, now that we've told Dovecot where the virtual mail users' `home` +folders are, we need to specify how to store the emails within them: +```sh +mail_location = maildir:~:LAYOUT=fs +``` +The two standard mailbox formats to choose from are `maildir` and `mbox`. +I highly recommend `maildir`; it's faster and more flexible than `mbox` +because it uses folders, whereas the older `mbox` uses a single database file. +The `~` tells Dovecot to directly put all mails in the account's home folder, +and `LAYOUT=fs` to user filesystem directories for you email folders. TODO? + + + + + +## MTA: Postfix + + +## MTA: Exim + + + +## MTA: OpenSMTPD + +OpenSMTPD is an MTA by the [OpenBSD](https://www.openbsd.org/) project, +who are known for their focus on security and minimalism. +I really like OpenSMTPD's flexible configuration format, +and its simple and intuitive way of passing around emails. + +It doesn't implement milter or any other filtering interface, +so DKIM signing and spam checking must be shoehorned in. +As such, only DKIMproxy can be used for DKIM signing. + +To begin, write your email domain on a single line in `/etc/smtpd/mailname`: +``` +example.com +``` +Next, you must fill in the `/etc/smtpd/aliases` file +which maps recipient addresses to system users: +``` +mailuser@example.com systemuser +``` +You can create a wildcard (catch-all) address for your domain +by omitting `mailuser`, such that any unknown recipients end up there. +You may also specify multiple system users as `systemuser1,systemuser2`. + +The first part of the actual config `/etc/smtpd/smtpd.conf` is as follows, +with DKIMproxy listening on port 19999 and outputting to 20000: +```sh +# Register TLS certificate to encrypt email transport: +pki "example.com" certificate "/etc/ssl/certs/example.com.pem" +pki "example.com" key "/etc/ssl/private/example.com.key" +pki "example.com" dhe auto + +table users "/etc/smtpd/aliases" + +# Relay emails submitted by client to DKIM signer on port 19999: +listen on eth0 port 465 smtps pki "example.com" tag SEND +listen on eth0 port 587 tls-require pki "example.com" tag SEND +accept tagged SEND from any for any relay via smtp://127.0.0.1:19999 + +# Relay outgoing DKIM-signed emails to their intended destination: +listen on lo port 20000 tag DKIM +accept tagged DKIM from local for any relay +``` +The lines starting with `pki` enable [DHE](https://en.wikipedia.org/wiki/Diffie%E2%80%93Hellman_key_exchange) +and tell where the public certificate and private key are for the domain. +We listen for client emails on both ports 465 (SMTPS) and 587 (SMTP), +although for the latter we refuse the submission if STARTTLS fails. + +If no spam filtering is desired, let OpenSMTPD deliver the email +by appending the following to the configuration file: +```sh +# Receive incoming emails and write them to the inbox: +listen on eth0 port 25 tls pki "example.com" +accept from any for domain "example.com" virtual <users> deliver to lmtp rcpt-to as vmail +``` +Here, we look up which system user corresponds to the recipient in `<users>`, +and deliver it to their maildir `mail` in their home folder. + + +### With SpamAssassin UNTESTED + +For SpamAssassin, we'll need to use [spampd](https://github.com/mpaperno/spampd) as an SMTP shim, +which we set to listen on port 29999 and relay to port 30000. +Add the `--nodetach` option in an init script or systemd service, +but leave it out when starting `spampd` by hand for testing: +```sh +$ spampd --host=127.0.0.1:29999 --relayhost=127.0.0.1:30000 [--nodetach] +``` +The rest of `smtpd`'s configuration is then as follows: +```sh +# Receive incoming emails and send them to spampd on port 29999: +listen on eth0 port 25 tls pki "example.com" tag SCAN +accept tagged SCAN from any for domain "example.com" relay via smtp://127.0.0.1:29999 + +# Receive filtered incoming emails and write them to the inbox: +listen on lo port 30000 tag DONE +accept tagged DONE from local for any virtual <users> deliver to maildir "~/mail" +``` +TODO + +### With Rspamd + + + + + + + + + +## DKIM signer: DKIMproxy + +[DKIMproxy](http://dkimproxy.sourceforge.net/) is a mature and simple SMTP proxy +that signs or verifies all emails passed through it +using the Perl Mail::DKIM module. + +In this guide we only use its signing feature, +which is controlled by `/etc/dkimproxy/dkimproxy_out.conf`. +This configuration file is pretty self-explanatory: +```sh +listen 127.0.0.1:19999 +relay 127.0.0.1:20000 + +domain example.com +selector mail + +keyfile /etc/dkim/private.key +signature dkim(c=relaxed/simple,a=rsa-sha256) +``` +This last line clearly tells it to create a DKIM signature, +because it also supports DKIM's predecessor, Yahoo! DomainKey. +The tag `a=rsa-sha256` specifies the signing algorithm, +while `c=relaxed/simple` gives the [canonicalization](https://tools.ietf.org/html/rfc6376#section-3.4) method. + +Canonicalization is a way to prevent the signature from becoming invalid +as the email might get reformatted by the MTAs on its journey. +Our choice `relaxed/simple` allows the header to be reformatted (`relaxed`), +but not the actual body of the message (`simple`). + + + + +## DKIM signer: OpenDKIM + +[OpenDKIM](http://www.opendkim.org/) is a milter program +that can both create and verify DKIM signatures. + + + + +## Spam filter: Rspamd + + + +## Spam filter: SpamAssassin + + + |