diff options
Diffstat (limited to 'content/articles/2019-email-server-tutorial.md')
| -rw-r--r-- | content/articles/2019-email-server-tutorial.md | 612 |
1 files changed, 0 insertions, 612 deletions
diff --git a/content/articles/2019-email-server-tutorial.md b/content/articles/2019-email-server-tutorial.md deleted file mode 100644 index cca8379..0000000 --- a/content/articles/2019-email-server-tutorial.md +++ /dev/null @@ -1,612 +0,0 @@ -+++ -title = "How to set up an email server in 2019" -+++ - -# How to set up an email server in 2019 - -This is a guide to set up your own email server in 2019. -This is a messy topic, and the available documentation is even messier, -so I compiled this guide in an attempt to make it all more accessible. - -So why run your own mail server? There are many reasons, -ranging from privacy concerns about popular providers, -to simply wanting to learn about this stuff for fun. -If you've arrived here, then I assume you've already found a reason. - -But beware that this is not for the faint of heart: -email is a horribly designed system, and even if you manage -to set up your own server, you may find that GMail flags all -your messages as spam, with little you can do about it. -This guide tries to implement all the security measures -that GMail seems to like, but *your mileage may vary*. - - - -## How email works - -The program on your device that you use to read and send emails -contains both a Mail User Agent (MUA) and a Mail Retrieval Agent (MRA), -although in practice the MRA is also often referred to as an MUA. -These mail agents can be in a dedicated app like [Thunderbird](https://www.thunderbird.net/en-US/) -or Microsoft Outlook, or it can simply be a web interface. - -When you check your inbox, your MRA asks for any new emails -from the server, which then sends them via the [IMAP](https://en.wikipedia.org/wiki/Internet_Message_Access_Protocol) protocol -(we ignore [POP3](https://en.wikipedia.org/wiki/Post_Office_Protocol), since it is rather [outdated](https://www.pop2imap.com/)). -The server-side program here is simply called the IMAP server, -which watches over the server's copy of your inbox. - -When you send an email, your MUA passes it along to the server's -Mail Submission Agent (MSA), which then immediately hands it over -to the Mail Transfer Agent (MTA) running on that same server. -In practice, the MSA's functionality is often regarded as part of the MTA. - -The MTA then works out where your email needs to go, -and sends it over a protocol called [SMTP](https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol) -to the MTA of the server responsible for the recipient of your message. -In complex networks there may be multiple MTAs in a chain. - -The final receiving MTA then hands over your email to the -Mail Delivery Agent (MDA) running on that same server. -The MDA checks whether it's spam, and then writes it to the server's storage, -which is being watched over by an IMAP server, thus completing the cycle. -Most MTA programs also have MDA features for simplicity, -although in this guide we'll be using our IMAP server's MDA helper. - -Long story short, the server we want to set up -must at least be running both an MTA for you to send emails -and an IMAP server for you to check your inbox, -one of which should also include MDA functionality. - - - -## Email security - -This messy base email system is horribly insecure on its own, -so we still need to tack on some more messy features. -In this context, "security" refers to both spam and privacy protection. -This guide covers all techniques mentioned here. - -Spam protection also means two things: -defending yourself against spammers, and -preventing that your emails' recipients' filters don't flag *you* as a spammer. -The former is optional, but the latter is not. - -Let's start with how to come across as a trustworthy mail server: - -The first feature is [Sender Policy Framework](https://en.wikipedia.org/wiki/Sender_Policy_Framework) (SPF), -which stops a spammer from filling in your email address -in the "sender" field of their message. -You do this by publishing a policy in a DNS record for how your domain -should relate to the IP address of your email server. -A filter enforcing SPF will use that to check that your server -is authorized to send messages with your email address. - -Then there's [DomainKeys Identified Mail](https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail) (DKIM), -which is a more comprehensive form of SPF. -It adds a cryptographic signature to all your emails, -which the receiver's spam filter will check against -the contents and a public key you need to publish in a DNS record. -You should implement *both* SPF and DKIM, despite their overlap. - -Lastly, we have [Domain-based Message Authentication, Reporting and Conformance](https://en.wikipedia.org/wiki/DMARC). -which uses another DNS record to specify whether you're using SPF and/or DKIM, -and whether super- and subdomains should be regarded as valid for authentication. -It also gives a contact address to which the receiver should send failure reports. -I highly recommend implementing this too. - -To defend yourself against spammers your server should enforce -the SPF, DKIM and DMARC policies of senders, in addition to some other checks. -All of that will be handled by your spam filter. - -There's one final problem to bring up: -your emails will be sent across the Internet unencrypted. - -For client-server communication, this is effectively solved -by IMAPS and SMTPS, which wrap IMAP and SMTP in [TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security). -To be clear, SMTPS only encrypts email submission by an MUA to an MTA (MSA). - -For server-server relaying of messages over SMTP, we have -STARTTLS, which is a system for opportunistic TLS, meaning that -communication is only encrypted if both parties agree -after a short unencrypted discussion. -That last part is vulnerable to [MitM](https://en.wikipedia.org/wiki/Man-in-the-middle_attack) attacks, -where anyone along the path of the email servers' discussion -can alter the exchange to block the use of encryption, -which sometimes actually [happens](https://www.eff.org/deeplinks/2014/11/starttls-downgrade-attacks) in practice. - -The only way to make sure that STARTTLS is used -is to refuse any emails unless the TLS handshake succeeds. -That's a risky approach, because fewer mail servers support -STARTTLS than you might expect or hope: I've had airline -booking confirmations, full of personal details, -and made with multi-billion-dollar companies, -sent across the Internet without any protection. -I'll show you how, but do it at your own risk. - - - -## Preparation - -Are you still here? Good, let's get started. Here's what you'll need: -* SSH root access to a 24/7-accessible server running Linux or *BSD -* an associated domain name for which you can create or edit DNS records -* a TLS certificate for encryption - -The TLS certificate is technically optional, -but you *really* should use one; it's 2019 for goodness' sake. -You may get away with using a self-signed certificate, -but instead I highly recommend getting a proper one from -[Let's Encrypt](https://letsencrypt.org/) (it's free!). - -In the rest of this guide I'll assume that -you have your public TLS certificate at `/etc/ssl/certs/example.com.pem`, -and your private encryption key at `/etc/ssl/private/example.com.pem`. -You should also create a file with [DH parameters](https://weakdh.org/sysadmin.html) -using the `openssl` tool, included in OpenSSL and LibreSSL: -```sh -openssl dhparam -out /etc/ssl/dhparam.pem 4096 -``` -This will take a while, because it's looking for sources -of randomness in your (very non-random) computer. -Just keep doing stuff with it running in the background. -The resulting file is public and expendable. - - - -There exist many options for most of the software roles described earlier. -This guide covers multiple options for most of these, -given here from most to least recommended by me: -* IMAP server + MDA: [Dovecot](https://www.dovecot.org/). -* MTA: [OpenSMTPD](https://opensmtpd.org/), [Postfix](http://www.postfix.org/), or [Exim](https://exim.org/). -* Spam filtering: [Rspamd](https://www.rspamd.com/), or [SpamAssassin](https://spamassassin.apache.org/). -* DKIM signing: [OpenDKIM](http://www.opendkim.org/), or [DKIMproxy](http://dkimproxy.sourceforge.net/). - -The MTA feeds arriving emails through the spam filter -before giving them to the MDA, which puts them in an inbox folder. -Likewise, it feeds outgoing emails through the DKIM signer. -The IMAP server then delivers your inbox contents to your MUA. - -All these MTAs include an MDA, but nevertheless -I recommend using Dovecot's one because it gives the most control. -This is also important because of how spam filtering works: -according to the email specification, an MTA assumes full responsibility -over any emails it accepts for delivery or relaying, -and promises that it will do its best to get it to its destination. - -The takeaway from this is that an MTA must not magically -swallow any emails it doesn't like, so if it uses a spam filter, -that filter may tag emails as "spam" or "not spam", -but the MTA is not allowed to act based on that verdict. -The MDA does not have this restriction, so only there -can we make the decision what to do with spam, -and Dovecot's MDA is by far the best equipped for that. - -When setting up an email server, you have the choice between -attaching email addresses to system users or virtual users. -With system users, if an email arrives for `foo@example.com`, -then MTA, MDA and IMAP server will all expect that there -exists a `foo` Unix user on the server to deliver it to. -With virtual users, this is not the case. - -This guide only covers virtual users, which I recommend -because it's more flexible, and unless you're using some -ancient mainframe there is no advantage to using system users. - -Therefore, create a global email user and group, -which are traditionally both called `vmail`. -Do *not* pass the "system user/group" flags, -because that will get you into trouble with Dovecot: -```sh -# GNU CoreUtils: -groupadd vmail -useradd -M -g vmail vmail -# BusyBox: -addgroup vmail -adduser -HD -G vmail vmail -# *BSD: -no clue, you can work it out -``` - - - - -## DNS records - -First things first, you should create an RSA keypair for DKIM, -which you do using the `openssl` utility again. -The minimum size is 1024 bits, but I recommend using 2048 bits. -Bigger is better, but because DNS is involved we can't stretch it -to 4096 bits without risking causing discomfort to [some](https://serverfault.com/questions/747185/dkim-can-i-use-a-rsa-key-larger-than-2048bit-i-e-4096) servers. -```sh -$ openssl genrsa -out /TODO/private.key 2048 -$ openssl rsa -in /path/private.key -out /path/public.key -``` - -Now we must set up all the necessary DNS records. -It may take up to a day for these to propagate over the Internet, -so I recommend doing this section now and the rest tomorrow. - -Now, this might sound obvious, but you must have an A and/or AAAA record -to associate your domain `example.com` with the server's IP address. -For email you also must have [reverse DNS](https://en.wikipedia.org/wiki/Reverse_DNS_lookup) set up correctly. -If you're renting your server remotely, you can often do this from -the provider's configuration tool, or you can just create a PTR-type DNS record. - -To inform the rest of the Internet that your server is an email server, -create an MX (Mail eXchanger) DNS record for your domain. -Note the dot at the end of the domain name: -``` -MX 42 example.com. -``` -When a message is sent to an email address ending in `@example.com`, -the sending server will query DNS for any MX records for `example.com`. -There it will find a domain name (in this case `example.com` again), -for which it will look up the IP address, and send the email to. -The domain name in the record must *not* have an associated CNAME record; -it must be directly translatable to an IP address. - -You may have multiple MX records, which must contain different -domain names, each with an associated preference number (`42` here). -The sending server will use all MX records with lower numbers first, -and if those servers are all unavailable, it will try a higher number. -If you have multiple mail servers (which is good for availability), -you can therefore declare those as follows: -``` -MX 13 mx1.example.com. -MX 42 mx2.example.com. -``` -Here, a server sending an email to your domain `example.com` -will try to send it to the IP address of `mx1.example.com` first, -and if that fails, it will move on to the higher numbered `mx2.example.com`. -If both `mx1` and `mx2` have the same number, then the sender -will randomly choose one, which is useful for load balancing. - -Next, publish an SPF policy record so you don't look like a spammer. -This is a TXT (*not* SPF!) record with the following contents: -``` -TXT "v=spf1 mx -all" -``` -Everything after the version `v=spf1` is a list of *mechanisms* -for a spam filter to try out in the given order. -The `-all` at the end says to reject your email -if all of the previous mechanisms fail verification. -I recommend only using the `mx` mechanism, which tells the verifier -to look at the A/AAAA addresses of the domains in your MX records. -This allows you to add, remove, or change your servers -without needing to update this record. - -Your DKIM policy must also be published in a TXT record -as follows, where `<public.key>` is the public RSA key `MI...AB` -stored in `/TODO/public.key`, with the newlines removed: -``` -TXT "v=DKIM1; t=s; h=sha256; p=<public.key>" -``` -Here, `v=DKIM1` is the version and must be the first tag. -The flag `t=s` enables strict mode, as recommended by the [DKIM spec](https://tools.ietf.org/html/rfc6376#page-29), -meaning emails sent from subdomains immediately fail verification. -The optional tag `h=sha256` blocks the use of the old SHA1 algorithm. - -Finally, there is the DMARC policy, which, unlike the others, -will need to be updated at the end of this guide. -For now, create another TXT record with these contents, -where `<admin>` is an email address of your choosing, -which may belong to the domain you're setting up for: -``` -TXT "v=DMARC1; aspf=s; adkim=s; p=none; sp=reject; fo=1; ruf=mailto:<admin>" -``` -The version tag `v=DMARC1` is required and must come first. -Next, `aspf=s` and `adkim=s` enable strict mode for both SPF and DKIM, -which once again blocks subdomains from passing the test. -Then `p=none` and `sp=reject` control what to do to failed messages -coming from the main domain and subdomains, respectively. -Unsurprisingly, `reject` means that delivery should be refused, -`none` asks to let it through anyway, and `quarantine` tells -the filter to take a closer look at the email or to put it in a spam folder. -Finally, `fo=1` asks the filter to create a forensic report -if any type of verification fails, and `ruf=` gives an address to send it to. - - - -## IMAP server: Dovecot - -[Dovecot](https://www.dovecot.org/) is a very popular IMAP server, -focused on being lightweight, simple, and secure. -Its documentation, while extensive and accurate, -is an absolute mess: the information you need is -usually either spread across multiple pages, -or buried somewhere two internal links deep. - -If you installed Dovecot via your package manager, -you'll probably have a lot of configuration files -in the `/etc/dovecot` directory and its subdirectories. -I want you to delete all of them. Yes, `rm -rf` that crap. - -Dovecot is simple to configure, and doesn't care where -you put its settings, so having all that chaos in `/etc/dovecot` -just makes things unnecessarily confusing. - -Create a new blank configuration file `/etc/dovecot/dovecot.conf`, -and start by filling in the details about your TLS certificate, -and making clear that unencrypted connections are unacceptable: -```sh -ssl = required -ssl_cert = </etc/ssl/certs/example.com.pem -ssl_key = </etc/ssl/private/example.com.key -ssl_dh = </etc/ssl/dhparam.pem - -ssl_prefer_server_ciphers = yes - -disable_plaintext_auth = yes -``` -The final `disable_plaintext_auth` option tells Dovecot -to reject any passwords that were sent unencrypted. -This means it must be [hashed](https://en.wikipedia.org/wiki/Cryptographic_hash_function) -or sent over a TLS connection, ideally both. - -Next, tell Dovecot which protocols to use -and where to expect them as follows: -```sh -protocols = lmtp imap - -service lmtp { - inet_listener lmtp { - address = 127.0.0.1 - port = 39999 - } - user = vmail -} -``` -LMTP is the Local Mail Transport Protocol, which is basically SMTP -but intended for exchange within a single server or cluster. -When an email is received, Dovecot will start a child process -under the `vmail` Unix user to deliver the message to its recipient. - -Since we set `ssl = required` earlier, MUAs (MRAs) will only get their mail -if the STARTTLS handshake was successful during the IMAP exchange, -or if they connect via IMAPS to force the use of encryption. -If you want to, you can set Dovecot to only accept IMAPS -by neutralizing the STARTTLS IMAP login listener: -```sh -service imap-login { - inet_listener imap { - port = 0 - } -} -``` - -Then we need to inform Dovecot which email users it should handle, -and what to do with their messages. Create a file `/etc/dovecot/users` for this, -which describes users in the same format as `/etc/passwd`: -``` -user:password:uid:gid::homedir -``` -You can leave `uid` and `gid` fields blank, since we're -using our `vmail` virtual mail user for all accounts. -You can even omit `homedir` if all users' directories -follow the same pattern, for example `/var/vmail/<user>/`. -The `user` name may contain the ending `@example.com`, but doesn't need to. -Create the password hash to put in the `password` field as follows: -```sh -# If your server is beefy and has lots of RAM: -doveadm pw -s ARGON2ID-CRYPT -# If you're using an old potato: -doveadm pw -s SHA512-CRYPT -``` -After you've entered your password, simply copy-paste the entire -hash string outputted by the program into the `password` field. - -Dovecot needs a file describing user accounts on two occasions: -* To check whether a user logging into the IMAP server - is valid and has given the right password. - This is handled by the `passdb` block(s) in the configuration. -* To know which email addresses the server is responsible for. - This is given by the `userdb` block(s) in the configuration. -These functions aren't necessarily fulfilled by the same file: -you can map multiple email addresses to one user acccount, -or multiple user accounts to one email address. -For simplicity, though, we'll use the same file: -```sh -passdb { - driver = passwd-file - args = scheme=ARGON2ID-CRYPT username_format=%n /etc/dovecot/users - #args = scheme=SHA512-CRYPT username_format=%n /etc/dovecot/users -} - -userdb { - driver = passwd-file - args = username_format=%n /etc/dovecot/users - # Use vmail for all users, and if all users' inboxes have the same form: - override_fields = uid=vmail gid=vmail home=/var/vmail/%n -} -``` -The `driver` option sets the kind of table Dovecot should expect. -We tell it to use a file in the `passwd`-like format described above, -but other common choices include SQL or the actual `/etc/passwd` file. -The options available in `args` depend on the chosen `driver`. - -The password hashing algorithm is given by `scheme`, while -`username_format` desribes the format of user names in the table: -if Dovecot receives an email for `someone@example.com`, -then for `%n` it will expect `someone` in `/etc/dovecot/users`, -but if you use `%u` it will want the full email address instead. - -In the `userdb` block we force the use of `vmail:vmail` -for all users, and tell Dovecot to put their mail in `/var/vmail/%n`, -where `%n` means the same as before. You can also use `%u` -for the full address again, or `%d` for the domain name. - -Optionally, you can create a catch-all inbox that will accept -all emails sent to your domain that don't match anyone in `/etc/dovecot/users`. -Just add this second `userdb` block after the first: -```sh -userdb { - driver = static - args = uid=vmail gid=vmail home=/var/vmail/catchall allow_all_users=yes -} -``` -The `static` driver means there is no table file: -all configuration is directly within this `userdb` block. -If we didn't specify `allow_all_users=yes`, then Dovecot -would check whether users existed using the `passdb` table. - -Finally, now that we've told Dovecot where the virtual mail users' `home` -folders are, we need to specify how to store the emails within them: -```sh -mail_location = maildir:~:LAYOUT=fs -``` -The two standard mailbox formats to choose from are `maildir` and `mbox`. -I highly recommend `maildir`; it's faster and more flexible than `mbox` -because it uses folders, whereas the older `mbox` uses a single database file. -The `~` tells Dovecot to directly put all mails in the account's home folder, -and `LAYOUT=fs` to user filesystem directories for you email folders. TODO? - - - - - -## MTA: Postfix - - -## MTA: Exim - - - -## MTA: OpenSMTPD - -OpenSMTPD is an MTA by the [OpenBSD](https://www.openbsd.org/) project, -who are known for their focus on security and minimalism. -I really like OpenSMTPD's flexible configuration format, -and its simple and intuitive way of passing around emails. - -It doesn't implement milter or any other filtering interface, -so DKIM signing and spam checking must be shoehorned in. -As such, only DKIMproxy can be used for DKIM signing. - -To begin, write your email domain on a single line in `/etc/smtpd/mailname`: -``` -example.com -``` -Next, you must fill in the `/etc/smtpd/aliases` file -which maps recipient addresses to system users: -``` -mailuser@example.com systemuser -``` -You can create a wildcard (catch-all) address for your domain -by omitting `mailuser`, such that any unknown recipients end up there. -You may also specify multiple system users as `systemuser1,systemuser2`. - -The first part of the actual config `/etc/smtpd/smtpd.conf` is as follows, -with DKIMproxy listening on port 19999 and outputting to 20000: -```sh -# Register TLS certificate to encrypt email transport: -pki "example.com" certificate "/etc/ssl/certs/example.com.pem" -pki "example.com" key "/etc/ssl/private/example.com.key" -pki "example.com" dhe auto - -table users "/etc/smtpd/aliases" - -# Relay emails submitted by client to DKIM signer on port 19999: -listen on eth0 port 465 smtps pki "example.com" tag SEND -listen on eth0 port 587 tls-require pki "example.com" tag SEND -accept tagged SEND from any for any relay via smtp://127.0.0.1:19999 - -# Relay outgoing DKIM-signed emails to their intended destination: -listen on lo port 20000 tag DKIM -accept tagged DKIM from local for any relay -``` -The lines starting with `pki` enable [DHE](https://en.wikipedia.org/wiki/Diffie%E2%80%93Hellman_key_exchange) -and tell where the public certificate and private key are for the domain. -We listen for client emails on both ports 465 (SMTPS) and 587 (SMTP), -although for the latter we refuse the submission if STARTTLS fails. - -If no spam filtering is desired, let OpenSMTPD deliver the email -by appending the following to the configuration file: -```sh -# Receive incoming emails and write them to the inbox: -listen on eth0 port 25 tls pki "example.com" -accept from any for domain "example.com" virtual <users> deliver to lmtp rcpt-to as vmail -``` -Here, we look up which system user corresponds to the recipient in `<users>`, -and deliver it to their maildir `mail` in their home folder. - - -### With SpamAssassin UNTESTED - -For SpamAssassin, we'll need to use [spampd](https://github.com/mpaperno/spampd) as an SMTP shim, -which we set to listen on port 29999 and relay to port 30000. -Add the `--nodetach` option in an init script or systemd service, -but leave it out when starting `spampd` by hand for testing: -```sh -$ spampd --host=127.0.0.1:29999 --relayhost=127.0.0.1:30000 [--nodetach] -``` -The rest of `smtpd`'s configuration is then as follows: -```sh -# Receive incoming emails and send them to spampd on port 29999: -listen on eth0 port 25 tls pki "example.com" tag SCAN -accept tagged SCAN from any for domain "example.com" relay via smtp://127.0.0.1:29999 - -# Receive filtered incoming emails and write them to the inbox: -listen on lo port 30000 tag DONE -accept tagged DONE from local for any virtual <users> deliver to maildir "~/mail" -``` -TODO - -### With Rspamd - - - - - - - - - -## DKIM signer: DKIMproxy - -[DKIMproxy](http://dkimproxy.sourceforge.net/) is a mature and simple SMTP proxy -that signs or verifies all emails passed through it -using the Perl Mail::DKIM module. - -In this guide we only use its signing feature, -which is controlled by `/etc/dkimproxy/dkimproxy_out.conf`. -This configuration file is pretty self-explanatory: -```sh -listen 127.0.0.1:19999 -relay 127.0.0.1:20000 - -domain example.com -selector mail - -keyfile /etc/dkim/private.key -signature dkim(c=relaxed/simple,a=rsa-sha256) -``` -This last line clearly tells it to create a DKIM signature, -because it also supports DKIM's predecessor, Yahoo! DomainKey. -The tag `a=rsa-sha256` specifies the signing algorithm, -while `c=relaxed/simple` gives the [canonicalization](https://tools.ietf.org/html/rfc6376#section-3.4) method. - -Canonicalization is a way to prevent the signature from becoming invalid -as the email might get reformatted by the MTAs on its journey. -Our choice `relaxed/simple` allows the header to be reformatted (`relaxed`), -but not the actual body of the message (`simple`). - - - - -## DKIM signer: OpenDKIM - -[OpenDKIM](http://www.opendkim.org/) is a milter program -that can both create and verify DKIM signatures. - - - - -## Spam filter: Rspamd - - - -## Spam filter: SpamAssassin - - - |