Post "Revisiting my email server in 2022"

7 files changed, 373 insertions, 35 deletions

diff --git a/content/blog/2020/email-server-extras.md b/content/blog/2020/email-server-extras.md
index 72299c9..5a72a84 100644
--- a/content/blog/2020/email-server-extras.md
+++ b/content/blog/2020/email-server-extras.md
@@ -7,11 +7,13 @@ draft: false
 
 #
 
-This sequel to my earlier [guide](/blog/2020/email-server/) discusses
-extra tips and tricks to extend your email setup.
-This page will be updated continuously as I come up with ideas.
+This sequel to my post
+"[Setting up an email server in 2020 with OpenSMTPD and Dovecot](/blog/2020/email-server/)"
+gives extra tips and tricks to extend your email setup.
+See also the sequel's sequel,
+"[Revisiting my email server in 2022](/blog/2022/email-server-revisited/)".
 
-Last updated 2020-04-29.
+Last updated on 2022-09-12.
 
 
 ## General
@@ -220,6 +222,13 @@ but I recommend against that for private servers: take a look at [this](https://
 You can configure OpenSMTPD to request a client certificate
 for sending emails, as a second factor for authentication.
 
+UPDATE: When I wrote this two years ago, it worked,
+but now it doesn't anymore, and I can't figure out why.
+It seems OpenSMTPD always rejects the client certificates for being self-signed,
+even if they can manually be verified for our CA using the `openssl` tool.
+I'm leaving this tutorial here for anyone who's interested,
+but it's unlikely I'll fix it anytime soon.
+
 
 #### Certificates
 
@@ -314,7 +323,17 @@ enter again when importing the certificate into the client.
 
 
 
-### Client certificates (instead of passwords)
+### ~~Client certificates (instead of passwords)~~
+
+UPDATE: Don't do this.
+As said above, OpenSMTPD's certificate verification is a mystery,
+so for all I know, if you follow the instructions in this subsection,
+you might find yourself running an *open* SMTP relay!
+That would be bad, because anyone on the Internet
+could send emails through your server with zero authentication.
+In theory, the client certificates act as authentication,
+but, again, the verification process is mysterious,
+so I'm just not confident enough to say.
 
 If you really want to, you can use the client certificates
 as a substitute for passwords. This is especially useful
diff --git a/content/blog/2020/email-server.md b/content/blog/2020/email-server.md
index a683fc7..ba03fcf 100644
--- a/content/blog/2020/email-server.md
+++ b/content/blog/2020/email-server.md
@@ -23,11 +23,12 @@ but your mileage may vary considerably. I hope you find it useful.
 This guide is aimed at people who are comfortable with
 the Linux/*BSD command line.
 
-When you're done, take a look at the
-[sequel](/blog/2020/email-server-extras/)
-for ideas to extend your setup.
+When you're done (if you get that far), take a look at the sequels
+"[Setting up an email server in 2020 with OpenSMTPD and Dovecot: extras](/blog/2020/email-server-extras/)"
+and "[Revisiting my email server in 2022](/blog/2022/email-server-revisited/)"
+for ideas on how to extend your setup.
 
-Last content update on 2020-04-29. Last correction on 2022-07-10.
+Last updated on 2022-09-12.
 
 
 
@@ -39,6 +40,7 @@ because you need to configure not one, but *two* server programs,
 I'll start by explaining the general structure of a mail server setup.
 
 
+
 ### How email works
 
 The programs involved in the exchange of emails are called [agents](https://en.wikipedia.org/wiki/Email_agent_(infrastructure)).
@@ -335,14 +337,14 @@ _dmarc.example.com. TXT "v=DMARC1; p=reject; sp=reject; pct=100; aspf=s; adkim=s
 The version tag `v=DMARC1` must come first,
 followed by `p=` and `sp=`, which control what to do to unverified messages
 coming from the main domain and subdomains, respectively.
-Unsurprisingly, `reject` means that delivery should be refused,
+Here, `reject` means that delivery should be refused,
 `none` asks to let it through anyway, and `quarantine` tells
 the filter to take a closer look or to put it in a spam folder.
 The percentage `pct=100` says how many of your emails to apply the policy to.
-Next, `aspf=s` and `adkim=s` enable strict mode for both SPF and DKIM,
-which blocks subdomains from passing the test.
-Finally, `fo=1` asks the filter to create a forensic report
-if any type of verification fails, and `ruf=` gives an address to send it to.
+Next, `aspf=s` and `adkim=s` enable strict mode for SPF and DKIM,
+which blocks subdomains from passing.
+Finally, `fo=1` asks for a forensic report if verification fails,
+and `ruf=` gives an address to send it to.
 If in doubt, see the [DMARC spec](https://tools.ietf.org/html/rfc7489).
 
 
@@ -520,12 +522,15 @@ and we'll let Dovecot manage the details by simply writing:
 @ vmail
 ```
 Then create a new file `/etc/smtpd/passwds`
-and fill it in according to the following format:
+and fill it in according to the following format,
+where `<user>` could be anything you want,
+not necessarily the same account name as in Dovecot:
 ```sh
-name@example.com <hash>
+<user> <hash>
 ```
 Generate the password hash with this command for each user.
-Be sure to use the same password for every account as in Dovecot:
+Again, the password doesn't need to be the same as in Dovecot,
+but for your own convenience it's probably best if it is:
 ```sh
 $ smtpctl encrypt '<password>'
 ```
@@ -550,8 +555,8 @@ example.com
 ```
 Then continue in `/etc/smtpd/smtpd.conf` by importing your TLS certificate:
 ```sh
-pki "prefet.ch" cert "/etc/ssl/certs/example.com.pem"
-pki "prefet.ch" key "/etc/ssl/private/example.com.key"
+pki "example.com" cert "/etc/ssl/certs/example.com.pem"
+pki "example.com" key "/etc/ssl/private/example.com.key"
 ```
 And tell OpenSMTPD which keys to use for the [Sender Rewriting Scheme](https://en.wikipedia.org/wiki/Sender_Rewriting_Scheme),
 which prevents forwarded emails from breaking SPF and looking like spam:
@@ -677,32 +682,40 @@ And... that's it! Of course, don't forget to start all the necessary daemons.
 
 Everything is set up now, so it's time to test. Fingers crossed!
 
-As mentioned earlier, you can check the correctness
+As mentioned earlier, you can check the validity
 of your DNS using the [MX Lookup](https://mxtoolbox.com/MXLookup.aspx) tool.
 You can use the same website to test OpenSMTPD
 with the [SMTP Diagnostics](https://mxtoolbox.com/diagnostic.aspx) tool.
 
 All decent email clients include an option to set the server for an email account.
-This guide excludes instructions for that,
+This guide doesn't include instructions for that,
 because it will vary a lot from client to client.
 If asked for your login type, choose plain/normal/unencrypted passwords.
 Don't worry, the client-server connection is TLS-encrypted,
 so nobody will be able to steal it.
 
-Next, to test sending and receiving messages,
-use the aptly-named [Is my email working?](http://ismyemailworking.com/) website.
-After that, specifically test that SPF, DKIM an DMARC
-are working correctly using the [DKIM validator](https://dkimvalidator.com/).
+Next, to verify that you can send and receive messages,
+and that your SPF/DKIM/DMARC is working,
+the following websites will be useful:
+
+* [~~Is my email working?~~](http://ismyemailworking.com/)
+  UPDATE: no longer available.
+* [DKIM validator](https://dkimvalidator.com/):
+  by sending an email, checks that SPF and DKIM are set up correctly.
+* [Email deliverability tool](https://mxtoolbox.com/deliverability):
+  tests basically everything.
+  As of writing, there's a bug that causes DKIM signatures to fail verification
+  even if you did it right; just ignore that.
+* [Learn and test DMARC](https://www.learndmarc.com/):
+  checks SPF, DKIM and DMARC in detail.
+
 If everything is good so far, congratulations!
 
 Now comes the big scary final test:
-sending an email to one of the "big guys", Google or Microsoft.
+sending an email to one of the "big guys", like Google, Yahoo or Microsoft.
 Their spam filters are very strict, so if you get through, great!
-Because these companies probably use AI-powered account-aware filters,
-you should *not* put anything identifiable in the test email.
-For example, if your name is "John Smith",
-do *not* put "John" nor "Smith" in the message,
-and do *not* send it from an addres like `john.smith@example.com`.
+Try to write your test message so that it doesn't look like spam,
+otherwise there's no point to this exercise.
 
 If something failed, then you have some investigating to do.
 Either one of the daemons is misconfigured, or there's a problem
diff --git a/content/blog/2022/email-server-revisited/index.md b/content/blog/2022/email-server-revisited/index.md
new file mode 100644
index 0000000..3f45993
--- /dev/null
+++ b/content/blog/2022/email-server-revisited/index.md
@@ -0,0 +1,304 @@
+---
+title: "Revisiting my email server in 2022"
+firstLetter: "R"
+date: 2022-07-25T10:56:53+02:00
+draft: false
+---
+
+#
+
+More than two years have passed since my first post about
+[setting up an email server in 2020 with OpenSMTPD and Dovecot](/blog/2020/email-server/)
+and [its sequel](/blog/2020/email-server-extras/).
+Since then, my server been going strong, with a few minor hiccups along the way.
+In this post, I'll explain some of the changes I made.
+
+This assumes you've followed the preceding guides:
+the configuration snippets given here
+should be interpreted as modifications to those guides,
+*not* as complete setups!
+
+Last updated on 2022-09-12.
+
+
+
+## More about DMARC
+
+When I wrote my original guide,
+I didn't properly understand how DMARC works:
+I misinterpreted it as an optional wrapper around SPF and DKIM.
+But oh God, I was wrong.
+Simon Andrews' article "[I figured out how DMARC works, and it almost broke me](https://simonandrews.ca/articles/how-to-set-up-spf-dkim-dmarc)"
+showed me the ugly truth,
+and I highly recommend reading it.
+Briefly, DMARC does two arguably unrelated things.
+
+Firstly, DMARC provides a way to diagnose issues with your SPF and DKIM configurations,
+in the form of reports that get sent to the `ruf=` and/or `rua=` email address(es)
+you put in the DNS record.
+Without this, there's no way of knowing why
+your emails are getting marked as spam.
+
+Secondly, it improves the trustworthiness of SPF and DKIM by enforcing *alignment*.
+This means something slightly different for SPF and DKIM,
+and boils down to fixing a glaring issue:
+
+* For some reason, in vanilla SMTP, it turns out that the email's `From:` header
+  doesn't need to agree with the address in the SMTP `MAIL FROM` command;
+  in other words, the server can claim a different sender that what's written in the message's header.
+  SPF only verifies the former (i.e. it takes the domain in `MAIL FROM`),
+  so one SMTP server can impersonate another.
+* An email's DKIM signature header states the domain of the signing server with the `d=` tag,
+  but, once again, that doesn't need to agree with the `From:` header's domain.
+  DKIM doesn't look at the latter, so an SMTP server can validly sign impersonated messages.
+
+DMARC's alignment refers to checking whether the domains match up for SPF and DKIM,
+thus ensuring that an SMTP server can't pretend to be someone else.
+It sounds obvious, but nope, apparently it wasn't before DMARC was made.
+
+
+
+## DKIMproxy
+
+To add DKIM signatures to my messages,
+I switched from Rspamd to [DKIMproxy](http://dkimproxy.sourceforge.net/).
+
+
+
+### Motivation
+
+To sign outgoing emails for DKIM, my original guide used Rspamd ---
+an unusual choice, since it's a spam filter designed to act on *incoming* messages.
+Later, in the [sequel](/blog/2020/email-server-extras/) to that guide,
+I needed some ugly workarounds to compensate for Rspamd's "smartness"
+when I tried to play around with authentication schemes in OpenSMTPD.
+Clearly, this wasn't ideal.
+
+However, the reason I cut my losses and switched to another DKIM signer
+was actually a bug in
+MXToolBox' [deliverability tool](https://mxtoolbox.com/deliverability).
+It appears that no matter what you do,
+this tool claims that your email's signature fails validation.
+I'm not the first to notice this issue:
+see e.g. [this question](https://serverfault.com/questions/1005818/dkim-validating-but-mxtoolbox-reports-as-dkim-signature-not-verified) on Server Fault.
+Other tools like the [DKIM validator](https://dkimvalidator.com/)
+say that my DKIM signatures are correct.
+
+There aren't many open-source alternatives out there for DKIM signing:
+the only ones I know of are [OpenDKIM](http://www.opendkim.org/)
+and DKIMproxy.
+The former is a so-called "milter",
+meaning it can only interact with MTAs via the milter API,
+which is only supported by [Sendmail](https://www.proofpoint.com/us/products/email-protection/open-source-email-solution)
+and [Postfix](https://www.postfix.org/).
+Since we're using OpenSMTPD,
+our only option is DKIMproxy,
+which consists of two daemons:
+`dkimproxy.out` to sign outgoing mail,
+and `dkimproxy.in` to verify incoming mail.
+We just need the former;
+Rspamd is still convenient for handling the latter's functionality.
+
+
+
+### DKIM settings
+
+Let's start by disabling Rspamd's DKIM signer
+in `/etc/rspamd/local.d/dkim_signing.conf`:
+```sh
+enabled = false;
+```
+Then configure `dkimproxy.out` as follows
+in `/etc/dkimproxy/dkimproxy_out.conf`.
+If you placed your DKIM public key
+in a TXT DNS record for `<selector>._domainkey.example.com.`,
+and stored your private key in `/path/to/dkim/private.key`,
+then:
+```sh
+# Receive emails on 10027, sign them, and forward them to 10028
+listen    127.0.0.1:10027
+relay     127.0.0.1:10028
+
+# Settings for email signing
+domain    example.com
+signature dkim(c=relaxed/relaxed,a=rsa-sha256)
+keyfile   /path/to/dkim/private.key
+selector  <selector>
+```
+Here, `rsa-sha256` is the signature algorithm
+(this is the best available, because DKIM is ancient),
+and `relaxed/relaxed` is the so-called *canonicalization* method,
+which is applied before signing and verification,
+to prevents failures if e.g. the email's whitespace gets changed in transit.
+
+
+
+### OpenSMTPD settings
+
+OpenSMTPD needs to send all outbound mail through `dkimproxy.out`.
+In `/etc/smtpd/smtpd.conf`, we tell it that all emails coming from the MUA
+must be relayed through `localhost:10027`, and then, after DKIM signing,
+picked up again on `localhost:10028`:
+```sh
+# Outbound
+listen on eth0 port 465 smtps       pki "example.com" auth <passwds> tag "TRUSTED"
+listen on eth0 port 587 tls-require pki "example.com" auth <passwds> tag "TRUSTED"
+action "SIGN" relay host "localhost:10027"
+match from any tag "TRUSTED" for any action "SIGN"
+
+listen on lo port 10028 tag "SIGNED"
+action "SEND" relay srs
+match from any tag "SIGNED" for any action "SEND"
+```
+The tag name `TRUSTED` reflects that only messages from trusted
+(i.e. authenticated) MUAs should be signed.
+After signing, emails get the tag `SIGNED`,
+and are sent to their destination as usual.
+
+
+
+## SMTP relay
+
+Instead of sending my emails directly to their destinations,
+I now send them to an SMTP relay server,
+which then passes them on to their actual destinations.
+
+
+
+### Motivation
+
+Large email providers such as Google, Microsoft and Yahoo
+manage many user accounts, so for them it makes sense
+to keep track of IP-based sender reputations.
+For example, if a number of low-quality emails are sent from a single IP
+to many of the accounts they manage,
+it's cheaper to simply blacklist that IP entirely at the MTA level,
+rather than passing each message through a computationally-intensive spam filter.
+
+But, as usual, Microsoft has to ruin everything with their draconic policies.
+In a stroke of genius,
+someone there decided to blindly ban IPs,
+seemingly in blocks belonging to VPS providers.
+One day, I tried to send an email to an Outlook-based account,
+and OpenSMTPD reported it had been unable to make the delivery,
+because Microsoft had thrown an error:
+
+<a href="microsoft-bounce.png">
+<img src="microsoft-bounce.png" style="width:100%;display:block;margin:auto;">
+</a>
+
+To their credit, they seem to be offering a way out.
+This approach is reasonable: preventively ban high-risk IP ranges,
+and allow "trustworthy" servers at the owner's request.
+I got error 5.7.511, asking me to send an email to a support address.
+If you're lucky, you may have a different error,
+and get the opportunity to use the slick [delist portal](https://sender.office.com/) instead.
+The URL in the bounce message links to [this list](https://go.microsoft.com/fwlink/?LinkId=526653) of error codes.
+
+I confess, I never actually bothered to forward the message to the provided address:
+my initial email was time-sensitive,
+so I couldn't afford to wait for Microsoft's response.
+Also, their customer support's stellar reputation precedes them,
+so I chose to use my time more wisely.
+Even if they would've resolved it nicely,
+there's nothing preventing Microsoft (or any other provider)
+from breaking my deliverability again in the future.
+Instead, I opted for a compromise.
+
+As a result of providers' IP reputation systems,
+a whole new business has appeared: SMTP relays.
+They offer to take the issue out of your hands:
+you send your emails through their servers,
+and they do their best to deliver them to large providers.
+SMTP relays are mostly used for sending marketing emails in bulk,
+but are also useful to avoid small-scale problems as described above.
+There are many SMTP relay services to choose from, at various prices.
+
+Using an SMTP relay results in more reliable delivery of your messages to large providers,
+but an obvious concern is privacy:
+the relay server can read all your outgoing emails (but not incoming),
+so you'll have to trust the service you choose.
+But it's no worse than using a major provider,
+and if you're sending sensitive material, why use email in the first place?
+Personally, I use [SMTP2GO](https://www.smtp2go.com/),
+but I can't say how good they are; do your own research.
+
+
+
+### OpenSMTPD settings
+
+Once you've chosen an SMTP relay service, let's say `relay.com`,
+and set up your account,
+they'll let you create credentials to use their SMTP servers.
+Suppose these credentials are `<username>` and `<password>`,
+create a file `/etc/smtpd/relaypw` with contents:
+
+```sh
+<label> <username>:<password>
+```
+
+Where `<label>` is a string of your choice.
+Note that `<password>` must be plaintext,
+because it needs to be provided to the relay server.
+To tell OpenSMTPD to use the relay,
+edit `/etc/smtpd/smtpd.conf` as follows,
+i.e. register the `relaypw` table and modify the `SEND` action:
+
+```sh
+table relaypw "/etc/smtpd/relaypw"
+
+#action "SEND" relay srs ### Replace this line with the following:
+action "SEND" relay host "smtps://<label>@relay.com:465" auth <relaypw> pki "example.com" srs
+```
+
+With `<label>` replaced by the label you chose earlier,
+and `relay.com:465` replaced by the host/port combination
+given in the relay service's documentation.
+Depending on what they support, you may also need to change the protocol `smtps://` (SMTP over TLS)
+to `smtp://` (SMTP with optional STARTTLS) or `smtp+tls://` (SMTP with mandatory STARTTLS).
+I recommend SMTPS.
+
+
+
+### DNS records
+
+If you've been paying attention so far, you have a burning question:
+what about SPF and stuff?
+Wasn't the point to prevent SMTP servers from sending emails on others' behalf?
+Well, yes, so you'll need to add some DNS records for the relay to work.
+The details depend on which service you choose,
+so they'll tell you what to do when you're setting up your account.
+As an example, based on my experience with SMTP2GO,
+you may need to add two CNAME records like:
+
+```sh
+emXXXXXX.example.com. CNAME return.relay.com.
+sXXXXXX._domainkey.example.com. CNAME dkim.relay.com.
+```
+
+Which, roughly speaking, respectively enable SPF and DKIM.
+Here, `XXXXXX` is an ID that the service will provide to you,
+since the DNS addresses must be unique.
+
+Technically, those two DNS records should be enough for SPF and DKIM,
+but in practice, it seems that different email providers/tools have slightly
+different interpretations of these standards,
+and can get confused when an email passes through multiple unaffiliated SMTP servers.
+Therefore, I recommend explicitly adding the relay to your SPF policy:
+
+```sh
+example.com TXT "v=spf1 mx include:spf.relay.com -all"
+```
+
+Your relay service might not publicly document their version of `spf.relay.com`,
+but you can find it by looking up your CNAME `emXXXXXX.example.com` (or equivalent)
+in MXToolBox' [ SPF tool](https://mxtoolbox.com/spf.aspx).
+
+I also recommend relaxing your DMARC domain policy for SPF and DKIM,
+such that your CNAME subdomains still pass the alignment checks:
+```sh
+_dmarc.example.com. TXT "v=DMARC1; aspf=r; adkim=r; ..."
+```
+
+Be sure to check that it's set up correctly
+using the website "[Learn and test DMARC](https://www.learndmarc.com/)".
diff --git a/content/blog/2022/email-server-revisited/microsoft-bounce.png b/content/blog/2022/email-server-revisited/microsoft-bounce.png
new file mode 100644
index 0000000..2d8a2e6
--- /dev/null
+++ b/content/blog/2022/email-server-revisited/microsoft-bounce.png
diff --git a/content/know/concept/calculus-of-variations/index.pdc b/content/know/concept/calculus-of-variations/index.pdc
index a8861cb..26c5753 100644
--- a/content/know/concept/calculus-of-variations/index.pdc
+++ b/content/know/concept/calculus-of-variations/index.pdc
@@ -14,7 +14,7 @@ markup: pandoc
 # Calculus of variations
 
 The **calculus of variations** lays the mathematical groundwork
-for Lagrangian mechanics.
+for [Lagrangian mechanics](/know/concept/lagrangian-mechanics/).
 
 Consider a **functional** $J$, mapping a function $f(x)$ to a scalar value
 by integrating over the so-called **Lagrangian** $L$,
@@ -28,6 +28,8 @@ If $J$ in some way measures the physical "cost" (e.g. energy) of
 the path $f(x)$ taken by a physical system,
 the **principle of least action** states that $f$ will be a minimum of $J[f]$,
 so for example the expended energy will be minimized.
+In practice, various cost metrics may be used,
+so maxima of $J[f]$ are also interesting to us.
 
 If $f(x, \varepsilon\!=\!0)$ is the optimal route, then a slightly
 different (and therefore worse) path between the same two points can be expressed
@@ -238,7 +240,7 @@ $$\begin{aligned}
 
 ## Constraints
 
-So far, for multiple functions $f_1, ... f_N$,
+So far, for multiple functions $f_1, ..., f_N$,
 we have been assuming that all $f_n$ are independent, and by extension all $\eta_n$.
 Suppose that we now have $M < N$ constraints $\phi_m$
 that all $f_n$ need to obey, introducing implicit dependencies between them.
diff --git a/content/know/concept/pulay-mixing/index.pdc b/content/know/concept/pulay-mixing/index.pdc
index 4e7a411..476932d 100644
--- a/content/know/concept/pulay-mixing/index.pdc
+++ b/content/know/concept/pulay-mixing/index.pdc
@@ -125,7 +125,7 @@ $$\begin{aligned}
 You might be confused by the absence of any $\rho_m^\mathrm{new}$
 in the creation of $\rho_{n+1}$, as if the iteration's outputs are being ignored.
 This is due to the first assumption,
-which states that $\rho_n^\mathrm{new}$ are $\rho_n$ are already similar,
+which states that $\rho_n^\mathrm{new}$ and $\rho_n$ are already similar,
 such that they are basically interchangeable.
 
 Speaking of which, about those assumptions:
diff --git a/content/know/concept/rayleigh-plesset-equation/index.pdc b/content/know/concept/rayleigh-plesset-equation/index.pdc
index 9325f3f..75feb13 100644
--- a/content/know/concept/rayleigh-plesset-equation/index.pdc
+++ b/content/know/concept/rayleigh-plesset-equation/index.pdc
@@ -20,7 +20,7 @@ inside an incompressible liquid.
 Notably, it leads to [cavitation](/know/concept/cavitation/).
 
 Consider the main
-[Navier-Stokes equations](/know/concept/navier-stokes-equations/)
+[Navier-Stokes equation](/know/concept/navier-stokes-equations/)
 for the velocity field $\va{v}$:
 
 $$\begin{aligned}