Categories: Numerical methods.
Some numerical problems are most easily solved iteratively, by generating a series of “solutions” , , etc. converging towards the true . Pulay mixing, also often called direct inversion in the iterative subspace (DIIS), can speed up the convergence for some types of problems, and also helps to avoid periodic divergences.
The key concept it relies on is the residual vector of the th iteration, which measures the error of the current . Its exact definition can vary, but it is generally the difference between the input of the th iteration and the raw resulting output :
It is not always clear what to do with . Directly using it as the next input () often leads to oscillation, and linear mixing () can take a very long time to converge properly. Pulay mixing offers an improvement.
The idea is to construct the next iteration’s input as a linear combination of the previous inputs , , …, , such that it is as close as possible to the optimal :
To do so, we make two assumptions. First, that the current is already close to , so such a linear combination makes sense. Second, that the iteration is linear, so the raw output is also a linear combination with the same coefficients:
We will revisit these assumptions later. The point is that can now also be written as a linear combination of old residuals :
The goal is to choose the coefficients such that the norm of the error , subject to the following constraint to preserve the normalization of :
We thus want to minimize the following quantity, where is a Lagrange multiplier:
By differentiating the right-hand side with respect to and demanding that the result is zero, we get a cheap-to-solve system of equations, in matrix form:
From this, we can also see that the Lagrange multiplier , where is the predicted residual of the next iteration, subject to the two assumptions. This fact makes a useful measure of convergence.
In practice, the earlier inputs , , etc. are much further from than , so usually only the most recent inputs are used. This also keeps the matrix small:
You might be confused by the absence of any in the creation of , as if the iteration’s outputs are being ignored. This is due to the first assumption, which states that and are already similar, such that they are basically interchangeable.
Although those assumptions will clearly become more realistic as , they might be very dubious at first. Consequently, the early iterations may get “trapped” in a suboptimal subspace spanned by , , etc. Think of it like this: we would be varying up to coefficients to try to optimize a -dimensional , where usually . It is almost impossible to find a decent optimum in this way!
This problem is easy to fix, by mixing in a small amount of the raw residuals to “nudge” towards the right subspace, where is a tunable parameter:
In this way, the raw outputs are (rightfully) included via , but we still give more weight to iterations with smaller residuals.
Pulay mixing is very effective for certain types of problems, e.g. density functional theory, where it can accelerate convergence by up to two orders of magnitude!
- P. Pulay, Convergence acceleration of iterative sequences. The case of SCF iteration, 1980, Elsevier.