The implemented algorithm is called RMM-DIIS. It implicitly calculates an approximation of the inverse Hessian matrix by taking into account information from previous iterations. On startup, the initial Hessian matrix is diagonal and equal to POTIM. Information from old steps (which can lead to linear dependencies) is automatically removed from the iteration history, if required. The number of vectors kept in the iterations history (which corresponds to the rank of the Hessian matrix must not exceed the degrees of freedom. Naively the number of degrees of freedom is 3*(NIONS-1). But symmetry arguments or constraints can reduce this number significantly. There are two algorithms build in to remove information from the iteration history. i) If NFREE is set in the INCAR file, only up to NFREE ionic steps are kept in the iteration history (the rank of the approximate Hessian matrix is not larger than NFREE). ii) If NFREE is not specified, the criterion whether information is removed from the iteration history is based on the eigenvalue spectrum of the inverse Hessian matrix: if one eigenvalue of the inverse Hessian matrix is larger than 8, information from previous steps is discarded. For complex problems NFREE can usually be set to a rather large value (i.e. 10-20), however systems of low dimensionality require a carful setting of NFREE (or preferably an exact counting of the number of degrees of freedom). To increase NFREE beyond 20 rarely improves convergence. If NFREE is set to too large, the RMM-DIIS algorithm might diverge.
The choice of a reasonable POTIM is also important and can speed up calculations significantly, we recommend to find an optimal POTIM using IBRION=2 or performing a few test calculations (see below).