The following sections contain some background information on the algorithms used in VASP. They do not contain a complete reference to all the things implemented in VASP but try to give hints on the most important topics. You should really understand at least the ideas touched here -- but it might be still possible to get good results without understanding all of it.
For a basic outline of pseudopotential plane wave programs we refer to [6,7]. Ultrasoft pseudopotentials are explained in [8,9,10,18]. An excellent introduction to PP plane wave codes - albeit in German - can be found in the thesis of J. Furthmüller . The best explanation of the algorithms found in VASP can be found in Ref. [13,14], these two papers give much more information than can be found in the following sections. And last but not least, you want might read the thesis of G. Kresse  (in German too) -- it contains a general discussion of PP including ultrasoft PP, and a discussion of the KS-functional and algorithms to calculate the KS-groundstate.