The memory requirements of VASP can easily exceed your computer facilities. In this case the first step is to estimate where the excessive memory requirements derive from. There are two possibilities:
NKDIM*NBANDS*NRPLWV*16The factor 16 arises from the fact that all quantities are COMPLEX*16.
4*(NGXF/2+1)*NGYF*NGZF*16Once again all quantities are COMPLEX*16.