Although VASP.4 was initially optimised for the T3D (and T3E), the support for shmem communication is now only very rudimentary, and might not even work. To make use of the efficient T3D (T3E) shmem communication scheme, specify T3D_SMA in the makefile. This might speed up communication by up to a factor of 2. But, mind that this can also cause problems on the T3E if VASP is used with data-streams:
export SCACHE_D_STREAMS=1The default makefile on the T3E, therefore does not use the optimised communication routines, because performance improvements due to data-streams are usually more important than optimised communication (it is thus safe to switch on data streaming on the T3E typing i.e. export SCACHE_D_STREAMS=1).