Segmentation fault after "entering main loop"
Posted: Mon Jul 15, 2013 12:26 pm
I?m trying to get VASP running on my university?s cluster system (DTU in Denmark). VASP compiles with MPI enabled and the Intel MKL library and it?s also able to calculate many small problems just fine. However, I?ve been struggling with a segmentation fault for a specific FeN relaxation problem for a very long time now, still without any clue about the cause. This specific calculation runs without problems on my local 6-core workstation (also running Open-MPI) as well as on another cluster with a precompiled version of VASP I?ve access to.
When running the problem on the university cluster on more than one node (e.g. nodes=2:ppn=8), the calculation always fails with a segfault after ?entering main loop?. If only one node is used, the calculation may pass this point. I've tried to using Open-MPI 1.6.5 compiled by myself as well as the version provided by the cluster. Also ?ulimit -s unlimited? is used as suggested elsewhere. So the problem seems to be related to the usage of Open-MPI across the InfiniBand network, but I?ve not been able to make any further progress from here, so help is really appreciated.
Below are the console output as well as my VASP Makefile and the INCAR and POSCAR files:
running on 16 total cores
distrk: each k-point on 16 cores, 1 groups
distr: one band on 1 cores, 16 groups
using from now: INCAR
vasp.5.3.3 18Dez12 (build Jul 15 2013 11:04:39) complex
POSCAR found type information on POSCAR Fe N
POSCAR found : 2 types and 48 ions
-----------------------------------------------------------------------------
| |
| W W AA RRRRR N N II N N GGGG !!! |
| W W A A R R NN N II NN N G G !!! |
| W W A A R R N N N II N N N G !!! |
| W WW W AAAAAA RRRRR N N N II N N N G GGG ! |
| WW WW A A R R N NN II N NN G G |
| W W A A R R N N II N N GGGG !!! |
| |
| For optimal performance we recommend that you set |
| NPAR = 4 - approx SQRT( number of cores) |
| (number of cores/NPAR must be integer) |
| This setting will greatly improve the performance of VASP for DFT. |
| The default NPAR=number of cores might be grossly inefficient |
| on modern multi-core architectures or massively parallel machines. |
| Do your own testing. |
| Unfortunately you need to use the default for hybrid, GW and RPA |
| calculations. |
| |
-----------------------------------------------------------------------------
-----------------------------------------------------------------------------
| |
| ADVICE TO THIS USER RUNNING 'VASP/VAMP' (HEAR YOUR MASTER'S VOICE ...): |
| |
| You have a (more or less) 'large supercell' and for larger cells |
| it might be more efficient to use real space projection opertators |
| So try LREAL= Auto in the INCAR file. |
| Mind: At the moment your POTCAR file does not contain real space |
| projectors, and has to be modified, BUT if you |
| want to do an extremely accurate calculation you might also keep the |
| reciprocal projection scheme (i.e. LREAL=.FALSE.) |
| |
-----------------------------------------------------------------------------
LDA part: xc-table for Pade appr. of Perdew
generate k-points for: 6 6 5
POSCAR, INCAR and KPOINTS ok, starting setup
FFT: planning ...
WAVECAR not read
entering main loop
N E dE d eps ncg rms rms(c)
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
libiomp5.so 00002AF0527942F5 Unknown Unknown Unknown
libiomp5.so 00002AF052790ED4 Unknown Unknown Unknown
libiomp5.so 00002AF05278FD14 Unknown Unknown Unknown
libiomp5.so 00002AF05278FDFD Unknown Unknown Unknown
libiomp5.so 00002AF05276D5E7 Unknown Unknown Unknown
libmkl_intel_thre 00002AF050728DA5 Unknown Unknown Unknown
libmkl_intel_thre 00002AF0506AB0C6 Unknown Unknown Unknown
libmkl_intel_lp64 00002AF04FF61A7F Unknown Unknown Unknown
vasp 0000000000649EF8 Unknown Unknown Unknown
vasp 00000000006859C4 Unknown Unknown Unknown
vasp 0000000000D14CE8 Unknown Unknown Unknown
vasp 0000000000D399AA Unknown Unknown Unknown
vasp 000000000045AEBB Unknown Unknown Unknown
vasp 000000000043599C Unknown Unknown Unknown
libc.so.6 0000003E32E1ECDD Unknown Unknown Unknown
vasp 0000000000435829 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
libc.so.6 0000003E32ECCDD7 Unknown Unknown Unknown
libiomp5.so 00002B9912C01B29 Unknown Unknown Unknown
libiomp5.so 00002B9912BE5332 Unknown Unknown Unknown
libiomp5.so 00002B9912BE1ED4 Unknown Unknown Unknown
libiomp5.so 00002B9912BE0D14 Unknown Unknown Unknown
libiomp5.so 00002B9912BE0DFD Unknown Unknown Unknown
libiomp5.so 00002B9912BBE5E7 Unknown Unknown Unknown
libmkl_intel_thre 00002B9910B79DA5 Unknown Unknown Unknown
libmkl_intel_thre 00002B9910AFC0C6 Unknown Unknown Unknown
libmkl_intel_lp64 00002B99103B2A7F Unknown Unknown Unknown
vasp 0000000000649EF8 Unknown Unknown Unknown
vasp 00000000006859C4 Unknown Unknown Unknown
vasp 0000000000D14CE8 Unknown Unknown Unknown
vasp 0000000000D399AA Unknown Unknown Unknown
vasp 000000000045AEBB Unknown Unknown Unknown
vasp 000000000043599C Unknown Unknown Unknown
libc.so.6 0000003E32E1ECDD Unknown Unknown Unknown
vasp 0000000000435829 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
mca_btl_openib.so 00002B14472E8ECE Unknown Unknown Unknown
libmpi.so.1 00002B14424763DA Unknown Unknown Unknown
libmpi.so.1 00002B14423C582D Unknown Unknown Unknown
mca_coll_tuned.so 00002B14487C5967 Unknown Unknown Unknown
libmpi.so.1 00002B14423D1796 Unknown Unknown Unknown
libmpi_f77.so.1 00002B1442154EEA Unknown Unknown Unknown
vasp 000000000048787F Unknown Unknown Unknown
vasp 00000000005D5D4F Unknown Unknown Unknown
vasp 0000000000D18521 Unknown Unknown Unknown
vasp 0000000000D399AA Unknown Unknown Unknown
vasp 000000000045AEBB Unknown Unknown Unknown
vasp 000000000043599C Unknown Unknown Unknown
libc.so.6 0000003E32E1ECDD Unknown Unknown Unknown
vasp 0000000000435829 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
mca_btl_sm.so 00002B9DF2B6B89F Unknown Unknown Unknown
libmpi.so.1 00002B9DED8D43DA Unknown Unknown Unknown
libmpi.so.1 00002B9DED82382D Unknown Unknown Unknown
mca_coll_tuned.so 00002B9DF3C23967 Unknown Unknown Unknown
libmpi.so.1 00002B9DED82F796 Unknown Unknown Unknown
libmpi_f77.so.1 00002B9DED5B2EEA Unknown Unknown Unknown
vasp 000000000048787F Unknown Unknown Unknown
vasp 00000000005D5D4F Unknown Unknown Unknown
vasp 0000000000D18521 Unknown Unknown Unknown
vasp 0000000000D399AA Unknown Unknown Unknown
vasp 000000000045AEBB Unknown Unknown Unknown
vasp 000000000043599C Unknown Unknown Unknown
libc.so.6 0000003E32E1ECDD Unknown Unknown Unknown
vasp 0000000000435829 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
libmlx4-rdmav2.so 00002B81E1AD6378 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
mca_btl_openib.so 00002AE907397FE6 Unknown Unknown Unknown
libmpi.so.1 00002AE9025253DA Unknown Unknown Unknown
libmpi.so.1 00002AE90247482D Unknown Unknown Unknown
mca_coll_tuned.so 00002AE908874967 Unknown Unknown Unknown
libmpi.so.1 00002AE902480796 Unknown Unknown Unknown
libmpi_f77.so.1 00002AE902203EEA Unknown Unknown Unknown
vasp 000000000048787F Unknown Unknown Unknown
vasp 00000000005D5D4F Unknown Unknown Unknown
vasp 0000000000D18521 Unknown Unknown Unknown
vasp 0000000000D399AA Unknown Unknown Unknown
vasp 000000000045AEBB Unknown Unknown Unknown
vasp 000000000043599C Unknown Unknown Unknown
libc.so.6 0000003E32E1ECDD Unknown Unknown Unknown
vasp 0000000000435829 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
libpthread.so.0 0000003E33A0C113 Unknown Unknown Unknown
libmlx4-rdmav2.so 00002AC77E1C4A2B Unknown Unknown Unknown
mca_btl_openib.so 00002AC77C49C52C Unknown Unknown Unknown
mca_btl_openib.so 00002AC77C49D089 Unknown Unknown Unknown
libmpi.so.1 00002AC77762A3DA Unknown Unknown Unknown
libmpi.so.1 00002AC77757982D Unknown Unknown Unknown
mca_coll_tuned.so 00002AC77D979967 Unknown Unknown Unknown
libmpi.so.1 00002AC777585796 Unknown Unknown Unknown
libmpi_f77.so.1 00002AC777308EEA Unknown Unknown Unknown
vasp 000000000048787F Unknown Unknown Unknown
vasp 00000000005D5D4F Unknown Unknown Unknown
vasp 0000000000D18521 Unknown Unknown Unknown
vasp 0000000000D399AA Unknown Unknown Unknown
vasp 000000000045AEBB Unknown Unknown Unknown
vasp 000000000043599C Unknown Unknown Unknown
libc.so.6 0000003E32E1ECDD Unknown Unknown Unknown
vasp 0000000000435829 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
mca_btl_openib.so 00002B163FB7107F Unknown Unknown Unknown
libmpi.so.1 00002B163ACFE3DA Unknown Unknown Unknown
libmpi.so.1 00002B163AC4D82D Unknown Unknown Unknown
mca_coll_tuned.so 00002B164104D967 Unknown Unknown Unknown
libmpi.so.1 00002B163AC59796 Unknown Unknown Unknown
libmpi_f77.so.1 00002B163A9DCEEA Unknown Unknown Unknown
vasp 000000000048787F Unknown Unknown Unknown
vasp 00000000005D5D4F Unknown Unknown Unknown
vasp 0000000000D18521 Unknown Unknown Unknown
vasp 0000000000D399AA Unknown Unknown Unknown
vasp 000000000045AEBB Unknown Unknown Unknown
vasp 000000000043599C Unknown Unknown Unknown
libc.so.6 0000003E32E1ECDD Unknown Unknown Unknown
vasp 0000000000435829 Unknown Unknown Unknown
--------------------------------------------------------------------------
mpirun noticed that process rank 9 with PID 19577 on node n-62-23-9 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
real 4m58.211s
user 69m43.740s
sys 3m7.745s
Makefile
========
.SUFFIXES: .inc .f .f90 .F
#-----------------------------------------------------------------------
# Makefile for Intel Fortran compiler for Pentium/Athlon/Opteron
# based systems
# we recommend this makefile for both Intel as well as AMD systems
# for AMD based systems appropriate BLAS (libgoto) and fftw libraries are
# however mandatory (whereas they are optional for Intel platforms)
# For Athlon we recommend
# ) to link against libgoto (and mkl as a backup for missing routines)
# ) odd enough link in libfftw3xf_intel.a (fftw interface for mkl)
# feedback is greatly appreciated
#
# The makefile was tested only under Linux on Intel and AMD platforms
# the following compiler versions have been tested:
# - ifc.7.1 works stable somewhat slow but reliably
# - ifc.8.1 fails to compile the code properly
# - ifc.9.1 recommended (both for 32 and 64 bit)
# - ifc.10.1 partially recommended (both for 32 and 64 bit)
# tested build 20080312 Package ID: l_fc_p_10.1.015
# the gamma only mpi version can not be compiles
# using ifc.10.1
# - ifc.11.1 partially recommended (some problems with Gamma only and intel fftw)
# Build 20090630 Package ID: l_cprof_p_11.1.046
# - ifort.12.1 strongly recommended (we use this to compile vasp)
# Version 12.1.5.339 Build 20120612
#
# it might be required to change some of library path ways, since
# LINUX installations vary a lot
#
# Hence check ***ALL*** options in this makefile very carefully
#-----------------------------------------------------------------------
#
# BLAS must be installed on the machine
# there are several options:
# 1) very slow but works:
# retrieve the lapackage from ftp.netlib.org
# and compile the blas routines (BLAS/SRC directory)
# please use g77 or f77 for the compilation. When I tried to
# use pgf77 or pgf90 for BLAS, VASP hang up when calling
# ZHEEV (however this was with lapack 1.1 now I use lapack 2.0)
# 2) more desirable: get an optimized BLAS
#
# the two most reliable packages around are presently:
# 2a) Intels own optimised BLAS (PIII, P4, PD, PC2, Itanium)
# http://developer.intel.com/software/products/mkl/
# this is really excellent, if you use Intel CPU's
#
# 2b) probably fastest SSE2 (4 GFlops on P4, 2.53 GHz, 16 GFlops PD,
# around 30 GFlops on Quad core)
# Kazushige Goto's BLAS
# http://www.cs.utexas.edu/users/kgoto/signup_first.html
# http://www.tacc.utexas.edu/resources/software/
#
#-----------------------------------------------------------------------
# all CPP processed fortran files have the extension .f90
SUFFIX=.f90
#-----------------------------------------------------------------------
# fortran compiler and linker
#-----------------------------------------------------------------------
FC=ifort
# fortran linker
FCL=$(FC)
#-----------------------------------------------------------------------
# whereis CPP ?? (I need CPP, can't use gcc with proper options)
# that's the location of gcc for SUSE 5.3
#
# CPP_ = /usr/lib/gcc-lib/i486-linux/2.7.2/cpp -P -C
#
# that's probably the right line for some Red Hat distribution:
#
# CPP_ = /usr/lib/gcc-lib/i386-redhat-linux/2.7.2.3/cpp -P -C
#
# SUSE X.X, maybe some Red Hat distributions:
CPP_ = ./preprocess <$*.F | /usr/bin/cpp -P -C -traditional >$*$(SUFFIX)
# this release should be fpp clean
# we now recommend fpp as preprocessor
# if this fails go back to cpp
CPP_=fpp -f_com=no -free -w0 $*.F $*$(SUFFIX)
#-----------------------------------------------------------------------
# possible options for CPP:
# NGXhalf charge density reduced in X direction
# wNGXhalf gamma point only reduced in X direction
# avoidalloc avoid ALLOCATE if possible
# PGF90 work around some for some PGF90 / IFC bugs
# CACHE_SIZE 1000 for PII,PIII, 5000 for Athlon, 8000-12000 P4, PD
# RPROMU_DGEMV use DGEMV instead of DGEMM in RPRO (depends on used BLAS)
# RACCMU_DGEMV use DGEMV instead of DGEMM in RACC (depends on used BLAS)
# tbdyn MD package of Tomas Bucko
#-----------------------------------------------------------------------
CPP = $(CPP_) -DHOST=\"LinuxIFC\" \
-DCACHE_SIZE=12000 -DPGF90 -Davoidalloc -DNGXhalf \
# -DRPROMU_DGEMV -DRACCMU_DGEMV
#-----------------------------------------------------------------------
# general fortran flags (there must a trailing blank on this line)
# byterecl is strictly required for ifc, since otherwise
# the WAVECAR file becomes huge
#-----------------------------------------------------------------------
FFLAGS = -FR -names lowercase -assume byterecl
#-----------------------------------------------------------------------
# optimization
# we have tested whether higher optimisation improves performance
# -axK SSE1 optimization, but also generate code executable on all mach.
# xK improves performance somewhat on XP, and a is required in order
# to run the code on older Athlons as well
# -xW SSE2 optimization
# -axW SSE2 optimization, but also generate code executable on all mach.
# -tpp6 P3 optimization
# -tpp7 P4 optimization
#-----------------------------------------------------------------------
# ifc.9.1, ifc.10.1 recommended
OFLAG=-O2 -ip
OFLAG_HIGH = $(OFLAG)
OBJ_HIGH =
OBJ_NOOPT =
DEBUG = -FR -O0
INLINE = $(OFLAG)
#-----------------------------------------------------------------------
# the following lines specify the position of BLAS and LAPACK
# we recommend to use mkl, that is simple and most likely
# fastest in Intel based machines
#-----------------------------------------------------------------------
# mkl path for ifc 11 compiler
#MKL_PATH=$(MKLROOT)/lib/em64t
# mkl path for ifc 12 compiler
MKL_PATH=$(MKLROOT)/lib/intel64
MKL_FFTW_PATH=$(HOME)/local/vasp/fftw3xf/
# BLAS
# setting -DRPROMU_DGEMV -DRACCMU_DGEMV in the CPP lines usually speeds up program execution
# BLAS= -Wl,--start-group $(MKL_PATH)/libmkl_intel_lp64.a $(MKL_PATH)/libmkl_intel_thread.a $(MKL_PATH)/libmkl_core.a -Wl,--end-group -
lguide
# faster linking and available from at least version 11
#BLAS= -lguide -mkl
# LAPACK, use vasp.5.lib/lapack_double
#LAPACK= ../vasp.5.lib/lapack_double.o
# LAPACK from mkl, usually faster and contains scaLAPACK as well
#LAPACK= $(MKL_PATH)/libmkl_intel_lp64.a
# here a tricky version, link in libgoto and use mkl as a backup
# also needs a special line for LAPACK
# this is the best thing you can do on AMD based systems !!!!!!
#BLAS = -Wl,--start-group /opt/libs/libgoto/libgoto.so $(MKL_PATH)/libmkl_intel_thread.a $(MKL_PATH)/libmkl_core.a -Wl,--end-group -
liomp5
#LAPACK= /opt/libs/libgoto/libgoto.so $(MKL_PATH)/libmkl_intel_lp64.a
#-----------------------------------------------------------------------
LIB = -L../vasp.5.lib -ldmy \
../vasp.5.lib/linpack_double.o $(LAPACK) \
$(BLAS)
# options for linking, nothing is required (usually)
LINK =
#-----------------------------------------------------------------------
# fft libraries:
# VASP.5.2 can use fftw.3.1.X (http://www.fftw.org)
# since this version is faster on P4 machines, we recommend to use it
#-----------------------------------------------------------------------
#FFT3D = fft3dfurth.o fft3dlib.o
# alternatively: fftw.3.1.X is slighly faster and should be used if available
#FFT3D = fftw3d.o fft3dlib.o /opt/libs/fftw-3.1.2/lib/libfftw3.a
# you may also try to use the fftw wrapper to mkl (but the path might vary a lot)
# it seems this is best for AMD based systems
#FFT3D = fftw3d.o fft3dlib.o $(MKL_FFTW_PATH)/libfftw3xf_gnu.a
#INCS = -I$(MKLROOT)/include/fftw
#=======================================================================
# MPI section, uncomment the following lines until
# general rules and compile lines
# presently we recommend OPENMPI, since it seems to offer better
# performance than lam or mpich
#
# !!! Please do not send me any queries on how to install MPI, I will
# certainly not answer them !!!!
#=======================================================================
#-----------------------------------------------------------------------
# fortran linker for mpi
#-----------------------------------------------------------------------
FC=mpif90
FCL=$(FC) -mkl
#-----------------------------------------------------------------------
# additional options for CPP in parallel version (see also above):
# NGZhalf charge density reduced in Z direction
# wNGZhalf gamma point only reduced in Z direction
# scaLAPACK use scaLAPACK (recommended if mkl is available)
# avoidalloc avoid ALLOCATE if possible
# PGF90 work around some for some PGF90 / IFC bugs
# CACHE_SIZE 1000 for PII,PIII, 5000 for Athlon, 8000-12000 P4, PD
# RPROMU_DGEMV use DGEMV instead of DGEMM in RPRO (depends on used BLAS)
# RACCMU_DGEMV use DGEMV instead of DGEMM in RACC (depends on used BLAS)
# tbdyn MD package of Tomas Bucko
#-----------------------------------------------------------------------
#-----------------------------------------------------------------------
CPP = $(CPP_) -DMPI -DHOST=\"LinuxIFC\" -DIFC \
-DCACHE_SIZE=4000 -DPGF90 -Davoidalloc -DNGZhalf \
-DMPI_BLOCK=8000 -Duse_collective
# -DscaLAPACK
## -DRPROMU_DGEMV -DRACCMU_DGEMV
#-----------------------------------------------------------------------
# location of SCALAPACK
# if you do not use SCALAPACK simply leave this section commented out
#-----------------------------------------------------------------------
# usually simplest link in mkl scaLAPACK
#BLACS= -lmkl_blacs_openmpi_lp64
#SCA= $(MKL_PATH)/libmkl_scalapack_lp64.a $(BLACS)
#-----------------------------------------------------------------------
# libraries
#-----------------------------------------------------------------------
#LIB = -L../vasp.5.lib -ldmy \
# ../vasp.5.lib/linpack_double.o \
# $(SCA) $(LAPACK) $(BLAS)
#-----------------------------------------------------------------------
# parallel FFT
#-----------------------------------------------------------------------
# FFT: fftmpi.o with fft3dlib of Juergen Furthmueller
#FFT3D = fftmpi.o fftmpi_map.o fft3dfurth.o fft3dlib.o
# alternatively: fftw.3.1.X is slighly faster and should be used if available
#FFT3D = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o /opt/libs/fftw-3.1.2/lib/libfftw3.a
# you may also try to use the fftw wrapper to mkl (but the path might vary a lot)
# it seems this is best for AMD based systems
FFT3D = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o $(MKL_FFTW_PATH)/libfftw3xf_gnu.a
INCS = -I$(MKLROOT)/include/fftw
#-----------------------------------------------------------------------
# general rules and compile lines
#-----------------------------------------------------------------------
BASIC= symmetry.o symlib.o lattlib.o random.o
SOURCE= base.o mpi.o smart_allocate.o xml.o \
constant.o jacobi.o main_mpi.o scala.o \
asa.o lattice.o poscar.o ini.o mgrid.o xclib.o vdw_nl.o xclib_grad.o \
radial.o pseudo.o gridq.o ebs.o \
mkpoints.o wave.o wave_mpi.o wave_high.o spinsym.o \
$(BASIC) nonl.o nonlr.o nonl_high.o dfast.o choleski2.o \
mix.o hamil.o xcgrad.o xcspin.o potex1.o potex2.o \
constrmag.o cl_shift.o relativistic.o LDApU.o \
paw_base.o metagga.o egrad.o pawsym.o pawfock.o pawlhf.o rhfatm.o hyperfine.o paw.o \
mkpoints_full.o charge.o Lebedev-Laikov.o stockholder.o dipol.o pot.o \
dos.o elf.o tet.o tetweight.o hamil_rot.o \
chain.o dyna.o k-proj.o sphpro.o us.o core_rel.o \
aedens.o wavpre.o wavpre_noio.o broyden.o \
dynbr.o hamil_high.o rmm-diis.o reader.o writer.o tutor.o xml_writer.o \
brent.o stufak.o fileio.o opergrid.o stepver.o \
chgloc.o fast_aug.o fock_multipole.o fock.o mkpoints_change.o sym_grad.o \
mymath.o internals.o npt_dynamics.o dynconstr.o dimer_heyden.o dvvtrajectory.o vdwforcefield.o \
nmr.o pead.o subrot.o subrot_scf.o \
force.o pwlhf.o gw_model.o optreal.o steep.o davidson.o david_inner.o \
electron.o rot.o electron_all.o shm.o pardens.o paircorrection.o \
optics.o constr_cell_relax.o stm.o finite_diff.o elpol.o \
hamil_lr.o rmm-diis_lr.o subrot_cluster.o subrot_lr.o \
lr_helper.o hamil_lrf.o elinear_response.o ilinear_response.o \
linear_optics.o \
setlocalpp.o wannier.o electron_OEP.o electron_lhf.o twoelectron4o.o \
mlwf.o ratpol.o screened_2e.o wave_cacher.o chi_base.o wpot.o \
local_field.o ump2.o ump2kpar.o fcidump.o ump2no.o \
bse_te.o bse.o acfdt.o chi.o sydmat.o dmft.o \
rmm-diis_mlr.o linear_response_NMR.o wannier_interpol.o linear_response.o
vasp: $(SOURCE) $(FFT3D) $(INC) main.o
rm -f vasp
$(FCL) -o vasp main.o $(SOURCE) $(FFT3D) $(LIB) $(LINK)
makeparam: $(SOURCE) $(FFT3D) makeparam.o main.F $(INC)
$(FCL) -o makeparam $(LINK) makeparam.o $(SOURCE) $(FFT3D) $(LIB)
zgemmtest: zgemmtest.o base.o random.o $(INC)
$(FCL) -o zgemmtest $(LINK) zgemmtest.o random.o base.o $(LIB)
dgemmtest: dgemmtest.o base.o random.o $(INC)
$(FCL) -o dgemmtest $(LINK) dgemmtest.o random.o base.o $(LIB)
ffttest: base.o smart_allocate.o mpi.o mgrid.o random.o ffttest.o $(FFT3D) $(INC)
$(FCL) -o ffttest $(LINK) ffttest.o mpi.o mgrid.o random.o smart_allocate.o base.o $(FFT3D) $(LIB)
kpoints: $(SOURCE) $(FFT3D) makekpoints.o main.F $(INC)
$(FCL) -o kpoints $(LINK) makekpoints.o $(SOURCE) $(FFT3D) $(LIB)
clean:
-rm -f *.g *.f *.o *.L *.mod ; touch *.F
main.o: main$(SUFFIX)
$(FC) $(FFLAGS)$(DEBUG) $(INCS) -c main$(SUFFIX)
xcgrad.o: xcgrad$(SUFFIX)
$(FC) $(FFLAGS) $(INLINE) $(INCS) -c xcgrad$(SUFFIX)
xcspin.o: xcspin$(SUFFIX)
$(FC) $(FFLAGS) $(INLINE) $(INCS) -c xcspin$(SUFFIX)
makeparam.o: makeparam$(SUFFIX)
$(FC) $(FFLAGS)$(DEBUG) $(INCS) -c makeparam$(SUFFIX)
makeparam$(SUFFIX): makeparam.F main.F
#
# MIND: I do not have a full dependency list for the include
# and MODULES: here are only the minimal basic dependencies
# if one strucuture is changed then touch_dep must be called
# with the corresponding name of the structure
#
base.o: base.inc base.F
mgrid.o: mgrid.inc mgrid.F
constant.o: constant.inc constant.F
lattice.o: lattice.inc lattice.F
setex.o: setexm.inc setex.F
pseudo.o: pseudo.inc pseudo.F
mkpoints.o: mkpoints.inc mkpoints.F
wave.o: wave.F
nonl.o: nonl.inc nonl.F
nonlr.o: nonlr.inc nonlr.F
$(OBJ_HIGH):
$(CPP)
$(FC) $(FFLAGS) $(OFLAG_HIGH) $(INCS) -c $*$(SUFFIX)
$(OBJ_NOOPT):
$(CPP)
$(FC) $(FFLAGS) $(INCS) -c $*$(SUFFIX)
fft3dlib_f77.o: fft3dlib_f77.F
$(CPP)
$(F77) $(FFLAGS_F77) -c $*$(SUFFIX)
.F.o:
$(CPP)
$(FC) $(FFLAGS) $(OFLAG) $(INCS) -c $*$(SUFFIX)
.F$(SUFFIX):
$(CPP)
$(SUFFIX).o:
$(FC) $(FFLAGS) $(OFLAG) $(INCS) -c $*$(SUFFIX)
# special rules
#-----------------------------------------------------------------------
# these special rules have been tested for ifc.11 and ifc.12 only
fft3dlib.o : fft3dlib.F
$(CPP)
$(FC) -FR -lowercase -O2 -c $*$(SUFFIX)
fft3dfurth.o : fft3dfurth.F
$(CPP)
$(FC) -FR -lowercase -O1 -c $*$(SUFFIX)
fftw3d.o : fftw3d.F
$(CPP)
$(FC) -FR -lowercase -O1 $(INCS) -c $*$(SUFFIX)
fftmpi.o : fftmpi.F
$(CPP)
$(FC) -FR -lowercase -O1 -c $*$(SUFFIX)
fftmpiw.o : fftmpiw.F
$(CPP)
$(FC) -FR -lowercase -O1 $(INCS) -c $*$(SUFFIX)
wave_high.o : wave_high.F
$(CPP)
$(FC) -FR -lowercase -O1 -c $*$(SUFFIX)
# the following rules are probably no longer required (-O3 seems to work)
wave.o : wave.F
$(CPP)
$(FC) -FR -lowercase -O2 -c $*$(SUFFIX)
paw.o : paw.F
$(CPP)
$(FC) -FR -lowercase -O2 -c $*$(SUFFIX)
cl_shift.o : cl_shift.F
$(CPP)
$(FC) -FR -lowercase -O2 -c $*$(SUFFIX)
us.o : us.F
$(CPP)
$(FC) -FR -lowercase -O2 -c $*$(SUFFIX)
LDApU.o : LDApU.F
$(CPP)
$(FC) -FR -lowercase -O2 -c $*$(SUFFIX)
INCAR
=====
ISMEAR = 1
IBRION = 2 ! Conjugate-gradient algorithm to relax ions
ISIF = 4 ! Relax ion positions and cell shape
VOSKOWN = 1
ISPIN = 2
MAGMOM = 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5
SYSTEM = (Fe+3)36(N+0.5)12
PREC = ACCURATE
EDIFF = 1E-05
NSW = 45
RANDOM_SEED = 1
POSCAR
======
Fe36N12
1
-8.8817841970012523E-16 -8.2188951999999986 0
-7.117773 4.1094475999999993 0
-5.4540000000000015E-16 0 -8.845378000000002
Fe N
36 12
Direct
0.22222233333333344 0.44444466666666665 1
0.22222233333333313 0.77777766666666648 1
0.22222200000000009 0.11111100000000002 1
0.1111110000000001 0.22222199999999998 0.24999999999999997
0.11111100000000004 0.888889 0.24999999999999997
0.4444446666666666 0.22222233333333336 0.24999999999999997
0.888889 0.77777799999999986 1
0.88888899999999982 0.11111099999999974 1
0.88888866666666666 0.44444433333333311 1
0.7777776666666667 0.55555533333333318 0.24999999999999997
0.77777766666666681 0.2222223333333333 0.24999999999999997
0.11111133333333331 0.55555566666666667 0.24999999999999997
0.55555566666666678 0.11111133333333328 1
0.55555566666666656 0.44444433333333316 1
0.55555533333333351 0.77777766666666681 1
0.44444433333333361 0.88888866666666666 0.24999999999999997
0.44444433333333361 0.55555566666666678 0.24999999999999997
0.77777799999999986 0.88888899999999982 0.24999999999999997
0.22222233333333341 0.4444446666666666 0.50000000000000011
0.22222233333333313 0.77777766666666648 0.50000000000000011
0.22222200000000003 0.11111099999999995 0.50000000000000011
0.11111100000000008 0.22222199999999995 0.75
0.11111099999999993 0.88888899999999982 0.75
0.44444466666666665 0.22222233333333333 0.75
0.888889 0.77777799999999986 0.50000000000000011
0.88888899999999982 0.11111099999999974 0.50000000000000011
0.88888866666666677 0.44444433333333322 0.50000000000000011
0.7777776666666667 0.55555533333333318 0.75
0.77777766666666681 0.2222223333333333 0.75
0.11111133333333335 0.55555566666666667 0.75
0.55555566666666678 0.11111133333333323 0.50000000000000011
0.55555566666666656 0.44444433333333316 0.50000000000000011
0.5555553333333334 0.77777766666666692 0.50000000000000011
0.44444433333333361 0.88888866666666666 0.75
0.44444433333333361 0.55555566666666678 0.75
0.77777799999999986 0.88888899999999982 0.75
0.33333333333333337 0.33333366666666664 0.37499999999999989
0.3333333333333332 0.99999966666666662 0.12499999999999999
1 0.66666699999999968 0.37499999999999989
1 0.333333 0.12499999999999999
0.66666666666666663 1.0000003333333334 0.37499999999999989
0.66666666666666663 0.66666633333333336 0.12499999999999999
0.33333333333333331 0.33333366666666658 0.875
0.33333333333333315 0.99999966666666662 0.62500000000000011
1 0.66666699999999968 0.875
1 0.333333 0.62500000000000011
0.66666666666666674 1.0000003333333336 0.875
0.66666666666666663 0.66666633333333336 0.62500000000000011
When running the problem on the university cluster on more than one node (e.g. nodes=2:ppn=8), the calculation always fails with a segfault after ?entering main loop?. If only one node is used, the calculation may pass this point. I've tried to using Open-MPI 1.6.5 compiled by myself as well as the version provided by the cluster. Also ?ulimit -s unlimited? is used as suggested elsewhere. So the problem seems to be related to the usage of Open-MPI across the InfiniBand network, but I?ve not been able to make any further progress from here, so help is really appreciated.
Below are the console output as well as my VASP Makefile and the INCAR and POSCAR files:
running on 16 total cores
distrk: each k-point on 16 cores, 1 groups
distr: one band on 1 cores, 16 groups
using from now: INCAR
vasp.5.3.3 18Dez12 (build Jul 15 2013 11:04:39) complex
POSCAR found type information on POSCAR Fe N
POSCAR found : 2 types and 48 ions
-----------------------------------------------------------------------------
| |
| W W AA RRRRR N N II N N GGGG !!! |
| W W A A R R NN N II NN N G G !!! |
| W W A A R R N N N II N N N G !!! |
| W WW W AAAAAA RRRRR N N N II N N N G GGG ! |
| WW WW A A R R N NN II N NN G G |
| W W A A R R N N II N N GGGG !!! |
| |
| For optimal performance we recommend that you set |
| NPAR = 4 - approx SQRT( number of cores) |
| (number of cores/NPAR must be integer) |
| This setting will greatly improve the performance of VASP for DFT. |
| The default NPAR=number of cores might be grossly inefficient |
| on modern multi-core architectures or massively parallel machines. |
| Do your own testing. |
| Unfortunately you need to use the default for hybrid, GW and RPA |
| calculations. |
| |
-----------------------------------------------------------------------------
-----------------------------------------------------------------------------
| |
| ADVICE TO THIS USER RUNNING 'VASP/VAMP' (HEAR YOUR MASTER'S VOICE ...): |
| |
| You have a (more or less) 'large supercell' and for larger cells |
| it might be more efficient to use real space projection opertators |
| So try LREAL= Auto in the INCAR file. |
| Mind: At the moment your POTCAR file does not contain real space |
| projectors, and has to be modified, BUT if you |
| want to do an extremely accurate calculation you might also keep the |
| reciprocal projection scheme (i.e. LREAL=.FALSE.) |
| |
-----------------------------------------------------------------------------
LDA part: xc-table for Pade appr. of Perdew
generate k-points for: 6 6 5
POSCAR, INCAR and KPOINTS ok, starting setup
FFT: planning ...
WAVECAR not read
entering main loop
N E dE d eps ncg rms rms(c)
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
libiomp5.so 00002AF0527942F5 Unknown Unknown Unknown
libiomp5.so 00002AF052790ED4 Unknown Unknown Unknown
libiomp5.so 00002AF05278FD14 Unknown Unknown Unknown
libiomp5.so 00002AF05278FDFD Unknown Unknown Unknown
libiomp5.so 00002AF05276D5E7 Unknown Unknown Unknown
libmkl_intel_thre 00002AF050728DA5 Unknown Unknown Unknown
libmkl_intel_thre 00002AF0506AB0C6 Unknown Unknown Unknown
libmkl_intel_lp64 00002AF04FF61A7F Unknown Unknown Unknown
vasp 0000000000649EF8 Unknown Unknown Unknown
vasp 00000000006859C4 Unknown Unknown Unknown
vasp 0000000000D14CE8 Unknown Unknown Unknown
vasp 0000000000D399AA Unknown Unknown Unknown
vasp 000000000045AEBB Unknown Unknown Unknown
vasp 000000000043599C Unknown Unknown Unknown
libc.so.6 0000003E32E1ECDD Unknown Unknown Unknown
vasp 0000000000435829 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
libc.so.6 0000003E32ECCDD7 Unknown Unknown Unknown
libiomp5.so 00002B9912C01B29 Unknown Unknown Unknown
libiomp5.so 00002B9912BE5332 Unknown Unknown Unknown
libiomp5.so 00002B9912BE1ED4 Unknown Unknown Unknown
libiomp5.so 00002B9912BE0D14 Unknown Unknown Unknown
libiomp5.so 00002B9912BE0DFD Unknown Unknown Unknown
libiomp5.so 00002B9912BBE5E7 Unknown Unknown Unknown
libmkl_intel_thre 00002B9910B79DA5 Unknown Unknown Unknown
libmkl_intel_thre 00002B9910AFC0C6 Unknown Unknown Unknown
libmkl_intel_lp64 00002B99103B2A7F Unknown Unknown Unknown
vasp 0000000000649EF8 Unknown Unknown Unknown
vasp 00000000006859C4 Unknown Unknown Unknown
vasp 0000000000D14CE8 Unknown Unknown Unknown
vasp 0000000000D399AA Unknown Unknown Unknown
vasp 000000000045AEBB Unknown Unknown Unknown
vasp 000000000043599C Unknown Unknown Unknown
libc.so.6 0000003E32E1ECDD Unknown Unknown Unknown
vasp 0000000000435829 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
mca_btl_openib.so 00002B14472E8ECE Unknown Unknown Unknown
libmpi.so.1 00002B14424763DA Unknown Unknown Unknown
libmpi.so.1 00002B14423C582D Unknown Unknown Unknown
mca_coll_tuned.so 00002B14487C5967 Unknown Unknown Unknown
libmpi.so.1 00002B14423D1796 Unknown Unknown Unknown
libmpi_f77.so.1 00002B1442154EEA Unknown Unknown Unknown
vasp 000000000048787F Unknown Unknown Unknown
vasp 00000000005D5D4F Unknown Unknown Unknown
vasp 0000000000D18521 Unknown Unknown Unknown
vasp 0000000000D399AA Unknown Unknown Unknown
vasp 000000000045AEBB Unknown Unknown Unknown
vasp 000000000043599C Unknown Unknown Unknown
libc.so.6 0000003E32E1ECDD Unknown Unknown Unknown
vasp 0000000000435829 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
mca_btl_sm.so 00002B9DF2B6B89F Unknown Unknown Unknown
libmpi.so.1 00002B9DED8D43DA Unknown Unknown Unknown
libmpi.so.1 00002B9DED82382D Unknown Unknown Unknown
mca_coll_tuned.so 00002B9DF3C23967 Unknown Unknown Unknown
libmpi.so.1 00002B9DED82F796 Unknown Unknown Unknown
libmpi_f77.so.1 00002B9DED5B2EEA Unknown Unknown Unknown
vasp 000000000048787F Unknown Unknown Unknown
vasp 00000000005D5D4F Unknown Unknown Unknown
vasp 0000000000D18521 Unknown Unknown Unknown
vasp 0000000000D399AA Unknown Unknown Unknown
vasp 000000000045AEBB Unknown Unknown Unknown
vasp 000000000043599C Unknown Unknown Unknown
libc.so.6 0000003E32E1ECDD Unknown Unknown Unknown
vasp 0000000000435829 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
libmlx4-rdmav2.so 00002B81E1AD6378 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
mca_btl_openib.so 00002AE907397FE6 Unknown Unknown Unknown
libmpi.so.1 00002AE9025253DA Unknown Unknown Unknown
libmpi.so.1 00002AE90247482D Unknown Unknown Unknown
mca_coll_tuned.so 00002AE908874967 Unknown Unknown Unknown
libmpi.so.1 00002AE902480796 Unknown Unknown Unknown
libmpi_f77.so.1 00002AE902203EEA Unknown Unknown Unknown
vasp 000000000048787F Unknown Unknown Unknown
vasp 00000000005D5D4F Unknown Unknown Unknown
vasp 0000000000D18521 Unknown Unknown Unknown
vasp 0000000000D399AA Unknown Unknown Unknown
vasp 000000000045AEBB Unknown Unknown Unknown
vasp 000000000043599C Unknown Unknown Unknown
libc.so.6 0000003E32E1ECDD Unknown Unknown Unknown
vasp 0000000000435829 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
libpthread.so.0 0000003E33A0C113 Unknown Unknown Unknown
libmlx4-rdmav2.so 00002AC77E1C4A2B Unknown Unknown Unknown
mca_btl_openib.so 00002AC77C49C52C Unknown Unknown Unknown
mca_btl_openib.so 00002AC77C49D089 Unknown Unknown Unknown
libmpi.so.1 00002AC77762A3DA Unknown Unknown Unknown
libmpi.so.1 00002AC77757982D Unknown Unknown Unknown
mca_coll_tuned.so 00002AC77D979967 Unknown Unknown Unknown
libmpi.so.1 00002AC777585796 Unknown Unknown Unknown
libmpi_f77.so.1 00002AC777308EEA Unknown Unknown Unknown
vasp 000000000048787F Unknown Unknown Unknown
vasp 00000000005D5D4F Unknown Unknown Unknown
vasp 0000000000D18521 Unknown Unknown Unknown
vasp 0000000000D399AA Unknown Unknown Unknown
vasp 000000000045AEBB Unknown Unknown Unknown
vasp 000000000043599C Unknown Unknown Unknown
libc.so.6 0000003E32E1ECDD Unknown Unknown Unknown
vasp 0000000000435829 Unknown Unknown Unknown
forrtl: error (78): process killed (SIGTERM)
Image PC Routine Line Source
mca_btl_openib.so 00002B163FB7107F Unknown Unknown Unknown
libmpi.so.1 00002B163ACFE3DA Unknown Unknown Unknown
libmpi.so.1 00002B163AC4D82D Unknown Unknown Unknown
mca_coll_tuned.so 00002B164104D967 Unknown Unknown Unknown
libmpi.so.1 00002B163AC59796 Unknown Unknown Unknown
libmpi_f77.so.1 00002B163A9DCEEA Unknown Unknown Unknown
vasp 000000000048787F Unknown Unknown Unknown
vasp 00000000005D5D4F Unknown Unknown Unknown
vasp 0000000000D18521 Unknown Unknown Unknown
vasp 0000000000D399AA Unknown Unknown Unknown
vasp 000000000045AEBB Unknown Unknown Unknown
vasp 000000000043599C Unknown Unknown Unknown
libc.so.6 0000003E32E1ECDD Unknown Unknown Unknown
vasp 0000000000435829 Unknown Unknown Unknown
--------------------------------------------------------------------------
mpirun noticed that process rank 9 with PID 19577 on node n-62-23-9 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
real 4m58.211s
user 69m43.740s
sys 3m7.745s
Makefile
========
.SUFFIXES: .inc .f .f90 .F
#-----------------------------------------------------------------------
# Makefile for Intel Fortran compiler for Pentium/Athlon/Opteron
# based systems
# we recommend this makefile for both Intel as well as AMD systems
# for AMD based systems appropriate BLAS (libgoto) and fftw libraries are
# however mandatory (whereas they are optional for Intel platforms)
# For Athlon we recommend
# ) to link against libgoto (and mkl as a backup for missing routines)
# ) odd enough link in libfftw3xf_intel.a (fftw interface for mkl)
# feedback is greatly appreciated
#
# The makefile was tested only under Linux on Intel and AMD platforms
# the following compiler versions have been tested:
# - ifc.7.1 works stable somewhat slow but reliably
# - ifc.8.1 fails to compile the code properly
# - ifc.9.1 recommended (both for 32 and 64 bit)
# - ifc.10.1 partially recommended (both for 32 and 64 bit)
# tested build 20080312 Package ID: l_fc_p_10.1.015
# the gamma only mpi version can not be compiles
# using ifc.10.1
# - ifc.11.1 partially recommended (some problems with Gamma only and intel fftw)
# Build 20090630 Package ID: l_cprof_p_11.1.046
# - ifort.12.1 strongly recommended (we use this to compile vasp)
# Version 12.1.5.339 Build 20120612
#
# it might be required to change some of library path ways, since
# LINUX installations vary a lot
#
# Hence check ***ALL*** options in this makefile very carefully
#-----------------------------------------------------------------------
#
# BLAS must be installed on the machine
# there are several options:
# 1) very slow but works:
# retrieve the lapackage from ftp.netlib.org
# and compile the blas routines (BLAS/SRC directory)
# please use g77 or f77 for the compilation. When I tried to
# use pgf77 or pgf90 for BLAS, VASP hang up when calling
# ZHEEV (however this was with lapack 1.1 now I use lapack 2.0)
# 2) more desirable: get an optimized BLAS
#
# the two most reliable packages around are presently:
# 2a) Intels own optimised BLAS (PIII, P4, PD, PC2, Itanium)
# http://developer.intel.com/software/products/mkl/
# this is really excellent, if you use Intel CPU's
#
# 2b) probably fastest SSE2 (4 GFlops on P4, 2.53 GHz, 16 GFlops PD,
# around 30 GFlops on Quad core)
# Kazushige Goto's BLAS
# http://www.cs.utexas.edu/users/kgoto/signup_first.html
# http://www.tacc.utexas.edu/resources/software/
#
#-----------------------------------------------------------------------
# all CPP processed fortran files have the extension .f90
SUFFIX=.f90
#-----------------------------------------------------------------------
# fortran compiler and linker
#-----------------------------------------------------------------------
FC=ifort
# fortran linker
FCL=$(FC)
#-----------------------------------------------------------------------
# whereis CPP ?? (I need CPP, can't use gcc with proper options)
# that's the location of gcc for SUSE 5.3
#
# CPP_ = /usr/lib/gcc-lib/i486-linux/2.7.2/cpp -P -C
#
# that's probably the right line for some Red Hat distribution:
#
# CPP_ = /usr/lib/gcc-lib/i386-redhat-linux/2.7.2.3/cpp -P -C
#
# SUSE X.X, maybe some Red Hat distributions:
CPP_ = ./preprocess <$*.F | /usr/bin/cpp -P -C -traditional >$*$(SUFFIX)
# this release should be fpp clean
# we now recommend fpp as preprocessor
# if this fails go back to cpp
CPP_=fpp -f_com=no -free -w0 $*.F $*$(SUFFIX)
#-----------------------------------------------------------------------
# possible options for CPP:
# NGXhalf charge density reduced in X direction
# wNGXhalf gamma point only reduced in X direction
# avoidalloc avoid ALLOCATE if possible
# PGF90 work around some for some PGF90 / IFC bugs
# CACHE_SIZE 1000 for PII,PIII, 5000 for Athlon, 8000-12000 P4, PD
# RPROMU_DGEMV use DGEMV instead of DGEMM in RPRO (depends on used BLAS)
# RACCMU_DGEMV use DGEMV instead of DGEMM in RACC (depends on used BLAS)
# tbdyn MD package of Tomas Bucko
#-----------------------------------------------------------------------
CPP = $(CPP_) -DHOST=\"LinuxIFC\" \
-DCACHE_SIZE=12000 -DPGF90 -Davoidalloc -DNGXhalf \
# -DRPROMU_DGEMV -DRACCMU_DGEMV
#-----------------------------------------------------------------------
# general fortran flags (there must a trailing blank on this line)
# byterecl is strictly required for ifc, since otherwise
# the WAVECAR file becomes huge
#-----------------------------------------------------------------------
FFLAGS = -FR -names lowercase -assume byterecl
#-----------------------------------------------------------------------
# optimization
# we have tested whether higher optimisation improves performance
# -axK SSE1 optimization, but also generate code executable on all mach.
# xK improves performance somewhat on XP, and a is required in order
# to run the code on older Athlons as well
# -xW SSE2 optimization
# -axW SSE2 optimization, but also generate code executable on all mach.
# -tpp6 P3 optimization
# -tpp7 P4 optimization
#-----------------------------------------------------------------------
# ifc.9.1, ifc.10.1 recommended
OFLAG=-O2 -ip
OFLAG_HIGH = $(OFLAG)
OBJ_HIGH =
OBJ_NOOPT =
DEBUG = -FR -O0
INLINE = $(OFLAG)
#-----------------------------------------------------------------------
# the following lines specify the position of BLAS and LAPACK
# we recommend to use mkl, that is simple and most likely
# fastest in Intel based machines
#-----------------------------------------------------------------------
# mkl path for ifc 11 compiler
#MKL_PATH=$(MKLROOT)/lib/em64t
# mkl path for ifc 12 compiler
MKL_PATH=$(MKLROOT)/lib/intel64
MKL_FFTW_PATH=$(HOME)/local/vasp/fftw3xf/
# BLAS
# setting -DRPROMU_DGEMV -DRACCMU_DGEMV in the CPP lines usually speeds up program execution
# BLAS= -Wl,--start-group $(MKL_PATH)/libmkl_intel_lp64.a $(MKL_PATH)/libmkl_intel_thread.a $(MKL_PATH)/libmkl_core.a -Wl,--end-group -
lguide
# faster linking and available from at least version 11
#BLAS= -lguide -mkl
# LAPACK, use vasp.5.lib/lapack_double
#LAPACK= ../vasp.5.lib/lapack_double.o
# LAPACK from mkl, usually faster and contains scaLAPACK as well
#LAPACK= $(MKL_PATH)/libmkl_intel_lp64.a
# here a tricky version, link in libgoto and use mkl as a backup
# also needs a special line for LAPACK
# this is the best thing you can do on AMD based systems !!!!!!
#BLAS = -Wl,--start-group /opt/libs/libgoto/libgoto.so $(MKL_PATH)/libmkl_intel_thread.a $(MKL_PATH)/libmkl_core.a -Wl,--end-group -
liomp5
#LAPACK= /opt/libs/libgoto/libgoto.so $(MKL_PATH)/libmkl_intel_lp64.a
#-----------------------------------------------------------------------
LIB = -L../vasp.5.lib -ldmy \
../vasp.5.lib/linpack_double.o $(LAPACK) \
$(BLAS)
# options for linking, nothing is required (usually)
LINK =
#-----------------------------------------------------------------------
# fft libraries:
# VASP.5.2 can use fftw.3.1.X (http://www.fftw.org)
# since this version is faster on P4 machines, we recommend to use it
#-----------------------------------------------------------------------
#FFT3D = fft3dfurth.o fft3dlib.o
# alternatively: fftw.3.1.X is slighly faster and should be used if available
#FFT3D = fftw3d.o fft3dlib.o /opt/libs/fftw-3.1.2/lib/libfftw3.a
# you may also try to use the fftw wrapper to mkl (but the path might vary a lot)
# it seems this is best for AMD based systems
#FFT3D = fftw3d.o fft3dlib.o $(MKL_FFTW_PATH)/libfftw3xf_gnu.a
#INCS = -I$(MKLROOT)/include/fftw
#=======================================================================
# MPI section, uncomment the following lines until
# general rules and compile lines
# presently we recommend OPENMPI, since it seems to offer better
# performance than lam or mpich
#
# !!! Please do not send me any queries on how to install MPI, I will
# certainly not answer them !!!!
#=======================================================================
#-----------------------------------------------------------------------
# fortran linker for mpi
#-----------------------------------------------------------------------
FC=mpif90
FCL=$(FC) -mkl
#-----------------------------------------------------------------------
# additional options for CPP in parallel version (see also above):
# NGZhalf charge density reduced in Z direction
# wNGZhalf gamma point only reduced in Z direction
# scaLAPACK use scaLAPACK (recommended if mkl is available)
# avoidalloc avoid ALLOCATE if possible
# PGF90 work around some for some PGF90 / IFC bugs
# CACHE_SIZE 1000 for PII,PIII, 5000 for Athlon, 8000-12000 P4, PD
# RPROMU_DGEMV use DGEMV instead of DGEMM in RPRO (depends on used BLAS)
# RACCMU_DGEMV use DGEMV instead of DGEMM in RACC (depends on used BLAS)
# tbdyn MD package of Tomas Bucko
#-----------------------------------------------------------------------
#-----------------------------------------------------------------------
CPP = $(CPP_) -DMPI -DHOST=\"LinuxIFC\" -DIFC \
-DCACHE_SIZE=4000 -DPGF90 -Davoidalloc -DNGZhalf \
-DMPI_BLOCK=8000 -Duse_collective
# -DscaLAPACK
## -DRPROMU_DGEMV -DRACCMU_DGEMV
#-----------------------------------------------------------------------
# location of SCALAPACK
# if you do not use SCALAPACK simply leave this section commented out
#-----------------------------------------------------------------------
# usually simplest link in mkl scaLAPACK
#BLACS= -lmkl_blacs_openmpi_lp64
#SCA= $(MKL_PATH)/libmkl_scalapack_lp64.a $(BLACS)
#-----------------------------------------------------------------------
# libraries
#-----------------------------------------------------------------------
#LIB = -L../vasp.5.lib -ldmy \
# ../vasp.5.lib/linpack_double.o \
# $(SCA) $(LAPACK) $(BLAS)
#-----------------------------------------------------------------------
# parallel FFT
#-----------------------------------------------------------------------
# FFT: fftmpi.o with fft3dlib of Juergen Furthmueller
#FFT3D = fftmpi.o fftmpi_map.o fft3dfurth.o fft3dlib.o
# alternatively: fftw.3.1.X is slighly faster and should be used if available
#FFT3D = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o /opt/libs/fftw-3.1.2/lib/libfftw3.a
# you may also try to use the fftw wrapper to mkl (but the path might vary a lot)
# it seems this is best for AMD based systems
FFT3D = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o $(MKL_FFTW_PATH)/libfftw3xf_gnu.a
INCS = -I$(MKLROOT)/include/fftw
#-----------------------------------------------------------------------
# general rules and compile lines
#-----------------------------------------------------------------------
BASIC= symmetry.o symlib.o lattlib.o random.o
SOURCE= base.o mpi.o smart_allocate.o xml.o \
constant.o jacobi.o main_mpi.o scala.o \
asa.o lattice.o poscar.o ini.o mgrid.o xclib.o vdw_nl.o xclib_grad.o \
radial.o pseudo.o gridq.o ebs.o \
mkpoints.o wave.o wave_mpi.o wave_high.o spinsym.o \
$(BASIC) nonl.o nonlr.o nonl_high.o dfast.o choleski2.o \
mix.o hamil.o xcgrad.o xcspin.o potex1.o potex2.o \
constrmag.o cl_shift.o relativistic.o LDApU.o \
paw_base.o metagga.o egrad.o pawsym.o pawfock.o pawlhf.o rhfatm.o hyperfine.o paw.o \
mkpoints_full.o charge.o Lebedev-Laikov.o stockholder.o dipol.o pot.o \
dos.o elf.o tet.o tetweight.o hamil_rot.o \
chain.o dyna.o k-proj.o sphpro.o us.o core_rel.o \
aedens.o wavpre.o wavpre_noio.o broyden.o \
dynbr.o hamil_high.o rmm-diis.o reader.o writer.o tutor.o xml_writer.o \
brent.o stufak.o fileio.o opergrid.o stepver.o \
chgloc.o fast_aug.o fock_multipole.o fock.o mkpoints_change.o sym_grad.o \
mymath.o internals.o npt_dynamics.o dynconstr.o dimer_heyden.o dvvtrajectory.o vdwforcefield.o \
nmr.o pead.o subrot.o subrot_scf.o \
force.o pwlhf.o gw_model.o optreal.o steep.o davidson.o david_inner.o \
electron.o rot.o electron_all.o shm.o pardens.o paircorrection.o \
optics.o constr_cell_relax.o stm.o finite_diff.o elpol.o \
hamil_lr.o rmm-diis_lr.o subrot_cluster.o subrot_lr.o \
lr_helper.o hamil_lrf.o elinear_response.o ilinear_response.o \
linear_optics.o \
setlocalpp.o wannier.o electron_OEP.o electron_lhf.o twoelectron4o.o \
mlwf.o ratpol.o screened_2e.o wave_cacher.o chi_base.o wpot.o \
local_field.o ump2.o ump2kpar.o fcidump.o ump2no.o \
bse_te.o bse.o acfdt.o chi.o sydmat.o dmft.o \
rmm-diis_mlr.o linear_response_NMR.o wannier_interpol.o linear_response.o
vasp: $(SOURCE) $(FFT3D) $(INC) main.o
rm -f vasp
$(FCL) -o vasp main.o $(SOURCE) $(FFT3D) $(LIB) $(LINK)
makeparam: $(SOURCE) $(FFT3D) makeparam.o main.F $(INC)
$(FCL) -o makeparam $(LINK) makeparam.o $(SOURCE) $(FFT3D) $(LIB)
zgemmtest: zgemmtest.o base.o random.o $(INC)
$(FCL) -o zgemmtest $(LINK) zgemmtest.o random.o base.o $(LIB)
dgemmtest: dgemmtest.o base.o random.o $(INC)
$(FCL) -o dgemmtest $(LINK) dgemmtest.o random.o base.o $(LIB)
ffttest: base.o smart_allocate.o mpi.o mgrid.o random.o ffttest.o $(FFT3D) $(INC)
$(FCL) -o ffttest $(LINK) ffttest.o mpi.o mgrid.o random.o smart_allocate.o base.o $(FFT3D) $(LIB)
kpoints: $(SOURCE) $(FFT3D) makekpoints.o main.F $(INC)
$(FCL) -o kpoints $(LINK) makekpoints.o $(SOURCE) $(FFT3D) $(LIB)
clean:
-rm -f *.g *.f *.o *.L *.mod ; touch *.F
main.o: main$(SUFFIX)
$(FC) $(FFLAGS)$(DEBUG) $(INCS) -c main$(SUFFIX)
xcgrad.o: xcgrad$(SUFFIX)
$(FC) $(FFLAGS) $(INLINE) $(INCS) -c xcgrad$(SUFFIX)
xcspin.o: xcspin$(SUFFIX)
$(FC) $(FFLAGS) $(INLINE) $(INCS) -c xcspin$(SUFFIX)
makeparam.o: makeparam$(SUFFIX)
$(FC) $(FFLAGS)$(DEBUG) $(INCS) -c makeparam$(SUFFIX)
makeparam$(SUFFIX): makeparam.F main.F
#
# MIND: I do not have a full dependency list for the include
# and MODULES: here are only the minimal basic dependencies
# if one strucuture is changed then touch_dep must be called
# with the corresponding name of the structure
#
base.o: base.inc base.F
mgrid.o: mgrid.inc mgrid.F
constant.o: constant.inc constant.F
lattice.o: lattice.inc lattice.F
setex.o: setexm.inc setex.F
pseudo.o: pseudo.inc pseudo.F
mkpoints.o: mkpoints.inc mkpoints.F
wave.o: wave.F
nonl.o: nonl.inc nonl.F
nonlr.o: nonlr.inc nonlr.F
$(OBJ_HIGH):
$(CPP)
$(FC) $(FFLAGS) $(OFLAG_HIGH) $(INCS) -c $*$(SUFFIX)
$(OBJ_NOOPT):
$(CPP)
$(FC) $(FFLAGS) $(INCS) -c $*$(SUFFIX)
fft3dlib_f77.o: fft3dlib_f77.F
$(CPP)
$(F77) $(FFLAGS_F77) -c $*$(SUFFIX)
.F.o:
$(CPP)
$(FC) $(FFLAGS) $(OFLAG) $(INCS) -c $*$(SUFFIX)
.F$(SUFFIX):
$(CPP)
$(SUFFIX).o:
$(FC) $(FFLAGS) $(OFLAG) $(INCS) -c $*$(SUFFIX)
# special rules
#-----------------------------------------------------------------------
# these special rules have been tested for ifc.11 and ifc.12 only
fft3dlib.o : fft3dlib.F
$(CPP)
$(FC) -FR -lowercase -O2 -c $*$(SUFFIX)
fft3dfurth.o : fft3dfurth.F
$(CPP)
$(FC) -FR -lowercase -O1 -c $*$(SUFFIX)
fftw3d.o : fftw3d.F
$(CPP)
$(FC) -FR -lowercase -O1 $(INCS) -c $*$(SUFFIX)
fftmpi.o : fftmpi.F
$(CPP)
$(FC) -FR -lowercase -O1 -c $*$(SUFFIX)
fftmpiw.o : fftmpiw.F
$(CPP)
$(FC) -FR -lowercase -O1 $(INCS) -c $*$(SUFFIX)
wave_high.o : wave_high.F
$(CPP)
$(FC) -FR -lowercase -O1 -c $*$(SUFFIX)
# the following rules are probably no longer required (-O3 seems to work)
wave.o : wave.F
$(CPP)
$(FC) -FR -lowercase -O2 -c $*$(SUFFIX)
paw.o : paw.F
$(CPP)
$(FC) -FR -lowercase -O2 -c $*$(SUFFIX)
cl_shift.o : cl_shift.F
$(CPP)
$(FC) -FR -lowercase -O2 -c $*$(SUFFIX)
us.o : us.F
$(CPP)
$(FC) -FR -lowercase -O2 -c $*$(SUFFIX)
LDApU.o : LDApU.F
$(CPP)
$(FC) -FR -lowercase -O2 -c $*$(SUFFIX)
INCAR
=====
ISMEAR = 1
IBRION = 2 ! Conjugate-gradient algorithm to relax ions
ISIF = 4 ! Relax ion positions and cell shape
VOSKOWN = 1
ISPIN = 2
MAGMOM = 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5
SYSTEM = (Fe+3)36(N+0.5)12
PREC = ACCURATE
EDIFF = 1E-05
NSW = 45
RANDOM_SEED = 1
POSCAR
======
Fe36N12
1
-8.8817841970012523E-16 -8.2188951999999986 0
-7.117773 4.1094475999999993 0
-5.4540000000000015E-16 0 -8.845378000000002
Fe N
36 12
Direct
0.22222233333333344 0.44444466666666665 1
0.22222233333333313 0.77777766666666648 1
0.22222200000000009 0.11111100000000002 1
0.1111110000000001 0.22222199999999998 0.24999999999999997
0.11111100000000004 0.888889 0.24999999999999997
0.4444446666666666 0.22222233333333336 0.24999999999999997
0.888889 0.77777799999999986 1
0.88888899999999982 0.11111099999999974 1
0.88888866666666666 0.44444433333333311 1
0.7777776666666667 0.55555533333333318 0.24999999999999997
0.77777766666666681 0.2222223333333333 0.24999999999999997
0.11111133333333331 0.55555566666666667 0.24999999999999997
0.55555566666666678 0.11111133333333328 1
0.55555566666666656 0.44444433333333316 1
0.55555533333333351 0.77777766666666681 1
0.44444433333333361 0.88888866666666666 0.24999999999999997
0.44444433333333361 0.55555566666666678 0.24999999999999997
0.77777799999999986 0.88888899999999982 0.24999999999999997
0.22222233333333341 0.4444446666666666 0.50000000000000011
0.22222233333333313 0.77777766666666648 0.50000000000000011
0.22222200000000003 0.11111099999999995 0.50000000000000011
0.11111100000000008 0.22222199999999995 0.75
0.11111099999999993 0.88888899999999982 0.75
0.44444466666666665 0.22222233333333333 0.75
0.888889 0.77777799999999986 0.50000000000000011
0.88888899999999982 0.11111099999999974 0.50000000000000011
0.88888866666666677 0.44444433333333322 0.50000000000000011
0.7777776666666667 0.55555533333333318 0.75
0.77777766666666681 0.2222223333333333 0.75
0.11111133333333335 0.55555566666666667 0.75
0.55555566666666678 0.11111133333333323 0.50000000000000011
0.55555566666666656 0.44444433333333316 0.50000000000000011
0.5555553333333334 0.77777766666666692 0.50000000000000011
0.44444433333333361 0.88888866666666666 0.75
0.44444433333333361 0.55555566666666678 0.75
0.77777799999999986 0.88888899999999982 0.75
0.33333333333333337 0.33333366666666664 0.37499999999999989
0.3333333333333332 0.99999966666666662 0.12499999999999999
1 0.66666699999999968 0.37499999999999989
1 0.333333 0.12499999999999999
0.66666666666666663 1.0000003333333334 0.37499999999999989
0.66666666666666663 0.66666633333333336 0.12499999999999999
0.33333333333333331 0.33333366666666658 0.875
0.33333333333333315 0.99999966666666662 0.62500000000000011
1 0.66666699999999968 0.875
1 0.333333 0.62500000000000011
0.66666666666666674 1.0000003333333336 0.875
0.66666666666666663 0.66666633333333336 0.62500000000000011