Vallard's Tech Notes

Today I compiled NWChem on my IBM Intel InfiniBand cluster.  I have no idea if my performance is optimal, but I do know that it works.

Here is the secret to my success:

First, you have to set some path variables to get things running.  This is set in my home directory:

cat ~/.bashrc

INTELCCROOT=/home/appls/compilers/intel/11.0/083

INTELFCROOT=/home/appls/compilers/intel/11.0/081

PGROOT=/home/appls/compilers/pgi
PGCC=$PGROOT/linux86-64/8.0-5/bin
PGFLEXLM=$PGROOT/linux86-64/8.0/bin
LM_LICENSE_FILE=$LM_LICENSE_FILE:$PGROOT/license.dat

#MPI_HOME=/home/appls/openmpi/gcc
#MPI_HOME=/home/appls/openmpi/pgi
MPI_HOME=/home/appls/openmpi/intel

NWCHEM=/home/appls/QChem/NWChem/intel/bin

PATH=$PATH:$MPI_HOME/bin:$HOME/bin:$INTELCCROOT/bin/intel64:$INTELFCROOT/bin/intel64:$PGCC:$PGFLEXLM:$NWCHEM

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$MPI_HOME/lib:$INTELCCROOT/lib/intel64:$INTELFCROOT/lib/intel64

Unfortunately, I’m leaving out a lot of the details here, but I used the Intel compilers and had previously built openmpi.  The way I built openmpi was:

./configure --prefix /home/appls/openmpi/intel CC=icc CXX=icpc F77=ifort FC=ifort

make -j8

make install

S

http://wiki.cse.ucdavis.edu/support:hpc:software:nwchem

http://www.mcsr.olemiss.edu/appssubpage.php?pagename=nwchem.inc

So, after I got that set up, I did the normal:

tar zxvf nwchem-5.1.tar.gz

Then I made a script that pretty much did all the work.  The script is called makeit.sh:

export TCGRSH=/usr/bin/ssh
export NWCHEM_TOP=/home/vallard/qchem/nwchem-5.1/
export NWCHEM_TARGET=LINUX64
export USE_MPI=y
export USE_MPIF=y
export MPI_LOC=/home/appls/openmpi/intel
export MPI_LIB=$MPI_LOC/lib
export LIBMPI="-L $MPI_LIB -lmpi -lopen-pal -lopen-rte -lmpi_f90 -lmpi_f77"
export MPI_INCLUDE=$MPI_LOC/include
export ARMCI_NETWORK=OPENIB
export LARGE_FILES
export NWCHEM_MODULES=all
export FC=ifort
export CC=icc
cd $NWCHEM_TOP/src
make CC=icc FC=ifort -j4

After kicking that off, it ran for almost 20 minutes compiling!  Forever!  I saw a lot of vector loop messages, but gave them no head, and fearlessly pressed forward.

After it was done compiling as root, I did the following:

[root@mgt vallard]# export NWCHEM_TOP=/home/vallard/qchem/nwchem-5.1/
[root@mgt vallard]# mkdir $NWCHEM/bin
[root@mgt vallard]# mkdir $NWCHEM/data
[root@mgt vallard]# cp /home/vallard/qchem/nwchem-5.1/bin/LINUX64/nwchem $NWCHEM/bin
^[[root@mgt vallard]# cp /home/vallard/qchem/nwchem-5.1/bin/LINUX64/depend.x $NWEM/bin/
[root@mgt vallard]# cd $NWCHEM_TOP/src/basis
[root@mgt basis]# cp -r libraries $NWCHEM/data/
[root@mgt basis]# cd $NWCHEM_TOP/src/
[root@mgt src]# cp -r data $NWCHEM
[root@mgt src]# cd $NWCHEM_TOP/src/nwpw/libraryps
[root@mgt libraryps]# cp -r pspw_default  $NWCHEM/data/
[root@mgt libraryps]# cp -r paw_default/  $NWCHEM/data/
[root@mgt libraryps]# cp -r TM  $NWCHEM/data/
[root@mgt libraryps]# cp -r HGH_LDA  $NWCHEM/data/

That got everything in place.  When done, I went to my compute node and tried one of the examples:

cd /home/vallard/qchem/nwchem-5.1/examples/dirdyvtst/h3
mpirun -np 32 -machinefile machinefile nwchem h3tr2.nw

After that it all seemed to work.  Any optimization info would be great!  Thanks