Installation on the HPC cluster Beskow

Beskow is a Cray machine running Intel processors. The procedure described below could arguably be one of the most complex methods due to the obsolete collection of modules and the use of wrappers which is designed to enable cross-compilation in all three compilation environments (Prg-cray, Prg-intel and Prg-gnu). Here we have managed to install FluidFFT in Prg-intel environment.

Load necessary modules:

module load gcc/6.1.0
module swap PrgEnv-cray PrgEnv-intel
module swap intel intel/18.0.0.128
module load mercurial
module load cray-hdf5-parallel

Ensure that gcc is loaded before intel. This is important so that Intel libraries get prepended to LD_LIBRARY_PATH and should supersede equivalent GCC libraries, since we intent to use Intel compilers.

Installing additional packages using Anaconda provided in Beskow can be troublesome. Install Miniconda instead, setup a virtual environment and install necessary python packages as described here:

wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh

module load craype-hugepages2M  # cray-hdf5-parallel

conda install ipython
conda install numpy
conda install h5py
conda install scipy matplotlib
conda install -c conda-forge pyfftw

pip install cython
export CRAYPE_LINK_TYPE=dynamic
module load craype-hugepages2M
CC=cc MPICC=cc pip install mpi4py -v --no-deps --no-binary mpi4py
module rm craype-hugepages2M

pip install pythran colorlog nose

Run a few tests:

salloc -A <project> -N 1 -t 10:00
export CRAY_ROOTFS=DSL
export OMP_NUM_THREADS=1
aprun -n 1 python -c 'import numpy; numpy.test()'
aprun -n 1 python -c 'import h5py; h5py.run_tests()'
aprun -n 2 python -m unittest fluiddyn.util.test.test_mpi

Create the file ~/.pythranrc with:

[pythran]
complex_hook = True

[compiler]
CC=gcc
CXX=g++

However do not run module load fftw as the latest version available now (2017-11-19) 3.3.4 and requires patching to use pfft. Also we need recent versions of GNU Autotools to properly build the packages, again not supplied with Beskow! We start with:

#!/bin/bash

# pkgdir=$HOME/.local/opt
pkgdir="/cfs/klemming/nobackup/${USER:0:1}/${USER}/opt/pkg"
srcdir=$PWD/autotools
export MAKEFLAGS="-j$(nproc)"

gnu_install() {
  cd $srcdir
  local pkgname=$1
  local pkgver=$2
  local pkgdir=$3

  mkdir -p $pkgdir
  if [ ! -d  ./$pkgname ]
  then
    [ ! -f ${pkgname}-${pkgver}.tar.gz ] && wget ftp://ftp.gnu.org/gnu/${pkgname}/${pkgname}-${pkgver}.tar.gz
    tar xvf ${pkgname}-${pkgver}.tar.gz
    mv ${pkgname}-${pkgver} ${pkgname}

    # git clone git://git.savannah.gnu.org/$pkgname.git --depth=1
  fi

  cd $pkgname
  rm -rf ${pkgdir}
  ./bootstrap
  ./configure --prefix=${pkgdir} CC=gcc
  make
  # make check
  make install
}

package() {
  mkdir -p $srcdir
  local LOCALDIR=$(dirname ${pkgdir})

  # Install GNU stow - a symlink manager: useful to manage local installations
  gnu_install stow "2.2.2" "$LOCALDIR"
  export PATH=$LOCALDIR/bin:$PATH
  rehash
  echo "End of installation: stow"
  sleep 3

  # Install GNU autotools one by one
  for i in autoconf,"2.69" automake,"1.15.1" libtool,"2.4.5"
  do
    pkgname=${i%,*};
    pkgver=${i#*,};
    gnu_install $pkgname $pkgver "$pkgdir/$pkgname"

    cd $pkgdir
    rm $pkgname/share/info/dir
    echo "stow -v $pkgname"
    stow -v $pkgname
    rehash
    echo "End of installation: $pkgname"

    sleep 3

  done

}

# Execute
package

This will install GNU stow, autoconf, automake, libtool. GNU stow will manage all local installation henceforth as demonstrated below. This is useful to keep the local directory clean, if something goes wrong, and to manage versions.

stow -v <directory>  # symlinks all files and directories to one level above
stow -Dv <directory>  # deletes all symlinks made before

Warning

Manual stow maybe required when run on the compute node due to Perl not being able to access the home directory.

Now let us proceed to build fftw3, p3dfft and pfft. For some reason, the cross- compilation wrappers do not work or fail to link when used with FFT libraries. Therefore we explicitly use Intel compilers in the following steps:

Warning

It is imperative to compile all the tools and FFT libraries in the compute node. This is because as I write now (Feb 2018) Beskow uses different CPU architectures in the login nodes / frontend (Sandy Bridge) compared to compute nodes (Haskell). We know for sure this can affect FFTW compilation, just to be sure, do everything on the compute node.

For fftw3:

#!/bin/bash

# Customizable variables
# ----------------------
pkgname='fftw'
# FFTW version
pkgver=3.3.7
# Directory in which the source tarball will be downloaded and extracted
srcdir=$PWD
# Directory to which the compiled FFTW library will be installed
pkgdir="/cfs/klemming/nobackup/${USER:0:1}/${USER}/opt/pkg/${pkgname}-${pkgver}"
export MAKEFLAGS="-j$(nproc)"

export CC="icc"
export MPICC="mpiicc"
export F77="ifort"

# Should be no reason to change anything below
# --------------------------------------------
download() {
  mkdir -p ${srcdir}
  cd ${srcdir}

  if [ ! -f ${pkgname}-${pkgver}.tar.gz ]; then
    wget http://www.fftw.org/${pkgname}-${pkgver}.tar.gz
  fi
  tar vxzf $pkgname-$pkgver.tar.gz
}

clean() {
  rm -rf ${pkgdir}

  cd ${srcdir}/${pkgname}-${pkgver}-double
  make clean

  cd ${srcdir}/${pkgname}-${pkgver}-single
  make clean
}

build() {
  cd ${srcdir}

  cp -a ${pkgname}-${pkgver} ${pkgname}-${pkgver}-double
  cp -a ${pkgname}-${pkgver} ${pkgname}-${pkgver}-single


  # do not use upstream default CFLAGS for -march/-mtune
  export CFLAGS="-O3 -fomit-frame-pointer -malign-double -fstrict-aliasing -ffast-math -xHost"

  CONFIGURE="./configure F77=$F77 CC=$CC MPICC=$MPICC \
	         --prefix=${pkgdir} \
                 --enable-shared \
		 --enable-threads \
		 --enable-openmp \
		 --enable-mpi "
                 # --host=x86_64-unknown-linux-gnu "

  # build double precision
  cd ${srcdir}/${pkgname}-${pkgver}-double
  $CONFIGURE --enable-sse2 --enable-avx --enable-avx2
  make

  # build & install single precision
  cd ${srcdir}/${pkgname}-${pkgver}-single
  $CONFIGURE --enable-float --enable-sse --enable-sse2 --enable-avx --enable-avx2
  make
}

check() {
  cd ${srcdir}/${pkgname}-${pkgver}-double
  make check

  cd ${srcdir}/${pkgname}-${pkgver}-single
  make check
}

package() {
  set -e
  cd ${srcdir}/${pkgname}-${pkgver}-double
  make install

  cd ${srcdir}/${pkgname}-${pkgver}-single
  make install

  set +e
  cd ${pkgdir}/..
  stow -v $pkgname-$pkgver
}


# Execute the functions above
# ---------------------------
if [ ! -d  ${srcdir}/${pkgname}-${pkgver} ]
then
  download
fi

clean
build
check
package

For p3dfft:

#!/bin/bash

# Customizable variables
# ----------------------
pkgname='p3dfft'
# P3DFFT version
pkgver=2.7.6
# Directory in which the source git repository will be downloaded
srcdir="${PWD}"
# Directory to which the compiled p3dfft library will be installed
pkgdir="/cfs/klemming/nobackup/${USER:0:1}/${USER}/opt/pkg/${pkgname}-${pkgver}"

# C and Fortran 90 MPI compilers
export CC="mpiicc"
export FC="mpiifort"
export CFLAGS="-xHost"
export LDFLAGS="-nofor-main"
# CC="cc"
# FC="ftn"

# FFTW
# ----
# fftwdir="/opt/fftw/3.3.4.0/haswell"
fftwdir="/cfs/klemming/nobackup/${USER:0:1}/${USER}/opt"
autotoolsdir="$fftwdir"

# Should be no reason to change anything below
# --------------------------------------------
git_clone() {
  git clone https://github.com/CyrilleBonamy/p3dfft.git ${srcdir}/${pkgname}-${pkgver} --depth=1
  # git clone https://github.com/sdsc/p3dfft.git ${srcdir}/${pkgname}-${pkgver} --depth=10
}

download() {
  mkdir -p ${srcdir}
  cd ${srcdir}

  if [ ! -f ${pkgname}-${pkgver}.tar.gz ]; then
    wget https://github.com/sdsc/p3dfft/archive/v${pkgver}.tar.gz -O ${pkgname}-${pkgver}.tar.gz
  fi
  tar vxzf ${pkgname}-${pkgver}.tar.gz
}

clean() {
  rm -rf ${srcdir}/${pkgname}-${pkgver}
  rm -rf ${pkgdir}
}

prepare() {
  cd ${srcdir}/${pkgname}-${pkgver}
  # Assuming you had installed and stowed libtool into $autotoolsdir
  cat "${autotoolsdir}/share/aclocal/libtool.m4" \
      "${autotoolsdir}/share/aclocal/ltoptions.m4" \
      "${autotoolsdir}/share/aclocal/ltversion.m4" >> aclocal.m4

  echo 'AC_CONFIG_MACRO_DIRS([m4])' >> configure.ac
  sed -i '1s/^/ACLOCAL_AMFLAGS\ \=\ -I\ m4\n/' Makefile.am
  echo 'ACLOCAL_AMFLAGS = -I m4' >> Makefile.am
}

build() {
  cd ${srcdir}/${pkgname}-${pkgver}

  libtoolize && autoheader && aclocal && autoconf && automake --add-missing
  ## If the above fails, use:
  # autoreconf -fvi
  ./configure \
    --prefix=${pkgdir} \
    --enable-intel \
    --enable-fftw --with-fftw=${fftwdir} \
    CC=${CC} FC=${FC} LDFLAGS=${LDFLAGS}
    # --host=x86_64-unknown-linux-gnu \

  make
}

package() {
  set -e
  cd ${srcdir}/${pkgname}-${pkgver}
  # make install
  ## If the above fails, use (with caution):
  make -i install

  set +e
  cd ${pkgdir}/..
  stow -v $pkgname-$pkgver
}


# Execute the functions above
# ---------------------------
clean
if [ ! -d  ${srcdir}/${pkgname}-${pkgver} ]
then
  ## Use any one of the following
  git_clone
  # download
fi
prepare
build
package

For pfft:

#!/bin/bash

# Customizable variables
# ----------------------
pkgname='pfft'
# PFFT version
pkgver="1.0.8-alpha"
# Directory in which the source git repository will be downloaded
srcdir="${PWD}"
# Directory to which the compiled pfft library will be installed
pkgdir="/cfs/klemming/nobackup/${USER:0:1}/${USER}/opt/pkg/${pkgname}-${pkgver}"
export MAKEFLAGS="-j$(nproc)"

# C and Fortran 90 MPI compilers
export CC=mpiicc
export FC=mpiifort
export CFLAGS="-xHost"
export LDFLAGS="-nofor-main"
# CC="cc"
# FC="ftn"

# FFTW
# ----
# FFTW == 3.3.4 requires patching, whereas 3.3.5 andlater versions should work
# as it is. See: https://github.com/mpip/pfft#install

# You can use the same fftw directory that for p3dfft
# fftwdir="/opt/fftw/3.3.4.0/haswell"
fftwdir="/cfs/klemming/nobackup/${USER:0:1}/${USER}/opt"

# Alternatively, set fftwdir as an empty string and mention fftw include and
# library directories seperately below
fftwinc=""
fftwlib=""

# Should be no reason to change anything below
# --------------------------------------------
git_clone() {
  git clone https://github.com/mpip/pfft.git ${srcdir}/${pkgname}-${pkgver} --depth=10
}

download() {
  mkdir -p ${srcdir}
  cd ${srcdir}

  if [ ! -f ${pkgname}-${pkgver}.tar.gz ]; then
    wget http://www.tu-chemnitz.de/~potts/workgroup/pippig/software/${pkgname}-${pkgver}.tar.gz
  fi
  tar vxzf ${pkgname}-${pkgver}.tar.gz
}

clean() {
  rm -rf ${srcdir}/${pkgname}-${pkgver}
  rm -rf ${pkgdir}
}

build() {
  cd ${srcdir}/${pkgname}-${pkgver}
  export LANG=C
  ./bootstrap.sh
  CONFIGURE="./configure \
            --prefix=${pkgdir} \
            CC=${CC} FC=${FC} MPICC=${CC} MPIFC=${FC} "
            # --host=x86_64-unknown-linux-gnu
  if [ -n "$fftwdir" ]; then
    CONFIGURE+="--with-fftw3=${fftwdir}"
  else
    CONFIGURE+="CPPFLAGS=-I${fftwinc}  LDFLAGS=-L${fftwlib}"
  fi
  echo ${CONFIGURE}
  ${CONFIGURE}
  make
}

package() {
  set -e
  cd ${srcdir}/${pkgname}-${pkgver}
  make install

  set +e
  cd ${pkgdir}/..
  stow -v $pkgname-$pkgver
}


# Execute the functions above
# ---------------------------
clean
if [ ! -d  ${srcdir}/${pkgname}-${pkgver} ]
then
  ## Use any one of the following
  git_clone
  # download
fi
build
package

Ensure that the stowed library location has been prepended to LD_LIBRARY_PATH. Finally, we can install fluidfft:

hg clone https://foss.heptapod.net/fluiddyn/fluidfft
cd fluidfft

Activate the virutal environment. Copy site.cfg.default to site.cfg within fluidfft directory or ~/.fluidfft-site.cfg with all libraries mentioned above set like:

use = True
dir = /cfs/klemming/nobackup/u/user
include_dir =
library_dir =

Install FluidFFT with:

export CRAYPE_LINK_TYPE=dynamic
module load craype-hugepages2M
MPICXX=CC LDSHARED="CC -shared" python setup.py develop
module rm craype-hugepages2M

Test your installation by allocating a compute node as follows:

salloc -A <project> -N 1 -t 05:00
export CRAY_ROOTFS=DSL
export OMP_NUM_THREADS=1
aprun -n 1 python -m unittest discover
aprun -n 2 python -m unittest discover