Error related to vasp6.4 with GPU installation on HPC

[mukhtiyar_singh]

Greetings all!
During installation of VASP6.4 with GPU Nvidia HPC-SDK 22.5 , We are facing an error during compilation, as mentioned below and attached makefile.include

During the build process with make, we are facing the following error:

Code: Select all

NVFORTRAN-F-0004-Unable to open MODULE file class_set.mod (subdftd4.F: 12)
NVFORTRAN/x86-64 Linux 22.5-0: compilation aborted 

.............................................................makefile.include ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

Code: Select all

# Default precompiler options
CPP_OPTIONS = -DHOST=\"LinuxNV\" \
              -DMPI -DMPI_INPLACE -DMPI_BLOCK=8000 -Duse_collective \
              -DscaLAPACK \
              -DCACHE_SIZE=4000 \
              -Davoidalloc \
              -Dvasp6 \
              -Duse_bse_te \
              -Dtbdyn \
              -Dqd_emulate \
              -Dfock_dblbuf \
              -D_OPENMP \
              -D_OPENACC \
              -DUSENCCL -DUSENCCLP2P

CPP         = nvfortran -Mpreprocess -Mfree -Mextend -E $(CPP_OPTIONS) $*$(FUFFIX)  > $*$(SUFFIX)

# N.B.: you might need to change the cuda-version here
#       to one that comes with your NVIDIA-HPC SDK

FC          = mpif90 -acc -gpu=cc60,cc70,cc80,cuda11.7 -mp
FCL         = mpif90 -acc -gpu=cc60,cc70,cc80,cuda11.7 -mp -c++libs

FREE        = -Mfree

FFLAGS      = -Mbackslash -Mlarge_arrays

OFLAG       = -fast

DEBUG       = -Mfree -O0 -traceback

OBJECTS     = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o

LLIBS       = -cudalib=cublas,cusolver,cufft,nccl -cuda

# Redefine the standard list of O1 and O2 objects
SOURCE_O1  := pade_fit.o minimax_dependence.o
SOURCE_O2  := pead.o

# For what used to be vasp.5.lib
CPP_LIB     = $(CPP)
FC_LIB      = nvfortran
CC_LIB      = nvc -w
CFLAGS_LIB  = -O
FFLAGS_LIB  = -O1 -Mfixed
FREE_LIB    = $(FREE)

OBJECTS_LIB = linpack_double.o

# For the parser library
CXX_PARS    = nvc++ --no_warnings

##
## Customize as of this point! Of course you may change the preceding
 ## part of this file as well if you like, but it should rarely be
## necessary ...
##
# When compiling on the target machine itself , change this to the
# relevant target when cross-compiling for another architecture
VASP_TARGET_CPU ?= -tp host
FFLAGS     += $(VASP_TARGET_CPU)

# Specify your NV HPC-SDK installation (mandatory)
#... first try to set it automatically
NVROOT      =$(shell which nvfortran | awk -F /compilers/bin/nvfortran '{ print $$1 }')

# If the above fails, then NVROOT needs to be set manually
#NVHPC      ?= /opt/nvidia/hpc_sdk
#NVVERSION   = 21.11
#NVROOT      = $(NVHPC)/Linux_x86_64/$(NVVERSION)

## Improves performance when using NV HPC-SDK >=21.11 and CUDA >11.2
#OFLAG_IN   = -fast -Mwarperf
#SOURCE_IN  := nonlr.o

# Software emulation of quadruple precsion (mandatory)
QD         ?= $(NVROOT)/compilers/extras/qd
LLIBS      += -L$(QD)/lib -lqdmod -lqd
INCS       += -I$(QD)/include/qd

# BLAS (mandatory)
BLAS        = -lblas

# LAPACK (mandatory)
LAPACK      = -llapack

# scaLAPACK (mandatory)
SCALAPACK   = -Mscalapack

LLIBS      += $(SCALAPACK) $(LAPACK) $(BLAS)

# FFTW (mandatory)
FFTW_ROOT  ?= /nlsasfs/home/harvest/muktiyar/Software/fftw-3.3.10/fftw_build
LLIBS      += -L$(FFTW_ROOT)/lib -lfftw3 -lfftw3_omp
INCS       += -I$(FFTW_ROOT)/include

# HDF5-support (optional but strongly recommended)
CPP_OPTIONS+= -DVASP_HDF5
HDF5_ROOT  ?=  /nlsasfs/home/harvest/muktiyar/Software/hdf5-1.14.2/hdf5_build
LLIBS      += -L$(HDF5_ROOT)/lib -lhdf5_fortran
INCS       += -I$(HDF5_ROOT)/include
# For the VASP-2-Wannier90 interface (optional)
CPP_OPTIONS    += -DVASP2WANNIER90
WANNIER90_ROOT ?= /nlsasfs/home/harvest/muktiyar/Software/wannier90/wannier90-3.1.0
LLIBS          += -L$(WANNIER90_ROOT)/lib -lwannier

#libeef
#CPP_OPTIONS  += -Dlibbeef
#LIBBEEF_ROOT ?= /nlsasfs/home/harvest/muktiyar/Software/libbeef/libeef_build
#LLIBS        += -L$(LIBBEEF_ROOT)/lib -lbeef

#dftd4
CPP_OPTIONS += -DDFTD4
DFTD4_ROOT  ?= /nlsasfs/home/harvest/muktiyar/Software/dftd4
LLIBS       += -L$(DFTD4_ROOT)/build -ldftd4
INCS        += -I$(DFTD4_ROOT)/libdftd4.a.p


# For the fftlib library (hardly any benefit for the OpenACC GPU port)
#CPP_OPTIONS+= -Dsysv
#FCL        += fftlib.o
#CXX_FFTLIB  = nvc++ -mp --no_warnings -std=c++11 -DFFTLIB_THREADSAFE
#INCS_FFTLIB = -I./include -I$(FFTW_ROOT)/include
#LIBS       += fftlib
#LLIBS      += -ldl

[merzuk.kaltak]

Dear mukhtiyar_singh,

According to dftd4's documentation one has to compile the dftd4 library with the same compiler as used for vasp.
Unfortunately, I encounter following error when using hpc_sdk 22.11:

Code: Select all

meson.build:30:0: ERROR: Value "c11" (of type "string") for combo option "C language standard to use" is not one of the choices. Possible choices are (as string): "none".

Looking on a corresponding github issue of dftd4 does not seem promising.
Have you been able to compile the dft4 code successfully with the nvidia compiler?

[mukhtiyar_singh]

No, we are tryign to do it with only vasp based on gpu for now.

[merzuk.kaltak]

Unfortunately this means there is not a lot we can do from our side.
You have only one choice left. That is, to compile the openacc version of vasp using hpc_sdk without the dftd4 library by commenting out the corresponding lines in makefile.include, like so:

Code: Select all

#CPP_OPTIONS += -DDFTD4
#DFTD4_ROOT  ?= /nlsasfs/home/harvest/muktiyar/Software/dftd4
#LLIBS       += -L$(DFTD4_ROOT)/build -ldftd4
#INCS        += -I$(DFTD4_ROOT)/libdftd4.a.p

Maybe reaching out to the developers of dftd4 could help with this problem.

[mukhtiyar_singh]

We had tried with dftd4 also and we got the same error. What we can do to solve it?please guide

[merzuk.kaltak]

Could you please try compiling vasp without linking to the dftd4 library as described in my previous post?
When doing so, it might be wise to wipe the build folder before, i.e.

Code: Select all

make veryclean

followed by

Code: Select all

make std gam ncl 

[paulfons]

I have done pretty the much the same thing as the current poster (with different directories for the dftd4 libraries of course). The GPU version (acc) build and runs fine without the DFTD4 module. I am using nvfortran 23.9-0.

The makefile.include file is attached below.

Code: Select all

# Default precompiler options
CPP_OPTIONS = -DHOST=\"LinuxNV\" \
              -DMPI -DMPI_INPLACE -DMPI_BLOCK=8000 -Duse_collective \
              -DscaLAPACK \
              -DCACHE_SIZE=4000 \
              -Davoidalloc \
              -Dvasp6 \
              -Duse_bse_te \
              -Dtbdyn \
              -Dqd_emulate \
              -Dfock_dblbuf \
              -D_OPENACC \
              -DUSENCCL -DUSENCCLP2P

CPP         = nvfortran -Mpreprocess -Mfree -Mextend -E $(CPP_OPTIONS) $*$(FUFFIX)  > $*$(SUFFIX)

# N.B.: you might need to change the cuda-version here
#       to one that comes with your NVIDIA-HPC SDK
FC          = mpif90 -acc -gpu=cc60,cc70,cc80,cuda12.2
FCL         = mpif90 -acc -gpu=cc60,cc70,cc80,cuda12.2 -c++libs

FREE        = -Mfree

FFLAGS      = -Mbackslash -Mlarge_arrays

OFLAG       = -fast

DEBUG       = -Mfree -O0 -traceback

OBJECTS     = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o

LLIBS       = -cudalib=cublas,cusolver,cufft,nccl -cuda

# Redefine the standard list of O1 and O2 objects
SOURCE_O1  := pade_fit.o minimax_dependence.o
SOURCE_O2  := pead.o

# For what used to be vasp.5.lib
CPP_LIB     = $(CPP)
FC_LIB      = nvfortran
CC_LIB      = nvc -w
CFLAGS_LIB  = -O
FFLAGS_LIB  = -O1 -Mfixed
FREE_LIB    = $(FREE)

OBJECTS_LIB = linpack_double.o

# For the parser library
CXX_PARS    = nvc++ --no_warnings

##
## Customize as of this point! Of course you may change the preceding
## part of this file as well if you like, but it should rarely be
## necessary ...
##
# When compiling on the target machine itself , change this to the
# relevant target when cross-compiling for another architecture
VASP_TARGET_CPU ?= -tp host
FFLAGS     += $(VASP_TARGET_CPU)

# Specify your NV HPC-SDK installation (mandatory)
#... first try to set it automatically
NVROOT      =$(shell which nvfortran | awk -F /compilers/bin/nvfortran '{ print $$1 }')

# If the above fails, then NVROOT needs to be set manually
#NVHPC      ?= /opt/nvidia/hpc_sdk
#NVVERSION   = 21.11
#NVROOT      = $(NVHPC)/Linux_x86_64/$(NVVERSION)

## Improves performance when using NV HPC-SDK >=21.11 and CUDA >11.2
#OFLAG_IN   = -fast -Mwarperf
#SOURCE_IN  := nonlr.o

# Software emulation of quadruple precsion (mandatory)
QD         ?= $(NVROOT)/compilers/extras/qd
LLIBS      += -L$(QD)/lib -lqdmod -lqd
INCS       += -I$(QD)/include/qd

# BLAS (mandatory)
BLAS        = -lblas

# LAPACK (mandatory)
LAPACK      = -llapack

# scaLAPACK (mandatory)
SCALAPACK   = -Mscalapack

LLIBS      += $(SCALAPACK) $(LAPACK) $(BLAS)

# FFTW (mandatory)
FFTW_ROOT  ?= /opt/fftw3
LLIBS      += -L$(FFTW_ROOT)/lib -lfftw3
INCS       += -I$(FFTW_ROOT)/include

# DFTD4
CPP_OPTIONS += -DDFTD4
LLIBS       += -L /data/Software/dftd4/install/lib64
INCS        += -L /data/Software/dftd4/install/include

# HDF5-support (optional but strongly recommended)
#CPP_OPTIONS+= -DVASP_HDF5
#HDF5_ROOT  ?= /path/to/your/hdf5/installation
#LLIBS      += -L$(HDF5_ROOT)/lib -lhdf5_fortran
#INCS       += -I$(HDF5_ROOT)/include

# For the VASP-2-Wannier90 interface (optional)
#CPP_OPTIONS    += -DVASP2WANNIER90
#WANNIER90_ROOT ?= /path/to/your/wannier90/installation
#LLIBS          += -L$(WANNIER90_ROOT)/lib -lwannier

[mukhtiyar_singh]

VASP 6.4 is not installed with Wannier90 and hdf5 interfacing only. We have tried it interface with only these two codes (All other are removed). But the error is coming again and again. Without any interfacing VASP6.4 is working properly on HPC with GPU. To interface these two codes(wanner90 and hdf5) I am following the same method with which my vasp 6.2 is working properly on my local mechine and also the same given on vaspwiki. But the same is not working with vasp6.3 and vasp6.4.