How to compile HPL-GPU: Difference between revisions

From HPCWIKI
Jump to navigation Jump to search
(Add categories: GPU, Guide)
({{Status}}/{{TOC}} 템플릿 제거)
 
(3 intermediate revisions by the same user not shown)
Line 1: Line 1:
== Background ==
== Overview ==
There are many combination to compile High Performance LINPACK (HPL) with different configurations such as different compiler, different basic linear algebra subprograms (BLAS), massage passing interface (MPI) libraries: for example
 
How to compile HPL-GPU에 대한 기술 문서입니다.
 
=== Summary ===
 
* 무엇인가? - How to compile HPL-GPU
* 왜 필요한가? - HPC 및 서버 환경에서 필수 개념
* 언제 사용하는가? - 서버 구성, 성능 튜닝, 문제 해결 시
 
== Purpose ==
 
이 문서가 존재하는 이유
 
* Goal: How to compile HPL-GPU에 대한 기술 정보 제공
* Scope: How to compile HPL-GPU의 개념, 사용법, 설정
* Non-goals: 다른 주제로의 확장


* Which compiler + HPL + which Blas (OpenBLAS / Intel MKL / CuBLAS) + which MPI (OpenMPI, MPICH, Intel MPI)
== Key Concepts ==


== Build High Performance LINPACK with CUDA ==
{| class="wikitable"
In this post, we are going to use GNU compiler, OpenBLAS, OpenMPI for HPL-GPU
! Concept
! Description
! Related
|-
| How to compile HPL-GPU
| HPC/서버 환경에서 중요한 기술 개념
| [[Linux]], [[Server]]
|}


=== Assumption ===
== Detailed Explanation ==


* The build system is Intel Skylake CPU, 128GB Memory, Nvidia GPUs on Ubuntu 20.04  
There are many combination to compile High Performance LINPACK (HPL) with different configurations such as different compiler, different basic linear algebra subprograms (BLAS), massage passing interface (MPI) libraries: for example
* Which compiler + HPL + which Blas (OpenBLAS / Intel MKL / [[CuBLAS]]) + which MPI (OpenMPI, MPICH, Intel MPI)
In this post, we are going to use GNU compiler, OpenBLAS, OpenMPI for HPL-GPU
* The build system is Intel Skylake CPU, 128GB Memory, [[NVIDIA GPUs|Nvidia GPUs]] on Ubuntu 20.04  
* [[NVIDIA driver]] is ready and [[CUDA]] 12.0 has been installed and [[library]] can be found at /usr/local/cuda  
* [[NVIDIA driver]] is ready and [[CUDA]] 12.0 has been installed and [[library]] can be found at /usr/local/cuda  
* We are using OpenMPI, OpenBlas and hpl-2.0_FERMI_v15 as a ingredient, gcc version 10, g++ version 10 and gfortran version 10
* We are using OpenMPI, OpenBlas and hpl-2.0_FERMI_v15 as a ingredient, gcc version 10, g++ version 10 and gfortran version 10
=== Step 1, Compile OpenBLAS<ref><nowiki>https://github.com/xianyi/OpenBLAS/wiki/User-Manual#compile-the-library</nowiki></ref> ===
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
_version=v0.3.23
_version=v0.3.23
Line 19: Line 42:
wget http://github.com/xianyi/OpenBLAS/archive/$FILE
wget http://github.com/xianyi/OpenBLAS/archive/$FILE
tar -xzvf $FILE
tar -xzvf $FILE
cd OpenBLAS-0.3.23
cd OpenBLAS-0.3.23
# OpenBLAS does not support f77, GCC or Intel are supported
# OpenBLAS does not support f77, GCC or Intel are supported
# type make to detect the CPU automatically. or type make TARGET=xxx to set target CPU, e.g. make TARGET=NEHALEM. The full target list is in file TargetList.txt.
# type make to detect the CPU automatically. or type make TARGET=xxx to set target CPU, e.g. make TARGET=NEHALEM. The full target list is in file TargetList.txt.
make  
make  
# To install the library, you can run "make PREFIX=/path/to/your/installation install".
# To install the library, you can run "make PREFIX=/path/to/your/installation install".
mkdir -p /opt/hpcmate/home/lib/OpenBLAS-0.3.23
mkdir -p /opt/hpcmate/home/lib/OpenBLAS-0.3.23
Line 31: Line 52:
</syntaxhighlight>now, compiled OpenBLAS's library and header file is available ~/lib/OpenBLAS-0.3.23
</syntaxhighlight>now, compiled OpenBLAS's library and header file is available ~/lib/OpenBLAS-0.3.23


=== Step 2, Compile OpenMPI<ref><nowiki>https://www.open-mpi.org/software/ompi/v2.0/</nowiki> </ref><ref><nowiki>https://sites.google.com/site/rangsiman1993/comp-env/program-install/install-openmpi</nowiki> </ref> ===
== Best Practices ==
<syntaxhighlight lang="bash">
FILE=openmpi-2.0.2.tar.gz
wget https://download.open-mpi.org/release/open-mpi/v2.0/$FILE
tar -xzvf $FILE


cd openmpi-2.0.2
* 최신 버전 사용 권장
./configure --prefix=/opt/hpcmate/home/lib/openmpi-2.0.2
* 공식 문서 참고
make all install
* 테스트 환경에서 먼저 검증
</syntaxhighlight>now, compiled OpenMPI 's library and header file is available ~/lib/openmpi-2.0.2


=== Step 3, Compile hpl-2.0_FERMI_v15 ===
== References ==
<syntaxhighlight lang="bash">
#download hpl-2.0_FERMI_v15.tgz and untar
cd hpl-2.0_FERMI_v15
</syntaxhighlight>Now we need to modify Makefile to fit our build environment by editing Make.CUDA file in hpl-2.0_FERMI_v15 folder.<syntaxhighlight lang="bash">
######Edit the below lines with your settings###########
#hpl-2.0_FERMI_v15 path
TOPdir      = /opt/hpcmate/home/try3/hpl-gpu/hpl-2.0_FERMI_v15


#MPI library path
* [https://wiki.hpcmate.com How to compile HPL-GPU]
MPdir        = /opt/hpcmate/home/lib/openmpi-2.0.2
MPinc        = -I$(MPdir)/include
MPlib        = $(MPdir)/lib/libmpi.so


#BLAS library path
== Related Pages ==
LAdir        = /opt/hpcmate/home/lib/OpenBLAS-0.3.23
LAinc        =
LAlib        = -L$(TOPdir)/src/cuda -ldgemm -L/usr/local/cuda/lib64 -lcuda -lcublas -lcudart -L$(LAdir)/lib -lopenblas -lpthread


# CC and Linker
* [[Linux]]
CC      = /opt/hpcmate/home/lib/openmpi-2.0.2/bin/mpicc
* [[Server]]
CCFLAGS = $(HPL_DEFS) -fomit-frame-pointer -O3 -funroll-loops -w -Wall -fopenmp
* [[Hardware]]
* [[Network]]


#################################################
[[Category:GPU]]
</syntaxhighlight>To remove Intel MKL dependency in source code at hpl-2.0_FERMI_v15/src/cuda/cuda_dgemm.c, we also need to change from intel MKL to OpenBLAS. To do that, we need to edit hpl-2.0_FERMI_v15/src/cuda/cuda_dgemm.c so that cuda_dgemm can use OpenBLAS instead of Intel MKL. <ref>http://hwengineer.blogspot.com/2018/03/power9-ac922-hpl-cuda-compile.html</ref>
== Knowledge Graph ==


<syntaxhighlight lang="bash">
Related
# find and edit following lines at hpl-2.0_FERMI_v15/src/cuda/cuda_dgemm.c as


...
→ [[CUDA]]
// handle2 = dlopen ("libmkl_intel_lp64.so", RTLD_LAZY);
→ [[GPU]]
handle2 = dlopen ("libopenblas.so", RTLD_LAZY);
→ [[NVIDIA GPU]]
...
→ [[NVLink]]
 
→ [[AMD GPUs]]
//dgemm_mkl = (void(*)())dlsym(handle, "dgemm");
→ [[MIG]]
dgemm_mkl = (void(*)())dlsym(handle, "dgemm_");
→ [[NVIDIA GPUs]]
 
→ [[Tensor Core]]
...
//handle = dlopen ("libmkl_intel_lp64.so", RTLD_LAZY);
handle = dlopen ("libopenblas.so", RTLD_LAZY);
 
...
//mkl_dtrsm = (void(*)())dlsym(handle2, "dtrsm");
mkl_dtrsm = (void(*)())dlsym(handle2, "dtrsm_");
 
...
</syntaxhighlight>otherwise,  we will face runtime error something like :
''libmkl_intel_lp64.so: cannot open shared object file: No such file or directory''
 
''libopenblas.so.0: undefined symbol: dtrsm''
 
''libopenblas.so.0: undefined symbol: dgemm''
 
 
Now we also need to setup required environment to point right path to compile HPL-GPU,<syntaxhighlight lang="bash">
# crate setenv.sh file with following contents
export CC=gcc
export CXX=g++
export F77=gfortran
export FC=gfortran
export FC90=gfortran
export PATH=/opt/hpcmate/home/lib/openmpi-2.0.2/bin:$PATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/hpcmate/home/lib/openmpi-2.0.2/lib:/opt/hpcmate/home/lib/OpenBLAS-0.3.23/lib:/opt/hpcmate/home/try3/hpl-gpu/hpl-2.0_FERMI_v15/src/cuda
</syntaxhighlight>Now we are ready to compile HPL-GPU, <syntaxhighlight lang="bash">
#set PATH and LD_LIBRARY_PATH
source setenv.sh #the definition above
 
#clean up
cd hpl-2.0_FERMI_v15
$make arch=CUDA clean
$make arch=CUDA
 
#for successful compilation, we will get ./bin/CUDA with xhpl and HPL.dat
 
</syntaxhighlight>
 
== Reminder ==
 
* Use the same compile tools and version to compile required libraries such as OpenMPI, OpenBlas and HPL-GPU
* Set the correct PATH and LD_LIBRARY_PATH to point right things
* MPdir, LAdir, CC should be matched with what you are trying to link
 
== Todo ==
 
* How to adjust HPL.dat
* Automated compile script  - we are going to share automated compile script of this post through repository.hpcmate.com. please [[contact]] support@hpcmate.com, if you need
 
== References ==
<references />
[[Category:GPU]]


[[Category:Guide]]
[[Category:Guide]]

Latest revision as of 13:24, 2 October 2026

Overview

How to compile HPL-GPU에 대한 기술 문서입니다.

Summary

  • 무엇인가? - How to compile HPL-GPU
  • 왜 필요한가? - HPC 및 서버 환경에서 필수 개념
  • 언제 사용하는가? - 서버 구성, 성능 튜닝, 문제 해결 시

Purpose

이 문서가 존재하는 이유

  • Goal: How to compile HPL-GPU에 대한 기술 정보 제공
  • Scope: How to compile HPL-GPU의 개념, 사용법, 설정
  • Non-goals: 다른 주제로의 확장

Key Concepts

Concept Description Related
How to compile HPL-GPU HPC/서버 환경에서 중요한 기술 개념 Linux, Server

Detailed Explanation

There are many combination to compile High Performance LINPACK (HPL) with different configurations such as different compiler, different basic linear algebra subprograms (BLAS), massage passing interface (MPI) libraries: for example

  • Which compiler + HPL + which Blas (OpenBLAS / Intel MKL / CuBLAS) + which MPI (OpenMPI, MPICH, Intel MPI)

In this post, we are going to use GNU compiler, OpenBLAS, OpenMPI for HPL-GPU

  • The build system is Intel Skylake CPU, 128GB Memory, Nvidia GPUs on Ubuntu 20.04
  • NVIDIA driver is ready and CUDA 12.0 has been installed and library can be found at /usr/local/cuda
  • We are using OpenMPI, OpenBlas and hpl-2.0_FERMI_v15 as a ingredient, gcc version 10, g++ version 10 and gfortran version 10
_version=v0.3.23
FILE=$_version.tar.gz 
wget http://github.com/xianyi/OpenBLAS/archive/$FILE
tar -xzvf $FILE
cd OpenBLAS-0.3.23
# OpenBLAS does not support f77, GCC or Intel are supported
# type make to detect the CPU automatically. or type make TARGET=xxx to set target CPU, e.g. make TARGET=NEHALEM. The full target list is in file TargetList.txt.
make 
# To install the library, you can run "make PREFIX=/path/to/your/installation install".
mkdir -p /opt/hpcmate/home/lib/OpenBLAS-0.3.23
make PREFIX=/opt/hpcmate/home/lib/OpenBLAS-0.3.23 install 
ls ~/lib/OpenBLAS-0.3.23/lib

now, compiled OpenBLAS's library and header file is available ~/lib/OpenBLAS-0.3.23

Best Practices

  • 최신 버전 사용 권장
  • 공식 문서 참고
  • 테스트 환경에서 먼저 검증

References

Related Pages

Knowledge Graph

Related

→ CUDA → GPU → NVIDIA GPU → NVLink → AMD GPUs → MIG → NVIDIA GPUs → Tensor Core

Retrieved from "http://wiki.hpcmate.com/index.php?title=How_to_compile_HPL-GPU&oldid=3789"