How to compile HPL-GPU: Difference between revisions

From HPCWIKI
Jump to navigation Jump to search
(Created page with "== Background == There are many combination to compile High Performance LINPACK (HPL) with different configurations such as different compiler, different basic linear algebra subprograms (BLAS), massage passing interface (MPI) libraries: for example * Which compiler + HPL + which Blas (OpenBLAS / Intel MKL / CuBLAS) + which MPI (OpenMPI, MPICH, Intel MPI) == Build High Performance LINPACK with CUDA == In this post, we are going to use GNU compiler, OpenBLAS, OpenMPI f...")
 
(Fix: remove --- horizontal lines (7 removed))
 
(3 intermediate revisions by the same user not shown)
Line 1: Line 1:
== Background ==
{{Status
There are many combination to compile High Performance LINPACK (HPL) with different configurations such as different compiler, different basic linear algebra subprograms (BLAS), massage passing interface (MPI) libraries: for example
|status=Draft
|owner=Knowledge Agent
|last_update=2026-07-16
|review=Pending
}}
 
{{TOC}}
 
== Overview ==
 
How to compile HPL-GPU에 대한 기술 문서입니다.
 
=== Summary ===
 
* 무엇인가? - How to compile HPL-GPU
* 왜 필요한가? - HPC 및 서버 환경에서 필수 개념
* 언제 사용하는가? - 서버 구성, 성능 튜닝, 문제 해결 시
 
 
== Purpose ==
 
이 문서가 존재하는 이유
 
* Goal: How to compile HPL-GPU에 대한 기술 정보 제공
* Scope: How to compile HPL-GPU의 개념, 사용법, 설정
* Non-goals: 다른 주제로의 확장
 
 
== Key Concepts ==


* Which compiler + HPL + which Blas (OpenBLAS / Intel MKL / CuBLAS) + which MPI (OpenMPI, MPICH, Intel MPI)
{| class="wikitable"
! Concept
! Description
! Related
|-
| How to compile HPL-GPU
| HPC/서버 환경에서 중요한 기술 개념
| [[Linux]], [[Server]]
|}


== Build High Performance LINPACK with CUDA ==
In this post, we are going to use GNU compiler, OpenBLAS, OpenMPI for HPL-GPU


=== Assumption ===
== Detailed Explanation ==


* The build system is Intel Skylake CPU, 128GB Memory, Nvidia GPUs on Ubuntu 20.04  
There are many combination to compile High Performance LINPACK (HPL) with different configurations such as different compiler, different basic linear algebra subprograms (BLAS), massage passing interface (MPI) libraries: for example
* Which compiler + HPL + which Blas (OpenBLAS / Intel MKL / CuBLAS) + which MPI (OpenMPI, MPICH, Intel MPI)
In this post, we are going to use GNU compiler, OpenBLAS, OpenMPI for HPL-GPU
* The build system is Intel Skylake CPU, 128GB Memory, [[NVIDIA GPUs|Nvidia GPUs]] on Ubuntu 20.04  
* [[NVIDIA driver]] is ready and [[CUDA]] 12.0 has been installed and [[library]] can be found at /usr/local/cuda  
* [[NVIDIA driver]] is ready and [[CUDA]] 12.0 has been installed and [[library]] can be found at /usr/local/cuda  
* We are using OpenMPI, OpenBlas and hpl-2.0_FERMI_v15 as a ingredient, gcc version 10, g++ version 10 and gfortran version 10
* We are using OpenMPI, OpenBlas and hpl-2.0_FERMI_v15 as a ingredient, gcc version 10, g++ version 10 and gfortran version 10
=== Step 1, Compile OpenBLAS<ref><nowiki>https://github.com/xianyi/OpenBLAS/wiki/User-Manual#compile-the-library</nowiki></ref> ===
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
_version=v0.3.23
_version=v0.3.23
Line 19: Line 54:
wget http://github.com/xianyi/OpenBLAS/archive/$FILE
wget http://github.com/xianyi/OpenBLAS/archive/$FILE
tar -xzvf $FILE
tar -xzvf $FILE
cd OpenBLAS-0.3.23
cd OpenBLAS-0.3.23
# OpenBLAS does not support f77, GCC or Intel are supported
# OpenBLAS does not support f77, GCC or Intel are supported
# type make to detect the CPU automatically. or type make TARGET=xxx to set target CPU, e.g. make TARGET=NEHALEM. The full target list is in file TargetList.txt.
# type make to detect the CPU automatically. or type make TARGET=xxx to set target CPU, e.g. make TARGET=NEHALEM. The full target list is in file TargetList.txt.
make  
make  
# To install the library, you can run "make PREFIX=/path/to/your/installation install".
# To install the library, you can run "make PREFIX=/path/to/your/installation install".
mkdir -p /opt/hpcmate/home/lib/OpenBLAS-0.3.23
mkdir -p /opt/hpcmate/home/lib/OpenBLAS-0.3.23
Line 31: Line 64:
</syntaxhighlight>now, compiled OpenBLAS's library and header file is available ~/lib/OpenBLAS-0.3.23
</syntaxhighlight>now, compiled OpenBLAS's library and header file is available ~/lib/OpenBLAS-0.3.23


=== Step 2, Compile OpenMPI<ref><nowiki>https://www.open-mpi.org/software/ompi/v2.0/</nowiki> </ref><ref><nowiki>https://sites.google.com/site/rangsiman1993/comp-env/program-install/install-openmpi</nowiki> </ref> ===
<syntaxhighlight lang="bash">
FILE=openmpi-2.0.2.tar.gz
wget https://download.open-mpi.org/release/open-mpi/v2.0/$FILE
tar -xzvf $FILE


cd openmpi-2.0.2
== Best Practices ==
./configure --prefix=/opt/hpcmate/home/lib/openmpi-2.0.2
make all install
</syntaxhighlight>now, compiled OpenMPI 's library and header file is available ~/lib/openmpi-2.0.2


=== Step 3, Compile hpl-2.0_FERMI_v15 ===
* 최신 버전 사용 권장
<syntaxhighlight lang="bash">
* 공식 문서 참고
#download hpl-2.0_FERMI_v15.tgz and untar
* 테스트 환경에서 먼저 검증
cd hpl-2.0_FERMI_v15
</syntaxhighlight>Now we need to modify Makefile to fit our build environment by editing Make.CUDA file in hpl-2.0_FERMI_v15 folder.<syntaxhighlight lang="bash">
######Edit the below lines with your settings###########
#hpl-2.0_FERMI_v15 path
TOPdir      = /opt/hpcmate/home/try3/hpl-gpu/hpl-2.0_FERMI_v15


#MPI library path
MPdir        = /opt/hpcmate/home/lib/openmpi-2.0.2
MPinc        = -I$(MPdir)/include
MPlib        = $(MPdir)/lib/libmpi.so


#BLAS library path
== References ==
LAdir        = /opt/hpcmate/home/lib/OpenBLAS-0.3.23
LAinc        =
LAlib        = -L$(TOPdir)/src/cuda -ldgemm -L/usr/local/cuda/lib64 -lcuda -lcublas -lcudart -L$(LAdir)/lib -lopenblas -lpthread
 
# CC and Linker
CC      = /opt/hpcmate/home/lib/openmpi-2.0.2/bin/mpicc
CCFLAGS = $(HPL_DEFS) -fomit-frame-pointer -O3 -funroll-loops -w -Wall -fopenmp
 
#################################################
</syntaxhighlight>To remove Intel MKL dependency in source code at hpl-2.0_FERMI_v15/src/cuda/cuda_dgemm.c, we also need to change from intel MKL to OpenBLAS. To do that, we need to edit hpl-2.0_FERMI_v15/src/cuda/cuda_dgemm.c so that cuda_dgemm can use OpenBLAS instead of Intel MKL. <ref>http://hwengineer.blogspot.com/2018/03/power9-ac922-hpl-cuda-compile.html</ref>
 
<syntaxhighlight lang="bash">
# find and edit following lines at hpl-2.0_FERMI_v15/src/cuda/cuda_dgemm.c as
 
...
// handle2 = dlopen ("libmkl_intel_lp64.so", RTLD_LAZY);
handle2 = dlopen ("libopenblas.so", RTLD_LAZY);
...


//dgemm_mkl = (void(*)())dlsym(handle, "dgemm");
* [https://wiki.hpcmate.com How to compile HPL-GPU]
dgemm_mkl = (void(*)())dlsym(handle, "dgemm_");


...
//handle = dlopen ("libmkl_intel_lp64.so", RTLD_LAZY);
handle = dlopen ("libopenblas.so", RTLD_LAZY);


...
== Related Pages ==
//mkl_dtrsm = (void(*)())dlsym(handle2, "dtrsm");
mkl_dtrsm = (void(*)())dlsym(handle2, "dtrsm_");


...
* [[Linux]]
</syntaxhighlight>otherwise,  we will face runtime error something like :
* [[Server]]
''libmkl_intel_lp64.so: cannot open shared object file: No such file or directory''
* [[Hardware]]
* [[Network]]


''libopenblas.so.0: undefined symbol: dtrsm''


''libopenblas.so.0: undefined symbol: dgemm''
[[Category:GPU]]
== Knowledge Graph ==


Related


Now we also need to setup required environment to point right path to compile HPL-GPU,<syntaxhighlight lang="bash">
→ [[CUDA]]
# crate setenv.sh file with following contents
→ [[GPU]]
export CC=gcc
→ [[NVIDIA GPU]]
export CXX=g++
→ [[NVLink]]
export F77=gfortran
→ [[AMD GPUs]]
export FC=gfortran
→ [[MIG]]
export FC90=gfortran
→ [[NVIDIA GPUs]]
export PATH=/opt/hpcmate/home/lib/openmpi-2.0.2/bin:$PATH
→ [[Tensor Core]]
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/hpcmate/home/lib/openmpi-2.0.2/lib:/opt/hpcmate/home/lib/OpenBLAS-0.3.23/lib:/opt/hpcmate/home/try3/hpl-gpu/hpl-2.0_FERMI_v15/src/cuda
</syntaxhighlight>Now we are ready to compile HPL-GPU, <syntaxhighlight lang="bash">
#set PATH and LD_LIBRARY_PATH
source setenv.sh #the definition above


#clean up
[[Category:Guide]]
cd hpl-2.0_FERMI_v15
$make arch=CUDA clean
$make arch=CUDA
 
#for successful compilation, we will get ./bin/CUDA with xhpl and HPL.dat
 
</syntaxhighlight>
 
== Reminder ==
 
* Use the same compile tools and version to compile required libraries such as OpenMPI, OpenBlas and HPL-GPU
* Set the correct PATH and LD_LIBRARY_PATH to point right things
* MPdir, LAdir, CC should be matched with what you are trying to link
 
== Todo ==
 
* How to adjust HPL.dat
* Automated compile script  - we are going to share automated compile script of this post through repository.hpcmate.com. please [[contact]] support@hpcmate.com, if you need
 
== References ==
<references />

Latest revision as of 11:28, 17 July 2026

Template:Status

Template:TOC

Overview

How to compile HPL-GPU에 대한 기술 문서입니다.

Summary

  • 무엇인가? - How to compile HPL-GPU
  • 왜 필요한가? - HPC 및 서버 환경에서 필수 개념
  • 언제 사용하는가? - 서버 구성, 성능 튜닝, 문제 해결 시


Purpose

이 문서가 존재하는 이유

  • Goal: How to compile HPL-GPU에 대한 기술 정보 제공
  • Scope: How to compile HPL-GPU의 개념, 사용법, 설정
  • Non-goals: 다른 주제로의 확장


Key Concepts

Concept Description Related
How to compile HPL-GPU HPC/서버 환경에서 중요한 기술 개념 Linux, Server


Detailed Explanation

There are many combination to compile High Performance LINPACK (HPL) with different configurations such as different compiler, different basic linear algebra subprograms (BLAS), massage passing interface (MPI) libraries: for example

  • Which compiler + HPL + which Blas (OpenBLAS / Intel MKL / CuBLAS) + which MPI (OpenMPI, MPICH, Intel MPI)

In this post, we are going to use GNU compiler, OpenBLAS, OpenMPI for HPL-GPU

  • The build system is Intel Skylake CPU, 128GB Memory, Nvidia GPUs on Ubuntu 20.04
  • NVIDIA driver is ready and CUDA 12.0 has been installed and library can be found at /usr/local/cuda
  • We are using OpenMPI, OpenBlas and hpl-2.0_FERMI_v15 as a ingredient, gcc version 10, g++ version 10 and gfortran version 10
_version=v0.3.23
FILE=$_version.tar.gz 
wget http://github.com/xianyi/OpenBLAS/archive/$FILE
tar -xzvf $FILE
cd OpenBLAS-0.3.23
# OpenBLAS does not support f77, GCC or Intel are supported
# type make to detect the CPU automatically. or type make TARGET=xxx to set target CPU, e.g. make TARGET=NEHALEM. The full target list is in file TargetList.txt.
make 
# To install the library, you can run "make PREFIX=/path/to/your/installation install".
mkdir -p /opt/hpcmate/home/lib/OpenBLAS-0.3.23
make PREFIX=/opt/hpcmate/home/lib/OpenBLAS-0.3.23 install 
ls ~/lib/OpenBLAS-0.3.23/lib

now, compiled OpenBLAS's library and header file is available ~/lib/OpenBLAS-0.3.23


Best Practices

  • 최신 버전 사용 권장
  • 공식 문서 참고
  • 테스트 환경에서 먼저 검증


References


Related Pages

Knowledge Graph

Related

→ CUDA → GPU → NVIDIA GPU → NVLink → AMD GPUs → MIG → NVIDIA GPUs → Tensor Core

Retrieved from "http://wiki.hpcmate.com/index.php?title=How_to_compile_HPL-GPU&oldid=3334"