Nvdia-smi tips and tricks: Difference between revisions

From HPCWIKI
Jump to navigation Jump to search
(Fix: remove --- horizontal lines (7 removed))
 
(3 intermediate revisions by the same user not shown)
Line 1: Line 1:
== Ouput example ==
{{Status
|status=Draft
|owner=Knowledge Agent
|last_update=2026-07-16
|review=Pending
}}
 
{{TOC}}
 
== Overview ==
 
Nvdia-smi tips and tricks에 대한 기술 문서입니다.
 
=== Summary ===
 
* 무엇인가? - Nvdia-smi tips and tricks
* 왜 필요한가? - HPC 및 서버 환경에서 필수 개념
* 언제 사용하는가? - 서버 구성, 성능 튜닝, 문제 해결 시
 
 
== Purpose ==
 
이 문서가 존재하는 이유
 
* Goal: Nvdia-smi tips and tricks에 대한 기술 정보 제공
* Scope: Nvdia-smi tips and tricks의 개념, 사용법, 설정
* Non-goals: 다른 주제로의 확장
 
 
== Key Concepts ==
 
{| class="wikitable"
! Concept
! Description
! Related
|-
| Nvdia-smi tips and tricks
| HPC/서버 환경에서 중요한 기술 개념
| [[Linux]], [[Server]]
|}
 
 
== Detailed Explanation ==
 
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
+---------------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------------+
Line 20: Line 63:
+---------------------------------------------------------------------------------------+
+---------------------------------------------------------------------------------------+
</syntaxhighlight>
</syntaxhighlight>
{| class="wikitable"
|+
!Property name
!Anotation
!Meaning
|-
|Performance State
|Perf
|States range from P0 (maxi-mum performance) to P12 (minimum performance).
|}


== Disable or enable GPU ==
<syntaxhighlight lang="bash">
##Disable GPU, where xx is PCIe bus number from lspci 
$sudo nvidia-smi -i 0000:xx:00.0 -pm 0
$sudo nvidia-smi drain -p 0000:xx:00.0 -m 1
## The device will still be visible with lspci after running the commands above.


#Enable GPU
== Best Practices ==
$sudo nvidia-smi drain -p 0000:xx:00.0 -m 0


* 최신 버전 사용 권장
* 공식 문서 참고
* 테스트 환경에서 먼저 검증


</syntaxhighlight>


== --gpus options ==
== References ==
without nvidia-container-toolkit, [[docker]] with --gpus options makes following error
docker: Error response from daemon: could not select device driver "" with capabilities: --gpus options
nvidia-container-toolkit package will solve the issue
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit


== Turn on / off ECC<ref>https://thelinuxcluster.com/2013/07/24/turning-off-and-on-ecc-ram-for-nvidia-gp-gpu-cards/</ref> ==
* [https://wiki.hpcmate.com Nvdia-smi tips and tricks]
To Turn off the ECC RAM
# nvidia-smi -g 0 --ecc-config=0
(repeat with -g x for each GPU ID)
To Turn back on ECC RAM
# nvidia-smi -g 0 --ecc-config=1
(repeat with -g x for each GPU ID)
To Reset ECC error<ref>https://developer.download.nvidia.com/compute/DCGM/docs/nvidia-smi-367.38.pdf</ref>
# nvidia-smi -g 0 --reset-ecc-errors=TYPE (0|VOLATILE or 1|AGGREGATE)


== Reset GPU ==
# nvidia-smi -g 0 --gpu-reset


== GPU mode ==
== Related Pages ==
The mode of the GPU is established directly at power-on, from settings stored in the GPU’s non-volatile memory.


gpumodeswitch changes the mode of the GPU by updating the GPU’s non-volatile memory settings.
* [[Linux]]
* [[Server]]
* [[Hardware]]
* [[Network]]


Compute mode is a configuration that is optimized for high-performance computing (HPC) applications, Compute mode can cause compatibility problems with OS and hypervisors when the GPU is used primarily as a graphics device.


Graphic mode
[[Category:Server]]
== Knowledge Graph ==


== Compute mode ==
Related
<code>#nvidia-smi -g0 -c <mode number></code>
{| class="wikitable"
|+
!number
!Mode
!Meaning
|-
|<code>0</code>
|<code>Default</code>
|Default mode GPU can be shared with several jobs,
|-
|<code>1</code>
|<code>Exclusive_Thread</code>
|Exclusive thread mode only is allowed to run one job, but in the same time, only one thread runs on exclusive thread mode GPU.
|-
|<code>2</code>
|<code>Prohibited</code>
|prohibited mode GPU is not allowed to run job,
|-
|<code>3</code>
|<code>Exclusive_Process</code>
|Exclusive process mode  is allowed to run one job, but in the same time, only one process runs on exclusive process mode GPU.
|}
 
== GPU parameter change ==
Enable or disable GPU's auto boost mode
 
$ sudo nvidia-smi --auto-boost-default=DISABLED
 
 
Set GPU clock speed


$ sudo nvidia-smi --applications-clocks=2505,875
→ [[Linux]]
→ [[Server]]
→ [[Hardware]]
→ [[Network]]


== References ==
[[Category:Reference]]
<references />

Latest revision as of 11:30, 17 July 2026

Template:Status

Template:TOC

Overview

Nvdia-smi tips and tricks에 대한 기술 문서입니다.

Summary

  • 무엇인가? - Nvdia-smi tips and tricks
  • 왜 필요한가? - HPC 및 서버 환경에서 필수 개념
  • 언제 사용하는가? - 서버 구성, 성능 튜닝, 문제 해결 시


Purpose

이 문서가 존재하는 이유

  • Goal: Nvdia-smi tips and tricks에 대한 기술 정보 제공
  • Scope: Nvdia-smi tips and tricks의 개념, 사용법, 설정
  • Non-goals: 다른 주제로의 확장


Key Concepts

Concept Description Related
Nvdia-smi tips and tricks HPC/서버 환경에서 중요한 기술 개념 Linux, Server


Detailed Explanation

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.86.05              Driver Version: 535.86.05    CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4090        Off | 00000000:D8:00.0 Off |                  Off |
| 30%   42C    P8              38W / 450W |      2MiB / 24564MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+


Best Practices

  • 최신 버전 사용 권장
  • 공식 문서 참고
  • 테스트 환경에서 먼저 검증


References


Related Pages

Knowledge Graph

Related

→ Linux → Server → Hardware → Network

Retrieved from "http://wiki.hpcmate.com/index.php?title=Nvdia-smi_tips_and_tricks&oldid=3409"