Nvdia-smi tips and tricks: Difference between revisions
Jump to navigation
Jump to search
| Line 30: | Line 30: | ||
|States range from P0 (maxi-mum performance) to P12 (minimum performance). | |States range from P0 (maxi-mum performance) to P12 (minimum performance). | ||
|} | |} | ||
== --gpus options == | |||
To use --gpus options with [[docker]] on Ubuntu | |||
without nvidia-container-toolkit, docker with --gpus options makes following error | |||
docker: Error response from daemon: could not select device driver "" with capabilities: --gpus options | |||
nvidia-container-toolkit package will solve the issue | |||
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit | |||
== Turn on / off ECC<ref>https://thelinuxcluster.com/2013/07/24/turning-off-and-on-ecc-ram-for-nvidia-gp-gpu-cards/</ref> == | == Turn on / off ECC<ref>https://thelinuxcluster.com/2013/07/24/turning-off-and-on-ecc-ram-for-nvidia-gp-gpu-cards/</ref> == | ||
Revision as of 12:37, 9 May 2024
Ouput example
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.86.05 Driver Version: 535.86.05 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 4090 Off | 00000000:D8:00.0 Off | Off |
| 30% 42C P8 38W / 450W | 2MiB / 24564MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+
| Property name | Anotation | Meaning |
|---|---|---|
| Performance State | Perf | States range from P0 (maxi-mum performance) to P12 (minimum performance). |
--gpus options
To use --gpus options with docker on Ubuntu
without nvidia-container-toolkit, docker with --gpus options makes following error
docker: Error response from daemon: could not select device driver "" with capabilities: --gpus options
nvidia-container-toolkit package will solve the issue
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
Turn on / off ECC[1]
To Turn off the ECC RAM
# nvidia-smi -g 0 --ecc-config=0 (repeat with -g x for each GPU ID)
To Turn back on ECC RAM
# nvidia-smi -g 0 --ecc-config=1 (repeat with -g x for each GPU ID)
To Reset ECC error[2]
# nvidia-smi -g 0 --reset-ecc-errors=TYPE (0|VOLATILE or 1|AGGREGATE)
Reset GPU
# nvidia-smi -g 0 --gpu-reset
GPU mode
The mode of the GPU is established directly at power-on, from settings stored in the GPU’s non-volatile memory.
gpumodeswitch changes the mode of the GPU by updating the GPU’s non-volatile memory settings.
Compute mode is a configuration that is optimized for high-performance computing (HPC) applications, Compute mode can cause compatibility problems with OS and hypervisors when the GPU is used primarily as a graphics device.
Graphic mode
Compute mode
#nvidia-smi -g0 -c <mode number>
| number | Mode | Meaning |
|---|---|---|
0
|
Default
|
Default mode GPU can be shared with several jobs, |
1
|
Exclusive_Thread
|
Exclusive thread mode only is allowed to run one job, but in the same time, only one thread runs on exclusive thread mode GPU. |
2
|
Prohibited
|
prohibited mode GPU is not allowed to run job, |
3
|
Exclusive_Process
|
Exclusive process mode is allowed to run one job, but in the same time, only one process runs on exclusive process mode GPU. |