Nvidia GPU Tips and Tricks: Difference between revisions

From HPCWIKI
Jump to navigation Jump to search
(Fix unclosed wikitable: close {|/|} + remove orphan Status; stops sections being swallowed into table)
 
(One intermediate revision by one other user not shown)
Line 29: Line 29:
|}
|}


== Detailed Explanation ==
== Xid 119 ==
Based on many cases, XID 119 error caused by many reasons. one of that is the incompatibility between Nvidia GSP firmware and [[PCIe ASPM (Active State Power Management)]], especially if you can see GPU device on PCI list ($lspci | grep -i vga) while nvidia-msi does not show the GPU.
 
From [[ASPM]] perspective,  host driver is trying to wakeup GPU in deep-P-state, GPU's GSP response has been timeout then host disconnected [[PCIe]] bus slot to disconnect the GPU (Fallen off the bus / Xid 79). simply idle deep-P-state GSP disconnection. 
 
<syntaxhighlight lang="bash">
#Xid 119 error example
$ sudo cat /var/log/syslog | grep -i NVRM
Oct 12 00:32:17 Venus kernel: [ 1074.666075] NVRM: GPU at PCI:0000:81:00: GPU-760dab3f-a851-fc4c-dba8-c400d7db4b17
Oct 12 00:32:17 Venus kernel: [ 1074.666086] NVRM: GPU Board Serial Number: 1714923003394
Oct 12 00:32:17 Venus kernel: [ 1074.666089] NVRM: Xid (PCI:0000:81:00): 119, pid=2816, name=nvidia-smi, Timeout after 6s of waiting for RPC response from GPU1 GSP! Expected function 103 (GSP_RM_ALLOC) (0x0 0x78).
Oct 12 00:32:17 Venus kernel: [ 1074.666114] NVRM: GPU1 GSP RPC buffer contains function 103 (GSP_RM_ALLOC) and data 0x0000000000000000 0x0000000000000078.
Oct 12 00:32:17 Venus kernel: [ 1074.666122] NVRM: GPU1 RPC history (CPU -> GSP):
Oct 12 00:32:17 Venus kernel: [ 1074.666126] NVRM:    entry function                  data0              data1              ts_start          ts_end            duration actively_polling
Oct 12 00:32:17 Venus kernel: [ 1074.666130] NVRM:      0    103  GSP_RM_ALLOC          0x0000000000000000 0x0000000000000078 0x00065d99d4274f2e 0x0000000000000000          y
Oct 12 00:32:17 Venus kernel: [ 1074.666141] NVRM:    -1    76  GSP_RM_CONTROL        0x0000000020800aff 0x0000000000000010 0x00065d999699a9c2 0x00065d999699ad04    834us 
Oct 12 00:32:17 Venus kernel: [ 1074.666152] NVRM:    -2    103  GSP_RM_ALLOC          0x0000000000002080 0x0000000000000004 0x00065d9996999382 0x00065d999699946b    233us 
Oct 12 00:32:17 Venus kernel: [ 1074.666161] NVRM:    -3    103  GSP_RM_ALLOC          0x0000000000000080 0x0000000000000038 0x00065d99969991fa 0x00065d999699929e    164us 
Oct 12 00:32:17 Venus kernel: [ 1074.666169] NVRM:    -4    103  GSP_RM_ALLOC          0x0000000000000000 0x0000000000000078 0x00065d999699910f 0x00065d99969991f6    231us 
Oct 12 00:32:17 Venus kernel: [ 1074.666177] NVRM:    -5    76  GSP_RM_CONTROL        0x000000000073028b 0x0000000000000038 0x00065d9996951efc 0x00065d9996951f73    119us 
Oct 12 00:32:17 Venus kernel: [ 1074.666185] NVRM:    -6    76  GSP_RM_CONTROL        0x0000000000730211 0x0000000000000010 0x00065d9996951e4d 0x00065d9996951ef3    166us 
Oct 12 00:32:17 Venus kernel: [ 1074.666194] NVRM:    -7    76  GSP_RM_CONTROL        0x000000000073028b 0x0000000000000038 0x00065d9996951d96 0x00065d9996951e41    171us 
Oct 12 00:32:17 Venus kernel: [ 1074.666202] NVRM: GPU1 RPC event history (CPU <- GSP):
Oct 12 00:32:17 Venus kernel: [ 1074.666206] NVRM:    entry function                  data0              data1              ts_start          ts_end            duration during_incomplete_rpc
Oct 12 00:32:17 Venus kernel: [ 1074.666211] NVRM:      0    4108 UCODE_LIBOS_PRINT    0x0000000000000000 0x0000000000000000 0x00065d99968fe0bd 0x00065d99968fe0bd         
Oct 12 00:32:17 Venus kernel: [ 1074.666220] NVRM:    -1    4108 UCODE_LIBOS_PRINT    0x0000000000000000 0x0000000000000000 0x00065d99968fdfc9 0x00065d99968fdfcb      2us 
Oct 12 00:32:17 Venus kernel: [ 1074.666229] NVRM:    -2    4098 GSP_RUN_CPU_SEQUENCER 0x000000000000061c 0x0000000000003fe2 0x00065d99968ec520 0x00065d99968ed9b7  5271us 
Oct 12 00:32:23 Venus kernel: [ 1080.710822] NVRM: Xid (PCI:0000:81:00): 119, pid=2816, name=nvidia-smi, Timeout after 6s of waiting for RPC response from GPU1 GSP! Expected function 103 (GSP_RM_ALLOC) (0x80 0x38).
Oct 12 00:32:29 Venus kernel: [ 1086.715510] NVRM: Xid (PCI:0000:81:00): 119, pid=2816, name=nvidia-smi, Timeout after 6s of waiting for RPC response from GPU1 GSP! Expected function 103 (GSP_RM_ALLOC) (0x2080 0x4).
Oct 12 00:32:35 Venus kernel: [ 1092.732051] NVRM: Rate limiting GSP RPC error prints for GPU at PCI:0000:81:00 (printing 1 of every 30).  The GPU likely needs to be reset.
Oct 12 00:39:36 Venus kernel: [ 1514.082939] NVRM: Xid (PCI:0000:81:00): 119, pid=3611, name=nvidia-smi, Timeout after 6s of waiting for RPC response from GPU1 GSP! Expected function 103 (GSP_RM_ALLOC) (0x80 0x38).
Oct 12 00:48:55 Venus kernel: [ 2073.280826] NVRM: Xid (PCI:0000:81:00): 119, pid=4684, name=nvidia-smi, Timeout after 6s of waiting for RPC response from GPU1 GSP! Expected function 76 (GSP_RM_CONTROL) (0x20801819 0x20).
Oct 12 01:09:34 Venus kernel: [  10.489443] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  570.169  Thu Jun 12 20:04:34 UTC 2025
Oct 12 01:09:44 Venus kernel: [  24.019501] NVRM: GPU 0000:81:00.0: RmInitAdapter failed! (0x25:0xffff:1545)
Oct 12 01:09:44 Venus kernel: [  24.019545] NVRM: GPU 0000:81:00.0: rm_init_adapter failed, device minor number 0
Oct 12 01:09:58 Venus kernel: [  37.212612] NVRM: GPU 0000:81:00.0: RmInitAdapter failed! (0x25:0xffff:1545)
Oct 12 01:09:58 Venus kernel: [  37.212667] NVRM: GPU 0000:81:00.0: rm_init_adapter failed, device minor number 0
Oct 12 01:10:05 Venus kernel: [  45.089470] NVRM: GPU 0000:81:00.0: RmInitAdapter failed! (0x25:0xffff:1545)
Oct 12 01:10:05 Venus kernel: [  45.089530] NVRM: GPU 0000:81:00.0: rm_init_adapter failed, device minor number 0
Oct 12 01:29:15 Venus kernel: [    9.350719] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  570.169  Thu Jun 12 20:04:34 UTC 2025
Oct 12 01:40:32 Venus kernel: [    8.597394] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  570.169  Thu Jun 12 20:04:34 UTC 2025
Oct 12 01:40:41 Venus kernel: [  20.848347] NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x25:0xffff:1545)
Oct 12 01:40:41 Venus kernel: [  20.848419] NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0
Oct 12 01:41:36 Venus kernel: [  75.328697] NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x25:0xffff:1545)
Oct 12 01:41:36 Venus kernel: [  75.328742] NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0


* [[NVIDIA GPUs]]
Based on many cases, XID 62 and 119 error caused by incompatible driver version issues. it's recommended to install right version of GPU driver based on customer [[CUDA]] application version.<syntaxhighlight lang="bash">
Remove Nvidia GPU driver on Ubuntu
$sudo apt-get remove --purge '^nvidia-.*'
apt search or get download GPU driver from nvidia then install and test
</syntaxhighlight>
</syntaxhighlight>
{| class="wikitable"
! colspan="1" rowspan="1" |XID
! colspan="1" rowspan="1" |[[NVIDIA GPU|Nvidia GPU]] Failure
![[Linux]] [[Kernel]] message
! colspan="7" rowspan="1" |Causes
|-
! colspan="1" rowspan="1" |
! colspan="1" rowspan="1" |
!
! colspan="1" rowspan="1" |HW Error
! colspan="1" rowspan="1" |Driver Error
! colspan="1" rowspan="1" |User App Error
! colspan="1" rowspan="1" |System Memory Corruption
! colspan="1" rowspan="1" |Bus Error


|}
The solution would be deactivate GSP - ASPM signaling using one of following way
 
1. Set GPU persistence mode (Most simple and recommended) along with custom systemd service to PM on after system bootup.
 
<nowiki>#</nowiki>sudo nvidia-smi -pm 1
 
 
2. Disable PCIe ASPM with [[kernel]] param
 
Add pcie-aspm=off in GRUB_CMDLINE_LINUX_DEFAULT then update-grub and reboot
 
 
3. Disable GSP feature (depending on version of driver, proprieatry vs open-kernel driver or GPU generation)
 
Create /etc/modprobe.d/gsp.conf 파일에 options nvidia NVreg_EnableGpuFirmware=0
 
== Best Practices ==
== Best Practices ==


Line 69: Line 109:
* [[Hardware]]
* [[Hardware]]
* [[Network]]
* [[Network]]
* [[NVIDIA GPUs]]


[[Category:GPU]]
[[Category:GPU]]

Latest revision as of 14:50, 14 October 2026

Overview

Nvidia GPU Tips and Tricks에 대한 기술 문서입니다.

Summary

  • 무엇인가? - Nvidia GPU Tips and Tricks
  • 왜 필요한가? - HPC 및 서버 환경에서 필수 개념
  • 언제 사용하는가? - 서버 구성, 성능 튜닝, 문제 해결 시

Purpose

이 문서가 존재하는 이유

  • Goal: Nvidia GPU Tips and Tricks에 대한 기술 정보 제공
  • Scope: Nvidia GPU Tips and Tricks의 개념, 사용법, 설정
  • Non-goals: 다른 주제로의 확장

Key Concepts

Concept Description Related
Nvidia GPU Tips and Tricks HPC/서버 환경에서 중요한 기술 개념 Linux, Server

Xid 119

Based on many cases, XID 119 error caused by many reasons. one of that is the incompatibility between Nvidia GSP firmware and PCIe ASPM (Active State Power Management), especially if you can see GPU device on PCI list ($lspci | grep -i vga) while nvidia-msi does not show the GPU.

From ASPM perspective, host driver is trying to wakeup GPU in deep-P-state, GPU's GSP response has been timeout then host disconnected PCIe bus slot to disconnect the GPU (Fallen off the bus / Xid 79). simply idle deep-P-state GSP disconnection. 

#Xid 119 error example
$ sudo cat /var/log/syslog | grep -i NVRM
Oct 12 00:32:17 Venus kernel: [ 1074.666075] NVRM: GPU at PCI:0000:81:00: GPU-760dab3f-a851-fc4c-dba8-c400d7db4b17
Oct 12 00:32:17 Venus kernel: [ 1074.666086] NVRM: GPU Board Serial Number: 1714923003394
Oct 12 00:32:17 Venus kernel: [ 1074.666089] NVRM: Xid (PCI:0000:81:00): 119, pid=2816, name=nvidia-smi, Timeout after 6s of waiting for RPC response from GPU1 GSP! Expected function 103 (GSP_RM_ALLOC) (0x0 0x78).
Oct 12 00:32:17 Venus kernel: [ 1074.666114] NVRM: GPU1 GSP RPC buffer contains function 103 (GSP_RM_ALLOC) and data 0x0000000000000000 0x0000000000000078.
Oct 12 00:32:17 Venus kernel: [ 1074.666122] NVRM: GPU1 RPC history (CPU -> GSP):
Oct 12 00:32:17 Venus kernel: [ 1074.666126] NVRM:     entry function                   data0              data1              ts_start           ts_end             duration actively_polling
Oct 12 00:32:17 Venus kernel: [ 1074.666130] NVRM:      0    103  GSP_RM_ALLOC          0x0000000000000000 0x0000000000000078 0x00065d99d4274f2e 0x0000000000000000          y
Oct 12 00:32:17 Venus kernel: [ 1074.666141] NVRM:     -1    76   GSP_RM_CONTROL        0x0000000020800aff 0x0000000000000010 0x00065d999699a9c2 0x00065d999699ad04    834us  
Oct 12 00:32:17 Venus kernel: [ 1074.666152] NVRM:     -2    103  GSP_RM_ALLOC          0x0000000000002080 0x0000000000000004 0x00065d9996999382 0x00065d999699946b    233us  
Oct 12 00:32:17 Venus kernel: [ 1074.666161] NVRM:     -3    103  GSP_RM_ALLOC          0x0000000000000080 0x0000000000000038 0x00065d99969991fa 0x00065d999699929e    164us  
Oct 12 00:32:17 Venus kernel: [ 1074.666169] NVRM:     -4    103  GSP_RM_ALLOC          0x0000000000000000 0x0000000000000078 0x00065d999699910f 0x00065d99969991f6    231us  
Oct 12 00:32:17 Venus kernel: [ 1074.666177] NVRM:     -5    76   GSP_RM_CONTROL        0x000000000073028b 0x0000000000000038 0x00065d9996951efc 0x00065d9996951f73    119us  
Oct 12 00:32:17 Venus kernel: [ 1074.666185] NVRM:     -6    76   GSP_RM_CONTROL        0x0000000000730211 0x0000000000000010 0x00065d9996951e4d 0x00065d9996951ef3    166us  
Oct 12 00:32:17 Venus kernel: [ 1074.666194] NVRM:     -7    76   GSP_RM_CONTROL        0x000000000073028b 0x0000000000000038 0x00065d9996951d96 0x00065d9996951e41    171us  
Oct 12 00:32:17 Venus kernel: [ 1074.666202] NVRM: GPU1 RPC event history (CPU <- GSP):
Oct 12 00:32:17 Venus kernel: [ 1074.666206] NVRM:     entry function                   data0              data1              ts_start           ts_end             duration during_incomplete_rpc
Oct 12 00:32:17 Venus kernel: [ 1074.666211] NVRM:      0    4108 UCODE_LIBOS_PRINT     0x0000000000000000 0x0000000000000000 0x00065d99968fe0bd 0x00065d99968fe0bd           
Oct 12 00:32:17 Venus kernel: [ 1074.666220] NVRM:     -1    4108 UCODE_LIBOS_PRINT     0x0000000000000000 0x0000000000000000 0x00065d99968fdfc9 0x00065d99968fdfcb      2us  
Oct 12 00:32:17 Venus kernel: [ 1074.666229] NVRM:     -2    4098 GSP_RUN_CPU_SEQUENCER 0x000000000000061c 0x0000000000003fe2 0x00065d99968ec520 0x00065d99968ed9b7   5271us  
Oct 12 00:32:23 Venus kernel: [ 1080.710822] NVRM: Xid (PCI:0000:81:00): 119, pid=2816, name=nvidia-smi, Timeout after 6s of waiting for RPC response from GPU1 GSP! Expected function 103 (GSP_RM_ALLOC) (0x80 0x38).
Oct 12 00:32:29 Venus kernel: [ 1086.715510] NVRM: Xid (PCI:0000:81:00): 119, pid=2816, name=nvidia-smi, Timeout after 6s of waiting for RPC response from GPU1 GSP! Expected function 103 (GSP_RM_ALLOC) (0x2080 0x4).
Oct 12 00:32:35 Venus kernel: [ 1092.732051] NVRM: Rate limiting GSP RPC error prints for GPU at PCI:0000:81:00 (printing 1 of every 30).  The GPU likely needs to be reset.
Oct 12 00:39:36 Venus kernel: [ 1514.082939] NVRM: Xid (PCI:0000:81:00): 119, pid=3611, name=nvidia-smi, Timeout after 6s of waiting for RPC response from GPU1 GSP! Expected function 103 (GSP_RM_ALLOC) (0x80 0x38).
Oct 12 00:48:55 Venus kernel: [ 2073.280826] NVRM: Xid (PCI:0000:81:00): 119, pid=4684, name=nvidia-smi, Timeout after 6s of waiting for RPC response from GPU1 GSP! Expected function 76 (GSP_RM_CONTROL) (0x20801819 0x20).
Oct 12 01:09:34 Venus kernel: [   10.489443] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  570.169  Thu Jun 12 20:04:34 UTC 2025
Oct 12 01:09:44 Venus kernel: [   24.019501] NVRM: GPU 0000:81:00.0: RmInitAdapter failed! (0x25:0xffff:1545)
Oct 12 01:09:44 Venus kernel: [   24.019545] NVRM: GPU 0000:81:00.0: rm_init_adapter failed, device minor number 0
Oct 12 01:09:58 Venus kernel: [   37.212612] NVRM: GPU 0000:81:00.0: RmInitAdapter failed! (0x25:0xffff:1545)
Oct 12 01:09:58 Venus kernel: [   37.212667] NVRM: GPU 0000:81:00.0: rm_init_adapter failed, device minor number 0
Oct 12 01:10:05 Venus kernel: [   45.089470] NVRM: GPU 0000:81:00.0: RmInitAdapter failed! (0x25:0xffff:1545)
Oct 12 01:10:05 Venus kernel: [   45.089530] NVRM: GPU 0000:81:00.0: rm_init_adapter failed, device minor number 0
Oct 12 01:29:15 Venus kernel: [    9.350719] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  570.169  Thu Jun 12 20:04:34 UTC 2025
Oct 12 01:40:32 Venus kernel: [    8.597394] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  570.169  Thu Jun 12 20:04:34 UTC 2025
Oct 12 01:40:41 Venus kernel: [   20.848347] NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x25:0xffff:1545)
Oct 12 01:40:41 Venus kernel: [   20.848419] NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0
Oct 12 01:41:36 Venus kernel: [   75.328697] NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x25:0xffff:1545)
Oct 12 01:41:36 Venus kernel: [   75.328742] NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0

The solution would be deactivate GSP - ASPM signaling using one of following way

1. Set GPU persistence mode (Most simple and recommended) along with custom systemd service to PM on after system bootup.

#sudo nvidia-smi -pm 1


2. Disable PCIe ASPM with kernel param

Add pcie-aspm=off in GRUB_CMDLINE_LINUX_DEFAULT then update-grub and reboot


3. Disable GSP feature (depending on version of driver, proprieatry vs open-kernel driver or GPU generation)

Create /etc/modprobe.d/gsp.conf 파일에 options nvidia NVreg_EnableGpuFirmware=0

Best Practices

  • 최신 버전 사용 권장
  • 공식 문서 참고
  • 테스트 환경에서 먼저 검증

References

Related Pages

Knowledge Graph

Related

→ CUDA → GPU → NVIDIA GPU → H100 → NVLink → AMD GPUs → A100 → MIG

Retrieved from "http://wiki.hpcmate.com/index.php?title=Nvidia_GPU_Tips_and_Tricks&oldid=4113"