RDMA: Difference between revisions

From HPCWIKI
Jump to navigation Jump to search
(Add categories: Linux, Reference)
(Phase 0.4: Expand RDMA)
 
(3 intermediate revisions by the same user not shown)
Line 1: Line 1:
RDMA (Remote Direct Memory Access), which means remote direct memory access, is a network communication protocol that was first applied in the field of high-performance computing, and has gradually become popular in data centers.
{{Status
|status=Draft
|owner=Knowledge Agent
|last_update=2026-07-16
|review=Pending
}}
 
{{TOC}}
 
== Overview ==
 
RDMA에 대한 기술 문서입니다.
 
=== Summary ===
 
* 무엇인가? - RDMA
* 왜 필요한가? - HPC 및 서버 환경에서 필수 개념
* 언제 사용하는가? - 서버 구성, 성능 튜닝, 문제 해결 시
 
 
== Purpose ==
 
이 문서가 존재하는 이유
 
* Goal: RDMA에 대한 기술 정보 제공
* Scope: RDMA의 개념, 사용법, 설정
* Non-goals: 다른 주제로의 확장




RDMA allows user programs to bypass the operating system kernel (CPU) and directly interact with the network card for network communication, thereby providing high bandwidth and extremely small latency.<ref>https://www.naddod.com/blog/rdma-high-speed-network-for-large-model-training</ref>
== Key Concepts ==
 
{| class="wikitable"
! Concept
! Description
! Related
|-
| RDMA
| HPC/서버 환경에서 중요한 기술 개념
| [[Linux]], [[Server]]
|}


[[File:RDMA FLOW.png|center|frameless|557x557px]]


== Types of RDMA ==
== Detailed Explanation ==


# Infiniband is a network protocol tailored specifically for RDMA, which can ensure the reliability of data transmission from the hardware level. Although InfiniBand technical specifications and standard specifications were officially published in 2000, InfiniBand Architecture (IBA) has been widely used on cluster supercomputers after 2005. The biggest reason for the slow development is that Infiniband requires its own dedicated hardware from L2 to L4. The cost of the enterprise is very high.  
RDMA (Remote Direct Memory Access), which means remote direct memory access, is a network communication protocol that was first applied in the field of high-performance computing, and has gradually become popular in data centers.
RDMA allows user programs to bypass the operating system [[kernel]] (CPU) and directly interact with the network card for network communication, thereby providing high bandwidth and extremely small latency.<ref>https://www.naddod.com/blog/rdma-high-speed-network-for-large-model-training</ref>
[[File:RDMA FLOW.png|center|frameless|557x557px]]
# Infiniband is a network protocol tailored specifically for RDMA, which can ensure the reliability of data transmission from the hardware level. Although [[InfiniBand]] technical specifications and standard specifications were officially published in 2000, InfiniBand Architecture (IBA) has been widely used on cluster supercomputers after 2005. The biggest reason for the slow development is that Infiniband requires its own dedicated hardware from L2 to L4. The cost of the enterprise is very high.  
# RoCE (RDMA over Converged Ethernet) is a protocol for implemtenting RDMA in an Ethernet network for a cheaper RDMA option. RoCE primarily provides RDMA capabilities over Ethernet.  
# RoCE (RDMA over Converged Ethernet) is a protocol for implemtenting RDMA in an Ethernet network for a cheaper RDMA option. RoCE primarily provides RDMA capabilities over Ethernet.  
## RoCEv1, which runs at the link layer and cannot be run over a routed network. Therefore, it requires the priority flow control (PFC) to be enabled. The RoCE v1 protocol is an Ethernet link layer protocol allowing two hosts in the same Ethernet broadcast domain (VLAN) to communicate. It uses Ethertype 0x8915, which limits the frame length to 1500 bytes for a standard Ethernet frame and 9000 bytes for an Ethernet jumbo frame.
## RoCEv1, which runs at the link layer and cannot be run over a routed network. Therefore, it requires the priority flow control (PFC) to be enabled. The RoCE v1 protocol is an Ethernet link layer protocol allowing two hosts in the same Ethernet broadcast domain (VLAN) to communicate. It uses Ethertype 0x8915, which limits the frame length to 1500 bytes for a standard Ethernet frame and 9000 bytes for an Ethernet jumbo frame.
## RoCEv2, which runs over layer 3. Because it is a routed solution, consider using explicit congestion notification (ECN) with RoCEv2 as ECN bits are communicated end-to-end across a routed network. By changing the packet encapsulation to include IP and UDP headers, RoCE v2 can now be used across both L2 and L3 networks. This enables Layer 3 routing, which brings RDMA to network with multiple subnets for great scalability. Therefore, RoCE v2 is also regarded as Routable RoCE (RRoCE). Owing to the arrival of RoCE v2, the IP multicast is now also possible.
## RoCEv2, which runs over layer 3. Because it is a routed solution, consider using explicit congestion notification (ECN) with RoCEv2 as ECN bits are communicated end-to-end across a routed network. By changing the packet encapsulation to include IP and UDP headers, RoCE v2 can now be used across both L2 and L3 networks. This enables Layer 3 routing, which brings RDMA to network with multiple subnets for great scalability. Therefore, RoCE v2 is also regarded as Routable RoCE (RRoCE). Owing to the arrival of RoCE v2, the IP multicast is now also possible.
# iWARP iWARP is a protocol for implementing RDMA across Internet Protocol (IP) networks. Its advantage is that it can run in today's standard TCP/IP network. RDMA can be used only by purchasing a network card that supports iWARP. For those with a slightly lower financial budget It is especially suitable for enterprises. But its disadvantage is that it is slightly worse than RoCE in performance. After all, you get what you pay for.
# iWARP iWARP is a protocol for implementing RDMA across Internet Protocol (IP) networks. Its advantage is that it can run in today's standard TCP/IP network. RDMA can be used only by purchasing a network card that supports iWARP. For those with a slightly lower financial budget It is especially suitable for enterprises. But its disadvantage is that it is slightly worse than RoCE in performance. After all, you get what you pay for.
== The Difference Between RDMA and TCP/IP<ref>https://community.fs.com/article/roce-vs-infiniband-vs-tcp-ip.html</ref><ref>https://community.fs.com/article/roce-rdma-over-converged-ethernet.html</ref> ==
{| class="wikitable"
{| class="wikitable"
!Contents
!Contents
Line 28: Line 63:
|Wore than InfiniBand and RoCE
|Wore than InfiniBand and RoCE
|Poor
|Poor
== Best Practices ==
* 최신 버전 사용 권장
* 공식 문서 참고
* 테스트 환경에서 먼저 검증
== References ==
* [https://wiki.hpcmate.com RDMA]
== Related Pages ==
* [[Linux]]
* [[Server]]
* [[Hardware]]
* [[Network]]
[[Category:Network]]
== Knowledge Graph ==
Related
→ [[Linux]]
→ [[Server]]
→ [[Hardware]]
→ [[Network]]
[[Category:Reference]]
== RoCE vs InfiniBand 비교 ==
{| class="wikitable"
! Feature ! RoCE v2 ! InfiniBand
|-
| 네트워크 | Ethernet | 전용 IB 네트워크
|-
| 스위치 | 표준 Ethernet 스위치 | IB 스위치
|-
| 레이턴시 | 1-2 μs | 0.5-1 μs
|-
|-
|High performance
| 대역폭 | 100/200/400 GbE | 200/400 Gb/s
|Equivalent to InfiniBand
|Excellent
|Slightly worse than InfiniBand (affected by TCP)
|Poor
|-
|-
|High stability
| 비용 | 낮음 (기존 인프라 활용) | 높음
|Good
|Excellent
|Poor
|Excellent
|-
|-
|Easy management
| 설정 복잡도 | 낮음 | 높음
|Hard
|Hard
|Harder than RoCE
|Easy
|-
|-
|Cost-efficiency
| ECN 지원 | 있음 | 있음
|Low
|High
|Medium
|Low
|-
|-
|Network device
| TSO/LRO | 지원 | 지원
|Network device
|IB switch
|Network device
|Network device
|}
|}
[[File:InfiniBand Vs. RoCEv2.png|center|frameless|808x808px|InfiniBand Vs. RoCEv2]]


== RDMA service ==
=== RoCE 설정 ===
RDMA service in [[Linux]] is a technology that allows applications to communicate directly with each other over a network without involving the operating system. It provides high-speed, low-latency data transfer between applications using a low-overhead protocol. This protocol is optimized for a variety of network configurations and can be used for remote direct memory access (RDMA) operations, such as streaming data or memory-mapped files. RDMA service in Linux is designed to reduce the overhead associated with networking and can provide improved performance and scalability. In addition, it can be used to offload certain networking tasks, such as '''packet processing''' and network I/O, improving scalability and freeing up the system’s resources for other tasks.


<syntaxhighlight lang="bash">
# RoCE v2 활성화
ethtool --set-ets eth0 pfc 1 0 0 0 0 1 0 0


To run RDMA, you must use pinned memory, i.e. memory that cannot be swapped between kernel components. It is recommended that every process that is not a root user run at low memory consumption (64KB). The amount of memory that can be locked must be increased in order for RDMA to function properly.<ref>https://www.systranbox.com/checking-for-rdma-capability-on-linux-how-to-determine-if-your-nic-is-rdma-ready/</ref>
# PFC (Priority Flow Control) 설정
ethtool --pause eth0 rx on tx on


== RDMA supported NIC ==
# RoCE 성능 테스트
To check if your NIC supports RDMA, you can use the lspci command in the Linux terminal, which will list all the PCI devices connected to your system. Additionally, you can look up your NIC’s specifications online to see if it supports RDMA.
ib_write_bw -d mlx5_0 -F
</syntaxhighlight>


== GPUDirect RDMA ==
=== IB 설정 ===
When training large-scale models in multi-node cluster, the cost of inter-node communication is significant.


The combination of InfiniBand and GPUs enables a feature called [[GPUDirect]] RDMA, which allows direct communication between GPUs across nodes without involving memory and CPU. In other words, the communication between GPUs of two nodes takes place directly through the InfiniBand network interface cards, bypassing the need to go through CPU and memory.
<syntaxhighlight lang="bash">
# IB 링크 상태 확인
ibstatus


== References ==
# IB 성능 테스트
<references />
ib_write_bw -d mlx5_1
[[Category:Linux]]
ib_send_lat -d mlx5_1


[[Category:Reference]]
# OpenMPI 연동
mpirun --mca btl_openib_allow_rc 1 -np 8 ./app
</syntaxhighlight>

Latest revision as of 13:37, 17 July 2026

Template:Status

Template:TOC

Overview

RDMA에 대한 기술 문서입니다.

Summary

  • 무엇인가? - RDMA
  • 왜 필요한가? - HPC 및 서버 환경에서 필수 개념
  • 언제 사용하는가? - 서버 구성, 성능 튜닝, 문제 해결 시


Purpose

이 문서가 존재하는 이유

  • Goal: RDMA에 대한 기술 정보 제공
  • Scope: RDMA의 개념, 사용법, 설정
  • Non-goals: 다른 주제로의 확장


Key Concepts

Concept Description Related
RDMA HPC/서버 환경에서 중요한 기술 개념 Linux, Server


Detailed Explanation

RDMA (Remote Direct Memory Access), which means remote direct memory access, is a network communication protocol that was first applied in the field of high-performance computing, and has gradually become popular in data centers. RDMA allows user programs to bypass the operating system kernel (CPU) and directly interact with the network card for network communication, thereby providing high bandwidth and extremely small latency.[1]

RDMA FLOW.png
  1. Infiniband is a network protocol tailored specifically for RDMA, which can ensure the reliability of data transmission from the hardware level. Although InfiniBand technical specifications and standard specifications were officially published in 2000, InfiniBand Architecture (IBA) has been widely used on cluster supercomputers after 2005. The biggest reason for the slow development is that Infiniband requires its own dedicated hardware from L2 to L4. The cost of the enterprise is very high.
  2. RoCE (RDMA over Converged Ethernet) is a protocol for implemtenting RDMA in an Ethernet network for a cheaper RDMA option. RoCE primarily provides RDMA capabilities over Ethernet.
    1. RoCEv1, which runs at the link layer and cannot be run over a routed network. Therefore, it requires the priority flow control (PFC) to be enabled. The RoCE v1 protocol is an Ethernet link layer protocol allowing two hosts in the same Ethernet broadcast domain (VLAN) to communicate. It uses Ethertype 0x8915, which limits the frame length to 1500 bytes for a standard Ethernet frame and 9000 bytes for an Ethernet jumbo frame.
    2. RoCEv2, which runs over layer 3. Because it is a routed solution, consider using explicit congestion notification (ECN) with RoCEv2 as ECN bits are communicated end-to-end across a routed network. By changing the packet encapsulation to include IP and UDP headers, RoCE v2 can now be used across both L2 and L3 networks. This enables Layer 3 routing, which brings RDMA to network with multiple subnets for great scalability. Therefore, RoCE v2 is also regarded as Routable RoCE (RRoCE). Owing to the arrival of RoCE v2, the IP multicast is now also possible.
  3. iWARP iWARP is a protocol for implementing RDMA across Internet Protocol (IP) networks. Its advantage is that it can run in today's standard TCP/IP network. RDMA can be used only by purchasing a network card that supports iWARP. For those with a slightly lower financial budget It is especially suitable for enterprises. But its disadvantage is that it is slightly worse than RoCE in performance. After all, you get what you pay for.
Contents RoCE Infiniband iWARP TCP/IP
High scalability Good Excellent Wore than InfiniBand and RoCE Poor


Best Practices

  • 최신 버전 사용 권장
  • 공식 문서 참고
  • 테스트 환경에서 먼저 검증


References


Related Pages

Knowledge Graph

Related

→ Linux → Server → Hardware → Network

RoCE vs InfiniBand 비교

Feature ! RoCE v2 ! InfiniBand
Ethernet | 전용 IB 네트워크
표준 Ethernet 스위치 | IB 스위치
1-2 μs | 0.5-1 μs
100/200/400 GbE | 200/400 Gb/s
낮음 (기존 인프라 활용) | 높음
낮음 | 높음
있음 | 있음
지원 | 지원

RoCE 설정

# RoCE v2 활성화
ethtool --set-ets eth0 pfc 1 0 0 0 0 1 0 0

# PFC (Priority Flow Control) 설정
ethtool --pause eth0 rx on tx on

# RoCE 성능 테스트
ib_write_bw -d mlx5_0 -F

IB 설정

# IB 링크 상태 확인
ibstatus

# IB 성능 테스트
ib_write_bw -d mlx5_1
ib_send_lat -d mlx5_1

# OpenMPI 연동
mpirun --mca btl_openib_allow_rc 1 -np 8 ./app
  1. ↑ https://www.naddod.com/blog/rdma-high-speed-network-for-large-model-training
Retrieved from "http://wiki.hpcmate.com/index.php?title=RDMA&oldid=3524"