RDMA: Difference between revisions

From HPCWIKI
Jump to navigation Jump to search
(Add categories: Linux, Reference)
(Template migration to LLM-Optimized Wiki Template)
Line 1: Line 1:
RDMA (Remote Direct Memory Access), which means remote direct memory access, is a network communication protocol that was first applied in the field of high-performance computing, and has gradually become popular in data centers.
{{Status
|status=Draft
|owner=Knowledge Agent
|last_update=2026-07-16
|review=Pending
}}
 
{{TOC}}
 
== Overview ==
 
RDMA에 대한 기술 문서입니다.
 
=== Summary ===
 
* 무엇인가? - RDMA
* 왜 필요한가? - HPC 및 서버 환경에서 필수 개념
* 언제 사용하는가? - 서버 구성, 성능 튜닝, 문제 해결 시
 
---
 
== Purpose ==
 
이 문서가 존재하는 이유
 
* Goal: RDMA에 대한 기술 정보 제공
* Scope: RDMA의 개념, 사용법, 설정
* Non-goals: 다른 주제로의 확장


---


RDMA allows user programs to bypass the operating system kernel (CPU) and directly interact with the network card for network communication, thereby providing high bandwidth and extremely small latency.<ref>https://www.naddod.com/blog/rdma-high-speed-network-for-large-model-training</ref>
== Key Concepts ==


[[File:RDMA FLOW.png|center|frameless|557x557px]]
{| class="wikitable"
! Concept
! Description
! Related
|-
| RDMA
| HPC/서버 환경에서 중요한 기술 개념
| [[Linux]], [[Server]]
|}
 
---


== Types of RDMA ==
== Detailed Explanation ==


RDMA (Remote Direct Memory Access), which means remote direct memory access, is a network communication protocol that was first applied in the field of high-performance computing, and has gradually become popular in data centers.
RDMA allows user programs to bypass the operating system [[kernel]] (CPU) and directly interact with the network card for network communication, thereby providing high bandwidth and extremely small latency.<ref>https://www.naddod.com/blog/rdma-high-speed-network-for-large-model-training</ref>
[[File:RDMA FLOW.png|center|frameless|557x557px]]
# Infiniband is a network protocol tailored specifically for RDMA, which can ensure the reliability of data transmission from the hardware level. Although InfiniBand technical specifications and standard specifications were officially published in 2000, InfiniBand Architecture (IBA) has been widely used on cluster supercomputers after 2005. The biggest reason for the slow development is that Infiniband requires its own dedicated hardware from L2 to L4. The cost of the enterprise is very high.  
# Infiniband is a network protocol tailored specifically for RDMA, which can ensure the reliability of data transmission from the hardware level. Although InfiniBand technical specifications and standard specifications were officially published in 2000, InfiniBand Architecture (IBA) has been widely used on cluster supercomputers after 2005. The biggest reason for the slow development is that Infiniband requires its own dedicated hardware from L2 to L4. The cost of the enterprise is very high.  
# RoCE (RDMA over Converged Ethernet) is a protocol for implemtenting RDMA in an Ethernet network for a cheaper RDMA option. RoCE primarily provides RDMA capabilities over Ethernet.  
# RoCE (RDMA over Converged Ethernet) is a protocol for implemtenting RDMA in an Ethernet network for a cheaper RDMA option. RoCE primarily provides RDMA capabilities over Ethernet.  
## RoCEv1, which runs at the link layer and cannot be run over a routed network. Therefore, it requires the priority flow control (PFC) to be enabled. The RoCE v1 protocol is an Ethernet link layer protocol allowing two hosts in the same Ethernet broadcast domain (VLAN) to communicate. It uses Ethertype 0x8915, which limits the frame length to 1500 bytes for a standard Ethernet frame and 9000 bytes for an Ethernet jumbo frame.
## RoCEv1, which runs at the link layer and cannot be run over a routed network. Therefore, it requires the priority flow control (PFC) to be enabled. The RoCE v1 protocol is an Ethernet link layer protocol allowing two hosts in the same Ethernet broadcast domain (VLAN) to communicate. It uses Ethertype 0x8915, which limits the frame length to 1500 bytes for a standard Ethernet frame and 9000 bytes for an Ethernet jumbo frame.
## RoCEv2, which runs over layer 3. Because it is a routed solution, consider using explicit congestion notification (ECN) with RoCEv2 as ECN bits are communicated end-to-end across a routed network. By changing the packet encapsulation to include IP and UDP headers, RoCE v2 can now be used across both L2 and L3 networks. This enables Layer 3 routing, which brings RDMA to network with multiple subnets for great scalability. Therefore, RoCE v2 is also regarded as Routable RoCE (RRoCE). Owing to the arrival of RoCE v2, the IP multicast is now also possible.
## RoCEv2, which runs over layer 3. Because it is a routed solution, consider using explicit congestion notification (ECN) with RoCEv2 as ECN bits are communicated end-to-end across a routed network. By changing the packet encapsulation to include IP and UDP headers, RoCE v2 can now be used across both L2 and L3 networks. This enables Layer 3 routing, which brings RDMA to network with multiple subnets for great scalability. Therefore, RoCE v2 is also regarded as Routable RoCE (RRoCE). Owing to the arrival of RoCE v2, the IP multicast is now also possible.
# iWARP iWARP is a protocol for implementing RDMA across Internet Protocol (IP) networks. Its advantage is that it can run in today's standard TCP/IP network. RDMA can be used only by purchasing a network card that supports iWARP. For those with a slightly lower financial budget It is especially suitable for enterprises. But its disadvantage is that it is slightly worse than RoCE in performance. After all, you get what you pay for.
# iWARP iWARP is a protocol for implementing RDMA across Internet Protocol (IP) networks. Its advantage is that it can run in today's standard TCP/IP network. RDMA can be used only by purchasing a network card that supports iWARP. For those with a slightly lower financial budget It is especially suitable for enterprises. But its disadvantage is that it is slightly worse than RoCE in performance. After all, you get what you pay for.
== The Difference Between RDMA and TCP/IP<ref>https://community.fs.com/article/roce-vs-infiniband-vs-tcp-ip.html</ref><ref>https://community.fs.com/article/roce-rdma-over-converged-ethernet.html</ref> ==
{| class="wikitable"
{| class="wikitable"
!Contents
!Contents
Line 28: Line 66:
|Wore than InfiniBand and RoCE
|Wore than InfiniBand and RoCE
|Poor
|Poor
|-
|High performance
|Equivalent to InfiniBand
|Excellent
|Slightly worse than InfiniBand (affected by TCP)
|Poor
|-
|High stability
|Good
|Excellent
|Poor
|Excellent
|-
|Easy management
|Hard
|Hard
|Harder than RoCE
|Easy
|-
|Cost-efficiency
|Low
|High
|Medium
|Low
|-
|Network device
|Network device
|IB switch
|Network device
|Network device
|}
[[File:InfiniBand Vs. RoCEv2.png|center|frameless|808x808px|InfiniBand Vs. RoCEv2]]


== RDMA service ==
---
RDMA service in [[Linux]] is a technology that allows applications to communicate directly with each other over a network without involving the operating system. It provides high-speed, low-latency data transfer between applications using a low-overhead protocol. This protocol is optimized for a variety of network configurations and can be used for remote direct memory access (RDMA) operations, such as streaming data or memory-mapped files. RDMA service in Linux is designed to reduce the overhead associated with networking and can provide improved performance and scalability. In addition, it can be used to offload certain networking tasks, such as '''packet processing''' and network I/O, improving scalability and freeing up the system’s resources for other tasks.
 
== Best Practices ==
 
* 최신 버전 사용 권장
* 공식 문서 참고
* 테스트 환경에서 먼저 검증
 
---


== References ==


To run RDMA, you must use pinned memory, i.e. memory that cannot be swapped between kernel components. It is recommended that every process that is not a root user run at low memory consumption (64KB). The amount of memory that can be locked must be increased in order for RDMA to function properly.<ref>https://www.systranbox.com/checking-for-rdma-capability-on-linux-how-to-determine-if-your-nic-is-rdma-ready/</ref>
* [https://wiki.hpcmate.com RDMA]


== RDMA supported NIC ==
---
To check if your NIC supports RDMA, you can use the lspci command in the Linux terminal, which will list all the PCI devices connected to your system. Additionally, you can look up your NIC’s specifications online to see if it supports RDMA.


== GPUDirect RDMA ==
== Related Pages ==
When training large-scale models in multi-node cluster, the cost of inter-node communication is significant.


The combination of InfiniBand and GPUs enables a feature called [[GPUDirect]] RDMA, which allows direct communication between GPUs across nodes without involving memory and CPU. In other words, the communication between GPUs of two nodes takes place directly through the InfiniBand network interface cards, bypassing the need to go through CPU and memory.
* [[Linux]]
* [[Server]]
* [[Hardware]]
* [[Network]]


== References ==
---
<references />
[[Category:Linux]]


[[Category:Network]]
[[Category:Reference]]
[[Category:Reference]]

Revision as of 15:17, 16 July 2026

Template:Status

Template:TOC

Overview

RDMA에 대한 기술 문서입니다.

Summary

  • 무엇인가? - RDMA
  • 왜 필요한가? - HPC 및 서버 환경에서 필수 개념
  • 언제 사용하는가? - 서버 구성, 성능 튜닝, 문제 해결 시

---

Purpose

이 문서가 존재하는 이유

  • Goal: RDMA에 대한 기술 정보 제공
  • Scope: RDMA의 개념, 사용법, 설정
  • Non-goals: 다른 주제로의 확장

---

Key Concepts

Concept Description Related
RDMA HPC/서버 환경에서 중요한 기술 개념 Linux, Server

---

Detailed Explanation

RDMA (Remote Direct Memory Access), which means remote direct memory access, is a network communication protocol that was first applied in the field of high-performance computing, and has gradually become popular in data centers. RDMA allows user programs to bypass the operating system kernel (CPU) and directly interact with the network card for network communication, thereby providing high bandwidth and extremely small latency.[1]

RDMA FLOW.png
  1. Infiniband is a network protocol tailored specifically for RDMA, which can ensure the reliability of data transmission from the hardware level. Although InfiniBand technical specifications and standard specifications were officially published in 2000, InfiniBand Architecture (IBA) has been widely used on cluster supercomputers after 2005. The biggest reason for the slow development is that Infiniband requires its own dedicated hardware from L2 to L4. The cost of the enterprise is very high.
  2. RoCE (RDMA over Converged Ethernet) is a protocol for implemtenting RDMA in an Ethernet network for a cheaper RDMA option. RoCE primarily provides RDMA capabilities over Ethernet.
    1. RoCEv1, which runs at the link layer and cannot be run over a routed network. Therefore, it requires the priority flow control (PFC) to be enabled. The RoCE v1 protocol is an Ethernet link layer protocol allowing two hosts in the same Ethernet broadcast domain (VLAN) to communicate. It uses Ethertype 0x8915, which limits the frame length to 1500 bytes for a standard Ethernet frame and 9000 bytes for an Ethernet jumbo frame.
    2. RoCEv2, which runs over layer 3. Because it is a routed solution, consider using explicit congestion notification (ECN) with RoCEv2 as ECN bits are communicated end-to-end across a routed network. By changing the packet encapsulation to include IP and UDP headers, RoCE v2 can now be used across both L2 and L3 networks. This enables Layer 3 routing, which brings RDMA to network with multiple subnets for great scalability. Therefore, RoCE v2 is also regarded as Routable RoCE (RRoCE). Owing to the arrival of RoCE v2, the IP multicast is now also possible.
  3. iWARP iWARP is a protocol for implementing RDMA across Internet Protocol (IP) networks. Its advantage is that it can run in today's standard TCP/IP network. RDMA can be used only by purchasing a network card that supports iWARP. For those with a slightly lower financial budget It is especially suitable for enterprises. But its disadvantage is that it is slightly worse than RoCE in performance. After all, you get what you pay for.
Contents RoCE Infiniband iWARP TCP/IP
High scalability Good Excellent Wore than InfiniBand and RoCE Poor

---

Best Practices

  • 최신 버전 사용 권장
  • 공식 문서 참고
  • 테스트 환경에서 먼저 검증

---

References

---

Related Pages

---

  1. ↑ https://www.naddod.com/blog/rdma-high-speed-network-for-large-model-training
Retrieved from "http://wiki.hpcmate.com/index.php?title=RDMA&oldid=2731"