RDMA

Overview

RDMA에 대한 기술 문서입니다.

Summary

• 무엇인가? - RDMA
• 왜 필요한가? - HPC 및 서버 환경에서 필수 개념
• 언제 사용하는가? - 서버 구성, 성능 튜닝, 문제 해결 시

Purpose

이 문서가 존재하는 이유

• Goal: RDMA에 대한 기술 정보 제공
• Scope: RDMA의 개념, 사용법, 설정
• Non-goals: 다른 주제로의 확장

Key Concepts

Concept	Description	Related
RDMA	HPC/서버 환경에서 중요한 기술 개념	Linux, Server

Detailed Explanation

RDMA (Remote Direct Memory Access), which means remote direct memory access, is a network communication protocol that was first applied in the field of high-performance computing, and has gradually become popular in data centers. RDMA allows user programs to bypass the operating system kernel (CPU) and directly interact with the network card for network communication, thereby providing high bandwidth and extremely small latency.

1. Infiniband is a network protocol tailored specifically for RDMA, which can ensure the reliability of data transmission from the hardware level. Although InfiniBand technical specifications and standard specifications were officially published in 2000, InfiniBand Architecture (IBA) has been widely used on cluster supercomputers after 2005. The biggest reason for the slow development is that Infiniband requires its own dedicated hardware from L2 to L4. The cost of the enterprise is very high.
2. RoCE (RDMA over Converged Ethernet) is a protocol for implemtenting RDMA in an Ethernet network for a cheaper RDMA option. RoCE primarily provides RDMA capabilities over Ethernet.
   1. RoCEv1, which runs at the link layer and cannot be run over a routed network. Therefore, it requires the priority flow control (PFC) to be enabled. The RoCE v1 protocol is an Ethernet link layer protocol allowing two hosts in the same Ethernet broadcast domain (VLAN) to communicate. It uses Ethertype 0x8915, which limits the frame length to 1500 bytes for a standard Ethernet frame and 9000 bytes for an Ethernet jumbo frame.
   2. RoCEv2, which runs over layer 3. Because it is a routed solution, consider using explicit congestion notification (ECN) with RoCEv2 as ECN bits are communicated end-to-end across a routed network. By changing the packet encapsulation to include IP and UDP headers, RoCE v2 can now be used across both L2 and L3 networks. This enables Layer 3 routing, which brings RDMA to network with multiple subnets for great scalability. Therefore, RoCE v2 is also regarded as Routable RoCE (RRoCE). Owing to the arrival of RoCE v2, the IP multicast is now also possible.
3. iWARP iWARP is a protocol for implementing RDMA across Internet Protocol (IP) networks. Its advantage is that it can run in today's standard TCP/IP network. RDMA can be used only by purchasing a network card that supports iWARP. For those with a slightly lower financial budget It is especially suitable for enterprises. But its disadvantage is that it is slightly worse than RoCE in performance. After all, you get what you pay for.

Contents	RoCE	Infiniband	iWARP	TCP/IP
High scalability	Good	Excellent	Wore than InfiniBand and RoCE	Poor Best Practices 최신 버전 사용 권장 공식 문서 참고 테스트 환경에서 먼저 검증 References RDMA Related Pages Linux Server Hardware Network Knowledge Graph Related → Linux → Server → Hardware → Network