Transformer Engine

Overview

Transformer Engine에 대한 기술 문서입니다.

Summary

• 무엇인가? - Transformer Engine
• 왜 필요한가? - HPC 및 서버 환경에서 필수 개념
• 언제 사용하는가? - 서버 구성, 성능 튜닝, 문제 해결 시

Purpose

이 문서가 존재하는 이유

• Goal: Transformer Engine에 대한 기술 정보 제공
• Scope: Transformer Engine의 개념, 사용법, 설정
• Non-goals: 다른 주제로의 확장

Key Concepts

Concept	Description	Related
Transformer Engine	HPC/서버 환경에서 중요한 기술 개념	Linux, Server

Detailed Explanation

Most deep learning frameworks train with FP32 by default. This is not essential, however, to achieve full accuracy for many deep learning models. Using mixed-precision training, which combines single-precision (FP32) with lower precision (FP16) format when training a model, results in significant speedups with minimal differences in accuracy as compared to FP32 training. With Hopper and Ada GPU architecture FP8 precision was introduced, which offers improved performance over FP16 with no degradation in accuracy. Although all major deep learning frameworks support FP16, FP8 support is not available natively in frameworks today. NVIDIA^® Transformer Engine is a library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit floating point (FP8) precision on Hopper and Ada GPUs, to provide better performance with lower memory utilization in both training and inference. TE addresses the problem of FP8 support by providing APIs, a Python API consisting of modules to easily build a Transformer layer as well as a framework-agnostic library in C++ including structs and kernels needed for FP8 training, greatly simplifying mixed precision training for users. TE is available at open source under Apache-2.0 license, TE supports

• Support for FP8 on NVIDIA Hopper and NVIDIA Ada GPUs
• Support for optimizations across all precisions (FP16, BF16) on NVIDIA Ampere GPU architecture generations and later
• Optimizations (e.g. fused kernels) for Transformer models
• Easy-to-use modules for building Transformer layers with FP8 support

Transformer Engine has been integrated with popular LLM frameworks such as:

• DeepSpeed
• Hugging Face Accelerate
• Lightning
• MosaicML Composer
• NVIDIA JAX Toolbox
• NVIDIA Megatron-LM
• NVIDIA NeMo Framework
• Amazon SageMaker Model Parallel Library
• Levanter
• Hugging Face Nanotron - Coming soon!
• Colossal-AI - Coming soon!

Best Practices

• 최신 버전 사용 권장
• 공식 문서 참고
• 테스트 환경에서 먼저 검증

References

• Transformer Engine

Related Pages

• Linux
• Server
• Hardware
• Network

Knowledge Graph

Related

→ Linux → Server → Hardware → Network