Transformer Engine: Difference between revisions

From HPCWIKI
Jump to navigation Jump to search
(Add categories: AI, Reference)
(Template migration to LLM-Optimized Wiki Template)
Line 1: Line 1:
== What is (TE) Transfermer Engine ==
{{Status
Most [[Deep Learning Frameworks|deep learning frameworks]] train with FP32 by default. This is not essential, however, to achieve full accuracy for many deep learning models. Using mixed-precision training, which combines <u>single-precision (FP32)</u> with <u>lower precision (FP16)</u> format when training a model, results in significant speedups with minimal differences in accuracy as compared to FP32 training. With Hopper and Ada GPU architecture <u>FP8 precision</u> was introduced, which offers improved performance over FP16 with no degradation in accuracy. Although all major [[Deep Learning Frameworks|deep learning frameworks]] [[support]] FP16, FP8 support is not available natively in frameworks today.
|status=Draft
|owner=Knowledge Agent
|last_update=2026-07-16
|review=Pending
}}
 
{{TOC}}
 
== Overview ==
 
Transformer Engine에 대한 기술 문서입니다.
 
=== Summary ===
 
* 무엇인가? - Transformer Engine
* 왜 필요한가? - HPC 및 서버 환경에서 필수 개념
* 언제 사용하는가? - 서버 구성, 성능 튜닝, 문제 해결 시
 
---
 
== Purpose ==
 
이 문서가 존재하는 이유
 
* Goal: Transformer Engine에 대한 기술 정보 제공
* Scope: Transformer Engine의 개념, 사용법, 설정
* Non-goals: 다른 주제로의 확장


---


== Key Concepts ==


NVIDIA<sup><small>®</small></sup> Transformer Engine is a [[library]] for accelerating [[Transformer model|Transformer models]] on NVIDIA GPUs, including using 8-bit floating point (FP8) precision on '''Hopper and Ada GPUs''', to provide better performance with lower memory utilization in both [[Training and Inference|training and inference]].<ref>https://docs.nvidia.com/deeplearning/transformer-engine/</ref>
{| class="wikitable"
! Concept
! Description
! Related
|-
| Transformer Engine
| HPC/서버 환경에서 중요한 기술 개념
| [[Linux]], [[Server]]
|}


---


== Detailed Explanation ==


Most [[Deep Learning Frameworks|deep learning frameworks]] train with FP32 by default. This is not essential, however, to achieve full accuracy for many deep learning models. Using mixed-precision training, which combines <u>single-precision (FP32)</u> with <u>lower precision (FP16)</u> format when training a model, results in significant speedups with minimal differences in accuracy as compared to FP32 training. With Hopper and Ada GPU architecture <u>FP8 precision</u> was introduced, which offers improved performance over FP16 with no degradation in accuracy. Although all major [[Deep Learning Frameworks|deep learning frameworks]] [[support]] FP16, FP8 support is not available natively in frameworks today.
NVIDIA<sup><small>®</small></sup> Transformer Engine is a [[library]] for accelerating [[Transformer model|Transformer models]] on [[NVIDIA GPUs]], including using 8-bit floating point (FP8) precision on '''Hopper and Ada GPUs''', to provide better performance with lower memory utilization in both [[Training and Inference|training and inference]].<ref>https://docs.nvidia.com/deeplearning/transformer-engine/</ref>
TE addresses the problem of FP8 support by providing APIs, a Python API consisting of modules to easily build a Transformer layer as well as a framework-agnostic library in C++ including structs and kernels needed for FP8 training, greatly simplifying mixed precision training for users.
TE addresses the problem of FP8 support by providing APIs, a Python API consisting of modules to easily build a Transformer layer as well as a framework-agnostic library in C++ including structs and kernels needed for FP8 training, greatly simplifying mixed precision training for users.
TE is available at [https://github.com/NVIDIA/TransformerEngine open source] under Apache-2.0 license, TE supports
TE is available at [https://github.com/NVIDIA/TransformerEngine open source] under Apache-2.0 license, TE supports
* Support for FP8 on NVIDIA Hopper and NVIDIA Ada GPUs
* Support for FP8 on NVIDIA Hopper and NVIDIA Ada GPUs
* Support for optimizations across all precisions (FP16, BF16) on NVIDIA [[Ampere GPU]] architecture generations and later
* Support for optimizations across all precisions (FP16, BF16) on NVIDIA [[Ampere GPU]] architecture generations and later
* Optimizations (e.g. fused kernels) for [[Transformer model|Transformer models]]
* Optimizations (e.g. fused kernels) for [[Transformer model|Transformer models]]
* Easy-to-use modules for building Transformer layers with FP8 support
* Easy-to-use modules for building Transformer layers with FP8 support
Transformer Engine has been integrated with popular LLM frameworks such as:<ref>https://github.com/NVIDIA/TransformerEngine</ref>
Transformer Engine has been integrated with popular LLM frameworks such as:<ref>https://github.com/NVIDIA/TransformerEngine</ref>
* DeepSpeed
* DeepSpeed
* Hugging Face Accelerate
* Hugging Face Accelerate
Line 30: Line 66:
* Hugging Face Nanotron - Coming soon!
* Hugging Face Nanotron - Coming soon!
* Colossal-AI - Coming soon!
* Colossal-AI - Coming soon!
* PeriFlow - Coming soon!
* GPT-NeoX - Coming soon!


== TE introduction videos ==
---


* [https://www.nvidia.com/en-us/on-demand/session/gtc24-s62457/ What's New in Transformer Engine and FP8 Training | GTC 2024]
== Best Practices ==
* [https://www.nvidia.com/en-us/on-demand/session/gtcspring23-s51393 FP8 Training with Transformer Engine | GTC 2023]
 
* [https://www.nvidia.com/en-us/on-demand/session/gtcspring23-s52166/ FP8 for Deep Learning | GTC 2023]
* 최신 버전 사용 권장
* 공식 문서 참고
* 테스트 환경에서 먼저 검증
 
---


== References ==
== References ==
<references />
[[Category:AI]]


* [https://wiki.hpcmate.com Transformer Engine]
---
== Related Pages ==
* [[Linux]]
* [[Server]]
* [[Hardware]]
* [[Network]]
---
[[Category:Server]]
[[Category:Reference]]
[[Category:Reference]]

Revision as of 15:26, 16 July 2026

Template:Status

Template:TOC

Overview

Transformer Engine에 대한 기술 문서입니다.

Summary

  • 무엇인가? - Transformer Engine
  • 왜 필요한가? - HPC 및 서버 환경에서 필수 개념
  • 언제 사용하는가? - 서버 구성, 성능 튜닝, 문제 해결 시

---

Purpose

이 문서가 존재하는 이유

  • Goal: Transformer Engine에 대한 기술 정보 제공
  • Scope: Transformer Engine의 개념, 사용법, 설정
  • Non-goals: 다른 주제로의 확장

---

Key Concepts

Concept Description Related
Transformer Engine HPC/서버 환경에서 중요한 기술 개념 Linux, Server

---

Detailed Explanation

Most deep learning frameworks train with FP32 by default. This is not essential, however, to achieve full accuracy for many deep learning models. Using mixed-precision training, which combines single-precision (FP32) with lower precision (FP16) format when training a model, results in significant speedups with minimal differences in accuracy as compared to FP32 training. With Hopper and Ada GPU architecture FP8 precision was introduced, which offers improved performance over FP16 with no degradation in accuracy. Although all major deep learning frameworks support FP16, FP8 support is not available natively in frameworks today. NVIDIA® Transformer Engine is a library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit floating point (FP8) precision on Hopper and Ada GPUs, to provide better performance with lower memory utilization in both training and inference.[1] TE addresses the problem of FP8 support by providing APIs, a Python API consisting of modules to easily build a Transformer layer as well as a framework-agnostic library in C++ including structs and kernels needed for FP8 training, greatly simplifying mixed precision training for users. TE is available at open source under Apache-2.0 license, TE supports

  • Support for FP8 on NVIDIA Hopper and NVIDIA Ada GPUs
  • Support for optimizations across all precisions (FP16, BF16) on NVIDIA Ampere GPU architecture generations and later
  • Optimizations (e.g. fused kernels) for Transformer models
  • Easy-to-use modules for building Transformer layers with FP8 support

Transformer Engine has been integrated with popular LLM frameworks such as:[2]

  • DeepSpeed
  • Hugging Face Accelerate
  • Lightning
  • MosaicML Composer
  • NVIDIA JAX Toolbox
  • NVIDIA Megatron-LM
  • NVIDIA NeMo Framework
  • Amazon SageMaker Model Parallel Library
  • Levanter
  • Hugging Face Nanotron - Coming soon!
  • Colossal-AI - Coming soon!

---

Best Practices

  • 최신 버전 사용 권장
  • 공식 문서 참고
  • 테스트 환경에서 먼저 검증

---

References

---

Related Pages

---

  1. ↑ https://docs.nvidia.com/deeplearning/transformer-engine/
  2. ↑ https://github.com/NVIDIA/TransformerEngine
Retrieved from "http://wiki.hpcmate.com/index.php?title=Transformer_Engine&oldid=2916"