THP (Transparent Huge Pages): Difference between revisions

From HPCWIKI
Jump to navigation Jump to search
(새 문서: Performance critical computing applications dealing with large memory working sets are already running on top of libhugetlbfs and in turn hugetlbfs. THP can be enabled system wide or restricted to certain tasks or even memory ranges inside task’s address space. Unless THP is completely disabled, there is <code>khugepaged</code> daemon that scans memory and collapses sequences of basic pages into huge pages == Need of application restart == The transparent_hugepage/enabled v...)
 
(Fix: remove --- horizontal lines (7 removed))
 
(16 intermediate revisions by 2 users not shown)
Line 1: Line 1:
Performance critical computing applications dealing with large memory working sets are already running on top of libhugetlbfs and in turn hugetlbfs.
{{Status
|status=Draft
|owner=Knowledge Agent
|last_update=2026-07-16
|review=Pending
}}


{{TOC}}


THP can be enabled system wide or restricted to certain tasks or even memory ranges inside task’s address space. Unless THP is completely disabled, there is <code>khugepaged</code> daemon that scans memory and collapses sequences of basic pages into huge pages
== Overview ==


== Need of application restart ==
THP (Transparent Huge Pages)에 대한 기술 문서입니다.
The transparent_hugepage/enabled values and tmpfs mount option only affect future behavior. So to make them effective you need to restart any application that could have been using hugepages. This also applies to the regions registered in khugepaged.


== Monitoring usage ==
=== Summary ===
 
* 무엇인가? - THP (Transparent Huge Pages)
* 왜 필요한가? - HPC 및 서버 환경에서 필수 개념
* 언제 사용하는가? - 서버 구성, 성능 튜닝, 문제 해결 시
 
 
== Purpose ==
 
이 문서가 존재하는 이유
 
* Goal: THP (Transparent Huge Pages)에 대한 기술 정보 제공
* Scope: THP (Transparent Huge Pages)의 개념, 사용법, 설정
* Non-goals: 다른 주제로의 확장
 
 
== Key Concepts ==
 
{| class="wikitable"
! Concept
! Description
! Related
|-
| THP (Transparent Huge Pages)
| HPC/서버 환경에서 중요한 기술 개념
| [[Linux]], [[Server]]
|}
 
 
== Detailed Explanation ==
 
Transparent Huge Pages (THP) is an alternative solution to HugeTLB. With THP, the [[kernel]] automatically assigns huge pages to processes, so huge pages do not need to be reserved manually
The THP feature has two modes of operation: system-wide and per-process. When THP is enabled system-wide, the kernel tries to assign huge pages to any process when it is possible to allocate huge pages and the process is using a large contiguous virtual memory area. If THP is enabled per-process, the kernel only assigns huge pages to individual processes' memory areas specified with the <code>madvise()</code> system call.
In [[Linux]] system, Memory is managed in blocks known as ''pages''. '''A page is 4096 bytes'''. 1MB of memory is equal to 256 pages; 1GB of memory is equal to 256,000 pages, etc
CPUs have a built-in ''memory management unit'' that contains a list of these pages.
Two ways to enable the system to manage large amounts of memory:
# Increase the number of page table entries in the hardware memory management unit
# Increase the page size   
The first method is expensive, since the hardware memory management unit in a modern processor only supports hundreds or thousands of page table entries. Additionally, hardware and memory management algorithms that work well with thousands of pages (megabytes of memory) may have difficulty performing well with millions (or even billions) of pages. This results in performance issues: when an application needs to use more memory pages than the memory management unit supports, the system falls back to slower, software-based memory management, which causes the entire system to run more slowly. it can empact to performance critical computing applications dealing with large memory working sets.
The first method is expensive, since the hardware memory management unit in a modern processor only supports hundreds or thousands of page table entries. Additionally, hardware and memory management algorithms that work well with thousands of pages (megabytes of memory) may have difficulty performing well with millions (or even billions) of pages. This results in performance issues: when an application needs to use more memory pages than the memory management unit supports, the system falls back to slower, software-based memory management, which causes the entire system to run more slowly. it can empact to performance critical computing applications dealing with large memory working sets.
{| class="wikitable"
{| class="wikitable"
|+
|+
!File
!Type
!Field
!Comments on THP
!Description
!Notes
|-
|-
| rowspan="2" |<code>/proc/meminfo</code>
|Database
|AnonHugePages
|database workloads often perform poorly with THP enabled, '''because they tend to have sparse rather than contiguous memory access patterns'''. When running MongoDB on Linux, THP should be disabled for best performance.<ref>https://www.mongodb.com/docs/manual/tutorial/transparent-huge-pages/</ref>
|The number of anonymous transparent huge pages currently used by the system
a Linux memory management feature, often slows down database performance<ref>https://www.pingcap.com/blog/transparent-huge-pages-why-we-disable-it-for-databases/</ref>
To identify what applications are using anonymous transparent huge pages, it is necessary to read <code>/proc/PID/smaps</code> and count the AnonHugePages fields for each mapping
| rowspan="2" |** Note that reading the smaps file is expensive and reading it frequently will incur overhead.
|-
|-
|ShmemPmdMapped
|Oracle
|The number of file transparent huge pages mapped to userspace.
|Transparen... [내용 계속]
To identify what applications are mapping file transparent huge pages, it is necessary to read <code>/proc/PID/smaps</code> and count the FileHugeMapped fields for each mapping
 
|-
 
|<code>/proc/vmstat</code>
== Best Practices ==
|
 
|used to monitor how successfully the system is providing huge pages for use
* 최신 버전 사용 권장
* 공식 문서 참고
* 테스트 환경에서 먼저 검증
 
 
== References ==
 
* [https://wiki.hpcmate.com THP (Transparent Huge Pages)]
 
 
== Related Pages ==
 
* [[Linux]]
* [[Server]]
* [[Hardware]]
* [[Network]]
 


; thp_fault_alloc
[[Category:Linux]]
: is incremented every time a huge page is successfully allocated to handle a page fault.
== Knowledge Graph ==
; thp_collapse_alloc
: is incremented by khugepaged when it has found a range of pages to collapse into one huge page and has successfully allocated a new huge page to store the data.
; thp_fault_fallback
: is incremented if a page fault fails to allocate a huge page and instead falls back to using small pages.
; thp_fault_fallback_charge
: is incremented if a page fault fails to charge a huge page and instead falls back to using small pages even though the allocation was successful.
; thp_collapse_alloc_failed
: is incremented if khugepaged found a range of pages that should be collapsed into one huge page but failed the allocation.
; thp_file_alloc
: is incremented every time a file huge page is successfully allocated.
; thp_file_fallback
: is incremented if a file huge page is attempted to be allocated but fails and instead falls back to using small pages.
; thp_file_fallback_charge
: is incremented if a file huge page cannot be charged and instead falls back to using small pages even though the allocation was successful.
; thp_file_mapped
: is incremented every time a file huge page is mapped into user address space.
; thp_split_page
: is incremented every time a huge page is split into base pages. This can happen for a variety of reasons but a common reason is that a huge page is old and is being reclaimed. This action implies splitting all PMD the page mapped with.
; thp_split_page_failed
: is incremented if kernel fails to split huge page. This can happen if the page was pinned by somebody.
; thp_deferred_split_page
: is incremented when a huge page is put onto split queue. This happens when a huge page is partially unmapped and splitting it would free up some memory. Pages on split queue are going to be split under memory pressure.
; thp_split_pmd
: is incremented every time a PMD split into table of PTEs. This can happen, for instance, when application calls mprotect() or munmap() on part of huge page. It doesn’t split huge page, only page table entry.
; thp_zero_page_alloc
: is incremented every time a huge zero page used for thp is successfully allocated. Note, it doesn’t count every map of the huge zero page, only its allocation.
; thp_zero_page_alloc_failed
: is incremented if kernel fails to allocate huge zero page and falls back to using small pages.
; thp_swpout
: is incremented every time a huge page is swapout in one piece without splitting.
; thp_swpout_fallback
: is incremented if a huge page has to be split before swapout. Usually because failed to allocate some continuous swap space for the huge page.
|As the system ages, allocating huge pages may be expensive as the system uses memory compaction to copy data around memory to free a huge page for use. There are some counters in <code>/proc/vmstat</code> to help monitor this overhead.


; compact_stall
Related
: is incremented every time a process stalls to run memory compaction so that a huge page is free for use.
; compact_success
: is incremented if the system compacted memory and freed a huge page for use.
; compact_fail
: is incremented if the system tries to compact memory but failed.


|-
→ [[Linux]]
|
→ [[Server]]
|
→ [[Hardware]]
|
→ [[Network]]
|
|-
|
|
|
|
|}


== Reference ==
[[Category:Configuration]]

Latest revision as of 11:30, 17 July 2026

Template:Status

Template:TOC

Overview

THP (Transparent Huge Pages)에 대한 기술 문서입니다.

Summary

  • 무엇인가? - THP (Transparent Huge Pages)
  • 왜 필요한가? - HPC 및 서버 환경에서 필수 개념
  • 언제 사용하는가? - 서버 구성, 성능 튜닝, 문제 해결 시


Purpose

이 문서가 존재하는 이유

  • Goal: THP (Transparent Huge Pages)에 대한 기술 정보 제공
  • Scope: THP (Transparent Huge Pages)의 개념, 사용법, 설정
  • Non-goals: 다른 주제로의 확장


Key Concepts

Concept Description Related
THP (Transparent Huge Pages) HPC/서버 환경에서 중요한 기술 개념 Linux, Server


Detailed Explanation

Transparent Huge Pages (THP) is an alternative solution to HugeTLB. With THP, the kernel automatically assigns huge pages to processes, so huge pages do not need to be reserved manually The THP feature has two modes of operation: system-wide and per-process. When THP is enabled system-wide, the kernel tries to assign huge pages to any process when it is possible to allocate huge pages and the process is using a large contiguous virtual memory area. If THP is enabled per-process, the kernel only assigns huge pages to individual processes' memory areas specified with the madvise() system call. In Linux system, Memory is managed in blocks known as pages. A page is 4096 bytes. 1MB of memory is equal to 256 pages; 1GB of memory is equal to 256,000 pages, etc CPUs have a built-in memory management unit that contains a list of these pages. Two ways to enable the system to manage large amounts of memory:

  1. Increase the number of page table entries in the hardware memory management unit
  2. Increase the page size   

The first method is expensive, since the hardware memory management unit in a modern processor only supports hundreds or thousands of page table entries. Additionally, hardware and memory management algorithms that work well with thousands of pages (megabytes of memory) may have difficulty performing well with millions (or even billions) of pages. This results in performance issues: when an application needs to use more memory pages than the memory management unit supports, the system falls back to slower, software-based memory management, which causes the entire system to run more slowly. it can empact to performance critical computing applications dealing with large memory working sets. The first method is expensive, since the hardware memory management unit in a modern processor only supports hundreds or thousands of page table entries. Additionally, hardware and memory management algorithms that work well with thousands of pages (megabytes of memory) may have difficulty performing well with millions (or even billions) of pages. This results in performance issues: when an application needs to use more memory pages than the memory management unit supports, the system falls back to slower, software-based memory management, which causes the entire system to run more slowly. it can empact to performance critical computing applications dealing with large memory working sets.

Type Comments on THP
Database database workloads often perform poorly with THP enabled, because they tend to have sparse rather than contiguous memory access patterns. When running MongoDB on Linux, THP should be disabled for best performance.[1]

a Linux memory management feature, often slows down database performance[2]

Oracle Transparen... [내용 계속]


Best Practices

  • 최신 버전 사용 권장
  • 공식 문서 참고
  • 테스트 환경에서 먼저 검증


References


Related Pages

Knowledge Graph

Related

→ Linux → Server → Hardware → Network

  1. ↑ https://www.mongodb.com/docs/manual/tutorial/transparent-huge-pages/
  2. ↑ https://www.pingcap.com/blog/transparent-huge-pages-why-we-disable-it-for-databases/
Retrieved from "http://wiki.hpcmate.com/index.php?title=THP_(Transparent_Huge_Pages)&oldid=3459"