Kernel tips and tricks: Difference between revisions

From HPCWIKI
Jump to navigation Jump to search
(Created page with " == perf interrupt took too long in system log == phenomenon system log shows "perf interrupt took too long (aaa > bbb), lowering kernel.perf_event_max_sample_rate to ccc" Action Nothing to worry about. It has to do with the Linux perf tool which is included in the kernel. The kernel automagically determines the sample rate that could be used without impacting system performance too much; and it logs this even when perf isn't active, or even installed. Messages like...")
 
(Fix: remove --- horizontal lines (7 removed))
 
(14 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{Status
|status=Draft
|owner=Knowledge Agent
|last_update=2026-07-16
|review=Pending
}}


== perf interrupt took too long in system log ==
{{TOC}}
phenomenon


system log  shows "perf interrupt took too long (aaa > bbb), lowering kernel.perf_event_max_sample_rate to ccc"
== Overview ==


Kernel tips and tricks에 대한 기술 문서입니다.


Action
=== Summary ===
 
* 무엇인가? - Kernel tips and tricks
* 왜 필요한가? - HPC 및 서버 환경에서 필수 개념
* 언제 사용하는가? - 서버 구성, 성능 튜닝, 문제 해결 시
 
 
== Purpose ==
 
이 문서가 존재하는 이유
 
* Goal: Kernel tips and tricks에 대한 기술 정보 제공
* Scope: Kernel tips and tricks의 개념, 사용법, 설정
* Non-goals: 다른 주제로의 확장
 
 
== Key Concepts ==
 
{| class="wikitable"
! Concept
! Description
! Related
|-
| Kernel tips and tricks
| HPC/서버 환경에서 중요한 기술 개념
| [[Linux]], [[Server]]
|}
 
 
== Detailed Explanation ==
 
* [[Kernel]]
In the ARM world from ARMv4 to ARMv7 floating-point [[support]] is called VFP, To determine whether you have appropriate floating point support, look for <code>vfpv3</code> in the “Features” line in <code>/proc/cpuinfo</code>.
# Identify the specific hardware error source using mcelog or the APEI kernel log (/var/log/kern.log.x)
# Update System Firmware - Manufacturers often release firmware updates to address hardware error-related issues. Applying the latest firmware ([[BIOS]]/UEFI/[[device firmware]]) can resolve known APEI generic hardware error problems
# Kernel Parameter Tuning - although this is not recommended as it may compromise system stability. “apei=off” kernel parameter can be used to disable APEI error handling altogether
# Hardware Diagnostics - memtest86 for memory testing or CPU stress tests for CPU-related issues
Symptoms
The "RT throttling activated" is a message from the operating system scheduler has identified some Real-Time threads that are hogging the CPU and starving other threads. The operating system does this in an attempt to throttle those real-time tasks and keep the operating system from being unresponsive.<ref>https://www.dell.com/support/kbdoc/ko-kr/000167765/scaleio-resource-contention-troubleshooting</ref>
From the kernel point of view, when a RT thread occupied the cpu by 950ms/1s (defined by /proc/sys/kernel/sched_rt_period_us and /proc/sys/kernel/sched_rt_runtime_us), whether the RT thread maybe is a business thread or other unknown thread. Current Linux kernel only outputs the print "sched: RT throttling activated" when RT throttling happen and it is hard to know what is the RT thread.
Simply, a normal thread can't get cpu, And at this moment, Kernel prints 'sched: RT throttling activated' log
[[Troubleshooting]]
# Linux patch to print more infor : For further analysis [https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwjh7-_H2qOAAxUxmFYBHT4yBDkQFnoECBcQAQ&url=https%3A%2F%2Flore.kernel.org%2Fall%2Ff3265adc26d4416dacf157f61fa60ad6%40h3c.com%2FT%2F&usg=AOvVaw0VhhVSVlLfR-OXnGELirNo&opi=89978449 A Linux Kernel patch] is available to print current RT task when RT throttling activated that help us to know what is the RT thread in the first time.
# To reach a only 50% CPU usage for real-time tasks and a larger period the values can be changed with the following commands<ref>https://wiki.linuxfoundation.org/realtime/documentation/technical_basics/sched_rt_throttling</ref><syntaxhighlight lang="bash">
# echo 2000000 > /proc/sys/kernel/sched_rt_period_us
# echo 1000000 > /proc/sys/kernel/sched_rt_runtime_us
</syntaxhighlight>
# Real-time throttling is disabled in case the real-time task runtime has the same length than the the period. This is done automatically by writing `-1` into `sched_rt_runtime_us`:<syntaxhighlight lang="bash">
# echo -1 > /proc/sys/kernel/sched_rt_runtime_us
</syntaxhighlight>
This mechanism is already implemented in mainline Linux.
 
 
== Best Practices ==
 
* 최신 버전 사용 권장
* 공식 문서 참고
* 테스트 환경에서 먼저 검증


Nothing to worry about. It has to do with the [[Linux]] perf tool which is included in the kernel. The kernel automagically determines the sample rate that could be used without impacting system performance too much; and it logs this even when perf isn't active, or even installed. Messages like this are triggered by high(er) system load or a cpu that is scaling<ref>https://bbs.archlinux.org/viewtopic.php?id=187636</ref>


== References ==
== References ==
<references />
 
* [https://wiki.hpcmate.com Kernel tips and tricks]
 
 
== Related Pages ==
 
* [[Linux]]
* [[Server]]
* [[Hardware]]
* [[Network]]
 
 
[[Category:Linux]]
== Knowledge Graph ==
 
Related
 
→ [[Linux Kernel Tuning]]
→ [[THP (Transparent Huge Pages)]]
→ [[Kernel boot param]]
→ [[Linux Kernel]]
→ [[Tunable kernel params]]
 
[[Category:Guide]]

Latest revision as of 11:29, 17 July 2026

Template:Status

Template:TOC

Overview

Kernel tips and tricks에 대한 기술 문서입니다.

Summary

  • 무엇인가? - Kernel tips and tricks
  • 왜 필요한가? - HPC 및 서버 환경에서 필수 개념
  • 언제 사용하는가? - 서버 구성, 성능 튜닝, 문제 해결 시


Purpose

이 문서가 존재하는 이유

  • Goal: Kernel tips and tricks에 대한 기술 정보 제공
  • Scope: Kernel tips and tricks의 개념, 사용법, 설정
  • Non-goals: 다른 주제로의 확장


Key Concepts

Concept Description Related
Kernel tips and tricks HPC/서버 환경에서 중요한 기술 개념 Linux, Server


Detailed Explanation

In the ARM world from ARMv4 to ARMv7 floating-point support is called VFP, To determine whether you have appropriate floating point support, look for vfpv3 in the “Features” line in /proc/cpuinfo.

  1. Identify the specific hardware error source using mcelog or the APEI kernel log (/var/log/kern.log.x)
  2. Update System Firmware - Manufacturers often release firmware updates to address hardware error-related issues. Applying the latest firmware (BIOS/UEFI/device firmware) can resolve known APEI generic hardware error problems
  3. Kernel Parameter Tuning - although this is not recommended as it may compromise system stability. “apei=off” kernel parameter can be used to disable APEI error handling altogether
  4. Hardware Diagnostics - memtest86 for memory testing or CPU stress tests for CPU-related issues

Symptoms The "RT throttling activated" is a message from the operating system scheduler has identified some Real-Time threads that are hogging the CPU and starving other threads. The operating system does this in an attempt to throttle those real-time tasks and keep the operating system from being unresponsive.[1] From the kernel point of view, when a RT thread occupied the cpu by 950ms/1s (defined by /proc/sys/kernel/sched_rt_period_us and /proc/sys/kernel/sched_rt_runtime_us), whether the RT thread maybe is a business thread or other unknown thread. Current Linux kernel only outputs the print "sched: RT throttling activated" when RT throttling happen and it is hard to know what is the RT thread. Simply, a normal thread can't get cpu, And at this moment, Kernel prints 'sched: RT throttling activated' log Troubleshooting

  1. Linux patch to print more infor : For further analysis A Linux Kernel patch is available to print current RT task when RT throttling activated that help us to know what is the RT thread in the first time.
  2. To reach a only 50% CPU usage for real-time tasks and a larger period the values can be changed with the following commands[2]
    # echo 2000000 > /proc/sys/kernel/sched_rt_period_us 
# echo 1000000 > /proc/sys/kernel/sched_rt_runtime_us
  3. Real-time throttling is disabled in case the real-time task runtime has the same length than the the period. This is done automatically by writing `-1` into `sched_rt_runtime_us`:
    # echo -1 > /proc/sys/kernel/sched_rt_runtime_us

This mechanism is already implemented in mainline Linux.


Best Practices

  • 최신 버전 사용 권장
  • 공식 문서 참고
  • 테스트 환경에서 먼저 검증


References


Related Pages

Knowledge Graph

Related

→ Linux Kernel Tuning → THP (Transparent Huge Pages) → Kernel boot param → Linux Kernel → Tunable kernel params

  1. ↑ https://www.dell.com/support/kbdoc/ko-kr/000167765/scaleio-resource-contention-troubleshooting
  2. ↑ https://wiki.linuxfoundation.org/realtime/documentation/technical_basics/sched_rt_throttling
Retrieved from "http://wiki.hpcmate.com/index.php?title=Kernel_tips_and_tricks&oldid=3358"