Kernel tips and tricks: Difference between revisions

From HPCWIKI
Jump to navigation Jump to search
No edit summary
(Fix: remove --- horizontal lines (7 removed))
 
(10 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== sched: RT throttling activated ==
{{Status
Symptoms
|status=Draft
|owner=Knowledge Agent
|last_update=2026-07-16
|review=Pending
}}
 
{{TOC}}
 
== Overview ==
 
Kernel tips and tricks에 대한 기술 문서입니다.
 
=== Summary ===
 
* 무엇인가? - Kernel tips and tricks
* 왜 필요한가? - HPC 및 서버 환경에서 필수 개념
* 언제 사용하는가? - 서버 구성, 성능 튜닝, 문제 해결 시
 
 
== Purpose ==
 
이 문서가 존재하는 이유


The "RT throttling activated" is a message from the operating system scheduler has identified some Real-Time threads that are hogging the CPU and starving other threads. The operating system does this in an attempt to throttle those real-time tasks and keep the operating system from being unresponsive.
* Goal: Kernel tips and tricks에 대한 기술 정보 제공
* Scope: Kernel tips and tricks의 개념, 사용법, 설정
* Non-goals: 다른 주제로의 확장




From the kernel point of view, when a RT thread occupied the cpu by 950ms/1s (defined by /proc/sys/kernel/sched_rt_period_us and /proc/sys/kernel/sched_rt_runtime_us), whether the RT thread maybe is a business thread or other unknown thread. Current Linux kernel only outputs the print "sched: RT throttling activated" when RT throttling happen and it is hard to know what is the RT thread.
== Key Concepts ==


Simply, a normal thread can't get cpu, And at this moment, Kernel prints 'sched: RT throttling activated' log
{| class="wikitable"
! Concept
! Description
! Related
|-
| Kernel tips and tricks
| HPC/서버 환경에서 중요한 기술 개념
| [[Linux]], [[Server]]
|}




== Detailed Explanation ==


* [[Kernel]]
In the ARM world from ARMv4 to ARMv7 floating-point [[support]] is called VFP, To determine whether you have appropriate floating point support, look for <code>vfpv3</code> in the “Features” line in <code>/proc/cpuinfo</code>.
# Identify the specific hardware error source using mcelog or the APEI kernel log (/var/log/kern.log.x)
# Update System Firmware - Manufacturers often release firmware updates to address hardware error-related issues. Applying the latest firmware ([[BIOS]]/UEFI/[[device firmware]]) can resolve known APEI generic hardware error problems
# Kernel Parameter Tuning - although this is not recommended as it may compromise system stability. “apei=off” kernel parameter can be used to disable APEI error handling altogether
# Hardware Diagnostics - memtest86 for memory testing or CPU stress tests for CPU-related issues
Symptoms
The "RT throttling activated" is a message from the operating system scheduler has identified some Real-Time threads that are hogging the CPU and starving other threads. The operating system does this in an attempt to throttle those real-time tasks and keep the operating system from being unresponsive.<ref>https://www.dell.com/support/kbdoc/ko-kr/000167765/scaleio-resource-contention-troubleshooting</ref>
From the kernel point of view, when a RT thread occupied the cpu by 950ms/1s (defined by /proc/sys/kernel/sched_rt_period_us and /proc/sys/kernel/sched_rt_runtime_us), whether the RT thread maybe is a business thread or other unknown thread. Current Linux kernel only outputs the print "sched: RT throttling activated" when RT throttling happen and it is hard to know what is the RT thread.
Simply, a normal thread can't get cpu, And at this moment, Kernel prints 'sched: RT throttling activated' log
[[Troubleshooting]]
[[Troubleshooting]]
# Linux patch to print more infor : For further analysis [https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwjh7-_H2qOAAxUxmFYBHT4yBDkQFnoECBcQAQ&url=https%3A%2F%2Flore.kernel.org%2Fall%2Ff3265adc26d4416dacf157f61fa60ad6%40h3c.com%2FT%2F&usg=AOvVaw0VhhVSVlLfR-OXnGELirNo&opi=89978449 A Linux Kernel patch] is available to print current RT task when RT throttling activated that help us to know what is the RT thread in the first time.
# Linux patch to print more infor : For further analysis [https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwjh7-_H2qOAAxUxmFYBHT4yBDkQFnoECBcQAQ&url=https%3A%2F%2Flore.kernel.org%2Fall%2Ff3265adc26d4416dacf157f61fa60ad6%40h3c.com%2FT%2F&usg=AOvVaw0VhhVSVlLfR-OXnGELirNo&opi=89978449 A Linux Kernel patch] is available to print current RT task when RT throttling activated that help us to know what is the RT thread in the first time.
Line 20: Line 62:
# echo -1 > /proc/sys/kernel/sched_rt_runtime_us
# echo -1 > /proc/sys/kernel/sched_rt_runtime_us
</syntaxhighlight>
</syntaxhighlight>
This mechanism is already implemented in mainline Linux.  
This mechanism is already implemented in mainline Linux.


== perf interrupt took too long in system log ==
Symptoms


The linux kernel gathers samples using ‘perf’ performance monitor without affecting the latencies. These include getting interrupt times. If interrupts take too long, a similar message to this prints:
== Best Practices ==
 
* 최신 버전 사용 권장
* 공식 문서 참고
* 테스트 환경에서 먼저 검증


system log  shows "perf interrupt took too long (aaa > bbb), lowering kernel.perf_event_max_sample_rate to ccc"


This essentially means that the machine was stuck on an interrupt for a long amount of time. This can be caused by a number of reasons<ref>https://discuss.aerospike.com/t/what-does-interrupt-took-too-long-mean/5818</ref>, including:
== References ==


# DISK IO interrupt taking long would be caused by a faulty, slow or overloaded disk. Alternatively this can be caused by an issue with a disk or raid controller.
* [https://wiki.hpcmate.com Kernel tips and tricks]
# Network IO interrupt taking too long would be caused most often by network driver issues being suboptimal. Alternatively, this can be caused by network issues, although the protocol switching should theoretically be preventing it.


Troubleshooting


The disk IO can be easily checked with disk IO stats (sysstat-sar and/or iostat) and confirmed. If the disk IO is not the reason for slow interrupts, the network IO will be. For this, the problem needs to be checked on the network and/or kernel side.
== Related Pages ==


* [[Linux]]
* [[Server]]
* [[Hardware]]
* [[Network]]


If there is no issue with the kernel drivers, the network would be most liekly at fault, most likely first hop. This then needs to be checked on the network side


[[Category:Linux]]
== Knowledge Graph ==


Impact
Related


This should not be a concern. perf is a tool to handle CPU performance. It has to do with the [[Linux]] perf tool which is included in the kernel. The kernel automagically determines the sample rate that could be used without impacting system performance too much; and it logs this even when perf isn't active, or even installed. Messages like this are triggered by high(er) system load or a cpu that is scaling<ref>https://bbs.archlinux.org/viewtopic.php?id=187636</ref>
→ [[Linux Kernel Tuning]]
→ [[THP (Transparent Huge Pages)]]
→ [[Kernel boot param]]
→ [[Linux Kernel]]
→ [[Tunable kernel params]]


== References ==
[[Category:Guide]]
<references />

Latest revision as of 11:29, 17 July 2026

Template:Status

Template:TOC

Overview

Kernel tips and tricks에 대한 기술 문서입니다.

Summary

  • 무엇인가? - Kernel tips and tricks
  • 왜 필요한가? - HPC 및 서버 환경에서 필수 개념
  • 언제 사용하는가? - 서버 구성, 성능 튜닝, 문제 해결 시


Purpose

이 문서가 존재하는 이유

  • Goal: Kernel tips and tricks에 대한 기술 정보 제공
  • Scope: Kernel tips and tricks의 개념, 사용법, 설정
  • Non-goals: 다른 주제로의 확장


Key Concepts

Concept Description Related
Kernel tips and tricks HPC/서버 환경에서 중요한 기술 개념 Linux, Server


Detailed Explanation

In the ARM world from ARMv4 to ARMv7 floating-point support is called VFP, To determine whether you have appropriate floating point support, look for vfpv3 in the “Features” line in /proc/cpuinfo.

  1. Identify the specific hardware error source using mcelog or the APEI kernel log (/var/log/kern.log.x)
  2. Update System Firmware - Manufacturers often release firmware updates to address hardware error-related issues. Applying the latest firmware (BIOS/UEFI/device firmware) can resolve known APEI generic hardware error problems
  3. Kernel Parameter Tuning - although this is not recommended as it may compromise system stability. “apei=off” kernel parameter can be used to disable APEI error handling altogether
  4. Hardware Diagnostics - memtest86 for memory testing or CPU stress tests for CPU-related issues

Symptoms The "RT throttling activated" is a message from the operating system scheduler has identified some Real-Time threads that are hogging the CPU and starving other threads. The operating system does this in an attempt to throttle those real-time tasks and keep the operating system from being unresponsive.[1] From the kernel point of view, when a RT thread occupied the cpu by 950ms/1s (defined by /proc/sys/kernel/sched_rt_period_us and /proc/sys/kernel/sched_rt_runtime_us), whether the RT thread maybe is a business thread or other unknown thread. Current Linux kernel only outputs the print "sched: RT throttling activated" when RT throttling happen and it is hard to know what is the RT thread. Simply, a normal thread can't get cpu, And at this moment, Kernel prints 'sched: RT throttling activated' log Troubleshooting

  1. Linux patch to print more infor : For further analysis A Linux Kernel patch is available to print current RT task when RT throttling activated that help us to know what is the RT thread in the first time.
  2. To reach a only 50% CPU usage for real-time tasks and a larger period the values can be changed with the following commands[2]
    # echo 2000000 > /proc/sys/kernel/sched_rt_period_us 
# echo 1000000 > /proc/sys/kernel/sched_rt_runtime_us
  3. Real-time throttling is disabled in case the real-time task runtime has the same length than the the period. This is done automatically by writing `-1` into `sched_rt_runtime_us`:
    # echo -1 > /proc/sys/kernel/sched_rt_runtime_us

This mechanism is already implemented in mainline Linux.


Best Practices

  • 최신 버전 사용 권장
  • 공식 문서 참고
  • 테스트 환경에서 먼저 검증


References


Related Pages

Knowledge Graph

Related

→ Linux Kernel Tuning → THP (Transparent Huge Pages) → Kernel boot param → Linux Kernel → Tunable kernel params

  1. ↑ https://www.dell.com/support/kbdoc/ko-kr/000167765/scaleio-resource-contention-troubleshooting
  2. ↑ https://wiki.linuxfoundation.org/realtime/documentation/technical_basics/sched_rt_throttling
Retrieved from "http://wiki.hpcmate.com/index.php?title=Kernel_tips_and_tricks&oldid=3358"