Skip to main content
UploadLanguage (EN)Support
BusinessMobileSocial MediaMarketingTechnologyArt & PhotosCareerDesignEducationPresentations & Public SpeakingGovernment & NonprofitHealthcareInternetLawLeadership & ManagementAutomotiveEngineeringSoftwareRecruiting & HRRetailSalesServicesScienceSmall Business & EntrepreneurshipFoodEnvironmentEconomy & FinanceData & AnalyticsInvestor RelationsSportsSpiritualNews & PoliticsTravelSelf ImprovementReal EstateEntertainment & HumorHealth & MedicineDevices & HardwareLifestyle
EC
Uploaded byEric Chou
118 views

COSCUP2025 藉由 sched_ext 實作客製化 Linux CPU 排程器.pdf

coscup 2025

Engineering◦
Downloaded 14 times
藉由 sched_ext 實作客製化 Linux CPU 排程器 講者: EricccTaiwan, charliechiou 2025/06/28 2025/08/09 COSCUP2025
周呈陽 成大電通所碩二 GitHub: EricccTaiwan LinkedIn: Eric Chou MS @ NCKUCCE 邱柏穎 成大電通所碩一 GitHub: charliechiou LinkedIn: Po-Ying Chiu MS @ NCKUCCE 2
3 ● A review of CFS and EEVDF ● Innovations and mechanisms behind sched_ext ● From custom FCFS/RR scheduler to introducing Machine Learning ● What’s Next ? Outline
4 ● A review of CFS and EEVDF ● Innovations and mechanisms behind sched_ext ● From custom FCFS/RR scheduler to introducing Machine Learning ● What’s Next ? Outline
CFS[1] and EEVDF[2] ● Both CFS (v2.6.23) and EEVDF (v6.6) are general-purpose schedulers in Linux kernel. ● CFS: Select the task with the earliest vruntime[3] , aiming to ensure fairness. ● EEVDF: Among eligible tasks (with lower vruntime than average vruntime), choose the one with the earliest virtual deadline. 5 CFS EEVDF Ref : 《Demystifying the Linux CPU Scheduler》 - by Ching-Chun (“Jserv”) Huang [1] Completely Fair Scheduler [2] Earliest Eligible Virtual Deadline First [3] virtual runtime
6 ● A review of CFS and EEVDF ● Innovations and mechanisms behind sched_ext ○ What is sched_ext ? ○ Why sched_ext ? ○ How sched_ext ? ○ sched_ext keynote ● From custom FCFS/RR scheduler to introducing Machine Learning ● What’s Next ? Outline
“ sched_ext (scx) is a Linux kernel feature which enables implementing kernel thread schedulers in eBPF and dynamically loading them. ” 7 What is sched_ext [1] ? [1] sched_ext: scheduler extension
● Sched_ext is an extensible scheduler class that allows for building of scheduling policies with eBPF. ● Implements scheduling policies as loadable eBPF programs that run in kernel context yet can be swapped at run-time. ● sched_ext was merged in Linux v6.12 in Sep 2024, which is the minimum required kernel. ○ Re: [PATCHSET v6] sched: Implement BPF extensible scheduler class ● Safety: If the eBPF program faults or violates verifier rules, the kernel automatically falls back to the default CFS/EEVDF scheduler. 8 What is sched_ext ? Ref : 《Demystifying the Linux CPU Scheduler》 - by Ching-Chun (“Jserv”) Huang High-Prio Low-Prio
● The default schedulers are tuned for best-effort throughput. ● Custom eBPF schedulers can be workload-aware. ○ e.g., scx_lavd for games (latency-critical tasks). ● The steps of customizing a CPU scheduler without sched_ext - by David Vernet [LKML, May 14, 2024] : 9 Why sched_ext ? ● With sched_ext, it’s just a “ 5 second compile job + 1 second to reload a safe BPF scheduler. ” 1. Tweak and recompile the kernel 2. Reinstall the kernel on the Steam Deck 3. Reboot the Steam Deck 4. Reload a game and let caches rewarm 5. Measure FPS $ meson compile -C $ meson install -C build
Interface 1: core kernel scheduler ⇒ scheduler class The core scheduler hands all SCHED_NORMAL events to the sched_ext class. Interface 2: sched_ext framework ⇒ eBPF scheduler The sched_ext framework forwards those events to the eBPF scheduler’s callback functions. Interface 3: eBPF scheduler ⇒ sched_ext framework The eBPF scheduler uses helpers to manage DiSpatch Queues (DSQs), enqueue or dequeue tasks, and kick CPUs. Interface 4: eBPF scheduler ⟺ user-space counterpart A user-space program exchanges metrics and settings with the scheduler via eBPF maps and ring buffers. 10 Ref : sched_ext: scheduler architecture and interfaces (Part 2) by Changwoo Min How sched_ext ?
“I'm also not a believer in the argument that has been used (multiple times) that the BPF scheduler would keep people from participating in scheduler development. I personally think the main thing that keeps people from participating is too high barriers to participation.“ — Linus Torvalds, [LKML, June 24, 2024] Benefits: ● Flexibility: ○ Customizable schedulers in user space. ● Agility: ○ Faster iteration and development. ● Accessibility: ○ Easier participation in scheduler innovation. 11 sched_ext keynote
12 ● A review of CFS and EEVDF ● Innovations and mechanisms behind sched_ext ● From custom FCFS/RR scheduler to introducing Machine Learning ○ FCFS/RR scheduler ○ ML-based load prediction and adaptation ● What’s Next ? Outline
13 scx_rustland_core
14 scx_rustland demo
15 10_000_000 20_000_000 30_000_000 Perfetto : https://ui.perfetto.dev/ FCFS/RR scheduler Sched_ext Perfetto With sched_ext, we can assign time slice in user space.
16 dispatched_task.slice_ns = u64::MAX; ● Starvation ! ● FCFS does not imply an infinite time slice scx_rlfifo: Clarify Round-Robin scheduling #1774 FCFS/RR scheduler
ML-based load prediction and adaptation 17 Are we learning yet ? *scx_rusty *Candle Burn or ? or scx_lavd Rust eBPF ● Main problems: ○ Scheduler ? ➡ scx_rusty ○ ML Framework ? ➡ Candle ○ Topology ? ➡ Level 2 Cache
ML-based load prediction and adaptation 18 Env. 1 : Ubuntu 25.04 (GNU/Linux 6.14.0-22-generic, x86_64) Early return :( ● Origial scx_rusty balance between NUMA node and Last Level Cache (LLC) ● Whether it would be useful to define domains in other terms is another issue scx_rusty: Domain detect didn't work #2214
ML-based load prediction and adaptation 19 Last Level Cache-based Balancing Level 2 Cache-based Balancing Env. 2: Ubuntu 25.04 (GNU/Linux 6.14.0-15-generic, arm) Load balancing performs poorly on domains closer to the core. Lower ! Higher !
ML-based load prediction and adaptation 20 Filter Pick tests Check Collect data here! Apply ML here! Sufficient ? $ stress-ng --cpu 30 -l 100 --timeout 120s --cpu-method matrixprod Task Selection Filter out ML Training Inferencing ML prevents selecting suboptimal tasks
ML-based load prediction and adaptation 21 Kernel compilation EEVDF scx_rusty L3 cache bal* L2 cache bal* L2 cache ML bal* ( Ours ) CPU 1417 % 1315 % 1070 % 1400 % time 1:36.24 1:42.99 2:01.38 1:37.07 Migrate times 55,873 217,361 428,263 457,935 $ sudo perf stat -e sched:sched_migrate_task wake lat 99.0th 2,988 1670 1,086 2,420 request lat 99.0th 14,032 10736 12,432 9,232 RPS 50.0th 1,870 3644 2,828 3,684 $ schbench -m 4 -t 4 -r 10 ● Migrate times increased from EEVDF (55,873) to scx_rusty (217,361) ● ML improves performance under L2 cache-based balancing ● Migrate times increased from L3 cache balancing (217,361) to L2 cache balancing (428,263) ● Request latency improved significantly Bold : best, Bold + underline : second-best, *bal: balance.
22 ● A review of CFS and EEVDF ● Innovations and mechanisms behind sched_ext ● From custom FCFS/RR scheduler to introducing Machine Learning ● What’s next? Outline
Upstream our scheduler: 1. Keep contributing — to both the kernel and upstream GitHub! 2. Improving ML-based scheduler! (e.g., collecting data from other schedulers, changing the workload, modifying load balance mechanism) 3. Retargeting upstream schedulers! (e.g., scx_rusty) 23 What’s Next ? PR: EricccTaiwan , charliechiou
Retargeting scx_rusty ● Most upstream sched_ext schedulers still require implementing both the eBPF and user-space parts — meaning you have to write code in both eBPF and Rust. This creates a high barrier for contributors, especially those unfamiliar with kernel eBPF development. ● With scx_rustland_core, the common eBPF logic is already implemented, so developers can focus entirely on writing Rust in user space without needing to touch eBPF. This greatly lowers the entry barrier and makes scheduler development more approachable. 24 Ref : Crafting a Linux kernel scheduler in Rust - Andrea Righi
Retargeting scx_rusty ● Our idea is to "retarget" an existing upstream scheduler — for example, scx_rusty — and reimplement it using scx_rustland_core as the base. Since scx_rusty already implements all of its load balancing logic in user space, replacing its BPF part with scx_rustland_core would be both feasible and meaningful. ● Andrea (scx maintainer) also mentioned that this aligns well with the current direction of the project: pushing for composable schedulers and scheduler libraries — with the goal of reusing as much code as possible across different schedulers. ● In the long run, the vision is to have a set of reusable building blocks that can be combined and assembled to form schedulers — like putting together pieces of a puzzle. ● So in addition to being an interesting technical project, we think this retargeting effort could make a valuable contribution to the upstream ecosystem — and it might also be a great topic for Linux Plumber Conference (LPC). 25
Extra: First Patch to Linux !! ● Typos ! (my first patch) ● kernel test robot : https://lore.kernel.org/oe-kbuild-all/?q=kernel+test+robot ● Refer to Jserv’s teaching slides / practice questions ! ● Ref: ○ https://hackmd.io/@eleanorLin/B1tNTBD3C ○ https://hackmd.io/@yanjiew/linux2023q1-1st_contrib ○ https://hackmd.io/@Max042004/first-linux-patch 26
Notes & Talks ● cce-underdogs/Talks Special Thanks: ● Ching-Chun (“Jserv”) Huang : Leading us into the world of Linux. ● Sched_ext community : Kindly answering our questions and reviewing our PRs. ● Chia-Ping (“chia7712”) Tsai : Introducing us to the “opensource4you” community. Thanks For Listening ! 27