Downloaded 43 times
![LivePatch example
23
static int livepatch_cmdline_proc_show(struct seq_file *m, void *v) { seq_printf(m, "%sn", "this has been live
patched");
return 0;
}
static struct klp_func funcs[] = { { .old_name = "cmdline_proc_show", .new_func = livepatch_cmdline_proc_show, }, { } };
static struct klp_object objs[] = { { /* name being NULL means vmlinux */ .funcs = funcs, }, { } };
static struct klp_patch patch = { .mod = THIS_MODULE, .objs = objs, }; static int livepatch_init(void) {
int ret;
ret = klp_register_patch(&patch); if (ret) return ret;
ret = klp_enable_patch(&patch);
if (ret) { WARN_ON(klp_unregister_patch(&patch)); return ret; } return 0;
}
static void livepatch_exit(void) { WARN_ON(klp_unregister_patch(&patch));
}
module_init(livepatch_init); module_exit(livepatch_exit);](https://image.slidesharecdn.com/linuxkernellivepatching-ruslanbilovol-180318224106/75/Linux-Kernel-Live-Patching-23-2048.jpg)
Uploaded byGlobalLogic Ukraine
Linux Kernel Live Patching
The document discusses Linux kernel live patching, its history, implementations like ksplice, kpatch, and kgraft, and the benefits of reducing downtime during hotfixes. It outlines the technical aspects of live patching, including consistency models, limitations, and recommendations for developers. Additionally, it provides details on commercial offerings and costs associated with live patching solutions from various vendors.
More Related Content
Linux Kernel Live Patching
- 1. Linux Kernel LivePatching Ruslan Bilovol
- 2. Agenda 2 • Why? • History •From Kprobes to LivePatch • LivePatch • Limitations • Current status in mainline Linux kernel • Kpatch • LivePatch writer recommendations • Demo • Questions
- 3. Why? 3 • High costof downtime caused by reboot • Apply hotfixes without reboot • Maximize uptime • Scheduled downtime (delay reboot)
- 4. Linux Kernel LivePatching: History • 2008: ksplice – university research, acquired by Oracle, source is closed • 2014: kPatch (Red Hat) – stopping the kernel for live patching – manual/automatic patches creation • 2014: kGraft (SUSE) – lazy kernel live patching – only manual patches creation • 2015: Livepatch (Linux kernel 4.0+) – combined consistency model of kGraft + kPatch 4
- 5. Commercial use 5 • OracleKsplice is available for Oracle Linux and RHEL servers with an Oracle Linux Premier Support license – $2299/node/year • RHEL Livepatching requires a Red Hat Enterprise Linux Server Premium Subscription – $1299/node/year • SUSE Livepatching is available as an add-on to SUSE Linux Enterprise Server 12 Priority Support subscription – $1,499/node/year • Canonical Livepatching service
- 6. From Kprobes toLivePatch
- 7. Kprobes 7 • Enables youto dynamically break into any kernel routine • Can trap at almost any kernel code address – you can probe even interrupt handlers • Probed instruction replaced with a breakpoint instruction • On trap executes "pre_handler", replaced instruction and "post_handler"
- 8.
- 9. Ftrace • "Kernel FunctionTracer" • Enables the kernel to trace every kernel function • Uses 'gcc -pg' option to insert “mcount” function call placed at the start of every kernel function • Big impact on performance (13%) 9
- 10. Dynamic Ftrace • "DynamicKernel Function Tracer" • mcount points to a simple return – Starting with gcc verson 4.6, the -mfentry has been added for x86, which calls " fentry " instead of "mcount" • Build-time tool parses ELF to find these location • On boot dynamic ftrace updates all the locations into NOPs 10
- 11. Dynamic Ftrace • ThenNOP sequence can be dynamically patched into a tracer call • Modules are processed as they are loaded and before they are executed • No measurable performance degradation 11
- 12. LivePatch? 12 • So wecan replace functions in running kernel? • ... • Live Kernel Patching!
- 13. LivePatch 13 • Need tobe more clever – SMP synchronization – task may sleep in a to-be-patched function – function semantic changes – data structure changes – what if patch contains multiple functions that depend on each other
- 14.
- 15. LivePatch 15 • Initially implementedfor x86-64 (kernel v4.0) – extended to PPC64 and s390 • Is a .ko module – just like any other kernel module • Uses dynamic FTRACE to replace original function • Creates /sys/kernel/livepatch dir – to monitor live patches status
- 16.
- 17. Hybrid Consistency Model 17 •Avoid calling a new function from old and vice versa • A thread calls either all old functions or all new ones • Per-thread migration – stack checking of sleeping tasks – new thread inherits its parent patched state – interrupt handler inherits interrupted thread state • Per-patched-set migration • Stack checking – need reliable stack information – frame pointers are not fast enough • Interrupt handler inherits the patched state of interrupted task
- 18.
- 19. ORC unwinder 19 • replacesGCC frame pointers approach – much simpler and faster – performance improvement (5-10% for some workloads) – 3.2% kernel .text savings • objtool for stack’s metadata validity on .o files and ORC data generation • disadvantage: needs more memory to store the ORC unwind tables • enabled by default since v4.14
- 20. Shadow variables 20 • associateadditional "shadow" data with existing data structures – allocated separately • based on global in-kernel hashtable • associates pointers to parent objects and a numeric identifier of the shadow data – there may be multiple shadow variables attached to parent object
- 21. Modules live patching 21 •patch can be inserted before module loading • LivePatch core gets notification on module load/unload – patches module before it get executing – unpatches module on module unload • module can be inserted-removed multiple times
- 22. Patch/unpatch callbacks 22 • executecallback functions when a kernel object is (un)patched • considered a "power feature" – safe updates to global data – "patches" to init and probe functions – patching otherwise unpatchable code (i.e. assembly) – ability refuse to load livepatch or a module • originally livepatch didn’t have such ability
- 23. LivePatch example 23 static intlivepatch_cmdline_proc_show(struct seq_file *m, void *v) { seq_printf(m, "%sn", "this has been live patched"); return 0; } static struct klp_func funcs[] = { { .old_name = "cmdline_proc_show", .new_func = livepatch_cmdline_proc_show, }, { } }; static struct klp_object objs[] = { { /* name being NULL means vmlinux */ .funcs = funcs, }, { } }; static struct klp_patch patch = { .mod = THIS_MODULE, .objs = objs, }; static int livepatch_init(void) { int ret; ret = klp_register_patch(&patch); if (ret) return ret; ret = klp_enable_patch(&patch); if (ret) { WARN_ON(klp_unregister_patch(&patch)); return ret; } return 0; } static void livepatch_exit(void) { WARN_ON(klp_unregister_patch(&patch)); } module_init(livepatch_init); module_exit(livepatch_exit);
- 24. LivePatch Limitations 24 • usefulfor small and critical fixes – e.g. can’t do kernel upgrade to a new version • Kprobes in the original function are ignored • can’t do semantics changes • can’t do data structure changes • inability to patch hand-written ASM – e.g. no fentry – can be hacked by patch/unpatch callbacks feature • vDSO can’t be patched (-pg is removed)
- 25. How it lookslike? 25
- 26. Current status inmainline Linux kernel Initial LivePatch support (x86-64) 26 v4.0 s390 architecture support v4.1 PPC64 architecture support v4.7 New consistency model (allows removal of livepatch modules) v4.12 ORC unwinder v4.14 Shadow variables v4.15 Patch/unpatch callbacks v4.15 Removed “immediate” feature v4.16 Atomic replace feature in progress (v4.17?) AArch64 architecture support (depends on GCC 8.x) in progress (v5.???)
- 27. kpatch Set ofTools
- 28. Why need automaticpatches generation? 28 • Human can produce errors • Inlining – inlined buggy function to other functions • GCC can do non-trivial optimization by default • Various IPA (inter-procedural optimization) – transform arguments passed by reference to arguments passed by value – create variants of a function with fewer arguments – optimize functions if values passed to them are constants • several clones of a function are possible
- 29. Kpatch 29 • Red Hatproject of live kernel patching – includes few userspace tools • Used in production (RH relies on it) • Now compatible with upstream Livepatch • Consists of 3 parts – kpatch/Livepatch kernel module stub – run-time patches management on target • systemd service to load livepatches on boot • ‘kpatch’ tool to manage patches – kpatch-build set of tools for automatic patches generation
- 30. kpatch-build 30 • build originalkernel – with -ffunction-sections -fdata-sections flags – use custom script as cross-compiler to collect .o files • build patched kernel – watch for changed objects • analyze changed objects – compare the objects at a section level – apply various rules to determine any object local sections that are dependencies of the changed section – use a lot of additional magic – create a diff of changed objects • attach changed functions to LivePatch kernel stub – finally generate .ko file
- 31. kpatch limitations 31 • can’tdo .data section changes (don’t remove static variables) • no cross-compilation support (patch is in upstream) • no out-of-src build • patching of original sources • supports only RHEL/Fedora/OracleLinux and Ubuntu/Debian – Yocto support is on the way • every patch must also be analyzed in-depth by a human • can’t generate patches with shadow variables, load/unload hooks • no out-of-tree modules support
- 32. LivePatch writer recommendations 32 •make changes as small as possible – e.g. this is “hot fix”, not a kernel upgrade to a new version • be careful with data structure changes • be careful with synchronization changes – you can create deadlock situation • watch for inlined functions • FUNC , line preprocessor macros – resulting code may be changed unexpectedly • code removal – Livepatch replaces function, not removes it • static variables – don’t remove them, keep unused (kpatch-build tracks .data changes) • https://github.com/dynup/kpatch/blob/master/doc/patch-author-guide.md
- 33.
- 34.
- 35.