16. 16
Ksplice behavior
• foo method is updated
• Find foo in running kernel
• Find a safe time to insert jmp
17. 17
Ksplice behavior
Limitation of Ksplice
1. Stop services < 1ms
2. Cannot patch common structures of kernel
3. Cannot patch common function (schedule, hrtimer)
18. Ksplice behavior
Building an update
Applying an update
Actions
18
Match pre code to running kernel
1. Discover symbol values
2. Safety check
Call stop_machine
1. Perform “safe time” check
2. Insert jmp instruction
Find what has been changed
Build pre and post source code to
get object code
Compare to find the list of changed
functions
Tell kernel to use new object code
Ksplice.ko
Load new object code
Ksplice-new.ko
20. kGraft
Target on…
not even for short time periods unlike
other technologies
Doesn't require stopping the kernel,
ever
1. kGraft patch can be built from C
source directly, without the need for
object code manipulation
2. Object-code based automated patch
generation is provided as an
alternative
Allows code review on kGraft
patch sources
Small amount of code thanks to
leveraging other Linux technologies,
no complex instruction decoders or
such
kGraft is lean
20
21. How does kGraft work?
1 A kGraft patch is a .ko kernel module in a KMP RPM
2 The .ko is inserted into the kernel using 'insmod' at RPM install or update time
3 kGraft replaces whole functions in the kernel
even while those functions may be executed
4 An updated kGraft RPM/module can replace an existing patch
21
22. kGraft
Limitations
1 kGraft is designed for fixing critical bugs
and thus primarily for simple changes
2 Changes in kernel data structure layout require special care
and depending on the size of the change, the change may not be possible to do without rebooting at all – same as with
other live patching tech
3 kGraft depends on a stable build environment
and thus best suited for Linux distributions, their customers or anyone who builds their own kernels, rather than 3 rd party
support companies
22
34. Kpatch
What’s Kpatch?
This applies a binary patch to kernel on-line
Patching is done without shutdown
Kpatch is a LIVE patching function for kernel
Security and stability fixes
Not for major kernel update
Possible to fail patching with big patch
1. Constantly used system calls
2. Data structures
Only for a small and critical issues
34
36. Kpatch
How to Patch
36
Kpatch uses Ftrace to patch
1. Hook the target function entry with registers
2. Change regs->ip to new function (change the flow)
37. Kpatch
Conflict of Old and New Functions
37
Kpatch ensures the old functions are not executed when patching
“Active Safeness Check”
Do stack dump to check the target functions are not executed, for each thread.
42. Kpatch
Stop_machine: Pros and Cons
Pros
Stop_machine stops all processes a while
It is critical for control/network appliances
In virtual environment, this takes longer
time
We need to wait all VCPUs are scheduled on the
host machine
Cons
42
Safe, simple and easy to review,
Good for the 1st version
Stop_machine-free kpatch is in discussion stage
push current stop_machine-based kpatch to upstream
43. 1 Human safety analysis required!
2 Not a general purpose upgrade tool
3
~80% of all CVE patches currently supported
1. Data structure changes, edge cases
2. Goal: 99%
4 stop_machine() latency: 1ms – 40ms
43
Kpatch
Limitations
Currently x86_64 only5
46. 46
Conclusion
1 kGraft/ Kpatch
RCU/ stop_machine
2 Implemented for x86 only as a reference architecture
powerpc, s390 and arm is already in the works
3 Only for a small and critical issues
Not for major kernel update
4 Two groups got together
Combine kpatch and kGraft in the Linux Kernel 4.0 .
48. 48
References
Linux Kernel
https://www.kernel.org/
LWN
https://lwn.net/
LKML.ORG - the Linux Kernel Mailing List Archive
https://lkml.org/
Kgraft official site
https://www.suse.com/promo/kgraft.html
Kpatch official site
http://rhelblog.redhat.com/2014/02/26/kpatch/
https://github.com/dynup/kpatch
Kpslice – Oracle official site
http://www.ksplice.com/
Kpatch Without Stop Machine
Masami Hiramatsu
49. 49
References
kpatch - Have your security and eat it too!
Josh Poimboeuf
Reboot adieu! Online Linux kernel patching
Udo Seidel
Oracle Ksplice for Oracle Linux
Ksplice-quickstart
Ksplice+ : Rebootless kernel updates in a distributed system
Sanjay Kulhari
KSPLICE: ZERO DOWNTIME UPDATES FOR ORACLE LINUX
ORACLE DATA SHEET
Note to Module Vendors With Respect to kGraft
SUSE SolidDriver Program
kGraft - Live patching of the Linux kernel
Vojtěch Pavlik
Ksplice: Automatic Rebootless Kernel Updates
Jeff Arnold and M. Frans Kaashoek
