Live patching technology allows updating the Linux kernel without downtime. Ksplice was an early live patching solution released in 2009 but was limited and had licensing issues. kGraft and Kpatch were later developed by SUSE and Red Hat respectively as open source live patching solutions. Both use object code comparison and replacement at runtime, but kGraft can patch without stopping processes while Kpatch uses stop_machine to ensure safe replacement. Live patching is useful for critical bugs but has limitations around data structure and common function changes.

1. Live Patching Technology
Linux Kernel 4.0
SZ LIN
szlin@cs.nctu.edu.tw
1

2. Time for Kernel 4.0
2

3. Time for Kernel 4.0
3

4. 4
Live Patching
0 down time.

5. Why you need Patch?
1 Bug Fixes
2
3
5
New Feature
Code Refactoring

6. Kernel update frequency
6

7. 7

8. Live Patching Technology
Oracle - 2009
Ksplice
SUSE - 2014
kGraft
RedHat - 2014
Kpatch
8

9. 9
kGraft & Kpatch

10. Ksplice & Kpatch
10

11. 11

12. 12
Ksplice License
GPLv2

13. 13
Ksplice License
Oracle Patents

14. 14

15. 15
Ksplice behavior
Original Kernel Patched Kernel

16. 16
Ksplice behavior
• foo method is updated
• Find foo in running kernel
• Find a safe time to insert jmp

17. 17
Ksplice behavior
Limitation of Ksplice
1. Stop services < 1ms
2. Cannot patch common structures of kernel
3. Cannot patch common function (schedule, hrtimer)

18. Ksplice behavior
Building an update
Applying an update
Actions
18
 Match pre code to running kernel
1. Discover symbol values
2. Safety check
 Call stop_machine
1. Perform “safe time” check
2. Insert jmp instruction
 Find what has been changed
 Build pre and post source code to
get object code
 Compare to find the list of changed
functions
 Tell kernel to use new object code
 Ksplice.ko
 Load new object code
 Ksplice-new.ko

19. 19
kGraft

20. kGraft
Target on…
not even for short time periods unlike
other technologies
Doesn't require stopping the kernel,
ever
1. kGraft patch can be built from C
source directly, without the need for
object code manipulation
2. Object-code based automated patch
generation is provided as an
alternative
Allows code review on kGraft
patch sources
Small amount of code thanks to
leveraging other Linux technologies,
no complex instruction decoders or
such
kGraft is lean
20

21. How does kGraft work?
1 A kGraft patch is a .ko kernel module in a KMP RPM
2 The .ko is inserted into the kernel using 'insmod' at RPM install or update time
3 kGraft replaces whole functions in the kernel
even while those functions may be executed
4 An updated kGraft RPM/module can replace an existing patch
21

22. kGraft
Limitations
1 kGraft is designed for fixing critical bugs
and thus primarily for simple changes
2 Changes in kernel data structure layout require special care
and depending on the size of the change, the change may not be possible to do without rebooting at all – same as with
other live patching tech
3 kGraft depends on a stable build environment
and thus best suited for Linux distributions, their customers or anyone who builds their own kernels, rather than 3 rd party
support companies
22

23. 23
kGraft
1 INT3/IPI-NMI self-modifying code
2 RCU-like update mechanism
3 mcount-based NOP space allocation
4 standard kernel module loading/linking mechanisms

24. kGraft
Object Code Comparison
2 kernel compilations
 objcopy
 nm
 readelf
O/S tools
24
 With & without patch
 Compiler flags
1. ffunction-sections
2. fdata-sections

25. kGraft Behavior (1/7)
25

26. kGraft Behavior (2/7)
26

27. kGraft Behavior (3/7)
27

28. kGraft Behavior (4/7)
28
RCU-like replacement

29. kGraft Behavior (5/7)
29
RCU-like replacement

30. kGraft Behavior (6/7)
30
RCU-like replacement

31. kGraft Behavior (7/7)
31

32. 32
kGraft
Toolbox
1 http://git.kernel.org/cgit/linux/kernel/git/jirislaby/kgraft.git/
2 http://github.com/useidel/kgraft-tools

33. 33
Kpatch

34. Kpatch
What’s Kpatch?
 This applies a binary patch to kernel on-line
 Patching is done without shutdown
Kpatch is a LIVE patching function for kernel
 Security and stability fixes
 Not for major kernel update
Possible to fail patching with big patch
1. Constantly used system calls
2. Data structures
Only for a small and critical issues
34

35. Kpatch
Overview
35
 Kpatch build:
 Build a binary patch module
 Kpatch.ko:
 The kernel module of Kpatch

36. Kpatch
How to Patch
36
 Kpatch uses Ftrace to patch
1. Hook the target function entry with registers
2. Change regs->ip to new function (change the flow)

37. Kpatch
Conflict of Old and New Functions
37
 Kpatch ensures the old functions are not executed when patching
 “Active Safeness Check”
 Do stack dump to check the target functions are not executed, for each thread.

38. Kpatch
Active Safeness Check With Stop_machine
38
 Kpatch uses stop_machine to check stacks

39. Kpatch
Active Safeness Check With Stop_machine
39

40. Kpatch
Active Safeness Check With Stop_machine
40

41. Kpatch
Active Safeness Check With Stop_machine
41

42. Kpatch
Stop_machine: Pros and Cons
Pros
 Stop_machine stops all processes a while
 It is critical for control/network appliances
 In virtual environment, this takes longer
time
 We need to wait all VCPUs are scheduled on the
host machine
Cons
42
 Safe, simple and easy to review,
Good for the 1st version
Stop_machine-free kpatch is in discussion stage
 push current stop_machine-based kpatch to upstream

43. 1 Human safety analysis required!
2 Not a general purpose upgrade tool
3
~80% of all CVE patches currently supported
1. Data structure changes, edge cases
2. Goal: 99%
4 stop_machine() latency: 1ms – 40ms
43
Kpatch
Limitations
Currently x86_64 only5

44. 44
Kpatch
Toolbox
1 http://github.com/dynup/kpatch/

45. 45
Conclusion

46. 46
Conclusion
1 kGraft/ Kpatch
RCU/ stop_machine
2 Implemented for x86 only as a reference architecture
powerpc, s390 and arm is already in the works
3 Only for a small and critical issues
Not for major kernel update
4 Two groups got together
Combine kpatch and kGraft in the Linux Kernel 4.0 .

47. 47
Thank you

48. 48
References
Linux Kernel
https://www.kernel.org/
LWN
https://lwn.net/
LKML.ORG - the Linux Kernel Mailing List Archive
https://lkml.org/
Kgraft official site
https://www.suse.com/promo/kgraft.html
Kpatch official site
http://rhelblog.redhat.com/2014/02/26/kpatch/
https://github.com/dynup/kpatch
Kpslice – Oracle official site
http://www.ksplice.com/
Kpatch Without Stop Machine
Masami Hiramatsu

49. 49
References
kpatch - Have your security and eat it too!
Josh Poimboeuf
Reboot adieu! Online Linux kernel patching
Udo Seidel
Oracle Ksplice for Oracle Linux
Ksplice-quickstart
Ksplice+ : Rebootless kernel updates in a distributed system
Sanjay Kulhari
KSPLICE: ZERO DOWNTIME UPDATES FOR ORACLE LINUX
ORACLE DATA SHEET
Note to Module Vendors With Respect to kGraft
SUSE SolidDriver Program
kGraft - Live patching of the Linux kernel
Vojtěch Pavlik
Ksplice: Automatic Rebootless Kernel Updates
Jeff Arnold and M. Frans Kaashoek