Kernel Recipes 2019 - ftrace: Where modifying a running kernel all ...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

©2019 VMware, Inc.
ftrace
Where modifying a running
kernel all started!
Steven Rostedt
Open Source Engineer
rostedt@goodmi...

2©2019 VMware, Inc.
Ftrace Function hooks
●
Allows attaching to a function in the kernel
– Function Tracer
– Function Grap...

3©2019 VMware, Inc.
Function Tracing
# cd /sys/kernel/tracing
# echo function > current_tracer
# cat trace
# tracer: funct...

4©2019 VMware, Inc.
Function Graph Tracing
# cd /sys/kernel/tracing
# echo function_graph > current_tracer
# cat trace
# t...

5©2019 VMware, Inc.
Dynamic Function Tracing
# cd /sys/kernel/tracing
# echo ‘*sched*’ > set_ftrace_filter
# echo function...

6©2019 VMware, Inc.
How does it work?
●
gcc’s profiler option: -pg
– Adds a special “mcount” call to all non-inlined funct...

7©2019 VMware, Inc.
How does it work?
●
gcc’s profiler option: -pg
– Adds a special “mcount” call to all non-inlined funct...

$8©2019 VMware, Inc. A Function Call asmlinkage __visible void __sched schedule(void) { struct task_struct *tsk = current; ...$

9©2019 VMware, Inc.
WARNING!
The following slides
may not be suitable for
some audiences

10©2019 VMware, Inc.
WARNING!
The next slide contains
ASSEMBLY!

11©2019 VMware, Inc.
Disassembled Function Call
<schedule>:
53 push %rbx
65 48 8b 1c 25 00 61 mov %gs:0x16100,%rbx
01 00
f...

12©2019 VMware, Inc.
Disassembled Function Call
<schedule>:
e8 1b d0 1e 00 callq ffffffff81c01930 <__fentry__>
ffffffff81a...

13©2019 VMware, Inc.
At Kernel Boot Up
<schedule>:
callq <__fentry__>
push %rbx <__fentry__>:
retq

14©2019 VMware, Inc.
Where are all the __fentry__ callers?
Can’t just leave them there

Too much overhead

Just calling ...

15©2019 VMware, Inc.
recordmcount
scripts/recordmcount.c (and there’s a perl version too!)
Reads the object files one at a...

16©2019 VMware, Inc.
recordmcount
scripts/recordmcount.c (and there’s a perl version too!)
Reads the object files one at a...

17©2019 VMware, Inc.
recordmcount (kernel/sched/core.o)
<__mcount_loc>:
&schedule
&yield
&preempt_schedule_common
&_cond_r...

18©2019 VMware, Inc.
recordmcount (kernel/sched/core.o)
<__mcount_loc>:
&schedule
&yield
&preempt_schedule_common
&_cond_r...

19©2019 VMware, Inc.
Linker Magic!
vmlinux.lds

include/asm-generic/vmlinux.lds.h
Magic Variables

__start_mcount_loc

...

20©2019 VMware, Inc.
Linker Magic!
vmlinux.lds

include/asm-generic/vmlinux.lds.h
Magic Variables

__start_mcount_loc

...

21©2019 VMware, Inc.
Linker Magic
vmlinux:
<__mcount_loc>:
&schedule
&yield
&preempt_schedule_common
&_cond_resched
&sched...

22©2019 VMware, Inc.
Linker Magic
<__start_mcount_loc>:
&schedule
&yield
&preempt_schedule_common
&_cond_resched
&schedule...

23©2019 VMware, Inc.
Linker Magic
<__start_mcount_loc>:
0xffffffff81a14910
0xffffffff81a149b0
0xffffffff81a14c00
0xfffffff...

24©2019 VMware, Inc.
Finding __fentry__
<schedule>:
callq <__fentry__>
[..]
<yield>:
callq <__fentry__>
[..]
<preempt_sche...

25©2019 VMware, Inc.
<schedule>:
callq <__fentry__>
[..]
<yield>:
callq <__fentry__>
[..]
<preempt_schedule_common>:
callq...

26©2019 VMware, Inc.
Finding __fentry__
<schedule>:
nop
[..]
<yield>:
nop
[..]
<preempt_schedule_common>:
nop
[..]
<_cond_...

27©2019 VMware, Inc.
Finding __fentry__
<schedule>:
nop
[..]
<yield>:
nop
[..]
<preempt_schedule_common>:
nop
[..]
<_cond_...

28©2019 VMware, Inc.
Finding __fentry__
vmlinux:
<schedule>:
nop
[..]
<yield>:
nop
[..]
<preempt_schedule_common>:
nop
[.....

29©2019 VMware, Inc.
Finding __fentry__
vmlinux:
<schedule>:
nop
[..]
<yield>:
nop
[..]
<preempt_schedule_common>:
nop
[.....

30©2019 VMware, Inc.
What about Tracing?
Need to know where to enable tracing
We threw away the __mcount_loc section

31©2019 VMware, Inc.
What about Tracing?
Need to know where to enable tracing
We threw away the __mcount_loc section

The...

$32©2019 VMware, Inc. struct dyn_ftrace struct dyn_ftrace { unsigned long ip; /* address of mcount call-site */ unsigned lo...$

$33©2019 VMware, Inc. struct dyn_ftrace struct dyn_ftrace { unsigned long ip; /* address of mcount call-site */ unsigned lo...$

$34©2019 VMware, Inc. struct dyn_ftrace struct dyn_ftrace { unsigned long ip; /* address of mcount call-site */ unsigned lo...$

35©2019 VMware, Inc.
Tracing data
Copy from __mcount_loc before deleting that section
Sorted for quick lookup
Allocated in...

36©2019 VMware, Inc.
Finding __fentry__
<schedule>:
nop
[..]
<yield>:
nop
[..]
<preempt_schedule_common>:
nop
[..]
<_cond_...

37©2019 VMware, Inc.
Finding __fentry__
ip = 0xffffffff81a14910
flags = 0
ip = 0xffffffff81a149b0
flags = 0
ip = 0xfffffff...

38©2019 VMware, Inc.
Finding __fentry__
ip = 0xffffffff81a14910
flags = 0
ip = 0xffffffff81a149b0
flags = 0
ip = 0xfffffff...

39©2019 VMware, Inc.
dyn_ftrace.flags
Bits 0-24: Counter for number of callbacks registered to function
Bit 25: Function i...

40©2019 VMware, Inc.
dyn_ftrace.flags
Bits 0-24: Counter for number of callbacks registered to function
Bit 25: Function i...

41©2019 VMware, Inc.
Finding __fentry__
<schedule>:
nop
[..]
<yield>:
nop
[..]
<preempt_schedule_common>:
nop
[..]
<_cond_...

42©2019 VMware, Inc.
Finding __fentry__
vmlinux: ip = 0xffffffff81a14910
flags = 0x40000001
ip = 0xffffffff81a149b0
flags ...

43©2019 VMware, Inc.
Finding __fentry__
<schedule>:
call ftrace_regs_caller
[..]
<yield>:
nop
[..]
<preempt_schedule_commo...

44©2019 VMware, Inc.
Modifying code at runtime!
Not the same as at boot up
SMP boxes need to take extra care
Other CPUs ma...

45©2019 VMware, Inc.
Modifying code at runtime!
<schedule>:
0f 1f 44 00 00 nop
53 push %rbx
65 48 8b 1c 25 00 61 mov %gs:0...

46©2019 VMware, Inc.
Modifying code at runtime!
<schedule>:
e8 1b d0 1e 00 callq ffffffff81c01930 <__fentry__>
53 push %rb...

47©2019 VMware, Inc.
Modifying code at runtime!
<schedule>:
0f 1f 44 00 00
53
65 48 8b 1c 25 00 61
01 00
48 8b 43 10
48 85...

48©2019 VMware, Inc.
Modifying code at runtime!
<schedule>:
0f 1f 44 00 00
53
65 48 8b 1c 25 00 61
01 00
48 8b 43 10
48 85...

49©2019 VMware, Inc.
Modifying code at runtime!
<schedule>:
0f 1f d0 1e 00
53
65 48 8b 1c 25 00 61
01 00
48 8b 43 10
48 85...

50©2019 VMware, Inc.
0f 1f d0 1e 00 ???
0f 1f d0 1e 00

51©2019 VMware, Inc.
0f 1f d0 1e 00 ???
BOOM!
CRASH!
General Protection Fault!
REBOOT!

52©2019 VMware, Inc.
How to go from this!
<schedule>:
0f 1f 44 00 00 nop
53 push %rbx
65 48 8b 1c 25 00 61 mov %gs:0x16100...

53©2019 VMware, Inc.
To this?
<schedule>:
e8 1b d0 1e 00 callq ffffffff81c01930 <__fentry__>
53 push %rbx
65 48 8b 1c 25 0...

54©2019 VMware, Inc.
Breakpoints!

55©2019 VMware, Inc.
Breakpoints!
<schedule>:
0f 1f 44 00 00 nop
53 push %rbx
65 48 8b 1c 25 00 61 mov %gs:0x16100,%rbx
01...

56©2019 VMware, Inc.
Breakpoints!
<schedule>:
<cc> 1f 44 00 00 <int3>nop
53 push %rbx
65 48 8b 1c 25 00 61 mov %gs:0x16100...

57©2019 VMware, Inc.
How this works
<schedule>:
<int3>nop
push %rbx
mov %gs:0x16100,%rbx
mov 0x10(%rbx),%rax
test %rax,%rax

58©2019 VMware, Inc.
How this works
<schedule>:
<int3>nop
push %rbx
mov %gs:0x16100,%rbx
mov 0x10(%rbx),%rax
test %rax,%ra...

59©2019 VMware, Inc.
How this works
<schedule>:
<int3>nop
push %rbx
mov %gs:0x16100,%rbx
mov 0x10(%rbx),%rax
test %rax,%ra...

60©2019 VMware, Inc.
How this works
<schedule>:
<int3>nop
push %rbx
mov %gs:0x16100,%rbx
mov 0x10(%rbx),%rax
test %rax,%ra...

61©2019 VMware, Inc.
How this works
<schedule>:
<int3>nop
push %rbx
mov %gs:0x16100,%rbx
mov 0x10(%rbx),%rax
test %rax,%ra...

62©2019 VMware, Inc.
Breakpoints!
<schedule>:
<cc> 1f 44 00 00 <int3>nop
53 push %rbx
65 48 8b 1c 25 00 61 mov %gs:0x16100...

63©2019 VMware, Inc.
Breakpoints!
<schedule>:
<cc>1b d0 1e 00 <int3>callq ffffffff81c01930 <__fentry__>
53 push %rbx
65 48...

64©2019 VMware, Inc.
Breakpoints!
<schedule>:
e8 1b d0 1e 00 callq ffffffff81c01930 <__fentry__>
53 push %rbx
65 48 8b 1c ...

65©2019 VMware, Inc.
Registering a callback with ftrace
Call register_ftrace_function()
Takes a ftrace_ops descriptor
Stat...

$66©2019 VMware, Inc. ftrace_ops structure struct ftrace_ops { ftrace_func_t func; struct ftrace_ops __rcu *next; unsigned ...$

67©2019 VMware, Inc.
ftrace_caller trampoline
<schedule>:
callq ftrace_caller
[..]
<yield>:
nop
[..]
<preempt_schedule_com...

68©2019 VMware, Inc.
ftrace_caller trampoline
<schedule>:
callq ftrace_caller
[..]
<yield>:
nop
[..]
<preempt_schedule_com...

69©2019 VMware, Inc.
ftrace_caller trampoline
<schedule>:
callq ftrace_caller
[..]
<yield>:
nop
[..]
<preempt_schedule_com...

70©2019 VMware, Inc.
Calling more that one callback on a function?
Direct calls to a single function are easy
Handling mor...

71©2019 VMware, Inc.
ftrace_caller trampoline
<schedule>:
callq ftrace_caller
[..]
<yield>:
nop
[..]
<preempt_schedule_com...

72©2019 VMware, Inc.
Multiple function callback example
Run function tracer on all functions
Run perf on just the scheduler

73©2019 VMware, Inc.
Multiple function callback example
Want to trace
schedule_idle()?
NO
list_func()
perf
Yes!
function t...

74©2019 VMware, Inc.
Multiple function callback example
Want to trace
__cond_resched()?
NO
list_func()
perf
Yes!
function ...

75©2019 VMware, Inc.
Multiple function callback example
Want to trace
yield()?
NO
list_func()
perf
Yes!
function tracer

76©2019 VMware, Inc.
Multiple function callback example
Want to trace
schedule()?
Yes!
list_func()
perf
Yes!
function trac...

77©2019 VMware, Inc.
ftrace_caller trampoline
<schedule>:
callq ftrace_caller
[..]
<yield>:
callq ftrace_caller
[..]
<pree...

78©2019 VMware, Inc.
ftrace_caller trampoline
<schedule>:
callq ftrace_caller
[..]
<yield>:
callq dynamic_trampoline
[..]
...

79©2019 VMware, Inc.
Problems with dynamic trampolines
When can you free them?
How do you know they are still not in use?

80©2019 VMware, Inc.
Dynamic Trampoline Problem
<schedule>:
callq dynamic_trampoline
push %rbx
mov %gs:0x16100,%rbx
vmlinu...

81©2019 VMware, Inc.
Dynamic Trampoline Problem
<schedule>:
callq dynamic_trampoline
push %rbx
mov %gs:0x16100,%rbx
vmlinu...

82©2019 VMware, Inc.
Dynamic Trampoline Problem
<schedule>:
callq dynamic_trampoline
push %rbx
mov %gs:0x16100,%rbx
vmlinu...

83©2019 VMware, Inc.
Dynamic Trampoline Problem
<schedule>:
callq dynamic_trampoline
push %rbx
mov %gs:0x16100,%rbx
vmlinu...

84©2019 VMware, Inc.
Dynamic Trampoline Problem
<schedule>:
nop
push %rbx
mov %gs:0x16100,%rbx
vmlinux:
<dynamic_trampolin...

85©2019 VMware, Inc.
Dynamic Trampoline Problem
<schedule>:
nop
push %rbx
mov %gs:0x16100,%rbx
vmlinux:
<dynamic_trampolin...

86©2019 VMware, Inc.
Dynamic Trampoline Problem
<schedule>:
nop
push %rbx
mov %gs:0x16100,%rbx
vmlinux:
<dynamic_trampolin...

87©2019 VMware, Inc.
Dynamic Trampoline Problem
<schedule>:
nop
push %rbx
mov %gs:0x16100,%rbx
vmlinux:
<dynamic_trampolin...

88©2019 VMware, Inc.
Dynamic Trampoline Problem
<schedule>:
nop
push %rbx
mov %gs:0x16100,%rbx
vmlinux:
<dynamic_trampolin...

89©2019 VMware, Inc.
Problems with dynamic trampolines
When can you free them?
How do you know they are still not in use?

90©2019 VMware, Inc.
Problems with dynamic trampolines
When can you free them?
How do you know they are still not in use?
...

91©2019 VMware, Inc.
call_rcu_tasks()
Added in Linux v3.18

Commit 8315f42295d2667 by Paul E. McKenney
synchronize_rcu_ta...

92©2019 VMware, Inc.
call_rcu_tasks()
Added in Linux v3.18

Commit 8315f42295d2667 by Paul E. McKenney
synchronize_rcu_ta...

93©2019 VMware, Inc.
call_rcu_tasks()
Added in Linux v3.18

Commit 8315f42295d2667 by Paul E. McKenney
synchronize_rcu_ta...

94©2019 VMware, Inc.
Dynamic Trampoline Solution
<schedule>:
nop
push %rbx
mov %gs:0x16100,%rbx
vmlinux:
<dynamic_trampoli...

95©2019 VMware, Inc.
Dynamic Trampoline Solution
<schedule>:
nop
push %rbx
mov %gs:0x16100,%rbx
vmlinux:
<dynamic_trampoli...

96©2019 VMware, Inc.
Dynamic Trampoline Solution
<schedule>:
nop
push %rbx
mov %gs:0x16100,%rbx
vmlinux:
<dynamic_trampoli...

97©2019 VMware, Inc.
Dynamic Trampoline Solution
<schedule>:
nop
push %rbx
mov %gs:0x16100,%rbx
vmlinux:
<dynamic_trampoli...

98©2019 VMware, Inc.
Dynamic Trampoline Solution
<schedule>:
nop
push %rbx
mov %gs:0x16100,%rbx
vmlinux:
<dynamic_trampoli...

99©2019 VMware, Inc.
Dynamic Trampoline Solution
<schedule>:
nop
push %rbx
mov %gs:0x16100,%rbx
vmlinux:
<dynamic_trampoli...

100©2019 VMware, Inc.
Dynamic Trampoline Solution
<schedule>:
nop
push %rbx
mov %gs:0x16100,%rbx
vmlinux:
<dynamic_trampol...

101©2019 VMware, Inc.
Dynamic Trampoline Solution
<schedule>:
nop
push %rbx
mov %gs:0x16100,%rbx
vmlinux:
<dynamic_trampol...

102©2019 VMware, Inc.
More uses of the function callback code
ftrace_regs_caller() gives all registers
A callback can modi...

103©2019 VMware, Inc.
Live Kernel Patching!
<schedule>:
callq ftrace_caller
[..]
Buggy schedule() function
<ftrace_caller>...

Thank You

Top clipped slide