The document discusses using eBPF for instrumentation and logging in Golang applications without source code modifications. It demonstrates attaching a BPF program to a function using uprobes to log function arguments each time the function is called. This allows tracing application flow and variables without recompilation like traditional debuggers. eBPF provides low overhead dynamic tracing of production applications.

1. GoSF
Nov 11, 2020
No-Instrumentation Golang
Logging with eBPF

2. ABOUT ME
👋 i’m Zain
@zainasgar
Co-Founder/CEO Pixie Labs &
Adjunct Professor of CS @ Stanford

3. DEVELOPER PROBLEM
You’re an application developer, and your program is misbehaving.
● No problem. You have logs! Right?
● Uh-oh, not in the spot you need. 😞
We’ve all been there:
● “I just wish I could see the variable x when Foo() is called”

4. - Delve is one of the main Go debuggers.
- You can also use GDB, but it has less support for Go runtime, data structures, etc.
- Install Delve to get started
go get github.com/go-delve/delve/cmd/dlv
First steps with a debugger
DEBUGGERS

5. Github Link: https://github.com/pixie-labs/pixie/tree/main/demos/simple-gotracing
Relevant Code:
// computeE computes the approximation of e by running a
fixed number of iterations.
func computeE(iterations int64) float64 {
res := 2.0
fact := 1.0
for i := int64(2); i < iterations; i++ {
fact *= float64(i)
res += 1 / fact
}
return res
}
Let’s look at test application
Use Case
GET /e?iters={iterations}

6. // computeE computes the approximation of e by running a
fixed number of iterations.
func computeE(iterations int64) float64 {
res := 2.0
fact := 1.0
for i := int64(2); i < iterations; i++ {
fact *= float64(i)
res += 1 / fact
}
return res
}
What if we just want to log the iterations?
Use Case
fmt.Printf("iterations: %dn”, iterations)

7. Running and using the debugger
% dlv exec app
Type 'help' for list of commands.
(dlv) b app.go:15
Breakpoint 1 set at 0x125ef40 for
main.computeE() ./app.go:15
(dlv) c
Starting server on: :9090
> main.computeE() ./app.go:15 (hits goroutine(34):1 total:1)
(PC: 0x125ef40)
Warning: debugging optimized function
...
14: for i := int64(2); i < iterations; i++ {
=> 15: fact *= float64(i)
16: res += 1 / fact
17: }
...
(dlv) print iterations
100000
(dlv) print i
2
Delve

8. (dlv) bt
0 0x000000000125ef40 in main.computeE
at ./app.go:15
1 0x000000000125f2b1 in main.main.func1
at ./app.go:39
2 0x0000000001232f24 in
net/http.HandlerFunc.ServeHTTP
at /opt/golang/src/net/http/server.go:2007
3 0x0000000001234dfd in
net/http.(*ServeMux).ServeHTTP
at /opt/golang/src/net/http/server.go:2387
4 0x0000000001235d74 in
net/http.serverHandler.ServeHTTP
at /opt/golang/src/net/http/server.go:2802
5 0x0000000001231f55 in net/http.(*conn).serve
at /opt/golang/src/net/http/server.go:1890
6 0x000000000105ab11 in runtime.goexit
at /opt/golang/src/runtime/asm_amd64.s:1357
(dlv) grs
Goroutine 1 - User:
/opt/golang/src/runtime/netpoll.go:184
internal/poll.runtime_pollWait (0x102aa45)
Goroutine 2 - User: /opt/golang/src/runtime/proc.go:305
runtime.gopark (0x1030360)
Goroutine 3 - User: /opt/golang/src/runtime/proc.go:305
runtime.gopark (0x1030360)
Goroutine 4 - User: /opt/golang/src/runtime/proc.go:305
runtime.gopark (0x1030360)
Goroutine 18 - User: /opt/golang/src/runtime/proc.go:305
runtime.gopark (0x1030360)
* Goroutine 34 - User: ./app.go:15 main.computeE
(0x125ef40) (thread 36175033)
Goroutine 35 - User:
/opt/golang/src/runtime/netpoll.go:184
internal/poll.runtime_pollWait (0x102aa45)
[7 goroutines]
Stack Goroutines
GETTING CONTEXT

9. RECAP ON DEBUGGERS
● Debuggers provide and easy way to get information about executing code
● No need to add fmt.Print and recompile.
○ Can easily step through code and get information.
● Dlv and GDB can connect to remote binaries to help debug machines running
on servers.
○ Unfortunately this turns out to be harder than expected when running in
environments like Kubernetes .
○ Checkout Solo.io, etc.

10. CAN WE RUN THIS SAFELY ON PRODUCTION CODE?
● Debuggers can be attached to production binaries by using a remote
attachment.
● Unfortunately, they can mutate, stop execution of the code and lead to
unexpected failures.
○ You probably don’t want this on prod
● What else can we do?

11. YOUR OPTIONS
Option 1: Add a log to your program, re-compile and re-deploy.
○ This can be simple log statements, or
○ More comprehensive like Open tracing.
Option 2: Debugger
○ GDB
○ Delve
Option 3: Linux tracing utility
○ strace/ftrace
○ LTTng/USDT
Option 4: eBPF 🤔

12. What is
Credit: https://ebpf.io/
eBPF

13. What are we going to build?
eBPF

14. [0] % objdump --syms app|grep computeE
00000000006609a0 g F .text 000000000000004b main.computeE
[0] % objdump -d app | less
00000000006609a0 <main.computeE>:
6609a0: 48 8b 44 24 08 mov 0x8(%rsp),%rax
6609a5: b9 02 00 00 00 mov $0x2,%ecx
6609aa: f2 0f 10 05 16 a6 0f movsd 0xfa616(%rip),%xmm0 #
75afc8 <$f64.3ff0000000000000>
6609b1: 00
6609b2: f2 0f 10 0d 36 a6 0f movsd 0xfa636(%rip),%xmm1 #
75aff0 <$f64.4000000000000000>
Diving into the details
eBPF

15. Using uprobes
eBPF
...
...
main.computeE
0x6609a0
{
App Binary
main.main
{
0x6609f0
...
...
main.computeE
Uprobe Hook
{
App Binary
main.main
{
0x6609f0
BPF
Program
Perf
Buffer Tracer
Binary

16. What does the BPF program look like?
● The function is simply invoked
whenever main.computeE is called.
● The registration is done via UProbes
● It attaches to every running version
of the binary
#include <uapi/linux/ptrace.h>
BPF_PERF_OUTPUT(trace);
inline int computeECalled(struct pt_regs *ctx)
{
// The input argument is stored in ax.
long val = ctx->ax;
trace.perf_submit(ctx, &val, sizeof(val));
return 0;
}
eBPF
Github Link: https://github.com/pixie-labs/pixie/tree/main/demos/simple-gotracing/trace_example

17. DEMO: eBPF

18. ● Utilizing tracepoints for dynamic logging allows for easy
instrumentation of production binaries
● The complexities of the Go ABI make it difficult to do. Especially
when you consider: interfaces, channels, etc.
What’s next?
eBPF

19. DEMO: HTTP Tracer

20. eBPF
https://github.com/pixie-labs/pixie
https://github.com/kinvolk/inspektor-gadget
Some related projects
https://github.com/draios/sysdig/wiki/eBPF

21. APPENDIX

22. eBPF
Dynamic
Tracepoint
(live injection)
func Foo(a, b int) int {
c := a + 2;
if c > threshold {
return Bar(b)
}
return 0
}
fmt.Printf("%v, %v, %v", b, c, threshold)
- Function Argument Tracing → “What are the arguments being passed to Foo(x, y, z)?”
- Function Input-Output Tracing → “What are the arguments and return value of calls to Foo(x, y, z)?”
- Function Latency Profiling → “What is the latency (call to return) of calls to Login(username)?”

23. At, less than 2% CPU overhead and Latency degradation
Less than 2%
overhead at
0% sampling !!

24. ...
...
main.computeE
0x6609a0
{
App Binary
main.main
{
0x6609f0
...
...
main.computeE
Uprobe Hook
{
App Binary
main.main
{
0x6609f0
BPF
Program
Perf
Buffer Tracer
Binary

25. COMPARING TOOLS
GDB, Delve etc. USDT, LTTng OpenTracing eBPF
Traceable
Code
Instrumented
Application Code
All application
code
Instrumented
Application Code
All application
code
Application
Disruption
High Low Low Low
Distributed
Analysis
No No Multi-Node No
Performance
Impact
High Low Low/Med Low