Exploring the debugging and profiling tools for AWS Lambda. We use the v8-profiler for NodeJS with IOpipe to see deep into a serverless application and trace function calls. https://www.iopipe.com

1. ServerlessConf NYC’17
Profiling & debugging
AWS Lambda
Erica Windisch
CTO, Founder
IOpipe, Inc.

2. Erica Windisch
CTO & Founder IOpipe, Inc.
Former maintainer of Docker &
OpenStack Building cloud, ops, and
infrastructure tooling for over 15
years.

3. (Thanks & credit to Katie Bayes!)

4. Traditional Instrumentation

5. Kernel
Hardware & Virtualization
Application
Detail
Accuracy
❖ Step-debugging
❖ CPU Profiling
❖ Heap snapshots &
allocation profiles

6. Kernel
Hardware & Virtualization
Application
Detail
Accuracy
❖ Step-debugging
❖ CPU Profiling
❖ Heap snapshots &
allocation profiles

7. Virtualization - Valgrind
https://www.cprogramming.com/debugging/valgrind.html

8. Kernel
Hardware & Virtualization
Application
Detail
Accuracy
❖ Step-debugging
❖ CPU Profiling
❖ Heap snapshots &
allocation profiles

9. Kernel-level tooling

10. Linux perf tools
https://github.com/brendangregg/perf-tools

11. — Brendan Gregg

12. Almost everything you need to know
about kernel-level performance
tracing & beyond
http://www.brendangregg.com/

13. Kernel
Hardware & Virtualization
Application
Detail
Accuracy
❖ Step-debugging
❖ CPU Profiling
❖ Heap snapshots &
allocation profiles

14. ptrace (i.e. gdb, strace, et al)
https://sysdig.com/blog/sysdig-vs-dtrace-vs-strace-a-technical-discussion/

15. There’s an excessive
number of tools
(and they don’t work on Lambda)

16. Instrumenting Lambda

17. Things that don’t work:
Linux eBPF!
$ iosnoop
Tracing block I/O. Ctrl-C to end.
/usr/bin/iosnoop: line 172: cd: /sys/kernel/debug/tracing: Permission denied
ERROR: accessing tracing. Root user? Kernel has FTRACE?
debugfs mounted? (mount -t debugfs debugfs /sys/kernel/debug)

18. Things that don’t work:
Linux eBPF!
λ$ ls -l /sys
⨯ Error: function response: Command failed: ls -l /sys
ls: cannot access /sys: No such file or directory

19. Things that don’t work:
Linux perf

20. Things that don’t work:
Linux perf
λ$ cat /proc/sys/kernel/perf_event_paranoid
2

21. Kernel-level instrumentation
is out of reach

22. HW+Virt instrumentation
is (mostly) out of reach.

23. Application instrumentation
is hard(er) on Lambda

24. Lambda Limitations
✤ Event-loop is out of our direct control
✤ Process command-line flags inaccessible
✤ Ptrace disabled!
✤ Linux kernel perf features not available on Lambda.
✤ Local dev+test is an evolving story
✤ Many languages, requiring different implementations
✤ Debugger breakpoints in prod aren’t generally viable

25. Local Debugging
& Instrumentation

27. (Thanks Paul Maddox, Sanath Kumar Ramesh, & AWS!)

28. Debugging Lambda
in the cloud
github.com/trek10inc/
aws-lambda-debugger

29. Profiling Lambda
in the cloud
github.com/iopipe/
iopipe-plugin-profiler

30. https://github.com/node-inspector/v8-profiler

33. (Thanks Pam Selle!)

38. iopipe.com