Over the past few years the Linux kernel has gained features that allow us to learn more about what's really happening on our servers and the applications that run on them. This talk will explore how these new features, particularly perf_events and ebpf, enable us to answer questions about what a Drupal site is doing in real time beyond what the standard logs, server performance tools, and even strace will reveal. Attendees will be provided a brief introduction to example uses of these tools to diagnose performance problems. This talk is intended for attendees that are familiar with Linux, the command line, and have used host observability tools in the past (top, netstat, etc).

1. Linux Server Deep Dives
Amin Astaneh
Drupalcon Amsterdam 2019

2. Who Am I?
● Senior Manager, SRE, Acquia
● Acquian since December 2010
● Champion DevOps, SRE,
operational, agile best practices

3. WARNING: This is NOT Your Usual Linux Talk
We won’t be talking about the usual suspects:
● top, ps, uptime
● sar, mpstat, iostat
● not even strace or lsof
● (well, maybe a little strace)
Past talk: https://bit.ly/2BNzNy5

4. Goal: An Introduction to Advanced Tooling
● Tools: perf_events, ebpf
● Origins and capabilities
● How to install these tools
● Demo of examples that you can use today
Aim is to provide inspiration on simple yet powerful ways to troubleshoot Drupal
from the infrastructure and performance side.
The classic tools answer what resources are being used.
These tools answer how resources are being used in much greater detail.

5. Before We Begin: Tool Caveats
1) These tools can introduce a performance overhead.
Keep that in mind when deciding to analyse your production workloads. Run
in non-production where possible.
2) Some tools require you to rebuild your services in order to use them.
Eg: mysqld, php, etc
3) Some tools require you to install debug packages to be useful.
4) These tools require root access.

6. Before We Begin: The Environment For This Talk
● Ubuntu 18.04 VM
● Drupal 8.7.8 installed running the Umami demo site
● Modest resources (1 core, 1GB RAM, 10GB HDD)
● No fancy caching like Varnish or Memcached

7. Before We Begin: Some Operating System Basics
Let’s talk about system calls (aka: syscalls).
It’s how programs interact with the kernel (in this case, Linux) to perform tasks,
such as:
● read or write to a file
● database calls, memcached, HTTP
● executing other programs
If you want the full list, run `man 2 syscalls`.
If you want to read about a specific one, run `man 2 name_of_syscall`.

9. The New Tools

10. perf_events
● It’s been around since 2009
● Part of the linux kernel since 2.6.31
● Originally called Performance Counters for Linux
● Enables capture of analysis of broad performance-related kernel events
● Not very well documented :(
● To install: linux-tools package

11. The Extended Berkeley Packet Filter (eBPF)
The Berkeley Packet Filter was originally simply that: a packet filter.
However, there are certain characteristics of the project as it evolved since 2014
that expanded upon its originally-intended usage:
● Filters were implemented as programs that ran in a kernel-mode VM;
● “BPF guarantees that the programs loaded into the kernel cannot crash, and
cannot run forever”
● eBPF programs can access in-kernel debugging features such as kprobes

12. What Does This Mean For eBPF?
You can use eBPF for in-depth performance analysis of a running server, not
just its network stack.
The toolkit provided by the BPF compiler collection (BCC) provides us an
accessible wealth of observability tools.
It also provides the means to write your own tools.

13. Installing BCC
● Ubuntu: sudo apt-get install bpfcc-tools linux-headers-$(uname -r)
● RHEL: yum install bcc-tools
● Amazon Linux: yum install bcc

14. Examples With Demos

15. `perf` tool
Allows you to monitor for specific OS events to trace/analyse
● counters- number of occurances that something happens
● tracing- real time tracking of events (usually syscalls)
● probing- monitor and capture specific events on the server
● reporting- analyse captured data

16. `perf stat`
Example of counters:
perf stat -e 'syscalls:sys_enter_*' COMMAND
(lists the count of system calls for COMMAND)
Example: let’s see what a `drush status` does to our system:
sudo perf stat -e 'syscalls:sys_enter_*' drush status 2>&1 | grep -v ' 0 '
Why does this matter? A certain module or feature may be badly performing,
and now you can find out why.

17. `perf trace`
● Say hello to a more performant replacement to strace!
● System call tracers print what is happening in real time
● Tracing PHP processes can be really useful for troubleshooting performance
problems quickly when you don’t have an APM installed
● perf trace has less overhead than strace, by a LOT

18. `perf trace` overhead
● How do we test that?
● Using dd, we can see that perf trace has a 2.5x slowdown
● Strace had a 62x showdown.
# dd if=/dev/zero of=/dev/null bs=512 count=10000k
5242880000 bytes (5.2 GB) copied, 3.53031 s, 1.5 GB/s
# perf stat -e 'syscalls:sys_enter_*' dd if=/dev/zero of=/dev/null bs=512 count=10000k
5242880000 bytes (5.2 GB) copied, 9.14225 s, 573 MB/s
# strace -c dd if=/dev/zero of=/dev/null bs=512 count=10000k
5242880000 bytes (5.2 GB) copied, 218.915 s, 23.9 MB/s

19. perf trace
You can see all syscalls on the system with `perf trace`!
For a single process, run `perf trace -p <PID>` or `perf trace <COMMAND>`

20. `perf record`
● You can sample all CPU activity on the system:
● perf record -a -F 1000 sleep 10
○ Record activity on all processors, 1000 times per second, for 10 seconds
● Then you can generate a report on the output data
● perf report
Note that you need to install debug packages in order to drill down into specific
library calls! (PKG-dbg, or PKG-dbgsym)

21. `perf top`
● Like the top command, but for kernel-level events
● Plain `perf top` will tell you what userspace and kernel functions are using the
most resources
● What is generating network traffic on the server?
● perf top -e net:net_dev_xmit -ns comm,pid

22. Dynamic Tracing with `perf probe`
This allows you to monitor for invokation of specific kernel functions.
● create a probe: perf probe --add <FUNCTION>
● record probe behavior: perf record -e probe:<FUNCTION> -aR sleep 1
● list probes: perf probe -l
● delete probes: perf probe -d <EVENT>
You probably won’t use these when getting started, but know that this exists.

23. Trace HTTP Outbound Connections in Real Time
Use tcpconnect to detect external calls performed by Drupal,
cronjobs, etc. Could also help in detecting intruders!
# tcpconnect
PID COMM IP SADDR DADDR DPORT
1957 php-fpm 4 192.168.122.229 143.204.214.36 80

24. Trace HTTP Requests in Real Time
Use tcptracer to detect all TCP connections on your server.
A very easy way to find abusive or high-throughput HTTP
clients as they happen!

25. How long do your HTTP client connections last?
tcplife prints out the latency and data transfers for each
connection, which again can be useful for analysing what
your clients are doing.

26. Trace File Accesses On Web Server
Use statsnoop to detect all file information accesses on
your server (stat family of syscalls)
# statsnoop | grep sites/default/files | egrep 'jpg|png|pdf|mp4'
PID COMM IP SADDR DADDR DPORT
1957 php-fpm 4 192.168.122.229 143.204.214.36 80

27. Monitor file reads and writes!
Use filetop to find how which specific files are getting the
most activity!

28. How large are your per-process I/O operations?
bitesize prints histograms of storage I/O operations for
each process. May be useful to find programs that are doing
excessive or inefficient operations.

29. How Long Does it Take For Filesystem Operations?
ext4dist, xfsdist, zfsdist, etc will generate histograms of
how long it takes to perform reads and write operations on
the filesystem.
This really breaks down the performance characteristics of
the filesystem beyond what iostat will tell you.

30. Find out if you need more memory!
More operating system theory:
A ‘page fault’ means that a access to data required reading
from the disk rather than what was in the page cache (stored
in RAM).
This is particularly important on servers expected to serve
a lot of file data, eg: a file server. Too little memory for
page cache affects performance.
This is in a way similar to nginx or varnish miss rates.
The cachestat tool enables you to monitor for this
condition. The dcstat tool is useful for directory cache.

31. Trace Creation of New Processes
Use pidpersec to determine the rate of new process creation.
High values may be revealing that something is wrong with
custom code such as cronjobs or scripts on the server.

32. Trace Creation of New Processes
Use execsnoop to detect all new processes on your server.
Quite useful for following up after use of pidpersec.
# execsnoop-bpfcc
PCOMM PID PPID RET ARGS
date 2647 2499 0 /bin/date
sleep 2648 2499 0 /bin/sleep 1
date 2649 2499 0 /bin/date
sleep 2650 2499 0 /bin/sleep 1

33. Spy On a User Session!
Use ttysnoop to watch another person’s shell session!
# to find the ttys in use
ps auxww --forest | egrep --color ‘^|pts’
# then to trace
ttysnoop /dev/pts/X

34. Spy On All User Sessions!
Similarly, you can use bashreadline to see all programs that
have been invoked from a bash shell. Useful for analysing
how jump hosts are being used.

35. Spy On SSL/TLS Connections!
sslsniff will print the data being written to and read from SSL_write() and
SSL_read() functions, basically intercepting encrypted traffic on the server!

36. In Summary
● perf_events and eBPF are pretty awesome additions to your toolkit
● You can see more details on Linux server activity than ever before
● You can start using these tools today :D
● Test in non-production first
● Have fun!

37. Further Reading

38. Further Reading
● Buy Brendan Gregg’s book on eBPF
○ http://www.brendangregg.com/bpf-performance-tools-book.html
● Perf Events Reference
○ https://perf.wiki.kernel.org/index.php/Main_Page
● bcc Github Project
○ https://github.com/iovisor/bcc
● Julia Evans’ Perf Cheatsheet
○ https://jvns.ca/perf-cheat-sheet.pdf
● Linux Syscall References
○ `man 2 syscalls`
○ `man 2 <SYSCALL>`

39. Thank You!
Amin Astaneh
Twitter: @aastaneh
Email: amin@aminastaneh.net

40. Join us for
contribution opportunities
Mentored
Contribution
First Time
Contributor Workshop
General
Contribution
#DrupalContributions

41. What did you think?
https://drupal.kuoni-congress.info/2019/program/
https://www.surveymonkey.com/r/DrupalConAmsterdam