This presentation shows the approach we use to diagnostic performance issues on the customer. We explain the USE method and how you can relate it with pt-stalk

1. A Support Engineer
Walkthrough on pt-stalk
Marcos Albe
Principal Support Engineer - Percona
Marcelo Altmann
Senior Support Engineer - Percona

2. 2
Thank You Sponsors!

3. 3
SAVE THE DATE!
CALL FOR PAPERS OPENING SOON!
www.perconalive.com
April 23-25, 2018
Santa Clara Convention Center

4. Who is speaking ?

5. 5
Who is Speaking ?
• Marcelo Altmann
- Senior Support Engineer @ Percona
- Resident specialist on Surf & Brazil leisure
•Marcos Albé
- Principal Support Engineer @ Percona
- Chief Barbeque Officer & Designated Beer Analyst

6. Agenda

7. 7
Agenda
• What do we USE
• Who’s stalkin’?
• How pt-stalk can be USEd
• Walkthrough

8. What do we USE

9. 9
We use one
of these...

10. 10
Utilization

11. 11
Utilization
Saturation

12. 12
Utilization
Saturation
Errors

13. 13

14. Who’s stalkin’?

15. 15
check out those
resources!

16. 16
Who’s stalkin’?
•Aspersa stalk + Aspersa collect in one! (thanks Daniel & Baron)
•# The name 'stalk' is because 'watch' is already taken, and 'stalk' is fun.
•Now part of pt-toolkit
•Monitors variable or a script’s single-valued output
•Triggers capture when set threshold is reached
- files from /proc: diskstats, interrupts, meminfo, slabinfo, procstat, procvmstat.
- vmstat, iostat, top, netstat, pmap, mpstat, sysctl, df, hostname, mysqladmin ext, mysqladmin debug, lsof).
- SHOW ENGINE INNODB STATUS, SHOW ENGINE INNODB MUTEX, SHOW GLOBAL VARIABLES, SHOW FULL
PROCESSLIST and SHOW OPEN TABLES.
•Production safe!
•Custom triggers and custom captures
•Companion tools (pt-sift, pt-mext, pt-diskstats, pt-summary); Bash dissection

17. 17
Who’s stalkin’: common parameters
•Stalking part
- --daemonize | --no-stalk
- --function=status
- --interval=1
- --variable=Threads_running
- --threshold=25
- --cycles=5
•Collecting part
- --run-time=30
- --sleep=300
- --iterations=
- --plugin=
- --dest=/var/lib/pt-stalk
• Values shown are the defaults for each (where given)

18. 18
Who’s stalkin’: work flow

19. 19
Who’s stalkin’: working examples I
Collect data during 60 seconds for 3 consecutive times, starting
immediately.
pt-stalk --no-stalk
--dest=$HOME/percona
--user=root
--ask-pass
--run-time=60
--sleep=60
--iterations=3

20. 20
Who’s stalkin’: working examples II
Run as a daemon and trigger collection if InnoDB has more than 5
pending fsyncs.
pt-stalk --daemonize
--function=status
--variable=Innodb_data_pending_fsyncs
--threshold=5
--cycles=2
--interval=1
--iterations=5
--plugin=/usr/local/bin/raid_card_status.sh

21. 21
Who’s stalkin’: custom triggers
#!/bin/bash
# Save this bash snippet as /root/delayed_slave_trg.sh;
trg_plugin() {
mysql $EXT_ARGV -Ee "SHOW SLAVE STATUS" |
grep "Seconds_Behind_Master" |
awk "{ print $2 }";
}
# use like
pt-stalk --function=/root/delayed_slave_trg.sh --etc --etc

22. 22
Who’s stalkin’: custom capture
#!/bin/bash
# save as /root/collect_perf.sh
before_collect() {
# run perf stat; match sleep length to --run-time
/usr/bin/perf stat --all-cpus
--output=${d}/${p}-stat_a
-- sleep ${OPT_RUN_TIME};
}
# use like
pt-stalk --plugin=/root/collect_perf.sh --etc --etc

23. How pt-stalk can be USE’d

24. 24
How pt-stalk can be USE’d
LinuxKernel
CPU
CPU
Interconnect
DRAM
I/O Bridge
I/O Controller Network Controller
I/O Bus
Expander Interconnect
Disk Disk Swap Port Port
Applications
System libraries
System Call Interface
VFS Sockets
Scheduler
File Systems TCP/UDP
Volume Manager IP
Virtual
Memory
Block Device Interface Ethernet
Device Drivers
Memory Bus
Interface Transports

25. 25
LinuxKernel
CPU
CPU
Interconnect
DRAM
I/O Bridge
I/O Controller Network Controller
I/O Bus
Expander Interconnect
Disk Disk Swap Port Port
Applications (MySQL!)
System libraries
System Call Interface
VFS Sockets
Scheduler
File Systems TCP/UDP
Volume Manager IP
Virtual
Memory
Block Device Interface Ethernet
Device Drivers
Memory Bus
Interface Transports
ps
df
/proc/diskstats
SHOW ENGINE INNODB STATUS
/proc/interrupts
iostat
lsof /proc/meminfo
mpstat
SHOW ENGINE INNODB MUTEX
mysqladmin ext
SHOW OPEN TABLES
pmap
/proc/procstat
SHOW FULL PROCESSLIST
/proc/vmstat
/proc/slabinfo
sysctl
SHOW ENGINE TOKUDB STATUS
top
SHOW GLOBAL VARIABLES
vmstat
SHOW SLAVE STATUS
netstat
strace
tcpdump
gdb (backtraces)
oprofile

26. 26
strace tcpdump
gdb (backtraces)
oprofile
How pt-stalk can be USE’d: Here Be Dragons!

27. A walkthrough

28. 28
A walkthrough
Using pt-stalk to solve problems with...
●...High IO
●...High CPU
●...Networking

29. 29
Useful links
●grep commands -
https://gist.github.com/altmannmarcelo/eec40a54568299c2cc11b6cf585
9203d
●Check List - http://www.brendangregg.com/USEmethod/use-linux.html

30. Questions ?
Marcos Albe
marcos.albe@percona.com
Marcelo Altmann
@altmannmarcelo
marcelo.altmann@percona.com